Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
189 changes: 189 additions & 0 deletions docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,189 @@
# ADR-TELEMETRY-ERRORS-0030: Agent-Consumable Error Telemetry

Date: 2026-09-03

## Status

Proposed

Three parts of this decision are not expressible with the current emitters.
Rule 5 selects soft, hard, and sampled assertion modes, but
`assert(condition, message, context)` has no mode or taxonomy-tag input. Its
desktop reporter calls Sentry `captureMessage` directly, while the cloud
reporter emits `invariant_assert` without taxonomy tags and caps repeated and
distinct messages. Rule 4 configures a fingerprint at the emitter, but
`ReportErrorOptions` has no fingerprint field. This stays Proposed until
assertion mode, assertion taxonomy tags, and emitter fingerprints are supported.

## Context

Real-user state exposes failures that tests and pre-deploy checks cannot see:
invariants, bad user states, expected-but-missing events, unexpected catches,
and degraded paths. These signals need common tags so agents can query them by
alert family and Datadog RUM can route alerts without one monitor per error
slug.

The frontend already provides `reportError`, mandates it in `src/AGENTS.md`,
and has production callers using snake_case `errorType` slugs and product
`surface` values. `src/base/assert.ts` is used much less widely, while many
catch blocks remain. The gap is coverage and discipline, not another reporting
API. Counts are deliberately omitted because they change independently of this
decision.

The tradeoffs considered were bounded indexed tags versus diagnostic context,
default grouping versus explicit fingerprints, family-level alerting versus
per-slug alerts, and failure counts versus rates with denominators. Those
tradeoffs lead to the low-cardinality tags and alternatives recorded below.

[ADR-TELEMETRY-DIAGNOSTICS-0019](TELEMETRY-DIAGNOSTICS-0019-recoverable-event-diagnostics.md)
proposes one diagnostics system and defers its concrete API while requiring
existing `reportError` infrastructure to be reused or evolved. This ADR narrows
that direction for error telemetry by defining tags for `reportError` and
`assert`; it does not add a second reporting path.

## Decision

1. **Keep `reportError` and `assert` as the only emitters.** Do not add another
abstraction. Widen coverage by converting swallowed or unexpected catches
to `reportError`, adding soft assertions at invariant boundaries, and
emitting `missing_event` observations where expected completion has a
bounded window.
2. **Use a fixed snake_case tag taxonomy:**
- `error_type`: written by `reportError` from `errorType`; a stable slug
shaped as `<category>_<operation>_<subject>[_detail]`, with the operation
written as a present participle
- `surface`: written by `reportError` from the required product owner in
`ReportErrorOptions`
- `failure_kind`: `invariant`, `bad_state`, `missing_event`,
`caught_unexpected`, or `degraded`
- `feature_area`: `workflow`, `queue`, `canvas`, `nodes`, `auth`, `cloud`,
`agent`, `crdt`, `billing`, `extensions`, `settings`, or `assets`
- `operation`: `load`, `save`, `execute`, `sync`, `import`, `export`,
`render`, `navigate`, `auth`, or `insert`
- `outcome`: `failed`, `recovered`, `aborted`, `timed_out`, `missing`, or
`degraded`
Comment on lines +63 to +64

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Question: should outcome:degraded exist beside aborted and recovered? Its registry reason says the operation was dropped safely, which reads like aborted. Production now writes recovered catches as caught_unexpected/recovered, caught_unexpected/degraded, and degraded/degraded. Please define each value or map rule 6's catch classes to tag pairs so family monitors can query one spelling.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 67aed68: rule 2 now defines each outcome value and rule 6 maps catch classes to failure_kind.

- `assert_mode`: `soft`, `hard`, or `sampled` for assertions
Comment thread
christian-byrne marked this conversation as resolved.
Resolved

`surface` identifies who owns the response; `feature_area` identifies the
failing product area used by alert families. The outcomes mean:
- `failed`: the requested work did not complete and no fallback ran.
- `recovered`: the requested work completed after recovery.
- `aborted`: the requested work stopped safely without applying its result.
- `timed_out`: a bounded wait expired.
- `missing`: an expected event did not arrive.
- `degraded`: work continued with reduced data or capability.

A conforming event always supplies `failure_kind`, `feature_area`, and
`outcome`. It supplies `operation` when the failure occurs during one of the
enumerated operations; invariants and bad states with no discrete operation
omit it. An assertion event is an `invariant` or `bad_state` check reported
with assertion semantics, including a semantic soft assertion emitted
through `reportError` instead of `assert()`. Assertion events supply
`assert_mode`; non-assertion events omit it. Existing taxonomy-tagged
emitters with other values or with `assert_mode` on non-assertion events do
not conform; reconcile them before acceptance. `ReportErrorOptions.tags`
remains optional for existing callers, but an event does not conform to
this taxonomy unless its tags satisfy this record shape.

Rule 2's lists are authoritative. After acceptance, add a value there and
append its audit record under Registry Amendments. Dynamic values such as
ids, names, and paths are forbidden in tags; permitted diagnostic values
belong in `context`. Other bounded snake_case tags may accompany these keys,
but monitors must not require them.

3. **Choose levels by impact.** Use `error` for `invariant`, `bad_state`,
`caught_unexpected`, and `outcome:failed`. Use `warning` for
`failure_kind:degraded`, `outcome:degraded`, `outcome:recovered`, and
`missing_event` unless the user lost work. When multiple rules match,
`error` takes precedence over `warning`.
4. **Retain default grouping unless a slug must subdivide an issue.** Sentry's
default grouping ignores tags, including `error_type`. When separate issues
are needed per slug, configure an emitter fingerprint containing both
`{{ default }}` and `errorType`. Never use a bare `[errorType]` fingerprint,
and do not add taxonomy fingerprints globally in `beforeSend`, so each
override stays reviewable at its call site.
5. **Use soft assertions by default in the browser.** Hard assertions are only
for cases where continuing corrupts document or CRDT state, authentication,
or billing. Sampled assertions are only for render and pointer hot paths and
never for data-loss invariants. Assertion messages are static strings.
6. **Classify every touched catch.** A catch is expected-handled,
degraded-recovered, unexpected, or rethrown. Only degraded-recovered and
unexpected catches report. A degraded-recovered catch reports
`failure_kind:degraded`; an unexpected catch reports
`failure_kind:caught_unexpected`. `outcome` records what happened to the
requested work. Never report and rethrow the same error.
7. **Do not include PII.** Prompts, workflow names or paths, file names, node
titles, emails, and tokens are forbidden in tags and context. Limit context
to ids, counts, enums, and booleans. Before acceptance, update the
`src/AGENTS.md` guidance that permits filenames and URLs in context.
8. **Route operational alerts through Datadog by family.** Following
[ADR-TELEMETRY-ROUTING-0013](TELEMETRY-ROUTING-0013-telemetry-routing-across-consumers.md),
use one RUM monitor per family rather than per slug:
- new invariant: `failure_kind:invariant`
- hard assertion: `assert_mode:hard`
- broken user operation: `outcome:failed` with `operation`
- unexpected catch: `failure_kind:caught_unexpected`
- missing completion: `failure_kind:missing_event`
- failure-rate deviation: `outcome:failed` divided by the paired success
actions from rule 9, grouped by `feature_area` and `operation`
- feature-area anomaly: conforming events grouped by `feature_area`
- recovered-degradation digest: `failure_kind:degraded` with
`outcome:recovered` or `outcome:degraded`

Sentry remains an error exploration and grouping sink, not the operational
alert route.

9. **Add denominators in a later phase.** Paired Datadog RUM success actions
should use the same bounded attributes as these error tags so Datadog
calculated metrics and monitors can measure failure rates.
10. **Keep this decision frontend-scoped.** Backend telemetry transports need
their own amendment before adopting this taxonomy.

## Consequences

### Positive

- Sentry issues become queryable by agents using stable tag conjunctions such
as `failure_kind:invariant feature_area:workflow release:X`.
- `feature_area` becomes shared vocabulary for error tags, future metric
attributes, and operational follow-up.
- Instrumentation changes stay small and reviewable: one feature area or catch
cluster per pull request, with no behavior change to recovery paths.
Comment on lines +142 to +151

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] docs/adr/README.md says to fill in all template sections, and the template splits Consequences into Positive and Negative. Please separate the benefits from accepted costs, including higher event volume and review-only conformance for untyped tags.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 67aed68: Consequences now has Positive and Negative sections, including untyped tags and migration costs.


### Negative

- Event volume rises, so alert families and existing filtering must keep noisy
deployment and chunk-load failures out of these signals.
- Tags remain untyped, so conformance depends on review until emitters enforce
the taxonomy.
- Existing nonconforming emitters must be migrated before the ADR is accepted.

## Alternatives Considered

- A new `reportInvariant()` or `Telemetry` wrapper was rejected because it
duplicates `reportError`, which already fans out to every configured sink.
Comment on lines +163 to +164

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Question: how should this relate to Proposed TELEMETRY-DIAGNOSTICS-0019? That ADR proposes one application-facing diagnostics system, while this one rejects another abstraction and keeps reportError as the emitter. Please state whether this supersedes 0019, narrows it, or must be reconciled before acceptance.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 67aed68: Context now says this ADR narrows TELEMETRY-DIAGNOSTICS-0019 for error telemetry without adding another reporting path.

- Kebab-case tags were rejected because existing slugs use snake_case.
- One Datadog monitor per `error_type` was rejected because it creates an
unbounded rule count and alert fatigue.
- A bare `[errorType]` fingerprint was rejected because it merges distinct root
causes.
- Global sampling was rejected because it can discard the rare invariants this
decision is intended to expose.
- Sentry-only alerting was rejected because accepted
[ADR-TELEMETRY-ROUTING-0013](TELEMETRY-ROUTING-0013-telemetry-routing-across-consumers.md)
assigns low-latency incident response to Datadog RUM and monitors.

## Registry Amendments

After acceptance, append `YYYY-MM-DD tag=value — reason — PR` lines here.

## References

- [`reportError`](../../src/platform/telemetry/reportError.ts)
- [`assert`](../../src/base/assert.ts)
- [Sentry tags](https://docs.sentry.io/platforms/javascript/enriching-events/tags/)
- [Sentry event grouping](https://docs.sentry.io/concepts/data-management/event-grouping/)
- [Google SRE: Monitoring Distributed Systems](https://sre.google/sre-book/monitoring-distributed-systems/)
- [TigerBeetle assertion guidance](https://github.com/tigerbeetle/tigerbeetle/blob/main/docs/TIGER_STYLE.md)
- [ADR-TELEMETRY-DIAGNOSTICS-0019: Recoverable Event Diagnostics](TELEMETRY-DIAGNOSTICS-0019-recoverable-event-diagnostics.md)
- [ADR-TELEMETRY-ROUTING-0013: Telemetry Routing Across Consumers](TELEMETRY-ROUTING-0013-telemetry-routing-across-consumers.md)
4 changes: 4 additions & 0 deletions src/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@
a migration of affected queries and alerts that accounts for old and new
releases reporting different names.

[ADR-TELEMETRY-ERRORS-0030](../docs/adr/TELEMETRY-ERRORS-0030-agent-consumable-error-telemetry.md)
proposes agent-consumable tags, assertion modes, catch classification, and PII
rules. It is Proposed: treat it as direction rather than a gate.

## Security

- Sanitize HTML with DOMPurify
Expand Down
Loading