Skip to content

09-error-handling.md

Latest commit

 

History

History
684 lines (585 loc) · 36 KB

09-error-handling.md

File metadata and controls

684 lines (585 loc) · 36 KB
Raw

09 — Error Handling Standards

Status: Active Derives from: ADR 0002 — Initial Architecture (Problem Details + Result<T> baseline), ADR 0032 — Exception Handling, Logging, and Observability Architecture (implementation patterns), 04-api-design.md § Error Responses. Public-read additions derive from ADR-0052. Tenant API authentication additions derive from ADR-0064. Public UI message mapping derives from ADR-0027 and the Accepted P02d-6 package. Bootstrap response ownership and renderer Retry-After qualification derive from ADR-0055. Delivery/recovery additions derive from ADR-0046. Background-job policy derives from ADR-0047 and ADR-0061. Hard-limit refusal follows ADR-0021, ADR-0045 and the accepted quota accounting.

How LearnStack represents, propagates, surfaces, and recovers from failures.

The conceptual deep dive and diagrams live in 33-cross-cutting-concerns.md. This standard contains the day-to-day rules.

Two-Track Model

flowchart LR
  bug[Bug / infra failure] --> exc[Exception thrown]
  expected[Expected outcome] --> result["Result<T> returned"]
  exc --> l1[L1 IExceptionHandler]
  result --> mapper["result.ToActionResult()"]
  l1 --> problemDetails[Problem Details response]
  mapper --> problemDetails
  problemDetails --> client[Client]
Loading
  • Exceptions for unexpected failures: bugs, transient infra issues, contract violations.
  • Result<T> for expected outcomes: validation failures, not-found, conflicts, business-rule violations.

Both end in RFC 7807 Problem Details at the API boundary.

Result Type

public sealed record Result<T> : IResultBase
{
    internal Result(bool isSuccess, T? value, Error? error, LocalizedMessage? successMessage = null) { ... }
    // The annotations are the contract, not decoration: they let a consumer
    // dereference Value or Error after one check without `!`. They do NOT flow
    // from IResultBase to its implementations, so both carry their own copy.
    // They are load-bearing only where T is a reference type: for a non-nullable
    // struct T — Result<None>, Result<CourseId> — `T?` is already non-nullable, so
    // flow analysis never had a warning to suppress. Nothing is lost; the
    // annotation is simply inert there.
    [MemberNotNullWhen(true, nameof(Value)), MemberNotNullWhen(false, nameof(Error))]
    public bool IsSuccess { get; }

    [MemberNotNullWhen(false, nameof(Value)), MemberNotNullWhen(true, nameof(Error))]
    public bool IsFailure => !IsSuccess;

    public T? Value { get; }
    public Error? Error { get; }
    public LocalizedMessage? SuccessMessage { get; }

    public static Result<T> Ok(T value, LocalizedMessage? message = null); // throws on null value
    public static Result<T> Fail(Error error);
}

public sealed record Error(
    LocalizedMessage Message,
    IReadOnlyDictionary<string, IReadOnlyList<LocalizedMessage>>? Details = null)
{
    public string Code => Message.Key[LocalizedMessage.RequiredPrefix.Length..];
}

public sealed record LocalizedMessage(string Key, IReadOnlyDictionary<string, string>? Params = null)
{
    public const string RequiredPrefix = "lockey_";
    // ctor enforces Key.StartsWith(RequiredPrefix); see Phase 02a Packet 2.
}

public readonly record struct None { public static None Value { get; } }

The LocalizedMessage's lockey_ prefix is invariant: the constructor rejects any key that does not start with lockey_. Resources used to resolve backend messages retain that prefix; it is not a prefix requirement for general frontend UI copy. Backend code never returns raw English. Error.Code is a stable, unprefixed projection of Message.Key (the lockey_ prefix is stripped). Routing logic (Result.ToActionResult(), Problem Details writers) reads Code; the consumer resolving a backend message reads Message.Key — two wire surfaces in sync by construction. This does not claim a shipped frontend message catalogue. See Frontend Error Handling for current SDK normalization and the public UI mapping implemented by P02d-6 Steps 1–3. Per Phase 02a Packet 2 and ADR-0032 § Error Model.

Result<T>'s primary constructor is internal; callers go through Ok / Fail so the success-must-carry-value rule (see § Forbidden) cannot be bypassed via positional record syntax. Result<None> is the canonical payload-less success shape.

Field-level errors in Error.Details flow as LocalizedMessage lists per key, so the lockey_ invariant covers every user-facing string the API ships — not just the top-level message.

Standard error codes (machine-readable, stable). The table uses the unprefixed shape that travels on the Problem Details code field; the matching localization key adds the lockey_ prefix.

Code Meaning HTTP
validation_failed Field-level validation failed 400
not_found Resource does not exist (or hidden cross-tenant) 404
unauthorized Authentication required or invalid 401
forbidden Authenticated but not permitted 403
tenant_mismatch Tenant context mismatch 404
concurrency_conflict Optimistic concurrency token mismatch 409
request_in_progress An earlier attempt with this idempotency key is still running 409
idempotency_key_reuse This idempotency key belongs to a different request 409
idempotency_outcome_unavailable The operation completed; its response was not retained 409
business_rule_violation Domain invariant violation 409
resource_scope_violation Resource-level authorization failure 403
rate_limited Too many requests 429
dependency_unavailable Required dependency unavailable, including invalid stored public configuration 503
audit_unavailable A MUST-class audit row could not be written durably 503
audit_unclassified_operation Operation absent from the audit catalogue 500
recording_consent_required Live session requires consent 409
unsupported_locale Well-formed public query locale absent or disabled for the tenant; implemented by P02d-4 400
feature_disabled Feature flag off for tenant 403
limit_exceeded A hard plan limit refuses a new counted resource 403
method_not_allowed Route matched, method did not — framework-minted 405
payload_too_large Request body over the published limit — framework-minted 413
unsupported_media_type Content-Type the endpoint does not accept — framework-minted 415
request_rejected Any other 4xx nothing else names — framework-minted 4xx as sent

ADR-0064 distinguishes HTTP credential failures from tenant refusal: invalid present credentials on known hosts return bare-challenge 401, while missing AccountSite credentials return masked not_found 404. Authenticated host/header disagreement remains tenant_mismatch; membership refusal remains not_found. Early 413/429 retain their existing precedence and shared Problem Details owner. The complete ongoing contract lives in API Design.

A framework-minted code is one no handler produced: routing or the server rejected the request before an Error could exist, so the code is derived from the status by HttpStatusMap.CanonicalCodeFor. See § API Surface below for the full mapping and for why a 4xx never carries internal_error.

The two audit codes carry the fail-closed rule of ADR-0033; their statuses are fixed by ADR-0044 § 11, which also adds them to the localization catalogue as lockey_audit_unavailable / lockey_audit_unclassified_operation under the prefix rule above. Both are explicit arms of HttpStatusMap.For(string) rather than fallthrough: a body whose code reads audit_unavailable under a 500 is exactly the status-and-code disagreement that map exists to prevent.

  • audit_unavailable (503) reaches the client as an exception, not a Result. IAuditStore.WritePendingAsync throws AuditWriteFailedException : InfrastructureException carrying the Error, and HttpStatusMap.For(Exception)'s existing LearnStackException known => For(known.Error) branch maps it — AuditLogBehavior's catch-and-rethrow contract is untouched. 04-api-design.md § Status Codes requires Retry-After on a 503. An explicit delta/date value is preserved; absent or empty values default to Retry-After: 30, the provider circuit breaker's default break duration. RetryAfter.Apply supplies that default where the status is decided, on both Problem Details paths — LearnStackExceptionHandler for an exception, ProblemDetailsActionResult for a result — rather than where each code is minted, so this 503, dependency_unavailable and the idempotency store's capacity refusal all carry it, and a 503 code added later inherits it. Packet 9 had said so before it was true; its fifth review found no 503 carrying the header and set it (Delivery Record (Packet 9)).
  • ADR-0033 Amendment 1 narrows when that 503 is returned at all. A standalone MUST-class write failure changes the response only when the operation would otherwise have succeeded — a standalone row recording an access that is being granted still answers 503 audit_unavailable. A standalone row recording an operation that is already being refused (a denied authorisation outcome, a rejected tenant assertion) keeps its own 403 / 404; the failure logs at Critical, increments the standalone-write-failure counter, and marks the audit health check unhealthy. A granted read-sensitive query is not one of those rows — since Packet 6 it rides the in-transaction path (ADR-0033 Amendment 2 § 7). The in-transaction class is untouched by that narrowing: a MUST row that cannot be written durably still rolls the business transaction back and answers 503.
  • audit_unclassified_operation (500) is a deployment defect — an IRequest<Result<T>> reached the pipeline's audit step with no catalogue entry. The caller cannot act on it, which is why it is a 5xx and not a 4xx. Absence is the defect, not silence: an operation that writes no row is registered as Off, and the classifier returns Unclassified only for a request type nothing registered (ADR-0044 Amendment 3).

Exceptions

Hierarchy

LearnStackException                  (base)
├── DomainException                  (domain invariant broken from inside, programmer error)
├── InfrastructureException          (DB, Valkey, SeaweedFS transient)
│   └── AuditWriteFailedException    (MUST-class audit row not durable → 503 audit_unavailable)
├── ProviderException                (upstream provider error)
│   ├── PaymentProviderException
│   ├── LiveClassProviderException
│   ├── StorageProviderException
│   └── ...
└── TenantContextMissingException    (no tenant resolved where one is required)

For "case branches that should be impossible" use the BCL System.Diagnostics.UnreachableException (.NET 7+) directly — adding a custom subclass collides with the BCL name and forces every use site to disambiguate with using aliases. The L1 IExceptionHandler treats it the same as any unhandled exception (Sentry-captured, 500 Problem Details).

Rules:

  • Throw LearnStackException subclasses, never raw Exception.
  • Constructors take a structured Error plus the underlying cause.
  • Stack traces flow into logs and Sentry, never to clients.
  • Don't catch Exception broadly; catch specific subclasses where you can act.

Retry vs. Don't Retry

Exception type Retry?
InfrastructureException (transient) Yes, with backoff
ProviderException (5xx, timeout) Yes, with backoff
ProviderException (4xx) No
DomainException No
TenantContextMissingException No

L1 Exception Handler

The first-line catch site is LearnStackExceptionHandler : IExceptionHandler (.NET 8+) per ADR-0032 § Sub-decision 1. Every backend host (LearnStack.Api, workers, background-service hosts) registers it the same way:

services.AddExceptionHandler<LearnStackExceptionHandler>();
services.AddProblemDetails();
// in pipeline:
app.UseExceptionHandler();

Responsibilities of the handler:

  • Map every LearnStackException subclass to its standard Error.Code and HTTP status (the table under § Result Type).
  • Build the RFC 7807 Problem Details body with correlationId set from the full W3C traceparent (Activity.Current.Id, the 00-<trace>-<span>-<flags> string) — not the bare 32-hex trace id — so the Problem Details body, the captured Sentry/LocalFile context, and ITenantContext.CorrelationId all carry the same handle. Falls back to HttpContext.TraceIdentifier when no Activity is current.
  • Call Activity.Current.RecordException(ex) + SetStatus(Error, ...) so Tempo sees the failure.
  • Dispatch to IErrorTrackingProvider.CaptureAsync only when ShouldCapture(ex) returns true (see § Sentry vs OpenTelemetry boundary).

The older app.UseExceptionHandler(lambda) and app.Use((ctx, next) => { try {...} catch {...} }) patterns are not used in new code. There is no ExceptionHandlingBehavior inside the MediatR pipeline — AuditLogBehavior's catch + rethrow + the L1 handler are the only two catch sites below the framework.

Sentry vs OpenTelemetry — Error Capture Boundary

Per ADR-0032 § Sub-decision 7, the two backends receive complementary signals:

Failure OTel span IErrorTrackingProvider Rationale
Unhandled Exception at L1 RecordException + SetStatus(Error) Capture Bug or infra; high-signal
LearnStackException subclass at L1 RecordException + SetStatus(Error) Capture Leaked from a failing layer
ProviderException with IsClientError == false (5xx upstream) RecordException + SetStatus(Error) Capture Upstream infra failure
ProviderException with IsClientError == true (4xx upstream) SetStatus(Error) only Skip Provider's user-error; not our bug
Result.Fail(validation_failed / forbidden / not_found / ...) SetStatus(Ok) (runtime completed; HTTP response is the appropriate 4xx Problem Details) Skip Expected outcome; metric counter only
Result.Fail(business_rule_violation) SetStatus(Ok) Skip Expected outcome; metric counter only
BadHttpRequestException with a 4xx status (oversized body, malformed chunking, bad header) SetStatus(Error) only Skip The client got the request wrong; same class as a provider's 4xx. Capturing it hands an anonymous caller a switch that fills the tracker one request at a time
OperationCanceledException (client disconnect) leave span Unset; no RecordException Skip Noise; not actionable. (ActivityStatusCode is Unset / Ok / Error — Unset is the right default for "we didn't finish but it wasn't a failure".)

The boundary is the L1 handler's ShouldCapture(Exception) switch. Modules never reference Sentry.SentrySdk directly — the architecture test Modules_Do_Not_Reference_Sentry_SDK_Directly enforces it.

API Surface

All API errors are RFC 7807 Problem Details:

{
  "type": "https://errors.learnstack.dev/validation",
  "title": "lockey_validation_failed",
  "status": 400,
  "code": "validation_failed",
  "messageKey": "lockey_validation_failed",
  "instance": "/api/v1/courses",
  "correlationId": "01H...",
  "errors": {
    "title": [
      { "key": "lockey_title_required" }
    ],
    "slug": [
      { "key": "lockey_slug_already_exists_in_tenant", "params": { "slug": "intro" } }
    ]
  }
}

Rules:

  • type is a stable URL.
  • code is the machine-readable identifier — the unprefixed Error.Code (Standards 04 § Problem Details). The codes a framework-level client error carries, where no handler ran and so no Error exists, are minted from the status by HttpStatusMap.CanonicalCodeFor: not_found (404), method_not_allowed (405), payload_too_large (413), unsupported_media_type (415), validation_failed (400 and 422), concurrency_conflict (409), rate_limited (429), dependency_unavailable (503), unauthorized (401), forbidden (403), and request_rejected for any other 4xx — never internal_error on a 4xx, which would put a code and a status that contradict each other in the same body. internal_error is reserved for 5xx, which is what the same method returns there.
  • messageKey is the LocalizedMessage.Key (always begins with lockey_) identifying a backend message for a consumer's supported error resources; it is not an arbitrary UI lookup instruction. The legacy detail field is omitted — backend never returns raw English.
  • instance is the request path.
  • correlationId is the full W3C traceparent (Activity.Current.Id), which embeds the trace id; falls back to the request id when no trace is active.
  • errors is field-level detail, each entry a LocalizedMessage payload (key + optional params), using the same backend-message contract as the top-level one. The frontend feature owns any supported field-message resources.

Validation Errors

  • FluentValidation produces field-level errors.
  • Always include all failures, not just the first one.
  • Field names match the request shape (camelCase).
  • Messages are localizable because they travel as keys: messageKey and each errors entry are LocalizedMessage payloads (key + optional params) for a consumer's supported error resources, as § API Surface states. The API returns no message text today; locale negotiation from Accept-Language for any message text the API composes later is Phase 04's.
  • ValidationBehavior returns Result.Fail(validation_failed, errors) — it does NOT throw FluentValidation.ValidationException. Per ADR-0032 § Sub-decision 3, the pipeline never raises a validation exception. The behavior aggregates ValidationResult failures into the Error.Details dictionary and short-circuits the request. The behavior's generic constraint is where TResponse : IResultBase; the static factory Result.FailFor<TResponse>(error) constructs the correct shape.

Domain Exceptions

Per ADR-0032 § Sub-decision 4, DomainException is reserved for programmer errors (bugs):

  • Aggregate invariant violations that signal a programming mistake (e.g. a domain method was called in an impossible order, an aggregate's invariant was bypassed).
  • Anything where "raising this exception means we have a bug to fix".

Expected business-rule violations return Result.Fail(business_rule_violation, ...) from the domain method — they are not exceptions. Examples that must be Result.Fail, not throws:

  • "Course capacity reached."
  • "Tenant plan limit exceeded."
  • "Cannot enrol learner: enrolment closed."
  • "Cannot publish course: missing required lesson."

Enforcement:

  • The Roslyn analyzer LearnStackException-DomainExceptionThrow flags every throw new DomainException(...) outside aggregate invariant guards as a Warning (Phase 02a) and as an Error after Phase 03 exit.
  • Architecture test Domain_Methods_Do_Not_Throw_For_Expected_Cases walks Result<T>-returning methods and asserts the analyzer's report is empty.

Sensitive Exception Messages

Exception messages never carry sensitive values such as credentials. Property-based runtime redaction does not scrub an exception's message. Use fixed messages and the existing closed descriptors at durable failure boundaries.

ADR-0032 Amendment 6 defines LS0002 (LearnStackException-SensitiveExceptionMessage): a bounded sensitive-name guard on directly thrown exception string arguments, across all production C# projects. The analyzer shares the runtime token matcher; it does not trace earlier values, inspect literal contents or cover JavaScript/TypeScript. Warning severity remains visible in CI until Phase 03 exit, when both LS0001 and LS0002 become Errors. A justified public-metadata false positive may be suppressed at that exact site; never silence the rule project-wide.

Controller Mapping — Result<T> → IActionResult

Per ADR-0032 § Sub-decision 6, the sanctioned shape is an explicit extension method:

[HttpPost("courses")]
public async Task<IActionResult> Create(
    CreateCourseCommand command, CancellationToken ct)
    => (await _mediator.Send(command, ct)).ToActionResult();

ResultExtensions.ToActionResult() lives in LearnStack.Api.Common. It matches on Error.Code and emits the Problem Details body with the right HTTP status (per the table in § Result Type). There is no action filter, no MediatR ResultUnwrapBehavior, no implicit conversion — the explicit pattern keeps the diff honest and the debug experience straightforward.

Frontend Error Handling

Boundaries

  • Root error boundary catches unexpected client errors and shows a recovery page with a correlation id.
  • Route group error.tsx shows a context-appropriate fallback.
  • Form errors render inline at the field level.
  • Toasts for transient, non-blocking failures.
  • Modals for action-required failures.

Mapping Problem Details → UI

type AppError =
  | { code: "validation_failed"; fieldErrors: Record<string, string[]> }
  | { code: "not_found"; resource?: string }
  | { code: "concurrency_conflict"; latestVersion?: number }
  | { code: "request_in_progress" }
  | { code: "idempotency_key_reuse" }
  | { code: "idempotency_outcome_unavailable" }
  | { code: "method_not_allowed" }
  | { code: "payload_too_large" }
  | { code: "unsupported_media_type" }
  | { code: "request_rejected" }
  | { code: "dependency_unavailable"; provider?: string; retryAfter?: number }
  | { code: "audit_unavailable"; retryAfter?: number }
  | { code: "audit_unclassified_operation"; correlationId?: string }
  | { code: "rate_limited"; retryAfter?: number }
  | { code: "forbidden" }
  | { code: "unauthorized" }
  | { code: "recording_consent_required" }
  | { code: "unknown"; correlationId?: string };

The SDK maps Problem Details payloads to AppError; UI code switches on code. It validates/preserves backend message keys as data and owns no translated resources. A feature that displays backend messages owns its supported error resources; general UI copy has its own feature identifiers.

ADR-0027 specifies P02d-6's closed page-outcome mapping to owned UI keys, with unknown outcomes mapped to the bounded unavailable state. It does not authorize arbitrary Problem Details keys, titles, field errors or parameters as UI lookup identifiers or visible copy. This public-page mapping is Accepted — 2026-10-09; P02d-6 Steps 1–3 implement it, with evidence in the Step 1, Step 2 and Step 3 delivery records.

Public Page Status and Recovery

P02d-6 G40 composition implemented — Steps 1–3, 2026-10-10. A missing/hidden content resource redirects locally with 307 to the same host's fixed /{locale}/status/not-found, whose admitted branded document returns 404. The browser URL changes. The page rechecks live host/locale admission, never calls Education, echoes no original target/query/cursor and supplies localized noindex metadata and a catalog recovery anchor. This is not a direct branded 404 at the original URL. A fresh bootstrap refusal remains neutral.

Known content-call failures are translated HTTP 200 noindex page states: invalid cursor has a relative reset link, rate limiting has a retry-later state, and transport/invalid-response/unavailable or unknown outcomes have the bounded unavailable state. They do not claim HTTP 400/429/503. Unexpected framework failures retain pre-stream 500/post-stream 200 behavior; error.tsx cannot choose arbitrary status. Plain same-host relative anchors trigger fresh document requests. Admission completes before loading boundaries flush; Standards 07 owns placement and the shared request-local loader.

ADR-0055 implementation delivered — 2026-10-10; production replacement proofs pass. All three steps completed both independent review rounds and focused fix verification. Middleware owns the sole live bootstrap and emits neutral 404 for invalid entry or API 404, 429 for valid API 429 and 503 for other site/transport/configuration failures. Responses are no-store with fixed plain-text copy and bodyless HEAD; failed bootstrap permits no Education operation or newly emitted tenant theme. RSC consumes only the validated same-request snapshot. An active-request context defect is a sanitized lifecycle failure, not API 404 or another read. Completion refusal cannot rewrite an already completed response.

Forward optional parsed Retry-After only for closed supported 429/503 error cases, as integer delta-seconds from zero through sixty inclusive. Preserve eligible values; omit malformed/date/negative/out-of-range values without clamping or inventing a retry. This qualifies renderer forwarding only; the API's selected refusal metadata and accounting remain unchanged. The Step 3 record records passing exact-wire failure/lifecycle proofs, both independent review rounds and focused fix verification.

User-Facing Copy

  • Never show stack traces.
  • Never show internal correlation ids without an explanatory message.
  • Always offer a next step (retry, contact support, navigate elsewhere).
  • Localize messages.

Provider Failures

  • Wrap every provider call with ProviderException mapping at the adapter boundary.
  • Translate provider-specific status codes to our normalized codes; set ProviderException.IsClientError based on the upstream status (true for 4xx, false for 5xx). The L1 handler uses this flag to decide whether to Sentry-capture (5xx) or not (4xx).
  • Don't leak provider names to end users (detail: "Recording could not be started. Please try again." not "LiveKit returned 503").
  • Capture provider raw response to logs (with redaction) for debugging.

Provider Resilience — Polly v8 ResiliencePipeline

Per ADR-0032 § Sub-decision 5, every provider adapter routes its outbound calls through a Polly v8 ResiliencePipeline carried by IProviderResilience<TPort>. The port is not decorated — C# forbids a type parameter as a base type, so no ResilientProviderAdapter<TPort> can satisfy : TPort. The composition root registers the adapter and its pipeline separately:

services.AddSingleton<ILiveClassProvider, LiveKitClient>();
services.AddProviderResilience<ILiveClassProvider>(configuration, "liveclass");

services.AddSingleton<IPaymentProvider, StripePaymentClient>();
services.AddProviderResilience<IPaymentProvider>(configuration, "payment");
// ...

The adapter takes IProviderResilience<TPort> as a constructor collaborator and wraps each SDK call in Pipeline.ExecuteAsync. ProviderResilience<TPort> reads Resilience:<portName>: from appsettings.{env}.json and builds a pipeline with:

  • Retry — exponential backoff with jitter; only retries ProviderException with IsClientError == false and InfrastructureException (transient).
  • Circuit breaker — opens on failureRatio over samplingDuration; shields the upstream from sustained pressure.
  • Timeout — bounds the longest single attempt.
  • Bulkhead — caps concurrent in-flight calls per upstream.

The adapter's only exception-related job is SDK → ProviderException translation. The retry / circuit-breaker / timeout policy lives in configuration and is carried by IProviderResilience<TPort>; the adapter invokes it and never authors one. The add-provider-adapter skill walks the canonical shape for every new adapter.

Configuration shape (excerpt):

{
  "Resilience": {
    "liveclass": {
      "retry": { "maxRetryAttempts": 2, "delaySeconds": 1, "useJitter": true },
      "circuitBreaker": { "failureRatio": 0.5, "samplingDurationSeconds": 30, "minimumThroughput": 10, "breakDurationSeconds": 30 },
      "timeout": { "totalSeconds": 10 }
    }
  }
}

Architecture test Adapters_Wrap_Provider_Exceptions asserts that SDK exception types (LiveKit.NET.LiveKitException, Stripe.StripeException, Meilisearch.MeilisearchApiError, …) never leave the LearnStack.Infrastructure.<Adapter> namespaces.

Background Jobs

Accepted ADR-0047 owns the job failure policy. The P02b-4 delivery records own the delivered implementation; the phase status owns remaining hosted qualification.

  • Count at most six normal admissions and six separate finalize admissions. Retries use the five fixed ordinal delays: 1 s, 5 s, 30 s, 5 min and 1 h. A duplicate vendor wakeup or recovery probe spends no business attempt.
  • Preserve top-level Result/exception classification for ordinary requests and invocation, Prepare and Finalize failures. Known completed external Failed/ Cancelled reports use Accepted ADR-0061's explicit policy. Owned lease loss, deadline or shutdown closes new checkpoint/COMMIT admission and retains work; an already admitted COMMIT keeps its actual result. Never infer permanent refusal or positive quiescence from cancellation or elapsed time.
  • Keep terminal state on the authoritative job row. Its fenced transition and terminal MUST audit commit together; prepared domain failure uses its audited finalizer. No duplicate dead-letter table is required.
  • Phase 03 owns audited operator inspection/reconciliation and replay permission; no operator surface is claimed here.

Accepted ADR-0061 owns the known completed external-failure branch. Step 7 implements and qualifies its report, storage and finalization contract before external activation; Step 9 owns full enforcement. Acceptance supplies no completed runtime proof.

Outbox

Accepted ADR-0046; P02b-2 producer runtime and enforcement complete and merged. All four steps completed both independent review rounds. The merge closeout owns CI/merge status. P02b-3 Steps 1–4 implement durable acceptance, shared inbox, independent subscriber execution/retries and terminal audit. Step 5 activates both real default subscriptions and implements the shared initializer/Seeder handoff.

  • Each committed normal claim increments producer attempts once, including an unknown crash outcome. Six total claims use retry delays 1 s, 5 s, 30 s, 5 min and 1 h. Expired claims recover with fresh tokens; claim six cannot publish again.
  • Required Pending subscriptions preserve rows before acquisition/validation. Only declared intentional absence may acknowledge without publication.
  • Unknown/retired/invalid records request explicit terminalization. Terminal requests never publish; terminal UPDATE and MUST audit share one transaction. Audit refusal retains terminal work for retry and makes health unhealthy.
  • Every success/failure/recovery/terminal transition is token- and expiry-fenced. A timeout bounds the wait; cancel and retain the task-owned scope until actual completion. Late results cannot acknowledge or overwrite a new owner.
  • Persist only closed stage/reason/category codes, optional allowlisted SQLSTATE and validated attempt traceparent in last_error, at most 256 UTF-8 bytes. Dispatcher logs/traces likewise exclude exception text/stacks, payloads, raw stored type and connection data. P02b-3 removes the former raw transport exception telemetry; subscriber execution and observation retain bounded diagnostics.
  • After durable acceptance, subscriber business failure spends only its independent ledger budget. Failed/unknown acceptance still spends producer attempts. Subscriber retry invokes one internal delivery executor, never a broadcast; operator recovery routes and producer reset/fencing are Phase 03.

Webhooks

  • Inbound webhook handlers return 200 on duplicate (idempotent).
  • Signature verification failure → 401 with no body.
  • Tenant resolution failure → 404 with no body.
  • Processing errors → 500 (provider will retry).
  • Always log the full event id for traceability.

Forbidden

  • Swallowing exceptions silently.
  • Returning null to signal an error.
  • Throwing Exception directly.
  • Throwing inside catch blocks without preserving the inner exception.
  • Including stack traces or query text in Problem Details.
  • Result<T> with IsSuccess = true but Value = null — model expected absence as Result.Fail with a not_found code, and payload-less success as Result<None>.
  • Localizing error codes (codes are stable English identifiers; only title and detail are localized).
  • Throwing DomainException for expected business-rule violations — use Result.Fail(business_rule_violation, ...) instead. The Roslyn analyzer flags violations.
  • Throwing FluentValidation.ValidationException from the ValidationBehavior. The behavior returns Result.Fail(validation_failed).
  • Importing Sentry.SentrySdk from any module assembly. Capture happens centrally via IErrorTrackingProvider; the L1 IExceptionHandler is the only sanctioned caller in application code.
  • Adding an ExceptionHandlingBehavior to the MediatR pipeline. The AuditLogBehavior + L1 IExceptionHandler cover the two needed catch sites; a third behavior would duplicate the responsibility.
  • Importing a provider SDK exception type (LiveKit.NET.LiveKitException, Stripe.StripeException, …) outside the adapter's LearnStack.Infrastructure.<Adapter> namespace.

The architecture tests and Roslyn analyzers that enforce the rules above are listed in 21-architecture-tests-catalogue.md § Cross-cutting: error handling, logging, observability; that catalogue is the canonical reference for every identifier.