diff --git a/.claude/skills/add-architecture-test/SKILL.md b/.claude/skills/add-architecture-test/SKILL.md index 5ca3049e..5e637542 100644 --- a/.claude/skills/add-architecture-test/SKILL.md +++ b/.claude/skills/add-architecture-test/SKILL.md @@ -129,11 +129,12 @@ Add yours to the file whose subject it shares: | `ApiConventionTests.cs` | Live majors, forwarded headers, required `Deployment:Mode`, unversioned route prefixes. | | `CrossCuttingFoundationTests.cs` | Pipeline order, `Result` returns, topic naming, and the direct-reference bans (Sentry, `DeploymentMode`, `IEventBus`, the cache clients, the Hub, provider SDK exceptions). | | `RequestSurfaceTests.cs` | What the step-4 authority ceiling admits: the two request markers, their permitted sets, the shape of the attributes themselves, and the ban on request shapes MediatR runs with no pipeline. | -| `PlatformAdminScopeConventionTests.cs` | The single sanctioned `BYPASSRLS` path: who may resolve the keyed platform data source, where connection strings are read, the entry gate, and what the scope must not touch. | +| `PlatformAdminScopeConventionTests.cs` | The platform-admin `BYPASSRLS` path: who may resolve its keyed data source, where connection strings are read, the entry gate, and what the scope must not touch. ADR-0046 separately confines the dispatcher credential. | | `RepositoryLayoutTests.cs` | `No_Source_Folder_Named_Verticals`, the single-frontend-app rule, and the commit-subject grammar — checked by running the `commit-msg` hook. | | `DomainEventHandlerTests.cs` | Complete domain-handler/helper/interface census; outbox-only reactions with planted forbidden-capability controls. | | `IntegrationEventContractTests.cs` | Production event inheritance, explicit identifier partition derivation and identifier-only payload/serializer shape. | | `OutboxProducerConventionTests.cs` | Producer/helper fence against extra connections, commit ownership and direct publication. | +| `OutboxDispatchConventionTests` (partial files) | Whole-production dispatcher credentials/publication, exact literal-SQL writers and memory-only observation/registration censuses, with compiled clean/dirty controls. | | `AggregateWriteTests.cs` | The one sanctioned cross-aggregate write, and the count of aggregate roots a handler's ports can write that keeps it at one. | | `CustomizationRegistryTests.cs` | The closed renderer registries that exist twice — in C# and in TypeScript — and stop the two drifting. | | `AuditConventionTests.cs` | The audit aggregate's shape, the closed-set columns against their `CHECK`s, append-only enforcement at the source level, the closed set of `audit_log` writers, and the `OperationType` table. | diff --git a/.claude/skills/add-audit-coverage/SKILL.md b/.claude/skills/add-audit-coverage/SKILL.md index 5bf0bc99..25be356b 100644 --- a/.claude/skills/add-audit-coverage/SKILL.md +++ b/.claude/skills/add-audit-coverage/SKILL.md @@ -105,12 +105,15 @@ parsed from the other, and three rules in `AuditCoverageTests` join them — whose command has since shipped **fails**. - **`(off-path)`** marks an operation that is not a MediatR request at all. Those rows sit outside the request-type join in both directions and register by slug rather than - by type. Seven rows carry it: Tenancy's `platform.admin_scope.enter`, + by type. Nine rows carry it: Tenancy's `platform.admin_scope.enter`, `tenancy.killswitch.toggle`, `tenancy.entitlement.refresh`, - `tenancy.tenant_assertion.reject` and `tenancy.tenant_assertion.anonymous_burst`, and - Audit's `audit.redaction.apply` and `audit.purge.apply`. The three whose writer ships are - registered through `DeclareOffPath`; the other four also carry `(planned)` and register - when their writer lands. + `tenancy.tenant_assertion.reject`, `tenancy.tenant_assertion.anonymous_burst`, + `platform.outbox.dead_letter` and `platform.delivery.dead_letter`, and + Audit's `audit.redaction.apply` and `audit.purge.apply`. The four whose writer + ships are registered through `DeclareOffPath`; the other five also carry + `(planned)` and register when their writer lands. ADR-0046 reserves the + Tenancy-owned producer terminal operation for P02b-2 and subscriber terminal + operation for P02b-3. - **A row carries both markers where both are true** — off the request path *and* ahead of the code that will write it. @@ -529,8 +532,13 @@ passes even when every policy is inert. See aggregate into another module's context inverts the dependency direction. - **`UPDATE`ing an audit row to add detail after the fact.** There is no second phase. The row is composed complete at the commit boundary, `IAuditStore` has **no update - method** — it has four *write* methods, `WritePendingAsync`, `WriteStandaloneAsync`, - `WriteBestEffortAsync` and `WritePlatformScopeAsync`, and that is a different thing — + method**. Its original four write methods are `WritePendingAsync`, + `WriteStandaloneAsync`, `WriteBestEffortAsync` and `WritePlatformScopeAsync`. + ADR-0046 adds the fifth, implemented in P02b-2 Step 2, + `WriteEventTerminalAsync(AuditEntryDraft, EventTerminalTransition, DbConnection, + DbTransaction, CancellationToken)` for confined terminal writers. Its restricted + witness binds audit identity to the actual fenced UPDATE; handlers cannot call + this seam. None is an update method, and `learnstack_app` holds no `UPDATE` privilege on `audit_log`. - **Truncating snapshots silently.** Over `JsonValue.MaxRowBytes` the value becomes the elision record of Step 4b, in the column's own JSON type. Never an empty object, never diff --git a/.claude/skills/add-integration-event/SKILL.md b/.claude/skills/add-integration-event/SKILL.md index 73c4cfbb..4c2d8185 100644 --- a/.claude/skills/add-integration-event/SKILL.md +++ b/.claude/skills/add-integration-event/SKILL.md @@ -3,10 +3,11 @@ name: add-integration-event description: > Publish or consume an integration event across modules using the outbox pattern, `IEventBus` (`InProcessEventBus` today; Dapr pub/sub → Kafka on the Phase 11 - trigger), and per-module inbox idempotency. + trigger), and shared transport-owned inbox idempotency. USE FOR: declaring a new versioned `IntegrationEventVN`, wiring a module to publish it via `IOutbox.EnqueueAsync`, and wiring another module to - consume it via `IIntegrationEventHandler` + `IInboxGuard`. DO NOT USE FOR: + consume it via `IIntegrationEventHandler` under the transport-owned inbox + boundary. DO NOT USE FOR: intra-module domain events (those use plain MediatR `INotification`, in-process), provider webhook receivers (they have their own pattern), or Hub-side events (those live in the `learnstack-hub` repo). @@ -31,7 +32,10 @@ collection and the completed-provisioning publisher. Step 4 implements the production censuses and catalogue enforcement; the [delivery record](../../../docs/roadmap/phase-02b-events-auth.md#p02b-1-step-4--producer-enforcement-and-closeout-2026-10-12) owns final verification and review status. -P02b-2/P02b-3 retain dispatcher/inbox decisions. +Accepted [ADR-0046](../../../docs/decisions/0046-durable-event-delivery-and-recovery.md) +owns dispatcher recovery, subscription readiness and the shared inbox contract. +P02b-2 implements producer dispatch; P02b-3 still owns durable acceptance, +independent execution, canonical inbox/ledger DDL and real subscriptions. ## When to use @@ -80,7 +84,7 @@ public sealed record EnrollmentCreatedIntegrationEventV1 : IntegrationEventBase public override string Topic => "learnstack.enrollment.enrollment"; // Declare EnrollmentId as the partition source. ADR-0057 accepts canonical - // identifier derivation; G3 durable publish ordering remains P02b-2. + // identifier derivation; ADR-0046 bounds eligible selection, not publish order. public override string PartitionKey => EnrollmentId.ToString("D"); } ``` @@ -153,7 +157,8 @@ Rules: causation and optional causal actor come from trusted execution. That boundary guarantees W3C Activity without a listener. Domain translation mints a new integration EventId and uses its domain EventId as causation. A non-marked - event does not erase an announced organization; G1 delivery scope stays open. + event does not erase an announced organization. ADR-0046 preserves that envelope + organization during delivery; a non-marked type never widens its scope. - The owner drains successful-save events FIFO before invariants/MUST audit and commit, at most 256 per transaction. Domain handlers only translate to outbox: no mutation/write stores, nested sends, further raises, publication or providers. @@ -186,94 +191,65 @@ neither derives a second answer. The architecture test `Integration_Event_TopicNames_FollowConvention` enforces the pattern; deviation fails the build. -### Step 4: Consumer — handler + inbox guard - -**Inherited target, pending P02b-2 G1 and P02b-3 delivery implementation.** This -example does not decide inbox ownership, deduplication identity or delivery scope. -The accepted first flow uses independent Customization `card@1`/`plain@1` -subscriptions for `TenantCreatedIntegrationEventV1`, not an Audit mirror. G1 must -prevent one subscription from suppressing its sibling before those handlers ship. -P02b-3 owns create-only preservation, per-effect MUST audit and the semantic -key/version Seeder handoff described in ADR-0057. - -In `.Application/IntegrationEvents/.cs`: - -```csharp -public sealed class QueueWelcomeNotificationOnEnrollmentCreated( - IInboxGuard inbox, - NotificationsDbContext db, - ILogger logger) - : IIntegrationEventHandler -{ - public async Task HandleAsync( - EnrollmentCreatedIntegrationEventV1 @event, CancellationToken ct) - { - if (await inbox.IsAlreadyProcessedAsync(@event.EventId, ct)) - { - logger.LogDebug("Duplicate event {EventId} skipped", @event.EventId); - return; - } - - // ... business logic ... - var dispatch = NotificationDispatch.QueueWelcome( - @event.LearnerId, @event.EnrollmentId); - db.NotificationDispatches.Add(dispatch); - - inbox.MarkAsProcessed(@event.EventId, @event.GetType().Name); - await db.SaveChangesAsync(ct); // atomic business write + inbox marker - } -} -``` - -Rules: - -- Under the inherited consumer target, **every** `IIntegrationEventHandler` invokes - `IInboxGuard.IsAlreadyProcessedAsync` - before any business logic. The architecture test - `Integration_Event_Handlers_Use_InboxGuard` holds every handler to it from - [Phase 02b](../../../docs/roadmap/phase-02b-events-auth.md), which ships the guard; it - is registered, not running, until then. -- `MarkAsProcessed` writes on the same **transaction** as the business write — the ambient - one the transport opens per delivery through `IUnitOfWork` - ([ADR-0040](../../../docs/decisions/0040-ambient-unit-of-work.md)) — so the inbox marker - and the business write commit atomically. -- The **transport** — not middleware — restores tenant context from - `@event.TenantId` before your handler runs, and puts the publisher's own back - afterwards. Read it through `ITenantContext` as usual; don't read it from - anywhere else. -- **Organization comes from the envelope, and it has to.** An earlier version of - this skill said it was deliberately not restored, reasoning that inventing an - organization scope would narrow queries the producer never narrowed. Under the - canonical Row Level Security policy the reasoning inverts: with - `app.organization_id` unset, an organization-scoped row evaluates - `false OR NULL OR NULL`, and a NULL policy result is false — so an absent - organization *hides* every organization-scoped row and `WITH CHECK` rejects - writing one. Widening is the `app.scope = 'tenant'` hatch, not an absent value. -- **The effective actor is always `UserId.SystemActor`.** A human named by the - envelope remains separate causal audit metadata (`CausalActorUserId`); an - asynchronous consumer never impersonates that human. -- **Your handler's constructor must do nothing but assign fields.** Each - subscription gets its own async DI scope and exactly one handler construction. - Constructor, handler and disposal failures are contained to that subscription, - so healthy siblings still run. +### Step 4: Consumer — transport-owned inbox boundary + +**Accepted target, pending P02b-3 implementation.** The transport owns the shared +`inbox_messages` guard; business handlers do not inject or call `IInboxGuard`. +Deduplication is `(tenant_id, subscription_id, event_id)`, with a stable versioned +subscription ID independent of handler type names. The separate delivery ledger +owns attempts, leases, terminal state and targeted replay. + +Before constructing a handler, the executor opens an app-role unit of work, +announces envelope tenant and organization, and acquires the uncommitted inbox +claim. Completed duplicates skip construction. The completed inbox marker, +business changes, produced outbox, MUST audit and fenced ledger completion commit +on that one transaction; rollback removes a new claim. A replay never resets a +completed inbox row or republishes to healthy siblings. + +The effective actor is `UserId.SystemActor`; an envelope's human actor remains +separate trusted causal metadata. Envelope organization is preserved even for a +non-marked event type. Restore context before resolving any handler, store or +module `DbContext`, and restore the caller's context afterwards. + +A handler constructor only assigns fields. Construction, execution and disposal +failures belong to that subscription's own retry budget. Timeouts poison the +transaction owner; late work cannot commit success. An uncertain physical COMMIT +is Indeterminate and recovers from atomic ledger/inbox state. Ordinary app-role +delivery refuses privileged erasure work; Phase 03 owns that separate audited +boundary. External effects need idempotency keyed by subscription/event, or an +outbox intent; database rollback alone cannot undo them. + +The first real flow has two independent Customization subscriptions: +`customization.initialize-content-type-default.v1` installs `card@1` content type; +`customization.initialize-taxonomy-default.v1` installs `plain@1` taxonomy. Both +remain Pending until P02b-3 supplies durable acceptance, create-only preservation, +per-effect MUST audit and the semantic key/version Seeder handoff. Neither may +suppress its sibling or overwrite authored, retired or deleted definitions. ### Step 5: Subscription registration -Today, expose the consumer assembly to the composition root so the -construction-free registry discovers and registers its concrete handlers: +The current baseline exposes the consumer assembly to the composition root so the +construction-free registry discovers and registers its concrete handlers. This +abridged assembly-registration example uses the shipped public module marker; +retain the other module assemblies in the actual composition root: ```csharp builder.AddLearnStackCrossCuttingFoundation( deploymentMode, - typeof(QueueWelcomeNotificationOnEnrollmentCreated).Assembly); + typeof(LearnStack.Modules.Customization.Application.AssemblyMarker).Assembly); ``` Today `InProcessEventBus` resolves the concrete `IIntegrationEventHandler` directly from DI in that subscription's async scope. MediatR is not involved. There is no shipped -`AddDaprSubscription` helper; do not invent one. Phase 11's Dapr adapter -invokes the same event-declared topic and handler contract, so the handler code -remains identical across transports. +`AddDaprSubscription` helper; do not invent one. P02b-2's construction-free +manifest implements stable subscription IDs, compatible +type aliases and required Pending/Active readiness. Pending holds the producer row +before acquisition or payload validation. Only explicit intentional absence may +acknowledge a known type without publication. P02b-3 makes Active publication mean +durable acceptance of all matching deliveries, independently of business handling. +Do not invent a registration API before its implementation ships. Phase 11 must +preserve these contracts when it adds Dapr. ### Step 6: Tests @@ -287,8 +263,10 @@ byte boundaries, SQL NULL metadata and terminal reuse. For delivered flows: 1. **Producer test** — exercise the handler, assert an `outbox_messages` row exists with the right `type`, `topic`, `payload`. -2. **Consumer test** — feed the event into the handler twice; assert the business - write happens exactly once (idempotency via `IInboxGuard`). +2. **Consumer test, P02b-3** — deliver duplicates through the real transport and + ledger, including concurrent duplicates and sibling subscriptions. Assert one + completed inbox/business effect per subscription/event, atomic rollback, + independent retry budgets and recovery after an uncertain final-claim COMMIT. ## Validation @@ -296,22 +274,22 @@ byte boundaries, SQL NULL metadata and terminal reuse. For delivered flows: - `LearnStack.Tests.Architecture` is green — `Integration_Event_TopicNames_FollowConvention` and `Integration_Events_Inherit_From_IntegrationEventBase` run today. `Integration_Event_Handlers_Use_InboxGuard` remains Registered for P02b-3's - consumer delivery after G1. Check + transport-owned consumer delivery. Check [the catalogue](../../../docs/standards/21-architecture-tests-catalogue.md) for their status rather than assuming a net is under you. -- An integration test confirms the round-trip: handler enqueues → outbox row - created → outbox processor dispatches → consumer handles + writes business - state + inbox row. -- Sending the same event twice writes the consumer's business state exactly once. +- P02b-2 proves producer recovery and terminal audit independently of Pending + subscriptions. P02b-3 proves the round-trip through durable acceptance and the + shared inbox; do not claim that proof before those components ship. +- P02b-3 duplicate delivery yields one committed business effect per subscription. ## Common pitfalls - **Crossing modules with a domain event.** Domain events stay inside the module that owns the aggregate. The moment another module needs to react, it must be an integration event. -- **Forgetting `IInboxGuard`.** The consumer rule remains Registered until - P02b-3; do not infer enforcement from its name. Consumer delivery must prove - redelivery cannot duplicate its business effect. +- **Moving the inbox guard into a handler.** The transport must claim before + construction. The rule remains Registered until P02b-3; its name alone proves + no enforcement. Redelivery must preserve independent sibling effects. - **Writing the outbox row in a separate transaction.** Use `IOutbox.EnqueueAsync` through the enrolled ambient `IUnitOfWork`; never `new TransactionScope`. - **Supplying a second topic at enqueue or subscription time.** The concrete diff --git a/.claude/skills/add-integration-test/SKILL.md b/.claude/skills/add-integration-test/SKILL.md index 16256df7..8beaa950 100644 --- a/.claude/skills/add-integration-test/SKILL.md +++ b/.claude/skills/add-integration-test/SKILL.md @@ -237,15 +237,19 @@ those shipped examples for Education rather than inventing an authenticated rout ### Step 3: Outbox round-trip -> **Steps 3 to 5 are the shape, not today's API.** `IOutbox` and the outbox -> dispatcher do not exist yet — Phase 02b owns both. `audit_log` shipped in Packet 9: -> for a real audit assertion, follow `AuditWorkflowTests`, which drives commands through +> **The `_fx.*` members in Steps 3 to 5 are illustrative.** P02b-1 implements +> `IOutbox`; follow `OutboxProducerTests` and `TenantCreatedProducerTests` for +> shipped producer fixtures. P02b-2 Step 3 implements the API-host dispatcher; +> both independent review rounds are complete, and `OutboxProcessorTests` owns its +> runtime proofs. Real subscriptions remain Pending; consumer/inbox assertions +> belong to P02b-3. `audit_log` shipped in Packet 9: for a real audit assertion, +> follow `AuditWorkflowTests`, which drives commands through > the seeder's composition root on `SchemaFixture` as `learnstack_app` and reads the > persisted columns. The durable `IIdempotencyStore` ships on the trigger > [ADR-0037 Amendment 1](../../../docs/decisions/0037-idempotency-key-contract.md) > names: the first `[Idempotent]` endpoint, or the first deployment running more -> than one instance. Packet 6 shipped the `idempotency_keys` table, not the store. The `_fx.*` members below are illustrative of what those phases will -> provide. The schema examples above use `PostgresFixture`, `SchemaFixture` and +> than one instance. Packet 6 shipped the `idempotency_keys` table, not the store. +> The schema examples above use `PostgresFixture`, `SchemaFixture` and > `SchemaQueries`; production public HTTP cases use the separately shipped fixtures > linked above. Write against those delivered APIs until the owning phase lands. @@ -268,8 +272,8 @@ public async Task CreateEnrollment_publishes_EnrollmentCreated_via_outbox() } ``` -To assert the consumer side too, run the outbox processor synchronously in the -test: +The future P02b-3 consumer-side assertion below is illustrative; the shipped +worker has no `_fx.RunOutboxProcessorOnceAsync()` fixture API: ```csharp await _fx.RunOutboxProcessorOnceAsync(); diff --git a/.claude/skills/code-review/SKILL.md b/.claude/skills/code-review/SKILL.md index af20dc97..df60c1f7 100644 --- a/.claude/skills/code-review/SKILL.md +++ b/.claude/skills/code-review/SKILL.md @@ -107,7 +107,7 @@ Walk the diff for correctness issues. Common shapes: | **Async/await** | `.Result` / `.Wait()` deadlock risk; cancellation token threaded through; `ConfigureAwait(false)` where the library expects it. | | **Race condition** | Read-modify-write without optimistic concurrency (`row_version`) or a transaction. Cross-instance state without `ICacheService` invalidation. | | **Resource leak** | `DbContext` / `HttpClient` / file stream not disposed; long-lived `DaprClient` reuse via DI, not `new`. | -| **Idempotency** | Command with external side effects (LiveKit room, Stripe charge) missing `Idempotency-Key` handling. Integration-event consumer missing `IInboxGuard.IsAlreadyProcessedAsync`. | +| **Idempotency** | Command with external side effects (LiveKit room, Stripe charge) missing `Idempotency-Key` handling. Under ADR-0046, transport claims the shared inbox before handler construction; P02b-3 implements it. Handlers do not call the guard themselves. | | **Outbox atomicity** | Immediate INSERT through the enrolled, announced ambient `IUnitOfWork`; source state/outbox/MUST audit share its transaction. SaveChanges flushes only. ADR-0057 generation, tenant, read-only, payload/metadata and duplicate refusals hold. | | **Tenant context restoration** | Background job + integration-event handler restores `TenantContext` before any work. | | **Edge cases** | Empty list, zero, negative, max-int, very-long string. Test or guard. | diff --git a/.claude/skills/local-dev-setup/SKILL.md b/.claude/skills/local-dev-setup/SKILL.md index ae8eaa45..dc33a7b3 100644 --- a/.claude/skills/local-dev-setup/SKILL.md +++ b/.claude/skills/local-dev-setup/SKILL.md @@ -105,14 +105,22 @@ P02d-5 Step 1 supplies the paired loopback launcher. Follow [README Quickstart](../../../README.md#3-prepare-local-https-and-start-the-applications) for `make public-env`, explicit mkcert trust/leaf creation and manual hosts aliases. Then run `make public-api` and `make public-web` in separate terminals. The API -helper loads only runtime database credentials and the paired hop network/secret; +helper loads application/dispatcher database credentials and the paired hop +network/secret; the web launcher captures native socket provenance before Next. Neither script shell-evaluates `.env`, edits hosts or installs trust. The optional web projection -must match the root private source. - -**Ordinary API startup is unchanged.** `dotnet run` still does not load `.env`. -For an API-only session, supply `ConnectionStrings:Default` through environment or -the project's `learnstack-api-dev` user-secrets store. Both empty hop lists remain +must match the root private source. README Quickstart owns the +[closed local demo database grammar](../../../README.md#3-prepare-local-https-and-start-the-applications) +and its supported field aliases; it does not promise general Npgsql parsing. + +**Ordinary API startup.** `dotnet run` does not load `.env`. For an API-only +session, supply `ConnectionStrings:Default` (`learnstack_app`) and +`ConnectionStrings:OutboxDispatcher` (`learnstack_outbox_admin`) through environment +or the project's `learnstack-api-dev` user-secrets store. The mandatory API worker +resolves its guarded source and verifies dispatcher connectivity at startup. +The dispatcher credential belongs only to the API; every Seeder invocation uses +an explicit application-only environment, and the web launcher receives neither +database credential. Both empty hop lists remain valid in every mode; a network-only or secret-only configuration is refused. The application connection must name `learnstack_app`; the data-source guard also refuses direct or transitive BYPASSRLS/superuser access. Migration credentials diff --git a/.claude/skills/standards-check/SKILL.md b/.claude/skills/standards-check/SKILL.md index 347b60f6..aa03371b 100644 --- a/.claude/skills/standards-check/SKILL.md +++ b/.claude/skills/standards-check/SKILL.md @@ -375,8 +375,12 @@ domain the diff doesn't touch. [30-api-gateway.md § 2](../../../docs/architecture/30-api-gateway.md). - [ ] Hub contract surface still satisfies ADR-0034's two invariants; any new endpoint carries its ADR and lands in both repositories. -- [ ] Outbox + inbox usage correct (atomic with aggregate write; inbox guard - in every consumer). +- [ ] Producer INSERT uses its announced transaction. ADR-0046 dispatcher + transitions use fresh tokens and atomic terminal audit; transport-owned shared + inbox acquisition precedes handler construction (P02b-3 pending). +- [ ] Manifest readiness preserves required Pending rows. Active publication + means durable acceptance; subscriber retry/terminal/replay state is independent + of the producer budget. Check catalogue status before claiming enforcement. ### Step 4 — Architecture-test coverage diff --git a/.claude/skills/wire-dapr-pubsub/SKILL.md b/.claude/skills/wire-dapr-pubsub/SKILL.md index fedc89f4..1b3e9724 100644 --- a/.claude/skills/wire-dapr-pubsub/SKILL.md +++ b/.claude/skills/wire-dapr-pubsub/SKILL.md @@ -21,7 +21,11 @@ ADR-0035's Phase 11 trigger has fired, also wire the Dapr adapter without changi that contract. The wiring contract is in [ADR-0038](../../../docs/decisions/0038-cross-cutting-port-and-event-contracts.md) and [29-dapr-integration.md](../../../docs/architecture/29-dapr-integration.md); -this skill is the **mechanical** check-list. +this skill is the **mechanical** check-list. Accepted +[ADR-0046](../../../docs/decisions/0046-durable-event-delivery-and-recovery.md) +adds construction-free stable subscription identities, Pending/Active readiness +and durable acceptance. Those consumer components remain P02b-3 work; Phase 11 +must preserve their contracts rather than reintroduce handler-owned deduplication. ## When to use @@ -126,7 +130,8 @@ await outbox.EnqueueAsync(new EnrollmentCreatedIntegrationEventV1 { ... }, ct); await db.SaveChangesAsync(ct); ``` -The `OutboxProcessor` (BackgroundService) polls `outbox_messages`, constructs an +The API-host `OutboxProcessor` BackgroundService implemented in P02b-2 Step 3 polls +`outbox_messages`, constructs an `IntegrationEventEnvelope`, and calls `IEventBus.PublishAsync(envelope)`. `DaprEventBus.PublishAsync` publishes to `envelope.Topic` with the envelope's delivery fields as metadata — see @@ -147,28 +152,40 @@ architecture test; the transport never re-derives or renames it. `InProcessEventBus` discovers handler types from assemblies passed at the composition root. Ensure the consumer assembly is included and let the registry -register the concrete handler type: +register the concrete handler type. This abridged assembly-registration example +uses the shipped public module marker; retain the other module assemblies in the +actual composition root: ```csharp builder.AddLearnStackCrossCuttingFoundation( deploymentMode, - typeof(CreateAuditEntryOnUserDeleted).Assembly); + typeof(LearnStack.Modules.Customization.Application.AssemblyMarker).Assembly); ``` There is no shipped `AddDaprSubscription` helper or `[Topic]` attribute. Do not invent one. The event declares its topic and the construction-free handler registry supplies the current subscription metadata. -The future Phase 11 subscription pipeline must preserve this behavior: +P02b-2 implements the declared manifest: stable versioned subscription IDs, compatible +aliases, and required Pending/Active readiness. Pending subscriptions hold producer +rows before acquisition. P02b-3 adds durable acceptance: Active publish returns only +after every matching delivery is durably accepted. A separate executor owns +independent per-subscription retries, +terminal state and targeted replay. Neither publication nor replay invokes healthy +sibling business handlers again. No current registration helper is claimed here. + +The future Phase 11 subscription pipeline must preserve the accepted behavior: 1. The sidecar **discovers** subscriptions with `GET /dapr/subscribe`, which returns the topic-to-route table; it then **delivers** each event with an HTTP `POST` to the route that table named. They are two different calls, and there is no endpoint called `/dapr/subscribe-endpoint`. -2. LearnStack's Phase 11 Dapr adapter deserialises the envelope, restores - `TenantContext` from the event and envelope, and resolves the matching - `IIntegrationEventHandler` directly from DI. -3. The `IIntegrationEventHandler` runs with `IInboxGuard` protection. +2. The adapter validates the envelope and durably accepts deliveries before + acknowledging the broker; transport acceptance is separate from business work. +3. The executor opens the common app-role transaction, restores tenant and + envelope organization, and acquires the shared inbox claim before resolving + `IIntegrationEventHandler`. Inbox completion, business state, produced + outbox, MUST audit and fenced delivery completion commit atomically. ### Step 5: Current transport (`InProcessEventBus`) @@ -229,20 +246,19 @@ await daprClient.PublishByteEventAsync( }); ``` -Kafka guarantees ordering **within a partition**. Cross-partition ordering does -not exist; design around it. +ADR-0046 guarantees deterministic eligible selection by availability, occurrence +and ID within one processor. It promises no enqueue, commit, retry or cross-process +order. Kafka partition ordering cannot strengthen the producer guarantee; document +any future broker-specific bound in Phase 11. ### Step 8: Observability -Phase 11 must add and verify the transport metrics governed by the observability -standard; they are not automatic in the current in-process implementation: - -- `learnstack_outbox_dispatch_duration_seconds{event_type}` -- `learnstack_outbox_dispatch_failed_total{event_type}` -- `learnstack_inbox_dedup_total{module, event_type}` - -Keep tags low-cardinality and confirm the names against the implementation and -dashboard when the adapter lands. +[Observability Standards](../../../docs/standards/10-observability.md#required-metrics) +owns event metric names, bounded dimensions, snapshot gauges and health deadlines. +P02b-2 implements producer dispatch diagnostics; P02b-3 implements subscriber +execution/dedup diagnostics and minimizes the existing transport diagnostics. +Phase 11 adds adapter binding evidence. Do not move these producer/subscriber +obligations to the adapter or infer delivery from a registered metric name. ## Validation @@ -253,8 +269,9 @@ dashboard when the adapter lands. - When doing Phase 02b consumer work, add/keep its registered inheritance and inbox-guard rules. When doing Phase 11 adapter work, add/keep the Dapr binding and direct-provider boundary rules owned by that phase. -- Today, an in-process integration test confirms publish → dispatch → consume → - inbox marker and the current transport's consumer span. +- The current baseline has no durable inbox. P02b-3 must prove acceptance plus + atomic inbox/business completion, concurrent duplicates, independent sibling + retries, sticky timeout and uncertain-COMMIT recovery on the real delivery path. - When Phase 11's adapter exists, its Dapr/Testcontainers binding test and transport metrics pass as additional validation. @@ -275,9 +292,9 @@ dashboard when the adapter lands. - **Switching on deployment mode before the trigger.** All modes use `InProcessEventBus` today. Add the Dapr branch only with the Phase 11 adapter and its integration suite. -- **Ordering assumption across topics.** Kafka does not order across topics or - cross-partition within a topic. If your design needs strict ordering, use - partition keys and document the assumption. +- **Assuming partition keys prove publish order.** The producer promises only + deterministic eligible selection. Retries and concurrent commits can reorder + delivery even when the future broker preserves its own partition order. - **Component YAML drift between dev and prod.** Keep one canonical YAML in `infra/dapr/components/`; environment overrides only via Vault-injected metadata fields (broker URL, auth). diff --git a/.env.example b/.env.example index f72ac3e7..67e25271 100644 --- a/.env.example +++ b/.env.example @@ -56,7 +56,7 @@ LEARNSTACK_OUTBOX_PW=learnstack-outbox-dev # FORCE ROW LEVEL SECURITY exists to defeat. # Default — every request-path and job-path DbContext. NOBYPASSRLS. # PlatformAdmin — resolvable only by PlatformAdminScope. BYPASSRLS. -# OutboxDispatcher — the worker running OutboxProcessor, and nothing else. +# OutboxDispatcher — API-owned OutboxProcessor only. Never Seeder or web. ConnectionStrings__Migration='Host=localhost;Port=5432;Database=learnstack;Username=learnstack_migration;Password=learnstack-migration-dev' ConnectionStrings__Default='Host=localhost;Port=5432;Database=learnstack;Username=learnstack_app;Password=learnstack-app-dev' ConnectionStrings__PlatformAdmin='Host=localhost;Port=5432;Database=learnstack;Username=learnstack_platform;Password=learnstack-platform-dev' @@ -109,7 +109,8 @@ ASPNETCORE_URLS=http://localhost:5080 # ─── Public renderer (ADR-0053; API and web share this private source) ─── # make public-env fills the empty secret with 32 random bytes encoded as ASCII. # No TLS/hosts/trust-store changes are automatic. See README Quickstart. -# make public-api loads only runtime DB credentials plus network AND secret. +# make public-api loads application/platform-admin/dispatcher DB credentials +# plus network AND secret. # pnpm dev/start loads these values; an optional web .env.local must match. LEARNSTACK_PUBLIC_API_ORIGIN=http://127.0.0.1:5080 LEARNSTACK_PUBLIC_HOP_SECRET= diff --git a/CLAUDE.md b/CLAUDE.md index aa206297..4acb64d5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -179,9 +179,26 @@ records passing verification and both completed review rounds. Step 4 adds production-census guards and catalogue enforcement; its [delivery record](docs/roadmap/phase-02b-events-auth.md#p02b-1-step-4--producer-enforcement-and-closeout-2026-10-12) records passing full backend verification and both completed review rounds. -**P02b-1 is complete; PR review and merge remain pending.** The -[packet closeout](docs/roadmap/phase-02b-events-auth.md#p02b-1-packet-closeout-2026-10-12) -owns delivery scope and the next P02b-2 decision pass. +**P02b-1 is complete and merged** through +[PR #29](https://github.com/HodeTech/LearnStack/pull/29) on 2026-10-12. The +[merge closeout](docs/roadmap/phase-02b-events-auth.md#p02b-1-merge-and-closeout-2026-10-12) +owns final verification. **P02b-2's revised decision package and ADR-0046 are +Accepted — 2026-10-12.** The four implementation steps and two independent review +rounds per step are authorized on development. The +[Step 1 record](docs/roadmap/phase-02b-events-auth.md#p02b-2-step-1--decision-schema-and-confined-composition-2026-10-12) +records both completed Step 1 reviews. Step 2 implements fenced transitions and +row-bound terminal MUST audit; both independent review rounds pass. +Step 3 implements hosted execution and observation; its +[delivery record](docs/roadmap/phase-02b-events-auth.md#p02b-2-step-3--hosted-lifecycle-and-observability-2026-10-12) +owns verification and both completed review rounds. Step 4 enforcement and local +closeout are complete; all four steps completed both independent review rounds. +The +[delivery record](docs/roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) +owns local closeout evidence and CI/merge status. +Acceptance registers six new catalogue rules; the Step 4 record owns their +delivery evidence. P02b-2 claims no inbox or subscriber delivery. P02b-3 still owns +durable acceptance, shared inbox/ledger, independent retries, both real default +subscriptions and Seeder handoff. Phase 06 retains the full renderer/Studio expansion. **Phase 01** shipped the .NET 10 solution scaffold under `backend/` @@ -381,11 +398,14 @@ For any task, read in this order: Then read the phase that is live and the one it stands on: - [docs/roadmap/phase-02b-events-auth.md](docs/roadmap/phase-02b-events-auth.md) - — **the live phase**. P02b-0 entry verification is complete; P02b-1's decision - package is Accepted — 2026-10-12; Steps 1–3 implement the producer runtime, - and Step 4 supplies production-census enforcement and packet closeout. - Each later packet accepts its own gates - before implementation. + — **the live phase**. P02b-0 entry verification is complete; P02b-1 is complete + and merged through PR #29 on 2026-10-12. Its four steps deliver the producer + runtime and production-census enforcement. P02b-2's revised package/ADR-0046 are + Accepted. Steps 1–2 completed both review rounds; Step 3 implements hosted + dispatch and observation, with both independent review rounds complete. Step 4 + enforcement and local closeout are complete; all four steps completed both + independent review rounds; the Step 4 record above owns CI/merge status. + P02b-3 owns consumer/inbox delivery; each later packet accepts its gates first. - [docs/roadmap/phase-02d-walking-skeleton.md](docs/roadmap/phase-02d-walking-skeleton.md) — complete and merged; the delivered public-site baseline and P02b-0 handoff. `02d` sorts after `02b`/`02c` but runs **before** them; the roadmap dependency @@ -630,9 +650,12 @@ rules: `ISecretProvider`. - Read `DeploymentMode` from inside a module — the composition root branches once, modules never. -- Write `audit_log`, `platform_entitlement_cache`, or `outbox_messages` - directly — use `IAuditStore`, `IEntitlementProvider.RefreshAsync`, - `IOutbox`. +- Write `audit_log`, `platform_entitlement_cache`, or producer `outbox_messages` + rows directly — use `IAuditStore`, `IEntitlementProvider.RefreshAsync`, `IOutbox`. + Only ADR-0046's confined dispatcher writes outbox status through fenced + parameterized SQL; its terminal UPDATE and narrow witness-bound audit INSERT + share one transaction. P02b-2 Step 2 implements that accepted exception; both + independent review rounds passed. - Write rows through EF Core's set-based APIs — `ExecuteUpdate`, `ExecuteDelete`, `ExecuteSql*`. The audit capture sees only what the `ChangeTracker` holds ([ADR-0044 § 7](docs/decisions/0044-audit-write-path.md)), so such a write commits with diff --git a/README.md b/README.md index 57c56b11..e840ca06 100644 --- a/README.md +++ b/README.md @@ -67,10 +67,18 @@ items, rules, custom fields and notification templates; their delivery is tracke **Phase 01, Phase 02a and Phase 02d are complete.** Phase 02d is merged through PR #28 on 2026-10-11. P02b-0's [entry audit](docs/roadmap/phase-02b-events-auth.md#p02b-0-entry-verification-2026-10-12) -is complete. **P02b-1 is complete; PR review and merge remain pending.** Its -[packet closeout](docs/roadmap/phase-02b-events-auth.md#p02b-1-packet-closeout-2026-10-12) -records transactional event production, enforcement and both review rounds per -step under Accepted ADR-0057. +is complete. **P02b-1 is complete and merged** through +[PR #29](https://github.com/HodeTech/LearnStack/pull/29) on 2026-10-12. Its +[merge closeout](docs/roadmap/phase-02b-events-auth.md#p02b-1-merge-and-closeout-2026-10-12) +records transactional event production, enforcement, completed reviews and final +CI verification under Accepted ADR-0057. P02b-2's revised package and ADR-0046 +are Accepted. Steps 1–2 completed both review rounds; Step 3 implements hosted +dispatch and observation, with both independent review rounds complete. Its +[delivery record](docs/roadmap/phase-02b-events-auth.md#p02b-2-step-3--hosted-lifecycle-and-observability-2026-10-12) +owns its runtime evidence. Step 4 enforcement and local closeout are complete; +all four steps completed both independent review rounds. CI and merge status +live in the +[delivery record](docs/roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12). [P02d-1](docs/roadmap/phase-02d-walking-skeleton.md#merge-and-closeout-2026-09-14) is **complete and merged**: Education domain, schema and isolation proofs. **P02d-2's decision package and ADR-0050/0051 are Accepted** as of 2026-10-02, @@ -120,8 +128,10 @@ proof pass real CI; the seventh required context is active. The owns CI enforcement, reviews and the positive phase-exit verdict. [PR #28](https://github.com/HodeTech/LearnStack/pull/28) is merged; the [merge closeout](docs/roadmap/phase-02d-walking-skeleton.md#p02d-7-merge-and-closeout-2026-10-11) -owns final verification. P02b-0 and P02b-1 are complete; dispatcher and consumer -work remain P02b-2/P02b-3. The +owns final verification. P02b-0 and P02b-1 are complete; P02b-2 producer dispatch +and enforcement are locally complete, with both independent review rounds complete +for all four steps; the Step 4 record above owns CI/merge status. Consumer +delivery remains P02b-3. The [P02d-4 decision package](docs/roadmap/phase-02d-walking-skeleton.md#p02d-4-decision-package-2026-10-03) is Accepted — 2026-10-03. All four implementation steps and their review loops are complete. P02d-5 supplies trusted server rendering; P02d-6 supplies public @@ -133,7 +143,7 @@ page consumers and P02d-7 the two-host browser demo. | **Customization** | Content types, level taxonomies, exact-definition and generation-cached batched display readers, text-card metadata validation and tenant-authored seeds | Remaining authoring capabilities across [Phases 04–08a](docs/roadmap/README.md) | | **Audit** | Classified write path and transactional durability for business changes | Operational hardening in [Phase 11](docs/roadmap/phase-11-production-hardening.md) | | **Education** | Course and Lesson aggregates, translations, protected-content policy, scoped authoring commands, complete demo seeds and isolated public GET/HEAD reads | Full versioned curriculum in [Phase 05](docs/roadmap/phase-05-education-learning-content.md) | -| **API foundation** | Error contracts, validation, tenancy, concurrency, observability and transactional event production | Authentication and durable event dispatch/consumption in [Phase 02b](docs/roadmap/phase-02b-events-auth.md) | +| **API foundation** | Error contracts, validation, tenancy, concurrency, observability, transactional event production and hosted producer dispatch with production-census enforcement (all four P02b-2 steps and both review rounds locally complete) | Authentication and durable event consumption in [Phase 02b](docs/roadmap/phase-02b-events-auth.md) | | **Frontend** | Native TLS ingress, live locale entry, trusted server caller, localized catalog/course/lesson pages, safe themes, accessibility controls/manual smoke and generated typed SDK | Full renderer/Studio and Playwright Test/axe in [Phase 06](docs/roadmap/phase-06-renderer-admin-studio.md) | **Four modules contain domain implementations:** Tenancy, Customization, Audit and @@ -212,6 +222,17 @@ foreign listener is refused; no foreign process is killed. If a hard kill leaves `.data/demo.lock`, confirm the earlier process tree has stopped before removing that lock manually. Missing/stale configuration or TLS is refused without reset. +Local demo database validation accepts only `Host`, `Port`, `Database`, +`Username` and `Password`, with case-insensitive keys. Supported aliases are +`Server` for `Host`; `User Name`, `Userid`, `User Id` and `Uid` for `Username`; +and `Pwd` and `Psw` for `Password`. Single- or double-quoted values preserve +whitespace and semicolons; double the matching quote to escape it. Duplicate +canonical fields and unsupported keys are refused. Host must be `localhost` or +`127.0.0.1`; port defaults to 5432 and must resolve to 5432. Role, password and +database must match the root configuration. This is the closed local demo +grammar, not general Npgsql connection-string support; +[demo-database.mjs](scripts/demo-database.mjs) owns validation. + For separate development/watch terminals, retain `make public-api` and `make public-web`. The CI lifecycle uses an isolated checkout/project and disposable configuration through `make demo DEMO_MODE=ci`. Install its locked managed browser @@ -225,7 +246,10 @@ Readiness is for the API and for the web listener. `pnpm dev/start` also use this launcher. Both processes read the same private root source; a web `.env.local` projection is optional and must match. No shell evaluates the values. Ordinary -no-hop API startup remains supported outside this paired renderer recipe. +no-hop API startup remains supported outside this paired renderer recipe; supply +both `ConnectionStrings:Default` (`learnstack_app`) and +`ConnectionStrings:OutboxDispatcher` (`learnstack_outbox_admin`). The mandatory +API worker verifies dispatcher connectivity at startup. **P02d-5 is complete: ingress, visitor budgets and public entry.** The web bootstraps the captured host through the API, redirects using enabled/default diff --git a/backend/src/LearnStack.Api/Composition/OutboxCompositionExtensions.cs b/backend/src/LearnStack.Api/Composition/OutboxCompositionExtensions.cs new file mode 100644 index 00000000..4ab9d192 --- /dev/null +++ b/backend/src/LearnStack.Api/Composition/OutboxCompositionExtensions.cs @@ -0,0 +1,55 @@ +using LearnStack.Api.Messaging; +using LearnStack.Infrastructure.Messaging; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Time; +using Microsoft.Extensions.DependencyInjection.Extensions; +using Npgsql; + +namespace LearnStack.Api.Composition; + +/// Owns the dispatcher credential in the API composition root. +public static class OutboxCompositionExtensions +{ + public static IServiceCollection AddLearnStackOutboxDispatcher( + this IServiceCollection services, IConfiguration configuration) + { + ArgumentNullException.ThrowIfNull(services); + ArgumentNullException.ThrowIfNull(configuration); + + var connectionString = configuration.GetConnectionString(OutboxDispatcherDataSource.ConnectionName); + if (!string.IsNullOrWhiteSpace(connectionString)) + { + OutboxDispatcherDataSource.Validate(connectionString); + } + + // The keyed source is container-owned. Lazy defers construction while + // construction-only test hosts remove the production worker; the worker's + // startup must resolve and open it, including when configuration is absent. + services.TryAddKeyedSingleton( + OutboxDispatcherDataSource.Key, + (_, _) => OutboxDispatcherDataSource.Build(connectionString)); + services.TryAddKeyedSingleton>( + OutboxDispatcherDataSource.Key, + (provider, key) => new Lazy( + () => provider.GetRequiredKeyedService(key))); + + services.TryAddSingleton(new OutboxDispatchOptions()); + services.TryAddSingleton(TimeProvider.System); + services.TryAddSingleton(); + services.TryAddSingleton(); + services.TryAddSingleton(provider => new OutboxDispatchStore( + provider.GetRequiredKeyedService>(OutboxDispatcherDataSource.Key), + provider.GetRequiredService(), + provider.GetRequiredService(), + provider.GetRequiredService(), + provider.GetRequiredService(), + provider.GetRequiredService())); + if (!services.Any(descriptor => descriptor.ServiceType == typeof(IHostedService) + && descriptor.ImplementationType == typeof(OutboxProcessor))) + { + services.AddHostedService(); + services.AddHealthChecks().AddCheck(OutboxHealthCheck.Name); + } + return services; + } +} diff --git a/backend/src/LearnStack.Api/Composition/PersistenceCompositionExtensions.cs b/backend/src/LearnStack.Api/Composition/PersistenceCompositionExtensions.cs index dde24e4c..ac3aca77 100644 --- a/backend/src/LearnStack.Api/Composition/PersistenceCompositionExtensions.cs +++ b/backend/src/LearnStack.Api/Composition/PersistenceCompositionExtensions.cs @@ -2,6 +2,7 @@ using LearnStack.Modules.Customization.Application.Contracts.Definitions; using LearnStack.Modules.Tenancy.Application.Contracts.Locales; using LearnStack.Modules.Tenancy.Application.Contracts.PublicReads; +using LearnStack.Modules.Tenancy.Application.Contracts.IntegrationEvents; using LearnStack.Modules.Education.Application.Audit; using LearnStack.Infrastructure.Audit; using LearnStack.Modules.Customization.Application.Audit; @@ -34,7 +35,7 @@ namespace LearnStack.Api.Composition; /// /// /// -/// One data source, and it must be the application role's. +/// Module contexts share the application-role data source. /// ConnectionStrings:Default names learnstack_app, which is /// NOBYPASSRLS and holds USAGE but not CREATE on schema /// public. ConnectionStrings:Migration is never read here — that @@ -61,8 +62,8 @@ namespace LearnStack.Api.Composition; /// /// Built lazily; validated eagerly when there is anything to validate. /// The data source is a singleton whose factory runs when something first needs a -/// connection, so a WebApplicationFactory test — and a deployment serving -/// only platform hosts — carries no database it does not use. The two checks above +/// connection, so a construction-only WebApplicationFactory test carries +/// no application database it does not use. The two checks above /// are not deferred with it: a present ConnectionStrings:Default is /// name-checked at AddLearnStackPersistence time, because a string naming /// learnstack_migration is precisely the ownership mistake this guard exists @@ -83,15 +84,16 @@ namespace LearnStack.Api.Composition; /// resolution boundary. /// /// -/// ConnectionStrings:OutboxDispatcher is still deliberately absent; it lands -/// with its consumer in Phase 02b. +/// ConnectionStrings:OutboxDispatcher is keyed and lazy through the separate +/// confined outbox composition. Its mandatory worker startup verifies connectivity; +/// module contexts and the Seeder never resolve it (ADR-0046). /// /// public static class PersistenceCompositionExtensions { private const string DefaultConnectionName = "Default"; - /// The one role a runtime process may connect as. + /// The role required by the application data source. internal const string RuntimeRole = ApplicationDataSource.RuntimeRole; public static IServiceCollection AddLearnStackPersistence( @@ -100,6 +102,15 @@ public static IServiceCollection AddLearnStackPersistence( ArgumentNullException.ThrowIfNull(services); ArgumentNullException.ThrowIfNull(configuration); + services.AddLearnStackOutboxDispatcher(configuration); + services.TryAddSingleton(provider => OutboxEventRegistry.Create( + [OutboxEventRegistration.Required( + typeof(TenantCreatedIntegrationEventV1), + [OutboxSubscriptionManifest.Pending("customization.initialize-content-type-default.v1"), + OutboxSubscriptionManifest.Pending("customization.initialize-taxonomy-default.v1")])], + [], provider.GetRequiredService(), + [typeof(TenantCreatedIntegrationEventV1).Assembly])); + var connectionString = configuration.GetConnectionString(DefaultConnectionName); // Validated at boot when the key is present; built on first use either way. diff --git a/backend/src/LearnStack.Api/Messaging/OutboxHealthCheck.cs b/backend/src/LearnStack.Api/Messaging/OutboxHealthCheck.cs new file mode 100644 index 00000000..3670f954 --- /dev/null +++ b/backend/src/LearnStack.Api/Messaging/OutboxHealthCheck.cs @@ -0,0 +1,14 @@ +using Microsoft.Extensions.Diagnostics.HealthChecks; + +namespace LearnStack.Api.Messaging; + +/// Private readiness observation; the public liveness route does not select this check. +public sealed class OutboxHealthCheck(OutboxObservation observation) : IHealthCheck +{ + public const string Name = "outbox"; + + public Task CheckHealthAsync(HealthCheckContext context, + CancellationToken cancellationToken = default) => Task.FromResult(observation.IsHealthy + ? HealthCheckResult.Healthy() + : HealthCheckResult.Unhealthy("Outbox dispatch is not healthy.")); +} diff --git a/backend/src/LearnStack.Api/Messaging/OutboxObservation.cs b/backend/src/LearnStack.Api/Messaging/OutboxObservation.cs new file mode 100644 index 00000000..0c471f8d --- /dev/null +++ b/backend/src/LearnStack.Api/Messaging/OutboxObservation.cs @@ -0,0 +1,132 @@ +using System.Collections.Frozen; +using System.Diagnostics; +using System.Diagnostics.Metrics; +using LearnStack.Infrastructure.Messaging; + +namespace LearnStack.Api.Messaging; + +/// Immutable process observation; timestamps use TimeProvider's monotonic clock. +public sealed record OutboxObservationSnapshot( + OutboxQueueSnapshot Queue, + bool Started = false, + bool Stopped = false, + long? LastProgressTimestamp = null, + long? LastSnapshotTimestamp = null, + int OutstandingPublishes = 0, + bool RefreshFailed = false, + bool LoopFailed = false, + bool TerminalAuditFailed = false); + +/// Bounded dispatch telemetry; observable callbacks only read memory. +public sealed class OutboxObservation : IDisposable +{ + public const string InstrumentationName = "learnstack.outbox"; + private readonly object _gate = new(); + private readonly TimeProvider _time; + private readonly OutboxDispatchOptions _options; + private readonly FrozenSet _labels; + private readonly Meter _meter = new(InstrumentationName); + private readonly ActivitySource _activities = new(InstrumentationName); + private readonly Histogram _duration; + private readonly Counter _failed; + private readonly Counter _deadletter; + private readonly Counter _leaseLost; + private OutboxObservationSnapshot _snapshot = new(new OutboxQueueSnapshot(0, 0, 0)); + + public OutboxObservation(TimeProvider time, OutboxDispatchOptions options, OutboxEventRegistry registry) + { + ArgumentNullException.ThrowIfNull(time); + ArgumentNullException.ThrowIfNull(options); + ArgumentNullException.ThrowIfNull(registry); + _time = time; + _options = options; + _labels = registry.EventTypeLabels.Concat(["unknown", "retired", "invalid"]) + .ToFrozenSet(StringComparer.Ordinal); + _duration = _meter.CreateHistogram("learnstack_outbox_dispatch_duration_seconds", "s"); + _failed = _meter.CreateCounter("learnstack_outbox_dispatch_failed_total"); + _deadletter = _meter.CreateCounter("learnstack_outbox_deadletter_total"); + _leaseLost = _meter.CreateCounter("learnstack_outbox_lease_lost_total"); + _meter.CreateObservableGauge("learnstack_outbox_pending_count", () => Snapshot.Queue.PendingCount); + _meter.CreateObservableGauge("learnstack_outbox_oldest_eligible_age_seconds", + () => Snapshot.Queue.OldestEligibleAgeSeconds, "s"); + _meter.CreateObservableGauge("learnstack_outbox_deferred_count", () => Snapshot.Queue.DeferredCount); + _meter.CreateObservableGauge("learnstack_outbox_heartbeat_age_seconds", () => HeartbeatAge.TotalSeconds, "s"); + } + + public OutboxObservationSnapshot Snapshot => Volatile.Read(ref _snapshot); + + public TimeSpan HeartbeatAge => Age(Snapshot.LastProgressTimestamp); + + public bool IsHealthy + { + get + { + var snapshot = Snapshot; + return snapshot.Started && !snapshot.Stopped && !snapshot.RefreshFailed + && !snapshot.LoopFailed && !snapshot.TerminalAuditFailed + && snapshot.OutstandingPublishes == 0 + && Age(snapshot.LastProgressTimestamp) < _options.ProgressTimeout + && Age(snapshot.LastSnapshotTimestamp) < _options.ProgressTimeout; + } + } + + internal void Started() => Update(value => value with { Started = true, Stopped = false }); + internal void Stopped() => Update(value => value with { Stopped = true }); + internal void Progress() => Update(value => value with + { + LastProgressTimestamp = _time.GetTimestamp(), + LoopFailed = false, + }); + internal void Refreshed(OutboxQueueSnapshot queue) => Update(value => value with + { + Queue = queue, + LastSnapshotTimestamp = _time.GetTimestamp(), + RefreshFailed = false, + }); + internal void RefreshFailed() => Update(value => value with { RefreshFailed = true }); + internal void LoopFailed() => Update(value => value with { LoopFailed = true }); + internal void TerminalAuditFailed() => Update(value => value with { TerminalAuditFailed = true }); + internal void TerminalAuditSucceeded() => Update(value => value with { TerminalAuditFailed = false }); + internal void Outstanding(int count) => Update(value => value with { OutstandingPublishes = count }); + internal void LeaseLost() => _leaseLost.Add(1); + internal void Failed(string label) => _failed.Add(1, EventType(label)); + internal void Deadletter(string label) => _deadletter.Add(1, EventType(label)); + internal void Duration(string label, TimeSpan duration) => + _duration.Record(Math.Max(0, duration.TotalSeconds), EventType(label)); + + internal Activity BeginAttempt(string correlation, string label) + { + var valid = ActivityContext.TryParse(correlation, traceState: null, out var parent); + var activity = _activities.StartActivity("outbox.dispatch", ActivityKind.Producer, parent); + if (activity is null) + { + // Audit needs a fresh valid attempt trace even with no telemetry listener. + activity = new Activity("outbox.dispatch").SetIdFormat(ActivityIdFormat.W3C); + if (valid) activity.SetParentId(parent.TraceId, parent.SpanId, parent.TraceFlags); + activity.Start(); + } + + activity.SetTag("event_type", SafeLabel(label)); + // Reconstruct the bounded identity; a future-version traceparent may + // contain additional fields that must not become diagnostic data. + if (valid) activity.SetTag("correlation.id", + $"00-{parent.TraceId}-{parent.SpanId}-{((parent.TraceFlags & ActivityTraceFlags.Recorded) != 0 ? "01" : "00")}"); + return activity; + } + + private KeyValuePair EventType(string label) => new("event_type", SafeLabel(label)); + private string SafeLabel(string label) => _labels.Contains(label) ? label : "invalid"; + private TimeSpan Age(long? timestamp) => timestamp is { } value + ? _time.GetElapsedTime(value, _time.GetTimestamp()) : _options.ProgressTimeout; + + private void Update(Func change) + { + lock (_gate) Volatile.Write(ref _snapshot, change(_snapshot)); + } + + public void Dispose() + { + _activities.Dispose(); + _meter.Dispose(); + } +} diff --git a/backend/src/LearnStack.Api/Messaging/OutboxProcessor.cs b/backend/src/LearnStack.Api/Messaging/OutboxProcessor.cs new file mode 100644 index 00000000..1e42450c --- /dev/null +++ b/backend/src/LearnStack.Api/Messaging/OutboxProcessor.cs @@ -0,0 +1,335 @@ +using System.Diagnostics; +using LearnStack.Infrastructure.Messaging; +using Npgsql; + +namespace LearnStack.Api.Messaging; + +/// One just-in-time claimant with bounded, task-owned publication overruns. +public sealed partial class OutboxProcessor : BackgroundService +{ + private readonly Lazy _source; + private readonly OutboxDispatchStore _store; + private readonly OutboxEnvelopeReader _reader; + private readonly OutboxEventRegistry _registry; + private readonly OutboxDispatchOptions _options; + private readonly TimeProvider _time; + private readonly IServiceScopeFactory _scopes; + private readonly OutboxObservation _observation; + private readonly ILogger _logger; + private readonly object _attemptGate = new(); + private readonly List _attempts = []; + + public OutboxProcessor( + [FromKeyedServices(OutboxDispatcherDataSource.Key)] Lazy source, + OutboxDispatchStore store, OutboxEnvelopeReader reader, OutboxEventRegistry registry, + OutboxDispatchOptions options, TimeProvider time, IServiceScopeFactory scopes, + OutboxObservation observation, ILogger logger) + { + ArgumentNullException.ThrowIfNull(source); + ArgumentNullException.ThrowIfNull(store); + ArgumentNullException.ThrowIfNull(reader); + ArgumentNullException.ThrowIfNull(registry); + ArgumentNullException.ThrowIfNull(options); + ArgumentNullException.ThrowIfNull(time); + ArgumentNullException.ThrowIfNull(scopes); + ArgumentNullException.ThrowIfNull(observation); + ArgumentNullException.ThrowIfNull(logger); + _source = source; + _store = store; + _reader = reader; + _registry = registry; + _options = options; + _time = time; + _scopes = scopes; + _observation = observation; + _logger = logger; + } + + public override async Task StartAsync(CancellationToken cancellationToken) + { + // .NET 10 runs ExecuteAsync in the background. Startup refusal therefore + // belongs here, before the host can report that it has started. + try + { + _options.Validate(); + using var timeout = new CancellationTokenSource(_options.SqlTimeout, _time); + using var deadline = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, timeout.Token); + await using var connection = await _source.Value.OpenConnectionAsync(deadline.Token).ConfigureAwait(false); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw new OperationCanceledException(cancellationToken); + } + catch (Exception) + { + // No connection data, provider exception or inner exception escapes boot. + throw new InvalidOperationException("Outbox dispatcher startup was refused. " + + "Configure ConnectionStrings:OutboxDispatcher in Npgsql key/value form " + + "for learnstack_outbox_admin and verify database connectivity."); + } + + _observation.Started(); + await base.StartAsync(cancellationToken).ConfigureAwait(false); + } + + protected override async Task ExecuteAsync(CancellationToken stoppingToken) + { + try + { + while (!stoppingToken.IsCancellationRequested) + { + var blocked = RetainedPartitions(); + var didWork = false; + try + { + if (OutstandingCount < _options.MaxOutstandingPublishes) + { + // Terminal requests and expired acquisitions never spend + // another publication attempt, and take priority over new work. + var claim = await _store.TryClaimTerminalAsync(blocked, stoppingToken).ConfigureAwait(false) + ?? await _store.TryAcquireRecoveryAsync(blocked, stoppingToken).ConfigureAwait(false) + ?? await _store.TryClaimAsync(blocked, stoppingToken).ConfigureAwait(false); + if (claim is not null) + { + didWork = true; + await DispatchAsync(claim, stoppingToken).ConfigureAwait(false); + } + + // Even an empty completed poll demonstrates real bounded work. + _observation.Progress(); + } + + await RefreshAsync(force: false, stoppingToken).ConfigureAwait(false); + } + catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested) + { + break; + } + catch (Exception) + { + _observation.LoopFailed(); + LogIterationFailure(_logger); + didWork = false; + } + + if (!didWork) + await Task.Delay(_options.IdlePoll, _time, stoppingToken).ConfigureAwait(false); + } + } + catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested) + { + // Normal hosted-service cancellation; live attempts own their cleanup. + } + finally + { + CancelAttempts(); + _observation.Stopped(); + } + } + + private async Task DispatchAsync(OutboxDispatchClaim claim, CancellationToken stoppingToken) + { + if (claim.Kind == OutboxClaimKind.Normal) + { + await DispatchNormalAsync(claim, stoppingToken).ConfigureAwait(false); + return; + } + + var label = claim.Kind == OutboxClaimKind.Terminal + ? _reader.Read(claim).EventTypeLabel : _registry.Resolve(claim.Type).EventTypeLabel; + using var activity = _observation.BeginAttempt(claim.CorrelationId, label); + if (claim.Kind == OutboxClaimKind.Recovery) + { + Fence(await _store.FailAsync(claim, OutboxFailureReason.UnknownOutcome, stoppingToken).ConfigureAwait(false)); + activity.SetTag("outcome", "recovered"); + LogAttempt(activity, label); + return; + } + + try + { + // BeginAttempt always starts a W3C Activity, including without listeners. + var terminalized = await _store.TryTerminalizeAsync(claim, activity.Id!, stoppingToken).ConfigureAwait(false); + Fence(terminalized); + if (terminalized) + { + _observation.TerminalAuditSucceeded(); + _observation.Deadletter(label); + } + activity.SetTag("outcome", terminalized ? "deadlettered" : "lease_lost"); + } + catch (Exception) + { + _observation.TerminalAuditFailed(); + activity.SetTag("outcome", "terminal_audit_failed"); + throw; + } + finally + { + LogAttempt(activity, label); + } + } + + private async Task DispatchNormalAsync(OutboxDispatchClaim claim, CancellationToken stoppingToken) + { + var read = _reader.Read(claim); + using var activity = _observation.BeginAttempt(claim.CorrelationId, read.EventTypeLabel); + var started = _time.GetTimestamp(); + try + { + if (read.TerminalReason is { } terminal) + { + Fence(await _store.RequestTerminalAsync(claim, terminal, stoppingToken).ConfigureAwait(false)); + activity.SetTag("outcome", "terminal_requested"); + return; + } + + if (read.Disposition == OutboxEventDisposition.IntentionallyUnsubscribed) + { + Fence(await _store.CompleteAsync(claim, stoppingToken).ConfigureAwait(false)); + activity.SetTag("outcome", "intentionally_unsubscribed"); + return; + } + + // The immutable registry excludes Pending identities in SQL, before + // acquisition. Refuse publication if this invariant ever changes. + if (read.Disposition != OutboxEventDisposition.Active || read.Envelope is null) + throw new InvalidOperationException("Outbox publication was not admitted."); + + var attempt = new OutboxPublishAttempt(_scopes, read.Envelope); + lock (_attemptGate) _attempts.Add(attempt); + bool succeeded; + try + { + succeeded = await attempt.Completion.WaitAsync(_options.PublishTimeout, _time, stoppingToken) + .ConfigureAwait(false); + } + catch (TimeoutException) + { + attempt.RequestCancellation(); + RetainedPartitions(); + _observation.Failed(read.EventTypeLabel); + activity.SetTag("outcome", "publish_timeout"); + // This is the only durable timeout transition. A late callback's + // result is observed solely by its owned task, never acknowledged. + Fence(await _store.FailAsync(claim, OutboxFailureReason.PublishTimeout, stoppingToken).ConfigureAwait(false)); + return; + } + catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested) + { + attempt.RequestCancellation(); + RetainedPartitions(); + activity.SetTag("outcome", "shutdown"); + // Expired-lease recovery owns this unknown result after shutdown. + throw; + } + + RetainedPartitions(); + stoppingToken.ThrowIfCancellationRequested(); + if (succeeded) + { + var completed = await _store.CompleteAsync(claim, stoppingToken).ConfigureAwait(false); + Fence(completed); + activity.SetTag("outcome", completed ? "published" : "lease_lost"); + } + else + { + _observation.Failed(read.EventTypeLabel); + Fence(await _store.FailAsync(claim, OutboxFailureReason.PublishFailed, stoppingToken).ConfigureAwait(false)); + activity.SetTag("outcome", "publish_failed"); + } + } + finally + { + _observation.Duration(read.EventTypeLabel, _time.GetElapsedTime(started, _time.GetTimestamp())); + LogAttempt(activity, read.EventTypeLabel); + } + } + + private void LogAttempt(Activity activity, string label) + { + if (_logger.IsEnabled(LogLevel.Information)) + { + var outcome = activity.GetTagItem("outcome") as string ?? "storage_failure"; + LogAttemptFinished(_logger, activity.Id, label, outcome); + } + } + + [LoggerMessage(Level = LogLevel.Warning, Message = "Outbox dispatch iteration failed with outcome iteration_failed.")] + private static partial void LogIterationFailure(ILogger logger); + + [LoggerMessage(Level = LogLevel.Information, + Message = "Outbox dispatch attempt {CorrelationId} finished with event type {EventType} and outcome {Outcome}.")] + private static partial void LogAttemptFinished(ILogger logger, string? correlationId, string eventType, string outcome); + + private async Task RefreshAsync(bool force, CancellationToken cancellationToken) + { + var last = _observation.Snapshot.LastSnapshotTimestamp; + if (!force && last is { } timestamp + && _time.GetElapsedTime(timestamp, _time.GetTimestamp()) < _options.SnapshotInterval) return; + try + { + var queue = await _store.ReadSnapshotAsync(RetainedPartitions(), cancellationToken).ConfigureAwait(false); + _observation.Refreshed(queue); + } + catch (Exception) + { + _observation.RefreshFailed(); + throw; + } + } + + private void Fence(bool changed) + { + if (!changed) _observation.LeaseLost(); + } + + private int OutstandingCount + { + get { lock (_attemptGate) return _attempts.Count; } + } + + private string[] RetainedPartitions() + { + lock (_attemptGate) + { + // Completion includes cancellation callbacks and asynchronous scope + // disposal; only then may this partition be acquired locally again. + _attempts.RemoveAll(attempt => attempt.Completion.IsCompleted); + _observation.Outstanding(_attempts.Count); + return _attempts.Select(attempt => attempt.PartitionKey).Distinct(StringComparer.Ordinal).ToArray(); + } + } + + private OutboxPublishAttempt[] CancelAttempts() + { + OutboxPublishAttempt[] attempts; + lock (_attemptGate) attempts = _attempts.ToArray(); + foreach (var attempt in attempts) attempt.RequestCancellation(); + return attempts; + } + + public override async Task StopAsync(CancellationToken cancellationToken) + { + // The base service and callback drain share one overall budget. Never add + // a second shutdown wait after the host has already exhausted the first. + using var timeout = new CancellationTokenSource(_options.ShutdownTimeout, _time); + using var deadline = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, timeout.Token); + try + { + await base.StopAsync(deadline.Token).ConfigureAwait(false); + var attempts = CancelAttempts(); + await Task.WhenAll(attempts.Select(attempt => attempt.Completion)).WaitAsync(deadline.Token) + .ConfigureAwait(false); + } + catch (OperationCanceledException) when (deadline.IsCancellationRequested) + { + CancelAttempts(); + } + finally + { + RetainedPartitions(); + _observation.Stopped(); + } + } +} diff --git a/backend/src/LearnStack.Api/Messaging/OutboxPublishAttempt.cs b/backend/src/LearnStack.Api/Messaging/OutboxPublishAttempt.cs new file mode 100644 index 00000000..9b6d660a --- /dev/null +++ b/backend/src/LearnStack.Api/Messaging/OutboxPublishAttempt.cs @@ -0,0 +1,87 @@ +using System.Diagnostics.CodeAnalysis; +using LearnStack.SharedKernel.Messaging; + +namespace LearnStack.Api.Messaging; + +/// Owns callback construction, cancellation callbacks and scope cleanup as one task. +[SuppressMessage("Design", "CA1001:Types that own disposable fields should be disposable", + Justification = "The owned task disposes its cancellation source only after callback and scope cleanup; caller disposal could invalidate a live attempt.")] +internal sealed class OutboxPublishAttempt +{ + private readonly object _gate = new(); + private readonly CancellationTokenSource _cancellation = new(); + private Task? _cancelCallbacks; + private bool _finishing; + + public OutboxPublishAttempt(IServiceScopeFactory scopes, IntegrationEventEnvelope envelope) + { + ArgumentNullException.ThrowIfNull(scopes); + ArgumentNullException.ThrowIfNull(envelope); + PartitionKey = envelope.PartitionKey; + // Resolution and even the synchronous prefix of PublishAsync belong to the + // retained task. Neither may monopolize the dispatcher's bounded wait. + Completion = Task.Run(() => PublishAsync(scopes, envelope), CancellationToken.None); + } + + public string PartitionKey { get; } + public Task Completion { get; } + + public void RequestCancellation() + { + lock (_gate) + { + if (_finishing || _cancelCallbacks is not null) return; + // CancelAsync changes token state immediately but runs registered + // callbacks asynchronously. A blocking registration cannot block StopAsync. + _cancelCallbacks = _cancellation.CancelAsync(); + } + } + + private async Task PublishAsync(IServiceScopeFactory scopes, IntegrationEventEnvelope envelope) + { + AsyncServiceScope? scope = null; + var succeeded = false; + try + { + scope = scopes.CreateAsyncScope(); + var bus = scope.Value.ServiceProvider.GetRequiredService(); + await bus.PublishAsync(envelope, _cancellation.Token).ConfigureAwait(false); + succeeded = !_cancellation.IsCancellationRequested; + } + catch (Exception) + { + // Provider diagnostics and payloads never cross this dispatch boundary. + succeeded = false; + } + finally + { + Task? callbacks; + lock (_gate) + { + _finishing = true; + callbacks = _cancelCallbacks; + } + + if (callbacks is not null) + { + try { await callbacks.ConfigureAwait(false); } + catch (Exception) { succeeded = false; } + } + + try + { + if (scope is { } ownedScope) await ownedScope.DisposeAsync().ConfigureAwait(false); + } + catch (Exception) + { + succeeded = false; + } + finally + { + _cancellation.Dispose(); + } + } + + return succeeded; + } +} diff --git a/backend/src/LearnStack.Infrastructure.Audit/PostgresAuditStore.cs b/backend/src/LearnStack.Infrastructure.Audit/PostgresAuditStore.cs index 177dcc78..f3d51356 100644 --- a/backend/src/LearnStack.Infrastructure.Audit/PostgresAuditStore.cs +++ b/backend/src/LearnStack.Infrastructure.Audit/PostgresAuditStore.cs @@ -74,6 +74,7 @@ public sealed class PostgresAuditStore : IAuditStore private readonly Lazy _dataSource; private readonly ILogger _logger; private readonly IAuditHealth _health; + private readonly IAuditCatalog _catalog; private readonly Counter _durableDuplicates; private readonly Counter _standaloneWriteFailures; @@ -82,7 +83,8 @@ public PostgresAuditStore( Lazy dataSource, ILogger logger, IMeterFactory meterFactory, - IAuditHealth health) + IAuditHealth health, + IAuditCatalog catalog) { ArgumentNullException.ThrowIfNull(meterFactory); @@ -90,6 +92,7 @@ public PostgresAuditStore( _dataSource = dataSource; _logger = logger; _health = health ?? throw new ArgumentNullException(nameof(health)); + _catalog = catalog ?? throw new ArgumentNullException(nameof(catalog)); var meter = meterFactory.Create(MeterName); _durableDuplicates = meter.CreateCounter(DurableDuplicateCounterName); @@ -376,6 +379,44 @@ public async Task WritePlatformScopeAsync( } } + /// + public async Task WriteEventTerminalAsync( + AuditEntryDraft entry, + EventTerminalTransition transition, + DbConnection connection, + DbTransaction transaction, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(transition); + transition.EnsureMatches(entry, connection, transaction); + + const string operation = "platform.outbox.dead_letter"; + if (!_catalog.TryGetOffPath(operation, out var declared) + || declared.Operation != operation || declared.ModuleName != "platform" + || declared.OperationClass != OperationClass.Must + || declared.OperationType != OperationType.SecurityEvent || declared.EntityType is not null + || entry.Operation != declared.Operation || entry.ModuleName != declared.ModuleName + || entry.OperationClass != declared.OperationClass || entry.OperationType != declared.OperationType + || transition.Kind != "producer" || transition.SubscriptionId is not null) + { + throw new ArgumentException("The terminal audit requires its declared producer MUST security event.", nameof(entry)); + } + + try + { + await using var command = connection.CreateCommand(); + command.CommandText = InsertSql; + command.Transaction = transaction; + Bind(command.Parameters, entry); + await command.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false); + } + catch (DbException failure) + { + throw new AuditWriteFailedException( + "The terminal audit could not be written; its state transition must roll back.", failure); + } + } + /// /// BEGIN; SET LOCAL app.tenant_id; SET LOCAL app.organization_id; INSERT; COMMIT /// on a connection of this call's own. @@ -429,8 +470,8 @@ private async Task WriteOwnTransactionAsync( /// Binds one row's parameters, for a command or for one command of a batch. /// /// Onto the parameter collection rather than the command, because the in-transaction - /// path writes through a and the other three through a - /// , and one binding is what keeps the four writing one row + /// path writes through a and the other four through a + /// , and one binding is what keeps the five writing one row /// shape. /// private static void Bind(DbParameterCollection parameters, AuditEntryDraft entry) diff --git a/backend/src/LearnStack.Infrastructure/Messaging/OutboxDispatchModels.cs b/backend/src/LearnStack.Infrastructure/Messaging/OutboxDispatchModels.cs new file mode 100644 index 00000000..9dc0f449 --- /dev/null +++ b/backend/src/LearnStack.Infrastructure/Messaging/OutboxDispatchModels.cs @@ -0,0 +1,21 @@ +using LearnStack.SharedKernel.Messaging; + +namespace LearnStack.Infrastructure.Messaging; + +public enum OutboxClaimKind { Normal, Recovery, Terminal } +public enum OutboxTerminalReason { UnknownType, RetiredType, InvalidEnvelope, RetryExhausted } +public enum OutboxFailureReason { PublishFailed, PublishTimeout, UnknownOutcome } + +/// A committed acquisition; its token is meaningful only until database-time expiry. +public sealed record OutboxDispatchClaim( + Guid Id, Guid TenantId, Guid? OrganizationId, string Token, int Attempts, + DateTimeOffset LockedUntil, OutboxClaimKind Kind, string Type, string Topic, + string PartitionKey, DateTimeOffset OccurredAt, string CorrelationId, + Guid? CausationId, Guid? ActorUserId, string Payload, bool MetadataIsNull); + +/// One database observation; gauge callbacks consume only this immutable value. +public sealed record OutboxQueueSnapshot(long PendingCount, long DeferredCount, double OldestEligibleAgeSeconds); + +public sealed record OutboxEnvelopeReadResult( + OutboxEventDisposition Disposition, string EventTypeLabel, + IntegrationEventEnvelope? Envelope, OutboxTerminalReason? TerminalReason); diff --git a/backend/src/LearnStack.Infrastructure/Messaging/OutboxDispatchOptions.cs b/backend/src/LearnStack.Infrastructure/Messaging/OutboxDispatchOptions.cs new file mode 100644 index 00000000..9ec2ffb5 --- /dev/null +++ b/backend/src/LearnStack.Infrastructure/Messaging/OutboxDispatchOptions.cs @@ -0,0 +1,30 @@ +namespace LearnStack.Infrastructure.Messaging; + +/// Finite execution bounds for ADR-0046's single dispatcher. +public sealed record OutboxDispatchOptions +{ + public TimeSpan SqlTimeout { get; init; } = TimeSpan.FromSeconds(5); + public TimeSpan PublishTimeout { get; init; } = TimeSpan.FromSeconds(10); + public TimeSpan LeaseDuration { get; init; } = TimeSpan.FromSeconds(30); + public TimeSpan IdlePoll { get; init; } = TimeSpan.FromMilliseconds(200); + public TimeSpan ProgressTimeout { get; init; } = TimeSpan.FromSeconds(45); + public TimeSpan SnapshotInterval { get; init; } = TimeSpan.FromSeconds(5); + public TimeSpan ShutdownTimeout { get; init; } = TimeSpan.FromSeconds(30); + public int MaxOutstandingPublishes { get; init; } = 4; + + public void Validate() + { + if (SqlTimeout <= TimeSpan.Zero || SqlTimeout > TimeSpan.FromSeconds(5) + || PublishTimeout <= TimeSpan.Zero || PublishTimeout > TimeSpan.FromSeconds(10) + || LeaseDuration <= PublishTimeout + SqlTimeout + SqlTimeout + || LeaseDuration > TimeSpan.FromSeconds(30) + || IdlePoll <= TimeSpan.Zero || IdlePoll > TimeSpan.FromSeconds(1) + || SnapshotInterval <= TimeSpan.Zero || SnapshotInterval > TimeSpan.FromSeconds(5) + || ProgressTimeout <= LeaseDuration || ProgressTimeout > TimeSpan.FromSeconds(45) + || ShutdownTimeout <= TimeSpan.Zero || ShutdownTimeout > TimeSpan.FromSeconds(30) + || MaxOutstandingPublishes is < 1 or > 4) + { + throw new ArgumentException("Outbox execution bounds are invalid."); + } + } +} diff --git a/backend/src/LearnStack.Infrastructure/Messaging/OutboxDispatchStore.cs b/backend/src/LearnStack.Infrastructure/Messaging/OutboxDispatchStore.cs new file mode 100644 index 00000000..bb8f11dc --- /dev/null +++ b/backend/src/LearnStack.Infrastructure/Messaging/OutboxDispatchStore.cs @@ -0,0 +1,299 @@ +using System.Data.Common; +using LearnStack.SharedKernel.Audit; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Time; +using Microsoft.Extensions.DependencyInjection; +using Npgsql; +using NpgsqlTypes; + +namespace LearnStack.Infrastructure.Messaging; + +/// Short, database-time fenced dispatcher transitions on its confined role. +public sealed class OutboxDispatchStore : EventTerminalTransition.MintingSource +{ + private const string Incomplete = "processed_at IS NULL AND dead_lettered_at IS NULL"; + private const string Normal = Incomplete + " AND terminal_reason IS NULL AND attempts < 6" + + " AND available_after <= clock_timestamp() AND locked_by IS NULL"; + private const string Recovery = Incomplete + " AND terminal_reason IS NULL" + + " AND (locked_by IS NOT NULL OR attempts >= 6)" + + " AND ((locked_by IS NOT NULL AND locked_until <= clock_timestamp())" + + " OR (locked_by IS NULL AND attempts >= 6))"; + private const string Admitted = " AND NOT (type = ANY(@pending)) AND NOT (partition_key = ANY(@blocked))"; + private const string ClaimColumns = "id, tenant_id, organization_id, locked_by, attempts, locked_until," + + " type, topic, partition_key, occurred_at, correlation_id, causation_id, actor_user_id, payload::text, metadata IS NULL"; + private const string Fence = "id = @id AND locked_by = @token AND locked_until > clock_timestamp()" + + " AND " + Incomplete; + + private readonly Lazy _source; + private readonly OutboxEventRegistry _registry; + private readonly OutboxDispatchOptions _options; + private readonly IGuidFactory _ids; + private readonly IClock _clock; + private readonly IServiceScopeFactory _scopes; + + public OutboxDispatchStore(Lazy source, OutboxEventRegistry registry, + OutboxDispatchOptions options, IGuidFactory ids, IClock clock, IServiceScopeFactory scopes) + { + ArgumentNullException.ThrowIfNull(source); + ArgumentNullException.ThrowIfNull(registry); + ArgumentNullException.ThrowIfNull(options); + ArgumentNullException.ThrowIfNull(ids); + ArgumentNullException.ThrowIfNull(clock); + ArgumentNullException.ThrowIfNull(scopes); + options.Validate(); + _source = source; + _registry = registry; + _options = options; + _ids = ids; + _clock = clock; + _scopes = scopes; + } + + public Task TryClaimAsync(IReadOnlyCollection blockedPartitions, + CancellationToken cancellationToken = default) => + AcquireAsync(OutboxClaimKind.Normal, Normal, blockedPartitions, cancellationToken); + + public Task TryAcquireRecoveryAsync(IReadOnlyCollection blockedPartitions, + CancellationToken cancellationToken = default) => + AcquireAsync(OutboxClaimKind.Recovery, Recovery, blockedPartitions, cancellationToken); + + public Task TryClaimTerminalAsync(IReadOnlyCollection blockedPartitions, + CancellationToken cancellationToken = default) => + AcquireAsync(OutboxClaimKind.Terminal, Incomplete + " AND terminal_reason IS NOT NULL" + + " AND available_after <= clock_timestamp()" + + " AND (locked_until IS NULL OR locked_until <= clock_timestamp())", blockedPartitions, cancellationToken); + + private async Task AcquireAsync(OutboxClaimKind kind, string eligibility, + IReadOnlyCollection blocked, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(blocked); + using var deadline = Deadline(cancellationToken); + await using var connection = await _source.Value.OpenConnectionAsync(deadline.Token).ConfigureAwait(false); + await using var transaction = await connection.BeginTransactionAsync(deadline.Token).ConfigureAwait(false); + await using var command = new NpgsqlCommand($""" + WITH candidate AS ( + SELECT id FROM outbox_messages WHERE {eligibility}{Admitted} + ORDER BY available_after, occurred_at, id FOR UPDATE SKIP LOCKED LIMIT 1 + ) + UPDATE outbox_messages SET locked_by = @token, + locked_until = clock_timestamp() + @lease, + attempts = attempts + @increment + WHERE id = (SELECT id FROM candidate) + RETURNING {ClaimColumns} + """, connection, transaction); + Configure(command); + AddAdmission(command, blocked); + command.Parameters.AddWithValue("token", _ids.NewUuidV4().ToString("D")); + command.Parameters.AddWithValue("lease", _options.LeaseDuration); + command.Parameters.AddWithValue("increment", kind == OutboxClaimKind.Normal ? 1 : 0); + OutboxDispatchClaim? result; + await using (var reader = await command.ExecuteReaderAsync(deadline.Token).ConfigureAwait(false)) + { + result = await reader.ReadAsync(deadline.Token).ConfigureAwait(false) ? ReadClaim(reader, kind) : null; + } + + await transaction.CommitAsync(deadline.Token).ConfigureAwait(false); + return result; + } + + public Task CompleteAsync(OutboxDispatchClaim claim, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(claim); + if (claim.Kind != OutboxClaimKind.Normal) + throw new ArgumentException("Only a normal publication claim can complete.", nameof(claim)); + return UpdateAsync(claim, "processed_at = clock_timestamp(), last_error = NULL," + + " locked_by = NULL, locked_until = NULL", " AND terminal_reason IS NULL", null, cancellationToken); + } + + public Task FailAsync(OutboxDispatchClaim claim, OutboxFailureReason reason, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(claim); + if (claim.Kind == OutboxClaimKind.Terminal) + throw new ArgumentException("Terminal requests cannot spend a publication retry.", nameof(claim)); + var descriptor = reason switch + { + OutboxFailureReason.PublishFailed => "publish:failed:provider", + OutboxFailureReason.PublishTimeout => "publish:timeout:deadline", + OutboxFailureReason.UnknownOutcome => "recovery:unknown:expired", + _ => throw new ArgumentOutOfRangeException(nameof(reason)), + }; + return UpdateAsync(claim, """ + terminal_reason = CASE WHEN attempts >= 6 THEN 'retry_exhausted' ELSE NULL END, + available_after = clock_timestamp() + CASE attempts + WHEN 1 THEN interval '1 second' WHEN 2 THEN interval '5 seconds' + WHEN 3 THEN interval '30 seconds' WHEN 4 THEN interval '5 minutes' + WHEN 5 THEN interval '1 hour' ELSE interval '0 seconds' END, + last_error = @descriptor, locked_by = NULL, locked_until = NULL + """, " AND terminal_reason IS NULL", descriptor, cancellationToken); + } + + public Task RequestTerminalAsync(OutboxDispatchClaim claim, OutboxTerminalReason reason, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(claim); + var code = TerminalCode(reason); + return UpdateAsync(claim, "terminal_reason = @reason, available_after = clock_timestamp()," + + " last_error = @descriptor, locked_by = NULL, locked_until = NULL", + " AND terminal_reason IS NULL", "terminal:" + code + ":closed", cancellationToken, code); + } + + private async Task UpdateAsync(OutboxDispatchClaim claim, string assignments, string extra, + string? descriptor, CancellationToken cancellationToken, string? reason = null) + { + using var deadline = Deadline(cancellationToken); + await using var connection = await _source.Value.OpenConnectionAsync(deadline.Token).ConfigureAwait(false); + await using var command = new NpgsqlCommand( + $"UPDATE outbox_messages SET {assignments} WHERE {Fence}{extra}", connection); + Configure(command); + AddFence(command, claim); + if (descriptor is not null) command.Parameters.AddWithValue("descriptor", descriptor); + if (reason is not null) command.Parameters.AddWithValue("reason", reason); + return await command.ExecuteNonQueryAsync(deadline.Token).ConfigureAwait(false) == 1; + } + + /// Fences, mints the actual returned-row witness, inserts MUST audit, then commits both. + public async Task TryTerminalizeAsync(OutboxDispatchClaim claim, string validatedAttemptTraceparent, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(claim); + if (claim.Kind != OutboxClaimKind.Terminal) + throw new ArgumentException("Terminalization requires its own acquisition.", nameof(claim)); + using var deadline = Deadline(cancellationToken); + await using var connection = await _source.Value.OpenConnectionAsync(deadline.Token).ConfigureAwait(false); + await using var transaction = await connection.BeginTransactionAsync(deadline.Token).ConfigureAwait(false); + await using var command = new NpgsqlCommand($""" + UPDATE outbox_messages SET dead_lettered_at = clock_timestamp(), + locked_by = NULL, locked_until = NULL + WHERE {Fence} AND terminal_reason IS NOT NULL + RETURNING id, tenant_id, organization_id, terminal_reason, attempts, + old.locked_by AS acquisition_token, correlation_id, causation_id, actor_user_id + """, connection, transaction); + Configure(command); + AddFence(command, claim); + EventTerminalTransition witness; + await using (var reader = await command.ExecuteReaderAsync(deadline.Token).ConfigureAwait(false)) + { + if (!await reader.ReadAsync(deadline.Token).ConfigureAwait(false)) return false; + witness = FromProducerUpdate(connection, transaction, reader, validatedAttemptTraceparent); + } + + await using var scope = _scopes.CreateAsyncScope(); + var audit = scope.ServiceProvider.GetRequiredService(); + var draft = new AuditEntryDraft + { + Id = AuditEntryId.From(_ids.NewUuidV7()), + TenantId = TenantId.From(witness.TenantId), + OrganizationId = witness.OrganizationId is { } organization ? OrganizationId.From(organization) : null, + ActorUserId = UserId.SystemActor, + ActorEmail = null, + ModuleName = "platform", + Operation = "platform.outbox.dead_letter", + OperationType = OperationType.SecurityEvent, + OperationClass = OperationClass.Must, + EntityType = null, + EntityId = null, + Outcome = AuditOutcome.Success, + ErrorKey = null, + Reason = null, + BeforeState = null, + AfterState = null, + Changes = null, + CorrelationId = witness.CorrelationId, + IpAddress = null, + UserAgent = null, + Timestamp = _clock.UtcNow, + Metadata = witness.Metadata, + }; + await audit.WriteEventTerminalAsync(draft, witness, connection, transaction, deadline.Token).ConfigureAwait(false); + try + { + await transaction.CommitAsync(deadline.Token).ConfigureAwait(false); + return true; + } + catch (Exception exception) when (exception is NpgsqlException or OperationCanceledException or IOException) + { + // COMMIT may have reached PostgreSQL. Reconcile on a fresh bounded connection; + // a durable terminal row already includes its audit and must never be inserted again. + try + { + if (await IsDurablyTerminalAsync(claim.Id).ConfigureAwait(false)) return true; + } + catch (Exception reconciliationFailure) + { + throw new AggregateException("The outbox terminal commit outcome remains unknown.", + exception, reconciliationFailure); + } + throw; + } + } + + private async Task IsDurablyTerminalAsync(Guid id) + { + using var deadline = Deadline(CancellationToken.None); + await using var connection = await _source.Value.OpenConnectionAsync(deadline.Token).ConfigureAwait(false); + await using var command = new NpgsqlCommand( + "SELECT EXISTS (SELECT 1 FROM outbox_messages WHERE id = @id AND dead_lettered_at IS NOT NULL)", connection); + Configure(command); + command.Parameters.AddWithValue("id", id); + return (bool)(await command.ExecuteScalarAsync(deadline.Token).ConfigureAwait(false))!; + } + + public async Task ReadSnapshotAsync(IReadOnlyCollection blockedPartitions, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(blockedPartitions); + using var deadline = Deadline(cancellationToken); + await using var connection = await _source.Value.OpenConnectionAsync(deadline.Token).ConfigureAwait(false); + await using var command = new NpgsqlCommand($""" + SELECT count(*), count(*) FILTER (WHERE type = ANY(@pending)), + COALESCE(GREATEST(0, extract(epoch FROM clock_timestamp() - min(occurred_at) + FILTER (WHERE {Normal}{Admitted}))), 0)::double precision + FROM outbox_messages WHERE {Incomplete} + """, connection); + Configure(command); + AddAdmission(command, blockedPartitions); + await using var reader = await command.ExecuteReaderAsync(deadline.Token).ConfigureAwait(false); + await reader.ReadAsync(deadline.Token).ConfigureAwait(false); + return new(reader.GetInt64(0), reader.GetInt64(1), reader.GetDouble(2)); + } + + private static OutboxDispatchClaim ReadClaim(DbDataReader row, OutboxClaimKind kind) => + new(row.GetGuid(0), row.GetGuid(1), row.IsDBNull(2) ? null : row.GetGuid(2), + row.GetString(3), row.GetInt32(4), row.GetFieldValue(5), kind, + row.GetString(6), row.GetString(7), row.GetString(8), row.GetFieldValue(9), + row.GetString(10), row.IsDBNull(11) ? null : row.GetGuid(11), + row.IsDBNull(12) ? null : row.GetGuid(12), row.GetString(13), row.GetBoolean(14)); + + private CancellationTokenSource Deadline(CancellationToken cancellationToken) + { + var source = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + source.CancelAfter(_options.SqlTimeout); + return source; + } + + private void Configure(NpgsqlCommand command) => + command.CommandTimeout = Math.Max(1, (int)Math.Ceiling(_options.SqlTimeout.TotalSeconds)); + + private void AddAdmission(NpgsqlCommand command, IReadOnlyCollection blocked) + { + command.Parameters.AddWithValue("pending", NpgsqlDbType.Array | NpgsqlDbType.Text, + _registry.PendingIdentities.ToArray()); + command.Parameters.AddWithValue("blocked", NpgsqlDbType.Array | NpgsqlDbType.Text, blocked.ToArray()); + } + + private static void AddFence(NpgsqlCommand command, OutboxDispatchClaim claim) + { + command.Parameters.AddWithValue("id", claim.Id); + command.Parameters.AddWithValue("token", claim.Token); + } + + private static string TerminalCode(OutboxTerminalReason reason) => reason switch + { + OutboxTerminalReason.UnknownType => "unknown_type", + OutboxTerminalReason.RetiredType => "retired_type", + OutboxTerminalReason.InvalidEnvelope => "invalid_envelope", + OutboxTerminalReason.RetryExhausted => "retry_exhausted", + _ => throw new ArgumentOutOfRangeException(nameof(reason)), + }; +} diff --git a/backend/src/LearnStack.Infrastructure/Messaging/OutboxDispatcherDataSource.cs b/backend/src/LearnStack.Infrastructure/Messaging/OutboxDispatcherDataSource.cs new file mode 100644 index 00000000..d4fdf4c6 --- /dev/null +++ b/backend/src/LearnStack.Infrastructure/Messaging/OutboxDispatcherDataSource.cs @@ -0,0 +1,104 @@ +using Npgsql; + +namespace LearnStack.Infrastructure.Messaging; + +/// Builds the grant-bounded pool reserved for outbox dispatch. +public static class OutboxDispatcherDataSource +{ + public const string ConnectionName = "OutboxDispatcher"; + public const string Key = "outbox-dispatcher"; + public const string Role = "learnstack_outbox_admin"; + + /// Builds a guarded pool without opening a connection. + public static NpgsqlDataSource Build(string? connectionString) + { + var parsed = ParseAndValidate(connectionString); + + try + { + var builder = new NpgsqlDataSourceBuilder(parsed.ConnectionString); + builder.UsePhysicalConnectionInitializer( + connection => RequireConfinedRole(connection, async: false).GetAwaiter().GetResult(), + connection => RequireConfinedRole(connection, async: true)); + return builder.Build(); + } + catch (Exception failure) when (failure is ArgumentException or FormatException or InvalidOperationException) + { + // Driver configuration errors may contain supplied values. Do not + // attach them as inner exceptions or repeat any connection field. + throw InvalidConfiguration(); + } + } + + /// Checks configuration without building the pool or connecting. + public static void Validate(string? connectionString) => _ = ParseAndValidate(connectionString); + + private static NpgsqlConnectionStringBuilder ParseAndValidate(string? connectionString) + { + if (string.IsNullOrWhiteSpace(connectionString)) + { + throw new InvalidOperationException( + $"ConnectionStrings:{ConnectionName} is not configured. See .env.example."); + } + + NpgsqlConnectionStringBuilder parsed; + try + { + parsed = new NpgsqlConnectionStringBuilder(connectionString); + } + catch (Exception failure) when (failure is ArgumentException or FormatException) + { + throw InvalidConfiguration(); + } + + if (!string.Equals(parsed.Username, Role, StringComparison.Ordinal)) + { + throw new InvalidOperationException( + $"ConnectionStrings:{ConnectionName} must name the dispatcher role. See .env.example."); + } + + return parsed; + } + + private static InvalidOperationException InvalidConfiguration() => new( + $"ConnectionStrings:{ConnectionName} is not a valid Npgsql key/value connection string. See .env.example."); + + private static async Task RequireConfinedRole(NpgsqlConnection connection, bool async) + { + await using var command = connection.CreateCommand(); + command.CommandTimeout = 5; + command.CommandText = + """ + SELECT EXISTS ( + SELECT 1 FROM pg_roles role + WHERE role.rolname = @role + AND session_user = @role AND current_user = @role + AND role.rolbypassrls + AND NOT role.rolsuper + AND NOT role.rolcreatedb + AND NOT role.rolcreaterole + AND NOT role.rolreplication + AND NOT EXISTS ( + SELECT 1 FROM pg_roles reachable + WHERE reachable.oid <> role.oid + AND pg_has_role(role.oid, reachable.oid, 'MEMBER')) + AND NOT EXISTS ( + SELECT 1 FROM pg_roles member + WHERE member.oid <> role.oid + -- Superusers implicitly belong to every role without a grant. + AND NOT member.rolsuper + AND pg_has_role(member.oid, role.oid, 'MEMBER'))) + """; + command.Parameters.AddWithValue("role", Role); + + var confined = async + ? await command.ExecuteScalarAsync() + : command.ExecuteScalar(); + + if (confined is not true) + { + throw new InvalidOperationException( + "The outbox dispatcher connection has an invalid role, attributes or memberships."); + } + } +} diff --git a/backend/src/LearnStack.Infrastructure/Messaging/OutboxEnvelopeReader.cs b/backend/src/LearnStack.Infrastructure/Messaging/OutboxEnvelopeReader.cs new file mode 100644 index 00000000..1a460595 --- /dev/null +++ b/backend/src/LearnStack.Infrastructure/Messaging/OutboxEnvelopeReader.cs @@ -0,0 +1,63 @@ +using System.Text.Json; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Messaging; + +namespace LearnStack.Infrastructure.Messaging; + +/// Validates a stored fact using only the registry's declared CLR contract. +public sealed class OutboxEnvelopeReader(OutboxEventRegistry registry) +{ + public OutboxEnvelopeReadResult Read(OutboxDispatchClaim claim) + { + ArgumentNullException.ThrowIfNull(claim); + var resolution = registry.Resolve(claim.Type); + // A Pending hold precedes validation, including deliberately malformed payloads. + if (resolution.Disposition == OutboxEventDisposition.Pending) + return new(resolution.Disposition, resolution.EventTypeLabel, null, null); + if (resolution.Disposition == OutboxEventDisposition.Unknown) + return new(resolution.Disposition, "unknown", null, OutboxTerminalReason.UnknownType); + if (resolution.Disposition == OutboxEventDisposition.Retired) + return new(resolution.Disposition, "retired", null, OutboxTerminalReason.RetiredType); + + try + { + if (!claim.MetadataIsNull || claim.TenantId == Guid.Empty + || claim.TenantId == TenantId.PlatformSentinel.Value || claim.OrganizationId == Guid.Empty) + return Invalid(); + + using var document = JsonDocument.Parse(claim.Payload); + var root = document.RootElement; + // Read the stored routing fields before deserialization: get-only + // properties would otherwise silently regenerate missing or altered values. + if (root.ValueKind != JsonValueKind.Object + || root.GetProperty(nameof(IIntegrationEvent.Topic)).GetString() != claim.Topic + || root.GetProperty(nameof(IIntegrationEvent.PartitionKey)).GetString() != claim.PartitionKey) + return Invalid(); + + var fact = root.Deserialize(resolution.EventType!, + IntegrationEventBase.PayloadJsonOptions) as IntegrationEventBase; + if (fact is null) return Invalid(); + var serialized = OutboxEventContract.Serialize(fact); + // Npgsql writes timestamps at microsecond precision by truncating + // sub-microsecond ticks; compare that stored precision, not a tolerance. + if (serialized.EventId != claim.Id || serialized.TenantId != claim.TenantId + || serialized.Topic != claim.Topic || serialized.PartitionKey != claim.PartitionKey + || serialized.OccurredAt.UtcTicks / 10 != claim.OccurredAt.UtcTicks / 10) + return Invalid(); + + var envelope = new IntegrationEventEnvelope(fact, claim.CorrelationId, + claim.OrganizationId, claim.CausationId, + claim.ActorUserId is { } actor ? UserId.From(actor) : null); + return new(resolution.Disposition, resolution.EventTypeLabel, envelope, null); + } + catch (Exception exception) when (exception is JsonException or ArgumentException + or InvalidOperationException or NotSupportedException or KeyNotFoundException) + { + // Stored data is untrusted; neither its text nor parser diagnostics leave this boundary. + return Invalid(); + } + + static OutboxEnvelopeReadResult Invalid() => + new(OutboxEventDisposition.Unknown, "invalid", null, OutboxTerminalReason.InvalidEnvelope); + } +} diff --git a/backend/src/LearnStack.Infrastructure/Messaging/OutboxEventRegistry.cs b/backend/src/LearnStack.Infrastructure/Messaging/OutboxEventRegistry.cs new file mode 100644 index 00000000..bad98752 --- /dev/null +++ b/backend/src/LearnStack.Infrastructure/Messaging/OutboxEventRegistry.cs @@ -0,0 +1,362 @@ +using System.Collections.Frozen; +using System.Diagnostics.CodeAnalysis; +using System.Reflection; +using LearnStack.SharedKernel.Messaging; + +namespace LearnStack.Infrastructure.Messaging; + +/// The declared delivery policy for one stored integration-event identity. +public enum OutboxEventDisposition +{ + Pending, + Active, + IntentionallyUnsubscribed, + Retired, + Unknown, +} + +/// Readiness of one stable, independently deduplicated subscription. +public enum OutboxSubscriptionReadiness +{ + Pending, + Active, +} + +/// Construction-free metadata for one subscription. +public sealed class OutboxSubscriptionManifest +{ + private OutboxSubscriptionManifest( + string id, OutboxSubscriptionReadiness readiness, Type? handlerType) + { + Id = id; + Readiness = readiness; + HandlerType = handlerType; + } + + public string Id { get; } + + public OutboxSubscriptionReadiness Readiness { get; } + + /// Only an Active subscription names a concrete handler. + public Type? HandlerType { get; } + + public static OutboxSubscriptionManifest Pending(string id) => + new(id, OutboxSubscriptionReadiness.Pending, null); + + public static OutboxSubscriptionManifest Active(string id, Type handlerType) => + new(id, OutboxSubscriptionReadiness.Active, handlerType); +} + +/// A concrete contract, its declared aliases and its delivery policy. +public sealed class OutboxEventRegistration +{ + private OutboxEventRegistration( + Type eventType, + IReadOnlyList subscriptions, + IReadOnlyList aliases, + bool intentionallyUnsubscribed) + { + EventType = eventType; + Subscriptions = subscriptions; + CompatibleAliases = aliases; + IsIntentionallyUnsubscribed = intentionallyUnsubscribed; + } + + public Type EventType { get; } + + public IReadOnlyList Subscriptions { get; } + + public IReadOnlyList CompatibleAliases { get; } + + public bool IsIntentionallyUnsubscribed { get; } + + public static OutboxEventRegistration Required( + Type eventType, + IEnumerable subscriptions, + params string[] compatibleAliases) + { + ArgumentNullException.ThrowIfNull(subscriptions); + ArgumentNullException.ThrowIfNull(compatibleAliases); + return new OutboxEventRegistration( + eventType, + Array.AsReadOnly(subscriptions.ToArray()), + Array.AsReadOnly(compatibleAliases.ToArray()), + intentionallyUnsubscribed: false); + } + + public static OutboxEventRegistration IntentionallyUnsubscribed( + Type eventType, + params string[] compatibleAliases) + { + ArgumentNullException.ThrowIfNull(compatibleAliases); + return new OutboxEventRegistration( + eventType, + Array.Empty(), + Array.AsReadOnly(compatibleAliases.ToArray()), + intentionallyUnsubscribed: true); + } +} + +/// A bounded lookup result; stored text never becomes a CLR type lookup. +public sealed record OutboxEventResolution( + OutboxEventDisposition Disposition, + Type? EventType, + string EventTypeLabel, + IReadOnlyList Subscriptions); + +/// +/// Exact stored-type resolver and readiness manifest for the durable outbox. +/// +/// +/// The registry is assembled from known contract types and the existing handler +/// registry. It never constructs a handler or calls Type.GetType on a row. +/// Pending identities include all declared compatible aliases, so the dispatcher +/// can exclude them before reading or validating a payload. +/// +public sealed class OutboxEventRegistry +{ + private static readonly IReadOnlyList NoSubscriptions = + Array.Empty(); + + private readonly FrozenDictionary _known; + private readonly FrozenSet _retired; + + private OutboxEventRegistry( + FrozenDictionary known, + FrozenSet retired, + IReadOnlyList pendingIdentities, + IReadOnlyList eventTypeLabels) + { + _known = known; + _retired = retired; + PendingIdentities = pendingIdentities; + EventTypeLabels = eventTypeLabels; + } + + /// Exact canonical and alias identities held before acquisition. + public IReadOnlyList PendingIdentities { get; } + + /// The finite labels allowed for declared event types in metrics. + public IReadOnlyList EventTypeLabels { get; } + + /// + /// Builds a closed policy from explicitly known assemblies and concrete types. + /// Active entries must match the construction-free handler census exactly. + /// + public static OutboxEventRegistry Create( + IEnumerable registrations, + IEnumerable retiredIdentities, + IntegrationEventHandlerRegistry handlers, + IEnumerable knownContractAssemblies) + { + ArgumentNullException.ThrowIfNull(registrations); + ArgumentNullException.ThrowIfNull(retiredIdentities); + ArgumentNullException.ThrowIfNull(handlers); + ArgumentNullException.ThrowIfNull(knownContractAssemblies); + + var assemblies = knownContractAssemblies.ToHashSet(); + if (assemblies.Count == 0) + { + throw new InvalidOperationException("The outbox registry needs a known contract assembly."); + } + + var known = new Dictionary(StringComparer.Ordinal); + var pending = new List(); + var subscriptionIds = new HashSet(StringComparer.Ordinal); + var activeHandlers = new HashSet<(Type EventType, Type HandlerType)>(); + var declaredTypes = new HashSet(); + + foreach (var registration in registrations) + { + if (registration is null) + { + throw new InvalidOperationException("An outbox contract registration is null."); + } + + var eventType = registration.EventType; + if (eventType is null || !assemblies.Contains(eventType.Assembly) + || !eventType.IsClass || eventType.IsAbstract || eventType.ContainsGenericParameters + || !typeof(IntegrationEventBase).IsAssignableFrom(eventType) + || !declaredTypes.Add(eventType)) + { + throw new InvalidOperationException( + "Outbox contracts must be distinct concrete IntegrationEventBase types " + + "from explicitly known contract assemblies."); + } + + var subscriptions = registration.Subscriptions; + if (registration.IsIntentionallyUnsubscribed == (subscriptions.Count > 0)) + { + throw new InvalidOperationException( + "An outbox contract needs required subscriptions or explicit intentional absence."); + } + + foreach (var subscription in subscriptions) + { + if (subscription is null || !ValidSubscriptionId(subscription.Id) + || !subscriptionIds.Add(subscription.Id)) + { + throw new InvalidOperationException( + "Outbox subscription IDs must be valid and globally unique."); + } + + if (subscription.Readiness == OutboxSubscriptionReadiness.Pending) + { + if (subscription.HandlerType is not null) + { + throw new InvalidOperationException("Pending subscriptions cannot name a handler."); + } + } + else if (subscription.Readiness == OutboxSubscriptionReadiness.Active + && subscription.HandlerType is + { + IsClass: true, IsAbstract: false, + ContainsGenericParameters: false + } handlerType) + { + if (!activeHandlers.Add((eventType, handlerType))) + { + throw new InvalidOperationException( + "One handler cannot stand for two independent Active subscriptions."); + } + } + else + { + throw new InvalidOperationException( + "An Active subscription needs a concrete handler type."); + } + } + + var disposition = registration.IsIntentionallyUnsubscribed + ? OutboxEventDisposition.IntentionallyUnsubscribed + : subscriptions.Any(static subscription => + subscription.Readiness == OutboxSubscriptionReadiness.Pending) + ? OutboxEventDisposition.Pending + : OutboxEventDisposition.Active; + + var resolution = new OutboxEventResolution( + disposition, eventType, eventType.Name, subscriptions); + AddIdentity(eventType.AssemblyQualifiedName, resolution); + foreach (var alias in registration.CompatibleAliases) + { + AddIdentity(alias, resolution); + } + } + + if (declaredTypes.Count == 0) + { + throw new InvalidOperationException("The outbox registry cannot have an empty contract census."); + } + + var retired = new HashSet(StringComparer.Ordinal); + foreach (var identity in retiredIdentities) + { + if (!ValidIdentity(identity) || known.ContainsKey(identity) || !retired.Add(identity)) + { + throw new InvalidOperationException( + "Retired outbox identities must be nonempty, unique and distinct from live identities."); + } + } + + var discovered = handlers.All.ToArray(); + if (discovered.Length != activeHandlers.Count + || discovered.Any(subscription => + !activeHandlers.Contains((subscription.EventType, subscription.HandlerType)))) + { + throw new InvalidOperationException( + "Every Active outbox subscription must have exactly one discovered handler, " + + "and every discovered handler must have an Active declaration."); + } + + return new OutboxEventRegistry( + known.ToFrozenDictionary(pair => pair.Key, pair => pair.Value, StringComparer.Ordinal), + retired.ToFrozenSet(StringComparer.Ordinal), + Array.AsReadOnly(pending.Order(StringComparer.Ordinal).ToArray()), + Array.AsReadOnly(declaredTypes.Select(type => type.Name) + .Distinct(StringComparer.Ordinal).Order(StringComparer.Ordinal).ToArray())); + + void AddIdentity(string? identity, OutboxEventResolution resolution) + { + if (!ValidIdentity(identity) || !known.TryAdd(identity, resolution)) + { + throw new InvalidOperationException( + "Canonical and compatible outbox identities must be nonempty and unique."); + } + + if (resolution.Disposition == OutboxEventDisposition.Pending) + { + pending.Add(identity); + } + } + } + + /// Looks up exact stored text without loading or reflecting over that text. + public OutboxEventResolution Resolve(string? storedIdentity) + { + if (storedIdentity is not null && _known.TryGetValue(storedIdentity, out var known)) + { + return known; + } + + return storedIdentity is not null && _retired.Contains(storedIdentity) + ? new OutboxEventResolution(OutboxEventDisposition.Retired, null, "retired", NoSubscriptions) + : new OutboxEventResolution(OutboxEventDisposition.Unknown, null, "unknown", NoSubscriptions); + } + + private static bool ValidIdentity([NotNullWhen(true)] string? identity) => + !string.IsNullOrWhiteSpace(identity) + && string.Equals(identity, identity.Trim(), StringComparison.Ordinal); + + private static bool ValidSubscriptionId(string? id) + { + if (id is null) return false; + + var firstDot = id.IndexOf('.'); + var lastDot = id.LastIndexOf('.'); + if (firstDot < 1 || lastDot <= firstDot + 1 || lastDot == id.Length - 1) + { + return false; + } + + var module = id.AsSpan(0, firstDot); + var subscription = id.AsSpan(firstDot + 1, lastDot - firstDot - 1); + var version = id.AsSpan(lastDot + 1); + return IsLowerLetter(module[0]) + && IsAsciiTail(module[1..], allowHyphen: false) + && IsLowerLetter(subscription[0]) + && IsAsciiTail(subscription[1..], allowHyphen: true) + && version.Length >= 2 && version[0] == 'v' + && version[1] is >= '1' and <= '9' + && IsDecimalTail(version[2..]); + } + + private static bool IsLowerLetter(char value) => value is >= 'a' and <= 'z'; + + private static bool IsAsciiTail(ReadOnlySpan value, bool allowHyphen) + { + foreach (var character in value) + { + if (!IsLowerLetter(character) + && character is not (>= '0' and <= '9') + && (!allowHyphen || character != '-')) + { + return false; + } + } + + return true; + } + + private static bool IsDecimalTail(ReadOnlySpan value) + { + foreach (var character in value) + { + if (character is not (>= '0' and <= '9')) + { + return false; + } + } + + return true; + } +} diff --git a/backend/src/LearnStack.Infrastructure/Persistence/Migrations/20261012180355_add_outbox_dispatch_state.Designer.cs b/backend/src/LearnStack.Infrastructure/Persistence/Migrations/20261012180355_add_outbox_dispatch_state.Designer.cs new file mode 100644 index 00000000..fbee5fa3 --- /dev/null +++ b/backend/src/LearnStack.Infrastructure/Persistence/Migrations/20261012180355_add_outbox_dispatch_state.Designer.cs @@ -0,0 +1,29 @@ +// +using LearnStack.Infrastructure.Persistence; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Migrations; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; +using Npgsql.EntityFrameworkCore.PostgreSQL.Metadata; + +#nullable disable + +namespace LearnStack.Infrastructure.Persistence.Migrations +{ + [DbContext(typeof(PlatformDbContext))] + [Migration("20261012180355_add_outbox_dispatch_state")] + partial class add_outbox_dispatch_state + { + /// + protected override void BuildTargetModel(ModelBuilder modelBuilder) + { +#pragma warning disable 612, 618 + modelBuilder + .HasAnnotation("ProductVersion", "10.0.12") + .HasAnnotation("Relational:MaxIdentifierLength", 63); + + NpgsqlModelBuilderExtensions.UseIdentityByDefaultColumns(modelBuilder); +#pragma warning restore 612, 618 + } + } +} diff --git a/backend/src/LearnStack.Infrastructure/Persistence/Migrations/20261012180355_add_outbox_dispatch_state.cs b/backend/src/LearnStack.Infrastructure/Persistence/Migrations/20261012180355_add_outbox_dispatch_state.cs new file mode 100644 index 00000000..84f41714 --- /dev/null +++ b/backend/src/LearnStack.Infrastructure/Persistence/Migrations/20261012180355_add_outbox_dispatch_state.cs @@ -0,0 +1,102 @@ +using Microsoft.EntityFrameworkCore.Migrations; + +#nullable disable + +namespace LearnStack.Infrastructure.Persistence.Migrations +{ + /// + public partial class add_outbox_dispatch_state : Migration + { + /// + protected override void Up(MigrationBuilder migrationBuilder) + { + // ADR-0046: legacy diagnostics are deliberately not retained. This + // owner-only, transactional backfill temporarily removes FORCE so + // it reaches every tenant, then restores it before the migration + // commits. Invalid legacy identities fail the CHECK without repair. + migrationBuilder.Sql(""" + ALTER TABLE outbox_messages NO FORCE ROW LEVEL SECURITY; + UPDATE outbox_messages + SET last_error = 'migration:legacy:redacted' + WHERE last_error IS NOT NULL; + ALTER TABLE outbox_messages FORCE ROW LEVEL SECURITY; + + ALTER TABLE outbox_messages + ADD COLUMN locked_by text NULL, + ADD COLUMN locked_until timestamptz NULL, + ADD COLUMN terminal_reason text NULL, + ADD COLUMN dead_lettered_at timestamptz NULL, + ADD CONSTRAINT ck_outbox_messages_identity CHECK ( + id <> '00000000-0000-0000-0000-000000000000'::uuid + AND tenant_id NOT IN ('00000000-0000-0000-0000-000000000000'::uuid, + '00000000-0000-7000-8000-000000000002'::uuid) + AND (organization_id IS NULL OR + organization_id <> '00000000-0000-0000-0000-000000000000'::uuid)), + ADD CONSTRAINT ck_outbox_messages_attempts CHECK (attempts >= 0), + ADD CONSTRAINT ck_outbox_messages_last_error_size CHECK ( + last_error IS NULL OR octet_length(last_error) <= 256), + ADD CONSTRAINT ck_outbox_messages_lease CHECK ( + (locked_by IS NULL AND locked_until IS NULL) OR + (locked_by IS NOT NULL AND length(locked_by) > 0 AND locked_until IS NOT NULL)), + ADD CONSTRAINT ck_outbox_messages_terminal_reason CHECK ( + terminal_reason IS NULL OR terminal_reason IN ( + 'unknown_type', 'retired_type', 'invalid_envelope', 'retry_exhausted')), + ADD CONSTRAINT ck_outbox_messages_state CHECK ( + (processed_at IS NULL OR + (dead_lettered_at IS NULL AND terminal_reason IS NULL AND locked_by IS NULL)) + AND (dead_lettered_at IS NULL OR + (processed_at IS NULL AND terminal_reason IS NOT NULL AND locked_by IS NULL))); + + DROP INDEX ix_outbox_messages_pending; + DROP INDEX ix_outbox_messages_tenant_pending; + CREATE INDEX ix_outbox_messages_pending + ON outbox_messages (available_after, occurred_at, id) + WHERE processed_at IS NULL AND dead_lettered_at IS NULL + AND terminal_reason IS NULL AND attempts < 6 AND locked_by IS NULL; + CREATE INDEX ix_outbox_messages_tenant_pending + ON outbox_messages (tenant_id, available_after) + WHERE processed_at IS NULL AND dead_lettered_at IS NULL; + CREATE INDEX ix_outbox_messages_recovery + ON outbox_messages (locked_until, occurred_at, id) + WHERE processed_at IS NULL AND dead_lettered_at IS NULL + AND terminal_reason IS NULL AND locked_by IS NOT NULL; + CREATE INDEX ix_outbox_messages_terminal + ON outbox_messages (available_after, occurred_at, id) + WHERE processed_at IS NULL AND dead_lettered_at IS NULL AND terminal_reason IS NOT NULL; + + GRANT UPDATE (locked_by, locked_until, terminal_reason, dead_lettered_at) + ON outbox_messages TO learnstack_outbox_admin; + """); + } + + /// + protected override void Down(MigrationBuilder migrationBuilder) + { + // Schema reversal for disposable databases only. Sanitized legacy + // diagnostics cannot be restored; live rollback must be forward-fix. + migrationBuilder.Sql(""" + REVOKE UPDATE (locked_by, locked_until, terminal_reason, dead_lettered_at) + ON outbox_messages FROM learnstack_outbox_admin; + DROP INDEX ix_outbox_messages_terminal; + DROP INDEX ix_outbox_messages_recovery; + DROP INDEX ix_outbox_messages_pending; + DROP INDEX ix_outbox_messages_tenant_pending; + ALTER TABLE outbox_messages + DROP CONSTRAINT ck_outbox_messages_state, + DROP CONSTRAINT ck_outbox_messages_terminal_reason, + DROP CONSTRAINT ck_outbox_messages_lease, + DROP CONSTRAINT ck_outbox_messages_last_error_size, + DROP CONSTRAINT ck_outbox_messages_attempts, + DROP CONSTRAINT ck_outbox_messages_identity, + DROP COLUMN dead_lettered_at, + DROP COLUMN terminal_reason, + DROP COLUMN locked_until, + DROP COLUMN locked_by; + CREATE INDEX ix_outbox_messages_pending + ON outbox_messages (available_after) WHERE processed_at IS NULL; + CREATE INDEX ix_outbox_messages_tenant_pending + ON outbox_messages (tenant_id, available_after) WHERE processed_at IS NULL; + """); + } + } +} diff --git a/backend/src/LearnStack.Infrastructure/Persistence/Migrations/20261012222654_align_outbox_recovery_index.Designer.cs b/backend/src/LearnStack.Infrastructure/Persistence/Migrations/20261012222654_align_outbox_recovery_index.Designer.cs new file mode 100644 index 00000000..61e61153 --- /dev/null +++ b/backend/src/LearnStack.Infrastructure/Persistence/Migrations/20261012222654_align_outbox_recovery_index.Designer.cs @@ -0,0 +1,29 @@ +// +using LearnStack.Infrastructure.Persistence; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Migrations; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; +using Npgsql.EntityFrameworkCore.PostgreSQL.Metadata; + +#nullable disable + +namespace LearnStack.Infrastructure.Persistence.Migrations +{ + [DbContext(typeof(PlatformDbContext))] + [Migration("20261012222654_align_outbox_recovery_index")] + partial class align_outbox_recovery_index + { + /// + protected override void BuildTargetModel(ModelBuilder modelBuilder) + { +#pragma warning disable 612, 618 + modelBuilder + .HasAnnotation("ProductVersion", "10.0.12") + .HasAnnotation("Relational:MaxIdentifierLength", 63); + + NpgsqlModelBuilderExtensions.UseIdentityByDefaultColumns(modelBuilder); +#pragma warning restore 612, 618 + } + } +} diff --git a/backend/src/LearnStack.Infrastructure/Persistence/Migrations/20261012222654_align_outbox_recovery_index.cs b/backend/src/LearnStack.Infrastructure/Persistence/Migrations/20261012222654_align_outbox_recovery_index.cs new file mode 100644 index 00000000..81cd4229 --- /dev/null +++ b/backend/src/LearnStack.Infrastructure/Persistence/Migrations/20261012222654_align_outbox_recovery_index.cs @@ -0,0 +1,36 @@ +using Microsoft.EntityFrameworkCore.Migrations; + +#nullable disable + +namespace LearnStack.Infrastructure.Persistence.Migrations +{ + /// + public partial class align_outbox_recovery_index : Migration + { + /// + protected override void Up(MigrationBuilder migrationBuilder) + { + // ADR-0046 requires the same total selection order for both recovery + // branches. Keep ordinary unleased Pending backlog outside the index. + migrationBuilder.Sql(""" + DROP INDEX ix_outbox_messages_recovery; + CREATE INDEX ix_outbox_messages_recovery + ON outbox_messages (available_after, occurred_at, id) + WHERE processed_at IS NULL AND dead_lettered_at IS NULL + AND terminal_reason IS NULL AND (locked_by IS NOT NULL OR attempts >= 6); + """); + } + + /// + protected override void Down(MigrationBuilder migrationBuilder) + { + migrationBuilder.Sql(""" + DROP INDEX ix_outbox_messages_recovery; + CREATE INDEX ix_outbox_messages_recovery + ON outbox_messages (locked_until, occurred_at, id) + WHERE processed_at IS NULL AND dead_lettered_at IS NULL + AND terminal_reason IS NULL AND locked_by IS NOT NULL; + """); + } + } +} diff --git a/backend/src/LearnStack.Infrastructure/Persistence/Migrations/PlatformDbContextModelSnapshot.cs b/backend/src/LearnStack.Infrastructure/Persistence/Migrations/PlatformDbContextModelSnapshot.cs index 025218fd..b1614d25 100644 --- a/backend/src/LearnStack.Infrastructure/Persistence/Migrations/PlatformDbContextModelSnapshot.cs +++ b/backend/src/LearnStack.Infrastructure/Persistence/Migrations/PlatformDbContextModelSnapshot.cs @@ -16,7 +16,7 @@ protected override void BuildModel(ModelBuilder modelBuilder) { #pragma warning disable 612, 618 modelBuilder - .HasAnnotation("ProductVersion", "10.0.0") + .HasAnnotation("ProductVersion", "10.0.12") .HasAnnotation("Relational:MaxIdentifierLength", 63); NpgsqlModelBuilderExtensions.UseIdentityByDefaultColumns(modelBuilder); diff --git a/backend/src/LearnStack.Infrastructure/Persistence/PlatformDbContext.cs b/backend/src/LearnStack.Infrastructure/Persistence/PlatformDbContext.cs index 65cd0b0b..cc0aeb98 100644 --- a/backend/src/LearnStack.Infrastructure/Persistence/PlatformDbContext.cs +++ b/backend/src/LearnStack.Infrastructure/Persistence/PlatformDbContext.cs @@ -11,9 +11,9 @@ namespace LearnStack.Infrastructure.Persistence; /// are reached through SharedKernel ports by any module's handler and read by /// infrastructure that belongs to none of them. IIdempotencyStore ships /// today, behind InMemoryIdempotencyStore until its ADR-0035 trigger -/// fires; IOutbox does not exist yet and lands with the dispatcher in -/// Phase 02b. -/// Nothing writes either table at runtime — Packet 6 shipped the tables. Putting +/// fires; IOutbox enqueues immediately on the ambient transaction under +/// ADR-0057. +/// Dispatcher leases and terminal state are decided by ADR-0046. Putting /// them in a module's context would make every other module's use of the outbox a /// dependency on that module — the shape /// 15-event-and-outbox.md @@ -21,15 +21,11 @@ namespace LearnStack.Infrastructure.Persistence; /// module. /// /// -/// It is the second DbContext, and that does not yet make -/// ADR-0040's -/// central property testable. That property is several contexts enlisted on one -/// connection so SET LOCAL protects every statement, and the enlistment -/// machinery — IUnitOfWork, the shared registration helper, the -/// TransactionBehavior body — lands in step 6; ADR-0040 § What Packet 6 -/// can and cannot prove says the property becomes observable in Phase 03, with -/// the second module context. This context exists for the reason below and -/// no other. +/// This migration-only context has no tracked entities. Runtime outbox SQL uses +/// the announced IUnitOfWork connection, while confined dispatch uses its +/// own role and short transactions. Module contexts enlist through the shared +/// ADR-0040 +/// registration and never acquire a dispatcher connection. /// /// /// Both tables are tenant-owned, tenant-wide despite living outside a diff --git a/backend/src/LearnStack.SharedKernel/Audit/EventTerminalTransition.cs b/backend/src/LearnStack.SharedKernel/Audit/EventTerminalTransition.cs new file mode 100644 index 00000000..f84db3c0 --- /dev/null +++ b/backend/src/LearnStack.SharedKernel/Audit/EventTerminalTransition.cs @@ -0,0 +1,168 @@ +using System.Buffers; +using System.Data; +using System.Data.Common; +using System.Diagnostics; +using System.Globalization; +using System.Text; +using System.Text.Json; +using LearnStack.SharedKernel.Identifiers; + +namespace LearnStack.SharedKernel.Audit; + +/// The immutable result of one fenced terminal UPDATE in its live transaction. +/// +/// Construction is private; the protected minting seam is confined to the producer +/// transition store by the production architecture census. It grants no friend-assembly +/// access to other SharedKernel authority. Subscriber minting belongs to P02b-3. +/// +public sealed class EventTerminalTransition +{ + private readonly DbConnection _connection; + private readonly DbTransaction _transaction; + + private EventTerminalTransition( + DbConnection connection, + DbTransaction transaction, + DbDataReader returnedRow, + string validatedAttemptTraceparent) + { + EnsureLiveTransaction(connection, transaction); + if (!TryCanonicalTraceparent(validatedAttemptTraceparent, out var attempt) + || !string.Equals(attempt, validatedAttemptTraceparent, StringComparison.Ordinal)) + { + throw new ArgumentException("The terminal attempt requires a canonical W3C traceparent."); + } + + if (returnedRow.FieldCount != 9) + { + throw new ArgumentException("The terminal UPDATE must return its complete bound row."); + } + + _connection = connection; + _transaction = transaction; + EventId = returnedRow.GetGuid(0); + TenantId = returnedRow.GetGuid(1); + OrganizationId = returnedRow.IsDBNull(2) ? null : returnedRow.GetGuid(2); + Reason = returnedRow.GetString(3); + Attempts = returnedRow.GetInt32(4); + AcquisitionToken = returnedRow.GetString(5); + + if (EventId == Guid.Empty || TenantId == Guid.Empty + || TenantId == Identifiers.TenantId.PlatformSentinel.Value + || OrganizationId == Guid.Empty || Attempts < 0 + || Reason is not ("unknown_type" or "retired_type" or "invalid_envelope" or "retry_exhausted") + || !Guid.TryParseExact(AcquisitionToken, "D", out var token) + || token == Guid.Empty + || !string.Equals(AcquisitionToken, token.ToString("D"), StringComparison.Ordinal)) + { + throw new ArgumentException("The terminal UPDATE returned invalid transition identity."); + } + + Guid? causationId = returnedRow.IsDBNull(7) ? null : returnedRow.GetGuid(7); + Guid? causalActorUserId = returnedRow.IsDBNull(8) ? null : returnedRow.GetGuid(8); + CorrelationId = TryCanonicalTraceparent( + returnedRow.IsDBNull(6) ? null : returnedRow.GetString(6), out var original) + && causationId != Guid.Empty && causalActorUserId != Guid.Empty + ? original : attempt; + Metadata = ComposeMetadata(causationId, causalActorUserId); + } + + public Guid EventId { get; } + public Guid TenantId { get; } + public Guid? OrganizationId { get; } + public string Reason { get; } + public int Attempts { get; } + public string AcquisitionToken { get; } + public string CorrelationId { get; } + public string Metadata { get; } + public string Kind { get; } = "producer"; + public string? SubscriptionId { get; } + + /// Refuses a draft or transaction that does not describe this exact transition. + public void EnsureMatches( + AuditEntryDraft entry, DbConnection connection, DbTransaction transaction) + { + ArgumentNullException.ThrowIfNull(entry); + ArgumentNullException.ThrowIfNull(connection); + ArgumentNullException.ThrowIfNull(transaction); + EnsureLiveTransaction(connection, transaction); + + if (!ReferenceEquals(_connection, connection) || !ReferenceEquals(_transaction, transaction) + || !entry.Id.IsInitialized() || entry.Id.Value == Guid.Empty + || !entry.TenantId.IsInitialized() || entry.TenantId.Value != TenantId + || (entry.OrganizationId is { } organization && !organization.IsInitialized()) + || entry.OrganizationId?.Value != OrganizationId + || !string.Equals(entry.Metadata, Metadata, StringComparison.Ordinal) + || !string.Equals(entry.CorrelationId, CorrelationId, StringComparison.Ordinal) + || entry.ActorUserId is not { } actor || !actor.IsInitialized() || actor != UserId.SystemActor + || entry.ActorEmail is not null + || entry.Outcome != AuditOutcome.Success || entry.ErrorKey is not null + || entry.Reason is not null || entry.EntityType is not null || entry.EntityId is not null + || entry.BeforeState is not null || entry.AfterState is not null || entry.Changes is not null + || entry.IpAddress is not null || entry.UserAgent is not null) + { + throw new ArgumentException("The terminal audit draft does not match its live transition.", nameof(entry)); + } + } + + /// The only construction capability; its production subtype and call site are closed. +#pragma warning disable CA1034 // Nesting permits private witness construction without SharedKernel friend assemblies. + public abstract class MintingSource +#pragma warning restore CA1034 + { + protected MintingSource() { } + + /// Reads the actual nine-column result of the successful fenced producer UPDATE. + protected static EventTerminalTransition FromProducerUpdate( + DbConnection connection, + DbTransaction transaction, + DbDataReader returnedRow, + string validatedAttemptTraceparent) + { + ArgumentNullException.ThrowIfNull(connection); + ArgumentNullException.ThrowIfNull(transaction); + ArgumentNullException.ThrowIfNull(returnedRow); + return new EventTerminalTransition(connection, transaction, returnedRow, validatedAttemptTraceparent); + } + } + + private static void EnsureLiveTransaction(DbConnection connection, DbTransaction transaction) + { + if (connection.State != ConnectionState.Open || !ReferenceEquals(transaction.Connection, connection)) + { + throw new ArgumentException("A terminal transition requires the caller's live transaction."); + } + } + + private string ComposeMetadata(Guid? causationId, Guid? causalActorUserId) + { + var buffer = new ArrayBufferWriter(); + using (var writer = new Utf8JsonWriter(buffer)) + { + writer.WriteStartObject(); + writer.WriteString("event_id", EventId); + writer.WriteString("reason", Reason); + writer.WriteNumber("attempts", Attempts); + if (causationId is { } causation && causation != Guid.Empty) + writer.WriteString("causation_id", causation); + if (causalActorUserId is { } actor && actor != Guid.Empty) + writer.WriteString("causal_actor_user_id", actor); + writer.WriteEndObject(); + } + + return Encoding.UTF8.GetString(buffer.WrittenSpan); + } + + private static bool TryCanonicalTraceparent(string? value, out string canonical) + { + if (ActivityContext.TryParse(value, traceState: null, out var context)) + { + canonical = string.Concat("00-", context.TraceId.ToHexString(), "-", context.SpanId.ToHexString(), + "-", ((byte)context.TraceFlags).ToString("x2", CultureInfo.InvariantCulture)); + return true; + } + + canonical = string.Empty; + return false; + } +} diff --git a/backend/src/LearnStack.SharedKernel/Audit/IAuditStore.cs b/backend/src/LearnStack.SharedKernel/Audit/IAuditStore.cs index 9503c46b..f8fcb625 100644 --- a/backend/src/LearnStack.SharedKernel/Audit/IAuditStore.cs +++ b/backend/src/LearnStack.SharedKernel/Audit/IAuditStore.cs @@ -4,7 +4,7 @@ namespace LearnStack.SharedKernel.Audit; /// -/// The only sanctioned way to put a row in audit_log. Four writes, and +/// The only sanctioned way to put a row in audit_log. Five writes, and /// no update. /// /// @@ -26,6 +26,11 @@ namespace LearnStack.SharedKernel.Audit; /// path does not own, as a role the other three never use, and before the /// operation it describes. "No update method" is untouched — this is a fourth write. /// +/// +/// ADR-0046 adds as the fifth write: a +/// terminal transition and its MUST row share the dispatcher's own transaction, +/// with a restricted witness binding the draft to the actual fenced UPDATE. +/// /// public interface IAuditStore { @@ -119,4 +124,20 @@ Task WritePlatformScopeAsync( DbConnection connection, DbTransaction transaction, CancellationToken cancellationToken = default); + + /// Inserts a MUST terminal audit on the successful transition's live transaction. + /// + /// The producer transition store is the only caller in P02b-2. The operation + /// must be declared off-path as MUST/SecurityEvent, and the draft must match + /// the actual UPDATE result carried by . + /// Any mismatch or write failure requires the caller to roll back the state + /// transition. This seam opens and commits no transaction. P02b-3 owns the + /// future subscriber writer; the existing four writes retain their contracts. + /// + Task WriteEventTerminalAsync( + AuditEntryDraft entry, + EventTerminalTransition transition, + DbConnection connection, + DbTransaction transaction, + CancellationToken cancellationToken = default); } diff --git a/backend/src/LearnStack.SharedKernel/Messaging/IIntegrationEvent.cs b/backend/src/LearnStack.SharedKernel/Messaging/IIntegrationEvent.cs index f5d7cc85..1c31a914 100644 --- a/backend/src/LearnStack.SharedKernel/Messaging/IIntegrationEvent.cs +++ b/backend/src/LearnStack.SharedKernel/Messaging/IIntegrationEvent.cs @@ -10,15 +10,15 @@ namespace LearnStack.SharedKernel.Messaging; /// implementing this interface directly, so that every event carries the same /// identity, tenancy and ordering fields and a consumer can rely on them without /// knowing the type. The architecture test -/// Integration_Events_Inherit_From_IntegrationEventBase is registered for -/// Phase 02b, -/// with the first event type for it to hold to that. +/// Integration_Events_Inherit_From_IntegrationEventBase is implemented by +/// P02b-2 against the declared concrete contract registry. /// public interface IIntegrationEvent { /// Identity for consumer-side deduplication. /// - /// Delivery is at-least-once, so this is what IInboxGuard keys on. + /// Delivery is at-least-once; P02b-3's transport-owned shared inbox uses this + /// value together with tenant and stable subscription identity. /// It is assigned once by the producer and never re-derived — a redelivery /// carries the same value, which is the entire point. /// @@ -51,14 +51,14 @@ public interface IIntegrationEvent string Topic { get; } /// - /// The ordering domain this event belongs to. + /// The partition identity this event belongs to. /// /// - /// Ordering is guaranteed per partition key and nowhere else, so a partition - /// key nobody sets is a guarantee nobody has. Normally the id of the - /// aggregate the event is about; the tenant id for events about the tenant as - /// a whole. Resolved once, by the producer that knows the domain, and never - /// re-derived downstream. + /// Normally the id of the aggregate the event is about; the tenant id for + /// events about the tenant as a whole. Resolved once by the producer, never + /// re-derived downstream. ADR-0046 bounds eligible selection within one + /// processor; this key alone promises no enqueue, commit, retry or + /// cross-process publication order. /// string PartitionKey { get; } } diff --git a/backend/src/LearnStack.SharedKernel/Messaging/IIntegrationEventHandler.cs b/backend/src/LearnStack.SharedKernel/Messaging/IIntegrationEventHandler.cs index a7b200d4..65a6ccb0 100644 --- a/backend/src/LearnStack.SharedKernel/Messaging/IIntegrationEventHandler.cs +++ b/backend/src/LearnStack.SharedKernel/Messaging/IIntegrationEventHandler.cs @@ -13,13 +13,14 @@ namespace LearnStack.SharedKernel.Messaging; /// production. Both transports resolve this same contract. /// /// -/// A handler must call IInboxGuard.IsAlreadyProcessedAsync before any -/// business logic. Delivery is at-least-once by design, so deduplication is -/// the consumer's obligation rather than the transport's. The guard, its -/// per-module inbox_messages table and the architecture test that holds -/// every handler to it, Integration_Event_Handlers_Use_InboxGuard, land in -/// Phase 02b; -/// the contract is shaped for it now so no handler is written twice. +/// Delivery is at-least-once. Under +/// ADR-0046, +/// the transport claims the shared IInboxGuard before constructing the +/// handler; handlers do not call it. Business state, inbox completion and fenced +/// delivery completion share one transaction. Durable acceptance, the shared +/// inbox_messages table and the nonempty production proof for +/// Integration_Event_Handlers_Use_InboxGuard belong to P02b-3. +/// The shipped in-process baseline does not yet provide durable deduplication. /// /// /// diff --git a/backend/src/LearnStack.Tools.Seeder/DemoDatabaseConfiguration.cs b/backend/src/LearnStack.Tools.Seeder/DemoDatabaseConfiguration.cs deleted file mode 100644 index 9d280687..00000000 --- a/backend/src/LearnStack.Tools.Seeder/DemoDatabaseConfiguration.cs +++ /dev/null @@ -1,32 +0,0 @@ -using Npgsql; - -namespace LearnStack.Tools.Seeder; - -/// Rejects stale local credentials before infrastructure or migrations run. -public static class DemoDatabaseConfiguration -{ - public static bool IsValid(IReadOnlyDictionary values) - { - try - { - foreach (var (key, role, password) in new[] - { - ("Migration", "learnstack_migration", "LEARNSTACK_MIGRATION_PW"), - ("Default", "learnstack_app", "LEARNSTACK_APP_PW"), - ("PlatformAdmin", "learnstack_platform", "LEARNSTACK_PLATFORM_PW"), - ("OutboxDispatcher", "learnstack_outbox_admin", "LEARNSTACK_OUTBOX_PW"), - }) - { - var builder = new NpgsqlConnectionStringBuilder(values[$"ConnectionStrings__{key}"]); - if (builder.Username != role || builder.Password != values[password] - || builder.Database != values["POSTGRES_DB"] || builder.Port != 5432 - || builder.Host is not ("localhost" or "127.0.0.1")) return false; - } - return true; - } - catch (Exception failure) when (failure is ArgumentException or KeyNotFoundException) - { - return false; - } - } -} diff --git a/backend/src/LearnStack.Tools.Seeder/Program.cs b/backend/src/LearnStack.Tools.Seeder/Program.cs index 2fe700e6..2ecae1cb 100644 --- a/backend/src/LearnStack.Tools.Seeder/Program.cs +++ b/backend/src/LearnStack.Tools.Seeder/Program.cs @@ -15,15 +15,6 @@ Console.WriteLine(JsonSerializer.Serialize(DemoManifest.Create(), JsonSerializerOptions.Web)); return 0; } -if (args is ["--validate-demo-config"]) -{ - // Configuration arrives via the environment, never credential-bearing argv. - var values = Environment.GetEnvironmentVariables().Cast() - .ToDictionary(row => (string)row.Key, row => (string)row.Value!, StringComparer.Ordinal); - if (DemoDatabaseConfiguration.IsValid(values)) return 0; - Console.Error.WriteLine("demo: database configuration disagrees with the selected local stack."); - return 2; -} if (args.Length != 0) { Console.Error.WriteLine("seed: unsupported arguments."); diff --git a/backend/src/Modules/Audit/LearnStack.Modules.Audit.Infrastructure/Persistence/Migrations/20261012180433_grant_outbox_terminal_audit.Designer.cs b/backend/src/Modules/Audit/LearnStack.Modules.Audit.Infrastructure/Persistence/Migrations/20261012180433_grant_outbox_terminal_audit.Designer.cs new file mode 100644 index 00000000..a461c101 --- /dev/null +++ b/backend/src/Modules/Audit/LearnStack.Modules.Audit.Infrastructure/Persistence/Migrations/20261012180433_grant_outbox_terminal_audit.Designer.cs @@ -0,0 +1,231 @@ +// +using System; +using System.Net; +using LearnStack.Modules.Audit.Infrastructure.Persistence; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Migrations; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; +using Npgsql.EntityFrameworkCore.PostgreSQL.Metadata; + +#nullable disable + +namespace LearnStack.Modules.Audit.Infrastructure.Persistence.Migrations +{ + [DbContext(typeof(AuditDbContext))] + [Migration("20261012180433_grant_outbox_terminal_audit")] + partial class grant_outbox_terminal_audit + { + /// + protected override void BuildTargetModel(ModelBuilder modelBuilder) + { +#pragma warning disable 612, 618 + modelBuilder + .HasAnnotation("ProductVersion", "10.0.12") + .HasAnnotation("Relational:MaxIdentifierLength", 63); + + NpgsqlModelBuilderExtensions.UseIdentityByDefaultColumns(modelBuilder); + + modelBuilder.Entity("LearnStack.Modules.Audit.Domain.AuditConfig", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("CreatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("created_at"); + + b.Property("CreatedBy") + .HasColumnType("uuid") + .HasColumnName("created_by"); + + b.Property("DeletedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("deleted_at"); + + b.Property("DeletedBy") + .HasColumnType("uuid") + .HasColumnName("deleted_by"); + + b.Property("IsEnabled") + .HasColumnType("boolean") + .HasColumnName("is_enabled"); + + b.Property("ModuleName") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("character varying(50)") + .HasColumnName("module"); + + b.Property("Operation") + .IsRequired() + .HasMaxLength(150) + .HasColumnType("character varying(150)") + .HasColumnName("operation"); + + b.Property("TenantId") + .HasColumnType("uuid") + .HasColumnName("tenant_id"); + + b.Property("UpdatedAt") + .HasColumnType("timestamp with time zone") + .HasColumnName("updated_at"); + + b.Property("UpdatedBy") + .HasColumnType("uuid") + .HasColumnName("updated_by"); + + b.Property("Version") + .IsConcurrencyToken() + .HasColumnType("bigint") + .HasDefaultValue(0L) + .HasColumnName("row_version"); + + b.HasKey("Id") + .HasName("pk_audit_config"); + + b.HasIndex("TenantId", "ModuleName", "Operation") + .IsUnique() + .HasDatabaseName("ux_audit_config_tenant_id_module_operation") + .HasFilter("deleted_at IS NULL"); + + b.ToTable("audit_config", (string)null); + }); + + modelBuilder.Entity("LearnStack.Modules.Audit.Domain.AuditEntry", b => + { + b.Property("Id") + .HasColumnType("uuid") + .HasColumnName("id"); + + b.Property("Timestamp") + .HasColumnType("timestamp with time zone") + .HasColumnName("timestamp"); + + b.Property("ActorEmail") + .HasMaxLength(320) + .HasColumnType("character varying(320)") + .HasColumnName("actor_email"); + + b.Property("ActorUserId") + .HasColumnType("uuid") + .HasColumnName("actor_user_id"); + + b.Property("AfterState") + .HasColumnType("jsonb") + .HasColumnName("after_state"); + + b.Property("BeforeState") + .HasColumnType("jsonb") + .HasColumnName("before_state"); + + b.Property("Changes") + .HasColumnType("jsonb") + .HasColumnName("changes"); + + b.Property("CorrelationId") + .HasMaxLength(100) + .HasColumnType("character varying(100)") + .HasColumnName("correlation_id"); + + b.Property("EntityId") + .HasColumnType("text") + .HasColumnName("entity_id"); + + b.Property("EntityType") + .HasMaxLength(200) + .HasColumnType("character varying(200)") + .HasColumnName("entity_type"); + + b.Property("ErrorKey") + .HasMaxLength(150) + .HasColumnType("character varying(150)") + .HasColumnName("error_key"); + + b.Property("IpAddress") + .HasColumnType("inet") + .HasColumnName("ip_address"); + + b.Property("Metadata") + .HasColumnType("jsonb") + .HasColumnName("metadata"); + + b.Property("ModuleName") + .IsRequired() + .HasMaxLength(50) + .HasColumnType("character varying(50)") + .HasColumnName("module"); + + b.Property("Operation") + .IsRequired() + .HasMaxLength(150) + .HasColumnType("character varying(150)") + .HasColumnName("operation"); + + b.Property("OperationClass") + .IsRequired() + .HasColumnType("text") + .HasColumnName("operation_class"); + + b.Property("OperationType") + .IsRequired() + .HasColumnType("text") + .HasColumnName("operation_type"); + + b.Property("OrganizationId") + .HasColumnType("uuid") + .HasColumnName("organization_id"); + + b.Property("Outcome") + .IsRequired() + .HasColumnType("text") + .HasColumnName("outcome"); + + b.Property("Reason") + .HasMaxLength(500) + .HasColumnType("character varying(500)") + .HasColumnName("reason"); + + b.Property("TenantId") + .HasColumnType("uuid") + .HasColumnName("tenant_id"); + + b.Property("UserAgent") + .HasMaxLength(512) + .HasColumnType("character varying(512)") + .HasColumnName("user_agent"); + + b.HasKey("Id", "Timestamp") + .HasName("audit_log_pkey"); + + b.HasIndex("CorrelationId") + .HasDatabaseName("ix_audit_log_correlation_id") + .HasFilter("correlation_id IS NOT NULL"); + + b.HasIndex("ActorUserId", "Timestamp") + .IsDescending(false, true) + .HasDatabaseName("ix_audit_log_actor_user_id_timestamp") + .HasFilter("actor_user_id IS NOT NULL"); + + b.HasIndex("TenantId", "Timestamp") + .IsDescending(false, true) + .HasDatabaseName("ix_audit_log_tenant_id_timestamp"); + + b.HasIndex("ModuleName", "Operation", "Timestamp") + .IsDescending(false, false, true) + .HasDatabaseName("ix_audit_log_module_operation_timestamp"); + + b.ToTable("audit_log", null, t => + { + t.HasCheckConstraint("ck_audit_log_operation_class", "operation_class IN ('Must', 'Should', 'May')"); + + t.HasCheckConstraint("ck_audit_log_operation_type", "operation_type IN ('Create', 'Update', 'Delete', 'ReadSensitive', 'SecurityEvent', 'PlatformAdmin', 'Action')"); + + t.HasCheckConstraint("ck_audit_log_outcome", "outcome IN ('success', 'denied', 'failed', 'indeterminate')"); + }); + }); +#pragma warning restore 612, 618 + } + } +} diff --git a/backend/src/Modules/Audit/LearnStack.Modules.Audit.Infrastructure/Persistence/Migrations/20261012180433_grant_outbox_terminal_audit.cs b/backend/src/Modules/Audit/LearnStack.Modules.Audit.Infrastructure/Persistence/Migrations/20261012180433_grant_outbox_terminal_audit.cs new file mode 100644 index 00000000..9d97f118 --- /dev/null +++ b/backend/src/Modules/Audit/LearnStack.Modules.Audit.Infrastructure/Persistence/Migrations/20261012180433_grant_outbox_terminal_audit.cs @@ -0,0 +1,25 @@ +using Microsoft.EntityFrameworkCore.Migrations; + +#nullable disable + +namespace LearnStack.Modules.Audit.Infrastructure.Persistence.Migrations +{ + /// + public partial class grant_outbox_terminal_audit : Migration + { + /// + protected override void Up(MigrationBuilder migrationBuilder) + { + // ADR-0046: the confined terminal writer shares the dispatcher's + // transaction. The role receives no audit read/update/delete access. + migrationBuilder.Sql("GRANT INSERT ON audit_log TO learnstack_outbox_admin;"); + } + + /// + protected override void Down(MigrationBuilder migrationBuilder) + { + // Disposable-schema reversal only; live recovery uses a forward fix. + migrationBuilder.Sql("REVOKE INSERT ON audit_log FROM learnstack_outbox_admin;"); + } + } +} diff --git a/backend/src/Modules/Tenancy/LearnStack.Modules.Tenancy.Application/Audit/TenancyAuditCatalogSource.cs b/backend/src/Modules/Tenancy/LearnStack.Modules.Tenancy.Application/Audit/TenancyAuditCatalogSource.cs index 22fe6e60..2d0df4f0 100644 --- a/backend/src/Modules/Tenancy/LearnStack.Modules.Tenancy.Application/Audit/TenancyAuditCatalogSource.cs +++ b/backend/src/Modules/Tenancy/LearnStack.Modules.Tenancy.Application/Audit/TenancyAuditCatalogSource.cs @@ -15,7 +15,7 @@ namespace LearnStack.Modules.Tenancy.Application.Audit; /// /// Only the operations whose command exists. /// The matrix carries -/// thirteen more rows marked (planned) — classification ahead of code — and +/// additional rows marked (planned) — classification ahead of code — and /// registering one of those would claim a writer that does not exist, which is the half /// of the join that has no way to notice. /// @@ -78,6 +78,8 @@ public void Describe(IAuditCatalogBuilder builder) // to name the same one. .DeclareOffPath( "platform.admin_scope.enter", OperationType.SecurityEvent, OperationClass.Must) + .DeclareOffPath( + "platform.outbox.dead_letter", OperationType.SecurityEvent, OperationClass.Must) .DeclareOffPath( "tenancy.tenant_assertion.reject", OperationType.SecurityEvent, OperationClass.Must) .DeclareOffPath( diff --git a/backend/tests/LearnStack.Tests.Architecture/AuditConventionTests.cs b/backend/tests/LearnStack.Tests.Architecture/AuditConventionTests.cs index 66a31adc..46adcd5a 100644 --- a/backend/tests/LearnStack.Tests.Architecture/AuditConventionTests.cs +++ b/backend/tests/LearnStack.Tests.Architecture/AuditConventionTests.cs @@ -305,7 +305,7 @@ public void Modules_Do_Not_Write_AuditLog_Directly() inserting.Should().Equal( ["LearnStack.Infrastructure.Audit/PostgresAuditStore.cs"], - "PostgresAuditStore's four writes are the only SQL that adds an audit row"); + "PostgresAuditStore's five writes are the only SQL that adds an audit row"); // The context's exemption is for the MAPPING: a change-tracker write placed inside // AuditDbContext would be invisible to the leg above, because its caller names only the @@ -515,13 +515,13 @@ public void The_AppendOnly_Sweep_Can_Actually_Fail() public void IAuditStore_Exposes_No_Update_Method() { // The other half of the same claim, and the one a reader checks first. ADR-0033 - // says four writes and no update; a fifth method whose name says otherwise would - // make the port contradict the table's own triggers. + // and ADR-0046 now authorize five writes and no update; another method would + // make the port contradict its closed write contract. typeof(IAuditStore).GetMethods() .Select(method => method.Name) .Should().BeEquivalentTo( "WritePendingAsync", "WriteStandaloneAsync", "WriteBestEffortAsync", - "WritePlatformScopeAsync"); + "WritePlatformScopeAsync", "WriteEventTerminalAsync"); } /// diff --git a/backend/tests/LearnStack.Tests.Architecture/CrossCuttingFoundationTests.cs b/backend/tests/LearnStack.Tests.Architecture/CrossCuttingFoundationTests.cs index 70b96952..5187b70a 100644 --- a/backend/tests/LearnStack.Tests.Architecture/CrossCuttingFoundationTests.cs +++ b/backend/tests/LearnStack.Tests.Architecture/CrossCuttingFoundationTests.cs @@ -774,6 +774,8 @@ private static bool FollowsTopicConvention(string topic) [Fact] public void Modules_Do_Not_Inject_IEventBus_Directly() { + OutboxDispatchConventionTests.AssertOnlyWorkerPublishes(); + // Standards 20 § IEventBus: the only sanctioned publisher is the // OutboxProcessor. A module that injects IEventBus gets a synchronous // cross-module call with no durability and no transactional atomicity — diff --git a/backend/tests/LearnStack.Tests.Architecture/OutboxDispatchBoundaryTests.cs b/backend/tests/LearnStack.Tests.Architecture/OutboxDispatchBoundaryTests.cs new file mode 100644 index 00000000..1475aa5f --- /dev/null +++ b/backend/tests/LearnStack.Tests.Architecture/OutboxDispatchBoundaryTests.cs @@ -0,0 +1,856 @@ +using FluentAssertions; +using LearnStack.Api.Composition; +using LearnStack.Infrastructure.Messaging; +using LearnStack.Infrastructure.MultiTenancy; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Mono.Cecil; +using Mono.Cecil.Cil; +using Npgsql; +using Xunit; + +namespace LearnStack.Tests.Architecture; + +public sealed partial class OutboxDispatchConventionTests +{ + private const string BoundaryApi = "LearnStack.Api"; + private const string BoundaryComposition = "LearnStack.Api.Composition.OutboxCompositionExtensions"; + private const string BoundaryPersistence = "LearnStack.Api.Composition.PersistenceCompositionExtensions"; + private const string BoundarySource = "LearnStack.Infrastructure.Messaging.OutboxDispatcherDataSource"; + private const string BoundaryWorker = "LearnStack.Api.Messaging.OutboxProcessor"; + private const string BoundaryAttempt = "LearnStack.Api.Messaging.OutboxPublishAttempt"; + private const string BoundaryPlatform = "LearnStack.Infrastructure.MultiTenancy.PlatformAdminScope"; + private const string BoundaryBus = "LearnStack.SharedKernel.Messaging.IEventBus"; + private const string BoundaryTransport = "LearnStack.Infrastructure.Messaging.InProcessEventBus"; + private const string BoundaryDataSource = "Npgsql.NpgsqlDataSource"; + private const string BoundaryLazySource = "System.Lazy`1"; + private const string BoundaryTask = "System.Threading.Tasks.Task"; + private const string BoundaryBooleanTask = "System.Threading.Tasks.Task`1"; + private const string BoundaryCancellation = "System.Threading.CancellationToken"; + private const string BoundaryClaim = "LearnStack.Infrastructure.Messaging.OutboxDispatchClaim"; + private const string BoundaryStrings = "System.Collections.Generic.IReadOnlyCollection`1"; + + /// + /// ADR-0046: census every production assembly, exact keyed factories and parameter + /// bindings. Role/connectivity/grant behavior is proved by the PostgreSQL suites. + /// + [Fact] + public void LearnStack_OutboxAdmin_Role_OnlyUsedBy_OutboxProcessor() + { + using var census = ProductionCensus.Read(); + CredentialOffenders(census).Should().BeEmpty( + "Fix: confine each keyed source to its exact production registration, lazy factory and consumer; " + + "the platform key never authorizes the dispatcher or its helpers"); + } + + internal static void AssertOnlyWorkerPublishes() + { + using var census = ProductionCensus.Read(); + PublicationOffenders(census).Should().BeEmpty( + "Fix: only OutboxProcessor.DispatchNormalAsync constructs the owned publication attempt, " + + "whose private PublishAsync is the sole production IEventBus publisher; producers use IOutbox"); + } + + [Fact] + public void Credential_census_rejects_generic_direct_delegate_async_and_extra_helper_resolution() + { + using var census = ProductionCensus.Read(); + var application = census.Modules.Single(module => module.Assembly.Name.Name == "LearnStack.Application"); + var extensions = typeof(ServiceProviderKeyedServiceExtensions).GetMethods() + .Where(method => method.Name is "GetRequiredKeyedService" or "GetKeyedService" or "GetKeyedServices") + .ToArray(); + var targets = extensions.Concat(typeof(IKeyedServiceProvider).GetMethods()) + .Concat(typeof(ServiceProvider).GetMethods().Where(method => method.Name is "GetRequiredKeyedService" or "GetKeyedService")); + foreach (var target in targets) + foreach (var opcode in new[] { OpCodes.Call, OpCodes.Callvirt, OpCodes.Ldftn, OpCodes.Ldvirtftn }) + AssertMutation(census, "extra keyed resolution through " + target.DeclaringType!.Name + "." + target.Name + "/" + opcode, + () => AddCallProbe(application, "CredentialCaller", application.ImportReference(target), opcode), CredentialOffenders); + + var resolver = application.ImportReference(extensions.First()); + AssertMutation(census, "a cross-assembly async keyed resolver", () => + AddAsyncCallProbe(application, "AsyncCredentialCaller", resolver), CredentialOffenders); + AssertMutation(census, "an unrelated helper on the sanctioned composition type", () => + { + var type = census.Types[(BoundaryApi, BoundaryComposition)]; + var method = AddCall(type, "UnrelatedCredentialHelper", resolver, OpCodes.Call); + return () => type.Methods.Remove(method); + }, CredentialOffenders); + AssertMutation(census, "a generated-name impersonation of the composition root", () => + { + var type = census.Types[(BoundaryApi, BoundaryComposition)]; + var generated = new TypeDefinition(string.Empty, "d__999", TypeAttributes.NestedPrivate, + type.Module.TypeSystem.Object); + type.NestedTypes.Add(generated); + AddCall(generated, "MoveNext", resolver, OpCodes.Call); + return () => type.NestedTypes.Remove(generated); + }, CredentialOffenders); + AssertMutation(census, "AnyKey enumeration even inside a sanctioned root", () => + { + var type = census.Types[(BoundaryApi, BoundaryComposition)]; + var method = AddCall(type, "AnyCredential", type.Module.ImportReference(typeof(KeyedService) + .GetProperty(nameof(KeyedService.AnyKey))!.GetMethod!), OpCodes.Call); + return () => type.Methods.Remove(method); + }, CredentialOffenders); + AssertMutation(census, "an outbox credential indexer in another production assembly", () => + AddStringProbe(application, "CredentialIndexer", "ConnectionStrings:OutboxDispatcher"), CredentialOffenders); + AssertMutation(census, "an outbox environment credential in another production assembly", () => + AddStringProbe(application, "CredentialEnvironment", "ConnectionStrings__OutboxDispatcher"), CredentialOffenders); + foreach (var alias in new[] { "OUTBOXDISPATCHER", "CONNECTIONSTRINGS:OUTBOXDISPATCHER", "CONNECTIONSTRINGS__OUTBOXDISPATCHER" }) + AssertMutation(census, "a case-insensitive configuration alias " + alias, () => + AddStringProbe(application, "UppercaseCredentialAlias", alias), CredentialOffenders); + foreach (var subject in new[] { (BoundaryApi, BoundaryWorker), (InfrastructureAssembly, Store) }) + foreach (var opcode in new[] { OpCodes.Ldfld, OpCodes.Ldflda }) + AssertMutation(census, "an extra privileged-source forwarding helper using " + opcode, () => + { + var type = census.Types[subject]; + var source = type.Fields.Single(field => field.Name == "_source"); + var method = new MethodDefinition("ForwardSource", MethodAttributes.Private, type.Module.TypeSystem.Void); + method.Body.Instructions.Add(Instruction.Create(opcode, source)); + method.Body.Instructions.Add(Instruction.Create(OpCodes.Pop)); + method.Body.Instructions.Add(Instruction.Create(OpCodes.Ret)); + type.Methods.Add(method); + return () => type.Methods.Remove(method); + }, CredentialOffenders); + var store = census.Types[(InfrastructureAssembly, Store)]; + var storeConstructor = store.Methods.Single(method => method.IsConstructor && !method.IsStatic); + AssertMutation(census, "a second consumer constructing the privileged store", () => + AddCallProbe(application, "AdditionalStoreConsumer", storeConstructor, OpCodes.Newobj), CredentialOffenders); + var resolveStore = typeof(ServiceProviderServiceExtensions).GetMethods().Single(method => method.Name == "GetRequiredService" + && method.IsGenericMethodDefinition && method.GetParameters().Length == 1).MakeGenericMethod(typeof(OutboxDispatchStore)); + AssertMutation(census, "an unkeyed locator escaping through the privileged store", () => + AddCallProbe(application, "LocatedStoreConsumer", application.ImportReference(resolveStore), OpCodes.Call), CredentialOffenders); + AssertMutation(census, "an unrelated helper on the approved worker using the privileged store", () => + { + var worker = census.Types[(BoundaryApi, BoundaryWorker)]; + var method = AddCall(worker, "UnrelatedStoreConsumer", store.Methods.Single(method => method.Name == "TryClaimAsync"), OpCodes.Call); + return () => worker.Methods.Remove(method); + }, CredentialOffenders); + var boundaryWorker = census.Types[(BoundaryApi, BoundaryWorker)]; + foreach (var name in new[] { "ExecuteAsync", "DispatchAsync", "DispatchNormalAsync", "RefreshAsync" }) + AssertMutation(census, "a public store-forwarding overload borrowing the approved worker name " + name, () => + AddBoundaryFieldGetter(boundaryWorker, name, "_store"), CredentialOffenders); + AssertMutation(census, "a public raw-source overload borrowing the approved worker name StartAsync", () => + AddBoundaryFieldGetter(boundaryWorker, "StartAsync", "_source"), CredentialOffenders); + foreach (var name in new[] { "AcquireAsync", "UpdateAsync", "TryTerminalizeAsync", "IsDurablyTerminalAsync", "ReadSnapshotAsync" }) + AssertMutation(census, "a public raw-source overload borrowing the approved store name " + name, () => + AddBoundaryFieldGetter(store, name, "_source"), CredentialOffenders); + foreach (var method in new[] { boundaryWorker.Methods.Single(item => item.Name == "StartAsync"), + store.Methods.Single(item => item.Name == "ReadSnapshotAsync") }) + { + AssertMutation(census, "a changed source-reader visibility", () => + ChangeAttributes(() => method.Attributes, value => method.Attributes = value, + (method.Attributes & ~MethodAttributes.MemberAccessMask) | MethodAttributes.Private), CredentialOffenders); + AssertMutation(census, "a static source reader replacing its approved instance binding", () => + ChangeAttributes(() => method.Attributes, value => method.Attributes = value, + method.Attributes | MethodAttributes.Static), CredentialOffenders); + AssertMutation(census, "a changed source-reader return type", () => + { + var original = method.ReturnType; + method.ReturnType = method.Module.TypeSystem.Object; + return () => method.ReturnType = original; + }, CredentialOffenders); + AssertMutation(census, "a changed source-reader parameter type", () => + { + var parameter = method.Parameters[0]; + var original = parameter.ParameterType; + parameter.ParameterType = method.Module.TypeSystem.Int32; + return () => parameter.ParameterType = original; + }, CredentialOffenders); + } + var dispatch = boundaryWorker.Methods.Single(method => method.Name == "DispatchAsync"); + AssertMutation(census, "a public store-consumer method replacing its private binding", () => + ChangeAttributes(() => dispatch.Attributes, value => dispatch.Attributes = value, + (dispatch.Attributes & ~MethodAttributes.MemberAccessMask) | MethodAttributes.Public), CredentialOffenders); + } + + [Fact] + public void Credential_census_rejects_crossed_keys_changed_closures_and_missing_subjects() + { + using var census = ProductionCensus.Read(); + foreach (var subject in new[] { (BoundaryApi, BoundaryWorker), (InfrastructureAssembly, BoundaryPlatform) }) + { + var parameter = census.Types[subject].Methods.Single(method => method.IsConstructor && !method.IsStatic) + .Parameters.Single(parameter => parameter.CustomAttributes.Any(attribute => + attribute.AttributeType.Name == "FromKeyedServicesAttribute")); + var attribute = parameter.CustomAttributes.Single(item => item.AttributeType.Name == "FromKeyedServicesAttribute"); + AssertMutation(census, "a consumer resolving the other privileged key", () => + { + var original = attribute.ConstructorArguments[0]; + attribute.ConstructorArguments[0] = new CustomAttributeArgument(original.Type, + subject.Item2 == BoundaryWorker ? PlatformAdminScope.PlatformDataSourceKey : OutboxDispatcherDataSource.Key); + return () => attribute.ConstructorArguments[0] = original; + }, CredentialOffenders); + AssertMutation(census, "a raw source replacing the required lazy binding", () => + { + var original = parameter.ParameterType; + parameter.ParameterType = census.Types[subject].Module.ImportReference(typeof(NpgsqlDataSource)); + return () => parameter.ParameterType = original; + }, CredentialOffenders); + } + + foreach (var key in new[] { OutboxDispatcherDataSource.Key, PlatformAdminScope.PlatformDataSourceKey }) + { + var descriptor = BoundaryRegistrations().Single(item => item.IsKeyedService && Equals(item.ServiceKey, key) + && item.ServiceType == typeof(Lazy)); + var factory = FindRuntimeMethod(census, descriptor.KeyedImplementationFactory!.Method)!; + var leafReference = DelegateTargets(factory).Single(); + var leaf = census.Types[Key(leafReference.DeclaringType)].Methods.Single(method => MethodId(method) == MethodId(leafReference)); + var call = leaf.Body.Instructions.Single(instruction => instruction.Operand is MethodReference method && IsKeyedResolution(method)); + AssertMutation(census, "a lazy callback substituting the other key", () => + ReplaceInstruction(call.Previous!, OpCodes.Ldstr, key == OutboxDispatcherDataSource.Key + ? PlatformAdminScope.PlatformDataSourceKey : OutboxDispatcherDataSource.Key), CredentialOffenders); + var captured = (FieldReference)call.Previous!.Operand; + var assignment = factory.Body.Instructions.Single(instruction => instruction.OpCode == OpCodes.Stfld + && instruction.Operand is FieldReference field && field.FullName == captured.FullName); + AssertMutation(census, "a factory capturing a different key instead of its registered key", () => + ReplaceInstruction(assignment.Previous!, OpCodes.Ldstr, "unrelated-key"), CredentialOffenders); + var provider = (FieldReference)call.Previous.Previous!.Previous!.Operand; + foreach (var field in new[] { captured, provider }) + { + AssertMutation(census, "the real lazy callback overwriting captured " + field.Name, () => + InsertCaptureMutation(leaf, field, address: false), CredentialOffenders); + AssertMutation(census, "the actual factory overwriting captured " + field.Name + " after its original assignment", () => + InsertCaptureMutation(factory, field, address: false, afterCapture: true), CredentialOffenders); + AssertMutation(census, "a closure helper overwriting captured " + field.Name, () => + { + var helper = new MethodDefinition("RewriteCapturedField", MethodAttributes.Private, leaf.Module.TypeSystem.Void); + helper.Body.Instructions.Add(Instruction.Create(OpCodes.Ret)); + leaf.DeclaringType.Methods.Add(helper); + var undo = InsertCaptureMutation(helper, field, address: false); + return () => { undo(); leaf.DeclaringType.Methods.Remove(helper); }; + }, CredentialOffenders); + AssertMutation(census, "the real lazy callback exposing the address of captured " + field.Name, () => + InsertCaptureMutation(leaf, field, address: true), CredentialOffenders); + var parameter = factory.Parameters[field == captured ? 1 : 0]; + foreach (var opcode in new[] { OpCodes.Starg, OpCodes.Starg_S, OpCodes.Ldarga, OpCodes.Ldarga_S }) + AssertMutation(census, "the actual factory mutating or exposing its " + parameter.Name + " argument through " + opcode, () => + { + var address = opcode == OpCodes.Ldarga || opcode == OpCodes.Ldarga_S; + var replacement = parameter.ParameterType.FullName == "System.Object" + ? Instruction.Create(OpCodes.Ldstr, PlatformAdminScope.PlatformDataSourceKey) : Instruction.Create(OpCodes.Ldnull); + var inserted = address + ? new[] { Instruction.Create(opcode, parameter), Instruction.Create(OpCodes.Pop) } + : [replacement, Instruction.Create(opcode, parameter)]; + for (var index = 0; index < inserted.Length; index++) factory.Body.Instructions.Insert(index, inserted[index]); + return () => { foreach (var instruction in inserted) factory.Body.Instructions.Remove(instruction); }; + }, CredentialOffenders); + } + } + + var root = census.Types[(BoundaryApi, BoundaryComposition)].Methods.Single(method => method.Name == "AddLearnStackOutboxDispatcher"); + var registrationKey = root.Body.Instructions.First(instruction => Equals(instruction.Operand, OutboxDispatcherDataSource.Key)); + AssertMutation(census, "a changed actual registration key", () => + ReplaceInstruction(registrationKey, OpCodes.Ldstr, PlatformAdminScope.PlatformDataSourceKey), CredentialOffenders); + var configurationKey = root.Body.Instructions.Single(instruction => Equals(instruction.Operand, "OutboxDispatcher")); + AssertMutation(census, "the dispatcher reading the platform configuration", () => + ReplaceInstruction(configurationKey, OpCodes.Ldstr, "PlatformAdmin"), CredentialOffenders); + var source = census.Types[(InfrastructureAssembly, BoundarySource)]; + AssertMutation(census, "a missing production data-source subject", () => + { + var module = source.Module; + module.Types.Remove(source); + return () => module.Types.Add(source); + }, CredentialOffenders); + } + + [Fact] + public void Credential_census_rejects_compiled_connection_string_indexers_and_composed_keys() + { + using var census = ProductionCensus.Read(); + var application = census.Modules.Single(module => module.Assembly.Name.Name == "LearnStack.Application"); + foreach (var name in new[] { "Concatenated", "ReadonlyName", "Interpolated", "Formatted", "Manager", "Root", "Section" }) + AssertMutation(census, "a compiled configuration credential indexer: " + name, () => + AddCompiledCredentialProbe(application, name), CredentialOffenders); + foreach (var name in new[] { "UnrelatedLiteral", "UnrelatedDynamic", "Dictionary", "CredentialDiagnostic" }) + { + var undo = AddCompiledCredentialProbe(application, name); + try { CredentialOffenders(census).Should().BeEmpty("Fix: unrelated configuration, dictionary indexers and diagnostics remain valid"); } + finally { undo(); } + } + } + + private static Action AddCompiledCredentialProbe(ModuleDefinition production, string name) + { + // Use the compiler's actual concat/interpolation/indexer IL in the same + // nonempty production census, rather than planting the missing full key. + var probes = ModuleDefinition.ReadModule(typeof(OutboxDispatchConventionTests).Assembly.Location, + new ReaderParameters { ReadingMode = ReadingMode.Immediate }); + var source = probes.GetTypes().Single(type => type.Name == nameof(CredentialIndexerProbes)); + var method = source.Methods.Single(item => item.Name == name); + // Decode IL while its tokens still belong to the source module. Moving a + // deferred body first would read those tokens through the production module. + _ = method.Body.Instructions.Count; + source.Methods.Remove(method); + var planted = AddProbe(production, "CompiledCredentialIndexer"); + planted.Methods.Add(method); + return () => { production.Types.Remove(planted); probes.Dispose(); }; + } + + private static class CredentialIndexerProbes + { + private static readonly string ConnectionName = "OutboxDispatcher"; + public static string? Concatenated(IConfiguration configuration, string name) => configuration["ConnectionStrings:" + name]; + public static string? ReadonlyName(IConfiguration configuration) => configuration["ConnectionStrings:" + ConnectionName]; + public static string? Interpolated(IConfiguration configuration, string name) => configuration[$"ConnectionStrings:{name}"]; + public static string? Formatted(IConfiguration configuration, string name) => configuration[string.Format(System.Globalization.CultureInfo.InvariantCulture, "ConnectionStrings:{0}", name)]; + public static string? Manager(ConfigurationManager configuration, string name) => configuration["ConnectionStrings:" + name]; + public static string? Root(ConfigurationRoot configuration, string name) => configuration["ConnectionStrings:" + name]; + public static string? Section(ConfigurationSection configuration, string name) => configuration["ConnectionStrings:" + name]; + public static string? UnrelatedLiteral(IConfiguration configuration) => configuration["Telemetry:OtlpEndpoint"]; + public static string? UnrelatedDynamic(IConfiguration configuration, string key) => configuration[key]; + public static string? Dictionary(IDictionary values, string name) => values["ConnectionStrings:" + name]; + public static string CredentialDiagnostic(string name) => $"ConnectionStrings:{name} is not configured."; + } + + [Fact] + public void Credential_registration_detector_rejects_missing_duplicate_wrong_key_and_non_singleton_sources() + { + IServiceCollection services = BoundaryRegistrations(); + RegistrationOffenders(services).Should().BeEmpty("Fix: the real production composition must be the clean control"); + var descriptor = services.Single(item => item.IsKeyedService && Equals(item.ServiceKey, OutboxDispatcherDataSource.Key) + && item.ServiceType == typeof(NpgsqlDataSource)); + services.Remove(descriptor); + RegistrationOffenders(services).Should().NotBeEmpty("Fix: a missing privileged source must fail"); + services.Add(descriptor); + services.Add(descriptor); + RegistrationOffenders(services).Should().NotBeEmpty("Fix: a duplicate privileged source must fail"); + services.Remove(descriptor); + services.Remove(descriptor); + var wrongKey = new ServiceCollection(); + wrongKey.AddKeyedSingleton("unrelated-key", (_, _) => throw new InvalidOperationException("Never resolved")); + services.Add(wrongKey.Single()); + RegistrationOffenders(services).Should().NotBeEmpty("Fix: a renamed privileged source must fail"); + services.Remove(wrongKey.Single()); + var scoped = new ServiceCollection(); + scoped.AddKeyedScoped(OutboxDispatcherDataSource.Key, (_, _) => throw new InvalidOperationException("Never resolved")); + services.Add(scoped.Single()); + RegistrationOffenders(services).Should().NotBeEmpty("Fix: a non-singleton privileged source must fail"); + services.Remove(scoped.Single()); + services.Add(descriptor); + RegistrationOffenders(services).Should().BeEmpty("Fix: restoring the actual descriptors must restore the clean control"); + } + + [Fact] + public void Publication_census_rejects_interface_concrete_delegate_async_and_unowned_attempt_paths() + { + using var census = ProductionCensus.Read(); + var port = census.Types[(KernelAssembly, BoundaryBus)].Methods.Single(method => method.Name == "PublishAsync"); + var concrete = census.Types[(InfrastructureAssembly, BoundaryTransport)].Methods.Single(method => method.Name == "PublishAsync"); + foreach (var assembly in new[] { BoundaryApi, "LearnStack.Application", "LearnStack.Tools.Seeder", "LearnStack.Modules.Tenancy.Application" }) + { + var module = census.Modules.Single(item => item.Assembly.Name.Name == assembly); + foreach (var target in new[] { port, concrete }) + foreach (var opcode in new[] { OpCodes.Call, OpCodes.Callvirt, OpCodes.Ldftn, OpCodes.Ldvirtftn }) + AssertMutation(census, assembly + " publication through " + target.DeclaringType.Name + "/" + opcode, + () => AddCallProbe(module, "ExtraPublisher", target, opcode), PublicationOffenders); + AssertMutation(census, assembly + " async publisher", () => AddAsyncCallProbe(module, "AsyncPublisher", port), PublicationOffenders); + } + var attempt = census.Types[(BoundaryApi, BoundaryAttempt)]; + var worker = census.Types[(BoundaryApi, BoundaryWorker)]; + var constructor = attempt.Methods.Single(method => method.IsConstructor && !method.IsStatic); + foreach (var target in new[] { constructor, attempt.Methods.Single(method => method.Name == "PublishAsync"), port }) + AssertMutation(census, "an unrelated method on the permitted worker/attempt type", () => + { + var owner = target == port ? attempt : worker; + var method = AddCall(owner, "UnrelatedPublisher", target, target == constructor ? OpCodes.Newobj : OpCodes.Call); + return () => owner.Methods.Remove(method); + }, PublicationOffenders); + AssertMutation(census, "a generated name impersonating the real publication owner", () => + { + var generated = new TypeDefinition(string.Empty, "d__999", TypeAttributes.NestedPrivate, attempt.Module.TypeSystem.Object); + attempt.NestedTypes.Add(generated); + AddCall(generated, "MoveNext", port, OpCodes.Callvirt); + return () => attempt.NestedTypes.Remove(generated); + }, PublicationOffenders); + var application = census.Modules.Single(module => module.Assembly.Name.Name == "LearnStack.Application"); + AssertMutation(census, "a second inherited event-bus implementation", () => + { + var type = AddProbe(application, "InheritedEventBus"); + type.BaseType = application.ImportReference(concrete.DeclaringType); + return () => application.Types.Remove(type); + }, PublicationOffenders); + AssertMutation(census, "an extra publish call inside the sanctioned method", () => + { + var owner = census.Types.Values.SelectMany(type => type.Methods).Single(method => method.HasBody + && method.Body.Instructions.Any(instruction => instruction.Operand is MethodReference target + && target.Name == "PublishAsync" && target.DeclaringType.FullName == BoundaryBus)); + var instruction = Instruction.Create(OpCodes.Callvirt, owner.Module.ImportReference(port)); + owner.Body.Instructions.Add(instruction); + return () => owner.Body.Instructions.Remove(instruction); + }, PublicationOffenders); + } + + [Fact] + public void Boundary_censuses_preserve_unkeyed_services_unrelated_publishers_and_both_real_keys() + { + using var census = ProductionCensus.Read(); + var module = census.Modules.Single(item => item.Assembly.Name.Name == "LearnStack.Application"); + var type = AddProbe(module, "CleanBoundaryHelper"); + var ordinaryResolver = typeof(ServiceProviderServiceExtensions).GetMethods().Single(method => + method.Name == "GetRequiredService" && method.IsGenericMethodDefinition && method.GetParameters().Length == 1); + AddCall(type, "ResolveOrdinaryService", module.ImportReference(ordinaryResolver), OpCodes.Call); + var unrelated = new MethodDefinition("PublishAsync", MethodAttributes.Public | MethodAttributes.Static, module.TypeSystem.Void); + unrelated.Body.Instructions.Add(Instruction.Create(OpCodes.Ret)); + type.Methods.Add(unrelated); + AddCall(type, "CallUnrelatedPublisher", unrelated, OpCodes.Call); + try + { + CredentialOffenders(census).Should().BeEmpty("Fix: both distinct production keys and ordinary unkeyed resolution remain valid"); + PublicationOffenders(census).Should().BeEmpty("Fix: unrelated methods named PublishAsync are not event-bus publication"); + } + finally { module.Types.Remove(type); } + } + + private static Action ReplaceInstruction(Instruction instruction, OpCode opcode, object operand) + { + var originalCode = instruction.OpCode; + var originalOperand = instruction.Operand; + instruction.OpCode = opcode; + instruction.Operand = operand; + return () => { instruction.OpCode = originalCode; instruction.Operand = originalOperand; }; + } + + private static Action InsertCaptureMutation(MethodDefinition method, FieldReference field, bool address, bool afterCapture = false) + { + var instructions = method.Body.Instructions; + var index = afterCapture ? instructions.IndexOf(instructions.Single(instruction => instruction.OpCode == OpCodes.Stfld + && instruction.Operand is FieldReference assigned && assigned.FullName == field.FullName)) + 1 : 0; + var replacement = field.FieldType.FullName == "System.Object" + ? Instruction.Create(OpCodes.Ldstr, PlatformAdminScope.PlatformDataSourceKey) : Instruction.Create(OpCodes.Ldnull); + // The factory still has the freshly captured closure on its stack; an + // instance callback/helper uses its own closure. Neither removes the real + // capture or changes the recognized resolver tail. + var inserted = new[] + { + Instruction.Create(afterCapture ? OpCodes.Dup : OpCodes.Ldarg_0), + address ? Instruction.Create(OpCodes.Ldflda, field) : replacement, + address ? Instruction.Create(OpCodes.Pop) : Instruction.Create(OpCodes.Stfld, field), + }; + for (var offset = 0; offset < inserted.Length; offset++) instructions.Insert(index + offset, inserted[offset]); + return () => { foreach (var instruction in inserted) instructions.Remove(instruction); }; + } + + private static Action AddStringProbe(ModuleDefinition module, string name, string value) + { + var type = AddProbe(module, name); + var method = new MethodDefinition("Run", MethodAttributes.Public | MethodAttributes.Static, module.TypeSystem.Void); + method.Body.Instructions.Add(Instruction.Create(OpCodes.Ldstr, value)); + method.Body.Instructions.Add(Instruction.Create(OpCodes.Pop)); + method.Body.Instructions.Add(Instruction.Create(OpCodes.Ret)); + type.Methods.Add(method); + return () => module.Types.Remove(type); + } + + private static Action AddBoundaryFieldGetter(TypeDefinition type, string name, string fieldName) + { + var field = type.Fields.Single(item => item.Name == fieldName); + var method = new MethodDefinition(name, MethodAttributes.Public, field.FieldType); + method.Body.Instructions.Add(Instruction.Create(OpCodes.Ldarg_0)); + method.Body.Instructions.Add(Instruction.Create(OpCodes.Ldfld, field)); + method.Body.Instructions.Add(Instruction.Create(OpCodes.Ret)); + type.Methods.Add(method); + return () => type.Methods.Remove(method); + } + + private static ServiceCollection BoundaryRegistrations() + { + var services = new ServiceCollection(); + // Real composition, no provider build, pool construction or connection. + services.AddLearnStackPersistence(new ConfigurationBuilder().Build()); + return services; + } + + private static SortedSet RegistrationOffenders(IEnumerable services) + { + var offenders = new SortedSet(StringComparer.Ordinal); + var expected = new[] + { + PlatformAdminScope.PlatformDataSourceKey + ":" + typeof(NpgsqlDataSource), + PlatformAdminScope.PlatformDataSourceKey + ":" + typeof(Lazy), + OutboxDispatcherDataSource.Key + ":" + typeof(NpgsqlDataSource), + OutboxDispatcherDataSource.Key + ":" + typeof(Lazy), + }; + var keyed = services.Where(descriptor => descriptor.IsKeyedService).ToArray(); + CheckSet(offenders, "keyed production registrations", keyed.Select(descriptor => + descriptor.ServiceKey + ":" + descriptor.ServiceType), expected); + foreach (var descriptor in keyed) + if (descriptor.Lifetime != ServiceLifetime.Singleton || descriptor.KeyedImplementationFactory is null) + offenders.Add("Keyed source must be a container-owned singleton factory: " + descriptor.ServiceType); + return offenders; + } + + private static SortedSet CredentialOffenders(ProductionCensus census) + { + var services = BoundaryRegistrations(); + var offenders = RegistrationOffenders(services); + var types = census.Types; + CheckSet(offenders, "production assemblies", census.Modules.Select(module => module.Assembly.Name.Name), ProductionAssemblies.Names()); + foreach (var subject in new[] { (BoundaryApi, BoundaryComposition), (BoundaryApi, BoundaryPersistence), + (BoundaryApi, BoundaryWorker), (InfrastructureAssembly, BoundaryPlatform), + (InfrastructureAssembly, BoundarySource), (InfrastructureAssembly, Store) }) + if (!types.ContainsKey(subject)) offenders.Add("Missing credential subject: " + subject.Item2); + if (offenders.Count != 0) return offenders; + + var outboxRoot = types[(BoundaryApi, BoundaryComposition)].Methods.Single(method => method.Name == "AddLearnStackOutboxDispatcher"); + var platformRoot = types[(BoundaryApi, BoundaryPersistence)].Methods.Single(method => method.Name == "AddLearnStackPersistence"); + var roots = new[] { (Root: outboxRoot, Key: OutboxDispatcherDataSource.Key), + (Root: platformRoot, Key: PlatformAdminScope.PlatformDataSourceKey) }; + var expectedResolvers = new List(); + var expectedFactories = new List(); + MethodDefinition? outboxBuildFactory = null; + + foreach (var (root, key) in roots) + { + var descriptors = services.Where(descriptor => descriptor.IsKeyedService && Equals(descriptor.ServiceKey, key)).ToArray(); + foreach (var descriptor in descriptors) + { + var factory = FindRuntimeMethod(census, descriptor.KeyedImplementationFactory!.Method); + if (factory is null) { offenders.Add("Missing keyed factory: " + key); continue; } + expectedFactories.Add(MethodId(root)); + if (!DelegateTargets(root).Any(target => MethodId(target) == MethodId(factory))) + offenders.Add(MethodId(root) + ": must directly register its actual factory " + MethodId(factory)); + if (descriptor.ServiceType == typeof(NpgsqlDataSource)) + { + if (key == OutboxDispatcherDataSource.Key) outboxBuildFactory = factory; + continue; + } + + var leaves = DelegateTargets(factory).ToArray(); + if (leaves.Length != 1 || !types.TryGetValue(Key(leaves[0].DeclaringType), out var leafType)) + { offenders.Add(MethodId(factory) + ": requires one actual lazy delegate"); continue; } + var leaf = leafType.Methods.SingleOrDefault(method => MethodId(method) == MethodId(leaves[0])); + if (leaf is null) { offenders.Add("Missing lazy delegate: " + leaves[0].FullName); continue; } + expectedResolvers.Add(MethodId(leaf)); + CheckCapturedKeyResolver(offenders, census, factory, leaf); + } + + // Pin registration keys independently from callback keys. A changed key in + // the in-memory IL cannot be hidden by the unmodified reflection descriptors. + CheckSet(offenders, MethodId(root) + " registration keys", root.Body.Instructions + .Where(instruction => instruction.Operand is string value + && (value == OutboxDispatcherDataSource.Key || value == PlatformAdminScope.PlatformDataSourceKey) + && instruction.Next?.Operand is not MethodReference { Name: "GetConnectionString" }) + .Select(instruction => (string)instruction.Operand), [key, key]); + var keyCalls = MethodOperands(root).Where(IsKeyedRegistration).ToArray(); + CheckSet(offenders, MethodId(root) + " registration service types", keyCalls.Select(GenericService), + [BoundaryDataSource, BoundaryLazySource]); + } + + var storeDescriptor = services.Single(descriptor => !descriptor.IsKeyedService && descriptor.ServiceType == typeof(OutboxDispatchStore)); + var storeFactory = FindRuntimeMethod(census, storeDescriptor.ImplementationFactory!.Method); + if (storeFactory is null) offenders.Add("Missing actual dispatcher-store factory"); + else + { + expectedResolvers.Add(MethodId(storeFactory)); + if (!DelegateTargets(outboxRoot).Any(target => MethodId(target) == MethodId(storeFactory))) + offenders.Add("Dispatcher store factory must be registered by the outbox root"); + CheckLiteralKeyResolver(offenders, storeFactory, BoundaryLazySource, OutboxDispatcherDataSource.Key); + } + + var sites = CallSites(census).ToArray(); + CheckSet(offenders, "keyed resolver callers", sites.Where(site => IsKeyedResolution(site.Target)).Select(site => site.Owner), expectedResolvers); + CheckSet(offenders, "keyed registration callers", sites.Where(site => IsKeyedRegistration(site.Target)).Select(site => site.Owner), expectedFactories); + CheckSet(offenders, "outbox pool construction", sites.Where(site => site.Target.DeclaringType.FullName == BoundarySource + && site.Target.Name == "Build").Select(site => site.Owner), outboxBuildFactory is null ? [] : [MethodId(outboxBuildFactory)]); + CheckSet(offenders, "outbox configuration validation", sites.Where(site => site.Target.DeclaringType.FullName == BoundarySource + && site.Target.Name == "Validate").Select(site => site.Owner), [MethodId(outboxRoot)]); + CheckSet(offenders, "connection-string reader census", sites.Where(site => IsConnectionStringRead(site.Target)) + .Select(site => site.Owner), [MethodId(outboxRoot), MethodId(platformRoot), MethodId(platformRoot)]); + + var injections = new List(); + foreach (var method in types.Values.SelectMany(type => type.Methods)) + { + foreach (var parameter in method.Parameters) + foreach (var attribute in parameter.CustomAttributes.Where(attribute => + attribute.AttributeType.FullName == "Microsoft.Extensions.DependencyInjection.FromKeyedServicesAttribute")) + injections.Add(MethodId(method) + ":" + parameter.ParameterType.FullName + ":" + + BoundaryAttributeValue(attribute.ConstructorArguments.FirstOrDefault())); + if (!method.HasBody) continue; + // GetConnectionString is the sole admitted credential-read member at + // the two roots above. Its indexer equivalent can compose the name, + // leaving no complete dispatcher literal in IL. Conservatively refuse + // a connection-string prefix alongside an explicit configuration indexer; + // this does not classify arbitrary key dataflow or unrelated indexers. + if (method.Body.Instructions.Any(instruction => instruction.Operand is MethodReference call && IsConfigurationIndexer(call)) + && method.Body.Instructions.Any(instruction => instruction.OpCode == OpCodes.Ldstr && instruction.Operand is string value + && value.StartsWith("ConnectionStrings:", StringComparison.OrdinalIgnoreCase))) + offenders.Add(MethodId(method) + ": connection-string indexer bypasses the declared credential readers"); + foreach (var instruction in method.Body.Instructions) + { + if (instruction.Operand is MethodReference call && IsConnectionStringRead(call)) + { + var key = instruction.Previous?.Operand as string; + var expected = method == outboxRoot ? new[] { "OutboxDispatcher" } + : method == platformRoot ? ["Default", "PlatformAdmin"] : []; + if (key is null || !expected.Contains(key, StringComparer.Ordinal)) + offenders.Add(MethodId(method) + ": unauthorized connection-string read"); + } + if (instruction.Operand is FieldReference field && field.DeclaringType.FullName == "Microsoft.Extensions.DependencyInjection.KeyedService" + && field.Name == "AnyKey") offenders.Add(MethodId(method) + ": AnyKey reopens both credentials"); + if (instruction.Operand is MethodReference anyKey && anyKey.DeclaringType.FullName == "Microsoft.Extensions.DependencyInjection.KeyedService" + && anyKey.Name == "get_AnyKey") offenders.Add(MethodId(method) + ": AnyKey reopens both credentials"); + if (instruction.Operand is string literal && (literal.Equals("OutboxDispatcher", StringComparison.OrdinalIgnoreCase) + || literal.Equals("ConnectionStrings:OutboxDispatcher", StringComparison.OrdinalIgnoreCase) + || literal.Equals("ConnectionStrings__OutboxDispatcher", StringComparison.OrdinalIgnoreCase)) && method != outboxRoot) + offenders.Add(MethodId(method) + ": dispatcher credential spelling outside its reader"); + } + } + var worker = types[(BoundaryApi, BoundaryWorker)]; + var workerConstructor = BoundaryWorkerConstructor(offenders, worker); + var scopeConstructor = types[(InfrastructureAssembly, BoundaryPlatform)].Methods.Single(method => method.IsConstructor && !method.IsStatic); + CheckSet(offenders, "keyed parameter bindings", injections, + [workerConstructor + ":" + BoundaryLazySource + ":" + OutboxDispatcherDataSource.Key, + MethodId(scopeConstructor) + ":" + BoundaryLazySource + ":" + PlatformAdminScope.PlatformDataSourceKey]); + CheckSourceFieldAccess(offenders, census, worker, workerConstructor, + [RequireBoundaryMethod(offenders, worker, "StartAsync", BoundaryTask, MethodAttributes.Public, BoundaryCancellation)]); + var store = types[(InfrastructureAssembly, Store)]; + var storeConstructor = RequireBoundaryMethod(offenders, store, ".ctor", "System.Void", MethodAttributes.Public, + BoundaryLazySource, "LearnStack.Infrastructure.Messaging.OutboxEventRegistry", + "LearnStack.Infrastructure.Messaging.OutboxDispatchOptions", "LearnStack.SharedKernel.Identifiers.IGuidFactory", + "LearnStack.SharedKernel.Time.IClock", "Microsoft.Extensions.DependencyInjection.IServiceScopeFactory"); + CheckSourceFieldAccess(offenders, census, store, storeConstructor, + [ + RequireBoundaryMethod(offenders, store, "AcquireAsync", "System.Threading.Tasks.Task`1<" + BoundaryClaim + ">", MethodAttributes.Private, + "LearnStack.Infrastructure.Messaging.OutboxClaimKind", "System.String", BoundaryStrings, BoundaryCancellation), + RequireBoundaryMethod(offenders, store, "UpdateAsync", BoundaryBooleanTask, MethodAttributes.Private, + BoundaryClaim, "System.String", "System.String", "System.String", BoundaryCancellation, "System.String"), + RequireBoundaryMethod(offenders, store, "TryTerminalizeAsync", BoundaryBooleanTask, MethodAttributes.Public, + BoundaryClaim, "System.String", BoundaryCancellation), + RequireBoundaryMethod(offenders, store, "IsDurablyTerminalAsync", BoundaryBooleanTask, MethodAttributes.Private, "System.Guid"), + RequireBoundaryMethod(offenders, store, "ReadSnapshotAsync", "System.Threading.Tasks.Task`1", + MethodAttributes.Public, BoundaryStrings, BoundaryCancellation), + ]); + if (storeFactory is not null) CheckStoreConsumers(offenders, census, outboxRoot, storeFactory); + return offenders; + } + + private static void CheckStoreConsumers(SortedSet offenders, ProductionCensus census, + MethodDefinition composition, MethodDefinition factory) + { + // A caller can obtain the same bypass through the unkeyed store facade. + // Confine typed consumers, including generic locator calls and wrappers; + // reflection and arbitrary runtime dataflow are not claimed by this census. + var owners = OwnedBodies(census).Where(item => item.Owner.DeclaringType.FullName != Store + && ReferencesStore(item.Body)).Select(item => MethodId(item.Owner)) + .Concat(census.Types.Values.SelectMany(type => type.Methods).Where(method => !method.HasBody + && method.DeclaringType.FullName != Store && ReferencesStore(method)).Select(MethodId)) + .Distinct(StringComparer.Ordinal); + var worker = census.Types[(BoundaryApi, BoundaryWorker)]; + var workerMethods = new[] + { + BoundaryWorkerConstructor(offenders, worker), + RequireBoundaryMethod(offenders, worker, "ExecuteAsync", BoundaryTask, MethodAttributes.Family, BoundaryCancellation), + RequireBoundaryMethod(offenders, worker, "DispatchAsync", BoundaryTask, MethodAttributes.Private, BoundaryClaim, BoundaryCancellation), + RequireBoundaryMethod(offenders, worker, "DispatchNormalAsync", BoundaryTask, MethodAttributes.Private, BoundaryClaim, BoundaryCancellation), + RequireBoundaryMethod(offenders, worker, "RefreshAsync", BoundaryTask, MethodAttributes.Private, "System.Boolean", BoundaryCancellation), + }; + CheckSet(offenders, "privileged store consumers", owners, + workerMethods.Concat([MethodId(composition), MethodId(factory)])); + // Only an attribute-linked Store state machine may carry its own captured + // instance. A similarly named nested type gains no authority from its name. + var stateMachines = OwnedBodies(census).Where(item => item.Body.Name == "MoveNext" && item.Body != item.Owner + && item.Owner.DeclaringType.FullName == Store).Select(item => Key(item.Body.DeclaringType)).ToHashSet(); + // Direct instance holders carry the capability. Generic wrappers (including + // the compiler's cached registration delegate) remain covered at every typed + // method/field-use site above, rather than receiving a generated-name exception. + var fields = census.Types.Values.SelectMany(type => type.Fields).Where(field => field.FieldType.FullName == Store + && !(stateMachines.Contains(Key(field.DeclaringType)) && field.Name == "<>4__this" + && field.FieldType.FullName == Store)).ToArray(); + CheckSet(offenders, "privileged store fields", fields.Select(field => TypeId(field.DeclaringType) + ":" + field.Name), + [TypeId(worker) + ":_store"]); + foreach (var field in fields) + if (!field.IsPrivate || !field.IsInitOnly) offenders.Add(field.FullName + ": store capability must remain private and readonly"); + } + + private static bool ReferencesStore(MethodDefinition method) + { + if (ContainsStore(method.ReturnType) || method.Parameters.Any(parameter => ContainsStore(parameter.ParameterType))) return true; + if (!method.HasBody) return false; + if (method.Body.Variables.Any(variable => ContainsStore(variable.VariableType))) return true; + return method.Body.Instructions.Any(instruction => instruction.Operand switch + { + GenericInstanceMethod generic => generic.GenericArguments.Any(ContainsStore) || ContainsStore(generic.DeclaringType) + || ContainsStore(generic.ReturnType) || generic.Parameters.Any(parameter => ContainsStore(parameter.ParameterType)), + MethodReference call => ContainsStore(call.DeclaringType) || ContainsStore(call.ReturnType) + || call.Parameters.Any(parameter => ContainsStore(parameter.ParameterType)), + FieldReference field => ContainsStore(field.FieldType), + TypeReference type => ContainsStore(type), + _ => false, + }); + } + + private static bool ContainsStore(TypeReference type) => type.FullName == Store + || (type is GenericInstanceType generic && generic.GenericArguments.Any(ContainsStore)) + || (type is TypeSpecification specification && ContainsStore(specification.ElementType)); + + private static void CheckSourceFieldAccess(SortedSet offenders, ProductionCensus census, + TypeDefinition type, string constructor, string[] readers) + { + var fields = type.Fields.Where(field => field.Name == "_source").ToArray(); + if (fields.Length != 1 || !fields[0].IsPrivate || !fields[0].IsInitOnly || fields[0].IsStatic + || fields[0].FieldType.FullName != BoundaryLazySource) + { offenders.Add(TypeId(type) + ": requires its private readonly lazy source"); return; } + // Close forwarding of an already injected capability as well as fresh keyed + // resolution. Exact access counts also catch another use in an allowed method. + var actual = OwnedBodies(census).SelectMany(item => item.Body.Body.Instructions + .Where(instruction => instruction.Operand is FieldReference field && SameType(field.DeclaringType, type) + && field.Name == fields[0].Name) + .Select(instruction => MethodId(item.Owner) + ":" + instruction.OpCode.Name)); + var expected = new[] { constructor + ":stfld" }.Concat(readers.Select(reader => reader + ":ldfld")); + CheckSet(offenders, TypeId(type) + " privileged source field access", actual, expected); + } + + private static string BoundaryWorkerConstructor(SortedSet offenders, TypeDefinition worker) => + RequireBoundaryMethod(offenders, worker, ".ctor", "System.Void", MethodAttributes.Public, + BoundaryLazySource, Store, "LearnStack.Infrastructure.Messaging.OutboxEnvelopeReader", + "LearnStack.Infrastructure.Messaging.OutboxEventRegistry", "LearnStack.Infrastructure.Messaging.OutboxDispatchOptions", + "System.TimeProvider", "Microsoft.Extensions.DependencyInjection.IServiceScopeFactory", "LearnStack.Api.Messaging.OutboxObservation", + "Microsoft.Extensions.Logging.ILogger`1<" + BoundaryWorker + ">"); + + private static string RequireBoundaryMethod(SortedSet offenders, TypeDefinition type, string name, + string result, MethodAttributes visibility, params string[] parameters) + { + // Expected authority comes from this fixed contract, never from all current + // methods sharing a permitted name. A new overload cannot authorize itself. + var expected = Key(type).Assembly + ":" + result + " " + type.FullName + "::" + name + "(" + string.Join(",", parameters) + ")"; + var declarations = type.Methods.Where(method => method.Name == name).ToArray(); + if (declarations.Length != 1 || declarations[0].IsStatic || declarations[0].HasGenericParameters + || (declarations[0].Attributes & MethodAttributes.MemberAccessMask) != visibility + || declarations[0].ReturnType.FullName != result || !ParametersMatch(declarations[0], parameters)) + offenders.Add(expected + ": requires exactly one instance declaration with its approved signature and visibility"); + return expected; + } + + private static void CheckCapturedKeyResolver(SortedSet offenders, ProductionCensus census, + MethodDefinition factory, MethodDefinition leaf) + { + // Recognize this one compiler-emitted closure: (provider, key) => + // new Lazy(() => provider.GetRequiredKeyedService(key)). + // No general dataflow interpreter or generated-name permission is involved. + var calls = leaf.Body.Instructions.Where(instruction => instruction.Operand is MethodReference call && IsKeyedResolution(call)).ToArray(); + var call = calls.Length == 1 ? calls[0] : null; + var keyField = call?.Previous?.Operand as FieldReference; + var providerField = call?.Previous?.Previous?.Previous?.Operand as FieldReference; + if (call?.Operand is not MethodReference target || target.Name != "GetRequiredKeyedService" + || GenericService(target) != BoundaryDataSource || keyField is null || providerField is null + || call.Previous!.OpCode != OpCodes.Ldfld || call.Previous.Previous!.OpCode != OpCodes.Ldarg_0 + || call.Previous.Previous.Previous!.OpCode != OpCodes.Ldfld + || !CapturesParameter(census, factory, keyField, 1) || !CapturesParameter(census, factory, providerField, 0)) + offenders.Add(MethodId(leaf) + ": lazy resolver must forward its own registered key and provider unchanged"); + } + + private static bool CapturesParameter(ProductionCensus census, MethodDefinition factory, FieldReference field, int parameter) + { + var argument = factory.Parameters[parameter]; + if (factory.Body.Instructions.Any(instruction => instruction.Operand is ParameterDefinition target && target == argument + && (instruction.OpCode == OpCodes.Starg || instruction.OpCode == OpCodes.Starg_S + || instruction.OpCode == OpCodes.Ldarga || instruction.OpCode == OpCodes.Ldarga_S))) return false; + // The matching original capture is insufficient: a later callback/helper + // write can replace the key or provider while preserving that instruction. + // Close every direct write and address escape of this exact captured field. + var mutations = census.Types.Values.SelectMany(type => type.Methods).Where(method => method.HasBody) + .SelectMany(method => method.Body.Instructions.Where(instruction => + instruction.Operand is FieldReference assigned && SameType(assigned.DeclaringType, field.DeclaringType) + && assigned.Name == field.Name && (instruction.OpCode == OpCodes.Stfld || instruction.OpCode == OpCodes.Stsfld + || instruction.OpCode == OpCodes.Ldflda || instruction.OpCode == OpCodes.Ldsflda)) + .Select(instruction => (Method: method, Instruction: instruction))).ToArray(); + return mutations.Length == 1 && mutations[0].Method == factory && mutations[0].Instruction.OpCode == OpCodes.Stfld + && LoadsArgument(mutations[0].Instruction.Previous, parameter + (factory.HasThis ? 1 : 0)); + } + + private static bool LoadsArgument(Instruction? instruction, int index) => instruction is not null && (index switch + { + 0 => instruction.OpCode == OpCodes.Ldarg_0, + 1 => instruction.OpCode == OpCodes.Ldarg_1, + 2 => instruction.OpCode == OpCodes.Ldarg_2, + 3 => instruction.OpCode == OpCodes.Ldarg_3, + _ => false, + }); + + private static void CheckLiteralKeyResolver(SortedSet offenders, MethodDefinition owner, string service, string key) + { + var calls = owner.Body.Instructions.Where(instruction => instruction.Operand is MethodReference call && IsKeyedResolution(call)).ToArray(); + if (calls.Length != 1 || calls[0].Operand is not MethodReference target + || target.Name != "GetRequiredKeyedService" || GenericService(target) != service + || calls[0].Previous?.OpCode != OpCodes.Ldstr || !Equals(calls[0].Previous.Operand, key)) + offenders.Add(MethodId(owner) + ": requires exactly its declared keyed service"); + } + + private static bool IsKeyedResolution(MethodReference method) => + method.DeclaringType.Namespace.StartsWith("Microsoft.Extensions.DependencyInjection", StringComparison.Ordinal) + && method.Name is "GetKeyedService" or "GetRequiredKeyedService" or "GetKeyedServices"; + + private static bool IsConnectionStringRead(MethodReference method) => + method.DeclaringType.FullName == "Microsoft.Extensions.Configuration.ConfigurationExtensions" && method.Name == "GetConnectionString"; + + private static bool IsConfigurationIndexer(MethodReference method) => method.Name == "get_Item" + && method.ReturnType.FullName == "System.String" && ParametersMatch(method, ["System.String"]) + && method.DeclaringType.FullName is "Microsoft.Extensions.Configuration.IConfiguration" + or "Microsoft.Extensions.Configuration.ConfigurationManager" or "Microsoft.Extensions.Configuration.ConfigurationRoot" + or "Microsoft.Extensions.Configuration.ConfigurationSection"; + + private static object? BoundaryAttributeValue(CustomAttributeArgument argument, int depth = 0) => depth >= 8 ? null + : argument.Value is CustomAttributeArgument nested ? BoundaryAttributeValue(nested, depth + 1) : argument.Value; + + private static bool IsKeyedRegistration(MethodReference method) => + method.DeclaringType.Namespace.StartsWith("Microsoft.Extensions.DependencyInjection", StringComparison.Ordinal) + && (method.Name.StartsWith("AddKeyed", StringComparison.Ordinal) || method.Name.StartsWith("TryAddKeyed", StringComparison.Ordinal) + || method.Name.StartsWith("KeyedSingleton", StringComparison.Ordinal) || method.Name.StartsWith("KeyedScoped", StringComparison.Ordinal) + || method.Name.StartsWith("KeyedTransient", StringComparison.Ordinal) + || method.Name == "DescribeKeyed" + || (method.Name == ".ctor" && method.DeclaringType.Name == "ServiceDescriptor" + && method.Parameters.Count >= 3 && method.Parameters[1].ParameterType.FullName == "System.Object")); + + private static string GenericService(MethodReference method) => method is GenericInstanceMethod generic + && generic.GenericArguments.Count == 1 ? generic.GenericArguments[0].FullName : "non-generic:" + method.FullName; + + private static IEnumerable MethodOperands(MethodDefinition method) => method.HasBody + ? method.Body.Instructions.Select(instruction => instruction.Operand).OfType() : []; + + private static IEnumerable DelegateTargets(MethodDefinition method) => method.HasBody + ? method.Body.Instructions.Where(instruction => instruction.OpCode == OpCodes.Ldftn || instruction.OpCode == OpCodes.Ldvirtftn) + .Select(instruction => instruction.Operand).OfType() : []; + + private static MethodDefinition? FindRuntimeMethod(ProductionCensus census, System.Reflection.MethodInfo method) => + census.Modules.Single(module => module.Assembly.Name.Name == method.Module.Assembly.GetName().Name) + .GetTypes().SelectMany(type => type.Methods).SingleOrDefault(candidate => candidate.MetadataToken.ToInt32() == method.MetadataToken); + + private static SortedSet PublicationOffenders(ProductionCensus census) + { + var offenders = new SortedSet(StringComparer.Ordinal); + var types = census.Types; + CheckSet(offenders, "production assemblies", census.Modules.Select(module => module.Assembly.Name.Name), ProductionAssemblies.Names()); + foreach (var subject in new[] { (KernelAssembly, BoundaryBus), (InfrastructureAssembly, BoundaryTransport), + (BoundaryApi, BoundaryWorker), (BoundaryApi, BoundaryAttempt) }) + if (!types.ContainsKey(subject)) offenders.Add("Missing publication subject: " + subject.Item2); + if (offenders.Count != 0) return offenders; + var port = types[(KernelAssembly, BoundaryBus)]; + var attempt = types[(BoundaryApi, BoundaryAttempt)]; + var worker = types[(BoundaryApi, BoundaryWorker)]; + CheckSet(offenders, "production event-bus implementations", types.Values.Where(type => !type.IsInterface + && !type.IsAbstract && Derives(type, port, types)).Select(TypeId), + [TypeId(types[(InfrastructureAssembly, BoundaryTransport)])]); + var publishers = attempt.Methods.Where(method => method.Name == "PublishAsync").ToArray(); + var constructors = attempt.Methods.Where(method => method.IsConstructor && !method.IsStatic).ToArray(); + var dispatchers = worker.Methods.Where(method => method.Name == "DispatchNormalAsync").ToArray(); + if (publishers.Length != 1 || !publishers[0].IsPrivate || publishers[0].IsStatic + || publishers[0].ReturnType.FullName != "System.Threading.Tasks.Task`1" + || !ParametersMatch(publishers[0], ["Microsoft.Extensions.DependencyInjection.IServiceScopeFactory", + "LearnStack.SharedKernel.Messaging.IntegrationEventEnvelope"]) || !attempt.IsSealed || attempt.IsPublic) + offenders.Add(BoundaryAttempt + ": requires its internal sealed, private publication boundary"); + if (constructors.Length != 1 || dispatchers.Length != 1) + offenders.Add(BoundaryAttempt + ": requires one constructor and one worker owner"); + if (offenders.Count != 0) return offenders; + var sites = CallSites(census).ToArray(); + CheckSet(offenders, "event-bus publication callers", sites.Where(site => site.Target.Name.EndsWith("PublishAsync", StringComparison.Ordinal) + && (SameType(site.Target.DeclaringType, port) || Derives(site.Target.DeclaringType, port, types))) + .Select(site => site.Owner), [MethodId(publishers[0])]); + CheckSet(offenders, "publication attempt constructors", sites.Where(site => site.Target.Name == ".ctor" + && SameType(site.Target.DeclaringType, attempt)).Select(site => site.Owner), [MethodId(dispatchers[0])]); + var callbacks = DelegateTargets(constructors[0]).ToArray(); + if (callbacks.Length != 1) offenders.Add(BoundaryAttempt + ": requires exactly one task-owned callback"); + CheckSet(offenders, "private publication callback callers", sites.Where(site => site.Target.Name == "PublishAsync" + && SameType(site.Target.DeclaringType, attempt)).Select(site => site.Owner), callbacks.Select(MethodId)); + return offenders; + } +} diff --git a/backend/tests/LearnStack.Tests.Architecture/OutboxDispatchConventionTests.cs b/backend/tests/LearnStack.Tests.Architecture/OutboxDispatchConventionTests.cs new file mode 100644 index 00000000..956bf997 --- /dev/null +++ b/backend/tests/LearnStack.Tests.Architecture/OutboxDispatchConventionTests.cs @@ -0,0 +1,604 @@ +using FluentAssertions; +using Mono.Cecil; +using Mono.Cecil.Cil; +using Xunit; + +namespace LearnStack.Tests.Architecture; + +/// ADR-0046: the dispatcher owns the restricted terminal-transition capability. +public sealed partial class OutboxDispatchConventionTests +{ + private const string KernelAssembly = "LearnStack.SharedKernel"; + private const string InfrastructureAssembly = "LearnStack.Infrastructure"; + private const string AuditAssembly = "LearnStack.Infrastructure.Audit"; + private const string Witness = "LearnStack.SharedKernel.Audit.EventTerminalTransition"; + private const string MintingSource = Witness + "/MintingSource"; + private const string Store = "LearnStack.Infrastructure.Messaging.OutboxDispatchStore"; + private const string AuditPort = "LearnStack.SharedKernel.Audit.IAuditStore"; + private const string AuditStore = "LearnStack.Infrastructure.Audit.PostgresAuditStore"; + private const string Mint = "FromProducerUpdate"; + private const string Terminalize = "TryTerminalizeAsync"; + private const string TerminalWrite = "WriteEventTerminalAsync"; + private static readonly string[] MintParameters = + [ + "System.Data.Common.DbConnection", "System.Data.Common.DbTransaction", + "System.Data.Common.DbDataReader", "System.String", + ]; + private static readonly string[] TerminalWriteParameters = + [ + "LearnStack.SharedKernel.Audit.AuditEntryDraft", Witness, + "System.Data.Common.DbConnection", "System.Data.Common.DbTransaction", + "System.Threading.CancellationToken", + ]; + private static readonly string[] ResolutionRoots = + [ + "LearnStack.Infrastructure.Messaging.OutboxEventRegistry", + "LearnStack.Infrastructure.Messaging.OutboxEnvelopeReader", + Store, + ]; + + /// + /// ADR-0046, Atomic terminal audit and privilege boundary: restricted minting and + /// fifth-write callers are closed over every production assembly. PostgreSQL tests + /// separately prove that the live UPDATE result and draft identity agree. + /// + [Fact] + public void Outbox_Terminal_Audit_Identity_Is_Row_Bound() + { + using var census = ProductionCensus.Read(); + TerminalOffenders(census).Should().BeEmpty( + "Fix: keep the immutable private witness, its sole dispatcher minting subtype, " + + "and the terminal writer/call sites closed; keep row/transaction identity checks in the real audit seam"); + } + + [Fact] + public void Terminal_census_detects_extra_derivers_constructors_and_mint_callers() + { + using var census = ProductionCensus.Read(); + var source = census.Types[(KernelAssembly, MintingSource)]; + var factory = source.Methods.Single(method => method.Name == Mint); + var constructor = census.Types[(KernelAssembly, Witness)].Methods.Single(method => method.IsConstructor); + var application = census.Modules.Single(module => module.Assembly.Name.Name == "LearnStack.Application"); + + AssertTerminalMutation(census, "a second minting subtype in another production assembly", () => + { + var type = AddProbe(application, "AdditionalMintingSource"); + type.BaseType = application.ImportReference(source); + return () => application.Types.Remove(type); + }); + AssertTerminalMutation(census, "an inherited minting subtype", () => + { + var parent = AddProbe(application, "AbstractMintingSource"); + parent.Attributes |= TypeAttributes.Abstract; + parent.BaseType = application.ImportReference(source); + var child = AddProbe(application, "InheritedMintingSource"); + child.BaseType = parent; + return () => { application.Types.Remove(child); application.Types.Remove(parent); }; + }); + foreach (var opcode in new[] { OpCodes.Call, OpCodes.Ldftn }) + { + AssertTerminalMutation(census, "a cross-assembly mint helper using " + opcode.Name, () => + AddCallProbe(application, "AdditionalMintCaller", factory, opcode)); + } + AssertTerminalMutation(census, "a direct private witness constructor call", () => + AddCallProbe(application, "AdditionalWitnessConstructor", constructor, OpCodes.Newobj)); + AssertTerminalMutation(census, "an async mint helper with an identified unauthorized owner", () => + AddAsyncCallProbe(application, "AsyncMintCaller", factory)); + AssertTerminalMutation(census, "a helper on the sanctioned dispatcher type", () => + { + var type = census.Types[(InfrastructureAssembly, Store)]; + var method = AddCall(type, "UnrelatedHelper", factory, OpCodes.Call); + return () => type.Methods.Remove(method); + }); + AssertTerminalMutation(census, "a generated name that has no real async owner", () => + { + var type = census.Types[(InfrastructureAssembly, Store)]; + var generated = new TypeDefinition(string.Empty, "d__999", TypeAttributes.NestedPrivate); + generated.BaseType = type.Module.TypeSystem.Object; + type.NestedTypes.Add(generated); + AddCall(generated, "MoveNext", factory, OpCodes.Call); + return () => type.NestedTypes.Remove(generated); + }); + } + + [Fact] + public void Terminal_census_detects_interface_concrete_delegate_async_and_extra_writer_paths() + { + using var census = ProductionCensus.Read(); + var port = census.Types[(KernelAssembly, AuditPort)]; + var writer = census.Types[(AuditAssembly, AuditStore)]; + var interfaceWrite = port.Methods.Single(method => method.Name == TerminalWrite); + var concreteWrite = writer.Methods.Single(method => method.Name == TerminalWrite); + var application = census.Modules.Single(module => module.Assembly.Name.Name == "LearnStack.Application"); + + foreach (var target in new[] { interfaceWrite, concreteWrite }) + foreach (var opcode in new[] { OpCodes.Call, OpCodes.Callvirt, OpCodes.Ldftn, OpCodes.Ldvirtftn }) + { + AssertTerminalMutation(census, "an extra " + target.DeclaringType.Name + " caller using " + opcode.Name, () => + AddCallProbe(application, "AdditionalTerminalWriter", target, opcode)); + } + AssertTerminalMutation(census, "an async helper with an identified unauthorized owner", () => + AddAsyncCallProbe(application, "AsyncTerminalWriter", interfaceWrite)); + AssertTerminalMutation(census, "a second IAuditStore implementation", () => + { + var type = AddProbe(application, "AdditionalAuditStore"); + type.Interfaces.Add(new InterfaceImplementation(application.ImportReference(port))); + var method = new MethodDefinition(TerminalWrite, MethodAttributes.Public, + application.ImportReference(interfaceWrite.ReturnType)); + foreach (var parameter in interfaceWrite.Parameters) + method.Parameters.Add(new ParameterDefinition(application.ImportReference(parameter.ParameterType))); + type.Methods.Add(method); + return () => application.Types.Remove(type); + }); + AssertTerminalMutation(census, "an inherited concrete IAuditStore implementation", () => + { + var type = AddProbe(application, "InheritedAuditStore"); + type.BaseType = application.ImportReference(writer); + return () => application.Types.Remove(type); + }); + } + + [Fact] + public void Terminal_census_detects_mutability_accessibility_missing_subjects_and_friend_access() + { + using var census = ProductionCensus.Read(); + var witness = census.Types[(KernelAssembly, Witness)]; + var source = census.Types[(KernelAssembly, MintingSource)]; + var factory = source.Methods.Single(method => method.Name == Mint); + var constructor = witness.Methods.Single(method => method.IsConstructor); + AssertTerminalMutation(census, "an unsealed witness", () => + ChangeAttributes(() => witness.Attributes, value => witness.Attributes = value, + witness.Attributes & ~TypeAttributes.Sealed)); + AssertTerminalMutation(census, "a public witness constructor", () => + ChangeAttributes(() => constructor.Attributes, value => constructor.Attributes = value, + (constructor.Attributes & ~MethodAttributes.MemberAccessMask) | MethodAttributes.Public)); + AssertTerminalMutation(census, "a mutable witness field", () => + { + var field = witness.Fields.First(field => !field.IsStatic); + return ChangeAttributes(() => field.Attributes, value => field.Attributes = value, + field.Attributes & ~FieldAttributes.InitOnly); + }); + AssertTerminalMutation(census, "a witness setter", () => + { + var property = witness.Properties.First(); + var setter = new MethodDefinition("set_" + property.Name, MethodAttributes.Public, witness.Module.TypeSystem.Void); + var original = property.SetMethod; + property.SetMethod = setter; + return () => property.SetMethod = original; + }); + AssertTerminalMutation(census, "a public minting seam", () => + ChangeAttributes(() => factory.Attributes, value => factory.Attributes = value, + (factory.Attributes & ~MethodAttributes.MemberAccessMask) | MethodAttributes.Public)); + AssertTerminalMutation(census, "a caller-identity mint signature", () => + { + var parameter = factory.Parameters[2]; + var original = parameter.ParameterType; + parameter.ParameterType = witness.Module.TypeSystem.String; + return () => parameter.ParameterType = original; + }); + AssertTerminalMutation(census, "a caller-identity witness constructor", () => + { + var parameter = constructor.Parameters[2]; + var original = parameter.ParameterType; + parameter.ParameterType = witness.Module.TypeSystem.String; + return () => parameter.ParameterType = original; + }); + foreach (var name in new[] { Mint, TerminalWrite }) + { + AssertTerminalMutation(census, "an empty " + name + " call-site census", () => + { + var owner = census.Types.Values.SelectMany(type => type.Methods).Single(method => method.HasBody + && method.Body.Instructions.Any(instruction => instruction.Operand is MethodReference target + && target.Name == name)); + var instruction = owner.Body.Instructions.Single(instruction => instruction.Operand is MethodReference target + && target.Name == name); + var index = owner.Body.Instructions.IndexOf(instruction); + owner.Body.Instructions.RemoveAt(index); + return () => owner.Body.Instructions.Insert(index, instruction); + }); + } + AssertTerminalMutation(census, "a missing dispatcher", () => + { + var type = census.Types[(InfrastructureAssembly, Store)]; + var module = type.Module; + module.Types.Remove(type); + return () => module.Types.Add(type); + }); + AssertTerminalMutation(census, "a SharedKernel friend assembly", () => + { + var module = witness.Module; + var attribute = new CustomAttribute(module.ImportReference( + typeof(System.Runtime.CompilerServices.InternalsVisibleToAttribute).GetConstructor([typeof(string)])!)); + attribute.ConstructorArguments.Add(new CustomAttributeArgument(module.TypeSystem.String, "ReviewProbe")); + module.Assembly.CustomAttributes.Add(attribute); + return () => module.Assembly.CustomAttributes.Remove(attribute); + }); + AssertTerminalMutation(census, "an audit-store constructor without the required catalogue", () => + { + var type = census.Types[(AuditAssembly, AuditStore)]; + var method = new MethodDefinition(".ctor", MethodAttributes.Public | MethodAttributes.SpecialName + | MethodAttributes.RTSpecialName, type.Module.TypeSystem.Void); + type.Methods.Add(method); + return () => type.Methods.Remove(method); + }); + } + + /// + /// ADR-0046: persisted type text selects a declared registry entry. This bounded + /// production helper walk bans named dynamic-loading/construction APIs; it does + /// not claim arbitrary reflection or string-dataflow analysis. + /// + [Fact] + public void Outbox_Type_Resolution_Is_Declared_And_Bounded() + { + using var census = ProductionCensus.Read(); + ResolutionOffenders(census).Should().BeEmpty( + "Fix: resolve stored identities only through the declared registry and deserialize its EventType; " + + "remove dynamic type loading or Activator construction from the resolver/validator/store helper graph"); + } + + [Fact] + public void Resolution_walk_detects_loading_construction_async_and_polymorphic_helpers() + { + using var census = ProductionCensus.Read(); + var application = census.Modules.Single(module => module.Assembly.Name.Name == "LearnStack.Application"); + var root = census.Types[(InfrastructureAssembly, ResolutionRoots[0])]; + var targets = new[] + { + typeof(Type).GetMethod(nameof(Type.GetType), [typeof(string)])!, + typeof(System.Reflection.Assembly).GetMethod(nameof(System.Reflection.Assembly.Load), [typeof(string)])!, + typeof(System.Reflection.Assembly).GetMethod(nameof(System.Reflection.Assembly.LoadFrom), [typeof(string)])!, + typeof(System.Reflection.Assembly).GetMethod(nameof(System.Reflection.Assembly.GetType), [typeof(string)])!, + typeof(System.Reflection.Assembly).GetMethod(nameof(System.Reflection.Assembly.CreateInstance), [typeof(string)])!, + typeof(Activator).GetMethod(nameof(Activator.CreateInstance), [typeof(Type)])!, + }; + foreach (var target in targets) + foreach (var opcode in new[] { OpCodes.Call, OpCodes.Ldftn }) + { + AssertResolutionMutation(census, target.DeclaringType!.Name + "." + target.Name + " using " + opcode.Name, () => + { + var helper = AddProbe(application, "DynamicResolutionHelper"); + AddCall(helper, "Resolve", application.ImportReference(target), opcode); + var field = new FieldDefinition("ReviewHelper", FieldAttributes.Private, root.Module.ImportReference(helper)); + root.Fields.Add(field); + return () => { root.Fields.Remove(field); application.Types.Remove(helper); }; + }); + } + AssertResolutionMutation(census, "an async cross-assembly resolver helper", () => + { + var helper = AddProbe(application, "AsyncResolutionHelper"); + var undoAsync = AddAsyncCall(helper, "Resolve", application.ImportReference(targets[0])); + var field = new FieldDefinition("ReviewHelper", FieldAttributes.Private, root.Module.ImportReference(helper)); + root.Fields.Add(field); + return () => { root.Fields.Remove(field); undoAsync(); application.Types.Remove(helper); }; + }); + foreach (var isInterface in new[] { true, false }) + { + AssertResolutionMutation(census, "a " + (isInterface ? "interface" : "virtual") + " resolver helper", () => + { + var contract = AddProbe(application, "ResolutionContract"); + if (isInterface) contract.Attributes |= TypeAttributes.Interface | TypeAttributes.Abstract; + var helper = AddProbe(application, "ResolutionImplementation"); + if (isInterface) helper.Interfaces.Add(new InterfaceImplementation(contract)); + else helper.BaseType = contract; + AddCall(helper, "Resolve", application.ImportReference(targets[0]), OpCodes.Call); + var field = new FieldDefinition("ReviewHelper", FieldAttributes.Private, root.Module.ImportReference(contract)); + root.Fields.Add(field); + return () => { root.Fields.Remove(field); application.Types.Remove(helper); application.Types.Remove(contract); }; + }); + } + } + + [Fact] + public void Resolution_walk_preserves_declared_type_metadata_and_clean_helpers() + { + using var census = ProductionCensus.Read(); + var application = census.Modules.Single(module => module.Assembly.Name.Name == "LearnStack.Application"); + var root = census.Types[(InfrastructureAssembly, ResolutionRoots[0])]; + var helper = AddProbe(application, "DeclaredTypeHelper"); + AddCall(helper, "ReadKnownAssembly", application.ImportReference( + typeof(Type).GetProperty(nameof(Type.Assembly))!.GetMethod!), OpCodes.Callvirt); + AddCall(helper, "ReadRuntimeType", application.ImportReference( + typeof(object).GetMethod(nameof(GetType))!), OpCodes.Callvirt); + var field = new FieldDefinition("ReviewHelper", FieldAttributes.Private, root.Module.ImportReference(helper)); + root.Fields.Add(field); + try + { + ResolutionOffenders(census).Should().BeEmpty( + "Fix: preserve metadata reads of already declared types and clean cross-assembly helpers"); + } + finally + { + root.Fields.Remove(field); + application.Types.Remove(helper); + } + } + + private static SortedSet TerminalOffenders(ProductionCensus census) + { + var offenders = new SortedSet(StringComparer.Ordinal); + var types = census.Types; + CheckSet(offenders, "production assemblies", census.Modules.Select(module => module.Assembly.Name.Name), ProductionAssemblies.Names()); + foreach (var key in new[] { (KernelAssembly, Witness), (KernelAssembly, MintingSource), + (InfrastructureAssembly, Store), (KernelAssembly, AuditPort), (AuditAssembly, AuditStore) }) + { + if (!types.ContainsKey(key)) offenders.Add("Missing production subject: " + key.Item2); + } + if (offenders.Count != 0) return offenders; + + var witness = types[(KernelAssembly, Witness)]; + var source = types[(KernelAssembly, MintingSource)]; + var store = types[(InfrastructureAssembly, Store)]; + var port = types[(KernelAssembly, AuditPort)]; + var auditStore = types[(AuditAssembly, AuditStore)]; + var constructors = witness.Methods.Where(method => method.IsConstructor && !method.IsStatic).ToArray(); + if (!witness.IsSealed || witness.BaseType?.FullName != "System.Object") offenders.Add(Witness + ": must be sealed with no custom base"); + if (constructors.Length != 1 || constructors.Any(method => !method.IsPrivate || !ParametersMatch(method, MintParameters))) + offenders.Add(Witness + ": requires exactly one private live-reader constructor"); + if (witness.Fields.All(field => field.IsStatic) || witness.Fields.Any(field => !field.IsStatic && !field.IsInitOnly)) + offenders.Add(Witness + ": instance fields must be nonempty and readonly"); + if (witness.Properties.Count == 0 || witness.Properties.Any(property => property.SetMethod is not null)) + offenders.Add(Witness + ": properties must be nonempty and getter-only"); + if (!source.IsNestedPublic || !source.IsAbstract || source.IsSealed) + offenders.Add(MintingSource + ": requires the public abstract nested minting capability"); + if (source.Methods.Where(method => method.IsConstructor && !method.IsStatic).Any(method => !method.IsFamily)) + offenders.Add(MintingSource + ": constructors must be protected"); + if (witness.Module.Assembly.CustomAttributes.Any(attribute => + attribute.AttributeType.FullName == "System.Runtime.CompilerServices.InternalsVisibleToAttribute")) + offenders.Add(KernelAssembly + ": friend assemblies reopen restricted authority"); + + var factories = source.Methods.Where(method => method.Name == Mint).ToArray(); + if (factories.Length != 1 || factories.Any(method => !method.IsFamily || !method.IsStatic + || method.ReturnType.FullName != Witness || !ParametersMatch(method, MintParameters))) + offenders.Add(MintingSource + "." + Mint + ": requires the protected static live-reader mint signature"); + var terminalizers = store.Methods.Where(method => method.Name == Terminalize).ToArray(); + if (terminalizers.Length != 1) offenders.Add(Store + "." + Terminalize + ": requires one concrete terminalization owner"); + CheckSet(offenders, "minting subtypes", types.Values.Where(type => Derives(type, source, types)).Select(TypeId), [TypeId(store)]); + CheckSet(offenders, "witness-producing methods", types.Values.SelectMany(type => type.Methods) + .Where(method => method.ReturnType.FullName == Witness).Select(MethodId), factories.Select(MethodId)); + + var writers = types.Values.Where(type => !type.IsInterface && !type.IsAbstract && Derives(type, port, types)).ToArray(); + CheckSet(offenders, "audit store implementations", writers.Select(TypeId), [TypeId(auditStore)]); + var declarations = types.Values.Where(type => SameType(type, port) || Derives(type, port, types)) + .SelectMany(type => type.Methods).Where(IsTerminalWriteName).ToArray(); + CheckSet(offenders, "terminal audit declarations", declarations.Select(method => TypeId(method.DeclaringType) + "." + method.Name), + [TypeId(port) + "." + TerminalWrite, TypeId(auditStore) + "." + TerminalWrite]); + foreach (var method in declarations.Where(method => method.ReturnType.FullName != "System.Threading.Tasks.Task" + || !ParametersMatch(method, TerminalWriteParameters))) + offenders.Add(MethodId(method) + ": wrong fifth-write signature"); + var auditConstructors = auditStore.Methods.Where(method => method.IsConstructor && !method.IsStatic).ToArray(); + if (auditConstructors.Length == 0 || auditConstructors.Any(method => method.Parameters.All(parameter => + parameter.ParameterType.FullName != "LearnStack.SharedKernel.Audit.IAuditCatalog"))) + offenders.Add(AuditStore + ": every constructor must require IAuditCatalog"); + + var sites = CallSites(census).ToArray(); + CheckSet(offenders, "witness constructor call sites", sites.Where(site => site.Target.Name == ".ctor" + && SameType(site.Target.DeclaringType, witness)).Select(site => site.Owner), factories.Select(MethodId)); + CheckSet(offenders, "mint call sites", sites.Where(site => site.Target.Name == Mint + && (SameType(site.Target.DeclaringType, source) || Derives(site.Target.DeclaringType, source, types))) + .Select(site => site.Owner), terminalizers.Select(MethodId)); + CheckSet(offenders, "terminal audit call sites", sites.Where(site => IsTerminalWriteName(site.Target) + && (SameType(site.Target.DeclaringType, port) || Derives(site.Target.DeclaringType, port, types))) + .Select(site => site.Owner), terminalizers.Select(MethodId)); + return offenders; + } + + private static SortedSet ResolutionOffenders(ProductionCensus census) + { + var offenders = new SortedSet(StringComparer.Ordinal); + var types = census.Types; + CheckSet(offenders, "production assemblies", census.Modules.Select(module => module.Assembly.Name.Name), ProductionAssemblies.Names()); + foreach (var name in ResolutionRoots) + if (!types.ContainsKey((InfrastructureAssembly, name))) offenders.Add("Missing production resolution subject: " + name); + if (offenders.Count != 0) return offenders; + + var pending = new Queue(ResolutionRoots.Select(name => types[(InfrastructureAssembly, name)])); + var visited = new HashSet<(string Assembly, string Type)>(); + while (pending.TryDequeue(out var type)) + { + if (!visited.Add(Key(type))) continue; + foreach (var method in Il.Methods(type).Select(item => item.Definition).Where(method => method.HasBody)) + foreach (var instruction in method.Body.Instructions) + { + if (instruction.Operand is MethodReference target && DynamicResolution(target)) + offenders.Add(MethodId(method) + " -> " + target.FullName); + } + foreach (var reference in Il.ReferencedTypeReferences(type)) + { + if (!types.TryGetValue(Key(reference), out var helper)) continue; + pending.Enqueue(helper); + if (helper.IsSealed) continue; + foreach (var implementation in types.Values.Where(candidate => !candidate.IsInterface + && Derives(candidate, helper, types))) + pending.Enqueue(implementation); + } + } + return offenders; + } + + private static bool DynamicResolution(MethodReference target) => target.DeclaringType.FullName switch + { + "System.Type" => target.Name == "GetType", + "System.Reflection.Assembly" => target.Name is "GetType" or "CreateInstance" or "Load" or "LoadFrom" or "LoadFile" or "UnsafeLoadFrom", + "System.Activator" => target.Name is "CreateInstance" or "CreateInstanceFrom", + _ => false, + }; + + private static IEnumerable<(string Owner, MethodReference Target)> CallSites(ProductionCensus census) + { + // Attribute mapping is deliberate: a nested type merely named + // d__123 must not inherit that owner's authority. + var asyncOwners = new Dictionary<(string Assembly, string Type), string>(); + foreach (var method in census.Types.Values.SelectMany(type => type.Methods)) + foreach (var attribute in method.CustomAttributes.Where(attribute => + attribute.AttributeType.FullName is "System.Runtime.CompilerServices.AsyncStateMachineAttribute" + or "System.Runtime.CompilerServices.IteratorStateMachineAttribute" + or "System.Runtime.CompilerServices.AsyncIteratorStateMachineAttribute")) + { + if (attribute.ConstructorArguments.FirstOrDefault().Value is TypeReference generated) + asyncOwners[Key(generated)] = MethodId(method); + } + foreach (var method in census.Types.Values.SelectMany(type => type.Methods).Where(method => method.HasBody)) + { + var owner = method.Name == "MoveNext" && asyncOwners.TryGetValue(Key(method.DeclaringType), out var original) + ? original : MethodId(method); + foreach (var instruction in method.Body.Instructions) + { + // Includes call/callvirt, newobj, ldftn and ldvirtftn. A delegate + // cannot hide a capability just because its invocation occurs later. + if (instruction.Operand is MethodReference target) yield return (owner, target); + } + } + } + + private static bool Derives(TypeReference candidate, TypeDefinition contract, + Dictionary<(string Assembly, string Type), TypeDefinition> types) + { + if (!types.TryGetValue(Key(candidate), out var definition)) return false; + var pending = new Queue(definition.Interfaces.Select(item => item.InterfaceType)); + if (definition.BaseType is { } parent) pending.Enqueue(parent); + var visited = new HashSet<(string Assembly, string Type)>(); + while (pending.TryDequeue(out var reference)) + { + if (!visited.Add(Key(reference))) continue; + if (SameType(reference, contract)) return true; + if (!types.TryGetValue(Key(reference), out var inherited)) continue; + foreach (var item in inherited.Interfaces) pending.Enqueue(item.InterfaceType); + if (inherited.BaseType is { } baseType) pending.Enqueue(baseType); + } + return false; + } + + private static bool IsTerminalWriteName(MethodReference method) => + method.Name == TerminalWrite || method.Name.EndsWith("." + TerminalWrite, StringComparison.Ordinal); + + private static bool ParametersMatch(MethodReference method, string[] names) => + method.Parameters.Select(parameter => parameter.ParameterType.FullName).SequenceEqual(names, StringComparer.Ordinal); + + private static bool SameType(TypeReference left, TypeReference right) => Key(left) == Key(right); + + private static (string Assembly, string Type) Key(TypeReference reference) + { + if (reference is TypeSpecification specification) return Key(specification.ElementType); + return (reference.Scope is AssemblyNameReference assembly ? assembly.Name + : reference.Scope is ModuleDefinition module ? module.Assembly.Name.Name : reference.Scope.Name, + reference.FullName); + } + + private static string TypeId(TypeReference type) => Key(type).Assembly + ":" + type.FullName; + private static string MethodId(MethodReference method) => Key(method.DeclaringType).Assembly + ":" + method.FullName; + + private static void CheckSet(SortedSet offenders, string label, IEnumerable actual, IEnumerable expected) + { + var found = actual.Order(StringComparer.Ordinal).ToArray(); + var required = expected.Order(StringComparer.Ordinal).ToArray(); + if (!found.SequenceEqual(required, StringComparer.Ordinal)) + offenders.Add(label + ": expected [" + string.Join(", ", required) + "]; actual [" + string.Join(", ", found) + "]"); + } + + private static void AssertTerminalMutation(ProductionCensus census, string description, Func mutate) => + AssertMutation(census, description, mutate, TerminalOffenders); + + private static void AssertResolutionMutation(ProductionCensus census, string description, Func mutate) => + AssertMutation(census, description, mutate, ResolutionOffenders); + + private static void AssertMutation(ProductionCensus census, string description, Func mutate, + Func> detector) + { + detector(census).Should().BeEmpty("Fix: the planted control requires a clean nonempty production baseline"); + var undo = mutate(); + try + { + detector(census).Should().NotBeEmpty("Fix: the same production detector must reject " + description); + } + finally + { + undo(); + } + detector(census).Should().BeEmpty("Fix: removing the planted offender must restore a clean production census"); + } + + private static Action ChangeAttributes(Func read, Action write, T replacement) + { + var original = read(); + write(replacement); + return () => write(original); + } + + private static TypeDefinition AddProbe(ModuleDefinition module, string name) + { + var type = new TypeDefinition("ReviewProbe", name, TypeAttributes.Public | TypeAttributes.Class, + module.TypeSystem.Object); + module.Types.Add(type); + return type; + } + + private static MethodDefinition AddCall(TypeDefinition type, string name, MethodReference target, OpCode opcode) + { + var method = new MethodDefinition(name, MethodAttributes.Public | MethodAttributes.Static, type.Module.TypeSystem.Void); + method.Body.Instructions.Add(Instruction.Create(opcode, type.Module.ImportReference(target))); + method.Body.Instructions.Add(Instruction.Create(OpCodes.Ret)); + type.Methods.Add(method); + return method; + } + + private static Action AddCallProbe(ModuleDefinition module, string name, MethodReference target, OpCode opcode) + { + var type = AddProbe(module, name); + AddCall(type, "Run", target, opcode); + return () => module.Types.Remove(type); + } + + private static Action AddAsyncCallProbe(ModuleDefinition module, string name, MethodReference target) + { + var type = AddProbe(module, name); + var undo = AddAsyncCall(type, "Run", target); + return () => { undo(); module.Types.Remove(type); }; + } + + private static Action AddAsyncCall(TypeDefinition type, string name, MethodReference target) + { + var module = type.Module; + var owner = new MethodDefinition(name, MethodAttributes.Public | MethodAttributes.Static, module.TypeSystem.Void); + type.Methods.Add(owner); + var generated = new TypeDefinition(string.Empty, "<" + name + ">d__999", TypeAttributes.NestedPrivate, + module.TypeSystem.Object); + type.NestedTypes.Add(generated); + AddCall(generated, "MoveNext", target, OpCodes.Call); + var attribute = new CustomAttribute(module.ImportReference( + typeof(System.Runtime.CompilerServices.AsyncStateMachineAttribute).GetConstructor([typeof(Type)])!)); + attribute.ConstructorArguments.Add(new CustomAttributeArgument(module.ImportReference(typeof(Type)), generated)); + owner.CustomAttributes.Add(attribute); + return () => { type.NestedTypes.Remove(generated); type.Methods.Remove(owner); }; + } + + private sealed class ProductionCensus : IDisposable + { + private ProductionCensus(List modules) => Modules = modules; + public List Modules { get; } + // Rebuild after each planted mutation so additions/removals are part of the + // same census rather than hidden behind the original type-map snapshot. + public Dictionary<(string Assembly, string Type), TypeDefinition> Types => + Modules.SelectMany(module => module.GetTypes()).ToDictionary(Key); + + public static ProductionCensus Read() + { + var modules = new List(); + try + { + foreach (var assembly in ProductionAssemblies.All()) modules.Add(ModuleDefinition.ReadModule(assembly.Location)); + return new ProductionCensus(modules); + } + catch + { + foreach (var module in modules) module.Dispose(); + throw; + } + } + + public void Dispose() + { + foreach (var module in Modules) module.Dispose(); + } + } +} diff --git a/backend/tests/LearnStack.Tests.Architecture/OutboxDispatchObservationTests.cs b/backend/tests/LearnStack.Tests.Architecture/OutboxDispatchObservationTests.cs new file mode 100644 index 00000000..55080d44 --- /dev/null +++ b/backend/tests/LearnStack.Tests.Architecture/OutboxDispatchObservationTests.cs @@ -0,0 +1,426 @@ +using System.Text.RegularExpressions; +using FluentAssertions; +using LearnStack.Api.Composition; +using LearnStack.Api.Messaging; +using LearnStack.Infrastructure.Messaging; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Diagnostics.HealthChecks; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Options; +using Mono.Cecil; +using Mono.Cecil.Cil; +using Xunit; + +namespace LearnStack.Tests.Architecture; + +public sealed partial class OutboxDispatchConventionTests +{ + private const string ApiAssembly = "LearnStack.Api"; + private const string Observation = "LearnStack.Api.Messaging.OutboxObservation"; + private const string Health = "LearnStack.Api.Messaging.OutboxHealthCheck"; + private static readonly string[] ObservationSubjects = + [ + Observation, Health, "LearnStack.Api.Messaging.OutboxObservationSnapshot", + "LearnStack.Api.Messaging.OutboxProcessor", "LearnStack.Api.Messaging.OutboxPublishAttempt", + ]; + + /// + /// ADR-0046: the real worker/check registrations and memory-only observation graph + /// cannot disappear or acquire persistence/service-locator capabilities. Runtime + /// tests separately prove deadlines, snapshot values, diagnostics and stale health. + /// + [Fact] + public void Outbox_Health_And_Metrics_Are_Bounded() + { + using var census = ProductionCensus.Read(); + ObservationOffenders(census).Should().BeEmpty( + "Fix: retain the production worker/observation subjects and keep gauge/health helpers memory-only"); + ObservationRegistrationOffenders(ObservationServices()).Should().BeEmpty( + "Fix: register exactly one mandatory dispatcher and private outbox health check without forcing its lazy source"); + } + + [Fact] + public void Observation_walk_detects_persistence_locators_async_and_polymorphic_helpers() + { + using var census = ProductionCensus.Read(); + var application = census.Modules.Single(module => module.Assembly.Name.Name == "LearnStack.Application"); + foreach (var rootName in new[] { Observation, Health }) + foreach (var capability in new[] { typeof(Npgsql.NpgsqlCommand), typeof(System.Data.Common.DbConnection), + typeof(IServiceProvider), typeof(IServiceScopeFactory) }) + { + AssertMutation(census, "a cross-assembly observation helper with " + capability.Name, () => + { + var root = census.Types[(ApiAssembly, rootName)]; + var helper = AddProbe(application, "ObservationPersistenceHelper"); + helper.Fields.Add(new FieldDefinition("Capability", FieldAttributes.Public, + application.ImportReference(capability))); + var field = new FieldDefinition("ReviewHelper", FieldAttributes.Private, root.Module.ImportReference(helper)); + root.Fields.Add(field); + return () => { root.Fields.Remove(field); application.Types.Remove(helper); }; + }, ObservationOffenders); + } + foreach (var isInterface in new[] { true, false }) + AssertMutation(census, "a polymorphic observation helper", () => + { + var root = census.Types[(ApiAssembly, Observation)]; + var contract = AddProbe(application, "ObservationContract"); + if (isInterface) contract.Attributes |= TypeAttributes.Interface | TypeAttributes.Abstract; + var implementation = AddProbe(application, "ObservationImplementation"); + if (isInterface) implementation.Interfaces.Add(new InterfaceImplementation(contract)); + else implementation.BaseType = contract; + var getter = application.ImportReference(typeof(IServiceScopeFactory).GetMethod(nameof(IServiceScopeFactory.CreateScope))!); + var undoAsync = AddAsyncCall(implementation, "Read", getter); + var field = new FieldDefinition("ReviewHelper", FieldAttributes.Private, root.Module.ImportReference(contract)); + root.Fields.Add(field); + return () => + { + root.Fields.Remove(field); + undoAsync(); + application.Types.Remove(implementation); + application.Types.Remove(contract); + }; + }, ObservationOffenders); + foreach (var subject in ObservationSubjects) + AssertMutation(census, "a missing observation subject " + subject, () => + { + var type = census.Types[(ApiAssembly, subject)]; + var module = type.Module; + module.Types.Remove(type); + return () => module.Types.Add(type); + }, ObservationOffenders); + } + + [Fact] + public void Observation_walk_keeps_clean_memory_helpers_and_checks_real_gauge_callbacks() + { + using var census = ProductionCensus.Read(); + var observation = census.Types[(ApiAssembly, Observation)]; + var callbacks = observation.Methods.SelectMany(method => method.HasBody + ? method.Body.Instructions : Enumerable.Empty()) + .Where(instruction => instruction.OpCode == OpCodes.Ldftn && instruction.Operand is MethodReference) + .Select(instruction => (MethodReference)instruction.Operand) + .Where(method => SameType(method.DeclaringType, observation)).ToArray(); + callbacks.Should().HaveCountGreaterThanOrEqualTo(4, + "Fix: the census must inspect the shipped observable callbacks, not an empty test-only graph"); + foreach (var callback in callbacks) + AssertMutation(census, "persistence inside an actual observation callback", () => + { + var method = observation.Methods.Single(method => method.FullName == callback.FullName); + var reference = observation.Module.ImportReference(typeof(Npgsql.NpgsqlCommand).GetConstructor(Type.EmptyTypes)!); + var instruction = Instruction.Create(OpCodes.Newobj, reference); + method.Body.Instructions.Insert(0, instruction); + return () => method.Body.Instructions.Remove(instruction); + }, ObservationOffenders); + + var helper = AddProbe(observation.Module, "MemoryObservationHelper"); + var field = new FieldDefinition("ReviewHelper", FieldAttributes.Private, helper); + helper.Fields.Add(new FieldDefinition("Count", FieldAttributes.Public, helper.Module.TypeSystem.Int32)); + observation.Fields.Add(field); + try { ObservationOffenders(census).Should().BeEmpty("Fix: pure memory helpers must remain admitted"); } + finally { observation.Fields.Remove(field); observation.Module.Types.Remove(helper); } + } + + [Fact] + public void Observation_registration_control_detects_missing_duplicate_or_replaced_worker_and_check() + { + IServiceCollection services = ObservationServices(); + ObservationRegistrationOffenders(services).Should().BeEmpty(); + var worker = services.Single(descriptor => descriptor.ServiceType == typeof(IHostedService) + && descriptor.ImplementationType == typeof(OutboxProcessor)); + services.Remove(worker); + ObservationRegistrationOffenders(services).Should().NotBeEmpty("Fix: missing mandatory execution must be detected"); + services.Add(worker); + services.Add(worker); + ObservationRegistrationOffenders(services).Should().NotBeEmpty("Fix: duplicate workers must be detected"); + services.Remove(worker); + foreach (var extra in new[] + { + ServiceDescriptor.Singleton(provider => ActivatorUtilities.CreateInstance(provider)), + ServiceDescriptor.Singleton(new UnrelatedHostedService()), + }) + { + services.Add(extra); + ObservationRegistrationOffenders(services).Should().NotBeEmpty("Fix: factory and instance hosted-service duplicates must be detected"); + services.Remove(extra); + } + ObservationRegistrationOffenders(services).Should().BeEmpty("Fix: restoring the descriptor set must restore the clean control"); + foreach (var replacement in new[] { "missing", "duplicate", "wrong-factory" }) + { + var modified = ObservationServices(); + modified.PostConfigure(options => + { + var registration = options.Registrations.Single(item => item.Name == OutboxHealthCheck.Name); + options.Registrations.Remove(registration); + if (replacement != "missing") + options.Registrations.Add(new HealthCheckRegistration(OutboxHealthCheck.Name, + _ => new UnrelatedHealthCheck(), HealthStatus.Unhealthy, [])); + if (replacement == "duplicate") options.Registrations.Add(registration); + }); + ObservationRegistrationOffenders(modified).Should().NotBeEmpty("Fix: detect " + replacement + " health registration"); + } + ObservationRegistrationOffenders(ObservationServices()).Should().BeEmpty(); + foreach (var replacement in new[] { "missing", "duplicate", "instance", "factory" }) + { + IServiceCollection modified = ObservationServices(); + var clock = modified.Single(descriptor => descriptor.ServiceType == typeof(TimeProvider)); + modified.Remove(clock); + if (replacement == "duplicate") { modified.Add(clock); modified.Add(clock); } + if (replacement == "instance") modified.AddSingleton(new AlternateTimeProvider()); + if (replacement == "factory") modified.AddSingleton(_ => new AlternateTimeProvider()); + ObservationRegistrationOffenders(modified).Should().NotBeEmpty("Fix: reject " + replacement + " production clock registration"); + } + var detachedServices = ObservationServices(); + using var detachedProvider = detachedServices.BuildServiceProvider(); + using var detached = new OutboxObservation(TimeProvider.System, new OutboxDispatchOptions(), + detachedProvider.GetRequiredService()); + detachedServices.PostConfigure(options => + { + options.Registrations.Remove(options.Registrations.Single(item => item.Name == OutboxHealthCheck.Name)); + options.Registrations.Add(new HealthCheckRegistration(OutboxHealthCheck.Name, + _ => new OutboxHealthCheck(detached), HealthStatus.Unhealthy, [])); + }); + ObservationRegistrationOffenders(detachedServices).Should().NotBeEmpty( + "Fix: a same-type check disconnected from the worker's singleton observation must fail"); + ObservationRegistrationOffenders(ObservationServices()).Should().BeEmpty( + "Fix: the exact production clock and singleton health binding must remain the clean control"); + } + + /// + /// ADR-0046: all literal SQL targeting outbox row mutations belongs to the producer, + /// exact transition methods or the one legacy backfill. This structural companion + /// does not claim to analyze dynamically constructed table names or SQL dataflow. + /// Real PostgreSQL tests prove predicates, fencing and atomicity. + /// + [Fact] + public void Outbox_Retry_And_Terminal_Transitions_Are_Fenced() + { + using var census = ProductionCensus.Read(); + OutboxSqlOffenders(census).Should().BeEmpty( + "Fix: keep outbox literal INSERT/status mutation SQL in the exact producer, dispatcher and backfill methods"); + } + + [Fact] + public void Outbox_SQL_census_detects_quoted_schema_CTE_extra_and_missing_writer_paths() + { + using var census = ProductionCensus.Read(); + var application = census.Modules.Single(module => module.Assembly.Name.Name == "LearnStack.Application"); + foreach (var sql in new[] + { + "update outbox_messages set attempts = 0", "UPDATE ONLY public.\"outbox_messages\" SET attempts=0", + "WITH c AS (SELECT 1) UpDaTe \"public\" . \"outbox_messages\" SET attempts=0", + "INSERT INTO public.outbox_messages (id) VALUES (NULL)", "DELETE FROM \"outbox_messages\"", + "DELETE FROM ONLY public.outbox_messages", "UPDATE /* review */ public.outbox_messages SET attempts=0", + "MERGE INTO outbox_messages USING x ON true WHEN MATCHED THEN DELETE", + "COPY public.outbox_messages FROM STDIN", "TRUNCATE TABLE public.outbox_messages", + "UPDATE outbox_messages/* review */ SET attempts=0", + "INSERT/* review */INTO outbox_messages(id) VALUES(NULL)", + "DELETE/* review */FROM outbox_messages WHERE true", + "TRUNCATE TABLE another_table, outbox_messages", + "TRUNCATE ONLY another_table, ONLY public.\"outbox_messages\" RESTART IDENTITY", + "TRUNCATE TABLE another_table, outbox_messages*", + }) + AssertMutation(census, "an unauthorized literal SQL writer: " + sql, () => AddSqlProbe(application, sql), OutboxSqlOffenders); + AssertMutation(census, "an extra helper on the allowed store", () => + { + var store = census.Types[(InfrastructureAssembly, Store)]; + var method = AddSqlMethod(store, "UnexpectedWrite", "UPDATE outbox_messages SET attempts=0"); + return () => store.Methods.Remove(method); + }, OutboxSqlOffenders); + AssertMutation(census, "an extra overload of an allowed transition", () => + { + var store = census.Types[(InfrastructureAssembly, Store)]; + var method = AddSqlMethod(store, "AcquireAsync", "UPDATE outbox_messages SET attempts=0"); + return () => store.Methods.Remove(method); + }, OutboxSqlOffenders); + AssertMutation(census, "a missing production mutation path", () => + { + var literal = OwnedBodies(census).Where(item => item.Owner.DeclaringType.FullName == Store) + .SelectMany(item => item.Body.Body.Instructions) + .First(instruction => instruction.OpCode == OpCodes.Ldstr && instruction.Operand is string sql && OutboxMutation.IsMatch(sql)); + var original = literal.Operand; + literal.Operand = "SELECT 1"; + return () => literal.Operand = original; + }, OutboxSqlOffenders); + } + + [Fact] + public void Outbox_SQL_census_preserves_reads_and_unrelated_table_writes() + { + using var census = ProductionCensus.Read(); + var application = census.Modules.Single(module => module.Assembly.Name.Name == "LearnStack.Application"); + foreach (var sql in new[] { "SELECT * FROM outbox_messages FOR UPDATE SKIP LOCKED", + "UPDATE another_outbox_messages SET attempts=0", "INSERT INTO outbox_messages_archive VALUES (1)", + "TRUNCATE TABLE another_table, outbox_messages_archive" }) + { + var undo = AddSqlProbe(application, sql); + try { OutboxSqlOffenders(census).Should().BeEmpty("Fix: admitted reads and unrelated table writes must stay clean"); } + finally { undo(); } + } + } + + private const string SqlGap = "(?:\\s|/\\*.*?\\*/|--[^\\r\\n]*(?:\\r?\\n|$))"; + private const string SqlIdentifier = "(?:\"[^\"]+\"|[a-z_][a-z0-9_$]*)"; + private const string SqlQualifier = "(?:" + SqlIdentifier + SqlGap + "*\\." + SqlGap + "*)?"; + private const string OutboxTarget = SqlQualifier + "(?:\"outbox_messages\"|outbox_messages)"; + private static readonly Regex OutboxMutation = new( + "\\b(?:INSERT" + SqlGap + "+INTO|UPDATE|DELETE" + SqlGap + "+FROM|MERGE" + SqlGap + "+INTO|COPY)" + + SqlGap + "+(?:ONLY" + SqlGap + "+)?" + OutboxTarget + "(?=" + SqlGap + "|[;(*]|$)" + + "|\\bTRUNCATE" + SqlGap + "+(?:TABLE" + SqlGap + "+)?" + + "(?:(?:ONLY" + SqlGap + "+)?" + SqlQualifier + SqlIdentifier + "(?:" + SqlGap + "*\\*)?" + + SqlGap + "*," + SqlGap + "*)*(?:ONLY" + SqlGap + "+)?" + OutboxTarget + + "(?=" + SqlGap + "|[,*;]|$)", + RegexOptions.IgnoreCase | RegexOptions.CultureInvariant | RegexOptions.Singleline, TimeSpan.FromSeconds(1)); + + private static SortedSet OutboxSqlOffenders(ProductionCensus census) + { + var offenders = new SortedSet(StringComparer.Ordinal); + CheckSet(offenders, "SQL production assemblies", census.Modules.Select(module => module.Assembly.Name.Name), ProductionAssemblies.Names()); + var writers = OwnedBodies(census).SelectMany(item => item.Body.Body.Instructions + .Where(instruction => instruction.OpCode == OpCodes.Ldstr && instruction.Operand is string sql && OutboxMutation.IsMatch(sql)) + .Select(_ => MethodId(item.Owner))); + var expected = new List(); + foreach (var (typeName, methodName) in new[] + { + ("LearnStack.Infrastructure.Messaging.PostgresOutbox", "EnqueueAsync"), + (Store, "AcquireAsync"), (Store, "UpdateAsync"), (Store, "TryTerminalizeAsync"), + ("LearnStack.Infrastructure.Persistence.Migrations.add_outbox_dispatch_state", "Up"), + }) + { + if (!census.Types.TryGetValue((InfrastructureAssembly, typeName), out var type)) + { offenders.Add("Missing SQL writer type: " + typeName); continue; } + var methods = type.Methods.Where(method => method.Name == methodName).ToArray(); + if (methods.Length != 1) offenders.Add("Missing or ambiguous SQL writer: " + typeName + "." + methodName); + else expected.Add(MethodId(methods[0])); + } + CheckSet(offenders, "literal outbox writer methods", writers, expected); + return offenders; + } + + private static IEnumerable<(MethodDefinition Body, MethodDefinition Owner)> OwnedBodies(ProductionCensus census) + { + var owners = census.Types.Values.SelectMany(type => type.Methods) + .SelectMany(method => method.CustomAttributes.Where(attribute => attribute.AttributeType.FullName is + "System.Runtime.CompilerServices.AsyncStateMachineAttribute" or "System.Runtime.CompilerServices.IteratorStateMachineAttribute" + or "System.Runtime.CompilerServices.AsyncIteratorStateMachineAttribute") + .Where(attribute => attribute.ConstructorArguments.FirstOrDefault().Value is TypeReference) + .Select(attribute => (Type: Key((TypeReference)attribute.ConstructorArguments[0].Value), Method: method))) + .ToDictionary(item => item.Type, item => item.Method); + foreach (var method in census.Types.Values.SelectMany(type => type.Methods).Where(method => method.HasBody)) + yield return (method, method.Name == "MoveNext" && owners.TryGetValue(Key(method.DeclaringType), out var owner) ? owner : method); + } + + private static SortedSet ObservationOffenders(ProductionCensus census) + { + var offenders = new SortedSet(StringComparer.Ordinal); + var types = census.Types; + CheckSet(offenders, "observation production assemblies", census.Modules.Select(module => module.Assembly.Name.Name), ProductionAssemblies.Names()); + foreach (var subject in ObservationSubjects) + if (!types.ContainsKey((ApiAssembly, subject))) offenders.Add("Missing observation subject: " + subject); + if (offenders.Count != 0) return offenders; + var pending = new Queue(new[] { types[(ApiAssembly, Observation)], types[(ApiAssembly, Health)] }); + var visited = new HashSet<(string Assembly, string Type)>(); + while (pending.TryDequeue(out var type)) + { + if (!visited.Add(Key(type))) continue; + foreach (var reference in Il.ReferencedTypeReferences(type)) + { + if (reference.Namespace.StartsWith("Npgsql", StringComparison.Ordinal) + || reference.Namespace.StartsWith("System.Data", StringComparison.Ordinal) + || reference.Namespace.StartsWith("Microsoft.EntityFrameworkCore", StringComparison.Ordinal) + || reference.FullName is "System.IServiceProvider" or "Microsoft.Extensions.DependencyInjection.IServiceScopeFactory" + or "Microsoft.Extensions.DependencyInjection.IKeyedServiceProvider") + offenders.Add(TypeId(type) + " -> " + reference.FullName); + if (!types.TryGetValue(Key(reference), out var helper)) continue; + pending.Enqueue(helper); + if (!helper.IsSealed) + foreach (var implementation in types.Values.Where(candidate => !candidate.IsInterface && Derives(candidate, helper, types))) + pending.Enqueue(implementation); + } + } + return offenders; + } + + private static ServiceCollection ObservationServices() + { + var services = new ServiceCollection(); + services.AddSingleton(IntegrationEventHandlerRegistry.Discover()); + services.AddLearnStackPersistence(new ConfigurationBuilder().Build()); + return services; + } + + private static SortedSet ObservationRegistrationOffenders(IServiceCollection services) + { + var offenders = new SortedSet(StringComparer.Ordinal); + // Include the framework health publisher in the complete descriptor set so + // factory/instance duplicates cannot hide behind the dispatcher type filter. + var workers = services.Where(descriptor => descriptor.ServiceType == typeof(IHostedService)).ToArray(); + CheckSet(offenders, "hosted-service descriptor types", workers.Select(descriptor => descriptor.ImplementationType?.FullName ?? "non-type registration"), + [typeof(OutboxProcessor).FullName!, "Microsoft.Extensions.Diagnostics.HealthChecks.HealthCheckPublisherHostedService"]); + if (workers.Any(descriptor => descriptor.Lifetime != ServiceLifetime.Singleton)) offenders.Add("Mandatory singleton worker registration"); + // The helper graph walks production contracts. Its one framework-owned + // virtual clock capability is admitted only as the shipped system instance. + var clocks = services.Where(descriptor => descriptor.ServiceType == typeof(TimeProvider)).ToArray(); + if (clocks.Length != 1 || clocks[0].Lifetime != ServiceLifetime.Singleton + || !ReferenceEquals(clocks[0].ImplementationInstance, TimeProvider.System)) + { + offenders.Add("Observation requires its exact singleton system clock registration"); + return offenders; + } + using var provider = services.BuildServiceProvider(); + var checks = provider.GetRequiredService>().Value.Registrations + .Where(check => check.Name == OutboxHealthCheck.Name).ToArray(); + var check = checks.Length == 1 ? checks[0].Factory(provider) : null; + if (check is not OutboxHealthCheck) offenders.Add("Private outbox health registration"); + else + { + var observation = provider.GetRequiredService(); + observation.Started(); + observation.Progress(); + observation.Refreshed(new OutboxQueueSnapshot(0, 0, 0)); + var context = new HealthCheckContext { Registration = checks[0] }; + if (check.CheckHealthAsync(context).GetAwaiter().GetResult().Status != HealthStatus.Healthy) + offenders.Add("Registered check does not follow healthy singleton observation"); + observation.TerminalAuditFailed(); + if (check.CheckHealthAsync(context).GetAwaiter().GetResult().Status != HealthStatus.Unhealthy) + offenders.Add("Registered check does not follow failed singleton observation"); + observation.TerminalAuditSucceeded(); + if (check.CheckHealthAsync(context).GetAwaiter().GetResult().Status != HealthStatus.Healthy) + offenders.Add("Registered check does not follow recovered singleton observation"); + } + if (provider.GetRequiredKeyedService>(OutboxDispatcherDataSource.Key).IsValueCreated) + offenders.Add("Observation forced dispatcher source construction"); + return offenders; + } + + private static Action AddSqlProbe(ModuleDefinition module, string sql) + { + var type = AddProbe(module, "OutboxSqlWriter"); + AddSqlMethod(type, "Write", sql); + return () => module.Types.Remove(type); + } + + private static MethodDefinition AddSqlMethod(TypeDefinition type, string name, string sql) + { + var method = new MethodDefinition(name, MethodAttributes.Public | MethodAttributes.Static, type.Module.TypeSystem.Void); + method.Body.Instructions.Add(Instruction.Create(OpCodes.Ldstr, sql)); + method.Body.Instructions.Add(Instruction.Create(OpCodes.Pop)); + method.Body.Instructions.Add(Instruction.Create(OpCodes.Ret)); + type.Methods.Add(method); + return method; + } + + private sealed class UnrelatedHealthCheck : IHealthCheck + { + public Task CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default) => + Task.FromResult(HealthCheckResult.Healthy()); + } + + private sealed class UnrelatedHostedService : IHostedService + { + public Task StartAsync(CancellationToken cancellationToken) => Task.CompletedTask; + public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask; + } + + private sealed class AlternateTimeProvider : TimeProvider; +} diff --git a/backend/tests/LearnStack.Tests.Architecture/PersistenceConventionTests.cs b/backend/tests/LearnStack.Tests.Architecture/PersistenceConventionTests.cs index d5bb70a7..00ba892c 100644 --- a/backend/tests/LearnStack.Tests.Architecture/PersistenceConventionTests.cs +++ b/backend/tests/LearnStack.Tests.Architecture/PersistenceConventionTests.cs @@ -170,12 +170,13 @@ public void Module_DbContexts_Enlist_In_The_Ambient_UnitOfWork() // SET LOCAL, so every read through it returns zero rows under the // corrected policy — silently. // - // Eight files under backend/src may reach for a connection at all: the five + // Nine files under backend/src may reach for a connection at all: the five // design-time factories, where a connection string is the point — one per // migration chain, and a module that ships a schema ships one; the shared // module helper, which passes a connection rather than a string; the API's // separately guarded platform-role builder; and the shared application-role - // builder used by both runtime hosts. A ninth is a new decision. + // builder used by both runtime hosts; and ADR-0046's confined dispatcher + // builder. Another site requires a new decision, not a wildcard exemption. // // The scan covers the raw constructors as well as `UseNpgsql` and // `AddDbContext`, because a call site that opened its own @@ -214,6 +215,7 @@ public void Module_DbContexts_Enlist_In_The_Ambient_UnitOfWork() // Both runtime hosts delegate application-role construction to this // shared guard; the API retains its separate platform-role builder. "Persistence/ApplicationDataSource.cs", + "Messaging/OutboxDispatcherDataSource.cs", ]); } diff --git a/backend/tests/LearnStack.Tests.Architecture/PlatformAdminScopeConventionTests.cs b/backend/tests/LearnStack.Tests.Architecture/PlatformAdminScopeConventionTests.cs index 0538cbb9..be114051 100644 --- a/backend/tests/LearnStack.Tests.Architecture/PlatformAdminScopeConventionTests.cs +++ b/backend/tests/LearnStack.Tests.Architecture/PlatformAdminScopeConventionTests.cs @@ -1,6 +1,9 @@ using System.Reflection; using FluentAssertions; +using LearnStack.Api.Messaging; +using LearnStack.Infrastructure.Messaging; using LearnStack.SharedKernel.Tenancy; +using Microsoft.Extensions.DependencyInjection; using Xunit; namespace LearnStack.Tests.Architecture; @@ -27,6 +30,12 @@ public sealed class PlatformAdminScopeConventionTests private const string CompositionFile = "LearnStack.Api/Composition/PersistenceCompositionExtensions.cs"; + // A distinct keyed credential, confined by ADR-0046. Naming it explicitly + // preserves this global resolution fence while permitting its approved root. + private const string OutboxCompositionFile = + "LearnStack.Api/Composition/OutboxCompositionExtensions.cs"; + private const string OutboxWorkerFile = "LearnStack.Api/Messaging/OutboxProcessor.cs"; + [Fact] public void Platform_DataSource_Resolved_Only_By_PlatformAdminScope() { @@ -48,9 +57,17 @@ public void Platform_DataSource_Resolved_Only_By_PlatformAdminScope() .ToList(); resolvers.Should().BeEquivalentTo( - [ScopeFile, CompositionFile], - "the scope injects the keyed source and the composition root registers it; a " - + "third resolver is a second path to a credential that sees every tenant"); + [ScopeFile, CompositionFile, OutboxCompositionFile, OutboxWorkerFile], + "the platform scope and the separately confined dispatcher are the only " + + "credential consumers, alongside their composition roots"); + + typeof(OutboxProcessor).GetConstructors().SelectMany(constructor => constructor.GetParameters()) + .Select(parameter => parameter.GetCustomAttribute()) + .OfType().Should().ContainSingle().Which.Key + .Should().Be(OutboxDispatcherDataSource.Key, + "permitting the worker file must not permit the platform credential"); + File.ReadAllText(Path.Combine(SourceScan.SourceRoot, OutboxWorkerFile)) + .Should().NotContain("PlatformAdmin"); // Leg 2 — IConfiguration.GetConnectionString is called in exactly one file under // backend/src. Not "every credential is read in one file": the two design-time @@ -65,8 +82,11 @@ public void Platform_DataSource_Resolved_Only_By_PlatformAdminScope() // is a rule matching its own vocabulary rather than a credential. SourceScan.FilesContaining(SourceScan.SourceRoot, "GetConnectionString", except: null) .Should().BeEquivalentTo( - [CompositionFile], - "one file reads credentials, so one file decides what is done with them"); + [CompositionFile, OutboxCompositionFile], + "each credential has an explicitly confined composition root"); + + File.ReadAllText(Path.Combine(SourceScan.SourceRoot, OutboxCompositionFile)) + .Should().NotContain("PlatformAdmin", "outbox composition must never read the platform credential"); // Leg 2b — the OTHER way to read the same value. GetConnectionString(name) is // sugar for configuration[$"ConnectionStrings:{name}"], so the indexer form diff --git a/backend/tests/LearnStack.Tests.Contract/DevelopmentWebApplicationFactory.cs b/backend/tests/LearnStack.Tests.Contract/DevelopmentWebApplicationFactory.cs index be0a0e27..31e7f255 100644 --- a/backend/tests/LearnStack.Tests.Contract/DevelopmentWebApplicationFactory.cs +++ b/backend/tests/LearnStack.Tests.Contract/DevelopmentWebApplicationFactory.cs @@ -1,5 +1,8 @@ +using LearnStack.Api.Messaging; using Microsoft.AspNetCore.Hosting; using Microsoft.AspNetCore.Mvc.Testing; +using Microsoft.AspNetCore.TestHost; +using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Hosting; namespace LearnStack.Tests.Contract; @@ -15,13 +18,24 @@ namespace LearnStack.Tests.Contract; /// gated MapOpenApi() on IsDevelopment(), so the document 404'd /// in CI without it. Packet 4 removed that gate — the document is the contract /// the SDK generates from and CI diffs, so it is served in every environment — -/// and the override now exists only to keep the contract suite on one known -/// environment rather than on whatever the runner happens to set. +/// and the environment override keeps the contract suite on one known environment. +/// This served-contract fixture also removes the dispatcher worker because it owns +/// no dispatcher database. Its retained private outbox health check intentionally +/// remains Unhealthy; the static /healthz liveness endpoint stays independent. /// public sealed class DevelopmentWebApplicationFactory : WebApplicationFactory { protected override void ConfigureWebHost(IWebHostBuilder builder) { builder.UseEnvironment(Environments.Development); + // This served-contract proof owns no dispatcher database. Remove only + // that worker; production startup tests retain the mandatory registration. + builder.ConfigureTestServices(services => + { + foreach (var descriptor in services.Where(descriptor => + descriptor.ServiceType == typeof(IHostedService) + && descriptor.ImplementationType == typeof(OutboxProcessor)).ToArray()) + services.Remove(descriptor); + }); } } diff --git a/backend/tests/LearnStack.Tests.Integration/ApiHandlerCompositionTests.cs b/backend/tests/LearnStack.Tests.Integration/ApiHandlerCompositionTests.cs index 9e36beb1..6a3a0c32 100644 --- a/backend/tests/LearnStack.Tests.Integration/ApiHandlerCompositionTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/ApiHandlerCompositionTests.cs @@ -31,12 +31,12 @@ namespace LearnStack.Tests.Integration; /// [Trait(Database.RequiresDocker.Key, Database.RequiresDocker.Value)] [Collection(Database.SharedSchema.Name)] -public sealed class ApiHandlerCompositionTests : IClassFixture> +public sealed class ApiHandlerCompositionTests : IClassFixture { private readonly WebApplicationFactory _factory; private readonly Database.SchemaFixture _schema; - public ApiHandlerCompositionTests(WebApplicationFactory factory, Database.SchemaFixture schema) + public ApiHandlerCompositionTests(ConstructionOnlyApiFactory factory, Database.SchemaFixture schema) { _factory = factory; _schema = schema; diff --git a/backend/tests/LearnStack.Tests.Integration/ApiVersioningHttpTests.cs b/backend/tests/LearnStack.Tests.Integration/ApiVersioningHttpTests.cs index 73b5b94d..57af6d07 100644 --- a/backend/tests/LearnStack.Tests.Integration/ApiVersioningHttpTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/ApiVersioningHttpTests.cs @@ -279,9 +279,9 @@ private async Task GetDocumentAsync(string major) /// declaring v2 — plus a second live major so the per-document /// filtering has something to filter. /// -public sealed class ApiVersioningFixture : WebApplicationFactory +public sealed class ApiVersioningFixture : ConstructionOnlyApiFactory { - protected override void ConfigureWebHost(IWebHostBuilder builder) + protected override void ConfigureTestHost(IWebHostBuilder builder) { ArgumentNullException.ThrowIfNull(builder); builder.UseEnvironment(Environments.Development); diff --git a/backend/tests/LearnStack.Tests.Integration/AuthorityCeilingHttpTests.cs b/backend/tests/LearnStack.Tests.Integration/AuthorityCeilingHttpTests.cs index e31b18aa..f23cc741 100644 --- a/backend/tests/LearnStack.Tests.Integration/AuthorityCeilingHttpTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/AuthorityCeilingHttpTests.cs @@ -175,13 +175,13 @@ public sealed class HostClassificationMeter : ICollectionFixtureA host whose resolver maps one tenant host and nothing else. -public sealed class AuthorityCeilingFixture : WebApplicationFactory +public sealed class AuthorityCeilingFixture : ConstructionOnlyApiFactory { public const string TenantHost = "school.example.com"; public static readonly Guid Tenant = Guid.Parse("018f4d40-0000-7000-8000-0000000000d1"); - protected override void ConfigureWebHost(IWebHostBuilder builder) + protected override void ConfigureTestHost(IWebHostBuilder builder) { ArgumentNullException.ThrowIfNull(builder); builder.UseEnvironment(Environments.Development); diff --git a/backend/tests/LearnStack.Tests.Integration/CompositionRootCatalogueTests.cs b/backend/tests/LearnStack.Tests.Integration/CompositionRootCatalogueTests.cs index 42dd7b3d..22dc27d8 100644 --- a/backend/tests/LearnStack.Tests.Integration/CompositionRootCatalogueTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/CompositionRootCatalogueTests.cs @@ -29,11 +29,11 @@ namespace LearnStack.Tests.Integration; /// exactly the entries the discovered sources declare — no source missing, none extra. /// /// -public sealed class CompositionRootCatalogueTests : IClassFixture> +public sealed class CompositionRootCatalogueTests : IClassFixture { private readonly WebApplicationFactory _factory; - public CompositionRootCatalogueTests(WebApplicationFactory factory) => _factory = factory; + public CompositionRootCatalogueTests(ConstructionOnlyApiFactory factory) => _factory = factory; [Fact] public void The_API_root_builds_the_discovered_catalogue() diff --git a/backend/tests/LearnStack.Tests.Integration/ConstructionOnlyApiFactory.cs b/backend/tests/LearnStack.Tests.Integration/ConstructionOnlyApiFactory.cs new file mode 100644 index 00000000..39a8832b --- /dev/null +++ b/backend/tests/LearnStack.Tests.Integration/ConstructionOnlyApiFactory.cs @@ -0,0 +1,43 @@ +using LearnStack.Api.Messaging; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.Mvc.Testing; +using Microsoft.AspNetCore.TestHost; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; + +namespace LearnStack.Tests.Integration; + +/// +/// Request/composition proofs own no background dispatcher. Production startup +/// proofs use an ordinary factory and must supply its confined credential. +/// +/// +/// The retained private outbox health check intentionally remains Unhealthy when +/// its worker is removed. The static /healthz liveness endpoint is independent. +/// +public class ConstructionOnlyApiFactory : WebApplicationFactory +{ + protected sealed override void ConfigureWebHost(IWebHostBuilder builder) + { + ArgumentNullException.ThrowIfNull(builder); + ConfigureTestHost(builder); + builder.ConfigureTestServices(RemoveDispatcher); + } + + protected virtual void ConfigureTestHost(IWebHostBuilder builder) + { + ArgumentNullException.ThrowIfNull(builder); + } + + internal static void RemoveDispatcher(IServiceCollection services) + { + // Leave every other hosted service intact, including telemetry. This is + // test composition, never a production flag or a blanket hosted-service ban. + foreach (var descriptor in services.Where(descriptor => + descriptor.ServiceType == typeof(IHostedService) + && descriptor.ImplementationType == typeof(OutboxProcessor)).ToArray()) + { + services.Remove(descriptor); + } + } +} diff --git a/backend/tests/LearnStack.Tests.Integration/ConstructionOnlyApiFactoryTests.cs b/backend/tests/LearnStack.Tests.Integration/ConstructionOnlyApiFactoryTests.cs new file mode 100644 index 00000000..09e17e8a --- /dev/null +++ b/backend/tests/LearnStack.Tests.Integration/ConstructionOnlyApiFactoryTests.cs @@ -0,0 +1,44 @@ +using FluentAssertions; +using LearnStack.Api.Messaging; +using LearnStack.Infrastructure.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Npgsql; +using Xunit; + +namespace LearnStack.Tests.Integration; + +public sealed class ConstructionOnlyApiFactoryTests +{ + [Fact] + public void Request_only_test_composition_removes_exactly_the_dispatcher_and_preserves_other_workers() + { + var services = new ServiceCollection(); + services.AddHostedService(); + services.AddHostedService(); + services.Count(descriptor => descriptor.ServiceType == typeof(IHostedService)).Should().Be(2); + + ConstructionOnlyApiFactory.RemoveDispatcher(services); + + services.Where(descriptor => descriptor.ServiceType == typeof(IHostedService)) + .Should().ContainSingle().Which.ImplementationType.Should().Be(); + } + + [Fact] + public async Task Construction_only_host_leaves_the_dispatcher_source_unresolved() + { + await using var host = new ConstructionOnlyApiFactory(); + using var client = host.CreateClient(); + + host.Services.GetServices().Should().NotContain(service => service is OutboxProcessor); + host.Services.GetRequiredKeyedService>(OutboxDispatcherDataSource.Key) + .IsValueCreated.Should().BeFalse(); + (await client.GetAsync("/healthz")).IsSuccessStatusCode.Should().BeTrue(); + } + + private sealed class OtherWorker : IHostedService + { + public Task StartAsync(CancellationToken cancellationToken) => Task.CompletedTask; + public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask; + } +} diff --git a/backend/tests/LearnStack.Tests.Integration/CrossCuttingFoundationHttpTests.cs b/backend/tests/LearnStack.Tests.Integration/CrossCuttingFoundationHttpTests.cs index e6a58822..194bfb41 100644 --- a/backend/tests/LearnStack.Tests.Integration/CrossCuttingFoundationHttpTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/CrossCuttingFoundationHttpTests.cs @@ -290,9 +290,9 @@ public void The_Audit_Health_State_Is_A_Singleton_Across_Request_Scopes() /// AddLearnStackCrossCuttingFoundation wiring so the L1 handler + /// ValidationBehavior tested here is the same code production runs. /// -public sealed class CrossCuttingHttpFixture : WebApplicationFactory +public sealed class CrossCuttingHttpFixture : ConstructionOnlyApiFactory { - protected override void ConfigureWebHost(IWebHostBuilder builder) + protected override void ConfigureTestHost(IWebHostBuilder builder) { ArgumentNullException.ThrowIfNull(builder); builder.UseEnvironment(Environments.Development); diff --git a/backend/tests/LearnStack.Tests.Integration/Database/AuditStoreTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/AuditStoreTests.cs index 70e33408..a474336a 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/AuditStoreTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/AuditStoreTests.cs @@ -407,7 +407,8 @@ public async Task A_best_effort_failure_is_logged_and_dropped() new Lazy(() => dataSource), logger, MeterFactory, - health); + health, + new AuditCatalog([])); // TenantB is a real tenant, but the draft's own announcement makes the row legal — // so to force a failure the row has to break something the announcement cannot @@ -756,7 +757,8 @@ public async Task A_best_effort_write_swallows_a_data_source_that_cannot_be_buil new Lazy(() => throw new InvalidOperationException("no credential")), NullLogger.Instance, MeterFactory, - new AuditHealth()); + new AuditHealth(), + new AuditCatalog([])); var bestEffort = async () => await store.WriteBestEffortAsync(Draft(organizationId: null)); @@ -940,7 +942,8 @@ public async Task A_standalone_failure_is_Critical_counted_and_leaves_the_check_ new Lazy(() => dataSource), logger, MeterFactory, - health); + health, + new AuditCatalog([])); var counted = 0L; using var listener = new MeterListener(); @@ -995,7 +998,8 @@ public async Task A_failure_the_store_cannot_translate_is_still_Critical_counted () => throw new InvalidOperationException("ConnectionStrings:Default is absent.")), logger, MeterFactory, - health); + health, + new AuditCatalog([])); var counted = 0L; using var listener = new MeterListener(); @@ -1112,7 +1116,7 @@ private static PostgresAuditStore Store(AuditStateCapture capture, NpgsqlDataSou private static PostgresAuditStore Store( AuditStateCapture capture, NpgsqlDataSource dataSource, IAuditHealth health) => new(capture, new Lazy(() => dataSource), - NullLogger.Instance, MeterFactory, health); + NullLogger.Instance, MeterFactory, health, new AuditCatalog([])); /// A real meter factory, so the counter the store increments is a real one. private static readonly IMeterFactory MeterFactory = diff --git a/backend/tests/LearnStack.Tests.Integration/Database/CommitResponseDroppingProxy.cs b/backend/tests/LearnStack.Tests.Integration/Database/CommitResponseDroppingProxy.cs new file mode 100644 index 00000000..21c4dde9 --- /dev/null +++ b/backend/tests/LearnStack.Tests.Integration/Database/CommitResponseDroppingProxy.cs @@ -0,0 +1,173 @@ +using System.Buffers.Binary; +using System.Net; +using System.Net.Sockets; +using Npgsql; + +namespace LearnStack.Tests.Integration.Database; + +/// +/// Forwards a real PostgreSQL session, dropping exactly one completed COMMIT's +/// acknowledgement. It introduces an unknown client outcome after physical commit. +/// +internal sealed class CommitResponseDroppingProxy : IAsyncDisposable +{ + private readonly TcpListener _listener = new(IPAddress.Loopback, 0); + private readonly CancellationTokenSource _stopping = new(TimeSpan.FromSeconds(30)); + private readonly List _sessions = []; + private readonly string _targetHost; + private readonly int _targetPort; + private readonly bool _rejectReconciliationConnections; + private readonly Task _accepting; + private int _droppedCommitResponses; + private int _acceptedConnections; + + public CommitResponseDroppingProxy(string connectionString, bool rejectReconciliationConnections = false) + { + _rejectReconciliationConnections = rejectReconciliationConnections; + var target = new NpgsqlConnectionStringBuilder(connectionString); + _targetHost = target.Host ?? throw new ArgumentException("The test proxy requires a PostgreSQL host.", nameof(connectionString)); + _targetPort = target.Port; + _listener.Start(); + ConnectionString = new NpgsqlConnectionStringBuilder(connectionString) + { + Host = IPAddress.Loopback.ToString(), + Port = ((IPEndPoint)_listener.LocalEndpoint).Port, + // Test-only plaintext forwarding makes protocol frames observable; + // production credentials and data-source validation are unchanged. + SslMode = SslMode.Disable, + GssEncryptionMode = GssEncryptionMode.Disable, + Pooling = false, + Timeout = 5, + }.ConnectionString; + _accepting = AcceptAsync(); + } + + public string ConnectionString { get; } + public int DroppedCommitResponses => Volatile.Read(ref _droppedCommitResponses); + public int AcceptedConnections => Volatile.Read(ref _acceptedConnections); + + public async ValueTask DisposeAsync() + { + await _stopping.CancelAsync(); + _listener.Stop(); + try + { + await _accepting.WaitAsync(TimeSpan.FromSeconds(5)); + // AcceptAsync alone mutates the list; it has completed before this read. + await Task.WhenAll(_sessions).WaitAsync(TimeSpan.FromSeconds(5)); + } + finally + { + _stopping.Dispose(); + } + } + + private async Task AcceptAsync() + { + try + { + while (!_stopping.IsCancellationRequested) + { + var client = await _listener.AcceptTcpClientAsync(_stopping.Token); + Interlocked.Increment(ref _acceptedConnections); + if (_rejectReconciliationConnections && DroppedCommitResponses != 0) + { + // Refuse the fresh durability probe only after the real + // terminal transaction committed and its response was dropped. + client.Dispose(); + continue; + } + _sessions.Add(ForwardSessionAsync(client)); + } + } + catch (OperationCanceledException) when (_stopping.IsCancellationRequested) + { + // Listener shutdown is an owned cancellation, not a swallowed fault. + } + catch (SocketException) when (_stopping.IsCancellationRequested) + { + // Stop can close the listener before cancellation reaches Accept. + } + } + + private async Task ForwardSessionAsync(TcpClient client) + { + using (client) + using (var server = new TcpClient()) + using (var stopping = CancellationTokenSource.CreateLinkedTokenSource(_stopping.Token)) + { + client.NoDelay = true; + server.NoDelay = true; + await server.ConnectAsync(_targetHost, _targetPort, stopping.Token); + await using var clientStream = client.GetStream(); + await using var serverStream = server.GetStream(); + var toServer = clientStream.CopyToAsync(serverStream, stopping.Token); + var toClient = ForwardResponsesAsync(serverStream, clientStream, stopping.Token); + var completed = await Task.WhenAny(toServer, toClient); + try + { + await completed; + } + catch (OperationCanceledException) when (_stopping.IsCancellationRequested) + { + // The proxy's finite lifetime or disposal owns this cancellation. + } + finally + { + await stopping.CancelAsync(); + client.Close(); + server.Close(); + await ObserveStoppedPumpAsync(toServer, stopping.Token); + await ObserveStoppedPumpAsync(toClient, stopping.Token); + } + } + } + + private async Task ForwardResponsesAsync(NetworkStream server, NetworkStream client, CancellationToken cancellationToken) + { + // Backend frames are a type byte, big-endian length including the length + // itself, then a body. CommandComplete 'C' carries a null-terminated tag. + // https://www.postgresql.org/docs/18/protocol-message-formats.html + var header = new byte[5]; + while (await server.ReadAsync(header.AsMemory(0, 1), cancellationToken) != 0) + { + await server.ReadExactlyAsync(header.AsMemory(1, 4), cancellationToken); + var length = BinaryPrimitives.ReadInt32BigEndian(header.AsSpan(1)); + if (length is < 4 or > 1_048_576) + throw new InvalidDataException("The test proxy received an invalid PostgreSQL response frame."); + + var body = new byte[length - 4]; + await server.ReadExactlyAsync(body, cancellationToken); + if (header[0] == (byte)'C' && body.AsSpan().SequenceEqual("COMMIT\0"u8) + && Interlocked.CompareExchange(ref _droppedCommitResponses, 1, 0) == 0) + { + // The real server has committed. Forward neither CommandComplete + // nor its following ReadyForQuery; disposal closes this socket. + return; + } + + await client.WriteAsync(header, cancellationToken); + await client.WriteAsync(body, cancellationToken); + } + } + + private static async Task ObserveStoppedPumpAsync(Task pump, CancellationToken stopped) + { + try + { + await pump; + } + catch (OperationCanceledException) when (stopped.IsCancellationRequested) + { + // The other pump ended and this socket's ownership has been canceled. + } + catch (IOException) when (stopped.IsCancellationRequested) + { + // Closing an owned socket also wakes a pending read/write with IO failure. + } + catch (ObjectDisposedException) when (stopped.IsCancellationRequested) + { + // Cleanup can close the socket before the canceled pump resumes. + } + } +} diff --git a/backend/tests/LearnStack.Tests.Integration/Database/CustomizationProjectionTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/CustomizationProjectionTests.cs index 1d4b70e2..2d61d4d7 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/CustomizationProjectionTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/CustomizationProjectionTests.cs @@ -27,8 +27,8 @@ namespace LearnStack.Tests.Integration.Database; [Trait(RequiresDocker.Key, RequiresDocker.Value)] [Collection(SharedSchema.Name)] -public sealed partial class CustomizationProjectionTests(SchemaFixture schema, WebApplicationFactory factory, ITestOutputHelper output) - : IClassFixture> +public sealed partial class CustomizationProjectionTests(SchemaFixture schema, ConstructionOnlyApiFactory factory, ITestOutputHelper output) + : IClassFixture { private static readonly Dictionary Label = new() { ["en"] = "Label", ["tr"] = "Etiket" }; private const string Profile = """ diff --git a/backend/tests/LearnStack.Tests.Integration/Database/CustomizationPublicationConcurrencyTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/CustomizationPublicationConcurrencyTests.cs index 6edd491a..e65838e4 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/CustomizationPublicationConcurrencyTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/CustomizationPublicationConcurrencyTests.cs @@ -26,8 +26,8 @@ namespace LearnStack.Tests.Integration.Database; [Trait(RequiresDocker.Key, RequiresDocker.Value)] [Collection(SharedSchema.Name)] -public sealed class CustomizationPublicationConcurrencyTests(SchemaFixture schema, WebApplicationFactory factory) - : IClassFixture> +public sealed class CustomizationPublicationConcurrencyTests(SchemaFixture schema, ConstructionOnlyApiFactory factory) + : IClassFixture { [Theory] [InlineData(false, false)] diff --git a/backend/tests/LearnStack.Tests.Integration/Database/DisposableSchemaDatabase.cs b/backend/tests/LearnStack.Tests.Integration/Database/DisposableSchemaDatabase.cs index d9ae4113..efb48f21 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/DisposableSchemaDatabase.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/DisposableSchemaDatabase.cs @@ -21,11 +21,13 @@ private DisposableSchemaDatabase(PostgresFixture postgres) MigrationConnectionString = For(postgres.MigrationConnectionString); AppConnectionString = For(postgres.AppConnectionString); PlatformConnectionString = For(postgres.PlatformConnectionString); + OutboxConnectionString = For(postgres.OutboxConnectionString); } public string MigrationConnectionString { get; } public string AppConnectionString { get; } public string PlatformConnectionString { get; } + public string OutboxConnectionString { get; } public static async Task CreateAsync( PostgresFixture postgres, bool applyMigrations = true) diff --git a/backend/tests/LearnStack.Tests.Integration/Database/DomainEventProductionTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/DomainEventProductionTests.cs index fdf073d3..59fbea34 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/DomainEventProductionTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/DomainEventProductionTests.cs @@ -797,7 +797,7 @@ public async Task Disposing_an_unfinished_scope_discards_collected_and_uncollect private static WebApplicationFactory Host( string connectionString, DispatchObservation observation, ITenantContext? context = null) => - new WebApplicationFactory().WithWebHostBuilder(builder => + new ConstructionOnlyApiFactory().WithWebHostBuilder(builder => { builder.UseSetting("ConnectionStrings:Default", connectionString); builder.ConfigureServices(services => diff --git a/backend/tests/LearnStack.Tests.Integration/Database/EducationWriterTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/EducationWriterTests.cs index 6be92811..86ec0d97 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/EducationWriterTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/EducationWriterTests.cs @@ -30,8 +30,8 @@ namespace LearnStack.Tests.Integration.Database; [Trait(RequiresDocker.Key, RequiresDocker.Value)] [Collection(SharedSchema.Name)] -public sealed class EducationWriterTests(SchemaFixture schema, WebApplicationFactory factory) - : IClassFixture> +public sealed class EducationWriterTests(SchemaFixture schema, ConstructionOnlyApiFactory factory) + : IClassFixture { private const string StringSchema = """{"$schema":"https://json-schema.org/draft/2020-12/schema","type":"object","properties":{"text":{"type":"string"}},"required":["text"],"additionalProperties":false}"""; private const string NumberSchema = """{"$schema":"https://json-schema.org/draft/2020-12/schema","type":"object","properties":{"text":{"type":"number"}},"required":["text"],"additionalProperties":false}"""; diff --git a/backend/tests/LearnStack.Tests.Integration/Database/NoOpAuditStore.cs b/backend/tests/LearnStack.Tests.Integration/Database/NoOpAuditStore.cs index 7cffe483..193efbc1 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/NoOpAuditStore.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/NoOpAuditStore.cs @@ -31,6 +31,14 @@ public Task WriteStandaloneAsync(AuditEntryDraft entry, CancellationToken cancel public Task WriteBestEffortAsync(AuditEntryDraft entry, CancellationToken cancellationToken = default) => Task.CompletedTask; + public Task WriteEventTerminalAsync( + AuditEntryDraft entry, + EventTerminalTransition transition, + System.Data.Common.DbConnection connection, + System.Data.Common.DbTransaction transaction, + CancellationToken cancellationToken = default) => + throw new NotSupportedException("The terminal-event path has its own cases."); + public Task WritePlatformScopeAsync( AuditEntryDraft entry, DbConnection connection, diff --git a/backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatchMigrationTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatchMigrationTests.cs new file mode 100644 index 00000000..e6e68601 --- /dev/null +++ b/backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatchMigrationTests.cs @@ -0,0 +1,356 @@ +using System.Data.Common; +using System.Reflection; +using System.Text.Json; +using FluentAssertions; +using LearnStack.Infrastructure.Messaging; +using LearnStack.Infrastructure.Persistence; +using LearnStack.Modules.Audit.Infrastructure.Persistence; +using LearnStack.SharedKernel.Tenancy; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Migrations; +using Npgsql; +using Xunit; + +namespace LearnStack.Tests.Integration.Database; + +[Trait(RequiresDocker.Key, RequiresDocker.Value)] +[Collection(SharedSchema.Name)] +public sealed class OutboxDispatchMigrationTests(SchemaFixture schema) +{ + private const string PlatformPredecessor = "20260828085701_create_platform_infrastructure_tables"; + private const string DispatchPredecessor = "20261012180355_add_outbox_dispatch_state"; + private const string AuditPredecessor = "20260914114341_audit_log_org_insert_scope"; + private const string SanitizedLegacy = "migration:legacy:redacted"; + private static readonly string[] PendingPlanIdentities = ["pending.contract", "pending.alias"]; + private static readonly string[] BlockedPlanPartitions = ["blocked"]; + + [Fact] + public async Task Legacy_facts_survive_up_down_reapply_and_diagnostics_are_redacted_across_tenants() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres, applyMigrations: false); + await using var context = Platform(database); + await context.GetService().MigrateAsync(PlatformPredecessor); + await InsertLegacyAsync(database, Guid.NewGuid(), SchemaFixture.TenantA, null, processed: false); + await InsertLegacyAsync(database, Guid.NewGuid(), SchemaFixture.TenantB, SchemaFixture.OrgB1, processed: true); + var facts = await FactsAsync(database); + + await context.Database.MigrateAsync(); + context.Database.HasPendingModelChanges().Should().BeFalse(); + (await FactsAsync(database)).Should().Be(facts, "migration must not change attempts, availability or event facts"); + await AssertLegacyDiagnosticsAndRlsAsync(database); + await AssertIndexAsync(database, "ix_outbox_messages_pending", + ["available_after, occurred_at, id", "dead_lettered_at IS NULL", "terminal_reason IS NULL", "attempts < 6", "locked_by IS NULL"]); + await AssertRecoveryIndexAsync(database); + await AssertIndexAsync(database, "ix_outbox_messages_terminal", ["terminal_reason IS NOT NULL"]); + + await context.GetService().MigrateAsync(PlatformPredecessor); + (await FactsAsync(database)).Should().Be(facts); + await using (var owner = await PostgresFixture.OpenAsync(database.MigrationConnectionString)) + { + (await SchemaQueries.CountAsync(owner, """ + SELECT count(*) FROM information_schema.columns + WHERE table_name = 'outbox_messages' + AND column_name IN ('locked_by','locked_until','terminal_reason','dead_lettered_at') + """)).Should().Be(0); + (await SchemaQueries.CountAsync(owner, """ + SELECT count(*) FROM pg_constraint + WHERE conrelid = 'outbox_messages'::regclass AND conname LIKE 'ck_outbox_messages_%' + """)).Should().Be(0); + } + + await context.Database.MigrateAsync(); + (await FactsAsync(database)).Should().Be(facts); + await AssertLegacyDiagnosticsAndRlsAsync(database); + } + + [Theory] + [InlineData(true)] + [InlineData(false)] + public async Task Recovery_index_covers_expired_and_exhausted_selection_without_scanning_pending_backlog(bool expiredLease) + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres, applyMigrations: false); + await using var context = Platform(database); + await context.GetService().MigrateAsync(DispatchPredecessor); + var target = Guid.Parse("10000000-0000-0000-0000-000000000002"); + await SeedRecoveryPlanAsync(database, target, expiredLease); + await using var dispatcher = await PostgresFixture.OpenAsync(database.OutboxConnectionString); + + // The applied predecessor is the negative control: its leased-only + // predicate cannot cover the exhausted branch and requires a full scan. + using (var previous = await ExplainRecoveryAsync(dispatcher)) + { + var nodes = PlanNodes(previous.RootElement[0].GetProperty("Plan")).ToArray(); + IndexNames(nodes).Should().NotContain("ix_outbox_messages_recovery"); + nodes.Sum(node => PlanRows(node, "Rows Removed by Filter")).Should().BeGreaterThanOrEqualTo(50_000); + } + + await context.Database.MigrateAsync(); + await AssertRecoveryPlanAsync(database, dispatcher, target); + await context.GetService().MigrateAsync(DispatchPredecessor); + await AssertIndexAsync(database, "ix_outbox_messages_recovery", ["locked_until, occurred_at, id", "locked_by IS NOT NULL"]); + await context.Database.MigrateAsync(); + await AssertRecoveryPlanAsync(database, dispatcher, target); + } + + [Theory] + [InlineData("id", "00000000-0000-0000-0000-000000000000")] + [InlineData("tenant", "00000000-0000-0000-0000-000000000000")] + [InlineData("tenant", "00000000-0000-7000-8000-000000000002")] + [InlineData("organization", "00000000-0000-0000-0000-000000000000")] + public async Task Invalid_legacy_identity_refuses_migration_without_repair(string member, string invalid) + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres, applyMigrations: false); + await using var context = Platform(database); + await context.GetService().MigrateAsync(PlatformPredecessor); + await InsertLegacyAsync(database, + member == "id" ? Guid.Parse(invalid) : Guid.NewGuid(), + member == "tenant" ? Guid.Parse(invalid) : SchemaFixture.TenantA, + member == "organization" ? Guid.Parse(invalid) : null, + processed: false); + var facts = await FactsAsync(database); + var apply = async () => await context.Database.MigrateAsync(); + var error = (await apply.Should().ThrowAsync()).Which; + error.ConstraintName.Should().Be("ck_outbox_messages_identity"); + (await FactsAsync(database)).Should().Be(facts); + (await context.Database.GetAppliedMigrationsAsync()).Last().Should().Be(PlatformPredecessor); + await using var owner = await PostgresFixture.OpenAsync(database.MigrationConnectionString); + await using var check = new NpgsqlCommand("SELECT relforcerowsecurity FROM pg_class WHERE oid = 'outbox_messages'::regclass", (NpgsqlConnection)owner); + (await check.ExecuteScalarAsync()).Should().Be(true, "the failed transactional backfill must restore FORCE RLS"); + await using var outbox = await PostgresFixture.OpenAsync(database.OutboxConnectionString); + (await SchemaQueries.ReadStringsAsync(outbox, "SELECT last_error FROM outbox_messages")) + .Should().ContainSingle().Which.Should().StartWith("private legacy"); + } + + [Theory] + [InlineData("last_error = repeat('é',128)", null)] + [InlineData("last_error = repeat('é',129)", "ck_outbox_messages_last_error_size")] + [InlineData("locked_by = 'token'", "ck_outbox_messages_lease")] + [InlineData("locked_by = 'token', locked_until = now() + interval '30 seconds'", null)] + [InlineData("locked_by = '', locked_until = now()", "ck_outbox_messages_lease")] + [InlineData("terminal_reason = 'unknown_type'", null)] + [InlineData("terminal_reason = 'not_declared'", "ck_outbox_messages_terminal_reason")] + [InlineData("dead_lettered_at = now()", "ck_outbox_messages_state")] + [InlineData("dead_lettered_at = now(), terminal_reason = 'retry_exhausted'", null)] + [InlineData("processed_at = now(), terminal_reason = 'unknown_type'", "ck_outbox_messages_state")] + [InlineData("attempts = -1", "ck_outbox_messages_attempts")] + [InlineData("attempts = 7", null)] + public async Task Dispatcher_state_checks_have_accepting_and_refusing_controls(string assignment, string? constraint) + { + // Literal theory statements are test-owned, never interpolated caller input. + // Legacy exhausted counts above six remain facts; normal claiming enforces + // the six-attempt ceiling, rather than a migration rewriting that history. + await using var connection = await PostgresFixture.OpenAsync(schema.Postgres.OutboxConnectionString); + await using var transaction = await connection.BeginTransactionAsync(); + await using var update = new NpgsqlCommand( + $"UPDATE outbox_messages SET {assignment} WHERE tenant_id = @tenant", + (NpgsqlConnection)connection, (NpgsqlTransaction)transaction); + update.Parameters.AddWithValue("tenant", SchemaFixture.TenantA); + var act = async () => (await update.ExecuteNonQueryAsync()).Should().BeGreaterThan(0, "a CHECK proof must touch a real row"); + if (constraint is null) + { + await act.Should().NotThrowAsync(); + } + else + { + (await act.Should().ThrowAsync()).Which.ConstraintName.Should().Be(constraint); + } + await transaction.RollbackAsync(); + } + + [Theory] + [InlineData("SELECT * FROM audit_log")] + [InlineData("UPDATE audit_log SET outcome = 'success'")] + [InlineData("DELETE FROM audit_log")] + [InlineData("SELECT * FROM tenants")] + [InlineData("UPDATE outbox_messages SET type = 'other'")] + [InlineData("UPDATE outbox_messages SET payload = '{}'::jsonb")] + [InlineData("DELETE FROM outbox_messages")] + public async Task Outbox_role_has_no_audit_read_mutation_or_business_access(string sql) + { + await using var connection = await PostgresFixture.OpenAsync(schema.Postgres.OutboxConnectionString); + await using var transaction = await connection.BeginTransactionAsync(); + await using var command = new NpgsqlCommand(sql, (NpgsqlConnection)connection, (NpgsqlTransaction)transaction); + var act = async () => await command.ExecuteNonQueryAsync(); + (await act.Should().ThrowAsync()).Which.SqlState.Should().Be(PostgresErrorCodes.InsufficientPrivilege); + await transaction.RollbackAsync(); + } + + [Fact] + public async Task Audit_grant_has_its_own_chain_and_is_removed_by_down_then_reapplied() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var context = new AuditDbContext(new DbContextOptionsBuilder() + .UseNpgsql(database.MigrationConnectionString, + provider => provider.MigrationsHistoryTable(AuditDbContextFactory.HistoryTable)).Options, + StaticTenantContextAccessor.Unresolved); + context.Database.HasPendingModelChanges().Should().BeFalse(); + await AssertAuditInsertGrantAsync(database, expected: true); + await context.GetService().MigrateAsync(AuditPredecessor); + await AssertAuditInsertGrantAsync(database, expected: false); + await context.Database.MigrateAsync(); + await AssertAuditInsertGrantAsync(database, expected: true); + } + + private static PlatformDbContext Platform(DisposableSchemaDatabase database) => + new(new DbContextOptionsBuilder().UseNpgsql(database.MigrationConnectionString, + provider => provider.MigrationsHistoryTable(PlatformDbContextFactory.HistoryTable)).Options); + + private static async Task InsertLegacyAsync(DisposableSchemaDatabase database, Guid id, Guid tenant, Guid? organization, bool processed) + { + await using var app = await PostgresFixture.OpenAsync(database.AppConnectionString); + await using var transaction = await app.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(app); + await SchemaQueries.SetTenantAsync(app, transaction, tenant); + await SchemaQueries.ExecuteAsync(app, transaction, """ + INSERT INTO outbox_messages + (id, tenant_id, organization_id, correlation_id, type, topic, partition_key, + payload, metadata, attempts, processed_at, available_after, last_error) + VALUES (@id, @tenant, @organization, 'bad-legacy-trace', 'legacy-contract', + 'learnstack.tenancy.tenant', @partition, '{"legacy":true}', '{"legacy":true}', + 7, CASE WHEN @processed THEN now() ELSE NULL END, now() + interval '5 minutes', + 'private legacy ' || repeat('é',1000)) + """, ("id", id), ("tenant", tenant), ("organization", organization.HasValue ? organization.Value : DBNull.Value), + ("partition", tenant.ToString()), ("processed", processed)); + await transaction.CommitAsync(); + } + + private static async Task FactsAsync(DisposableSchemaDatabase database) + { + await using var outbox = await PostgresFixture.OpenAsync(database.OutboxConnectionString); + await using var command = new NpgsqlCommand(""" + SELECT jsonb_agg(to_jsonb(o) - 'last_error' - 'locked_by' - 'locked_until' + - 'terminal_reason' - 'dead_lettered_at' ORDER BY id)::text FROM outbox_messages o + """, (NpgsqlConnection)outbox); + return (string)(await command.ExecuteScalarAsync())!; + } + + private static async Task AssertLegacyDiagnosticsAndRlsAsync(DisposableSchemaDatabase database) + { + await using var outbox = await PostgresFixture.OpenAsync(database.OutboxConnectionString); + (await SchemaQueries.ReadStringsAsync(outbox, "SELECT last_error FROM outbox_messages ORDER BY id")) + .Should().Equal(SanitizedLegacy, SanitizedLegacy); + await using var app = await PostgresFixture.OpenAsync(database.AppConnectionString); + (await SchemaQueries.CountAsync(app, "SELECT count(*) FROM outbox_messages")).Should().Be(0, + "unresolved app context must remain fail-closed after the owner backfill"); + await using var owner = await PostgresFixture.OpenAsync(database.MigrationConnectionString); + await using var command = new NpgsqlCommand("SELECT relrowsecurity AND relforcerowsecurity FROM pg_class WHERE oid = 'outbox_messages'::regclass", (NpgsqlConnection)owner); + (await command.ExecuteScalarAsync()).Should().Be(true); + } + + private static async Task AssertIndexAsync(DisposableSchemaDatabase database, string name, string[] fragments) + { + await using var owner = await PostgresFixture.OpenAsync(database.MigrationConnectionString); + await using var command = new NpgsqlCommand("SELECT indexdef FROM pg_indexes WHERE schemaname = 'public' AND indexname = @name", (NpgsqlConnection)owner); + command.Parameters.AddWithValue("name", name); + var definition = (string)(await command.ExecuteScalarAsync())!; + foreach (var fragment in fragments) + { + definition.Should().Contain(fragment); + } + } + + private static Task AssertRecoveryIndexAsync(DisposableSchemaDatabase database) => + AssertIndexAsync(database, "ix_outbox_messages_recovery", + ["available_after, occurred_at, id", "processed_at IS NULL", "dead_lettered_at IS NULL", + "terminal_reason IS NULL", "(locked_by IS NOT NULL) OR (attempts >= 6)"]); + + private static async Task SeedRecoveryPlanAsync(DisposableSchemaDatabase database, Guid target, bool expiredLease) + { + await using (var app = await PostgresFixture.OpenAsync(database.AppConnectionString)) + await using (var transaction = await app.BeginTransactionAsync()) + { + await SchemaQueries.AssertApplicationRoleAsync(app); + await SchemaQueries.SetTenantAsync(app, transaction, SchemaFixture.TenantA); + await SchemaQueries.ExecuteAsync(app, transaction, """ + INSERT INTO outbox_messages + (id, tenant_id, correlation_id, type, topic, partition_key, payload, available_after) + SELECT uuidv7(), @tenant, 'plan-proof', 'pending.contract', 'test.topic', 'allowed', '{}', '2000-01-01' + FROM generate_series(1, 50000); + INSERT INTO outbox_messages + (id, tenant_id, correlation_id, type, topic, partition_key, payload, + available_after, occurred_at, attempts, locked_by, locked_until) + SELECT candidate.id, @tenant, 'plan-proof', candidate.type, 'test.topic', candidate.partition_key, '{}', + candidate.available_after, candidate.occurred_at, CASE WHEN @expired THEN 1 ELSE 6 END, + CASE WHEN @expired THEN 'expired-token' ELSE NULL END, + CASE WHEN @expired THEN clock_timestamp() - interval '1 hour' ELSE NULL END + FROM (VALUES + (@target, 'active.contract', 'allowed', '2001-01-01'::timestamptz, '2001-01-01'::timestamptz), + ('10000000-0000-0000-0000-000000000003'::uuid, 'active.contract', 'allowed', '2001-01-01'::timestamptz, '2001-01-01'::timestamptz), + ('10000000-0000-0000-0000-000000000001'::uuid, 'active.contract', 'allowed', '2001-01-01'::timestamptz, '2002-01-01'::timestamptz), + (uuidv7(), 'active.contract', 'allowed', '2002-01-01'::timestamptz, '2000-01-01'::timestamptz), + (uuidv7(), 'pending.contract', 'allowed', '2000-01-01'::timestamptz, '2000-01-01'::timestamptz), + (uuidv7(), 'pending.alias', 'allowed', '2000-01-01'::timestamptz, '2000-01-01'::timestamptz), + (uuidv7(), 'active.contract', 'blocked', '2000-01-01'::timestamptz, '2000-01-01'::timestamptz) + ) AS candidate(id, type, partition_key, available_after, occurred_at); + INSERT INTO outbox_messages + (id, tenant_id, correlation_id, type, topic, partition_key, payload, + available_after, locked_by, locked_until) + VALUES (uuidv7(), @tenant, 'plan-proof', 'active.contract', 'test.topic', 'allowed', '{}', + '2000-01-01', 'live-token', clock_timestamp() + interval '1 hour'); + """, ("tenant", SchemaFixture.TenantA), ("target", target), ("expired", expiredLease)); + await transaction.CommitAsync(); + } + await using var owner = await PostgresFixture.OpenAsync(database.MigrationConnectionString); + await SchemaQueries.ExecuteAsync(owner, null, "ANALYZE outbox_messages"); + } + + private static NpgsqlCommand RecoverySelection(DbConnection dispatcher) + { + // Read the production predicates so an eligibility/admission regression + // cannot leave this proof explaining an independently copied WHERE clause. + static string Predicate(string name) => (string)typeof(OutboxDispatchStore) + .GetField(name, BindingFlags.Static | BindingFlags.NonPublic)!.GetRawConstantValue()!; + var command = new NpgsqlCommand($""" + SELECT id FROM outbox_messages WHERE {Predicate("Recovery")}{Predicate("Admitted")} + ORDER BY available_after, occurred_at, id FOR UPDATE SKIP LOCKED LIMIT 1 + """, (NpgsqlConnection)dispatcher); + command.Parameters.AddWithValue("pending", PendingPlanIdentities); + command.Parameters.AddWithValue("blocked", BlockedPlanPartitions); + return command; + } + + private static async Task ExplainRecoveryAsync(DbConnection dispatcher) + { + await using var command = RecoverySelection(dispatcher); + command.CommandText = "EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) " + command.CommandText; + return JsonDocument.Parse((string)(await command.ExecuteScalarAsync())!); + } + + private static async Task AssertRecoveryPlanAsync(DisposableSchemaDatabase database, DbConnection dispatcher, Guid target) + { + await AssertRecoveryIndexAsync(database); + using var plan = await ExplainRecoveryAsync(dispatcher); + var nodes = PlanNodes(plan.RootElement[0].GetProperty("Plan")).ToArray(); + IndexNames(nodes).Should().ContainSingle(name => name == "ix_outbox_messages_recovery"); + nodes.Should().NotContain(node => node.GetProperty("Node Type").GetString() == "Sort" + || node.GetProperty("Node Type").GetString() == "Seq Scan"); + nodes.Sum(node => PlanRows(node, "Rows Removed by Filter")).Should().BeLessThanOrEqualTo(4, + "only the two Pending identities, blocked partition and live lease precede the first eligible row"); + await using var selection = RecoverySelection(dispatcher); + (await selection.ExecuteScalarAsync()).Should().Be(target, "recovery keeps the global three-column order and admission exclusions"); + } + + private static IEnumerable PlanNodes(JsonElement node) + { + yield return node; + if (!node.TryGetProperty("Plans", out var children)) yield break; + foreach (var child in children.EnumerateArray()) + foreach (var descendant in PlanNodes(child)) + yield return descendant; + } + + private static double PlanRows(JsonElement node, string property) => + node.TryGetProperty(property, out var value) ? value.GetDouble() : 0; + + private static IEnumerable IndexNames(IEnumerable nodes) => + nodes.Where(node => node.TryGetProperty("Index Name", out _)) + .Select(node => node.GetProperty("Index Name").GetString()); + + private static async Task AssertAuditInsertGrantAsync(DisposableSchemaDatabase database, bool expected) + { + await using var owner = await PostgresFixture.OpenAsync(database.MigrationConnectionString); + await using var command = new NpgsqlCommand("SELECT has_table_privilege('learnstack_outbox_admin', 'audit_log', 'INSERT')", (NpgsqlConnection)owner); + (await command.ExecuteScalarAsync()).Should().Be(expected); + } +} diff --git a/backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatchStoreTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatchStoreTests.cs new file mode 100644 index 00000000..c8629cbe --- /dev/null +++ b/backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatchStoreTests.cs @@ -0,0 +1,1186 @@ +using System.Data.Common; +using System.Text.Json; +using System.Text.Json.Nodes; +using FluentAssertions; +using LearnStack.Infrastructure.Audit; +using LearnStack.Infrastructure.Messaging; +using LearnStack.Modules.Tenancy.Application.Contracts.IntegrationEvents; +using LearnStack.Modules.Tenancy.Application.Contracts.Tenant; +using LearnStack.SharedKernel.Audit; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Messaging; +using LearnStack.SharedKernel.Persistence; +using LearnStack.SharedKernel.Time; +using LearnStack.Tools.Seeder; +using MediatR; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using NpgsqlTypes; +using Xunit; + +namespace LearnStack.Tests.Integration.Database; + +/// Real committed leases, fenced transitions and witness-bound terminal audit. +[Trait(RequiresDocker.Key, RequiresDocker.Value)] +[Collection(SharedSchema.Name)] +public sealed class OutboxDispatchStoreTests(SchemaFixture schema) +{ + private const string Traceparent = "00-11111111111111111111111111111111-2222222222222222-01"; + private const string AttemptTraceparent = "00-33333333333333333333333333333333-4444444444444444-01"; + private const string EarlierTenantIdentity = "LearnStack.Previous.TenantCreatedIntegrationEventV1, LearnStack.Previous.Contracts"; + private static readonly DateTimeOffset Occurred = new(2026, 10, 1, 12, 0, 0, TimeSpan.Zero); + private static readonly IReadOnlyCollection NoBlockedPartitions = Array.Empty(); + + [Fact] + public async Task Two_independent_stores_contend_for_one_committed_live_lease() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + var first = test.Store(); + var second = test.Store(); + + var claims = await Task.WhenAll(first.TryClaimAsync(NoBlockedPartitions), second.TryClaimAsync(NoBlockedPartitions)); + + var claim = claims.OfType().Should().ContainSingle().Which; + claim.Id.Should().Be(id); + claim.Kind.Should().Be(OutboxClaimKind.Normal); + claim.Attempts.Should().Be(1); + Guid.ParseExact(claim.Token, "D").Should().NotBeEmpty(); + var durable = await test.ReadAsync(id); + durable.Token.Should().Be(claim.Token); + durable.Attempts.Should().Be(1); + durable.LockedUntil.Should().Be(claim.LockedUntil); + // Reading the winner must not release the database lease. This second poll + // catches a release-on-read implementation even if the first race serialized. + (await first.TryClaimAsync(NoBlockedPartitions)).Should().BeNull(); + (await second.TryClaimAsync(NoBlockedPartitions)).Should().BeNull(); + (await test.ReadAsync(id)).Should().Be(durable); + } + + [Fact] + public async Task Concurrent_stores_claim_distinct_rows_without_owning_a_batch() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var ids = new[] { await test.InsertAsync(), await test.InsertAsync() }; + + var claims = await Task.WhenAll(test.Store().TryClaimAsync(NoBlockedPartitions), test.Store().TryClaimAsync(NoBlockedPartitions)); + + claims.Should().OnlyContain(claim => claim != null); + claims.Select(claim => claim!.Id).Should().BeEquivalentTo(ids); + claims.Select(claim => claim!.Token).Should().OnlyHaveUniqueItems(); + (await test.Store().TryClaimAsync(NoBlockedPartitions)).Should().BeNull(); + } + + [Fact] + public async Task A_live_success_commits_processed_state_and_clears_its_lease_once() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + var store = test.Store(); + var claim = (await store.TryClaimAsync(NoBlockedPartitions))!; + + (await store.CompleteAsync(claim)).Should().BeTrue(); + + var row = await test.ReadAsync(id); + row.ProcessedAt.Should().NotBeNull(); + row.DeadLetteredAt.Should().BeNull(); + row.TerminalReason.Should().BeNull(); + row.Token.Should().BeNull(); + row.LockedUntil.Should().BeNull(); + row.Attempts.Should().Be(1); + (await store.CompleteAsync(claim)).Should().BeFalse(); + (await store.FailAsync(claim, OutboxFailureReason.PublishFailed)).Should().BeFalse(); + (await store.TryClaimAsync(NoBlockedPartitions)).Should().BeNull(); + (await test.ReadAsync(id)).Should().Be(row); + (await test.AuditCountAsync()).Should().Be(0); + } + + [Theory] + [InlineData("complete", OutboxClaimKind.Recovery)] + [InlineData("complete", OutboxClaimKind.Terminal)] + [InlineData("fail", OutboxClaimKind.Terminal)] + [InlineData("terminalize", OutboxClaimKind.Normal)] + [InlineData("terminalize", OutboxClaimKind.Recovery)] + public async Task Wrong_claim_kind_refuses_before_changing_the_row_or_audit(string operation, OutboxClaimKind kind) + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + var store = test.Store(); + var claim = (await store.TryClaimAsync(NoBlockedPartitions))! with { Kind = kind }; + var before = await test.ReadAsync(id); + Func act = () => operation switch + { + "complete" => store.CompleteAsync(claim), + "fail" => store.FailAsync(claim, OutboxFailureReason.PublishFailed), + "terminalize" => store.TryTerminalizeAsync(claim, AttemptTraceparent), + _ => throw new ArgumentOutOfRangeException(nameof(operation)), + }; + + await act.Should().ThrowAsync(); + + (await test.ReadAsync(id)).Should().Be(before); + (await test.AuditCountAsync()).Should().Be(0); + } + + [Theory] + [InlineData("success")] + [InlineData("failure")] + [InlineData("terminal-request")] + public async Task Expiry_alone_refuses_every_transition_without_changing_state(string transition) + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + var store = test.Store(); + var claim = (await store.TryClaimAsync(NoBlockedPartitions))!; + await test.ExpireAsync(id); + var before = await test.ReadAsync(id); + + (await TransitionAsync(store, claim, transition)).Should().BeFalse(); + + (await test.ReadAsync(id)).Should().Be(before); + (await test.AuditCountAsync()).Should().Be(0); + (await store.TryClaimAsync(NoBlockedPartitions)).Should().BeNull("an expired claim requires recovery before publication"); + } + + [Theory] + [InlineData("success")] + [InlineData("failure")] + [InlineData("terminal-request")] + public async Task Recovery_by_the_same_store_replaces_the_token_and_fences_every_stale_transition(string transition) + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + var store = test.Store(); + var stale = (await store.TryClaimAsync(NoBlockedPartitions))!; + await test.ExpireAsync(id); + + var recovery = (await store.TryAcquireRecoveryAsync(NoBlockedPartitions))!; + + recovery.Id.Should().Be(id); + recovery.Kind.Should().Be(OutboxClaimKind.Recovery); + recovery.Token.Should().NotBe(stale.Token); + recovery.Attempts.Should().Be(stale.Attempts); + var before = await test.ReadAsync(id); + (await TransitionAsync(store, stale, transition)).Should().BeFalse(); + (await test.ReadAsync(id)).Should().Be(before); + (await test.AuditCountAsync()).Should().Be(0); + (await store.FailAsync(recovery, OutboxFailureReason.UnknownOutcome)).Should().BeTrue(); + await test.MakeDueAsync(id); + var fresh = (await store.TryClaimAsync(NoBlockedPartitions))!; + fresh.Token.Should().NotBe(stale.Token).And.NotBe(recovery.Token); + fresh.Attempts.Should().Be(2); + (await store.CompleteAsync(fresh)).Should().BeTrue(); + } + + [Fact] + public async Task Expired_terminal_acquisition_loses_its_fence_before_and_after_fresh_terminal_reacquisition() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + var store = test.Store(); + var normal = (await store.TryClaimAsync(NoBlockedPartitions))!; + (await store.RequestTerminalAsync(normal, OutboxTerminalReason.InvalidEnvelope)).Should().BeTrue(); + var stale = (await store.TryClaimTerminalAsync(NoBlockedPartitions))!; + await test.ExpireAsync(id); + var expired = await test.ReadAsync(id); + + (await store.TryTerminalizeAsync(stale, AttemptTraceparent)).Should().BeFalse(); + (await test.ReadAsync(id)).Should().Be(expired); + var fresh = (await store.TryClaimTerminalAsync(NoBlockedPartitions))!; + fresh.Token.Should().NotBe(stale.Token); + fresh.Attempts.Should().Be(1); + var reacquired = await test.ReadAsync(id); + (await store.TryTerminalizeAsync(stale, AttemptTraceparent)).Should().BeFalse(); + (await test.ReadAsync(id)).Should().Be(reacquired); + (await test.AuditCountAsync()).Should().Be(0); + (await store.TryTerminalizeAsync(fresh, AttemptTraceparent)).Should().BeTrue(); + (await test.AuditCountAsync()).Should().Be(1); + } + + [Theory] + [InlineData(1, 1)] + [InlineData(2, 5)] + [InlineData(3, 30)] + [InlineData(4, 300)] + [InlineData(5, 3600)] + public async Task Committed_failures_schedule_the_five_database_time_backoffs_without_sleeping(int attempt, int seconds) + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(attempts: attempt - 1); + // A caller clock decades away must not decide expiry or availability. + var store = test.Store(clock: new FixedClock(Occurred.AddYears(50))); + var claim = (await store.TryClaimAsync(NoBlockedPartitions))!; + claim.Attempts.Should().Be(attempt); + var before = await test.DatabaseNowAsync(); + + (await store.FailAsync(claim, OutboxFailureReason.PublishFailed)).Should().BeTrue(); + + var after = await test.DatabaseNowAsync(); + var row = await test.ReadAsync(id); + AssertBackoff(row, attempt, seconds, before, after); + (await store.TryClaimAsync(NoBlockedPartitions)).Should().BeNull(); + await test.MakeDueAsync(id); + (await store.TryClaimAsync(NoBlockedPartitions))!.Attempts.Should().Be(attempt + 1); + } + + [Theory] + [InlineData(1, 1)] + [InlineData(2, 5)] + [InlineData(3, 30)] + [InlineData(4, 300)] + [InlineData(5, 3600)] + public async Task Expired_claims_recover_into_backoff_without_spending_another_attempt(int attempt, int seconds) + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(attempts: attempt - 1); + var store = test.Store(); + var crashed = (await store.TryClaimAsync(NoBlockedPartitions))!; + await test.ExpireAsync(id); + (await store.TryClaimAsync(NoBlockedPartitions)).Should().BeNull(); + var recovery = (await store.TryAcquireRecoveryAsync(NoBlockedPartitions))!; + recovery.Attempts.Should().Be(attempt); + recovery.Token.Should().NotBe(crashed.Token); + var before = await test.DatabaseNowAsync(); + + (await store.FailAsync(recovery, OutboxFailureReason.UnknownOutcome)).Should().BeTrue(); + + AssertBackoff(await test.ReadAsync(id), attempt, seconds, before, await test.DatabaseNowAsync()); + (await store.TryAcquireRecoveryAsync(NoBlockedPartitions)).Should().BeNull(); + (await store.TryClaimAsync(NoBlockedPartitions)).Should().BeNull(); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Failed_or_crashed_sixth_claim_requests_terminalization_and_never_claims_a_seventh(bool crash) + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + var store = test.Store(); + for (var attempt = 1; attempt <= 5; attempt++) + { + var earlier = (await store.TryClaimAsync(NoBlockedPartitions))!; + earlier.Attempts.Should().Be(attempt); + (await store.FailAsync(earlier, OutboxFailureReason.PublishFailed)).Should().BeTrue(); + await test.MakeDueAsync(id); + } + var sixth = (await store.TryClaimAsync(NoBlockedPartitions))!; + sixth.Attempts.Should().Be(6); + if (crash) + { + await test.ExpireAsync(id); + (await store.TryClaimAsync(NoBlockedPartitions)).Should().BeNull(); + sixth = (await store.TryAcquireRecoveryAsync(NoBlockedPartitions))!; + sixth.Kind.Should().Be(OutboxClaimKind.Recovery); + } + + (await store.FailAsync(sixth, crash ? OutboxFailureReason.UnknownOutcome : OutboxFailureReason.PublishTimeout)).Should().BeTrue(); + + var requested = await test.ReadAsync(id); + requested.Attempts.Should().Be(6); + requested.TerminalReason.Should().Be("retry_exhausted"); + requested.DeadLetteredAt.Should().BeNull(); + requested.Token.Should().BeNull(); + await test.MakeDueAsync(id); + (await store.TryClaimAsync(NoBlockedPartitions)).Should().BeNull(); + var terminal = (await store.TryClaimTerminalAsync(NoBlockedPartitions))!; + terminal.Kind.Should().Be(OutboxClaimKind.Terminal); + terminal.Attempts.Should().Be(6); + (await store.TryTerminalizeAsync(terminal, AttemptTraceparent)).Should().BeTrue(); + (await store.TryClaimAsync(NoBlockedPartitions)).Should().BeNull(); + (await test.ReadAsync(id)).Attempts.Should().Be(6); + (await test.AuditCountAsync()).Should().Be(1); + } + + [Fact] + public async Task Exhausted_unleased_legacy_work_is_terminal_only_without_an_extra_attempt() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(attempts: 8); + var store = test.Store(); + + (await store.TryClaimAsync(NoBlockedPartitions)).Should().BeNull(); + var recovery = (await store.TryAcquireRecoveryAsync(NoBlockedPartitions))!; + recovery.Attempts.Should().Be(8); + (await store.FailAsync(recovery, OutboxFailureReason.UnknownOutcome)).Should().BeTrue(); + var terminal = (await store.TryClaimTerminalAsync(NoBlockedPartitions))!; + + terminal.Id.Should().Be(id); + terminal.Attempts.Should().Be(8); + (await store.TryTerminalizeAsync(terminal, AttemptTraceparent)).Should().BeTrue(); + (await test.ReadAsync(id)).TerminalReason.Should().Be("retry_exhausted"); + (await test.ReadAsync(id)).Attempts.Should().Be(8); + } + + [Theory] + [InlineData(OutboxTerminalReason.UnknownType, "unknown_type")] + [InlineData(OutboxTerminalReason.RetiredType, "retired_type")] + [InlineData(OutboxTerminalReason.InvalidEnvelope, "invalid_envelope")] + public async Task A_terminal_request_before_six_attempts_cannot_be_claimed_for_publication(OutboxTerminalReason reason, string storedReason) + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + var store = test.Store(); + var claim = (await store.TryClaimAsync(NoBlockedPartitions))!; + + (await store.RequestTerminalAsync(claim, reason)).Should().BeTrue(); + + var row = await test.ReadAsync(id); + row.TerminalReason.Should().Be(storedReason); + row.Attempts.Should().Be(1); + row.ProcessedAt.Should().BeNull(); + row.DeadLetteredAt.Should().BeNull(); + row.Token.Should().BeNull(); + row.LockedUntil.Should().BeNull(); + await test.MakeDueAsync(id); + (await store.TryClaimAsync(NoBlockedPartitions)).Should().BeNull(); + (await store.TryAcquireRecoveryAsync(NoBlockedPartitions)).Should().BeNull(); + var terminal = (await store.TryClaimTerminalAsync(NoBlockedPartitions))!; + terminal.Attempts.Should().Be(1); + terminal.Token.Should().NotBe(claim.Token); + (await store.TryTerminalizeAsync(terminal, AttemptTraceparent)).Should().BeTrue(); + } + + [Fact] + public async Task Refused_audit_rolls_back_the_terminal_update_and_retains_the_request_for_audited_retry() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(organization: SchemaFixture.OrgA1); + var store = test.Store(); + var normal = (await store.TryClaimAsync(NoBlockedPartitions))!; + (await store.RequestTerminalAsync(normal, OutboxTerminalReason.InvalidEnvelope)).Should().BeTrue(); + var terminal = (await store.TryClaimTerminalAsync(NoBlockedPartitions))!; + var before = await test.ReadAsync(id); + + try + { + await test.AuditInsertGrantAsync(grant: false); + var terminalize = () => store.TryTerminalizeAsync(terminal, AttemptTraceparent); + await terminalize.Should().ThrowAsync(); + (await test.ReadAsync(id)).Should().Be(before, "the failed INSERT and terminal UPDATE share one transaction"); + (await test.AuditCountAsync()).Should().Be(0); + var snapshot = await store.ReadSnapshotAsync(NoBlockedPartitions); + snapshot.PendingCount.Should().Be(1); + snapshot.DeferredCount.Should().Be(0); + snapshot.OldestEligibleAgeSeconds.Should().Be(0); + await test.MakeDueAsync(id); + (await store.TryClaimAsync(NoBlockedPartitions)).Should().BeNull(); + await test.ExpireAsync(id); + (await store.TryClaimAsync(NoBlockedPartitions)).Should().BeNull("terminal intent survives time advancing"); + } + finally + { + await test.AuditInsertGrantAsync(grant: true); + } + + var retry = (await store.TryClaimTerminalAsync(NoBlockedPartitions))!; + retry.Token.Should().NotBe(terminal.Token); + retry.Attempts.Should().Be(1); + (await store.TryTerminalizeAsync(retry, AttemptTraceparent)).Should().BeTrue(); + (await test.AuditCountAsync()).Should().Be(1); + (await test.ReadAsync(id)).DeadLetteredAt.Should().NotBeNull(); + (await store.TryTerminalizeAsync(terminal, AttemptTraceparent)).Should().BeFalse(); + (await store.TryTerminalizeAsync(retry, AttemptTraceparent)).Should().BeFalse(); + (await store.TryClaimTerminalAsync(NoBlockedPartitions)).Should().BeNull(); + (await test.AuditCountAsync()).Should().Be(1, "a committed terminal retry cannot append another audit"); + } + + [Fact] + public async Task A_lost_physical_COMMIT_response_reconciles_durable_terminal_state_without_duplicate_audit() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(organization: SchemaFixture.OrgA1); + var direct = test.Store(); + var normal = (await direct.TryClaimAsync(NoBlockedPartitions))!; + (await direct.RequestTerminalAsync(normal, OutboxTerminalReason.InvalidEnvelope)).Should().BeTrue(); + var terminal = (await direct.TryClaimTerminalAsync(NoBlockedPartitions))!; + // Acquisitions use the direct source. The first explicit COMMIT crossing + // this separate proxy can therefore only be the audited terminal COMMIT. + await using var proxy = new CommitResponseDroppingProxy(test.OutboxConnectionString); + await using var source = OutboxDispatcherDataSource.Build(proxy.ConnectionString); + var uncertain = test.Store(dispatchSource: source); + + (await uncertain.TryTerminalizeAsync(terminal, AttemptTraceparent).WaitAsync(TimeSpan.FromSeconds(15))) + .Should().BeTrue("the fresh reconciliation connection observes the already committed terminal state"); + + proxy.DroppedCommitResponses.Should().Be(1, "a real server COMMIT acknowledgement must have been suppressed"); + proxy.AcceptedConnections.Should().BeGreaterThanOrEqualTo(2, + "the broken commit connection cannot answer the durability reconciliation"); + var row = await test.ReadAsync(id); + row.DeadLetteredAt.Should().NotBeNull(); + row.ProcessedAt.Should().BeNull(); + row.Token.Should().BeNull(); + (await test.AuditCountAsync()).Should().Be(1); + (await uncertain.TryTerminalizeAsync(terminal, AttemptTraceparent)).Should().BeFalse(); + (await uncertain.TryClaimTerminalAsync(NoBlockedPartitions)).Should().BeNull(); + (await test.AuditCountAsync()).Should().Be(1); + proxy.DroppedCommitResponses.Should().Be(1); + } + + [Fact] + public async Task A_lost_COMMIT_response_and_failed_reconciliation_preserve_both_failures_without_duplicate_audit() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(organization: SchemaFixture.OrgA1); + var direct = test.Store(); + var normal = (await direct.TryClaimAsync(NoBlockedPartitions))!; + (await direct.RequestTerminalAsync(normal, OutboxTerminalReason.InvalidEnvelope)).Should().BeTrue(); + var terminal = (await direct.TryClaimTerminalAsync(NoBlockedPartitions))!; + await using var proxy = new CommitResponseDroppingProxy(test.OutboxConnectionString, + rejectReconciliationConnections: true); + await using var source = OutboxDispatcherDataSource.Build(proxy.ConnectionString); + var uncertain = test.Store(dispatchSource: source); + + var attempt = () => uncertain.TryTerminalizeAsync(terminal, AttemptTraceparent) + .WaitAsync(TimeSpan.FromSeconds(15)); + var exception = (await attempt.Should().ThrowAsync()).Which; + + exception.Message.Should().StartWith("The outbox terminal commit outcome remains unknown."); + exception.InnerExceptions.Should().HaveCount(2); + var commitFailure = exception.InnerExceptions[0]; + var reconciliationFailure = exception.InnerExceptions[1]; + commitFailure.Should().BeOfType(); + reconciliationFailure.Should().BeOfType().And.NotBeSameAs(commitFailure); + commitFailure.StackTrace.Should().Contain("NpgsqlTransaction.Commit"); + reconciliationFailure.StackTrace.Should().Contain("NpgsqlDataSource.OpenConnectionAsync") + .And.Contain("IsDurablyTerminalAsync"); + proxy.DroppedCommitResponses.Should().Be(1); + proxy.AcceptedConnections.Should().BeGreaterThanOrEqualTo(2, + "the durability probe must open a fresh connection after the committed session breaks"); + // A separate healthy source establishes the physical outcome even though + // the caller cannot reconcile it through the failed connection path. + var row = await test.ReadAsync(id); + row.DeadLetteredAt.Should().NotBeNull(); + row.ProcessedAt.Should().BeNull(); + row.TerminalReason.Should().Be("invalid_envelope"); + row.Token.Should().BeNull(); + row.LockedUntil.Should().BeNull(); + (await test.AuditCountAsync()).Should().Be(1); + using var metadata = JsonDocument.Parse((await test.ReadAuditAsync()).Metadata); + metadata.RootElement.GetProperty("event_id").GetGuid().Should().Be(id); + (await direct.TryTerminalizeAsync(terminal, AttemptTraceparent)).Should().BeFalse(); + (await direct.TryClaimTerminalAsync(NoBlockedPartitions)).Should().BeNull(); + (await test.ReadAsync(id)).Should().Be(row); + (await test.AuditCountAsync()).Should().Be(1); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Terminal_audit_uses_the_stored_tenant_organization_and_validated_causal_metadata(bool invalidCausalMetadata) + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var causation = invalidCausalMetadata ? Guid.Empty : Guid.CreateVersion7(); + var causalActor = invalidCausalMetadata ? Guid.Empty : SchemaFixture.Actor; + var id = await test.InsertAsync(organization: SchemaFixture.OrgA1, + correlation: invalidCausalMetadata ? "private-invalid-trace" : Traceparent, + causation: causation, actor: causalActor, payload: "{}"); + var store = test.Store(); + var claim = (await store.TryClaimAsync(NoBlockedPartitions))!; + (await store.RequestTerminalAsync(claim, OutboxTerminalReason.InvalidEnvelope)).Should().BeTrue(); + var terminal = (await store.TryClaimTerminalAsync(NoBlockedPartitions))!; + + (await store.TryTerminalizeAsync(terminal, AttemptTraceparent)).Should().BeTrue(); + + var audit = await test.ReadAuditAsync(); + audit.Tenant.Should().Be(SchemaFixture.TenantA); + audit.Organization.Should().Be(SchemaFixture.OrgA1); + audit.Actor.Should().Be(UserId.SystemActor.Value); + audit.Module.Should().Be("platform"); + audit.Operation.Should().Be("platform.outbox.dead_letter"); + audit.Class.Should().Be("Must"); + audit.Type.Should().Be("SecurityEvent"); + audit.Outcome.Should().Be("success"); + audit.Reason.Should().BeNull("the platform-entry reason field is not a terminal descriptor"); + audit.Correlation.Should().Be(invalidCausalMetadata ? AttemptTraceparent : Traceparent); + using var metadata = JsonDocument.Parse(audit.Metadata); + metadata.RootElement.GetProperty("event_id").GetGuid().Should().Be(id); + metadata.RootElement.GetProperty("reason").GetString().Should().Be("invalid_envelope"); + metadata.RootElement.GetProperty("attempts").GetInt32().Should().Be(1); + if (invalidCausalMetadata) + { + metadata.RootElement.TryGetProperty("causation_id", out _).Should().BeFalse(); + metadata.RootElement.TryGetProperty("causal_actor_user_id", out _).Should().BeFalse(); + audit.Metadata.Should().NotContain("private-invalid-trace"); + } + else + { + metadata.RootElement.GetProperty("causation_id").GetGuid().Should().Be(causation); + metadata.RootElement.GetProperty("causal_actor_user_id").GetGuid().Should().Be(causalActor); + } + (await test.ReadAsync(id)).DeadLetteredAt.Should().NotBeNull(); + } + + [Theory] + [InlineData("tenant")] + [InlineData("organization")] + [InlineData("event")] + [InlineData("operation")] + [InlineData("class")] + [InlineData("type")] + [InlineData("transaction")] + public async Task Substituting_a_draft_or_transaction_against_the_actual_UPDATE_witness_rolls_back_both_writes(string substitution) + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(organization: SchemaFixture.OrgA1); + var calls = 0; + var store = test.Store(decorate: inner => new TerminalAuditDecorator(inner, + async (draft, witness, connection, transaction, cancellationToken) => + { + calls++; + witness.EventId.Should().Be(id); + witness.TenantId.Should().Be(SchemaFixture.TenantA); + witness.OrganizationId.Should().Be(SchemaFixture.OrgA1); + // This witness came from the real successful UPDATE: its state is + // visible on that live bypass-role transaction before the INSERT. + await using var truth = new NpgsqlCommand( + "SELECT dead_lettered_at IS NOT NULL FROM outbox_messages WHERE id = @id", + (NpgsqlConnection)connection, (NpgsqlTransaction)transaction); + truth.Parameters.AddWithValue("id", id); + (await truth.ExecuteScalarAsync(cancellationToken)).Should().Be(true); + + if (substitution == "transaction") + { + await using var otherConnection = await test.DispatchSource.OpenConnectionAsync(cancellationToken); + await using var otherTransaction = await otherConnection.BeginTransactionAsync(cancellationToken); + await inner.WriteEventTerminalAsync(draft, witness, otherConnection, otherTransaction, cancellationToken); + return; + } + + var changed = substitution switch + { + "tenant" => draft with { TenantId = TenantId.From(SchemaFixture.TenantB) }, + "organization" => draft with { OrganizationId = OrganizationId.From(SchemaFixture.OrgA2) }, + "event" => draft with { Metadata = ReplaceEventIdentity(draft.Metadata!) }, + "operation" => draft with { Operation = "platform.delivery.dead_letter" }, + "class" => draft with { OperationClass = OperationClass.Should }, + "type" => draft with { OperationType = OperationType.Update }, + _ => throw new ArgumentOutOfRangeException(nameof(substitution)), + }; + await inner.WriteEventTerminalAsync(changed, witness, connection, transaction, cancellationToken); + })); + var normal = (await store.TryClaimAsync(NoBlockedPartitions))!; + (await store.RequestTerminalAsync(normal, OutboxTerminalReason.InvalidEnvelope)).Should().BeTrue(); + var terminal = (await store.TryClaimTerminalAsync(NoBlockedPartitions))!; + var before = await test.ReadAsync(id); + + var attempt = () => store.TryTerminalizeAsync(terminal, AttemptTraceparent); + await attempt.Should().ThrowAsync(); + + calls.Should().Be(1, "the refusal must exercise the real audit seam after a successful terminal UPDATE"); + (await test.ReadAsync(id)).Should().Be(before); + (await test.AuditCountAsync()).Should().Be(0); + // The same still-live terminal claim succeeds with the real unmodified seam. + (await test.Store().TryTerminalizeAsync(terminal, AttemptTraceparent)).Should().BeTrue(); + (await test.AuditCountAsync()).Should().Be(1); + } + + [Fact] + public async Task A_witness_from_a_committed_terminal_transaction_cannot_be_reused_on_a_later_real_transition() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + EventTerminalTransition? previous = null; + var calls = 0; + var store = test.Store(decorate: inner => new TerminalAuditDecorator(inner, + async (draft, witness, connection, transaction, cancellationToken) => + { + calls++; + var supplied = previous ?? witness; + await inner.WriteEventTerminalAsync(draft, supplied, connection, transaction, cancellationToken); + previous = witness; + })); + var firstId = await test.InsertAsync(); + var first = (await store.TryClaimAsync(NoBlockedPartitions))!; + (await store.RequestTerminalAsync(first, OutboxTerminalReason.InvalidEnvelope)).Should().BeTrue(); + var firstTerminal = (await store.TryClaimTerminalAsync(NoBlockedPartitions))!; + (await store.TryTerminalizeAsync(firstTerminal, AttemptTraceparent)).Should().BeTrue(); + previous.Should().NotBeNull(); + previous!.EventId.Should().Be(firstId); + var secondId = await test.InsertAsync(); + var second = (await store.TryClaimAsync(NoBlockedPartitions))!; + (await store.RequestTerminalAsync(second, OutboxTerminalReason.InvalidEnvelope)).Should().BeTrue(); + var secondTerminal = (await store.TryClaimTerminalAsync(NoBlockedPartitions))!; + var before = await test.ReadAsync(secondId); + + var attempt = () => store.TryTerminalizeAsync(secondTerminal, AttemptTraceparent); + await attempt.Should().ThrowAsync(); + + calls.Should().Be(2); + (await test.ReadAsync(secondId)).Should().Be(before); + (await test.AuditCountAsync()).Should().Be(1); + (await test.Store().TryTerminalizeAsync(secondTerminal, AttemptTraceparent)).Should().BeTrue(); + (await test.AuditCountAsync()).Should().Be(2); + } + + [Theory] + [InlineData(false, false)] + [InlineData(true, false)] + [InlineData(false, true)] + [InlineData(true, true)] + public async Task Real_TenantCreated_contracts_and_aliases_are_held_before_any_payload_validation(bool alias, bool malformed) + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres, seedTenant: false); + await using (var scope = test.Provider.CreateAsyncScope()) + { + var result = await scope.ServiceProvider.GetRequiredService().Send(new ProvisionTenantCommand( + TenantId.From(SchemaFixture.TenantA), "dispatch-probe", "Dispatch probe", + OrganizationId.From(SchemaFixture.OrgA1), "hq", "HQ")); + result.IsSuccess.Should().BeTrue(); + } + var produced = await test.SingleIdAsync(); + var fact = new TenantCreatedIntegrationEventV1 + { + EventId = Guid.CreateVersion7(), + TenantId = SchemaFixture.TenantA, + OccurredAt = Occurred, + }; + var id = await test.InsertAsync(fact: fact, + type: alias ? EarlierTenantIdentity : typeof(TenantCreatedIntegrationEventV1).AssemblyQualifiedName, + payload: malformed ? "{}" : fact.ToPayloadJson()); + var beforeProduced = await test.ReadAsync(produced); + var before = await test.ReadAsync(id); + var store = test.Store(registry: PendingRegistry()); + + for (var poll = 0; poll < 3; poll++) + { + (await store.TryClaimAsync(NoBlockedPartitions)).Should().BeNull(); + (await store.TryAcquireRecoveryAsync(NoBlockedPartitions)).Should().BeNull(); + (await store.TryClaimTerminalAsync(NoBlockedPartitions)).Should().BeNull(); + } + + (await test.ReadAsync(produced)).Should().Be(beforeProduced); + (await test.ReadAsync(id)).Should().Be(before); + before.Attempts.Should().Be(0); + before.Token.Should().BeNull(); + before.TerminalReason.Should().BeNull(); + var snapshot = await store.ReadSnapshotAsync(NoBlockedPartitions); + snapshot.PendingCount.Should().Be(2); + snapshot.DeferredCount.Should().Be(2); + snapshot.OldestEligibleAgeSeconds.Should().Be(0); + (await test.AuditCountAsync()).Should().Be(0, "provisioning audits are separate from the terminal operation"); + } + + [Theory] + [InlineData("Topic", false)] + [InlineData("Topic", true)] + [InlineData("PartitionKey", false)] + [InlineData("PartitionKey", true)] + public async Task Stored_routing_corruption_requests_terminal_without_acknowledging_the_fact(string property, bool missing) + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var fact = new DispatchEventV1 + { + EventId = Guid.CreateVersion7(), + TenantId = SchemaFixture.TenantA, + OccurredAt = Occurred, + SubjectId = Guid.CreateVersion7(), + }; + var payload = JsonNode.Parse(fact.ToPayloadJson())!.AsObject(); + if (missing) payload.Remove(property); + else payload[property] = "private-invalid-routing"; + var id = await test.InsertAsync(fact: fact, payload: payload.ToJsonString()); + var registry = EligibleRegistry(); + var store = test.Store(registry: registry); + var claim = (await store.TryClaimAsync(NoBlockedPartitions))!; + + var result = new OutboxEnvelopeReader(registry).Read(claim); + + result.TerminalReason.Should().Be(OutboxTerminalReason.InvalidEnvelope); + result.Envelope.Should().BeNull(); + (await store.RequestTerminalAsync(claim, result.TerminalReason!.Value)).Should().BeTrue(); + (await store.TryClaimAsync(NoBlockedPartitions)).Should().BeNull(); + var terminal = (await store.TryClaimTerminalAsync(NoBlockedPartitions))!; + (await store.TryTerminalizeAsync(terminal, AttemptTraceparent)).Should().BeTrue(); + var row = await test.ReadAsync(id); + row.ProcessedAt.Should().BeNull(); + row.DeadLetteredAt.Should().NotBeNull(); + (await test.AuditCountAsync()).Should().Be(1); + (await test.ReadAuditAsync()).Metadata.Should().NotContain("private-invalid-routing"); + } + + [Fact] + public async Task An_intentionally_unsubscribed_contract_remains_eligible_for_explicit_acknowledgement() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + var registry = EligibleRegistry(); + registry.Resolve(typeof(DispatchEventV1).AssemblyQualifiedName).Disposition + .Should().Be(OutboxEventDisposition.IntentionallyUnsubscribed); + var store = test.Store(registry: registry); + + var claim = (await store.TryClaimAsync(NoBlockedPartitions))!; + claim.Id.Should().Be(id); + (await store.CompleteAsync(claim)).Should().BeTrue(); + + (await test.ReadAsync(id)).ProcessedAt.Should().NotBeNull(); + (await test.AuditCountAsync()).Should().Be(0); + } + + [Fact] + public async Task Selection_orders_by_availability_then_occurrence_then_ID_regardless_of_insert_order() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var low = Guid.Parse("10000000-0000-7000-8000-000000000001"); + var high = Guid.Parse("20000000-0000-7000-8000-000000000001"); + var olderOccurrence = Guid.Parse("30000000-0000-7000-8000-000000000001"); + var earlierAvailability = Guid.Parse("40000000-0000-7000-8000-000000000001"); + await test.InsertAsync(id: high, available: Occurred, occurred: Occurred); + await test.InsertAsync(id: low, available: Occurred, occurred: Occurred); + await test.InsertAsync(id: olderOccurrence, available: Occurred, occurred: Occurred.AddMinutes(-1)); + await test.InsertAsync(id: earlierAvailability, available: Occurred.AddMinutes(-1), occurred: Occurred.AddMinutes(1)); + var store = test.Store(); + var selected = new List(); + + for (var poll = 0; poll < 4; poll++) + { + var claim = (await store.TryClaimAsync(NoBlockedPartitions))!; + selected.Add(claim.Id); + (await store.CompleteAsync(claim)).Should().BeTrue(); + } + + selected.Should().Equal(earlierAvailability, olderOccurrence, low, high); + } + + [Fact] + public async Task Delayed_failure_allows_later_work_on_the_same_partition_to_overtake() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var partition = Guid.CreateVersion7(); + var first = await test.InsertAsync(partition: partition, available: Occurred); + var second = await test.InsertAsync(partition: partition, available: Occurred.AddMinutes(1)); + var store = test.Store(); + var failed = (await store.TryClaimAsync(NoBlockedPartitions))!; + failed.Id.Should().Be(first); + (await store.FailAsync(failed, OutboxFailureReason.PublishFailed)).Should().BeTrue(); + + var later = (await store.TryClaimAsync(NoBlockedPartitions))!; + + later.Id.Should().Be(second); + later.PartitionKey.Should().Be(failed.PartitionKey); + (await store.CompleteAsync(later)).Should().BeTrue(); + (await test.ReadAsync(first)).ProcessedAt.Should().BeNull(); + } + + [Fact] + public async Task Locally_blocked_partitions_exclude_normal_recovery_and_terminal_acquisition() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var blocked = Guid.CreateVersion7(); + var id = await test.InsertAsync(partition: blocked); + IReadOnlyCollection partitions = [blocked.ToString("D")]; + var store = test.Store(); + (await store.TryClaimAsync(partitions)).Should().BeNull(); + var normal = (await store.TryClaimAsync(NoBlockedPartitions))!; + await test.ExpireAsync(id); + (await store.TryAcquireRecoveryAsync(partitions)).Should().BeNull(); + var recovery = (await store.TryAcquireRecoveryAsync(NoBlockedPartitions))!; + (await store.RequestTerminalAsync(recovery, OutboxTerminalReason.InvalidEnvelope)).Should().BeTrue(); + (await store.TryClaimTerminalAsync(partitions)).Should().BeNull(); + var terminal = (await store.TryClaimTerminalAsync(NoBlockedPartitions))!; + terminal.Token.Should().NotBe(normal.Token).And.NotBe(recovery.Token); + (await store.TryTerminalizeAsync(terminal, AttemptTraceparent)).Should().BeTrue(); + } + + [Fact] + public async Task Queue_snapshot_counts_every_pending_state_but_ages_only_publish_eligible_work() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var now = await test.DatabaseNowAsync(); + var due = await test.InsertAsync(occurred: now.AddMinutes(-5)); + await test.InsertAsync(available: now.AddHours(1), occurred: now.AddHours(-10)); + var leased = await test.InsertAsync(occurred: now.AddHours(-9)); + await test.SetStateAsync(leased, "locked_by = '11111111-1111-4111-8111-111111111111', locked_until = now() + interval '30 seconds'"); + var expired = await test.InsertAsync(occurred: now.AddHours(-8)); + await test.SetStateAsync(expired, "locked_by = '22222222-2222-4222-8222-222222222222', locked_until = now() - interval '1 second'"); + var requested = await test.InsertAsync(occurred: now.AddHours(-7)); + await test.SetStateAsync(requested, "terminal_reason = 'invalid_envelope'"); + await test.InsertAsync(attempts: 6, occurred: now.AddHours(-6)); + await test.InsertAsync(fact: new TenantCreatedIntegrationEventV1 + { + EventId = Guid.CreateVersion7(), + TenantId = SchemaFixture.TenantA, + OccurredAt = now.AddHours(-5), + }); + var blockedPartition = Guid.CreateVersion7(); + await test.InsertAsync(partition: blockedPartition, occurred: now.AddHours(-4)); + var processed = await test.InsertAsync(occurred: now.AddHours(-3)); + await test.SetStateAsync(processed, "processed_at = now()"); + var terminal = await test.InsertAsync(occurred: now.AddHours(-2)); + await test.SetStateAsync(terminal, "terminal_reason = 'unknown_type', dead_lettered_at = now()"); + var registry = OutboxEventRegistry.Create( + [OutboxEventRegistration.IntentionallyUnsubscribed(typeof(DispatchEventV1)), + OutboxEventRegistration.Required(typeof(TenantCreatedIntegrationEventV1), + [OutboxSubscriptionManifest.Pending("customization.initialize-content-type-default.v1")])], + [], IntegrationEventHandlerRegistry.Discover(), + [typeof(DispatchEventV1).Assembly, typeof(TenantCreatedIntegrationEventV1).Assembly]); + var store = test.Store(registry: registry); + IReadOnlyCollection blocked = [blockedPartition.ToString("D")]; + var before = await test.DatabaseNowAsync(); + + var snapshot = await store.ReadSnapshotAsync(blocked); + + var after = await test.DatabaseNowAsync(); + snapshot.PendingCount.Should().Be(8, "delayed, leased, expired, requested, exhausted, deferred and blocked rows are pending"); + snapshot.DeferredCount.Should().Be(1); + snapshot.OldestEligibleAgeSeconds.Should().BeInRange( + (before - now.AddMinutes(-5)).TotalSeconds, (after - now.AddMinutes(-5)).TotalSeconds); + var claim = (await store.TryClaimAsync(blocked))!; + claim.Id.Should().Be(due, "snapshot eligibility must equal normal acquisition eligibility"); + (await store.CompleteAsync(claim)).Should().BeTrue(); + var excludedOnly = await store.ReadSnapshotAsync(blocked); + excludedOnly.PendingCount.Should().Be(7); + excludedOnly.DeferredCount.Should().Be(1); + excludedOnly.OldestEligibleAgeSeconds.Should().Be(0); + (await store.ReadSnapshotAsync(NoBlockedPartitions)).OldestEligibleAgeSeconds.Should().BeGreaterThan(14_000, + "removing the local hold admits the old blocked partition"); + } + + [Fact] + public async Task Empty_and_future_occurrence_snapshots_never_report_negative_eligible_age() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var store = test.Store(); + (await store.ReadSnapshotAsync(NoBlockedPartitions)).Should().Be(new OutboxQueueSnapshot(0, 0, 0)); + var now = await test.DatabaseNowAsync(); + await test.InsertAsync(occurred: now.AddHours(1)); + + var snapshot = await store.ReadSnapshotAsync(NoBlockedPartitions); + + snapshot.PendingCount.Should().Be(1); + snapshot.DeferredCount.Should().Be(0); + snapshot.OldestEligibleAgeSeconds.Should().Be(0); + } + + [Fact] + public async Task Reserved_subscriber_terminal_operation_remains_unregistered() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var catalog = test.Provider.GetRequiredService(); + catalog.TryGetOffPath("platform.outbox.dead_letter", out var producer).Should().BeTrue(); + producer.OperationClass.Should().Be(OperationClass.Must); + producer.OperationType.Should().Be(OperationType.SecurityEvent); + catalog.TryGetOffPath("platform.delivery.dead_letter", out _).Should().BeFalse(); + } + + private static Task TransitionAsync(OutboxDispatchStore store, OutboxDispatchClaim claim, string transition) => transition switch + { + "success" => store.CompleteAsync(claim), + "failure" => store.FailAsync(claim, OutboxFailureReason.PublishFailed), + "terminal-request" => store.RequestTerminalAsync(claim, OutboxTerminalReason.InvalidEnvelope), + _ => throw new ArgumentOutOfRangeException(nameof(transition)), + }; + + private static void AssertBackoff(Row row, int attempts, int seconds, DateTimeOffset before, DateTimeOffset after) + { + // Both endpoints are PostgreSQL readings, not elapsed wall-clock assertions. + // The interval must be the documented delay added by the intervening SQL. + row.AvailableAfter.Should().BeOnOrAfter(before.AddSeconds(seconds)); + row.AvailableAfter.Should().BeOnOrBefore(after.AddSeconds(seconds)); + row.Attempts.Should().Be(attempts); + row.Token.Should().BeNull(); + row.LockedUntil.Should().BeNull(); + row.TerminalReason.Should().BeNull(); + row.ProcessedAt.Should().BeNull(); + row.LastError.Should().NotBeNullOrWhiteSpace(); + } + + private static string ReplaceEventIdentity(string metadata) + { + var values = JsonSerializer.Deserialize>(metadata)!; + values["event_id"] = JsonSerializer.SerializeToElement(Guid.CreateVersion7()); + return JsonSerializer.Serialize(values); + } + + private static OutboxEventRegistry EligibleRegistry() => OutboxEventRegistry.Create( + [OutboxEventRegistration.IntentionallyUnsubscribed(typeof(DispatchEventV1))], [], + IntegrationEventHandlerRegistry.Discover(), [typeof(DispatchEventV1).Assembly]); + + private static OutboxEventRegistry PendingRegistry() => OutboxEventRegistry.Create( + [OutboxEventRegistration.Required(typeof(TenantCreatedIntegrationEventV1), + [ + OutboxSubscriptionManifest.Pending("customization.initialize-content-type-default.v1"), + OutboxSubscriptionManifest.Pending("customization.initialize-taxonomy-default.v1"), + ], EarlierTenantIdentity)], [], IntegrationEventHandlerRegistry.Discover(), + [typeof(TenantCreatedIntegrationEventV1).Assembly]); + + [IntegrationEventPartition(nameof(SubjectId))] + public sealed record DispatchEventV1 : IntegrationEventBase + { + public required Guid SubjectId { get; init; } + public override string Topic => "learnstack.test.dispatch"; + public override string PartitionKey => SubjectId.ToString("D"); + } + + private sealed record Row( + Guid Id, int Attempts, string? Token, DateTimeOffset? LockedUntil, + DateTimeOffset AvailableAfter, DateTimeOffset? ProcessedAt, + string? TerminalReason, DateTimeOffset? DeadLetteredAt, string? LastError, + string Type, string Payload, string? Metadata); + + private sealed record AuditRow( + Guid Tenant, Guid? Organization, Guid Actor, string Module, string Operation, + string Type, string Class, string Outcome, string? Reason, string Correlation, string Metadata); + + private sealed class TestDatabase : IAsyncDisposable + { + private readonly DisposableSchemaDatabase _database; + private readonly NpgsqlDataSource _appSource; + + private TestDatabase(DisposableSchemaDatabase database) + { + _database = database; + _appSource = NpgsqlDataSource.Create(database.AppConnectionString); + DispatchSource = OutboxDispatcherDataSource.Build(database.OutboxConnectionString); + Provider = SeedComposition.Build(_appSource, null, NullLoggerFactory.Instance); + } + + public NpgsqlDataSource DispatchSource { get; } + public ServiceProvider Provider { get; } + public string OutboxConnectionString => _database.OutboxConnectionString; + + public static async Task CreateAsync(PostgresFixture postgres, bool seedTenant = true) + { + var test = new TestDatabase(await DisposableSchemaDatabase.CreateAsync(postgres)); + try + { + if (seedTenant) await test.SeedTenantAsync(); + return test; + } + catch + { + await test.DisposeAsync(); + throw; + } + } + + public OutboxDispatchStore Store( + OutboxEventRegistry? registry = null, IClock? clock = null, + Func? decorate = null, NpgsqlDataSource? dispatchSource = null) => new( + new Lazy(() => dispatchSource ?? DispatchSource), registry ?? EligibleRegistry(), + new OutboxDispatchOptions(), new SystemGuidFactory(), clock ?? new FixedClock(Occurred), + decorate is null ? Provider.GetRequiredService() + : new DecoratingScopeFactory(Provider.GetRequiredService(), decorate)); + + public async Task InsertAsync( + Guid? id = null, Guid? organization = null, int attempts = 0, + DateTimeOffset? available = null, DateTimeOffset? occurred = null, Guid? partition = null, + string? type = null, string? payload = null, string? correlation = null, + Guid? causation = null, Guid? actor = null, IntegrationEventBase? fact = null, + string? metadata = null) + { + fact ??= new DispatchEventV1 + { + EventId = id ?? Guid.CreateVersion7(), + TenantId = SchemaFixture.TenantA, + OccurredAt = occurred ?? Occurred, + SubjectId = partition ?? Guid.CreateVersion7(), + }; + await using var connection = await _appSource.OpenConnectionAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await SchemaQueries.SetTenantAsync(connection, transaction, fact.TenantId); + if (organization is { } org) + await SchemaQueries.SetSettingAsync(connection, transaction, "app.organization_id", org.ToString("D")); + await using var insert = new NpgsqlCommand( + """ + INSERT INTO outbox_messages + (id, tenant_id, organization_id, occurred_at, correlation_id, + causation_id, actor_user_id, type, topic, partition_key, + payload, metadata, attempts, available_after) + VALUES (@id, @tenant, @organization, @occurred, @correlation, + @causation, @actor, @type, @topic, @partition, + @payload, @metadata, @attempts, @available) + """, connection, transaction); + insert.Parameters.AddWithValue("id", fact.EventId); + insert.Parameters.AddWithValue("tenant", fact.TenantId); + insert.Parameters.Add(new NpgsqlParameter("organization", NpgsqlDbType.Uuid) { Value = (object?)organization ?? DBNull.Value }); + insert.Parameters.AddWithValue("occurred", fact.OccurredAt); + insert.Parameters.AddWithValue("correlation", correlation ?? Traceparent); + insert.Parameters.Add(new NpgsqlParameter("causation", NpgsqlDbType.Uuid) { Value = (object?)causation ?? DBNull.Value }); + insert.Parameters.Add(new NpgsqlParameter("actor", NpgsqlDbType.Uuid) { Value = (object?)actor ?? DBNull.Value }); + insert.Parameters.AddWithValue("type", type ?? fact.GetType().AssemblyQualifiedName!); + insert.Parameters.AddWithValue("topic", fact.Topic); + insert.Parameters.AddWithValue("partition", fact.PartitionKey); + insert.Parameters.Add(new NpgsqlParameter("payload", NpgsqlDbType.Jsonb) { Value = payload ?? fact.ToPayloadJson() }); + insert.Parameters.Add(new NpgsqlParameter("metadata", NpgsqlDbType.Jsonb) { Value = (object?)metadata ?? DBNull.Value }); + insert.Parameters.AddWithValue("attempts", attempts); + insert.Parameters.AddWithValue("available", available ?? Occurred); + (await insert.ExecuteNonQueryAsync()).Should().Be(1); + await transaction.CommitAsync(); + return fact.EventId; + } + + public async Task ReadAsync(Guid id) + { + await using var connection = await DispatchSource.OpenConnectionAsync(); + await using var command = new NpgsqlCommand( + """ + SELECT id, attempts, locked_by, locked_until, available_after, + processed_at, terminal_reason, dead_lettered_at, last_error, + type, payload::text, metadata::text + FROM outbox_messages WHERE id = @id + """, connection); + command.Parameters.AddWithValue("id", id); + await using var reader = await command.ExecuteReaderAsync(); + (await reader.ReadAsync()).Should().BeTrue(); + return new Row(reader.GetGuid(0), reader.GetInt32(1), Text(reader, 2), Timestamp(reader, 3), + reader.GetFieldValue(4), Timestamp(reader, 5), Text(reader, 6), Timestamp(reader, 7), + Text(reader, 8), reader.GetString(9), reader.GetString(10), Text(reader, 11)); + } + + public async Task SingleIdAsync() + { + await using var connection = await DispatchSource.OpenConnectionAsync(); + await using var command = new NpgsqlCommand("SELECT id FROM outbox_messages", connection); + await using var reader = await command.ExecuteReaderAsync(); + (await reader.ReadAsync()).Should().BeTrue(); + var id = reader.GetGuid(0); + (await reader.ReadAsync()).Should().BeFalse(); + return id; + } + + public Task ExpireAsync(Guid id) => UpdateAsync(id, "locked_until = now() - interval '1 second'"); + public Task MakeDueAsync(Guid id) => UpdateAsync(id, "available_after = now() - interval '1 second'"); + public Task SetStateAsync(Guid id, string assignment) => UpdateAsync(id, assignment); + + public async Task DatabaseNowAsync() + { + await using var connection = await DispatchSource.OpenConnectionAsync(); + await using var command = new NpgsqlCommand("SELECT clock_timestamp()", connection); + await using var reader = await command.ExecuteReaderAsync(); + (await reader.ReadAsync()).Should().BeTrue(); + return reader.GetFieldValue(0); + } + + public async Task AuditInsertGrantAsync(bool grant) + { + await using var connection = (NpgsqlConnection)await PostgresFixture.OpenAsync(_database.MigrationConnectionString); + await using var command = new NpgsqlCommand(grant + ? "GRANT INSERT ON audit_log TO learnstack_outbox_admin" + : "REVOKE INSERT ON audit_log FROM learnstack_outbox_admin", connection); + await command.ExecuteNonQueryAsync(); + } + + public async Task AuditCountAsync() + { + await using var connection = await _appSource.OpenConnectionAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await SchemaQueries.SetTenantAsync(connection, transaction, SchemaFixture.TenantA); + await SchemaQueries.SetSettingAsync(connection, transaction, "app.scope", "tenant"); + await using var command = new NpgsqlCommand( + "SELECT count(*) FROM audit_log WHERE operation = 'platform.outbox.dead_letter'", connection, transaction); + return (long)(await command.ExecuteScalarAsync())!; + } + + public async Task ReadAuditAsync() + { + await using var connection = await _appSource.OpenConnectionAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await SchemaQueries.SetTenantAsync(connection, transaction, SchemaFixture.TenantA); + await SchemaQueries.SetSettingAsync(connection, transaction, "app.organization_id", SchemaFixture.OrgA1.ToString("D")); + await using var command = new NpgsqlCommand( + """ + SELECT tenant_id, organization_id, actor_user_id, module, operation, + operation_type, operation_class, outcome, reason, correlation_id, metadata::text + FROM audit_log WHERE operation = 'platform.outbox.dead_letter' + """, connection, transaction); + await using var reader = await command.ExecuteReaderAsync(); + (await reader.ReadAsync()).Should().BeTrue(); + var row = new AuditRow(reader.GetGuid(0), reader.IsDBNull(1) ? null : reader.GetGuid(1), reader.GetGuid(2), + reader.GetString(3), reader.GetString(4), reader.GetString(5), reader.GetString(6), reader.GetString(7), + Text(reader, 8), reader.GetString(9), reader.GetString(10)); + (await reader.ReadAsync()).Should().BeFalse(); + return row; + } + + public async ValueTask DisposeAsync() + { + await Provider.DisposeAsync(); + await DispatchSource.DisposeAsync(); + await _appSource.DisposeAsync(); + await _database.DisposeAsync(); + } + + private async Task UpdateAsync(Guid id, string assignment) + { + // Assignments are closed test-owned statements; values remain parameters. + await using var connection = await DispatchSource.OpenConnectionAsync(); + await using var command = new NpgsqlCommand($"UPDATE outbox_messages SET {assignment} WHERE id = @id", connection); + command.Parameters.AddWithValue("id", id); + (await command.ExecuteNonQueryAsync()).Should().Be(1); + } + + private async Task SeedTenantAsync() + { + await using var connection = await _appSource.OpenConnectionAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await SchemaQueries.SetTenantAsync(connection, transaction, SchemaFixture.TenantA); + await SchemaQueries.ExecuteAsync(connection, transaction, + """ + INSERT INTO tenants (id, slug, display_name, status, created_at, created_by, row_version) + VALUES (@tenant, 'dispatch-probe', 'Dispatch probe', 'Trial', now(), @actor, 0); + INSERT INTO organizations (id, tenant_id, slug, display_name, status, created_at, created_by, row_version) + VALUES (@organization, @tenant, 'hq', 'HQ', 'Active', now(), @actor, 0); + """, ("tenant", SchemaFixture.TenantA), ("organization", SchemaFixture.OrgA1), ("actor", UserId.SystemActor.Value)); + await transaction.CommitAsync(); + } + + private static string? Text(NpgsqlDataReader reader, int ordinal) => reader.IsDBNull(ordinal) ? null : reader.GetString(ordinal); + private static DateTimeOffset? Timestamp(NpgsqlDataReader reader, int ordinal) => + reader.IsDBNull(ordinal) ? null : reader.GetFieldValue(ordinal); + } + + private sealed class DecoratingScopeFactory(IServiceScopeFactory inner, Func decorate) : IServiceScopeFactory + { + public IServiceScope CreateScope() => new DecoratingScope(inner.CreateScope(), decorate); + } + + private sealed class DecoratingScope : IServiceScope, IAsyncDisposable + { + private readonly IServiceScope _inner; + + public DecoratingScope(IServiceScope inner, Func decorate) + { + _inner = inner; + ServiceProvider = new DecoratingProvider(inner.ServiceProvider, decorate); + } + + public IServiceProvider ServiceProvider { get; } + public void Dispose() => _inner.Dispose(); + public ValueTask DisposeAsync() => _inner is IAsyncDisposable asyncDisposable + ? asyncDisposable.DisposeAsync() : DisposeSynchronously(); + + private ValueTask DisposeSynchronously() + { + _inner.Dispose(); + return ValueTask.CompletedTask; + } + } + + private sealed class DecoratingProvider(IServiceProvider inner, Func decorate) : IServiceProvider + { + public object? GetService(Type serviceType) => serviceType == typeof(IAuditStore) + ? decorate(inner.GetRequiredService()) : inner.GetService(serviceType); + } + + private sealed class TerminalAuditDecorator( + IAuditStore inner, + Func terminal) : IAuditStore + { + public Task WritePendingAsync(IUnitOfWork unitOfWork, CancellationToken cancellationToken = default) => + inner.WritePendingAsync(unitOfWork, cancellationToken); + public Task WriteStandaloneAsync(AuditEntryDraft entry, CancellationToken cancellationToken = default) => + inner.WriteStandaloneAsync(entry, cancellationToken); + public Task WriteBestEffortAsync(AuditEntryDraft entry, CancellationToken cancellationToken = default) => + inner.WriteBestEffortAsync(entry, cancellationToken); + public Task WritePlatformScopeAsync(AuditEntryDraft entry, DbConnection connection, DbTransaction transaction, + CancellationToken cancellationToken = default) => inner.WritePlatformScopeAsync(entry, connection, transaction, cancellationToken); + public Task WriteEventTerminalAsync(AuditEntryDraft entry, EventTerminalTransition transition, DbConnection connection, + DbTransaction transaction, CancellationToken cancellationToken = default) => + terminal(entry, transition, connection, transaction, cancellationToken); + } +} diff --git a/backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatcherRoleGuardTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatcherRoleGuardTests.cs new file mode 100644 index 00000000..6329129f --- /dev/null +++ b/backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatcherRoleGuardTests.cs @@ -0,0 +1,211 @@ +using FluentAssertions; +using LearnStack.Infrastructure.Messaging; +using LearnStack.Infrastructure.Persistence; +using Npgsql; +using Xunit; + +namespace LearnStack.Tests.Integration.Database; + +// Role mutations belong to this class's container, never the shared schema suite. +[Trait(RequiresDocker.Key, RequiresDocker.Value)] +public sealed class OutboxDispatcherRoleGuardTests(PostgresFixture postgres) : IClassFixture +{ + [Fact] + public async Task The_actual_outbox_role_connects_with_its_exact_bounded_attributes() + { + await using var source = OutboxDispatcherDataSource.Build(postgres.OutboxConnectionString); + await using var connection = await source.OpenConnectionAsync(); + await using var command = new NpgsqlCommand( + """ + SELECT session_user, current_user, rolbypassrls, + rolsuper, rolcreatedb, rolcreaterole, rolreplication + FROM pg_roles WHERE rolname = current_user + """, connection); + await using var reader = await command.ExecuteReaderAsync(); + + (await reader.ReadAsync()).Should().BeTrue(); + reader.GetString(0).Should().Be(OutboxDispatcherDataSource.Role); + reader.GetString(1).Should().Be(OutboxDispatcherDataSource.Role); + reader.GetBoolean(2).Should().BeTrue(); + for (var ordinal = 3; ordinal <= 6; ordinal++) reader.GetBoolean(ordinal).Should().BeFalse(); + (await reader.ReadAsync()).Should().BeFalse(); + } + + [Theory] + [InlineData("NOBYPASSRLS", "BYPASSRLS", "NOT rolbypassrls", false)] + [InlineData("NOBYPASSRLS", "BYPASSRLS", "NOT rolbypassrls", true)] + [InlineData("SUPERUSER", "NOSUPERUSER", "rolsuper", false)] + [InlineData("SUPERUSER", "NOSUPERUSER", "rolsuper", true)] + [InlineData("CREATEDB", "NOCREATEDB", "rolcreatedb", false)] + [InlineData("CREATEDB", "NOCREATEDB", "rolcreatedb", true)] + [InlineData("CREATEROLE", "NOCREATEROLE", "rolcreaterole", false)] + [InlineData("CREATEROLE", "NOCREATEROLE", "rolcreaterole", true)] + [InlineData("REPLICATION", "NOREPLICATION", "rolreplication", false)] + [InlineData("REPLICATION", "NOREPLICATION", "rolreplication", true)] + public async Task Every_physical_connection_refuses_an_invalid_attribute_and_accepts_its_restoration( + string altered, string restored, string predicate, bool synchronous) + { + await using var source = OutboxDispatcherDataSource.Build(postgres.OutboxConnectionString); + // Occupy the first physical connection so the rejection exercises a new + // connector in an already-used pool, not just the pool's first connection. + await using var first = await source.OpenConnectionAsync(); + try + { + await postgres.ExecuteAsSuperuserAsync($"ALTER ROLE learnstack_outbox_admin {altered}"); + await using var raw = await PostgresFixture.OpenAsync(postgres.OutboxConnectionString); + await using var truth = new NpgsqlCommand( + $"SELECT {predicate} FROM pg_roles WHERE rolname = current_user", (NpgsqlConnection)raw); + (await truth.ExecuteScalarAsync()).Should().Be(true, "the invalid server attribute must actually be present"); + + await AssertGuardRefusesAsync(source, synchronous); + } + finally + { + await postgres.ExecuteAsSuperuserAsync($"ALTER ROLE learnstack_outbox_admin {restored}"); + } + + await using var healthy = await source.OpenConnectionAsync(); + healthy.State.Should().Be(System.Data.ConnectionState.Open); + healthy.ProcessID.Should().NotBe(first.ProcessID); + } + + [Theory] + [InlineData(false, false)] + [InlineData(false, true)] + [InlineData(true, false)] + [InlineData(true, true)] + public async Task Foreign_membership_is_refused_directly_and_through_a_bridge_then_restoration_connects( + bool throughBridge, bool synchronous) + { + await using var source = OutboxDispatcherDataSource.Build(postgres.OutboxConnectionString); + try + { + await postgres.ExecuteAsSuperuserAsync(throughBridge + ? "CREATE ROLE outbox_guard_bridge NOLOGIN; " + + "GRANT learnstack_migration TO outbox_guard_bridge; " + + "GRANT outbox_guard_bridge TO learnstack_outbox_admin" + : "GRANT learnstack_migration TO learnstack_outbox_admin"); + await using var raw = await PostgresFixture.OpenAsync(postgres.OutboxConnectionString); + await using var truth = new NpgsqlCommand( + "SELECT pg_has_role(current_user, 'learnstack_migration', 'MEMBER')", (NpgsqlConnection)raw); + (await truth.ExecuteScalarAsync()).Should().Be(true, "the ownership escalation must be reachable"); + + await AssertGuardRefusesAsync(source, synchronous); + } + finally + { + await postgres.ExecuteAsSuperuserAsync(throughBridge + ? "DROP ROLE IF EXISTS outbox_guard_bridge" + : "REVOKE learnstack_migration FROM learnstack_outbox_admin"); + } + + await using var healthy = await source.OpenConnectionAsync(); + healthy.State.Should().Be(System.Data.ConnectionState.Open); + } + + [Theory] + [InlineData(false, false)] + [InlineData(false, true)] + [InlineData(true, false)] + [InlineData(true, true)] + public async Task App_membership_in_the_dispatcher_is_refused_directly_and_transitively_then_restoration_connects( + bool throughBridge, bool synchronous) + { + await using var source = OutboxDispatcherDataSource.Build(postgres.OutboxConnectionString); + await using var first = await source.OpenConnectionAsync(); + await using var appSource = ApplicationDataSource.Build(postgres.AppConnectionString); + // The app guard checks new physical connections. A later grant can still + // escalate an existing session, so the dispatcher checks incoming grants too. + await using var app = await appSource.OpenConnectionAsync(); + try + { + await postgres.ExecuteAsSuperuserAsync(throughBridge + ? "CREATE ROLE outbox_guard_member_bridge NOLOGIN; " + + "GRANT learnstack_outbox_admin TO outbox_guard_member_bridge; " + + "GRANT outbox_guard_member_bridge TO learnstack_app" + : "GRANT learnstack_outbox_admin TO learnstack_app"); + await using var switchRole = new NpgsqlCommand("SET ROLE learnstack_outbox_admin", app); + await switchRole.ExecuteNonQueryAsync(); + await using var truth = new NpgsqlCommand( + """ + SELECT session_user = 'learnstack_app' + AND current_user = 'learnstack_outbox_admin' AND rolbypassrls + FROM pg_roles WHERE rolname = current_user + """, app); + (await truth.ExecuteScalarAsync()).Should().Be(true, + "the direct or transitive grant must really let an existing app session acquire the bypass role"); + + await AssertGuardRefusesAsync(source, synchronous); + } + finally + { + await postgres.ExecuteAsSuperuserAsync(throughBridge + ? "DROP ROLE IF EXISTS outbox_guard_member_bridge" + : "REVOKE learnstack_outbox_admin FROM learnstack_app"); + await using var resetRole = new NpgsqlCommand("RESET ROLE", app); + await resetRole.ExecuteNonQueryAsync(); + } + + await using var healthy = await source.OpenConnectionAsync(); + healthy.State.Should().Be(System.Data.ConnectionState.Open); + healthy.ProcessID.Should().NotBe(first.ProcessID); + } + + [Fact] + public async Task A_startup_role_switch_cannot_change_the_dispatcher_current_role() + { + try + { + await postgres.ExecuteAsSuperuserAsync( + "CREATE ROLE outbox_guard_switched NOLOGIN; " + + "GRANT outbox_guard_switched TO learnstack_outbox_admin"); + var switched = new NpgsqlConnectionStringBuilder(postgres.OutboxConnectionString) + { + Options = "-c role=outbox_guard_switched", + }.ConnectionString; + await using var raw = await PostgresFixture.OpenAsync(switched); + await using var truth = new NpgsqlCommand("SELECT session_user, current_user", (NpgsqlConnection)raw); + await using (var reader = await truth.ExecuteReaderAsync()) + { + (await reader.ReadAsync()).Should().BeTrue(); + reader.GetString(0).Should().Be(OutboxDispatcherDataSource.Role); + reader.GetString(1).Should().Be("outbox_guard_switched"); + } + + await using var source = OutboxDispatcherDataSource.Build(switched); + await AssertGuardRefusesAsync(source, synchronous: false); + } + finally + { + await postgres.ExecuteAsSuperuserAsync("DROP ROLE IF EXISTS outbox_guard_switched"); + } + + await using var restored = OutboxDispatcherDataSource.Build(postgres.OutboxConnectionString); + await using var healthy = await restored.OpenConnectionAsync(); + healthy.State.Should().Be(System.Data.ConnectionState.Open); + } + + private static async Task AssertGuardRefusesAsync(NpgsqlDataSource source, bool synchronous) + { + InvalidOperationException failure; + if (synchronous) + { + var open = () => + { + using var connection = source.OpenConnection(); + }; + failure = open.Should().Throw().Which; + } + else + { + var open = async () => + { + await using var connection = await source.OpenConnectionAsync(); + }; + failure = (await open.Should().ThrowAsync()).Which; + } + + failure.Message.Should().Be("The outbox dispatcher connection has an invalid role, attributes or memberships."); + failure.InnerException.Should().BeNull(); + } +} diff --git a/backend/tests/LearnStack.Tests.Integration/Database/OutboxProcessorTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/OutboxProcessorTests.cs new file mode 100644 index 00000000..63e0af8f --- /dev/null +++ b/backend/tests/LearnStack.Tests.Integration/Database/OutboxProcessorTests.cs @@ -0,0 +1,947 @@ +using System.Collections.Concurrent; +using System.Diagnostics; +using System.Net; +using FluentAssertions; +using LearnStack.Api.Messaging; +using LearnStack.Infrastructure.Messaging; +using LearnStack.Modules.Tenancy.Application.Contracts.IntegrationEvents; +using LearnStack.Modules.Tenancy.Application.Contracts.Tenant; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Messaging; +using LearnStack.SharedKernel.Tenancy; +using MediatR; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.Mvc.Testing; +using Microsoft.AspNetCore.TestHost; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; +using Microsoft.Extensions.Diagnostics.HealthChecks; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using Npgsql; +using NpgsqlTypes; +using Xunit; + +namespace LearnStack.Tests.Integration.Database; + +/// The registered production worker, real PostgreSQL roles and actual transport scopes. +[Trait(RequiresDocker.Key, RequiresDocker.Value)] +[Collection(SharedSchema.Name)] +public sealed class OutboxProcessorTests(SchemaFixture schema) +{ + private const string Traceparent = "00-11111111111111111111111111111111-2222222222222222-01"; + private const string SensitiveCanary = "private-outbox-canary-84e315"; + private static readonly DateTimeOffset Occurred = new(2026, 10, 1, 12, 0, 0, TimeSpan.Zero); + private static readonly TimeSpan ProofDeadline = TimeSpan.FromSeconds(10); + private static readonly OutboxDispatchOptions Bounds = new() + { + SqlTimeout = TimeSpan.FromSeconds(1), + PublishTimeout = TimeSpan.FromMilliseconds(150), + LeaseDuration = TimeSpan.FromSeconds(3), + IdlePoll = TimeSpan.FromMilliseconds(20), + ProgressTimeout = TimeSpan.FromSeconds(5), + SnapshotInterval = TimeSpan.FromMilliseconds(20), + ShutdownTimeout = TimeSpan.FromMilliseconds(200), + }; + + [Fact] + public async Task The_actual_API_registers_exactly_one_worker_and_boots_its_guarded_source() + { + await using var test = await WorkerDatabase.CreateAsync(schema.Postgres); + test.Start(); + + test.Services.GetServices().OfType() + .Should().ContainSingle().Which.Should().BeSameAs(test.Worker); + await UntilAsync(() => test.Observation.IsHealthy); + var source = test.Services.GetRequiredKeyedService>(OutboxDispatcherDataSource.Key); + source.IsValueCreated.Should().BeTrue("startup must connect even when there are no rows"); + await using var connection = await source.Value.OpenConnectionAsync(); + await using var role = new NpgsqlCommand("SELECT session_user, current_user", connection); + await using var reader = await role.ExecuteReaderAsync(); + (await reader.ReadAsync()).Should().BeTrue(); + reader.GetString(0).Should().Be(OutboxDispatcherDataSource.Role); + reader.GetString(1).Should().Be(OutboxDispatcherDataSource.Role); + } + + [Fact] + public async Task Two_independent_actual_processors_cannot_publish_or_claim_one_live_lease_twice() + { + await using var test = await WorkerDatabase.CreateAsync(schema.Postgres); + var liveBounds = Bounds with + { + PublishTimeout = TimeSpan.FromSeconds(10), + LeaseDuration = TimeSpan.FromSeconds(30), + ProgressTimeout = TimeSpan.FromSeconds(45), + }; + var partition = Guid.CreateVersion7(); + var held = test.Callbacks.Hold(partition); + test.Start(liveBounds); + await using var contender = new WorkerApiFactory(test, test.OutboxConnectionString, liveBounds); + OutboxProcessor? secondWorker = null; + try + { + var secondServices = contender.Services; + secondWorker = secondServices.GetServices().OfType() + .Should().ContainSingle().Which; + secondWorker.Should().NotBeSameAs(test.Worker); + var secondObservation = secondServices.GetRequiredService(); + await UntilAsync(() => test.Observation.IsHealthy && secondObservation.IsHealthy); + + // Host construction and the first empty polls finish before the live + // lease begins. The whole contention proof shares a shorter deadline. + using var proof = new CancellationTokenSource(TimeSpan.FromSeconds(5)); + var id = await test.InsertAsync(partition: partition).WaitAsync(proof.Token); + await held.Entered.Task.WaitAsync(proof.Token); + var acquired = await test.ReadAsync(id).WaitAsync(proof.Token); + acquired.Attempts.Should().Be(1); + acquired.Token.Should().NotBeNullOrWhiteSpace(); + var firstPrevious = test.Observation.Snapshot.LastSnapshotTimestamp; + var secondPrevious = secondObservation.Snapshot.LastSnapshotTimestamp; + await UntilAsync(() => test.Observation.Snapshot.LastSnapshotTimestamp > firstPrevious + || secondObservation.Snapshot.LastSnapshotTimestamp > secondPrevious, proof.Token); + // Either warm processor may win; the other must continue real polls + // while the winning processor is awaiting its callback. + var observer = test.Observation.Snapshot.LastSnapshotTimestamp > firstPrevious + ? test.Observation : secondObservation; + for (var poll = 0; poll < 2; poll++) + { + var previous = observer.Snapshot.LastSnapshotTimestamp; + await UntilAsync(() => observer.Snapshot.LastSnapshotTimestamp > previous, proof.Token); + } + + (await test.LeaseIsLiveAsync(id).WaitAsync(proof.Token)).Should().BeTrue("the contention assertion must run before expiry or a timeout"); + (await test.ReadAsync(id).WaitAsync(proof.Token)).Should().Be(acquired); + test.Callbacks.Constructions.Should().Be(1); + observer.Snapshot.Queue.PendingCount.Should().Be(1); + observer.Snapshot.Queue.OldestEligibleAgeSeconds.Should().Be(0); + held.ReleaseCallback.TrySetResult(); + await UntilAsync(async () => (await test.ReadAsync(id)).ProcessedAt is not null, proof.Token); + var completed = await test.ReadAsync(id).WaitAsync(proof.Token); + completed.Attempts.Should().Be(1); + completed.Token.Should().BeNull(); + test.Callbacks.Constructions.Should().Be(1); + } + finally + { + held.ReleaseCallback.TrySetResult(); + held.ReleaseCleanup.TrySetResult(); + await test.Worker.StopAsync(CancellationToken.None).WaitAsync(ProofDeadline); + if (secondWorker is not null) await secondWorker.StopAsync(CancellationToken.None).WaitAsync(ProofDeadline); + await test.Callbacks.AwaitCleanupAsync().WaitAsync(ProofDeadline); + } + } + + [Theory] + [InlineData("missing")] + [InlineData("malformed")] + [InlineData("wrong-user")] + [InlineData("unreachable")] + public async Task Actual_host_startup_rejects_invalid_credentials_without_provider_or_secret_text(string invalid) + { + await using var test = await WorkerDatabase.CreateAsync(schema.Postgres); + var credential = invalid switch + { + "missing" => string.Empty, + "malformed" => "Password=" + SensitiveCanary + ";invalid-field=value", + "wrong-user" => new NpgsqlConnectionStringBuilder(test.OutboxConnectionString) + { + Username = "learnstack_app", + Password = SensitiveCanary, + }.ConnectionString, + "unreachable" => new NpgsqlConnectionStringBuilder(test.OutboxConnectionString) + { + Host = "127.0.0.1", + Port = 1, + Password = SensitiveCanary, + Timeout = 1, + }.ConnectionString, + _ => throw new ArgumentOutOfRangeException(nameof(invalid)), + }; + await using var factory = new WorkerApiFactory(test, credential); + + var failure = await Record.ExceptionAsync(() => Task.Run(() => _ = factory.Services).WaitAsync(ProofDeadline)); + + failure.Should().NotBeNull(); + failure.Should().BeAssignableTo(); + failure!.ToString().Should().NotContain(SensitiveCanary).And.NotContain(credential == string.Empty ? "Password=" : credential); + if (invalid == "missing") + failure.Message.Should().Contain("ConnectionStrings:OutboxDispatcher") + .And.Contain("Npgsql key/value").And.Contain("learnstack_outbox_admin"); + test.Logs.Text.Should().NotContain(SensitiveCanary); + } + + [Fact] + public async Task Real_provisioning_rows_remain_Pending_across_hosted_polls_without_construction_or_mutation() + { + await using var test = await WorkerDatabase.CreateAsync(schema.Postgres, seedTenant: false); + test.Start(); + await using (var scope = test.Services.CreateAsyncScope()) + { + var result = await scope.ServiceProvider.GetRequiredService().Send(new ProvisionTenantCommand( + TenantId.From(SchemaFixture.TenantA), "worker-probe", "Worker probe", + OrganizationId.From(SchemaFixture.OrgA1), "hq", "HQ")); + result.IsSuccess.Should().BeTrue(); + } + var id = await test.SingleIdAsync(); + var before = await test.ReadAsync(id); + for (var iteration = 0; iteration < 3; iteration++) await test.NextSnapshotAsync(); + + (await test.ReadAsync(id)).Should().Be(before); + before.Type.Should().Be(typeof(TenantCreatedIntegrationEventV1).AssemblyQualifiedName); + before.Attempts.Should().Be(0); + test.Callbacks.Constructions.Should().Be(0); + test.Observation.Snapshot.Queue.PendingCount.Should().Be(1); + test.Observation.Snapshot.Queue.DeferredCount.Should().Be(1); + (await test.ReadTerminalAuditsAsync()).Should().BeEmpty(); + } + + [Fact] + public async Task Successful_claims_keep_the_snapshot_until_its_interval_and_refresh_at_the_boundary() + { + await using var test = await WorkerDatabase.CreateAsync(schema.Postgres); + var time = new SnapshotTimeProvider(); + var bounds = Bounds with + { + SnapshotInterval = TimeSpan.FromSeconds(1), + PublishTimeout = TimeSpan.FromSeconds(10), + LeaseDuration = TimeSpan.FromSeconds(30), + ProgressTimeout = TimeSpan.FromSeconds(45), + }; + test.Start(bounds, time); + await UntilAsync(() => test.Observation.IsHealthy); + var initial = test.Observation.Snapshot; + initial.LastSnapshotTimestamp.Should().Be(0); + + var first = await test.InsertAsync(); + await UntilAsync(async () => (await test.ReadAsync(first)).ProcessedAt is not null); + var secondPartition = Guid.CreateVersion7(); + var second = test.Callbacks.Hold(secondPartition); + var secondId = await test.InsertAsync(partition: secondPartition); + await second.Entered.Task.WaitAsync(ProofDeadline); + // Entering the next callback proves the preceding iteration (including + // its refresh decision) completed. Wall-clock duration cannot expire this clock. + test.Observation.Snapshot.Queue.Should().BeSameAs(initial.Queue); + + time.Advance(bounds.SnapshotInterval - TimeSpan.FromTicks(1)); + second.ReleaseCallback.TrySetResult(); + await UntilAsync(async () => (await test.ReadAsync(secondId)).ProcessedAt is not null); + var thirdPartition = Guid.CreateVersion7(); + var third = test.Callbacks.Hold(thirdPartition); + var thirdId = await test.InsertAsync(partition: thirdPartition); + await third.Entered.Task.WaitAsync(ProofDeadline); + test.Observation.Snapshot.Queue.Should().BeSameAs(initial.Queue); + test.Observation.Snapshot.LastSnapshotTimestamp.Should().Be(0); + + time.Advance(TimeSpan.FromTicks(1)); + third.ReleaseCallback.TrySetResult(); + await UntilAsync(() => test.Observation.Snapshot.LastSnapshotTimestamp == time.GetTimestamp()); + + (await test.ReadAsync(thirdId)).ProcessedAt.Should().NotBeNull(); + test.Observation.Snapshot.Queue.Should().NotBeSameAs(initial.Queue); + test.Observation.Snapshot.Queue.PendingCount.Should().Be(0); + test.Observation.Snapshot.LastSnapshotTimestamp.Should().Be(bounds.SnapshotInterval.Ticks); + test.Callbacks.Constructions.Should().Be(3); + test.Observation.IsHealthy.Should().BeTrue(); + } + + [Fact] + public async Task A_poison_first_row_terminalizes_with_MUST_audit_and_the_following_real_handler_progresses() + { + await using var test = await WorkerDatabase.CreateAsync(schema.Postgres); + using var traces = new DispatcherTraces(); + var poison = await test.InsertAsync(type: SensitiveCanary, payload: "{\"private\":\"" + SensitiveCanary + "\"}", correlation: SensitiveCanary); + var following = await test.InsertAsync(); + test.Start(); + + await UntilAsync(async () => (await test.ReadAsync(poison)).DeadLetteredAt is not null + && (await test.ReadAsync(following)).ProcessedAt is not null); + + var row = await test.ReadAsync(poison); + row.TerminalReason.Should().Be("unknown_type"); + row.Attempts.Should().Be(1); + row.ProcessedAt.Should().BeNull(); + test.Callbacks.Constructions.Should().Be(1); + var audit = (await test.ReadTerminalAuditsAsync()).Should().ContainSingle().Which; + audit.OperationClass.Should().Be("Must"); + audit.OperationType.Should().Be("SecurityEvent"); + audit.Metadata.Should().Contain(poison.ToString("D")).And.NotContain(SensitiveCanary); + audit.Correlation.Should().NotBe(SensitiveCanary); + ActivityContext.TryParse(audit.Correlation, null, out _).Should().BeTrue(); + test.Logs.Text.Should().NotContain(SensitiveCanary); + row.LastError.Should().Be("terminal:unknown_type:closed"); + traces.Records.Should().NotBeEmpty(); + string.Join("\n", traces.Records).Should().NotContain(SensitiveCanary); + await test.NextSnapshotAsync(); + (await test.ReadTerminalAuditsAsync()).Should().ContainSingle(); + } + + [Fact] + public async Task Ordinary_handler_failure_persists_a_closed_descriptor_and_allows_following_work() + { + await using var test = await WorkerDatabase.CreateAsync(schema.Postgres); + using var traces = new DispatcherTraces(); + var partition = Guid.CreateVersion7(); + test.Callbacks.Fail(partition, new InvalidOperationException(SensitiveCanary)); + var failed = await test.InsertAsync(partition: partition); + var following = await test.InsertAsync(); + test.Start(); + await UntilAsync(async () => (await test.ReadAsync(failed)).LastError == "publish:failed:provider"); + await test.PostponeAsync(failed); + await UntilAsync(async () => (await test.ReadAsync(following)).ProcessedAt is not null); + + var row = await test.ReadAsync(failed); + row.Attempts.Should().Be(1); + row.Token.Should().BeNull(); + row.ProcessedAt.Should().BeNull(); + test.Callbacks.Constructions.Should().Be(2); + test.Logs.Text.Should().NotContain(SensitiveCanary); + traces.Records.Should().NotBeEmpty(); + string.Join("\n", traces.Records).Should().NotContain(SensitiveCanary); + } + + [Fact] + public async Task Explicit_intentional_absence_acknowledges_without_constructing_a_handler() + { + await using var test = await WorkerDatabase.CreateAsync(schema.Postgres); + var fact = new UnsubscribedEventV1 + { + EventId = Guid.CreateVersion7(), + TenantId = SchemaFixture.TenantA, + OccurredAt = Occurred, + SubjectId = Guid.CreateVersion7(), + }; + var id = await test.InsertAsync(fact: fact); + test.Start(); + + await UntilAsync(async () => (await test.ReadAsync(id)).ProcessedAt is not null); + + (await test.ReadAsync(id)).Attempts.Should().Be(1); + test.Callbacks.Constructions.Should().Be(0); + (await test.ReadTerminalAuditsAsync()).Should().BeEmpty(); + } + + [Fact] + public async Task Timeout_persists_failure_and_holds_the_partition_until_actual_callback_scope_cleanup() + { + await using var test = await WorkerDatabase.CreateAsync(schema.Postgres); + var partition = Guid.CreateVersion7(); + var held = test.Callbacks.Hold(partition, holdCleanup: true); + var first = await test.InsertAsync(partition: partition); + var sibling = await test.InsertAsync(partition: partition); + var unrelated = await test.InsertAsync(); + test.Start(); + await held.Entered.Task.WaitAsync(ProofDeadline); + await UntilAsync(async () => (await test.ReadAsync(first)).LastError == "publish:timeout:deadline"); + await test.PostponeAsync(first); + await UntilAsync(async () => (await test.ReadAsync(unrelated)).ProcessedAt is not null); + + test.Observation.IsHealthy.Should().BeFalse(); + (await test.ReadAsync(sibling)).Attempts.Should().Be(0); + held.Disposals.Should().Be(0); + held.CancellationToken.IsCancellationRequested.Should().BeTrue(); + held.ReleaseCallback.TrySetResult(); + await held.CleanupEntered.Task.WaitAsync(ProofDeadline); + await test.NextSnapshotAsync(); + (await test.ReadAsync(sibling)).Attempts.Should().Be(0, "partition ownership includes asynchronous scope cleanup"); + held.Disposals.Should().Be(0); + _ = held.CancellationToken.WaitHandle; + + held.ReleaseCleanup.TrySetResult(); + await UntilAsync(async () => (await test.ReadAsync(sibling)).ProcessedAt is not null); + await UntilAsync(() => test.Observation.Snapshot.OutstandingPublishes == 0); + var failed = await test.ReadAsync(first); + failed.ProcessedAt.Should().BeNull("late callback success cannot acknowledge a timed-out claim"); + failed.Attempts.Should().Be(1); + failed.Token.Should().BeNull(); + held.Disposals.Should().Be(2, "the released partition subsequently constructs and disposes its sibling's handler"); + } + + [Fact] + public async Task Four_retained_callbacks_stop_all_acquisition_and_releasing_one_restores_capacity() + { + await using var test = await WorkerDatabase.CreateAsync(schema.Postgres); + var held = new List(); + var heldIds = new List(); + for (var index = 0; index < 4; index++) + { + var partition = Guid.CreateVersion7(); + held.Add(test.Callbacks.Hold(partition)); + heldIds.Add(await test.InsertAsync(partition: partition)); + } + var fifth = await test.InsertAsync(); + test.Start(); + await Task.WhenAll(held.Select(item => item.Entered.Task)).WaitAsync(ProofDeadline); + await UntilAsync(() => test.Observation.Snapshot.OutstandingPublishes == 4); + foreach (var id in heldIds) await test.PostponeAsync(id); + await test.NextSnapshotAsync(); + + (await test.ReadAsync(fifth)).Attempts.Should().Be(0, "the four retained callbacks exhaust the global allowance"); + test.Callbacks.Constructions.Should().Be(4); + test.Observation.IsHealthy.Should().BeFalse(); + held[0].ReleaseCallback.TrySetResult(); + await UntilAsync(async () => (await test.ReadAsync(fifth)).ProcessedAt is not null); + test.Observation.Snapshot.OutstandingPublishes.Should().Be(3); + (await test.ReadAsync(heldIds[0])).ProcessedAt.Should().BeNull(); + } + + [Fact] + public async Task Shutdown_uses_one_finite_wait_for_all_four_retained_callbacks() + { + await using var test = await WorkerDatabase.CreateAsync(schema.Postgres); + var time = new ShutdownTimeProvider(Bounds.ShutdownTimeout); + var held = new List(); + for (var index = 0; index < 4; index++) + { + var partition = Guid.CreateVersion7(); + held.Add(test.Callbacks.Hold(partition)); + await test.InsertAsync(partition: partition); + } + test.Start(time: time); + await Task.WhenAll(held.Select(item => item.Entered.Task)).WaitAsync(ProofDeadline); + await UntilAsync(() => test.Observation.Snapshot.OutstandingPublishes == 4); + time.ControlShutdownTimers(); + var stop = test.Worker.StopAsync(CancellationToken.None); + await time.BudgetRegistered.Task.WaitAsync(ProofDeadline); + await test.Worker.ExecuteTask!.WaitAsync(ProofDeadline); + + // The worker has exited, but all callback scopes remain live. Expire + // exactly the first deadline; a fresh drain budget must not get time. + stop.IsCompleted.Should().BeFalse(); + time.Advance(Bounds.ShutdownTimeout); + await stop.WaitAsync(ProofDeadline); + + test.Observation.Snapshot.OutstandingPublishes.Should().Be(4); + foreach (var callback in held) + { + callback.Disposals.Should().Be(0); + callback.CancellationToken.IsCancellationRequested.Should().BeTrue(); + _ = callback.CancellationToken.WaitHandle; + } + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Timeout_and_shutdown_bound_the_wait_and_keep_live_scope_and_CTS_owned_until_completion(bool synchronousBlock) + { + await using var test = await WorkerDatabase.CreateAsync(schema.Postgres); + var partition = Guid.CreateVersion7(); + var held = test.Callbacks.Hold(partition, synchronousBlock: synchronousBlock); + var id = await test.InsertAsync(partition: partition); + test.Start(); + await held.Entered.Task.WaitAsync(ProofDeadline); + await UntilAsync(async () => (await test.ReadAsync(id)).LastError == "publish:timeout:deadline"); + await UntilAsync(() => test.Observation.Snapshot.OutstandingPublishes == 1); + var stop = Stopwatch.StartNew(); + + await test.Worker.StopAsync(CancellationToken.None).WaitAsync(TimeSpan.FromSeconds(2)); + + stop.Elapsed.Should().BeLessThan(TimeSpan.FromSeconds(2)); + held.Disposals.Should().Be(0); + held.CancellationToken.IsCancellationRequested.Should().BeTrue(); + _ = held.CancellationToken.WaitHandle; + held.ReleaseCallback.TrySetResult(); + await held.Completed.Task.WaitAsync(ProofDeadline); + await UntilAsync(() => TokenSourceDisposed(held.CancellationToken)); + held.Disposals.Should().Be(1); + (await test.ReadAsync(id)).ProcessedAt.Should().BeNull(); + } + + [Fact] + public async Task A_failed_terminal_audit_reports_unhealthy_while_the_actual_healthz_remains_live() + { + await using var test = await WorkerDatabase.CreateAsync(schema.Postgres); + test.Start(); + await UntilAsync(() => test.Observation.IsHealthy); + await test.AuditInsertGrantAsync(grant: false); + var id = await test.InsertAsync(type: SensitiveCanary, correlation: SensitiveCanary); + await UntilAsync(() => test.Observation.Snapshot.TerminalAuditFailed); + await UntilAsync(() => test.Logs.Text.Contains("Outbox dispatch iteration failed with outcome iteration_failed.", + StringComparison.Ordinal)); + + test.Observation.IsHealthy.Should().BeFalse(); + test.Logs.Text.Should().NotContain(SensitiveCanary); + var row = await test.ReadAsync(id); + row.TerminalReason.Should().Be("unknown_type"); + row.DeadLetteredAt.Should().BeNull(); + row.ProcessedAt.Should().BeNull(); + (await test.ReadTerminalAuditsAsync()).Should().BeEmpty(); + var health = await test.Services.GetRequiredService().CheckHealthAsync( + registration => registration.Name == OutboxHealthCheck.Name); + health.Entries.Keys.Should().ContainSingle().Which.Should().Be(OutboxHealthCheck.Name); + health.Status.Should().Be(HealthStatus.Unhealthy); + var response = await test.Client.GetAsync("/healthz"); + response.StatusCode.Should().Be(HttpStatusCode.OK); + } + + private static bool TokenSourceDisposed(CancellationToken token) + { + try { _ = token.WaitHandle; return false; } + catch (ObjectDisposedException) { return true; } + } + + private static Task UntilAsync(Func condition, CancellationToken cancellationToken = default) + => UntilAsync(() => Task.FromResult(condition()), cancellationToken); + + private static async Task UntilAsync(Func> condition, CancellationToken cancellationToken = default) + { + using var deadline = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + deadline.CancelAfter(ProofDeadline); + while (!await condition().WaitAsync(deadline.Token)) await Task.Delay(TimeSpan.FromMilliseconds(10), deadline.Token); + } + + [IntegrationEventPartition(nameof(SubjectId))] + public sealed record WorkerEventV1 : IntegrationEventBase + { + public required Guid SubjectId { get; init; } + public override string Topic => "learnstack.test.worker"; + public override string PartitionKey => SubjectId.ToString("D"); + } + + [IntegrationEventPartition(nameof(SubjectId))] + public sealed record UnsubscribedEventV1 : IntegrationEventBase + { + public required Guid SubjectId { get; init; } + public override string Topic => "learnstack.test.unsubscribed"; + public override string PartitionKey => SubjectId.ToString("D"); + } + + /// One discovered, configurable real-bus handler; no production subscription is activated. + public sealed class WorkerHandler(CallbackControls controls, ITenantContextAccessor accessor) + : IIntegrationEventHandler, IAsyncDisposable + { + private readonly CallbackControls _controls = controls.RecordConstruction(); + private CallbackBarrier? _barrier; + + public Task HandleAsync(WorkerEventV1 integrationEvent, CancellationToken cancellationToken) + { + accessor.Current.Should().NotBeNull(); + accessor.Current!.TenantId.Value.Should().Be(integrationEvent.TenantId); + accessor.Current.UserId.Should().Be(UserId.SystemActor); + _barrier = _controls.Enter(integrationEvent.SubjectId, cancellationToken); + if (_barrier.Failure is { } failure) throw failure; + if (_barrier.SynchronousBlock) _barrier.ReleaseCallback.Task.GetAwaiter().GetResult(); + return _barrier.ReleaseCallback.Task; + } + + public async ValueTask DisposeAsync() + { + if (_barrier is null) return; + _barrier.CleanupEntered.TrySetResult(); + await _barrier.ReleaseCleanup.Task; + _barrier.Disposed(); + } + } + + public sealed class CallbackControls + { + private readonly ConcurrentDictionary _partitions = new(); + private int _constructions; + public int Constructions => Volatile.Read(ref _constructions); + + public CallbackBarrier Hold(Guid partition, bool holdCleanup = false, bool synchronousBlock = false) + { + var barrier = new CallbackBarrier(holdCallback: true, holdCleanup, synchronousBlock); + _partitions.TryAdd(partition, barrier).Should().BeTrue(); + return barrier; + } + + public void Fail(Guid partition, Exception failure) => _partitions.TryAdd(partition, + new CallbackBarrier(false, false, false) { Failure = failure }).Should().BeTrue(); + + public CallbackControls RecordConstruction() + { + Interlocked.Increment(ref _constructions); + return this; + } + + public CallbackBarrier Enter(Guid partition, CancellationToken token) + { + var barrier = _partitions.GetOrAdd(partition, _ => new CallbackBarrier(false, false, false)); + barrier.CancellationToken = token; + barrier.Entered.TrySetResult(); + return barrier; + } + + public void ReleaseAll() + { + foreach (var barrier in _partitions.Values) + { + barrier.ReleaseCallback.TrySetResult(); + barrier.ReleaseCleanup.TrySetResult(); + } + } + + public Task AwaitCleanupAsync() => Task.WhenAll(_partitions.Values + .Where(barrier => barrier.Entered.Task.IsCompleted).Select(barrier => barrier.Completed.Task)); + } + + public sealed class CallbackBarrier + { + private int _disposals; + public CallbackBarrier(bool holdCallback, bool holdCleanup, bool synchronousBlock) + { + SynchronousBlock = synchronousBlock; + if (!holdCallback) ReleaseCallback.TrySetResult(); + if (!holdCleanup) ReleaseCleanup.TrySetResult(); + } + + public Exception? Failure { get; init; } + public bool SynchronousBlock { get; } + public int Disposals => Volatile.Read(ref _disposals); + public CancellationToken CancellationToken { get; set; } + public TaskCompletionSource Entered { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + public TaskCompletionSource ReleaseCallback { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + public TaskCompletionSource CleanupEntered { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + public TaskCompletionSource ReleaseCleanup { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + public TaskCompletionSource Completed { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + public void Disposed() + { + Interlocked.Increment(ref _disposals); + Completed.TrySetResult(); + } + } + + private sealed record Row(int Attempts, string? Token, DateTimeOffset? ProcessedAt, + string? TerminalReason, DateTimeOffset? DeadLetteredAt, string? LastError, + DateTimeOffset AvailableAfter, string Type, string Payload); + private sealed record TerminalAudit(string OperationType, string OperationClass, string Correlation, string Metadata); + + private sealed class SnapshotTimeProvider : TimeProvider + { + private long _timestamp; + public override long TimestampFrequency => TimeSpan.TicksPerSecond; + public override long GetTimestamp() => Interlocked.Read(ref _timestamp); + public void Advance(TimeSpan duration) => Interlocked.Add(ref _timestamp, duration.Ticks); + } + + /// Controls shutdown deadlines while all other host timers keep running normally. + private sealed class ShutdownTimeProvider(TimeSpan shutdownTimeout) : TimeProvider + { + private readonly object _gate = new(); + private readonly List _timers = []; + private bool _controlled; + private long? _origin; + private TimeSpan _elapsed; + private TimeSpan _lastRegistration; + public TaskCompletionSource BudgetRegistered { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + + public void ControlShutdownTimers() { lock (_gate) _controlled = true; } + + public override ITimer CreateTimer(TimerCallback callback, object? state, TimeSpan dueTime, TimeSpan period) + { + lock (_gate) + { + if (!_controlled || dueTime != shutdownTimeout) + return System.CreateTimer(callback, state, dueTime, period); + period.Should().Be(Timeout.InfiniteTimeSpan); + _origin ??= Stopwatch.GetTimestamp(); + var registration = _timers.Count == 0 ? TimeSpan.Zero : Stopwatch.GetElapsedTime(_origin.Value); + // Preserve the real ordering even below TimeSpan's resolution: + // a later budget cannot expire at the first budget's deadline. + if (_timers.Count > 0 && registration <= _lastRegistration) + registration = _lastRegistration + TimeSpan.FromTicks(1); + _lastRegistration = registration; + var timer = new ShutdownTimer(this, callback, state, + (_elapsed > registration ? _elapsed : registration) + dueTime); + _timers.Add(timer); + BudgetRegistered.TrySetResult(); + return timer; + } + } + + public void Advance(TimeSpan duration) + { + ShutdownTimer[] due; + lock (_gate) + { + _elapsed += duration; + due = _timers.Where(timer => timer.Due is { } at && at <= _elapsed).ToArray(); + foreach (var timer in due) timer.Due = null; + } + foreach (var timer in due) timer.Fire(); + } + + private sealed class ShutdownTimer(ShutdownTimeProvider owner, TimerCallback callback, + object? state, TimeSpan due) : ITimer + { + private bool _disposed; + public TimeSpan? Due { get; set; } = due; + public void Fire() => callback(state); + public bool Change(TimeSpan dueTime, TimeSpan period) + { + lock (owner._gate) + { + if (_disposed) return false; + period.Should().Be(Timeout.InfiniteTimeSpan); + Due = dueTime == Timeout.InfiniteTimeSpan ? null : owner._elapsed + dueTime; + return true; + } + } + public void Dispose() + { + lock (owner._gate) { _disposed = true; Due = null; } + } + public ValueTask DisposeAsync() { Dispose(); return ValueTask.CompletedTask; } + } + } + + private sealed class WorkerApiFactory(WorkerDatabase database, string? credential, + OutboxDispatchOptions? bounds = null, TimeProvider? time = null) : WebApplicationFactory + { + protected override void ConfigureWebHost(IWebHostBuilder builder) + { + builder.UseEnvironment(Environments.Development); + builder.UseSetting("ConnectionStrings:Default", database.AppConnectionString); + builder.UseSetting("ConnectionStrings:PlatformAdmin", database.PlatformConnectionString); + builder.UseSetting("ConnectionStrings:OutboxDispatcher", credential); + builder.ConfigureTestServices(services => + { + services.RemoveAll(); + services.AddSingleton(bounds ?? Bounds); + if (time is not null) + { + services.RemoveAll(); + services.AddSingleton(time); + } + services.RemoveAll(); + var handlers = IntegrationEventHandlerRegistry.Discover(typeof(WorkerHandler).Assembly); + handlers.All.Should().ContainSingle().Which.HandlerType.Should().Be(); + services.AddSingleton(handlers); + services.RemoveAll(); + services.AddSingleton(OutboxEventRegistry.Create([ + OutboxEventRegistration.Required(typeof(WorkerEventV1), + [OutboxSubscriptionManifest.Active("test.worker.v1", typeof(WorkerHandler))]), + OutboxEventRegistration.IntentionallyUnsubscribed(typeof(UnsubscribedEventV1)), + OutboxEventRegistration.Required(typeof(TenantCreatedIntegrationEventV1), [ + OutboxSubscriptionManifest.Pending("customization.initialize-content-type-default.v1"), + OutboxSubscriptionManifest.Pending("customization.initialize-taxonomy-default.v1"), + ]), + ], [], handlers, [typeof(WorkerEventV1).Assembly, typeof(TenantCreatedIntegrationEventV1).Assembly])); + services.AddSingleton(database.Callbacks); + services.AddScoped(); + services.AddSingleton>(database.Logs); + }); + } + } + + private sealed class WorkerDatabase : IAsyncDisposable + { + private readonly DisposableSchemaDatabase _database; + private readonly NpgsqlDataSource _appSource; + private readonly NpgsqlDataSource _dispatchSource; + private WorkerApiFactory? _factory; + private HttpClient? _client; + private int _insertSequence; + private WorkerDatabase(DisposableSchemaDatabase database) + { + _database = database; + _appSource = NpgsqlDataSource.Create(database.AppConnectionString); + _dispatchSource = OutboxDispatcherDataSource.Build(database.OutboxConnectionString); + } + + public string AppConnectionString => _database.AppConnectionString; + public string PlatformConnectionString => _database.PlatformConnectionString; + public string OutboxConnectionString => _database.OutboxConnectionString; + public CallbackControls Callbacks { get; } = new(); + public RecordingLogger Logs { get; } = new(); + public IServiceProvider Services => _factory!.Services; + public OutboxProcessor Worker => Services.GetServices().OfType().Single(); + public OutboxObservation Observation => Services.GetRequiredService(); + public HttpClient Client => _client!; + + public static async Task CreateAsync(PostgresFixture postgres, bool seedTenant = true) + { + var test = new WorkerDatabase(await DisposableSchemaDatabase.CreateAsync(postgres)); + try + { + if (seedTenant) await test.SeedTenantAsync(); + return test; + } + catch { await test.DisposeAsync(); throw; } + } + + public void Start(OutboxDispatchOptions? bounds = null, TimeProvider? time = null) + { + _factory = new WorkerApiFactory(this, OutboxConnectionString, bounds, time); + _client = _factory.CreateClient(new WebApplicationFactoryClientOptions { BaseAddress = new Uri("http://localhost") }); + Services.GetRequiredService().Should().BeOfType(); + } + + public async Task NextSnapshotAsync() + { + var before = Observation.Snapshot.LastSnapshotTimestamp ?? long.MinValue; + await UntilAsync(() => Observation.Snapshot.LastSnapshotTimestamp > before); + } + + public async Task InsertAsync(Guid? partition = null, string? type = null, + string? payload = null, string? correlation = null, IntegrationEventBase? fact = null) + { + fact ??= new WorkerEventV1 + { + EventId = Guid.CreateVersion7(), + TenantId = SchemaFixture.TenantA, + OccurredAt = Occurred, + SubjectId = partition ?? Guid.CreateVersion7(), + }; + await using var connection = await _appSource.OpenConnectionAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await SchemaQueries.SetTenantAsync(connection, transaction, fact.TenantId); + await using var insert = new NpgsqlCommand(""" + INSERT INTO outbox_messages + (id, tenant_id, occurred_at, correlation_id, type, topic, partition_key, payload, available_after) + VALUES (@id, @tenant, @occurred, @correlation, @type, @topic, @partition, @payload, @available) + """, connection, transaction); + insert.Parameters.AddWithValue("id", fact.EventId); + insert.Parameters.AddWithValue("tenant", fact.TenantId); + insert.Parameters.AddWithValue("occurred", fact.OccurredAt); + insert.Parameters.AddWithValue("correlation", correlation ?? Traceparent); + insert.Parameters.AddWithValue("type", type ?? fact.GetType().AssemblyQualifiedName!); + insert.Parameters.AddWithValue("topic", fact.Topic); + insert.Parameters.AddWithValue("partition", fact.PartitionKey); + insert.Parameters.Add(new NpgsqlParameter("payload", NpgsqlDbType.Jsonb) { Value = payload ?? fact.ToPayloadJson() }); + insert.Parameters.AddWithValue("available", Occurred.AddMilliseconds(Interlocked.Increment(ref _insertSequence))); + (await insert.ExecuteNonQueryAsync()).Should().Be(1); + await transaction.CommitAsync(); + return fact.EventId; + } + + public async Task ReadAsync(Guid id) + { + await using var connection = await _dispatchSource.OpenConnectionAsync(); + await using var read = new NpgsqlCommand(""" + SELECT attempts, locked_by, processed_at, terminal_reason, dead_lettered_at, + last_error, available_after, type, payload::text FROM outbox_messages WHERE id = @id + """, connection); + read.Parameters.AddWithValue("id", id); + await using var reader = await read.ExecuteReaderAsync(); + (await reader.ReadAsync()).Should().BeTrue(); + return new Row(reader.GetInt32(0), Text(reader, 1), Timestamp(reader, 2), Text(reader, 3), + Timestamp(reader, 4), Text(reader, 5), reader.GetFieldValue(6), reader.GetString(7), reader.GetString(8)); + } + + public async Task SingleIdAsync() + { + await using var connection = await _dispatchSource.OpenConnectionAsync(); + await using var read = new NpgsqlCommand("SELECT id FROM outbox_messages", connection); + await using var reader = await read.ExecuteReaderAsync(); + (await reader.ReadAsync()).Should().BeTrue(); + var id = reader.GetGuid(0); + (await reader.ReadAsync()).Should().BeFalse(); + return id; + } + + public async Task LeaseIsLiveAsync(Guid id) + { + await using var connection = await _dispatchSource.OpenConnectionAsync(); + await using var read = new NpgsqlCommand( + "SELECT locked_until > clock_timestamp() FROM outbox_messages WHERE id = @id", connection); + read.Parameters.AddWithValue("id", id); + return await read.ExecuteScalarAsync() is true; + } + + public async Task PostponeAsync(Guid id) + { + await using var connection = await _dispatchSource.OpenConnectionAsync(); + await using var update = new NpgsqlCommand( + "UPDATE outbox_messages SET available_after = clock_timestamp() + interval '1 hour' WHERE id = @id", connection); + update.Parameters.AddWithValue("id", id); + (await update.ExecuteNonQueryAsync()).Should().Be(1); + } + + public async Task AuditInsertGrantAsync(bool grant) + { + await using var connection = await PostgresFixture.OpenAsync(_database.MigrationConnectionString); + await using var command = connection.CreateCommand(); + command.CommandText = grant ? "GRANT INSERT ON audit_log TO learnstack_outbox_admin" + : "REVOKE INSERT ON audit_log FROM learnstack_outbox_admin"; + await command.ExecuteNonQueryAsync(); + } + + public async Task> ReadTerminalAuditsAsync() + { + await using var connection = await _appSource.OpenConnectionAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await SchemaQueries.SetTenantAsync(connection, transaction, SchemaFixture.TenantA); + await using var read = new NpgsqlCommand(""" + SELECT operation_type, operation_class, correlation_id, metadata::text + FROM audit_log WHERE operation = 'platform.outbox.dead_letter' + """, connection, transaction); + var rows = new List(); + await using var reader = await read.ExecuteReaderAsync(); + while (await reader.ReadAsync()) rows.Add(new TerminalAudit(reader.GetString(0), reader.GetString(1), reader.GetString(2), reader.GetString(3))); + return rows; + } + + public async ValueTask DisposeAsync() + { + Callbacks.ReleaseAll(); + if (_factory is not null) + { + await Worker.StopAsync(CancellationToken.None).WaitAsync(ProofDeadline); + await Callbacks.AwaitCleanupAsync().WaitAsync(ProofDeadline); + _client?.Dispose(); + await _factory.DisposeAsync(); + } + await _dispatchSource.DisposeAsync(); + await _appSource.DisposeAsync(); + await _database.DisposeAsync(); + } + + private async Task SeedTenantAsync() + { + await using var connection = await _appSource.OpenConnectionAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await SchemaQueries.SetTenantAsync(connection, transaction, SchemaFixture.TenantA); + await SchemaQueries.ExecuteAsync(connection, transaction, """ + INSERT INTO tenants (id, slug, display_name, status, created_at, created_by, row_version) + VALUES (@tenant, 'worker-probe', 'Worker probe', 'Trial', now(), @actor, 0); + INSERT INTO organizations (id, tenant_id, slug, display_name, status, created_at, created_by, row_version) + VALUES (@organization, @tenant, 'hq', 'HQ', 'Active', now(), @actor, 0); + """, ("tenant", SchemaFixture.TenantA), ("organization", SchemaFixture.OrgA1), ("actor", UserId.SystemActor.Value)); + await transaction.CommitAsync(); + } + + private static string? Text(NpgsqlDataReader reader, int ordinal) => reader.IsDBNull(ordinal) ? null : reader.GetString(ordinal); + private static DateTimeOffset? Timestamp(NpgsqlDataReader reader, int ordinal) => + reader.IsDBNull(ordinal) ? null : reader.GetFieldValue(ordinal); + } + + private sealed class DispatcherTraces : IDisposable + { + private readonly ActivityListener _listener; + private readonly ConcurrentQueue _records = new(); + public DispatcherTraces() + { + _listener = new ActivityListener + { + ShouldListenTo = source => source.Name == OutboxObservation.InstrumentationName, + Sample = (ref ActivityCreationOptions _) => ActivitySamplingResult.AllDataAndRecorded, + ActivityStopped = activity => _records.Enqueue(string.Join("\n", new[] + { + activity.DisplayName, activity.Id, activity.ParentId, activity.TraceStateString, + activity.StatusDescription, + string.Join(";", activity.TagObjects.Select(tag => tag.Key + "=" + tag.Value)), + string.Join(";", activity.Events.Select(@event => @event.Name + ":" + + string.Join(",", @event.Tags.Select(tag => tag.Key + "=" + tag.Value)))), + })), + }; + ActivitySource.AddActivityListener(_listener); + } + public IReadOnlyCollection Records => _records.ToArray(); + public void Dispose() => _listener.Dispose(); + } + + private sealed class RecordingLogger : ILogger + { + private readonly ConcurrentQueue _messages = new(); + public string Text => string.Join('\n', _messages); + public IDisposable? BeginScope(TState state) where TState : notnull => null; + public bool IsEnabled(LogLevel logLevel) => true; + public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, + Func formatter) => _messages.Enqueue(formatter(state, exception) + exception); + } +} diff --git a/backend/tests/LearnStack.Tests.Integration/Database/OutboxProducerTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/OutboxProducerTests.cs index 549cb9fd..4dcb6719 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/OutboxProducerTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/OutboxProducerTests.cs @@ -55,7 +55,7 @@ public async Task Both_roots_insert_the_complete_runtime_event_before_business_a await using var dataSource = NpgsqlDataSource.Create(database.AppConnectionString); var context = new ActingContext(OrganizationId.From(SchemaFixture.OrgA1)); await using var seed = Compose(dataSource, context); - await using var host = new WebApplicationFactory().WithWebHostBuilder(builder => + await using var host = new ConstructionOnlyApiFactory().WithWebHostBuilder(builder => { builder.UseSetting("ConnectionStrings:Default", database.AppConnectionString); builder.ConfigureServices(services => services.AddSingleton( @@ -153,7 +153,7 @@ public async Task Enqueue_and_owner_completion_never_publish_to_the_transport() { await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); var transport = new CountingEventBus(); - await using var host = new WebApplicationFactory().WithWebHostBuilder(builder => + await using var host = new ConstructionOnlyApiFactory().WithWebHostBuilder(builder => { builder.UseSetting("ConnectionStrings:Default", database.AppConnectionString); builder.ConfigureServices(services => diff --git a/backend/tests/LearnStack.Tests.Integration/Database/PlatformAdminScopeTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/PlatformAdminScopeTests.cs index 029cbd46..8050a9f9 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/PlatformAdminScopeTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/PlatformAdminScopeTests.cs @@ -711,6 +711,7 @@ private static IServiceScopeFactory AuditStoreScopeFactory() => .AddLogging() .AddScoped() .AddSingleton() + .AddSingleton(new AuditCatalog([new TenancyAuditCatalogSource()])) .AddSingleton(new Lazy( () => throw new InvalidOperationException( "The platform-scope row is written on the scope's own connection, so " @@ -741,6 +742,14 @@ public Task WriteBestEffortAsync( AuditEntryDraft entry, CancellationToken cancellationToken = default) => throw new NotSupportedException(); + public Task WriteEventTerminalAsync( + AuditEntryDraft entry, + EventTerminalTransition transition, + System.Data.Common.DbConnection connection, + System.Data.Common.DbTransaction transaction, + CancellationToken cancellationToken = default) => + throw new NotSupportedException("The terminal-event path has its own cases."); + public Task WritePlatformScopeAsync( AuditEntryDraft entry, System.Data.Common.DbConnection connection, diff --git a/backend/tests/LearnStack.Tests.Integration/Database/PlatformSchemaTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/PlatformSchemaTests.cs index 18953605..27fdc994 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/PlatformSchemaTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/PlatformSchemaTests.cs @@ -80,14 +80,13 @@ public async Task TheApplicationRoleCanOnlyEnqueue(string statement) } [Fact] - public async Task TheDispatcherHoldsExactlyTheFourColumnUpdateGrant() + public async Task TheDispatcherHoldsExactlyTheEightStatusColumnUpdateGrant() { // BYPASSRLS bypasses policies, not GRANTs, so this column list is the // whole of the bound on learnstack_outbox_admin. SELECT ... FOR UPDATE // SKIP LOCKED works with a column-level grant, so no table-wide UPDATE is - // needed — and when locked_by / locked_until land in Phase 02b, that - // migration extends this grant or the dispatcher fails at runtime with - // `permission denied for table`. + // needed. ADR-0046 extends the original four columns only with the lease + // pair and terminal request/commit markers; payload and identity stay read-only. await using var connection = await PostgresFixture.OpenAsync(_schema.Postgres.MigrationConnectionString); await using var command = new NpgsqlCommand( """ @@ -99,7 +98,7 @@ FROM information_schema.column_privileges """, (NpgsqlConnection)connection); (await command.ExecuteScalarAsync()).Should() - .Be("attempts,available_after,last_error,processed_at"); + .Be("attempts,available_after,dead_lettered_at,last_error,locked_by,locked_until,processed_at,terminal_reason"); } [Fact] diff --git a/backend/tests/LearnStack.Tests.Integration/Database/ProducerExecutionTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/ProducerExecutionTests.cs index 07914ece..1e786492 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/ProducerExecutionTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/ProducerExecutionTests.cs @@ -452,7 +452,7 @@ public async Task Both_compositions_reconcile_an_independent_scope_inside_anothe await SeedTenantAsync(database.AppConnectionString); await using var dataSource = NpgsqlDataSource.Create(database.AppConnectionString); await using var seed = Compose(dataSource); - await using var host = new WebApplicationFactory().WithWebHostBuilder(builder => + await using var host = new ConstructionOnlyApiFactory().WithWebHostBuilder(builder => { builder.UseSetting("ConnectionStrings:Default", database.AppConnectionString); builder.ConfigureServices(services => services.AddSingleton( diff --git a/backend/tests/LearnStack.Tests.Integration/Database/PublicLocalLaunchTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/PublicLocalLaunchTests.cs index 35c588b4..636f9a84 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/PublicLocalLaunchTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/PublicLocalLaunchTests.cs @@ -11,7 +11,7 @@ namespace LearnStack.Tests.Integration.Database; -/// Exercises the actual local API launcher with only the application credential. +/// Exercises the actual local API launcher with explicit application and dispatcher credentials. [Collection(PublicReadTestGroup.Name)] [Trait(RequiresDocker.Key, RequiresDocker.Value)] public sealed class PublicLocalLaunchTests(PublicReadFixture fixture) @@ -107,6 +107,7 @@ await File.WriteAllTextAsync(script, """ process.StartInfo.ArgumentList.Add("api"); } process.StartInfo.Environment["ConnectionStrings__Default"] = fixture.AppConnectionString; + process.StartInfo.Environment["ConnectionStrings__OutboxDispatcher"] = fixture.OutboxConnectionString; process.StartInfo.Environment["LEARNSTACK_PUBLIC_API_ORIGIN"] = "http://127.0.0.1:0"; process.StartInfo.Environment["LEARNSTACK_PUBLIC_HOP_SECRET"] = Secret; // API configuration validates paths but does not open renderer TLS files. diff --git a/backend/tests/LearnStack.Tests.Integration/Database/PublicLocaleEvolutionHttpTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/PublicLocaleEvolutionHttpTests.cs index 4f85383d..db972ad6 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/PublicLocaleEvolutionHttpTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/PublicLocaleEvolutionHttpTests.cs @@ -32,8 +32,8 @@ namespace LearnStack.Tests.Integration.Database; [Collection(SharedSchema.Name)] [Trait(RequiresDocker.Key, RequiresDocker.Value)] -public sealed class PublicLocaleEvolutionHttpTests(SchemaFixture schema, WebApplicationFactory factory) - : IClassFixture> +public sealed class PublicLocaleEvolutionHttpTests(SchemaFixture schema, ConstructionOnlyApiFactory factory) + : IClassFixture { private const string Root = "/api/v1/public/courses"; private const string Site = "/api/v1/public/site"; diff --git a/backend/tests/LearnStack.Tests.Integration/Database/PublicReadFixture.cs b/backend/tests/LearnStack.Tests.Integration/Database/PublicReadFixture.cs index f52dd5d0..3ef2d216 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/PublicReadFixture.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/PublicReadFixture.cs @@ -26,10 +26,11 @@ public sealed class PublicReadTestGroup : ICollectionFixture } /// Production API composition, seeded through real writers in an isolated four-role database. -public sealed class PublicReadFixture : WebApplicationFactory, IAsyncLifetime +public sealed class PublicReadFixture : ConstructionOnlyApiFactory, IAsyncLifetime { private readonly PostgresFixture _postgres = new(); public string AppConnectionString => _postgres.AppConnectionString; + public string OutboxConnectionString => _postgres.OutboxConnectionString; public PublicReadObservation Observation { get; } = new(); public async Task InitializeAsync() @@ -68,7 +69,7 @@ protected override void Dispose(bool disposing) } } - protected override void ConfigureWebHost(IWebHostBuilder builder) + protected override void ConfigureTestHost(IWebHostBuilder builder) { ArgumentNullException.ThrowIfNull(builder); builder.UseEnvironment(Environments.Development); diff --git a/backend/tests/LearnStack.Tests.Integration/Database/PublicReadTransactionTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/PublicReadTransactionTests.cs index 136e2ca2..b0383421 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/PublicReadTransactionTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/PublicReadTransactionTests.cs @@ -315,7 +315,7 @@ private ServiceProvider BuildProvider(ILoggerFactory? injectedLogs = null) private static PostgresAuditStore Store(IServiceProvider services, AuditStateCapture capture) => new( capture, new Lazy(services.GetRequiredService), NullLogger.Instance, - services.GetRequiredService(), new AuditHealth()); + services.GetRequiredService(), new AuditHealth(), new AuditCatalog([])); private static async Task AssertRoleAsync(IUnitOfWork unit) { diff --git a/backend/tests/LearnStack.Tests.Integration/Database/TenancySchemaTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/TenancySchemaTests.cs index 9aaed12b..caf1b085 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/TenancySchemaTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/TenancySchemaTests.cs @@ -1227,11 +1227,12 @@ AND grantee IN ('learnstack_app', 'learnstack_platform', 'learnstack_outbox_admi "learnstack_platform tenant_level_taxonomy_items SELECT", "learnstack_platform customization_generations SELECT", "learnstack_outbox_admin outbox_messages SELECT", + "learnstack_outbox_admin audit_log INSERT", // The Audit chain. learnstack_platform's column-restricted UPDATE on // audit_log does NOT appear here — information_schema.role_table_grants // reports table-level privileges only — so the six-column list has its own - // assertion in AuditSchemaTests, the way the dispatcher's four-column grant + // assertion in AuditSchemaTests, the way the dispatcher's eight-column grant // on outbox_messages does in PlatformSchemaTests. A matrix row missing its // column half reads as a narrower grant than the one that shipped. "learnstack_app audit_log INSERT,SELECT", @@ -1248,6 +1249,6 @@ AND grantee IN ('learnstack_app', 'learnstack_platform', 'learnstack_outbox_admi "learnstack_platform lesson_translations SELECT", ], "every line is one line of the migration chains' grant matrices, and " - + "learnstack_outbox_admin holds nothing beyond the outbox"); + + "learnstack_outbox_admin holds only outbox SELECT/status UPDATE and terminal audit INSERT"); } } diff --git a/backend/tests/LearnStack.Tests.Integration/Database/TenancyWriterTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/TenancyWriterTests.cs index 42209fab..c1d1b1b6 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/TenancyWriterTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/TenancyWriterTests.cs @@ -32,8 +32,8 @@ namespace LearnStack.Tests.Integration.Database; [Trait(RequiresDocker.Key, RequiresDocker.Value)] [Collection(SharedSchema.Name)] -public sealed class TenancyWriterTests(SchemaFixture schema, WebApplicationFactory factory) - : IClassFixture> +public sealed class TenancyWriterTests(SchemaFixture schema, ConstructionOnlyApiFactory factory) + : IClassFixture { private const string Theme = """{"primary":"#2345aa","background":"#ffffff","foreground":"#111111","muted":"#555555"}"""; private const string OtherTheme = """{"primary":"#663399","background":"#ffffff","foreground":"#000000","muted":"#444444"}"""; diff --git a/backend/tests/LearnStack.Tests.Integration/Database/TenantAssertionAuditTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/TenantAssertionAuditTests.cs index db2c97d5..bdbdf611 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/TenantAssertionAuditTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/TenantAssertionAuditTests.cs @@ -48,7 +48,8 @@ public async Task An_authenticated_mismatch_lands_a_denied_row_the_tenant_can_re new Lazy(() => dataSource), NullLogger.Instance, MeterFactory, - new AuditHealth()); + new AuditHealth(), + new AuditCatalog([])); var recorder = Recorder(store, threshold: 10); @@ -99,7 +100,8 @@ public async Task An_anonymous_burst_lands_one_row_however_long_the_run_is() new Lazy(() => dataSource), NullLogger.Instance, MeterFactory, - new AuditHealth()); + new AuditHealth(), + new AuditCatalog([])); var recorder = Recorder(store, threshold: 3); diff --git a/backend/tests/LearnStack.Tests.Integration/Database/TenantCreatedProducerTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/TenantCreatedProducerTests.cs index a61c1075..e592eca8 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/TenantCreatedProducerTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/TenantCreatedProducerTests.cs @@ -44,7 +44,7 @@ public async Task Provisioning_business_outbox_and_both_MUST_rows_share_one_comm await using var source = NpgsqlDataSource.Create(database.AppConnectionString); await using var seed = SeedComposition.Build(source, null, NullLoggerFactory.Instance); - await using var host = new WebApplicationFactory().WithWebHostBuilder(builder => + await using var host = new ConstructionOnlyApiFactory().WithWebHostBuilder(builder => { builder.UseSetting("ConnectionStrings:Default", database.AppConnectionString); builder.ConfigureServices(services => services.AddSingleton( diff --git a/backend/tests/LearnStack.Tests.Integration/Database/TenantIsolationHttpTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/TenantIsolationHttpTests.cs index a541fde4..0ee86fc8 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/TenantIsolationHttpTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/TenantIsolationHttpTests.cs @@ -326,7 +326,7 @@ private async Task> GetAsync(string host, string route) /// SeedRunner and serves HTTP, and sharing would make the schema suite's exact row /// counts depend on whether these tests ran first. /// -public sealed class TenantIsolationFixture : WebApplicationFactory, IAsyncLifetime +public sealed class TenantIsolationFixture : ConstructionOnlyApiFactory, IAsyncLifetime { private readonly PostgresFixture _postgres = new(); @@ -442,7 +442,7 @@ public HttpClient ClientFor(string host) return client; } - protected override void ConfigureWebHost(IWebHostBuilder builder) + protected override void ConfigureTestHost(IWebHostBuilder builder) { ArgumentNullException.ThrowIfNull(builder); builder.UseEnvironment(Environments.Development); diff --git a/backend/tests/LearnStack.Tests.Integration/DemoManifestTests.cs b/backend/tests/LearnStack.Tests.Integration/DemoManifestTests.cs index 33902e19..4a3886ee 100644 --- a/backend/tests/LearnStack.Tests.Integration/DemoManifestTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/DemoManifestTests.cs @@ -1,7 +1,6 @@ using System.Text.Json; using FluentAssertions; using LearnStack.Tools.Seeder; -using Npgsql; using Xunit; namespace LearnStack.Tests.Integration; @@ -60,53 +59,4 @@ public void Demo_manifest_covers_every_enabled_seed_lane_and_nine_distinct_real_ manifest.Select(row => row.Fields.Count).Distinct().Should().HaveCount(2, "the browser proof must observe two genuinely distinct authored shapes"); } - - [Theory] - [InlineData("Default", "Username=learnstack_platform")] - [InlineData("Migration", "Username=learnstack_app")] - [InlineData("Default", "Host=remote.example")] - [InlineData("Default", "Port=5433")] - [InlineData("Default", "Database=other")] - [InlineData("Default", "Password=stale")] - public void Demo_configuration_refuses_stale_or_privileged_connection_values(string key, string replacement) - { - var values = Configuration(); - DemoDatabaseConfiguration.IsValid(values).Should().BeTrue(); - values[$"ConnectionStrings__{key}"] += ";" + replacement; - DemoDatabaseConfiguration.IsValid(values).Should().BeFalse(); - } - - [Fact] - public void Demo_configuration_refuses_missing_and_malformed_values() - { - DemoDatabaseConfiguration.IsValid(new Dictionary()).Should().BeFalse(); - var values = Configuration(); - values["ConnectionStrings__Default"] = "not a connection string"; - DemoDatabaseConfiguration.IsValid(values).Should().BeFalse(); - } - - private static Dictionary Configuration() - { - var values = new Dictionary(StringComparer.Ordinal) { ["POSTGRES_DB"] = "demo" }; - foreach (var (suffix, role, key) in new[] - { - ("Migration", "learnstack_migration", "LEARNSTACK_MIGRATION_PW"), - ("Default", "learnstack_app", "LEARNSTACK_APP_PW"), - ("PlatformAdmin", "learnstack_platform", "LEARNSTACK_PLATFORM_PW"), - ("OutboxDispatcher", "learnstack_outbox_admin", "LEARNSTACK_OUTBOX_PW"), - }) - { - // Ephemeral parser inputs only: no service uses these credentials. - var password = Guid.NewGuid().ToString("N"); - values[key] = password; - values[$"ConnectionStrings__{suffix}"] = new NpgsqlConnectionStringBuilder - { - Host = "localhost", - Database = "demo", - Username = role, - Password = password, - }.ConnectionString; - } - return values; - } } diff --git a/backend/tests/LearnStack.Tests.Integration/DeploymentModeCompositionTests.cs b/backend/tests/LearnStack.Tests.Integration/DeploymentModeCompositionTests.cs index 0edc79b1..66994c25 100644 --- a/backend/tests/LearnStack.Tests.Integration/DeploymentModeCompositionTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/DeploymentModeCompositionTests.cs @@ -133,7 +133,7 @@ public void Tenant_Context_Resolution_Forwards_Each_Access_To_The_Accessor() /// itself reads. /// private static WebApplicationFactory For(string mode) => - new WebApplicationFactory().WithWebHostBuilder(builder => + new ConstructionOnlyApiFactory().WithWebHostBuilder(builder => { builder.UseSetting("Deployment:Mode", mode); builder.UseSetting("ErrorTracking:Sentry:Dsn", DevelopmentShapedDsn); diff --git a/backend/tests/LearnStack.Tests.Integration/ErrorShapeHttpTests.cs b/backend/tests/LearnStack.Tests.Integration/ErrorShapeHttpTests.cs index 7d74975e..8a44ac33 100644 --- a/backend/tests/LearnStack.Tests.Integration/ErrorShapeHttpTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/ErrorShapeHttpTests.cs @@ -345,9 +345,9 @@ private static async Task AssertProblemAsync( } } -public sealed class ErrorShapeFixture : WebApplicationFactory +public sealed class ErrorShapeFixture : ConstructionOnlyApiFactory { - protected override void ConfigureWebHost(IWebHostBuilder builder) + protected override void ConfigureTestHost(IWebHostBuilder builder) { ArgumentNullException.ThrowIfNull(builder); builder.UseEnvironment(Environments.Development); diff --git a/backend/tests/LearnStack.Tests.Integration/HostClassificationHttpTests.cs b/backend/tests/LearnStack.Tests.Integration/HostClassificationHttpTests.cs index 93470d95..5468047d 100644 --- a/backend/tests/LearnStack.Tests.Integration/HostClassificationHttpTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/HostClassificationHttpTests.cs @@ -305,7 +305,7 @@ public async Task An_Unclassified_Prefix_Is_Served_Whatever_Its_Host() } /// A host whose resolver is a stub, so no database is involved. -public sealed class HostClassificationFixture : WebApplicationFactory +public sealed class HostClassificationFixture : ConstructionOnlyApiFactory { public const string OrganizationHost = "branch.example.com"; public const string TenantHost = "school.example.com"; @@ -318,7 +318,7 @@ public sealed class HostClassificationFixture : WebApplicationFactory public StubResolver Resolver { get; } = new(); - protected override void ConfigureWebHost(IWebHostBuilder builder) + protected override void ConfigureTestHost(IWebHostBuilder builder) { ArgumentNullException.ThrowIfNull(builder); builder.UseEnvironment(Environments.Development); diff --git a/backend/tests/LearnStack.Tests.Integration/IdempotencyHttpTests.cs b/backend/tests/LearnStack.Tests.Integration/IdempotencyHttpTests.cs index b0baf7b1..ce9ae6c5 100644 --- a/backend/tests/LearnStack.Tests.Integration/IdempotencyHttpTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/IdempotencyHttpTests.cs @@ -508,7 +508,7 @@ private async Task PostAsync( /// which made every test in the class depend on the order the others ran in — /// Standards 06 § Forbidden bars exactly that. /// -public sealed class IdempotencyFixture : WebApplicationFactory +public sealed class IdempotencyFixture : ConstructionOnlyApiFactory { public static readonly Guid TenantA = Guid.Parse("018f4d40-0000-7000-8000-00000000000a"); public static readonly Guid TenantB = Guid.Parse("018f4d40-0000-7000-8000-00000000000b"); @@ -518,7 +518,7 @@ public sealed class IdempotencyFixture : WebApplicationFactory internal const string TenantHeader = "X-Test-Tenant"; internal const string UserHeader = "X-Test-User"; - protected override void ConfigureWebHost(IWebHostBuilder builder) + protected override void ConfigureTestHost(IWebHostBuilder builder) { ArgumentNullException.ThrowIfNull(builder); builder.UseEnvironment(Environments.Development); diff --git a/backend/tests/LearnStack.Tests.Integration/RateLimitingHttpTests.cs b/backend/tests/LearnStack.Tests.Integration/RateLimitingHttpTests.cs index ad6f7b33..c82b9de4 100644 --- a/backend/tests/LearnStack.Tests.Integration/RateLimitingHttpTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/RateLimitingHttpTests.cs @@ -115,9 +115,9 @@ public async Task A_Forwarded_For_Header_Does_Not_Buy_A_Fresh_Budget() /// A host used by nothing else, so its exhausted budget poisons nothing. -public class RateLimitedHostFixture : WebApplicationFactory +public class RateLimitedHostFixture : ConstructionOnlyApiFactory { - protected override void ConfigureWebHost(IWebHostBuilder builder) + protected override void ConfigureTestHost(IWebHostBuilder builder) { ArgumentNullException.ThrowIfNull(builder); builder.UseEnvironment(Environments.Development); diff --git a/backend/tests/LearnStack.Tests.Integration/SmokeTests.cs b/backend/tests/LearnStack.Tests.Integration/SmokeTests.cs index e53dc60b..d95d94a9 100644 --- a/backend/tests/LearnStack.Tests.Integration/SmokeTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/SmokeTests.cs @@ -5,13 +5,13 @@ namespace LearnStack.Tests.Integration; -public sealed class SmokeTests : IClassFixture> +public sealed class SmokeTests : IClassFixture { private static readonly Uri HealthzPath = new("/healthz", UriKind.Relative); private readonly WebApplicationFactory _factory; - public SmokeTests(WebApplicationFactory factory) + public SmokeTests(ConstructionOnlyApiFactory factory) { _factory = factory; } diff --git a/backend/tests/LearnStack.Tests.Integration/TenantAssertionHttpTests.cs b/backend/tests/LearnStack.Tests.Integration/TenantAssertionHttpTests.cs index f7e7d2be..b19fd067 100644 --- a/backend/tests/LearnStack.Tests.Integration/TenantAssertionHttpTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/TenantAssertionHttpTests.cs @@ -319,9 +319,9 @@ public async Task The_Same_Request_Without_The_Header_Is_Served() /// public sealed class TenantWideFixture : ResolvedTenantFixture { - protected override void ConfigureWebHost(IWebHostBuilder builder) + protected override void ConfigureTestHost(IWebHostBuilder builder) { - base.ConfigureWebHost(builder); + base.ConfigureTestHost(builder); builder.ConfigureTestServices(services => { services.RemoveAll(); @@ -468,13 +468,13 @@ public void A_Threshold_Below_One_Is_Refused_At_Boot() /// misconfigured variants take their override through /// , which a case builds directly. /// -public sealed class RegisteredRecorderFixture : WebApplicationFactory; +public sealed class RegisteredRecorderFixture : ConstructionOnlyApiFactory; /// The real host with one burst setting bent out of shape. public sealed class MisconfiguredBurstFixture(string key, string value) - : WebApplicationFactory + : ConstructionOnlyApiFactory { - protected override void ConfigureWebHost(IWebHostBuilder builder) + protected override void ConfigureTestHost(IWebHostBuilder builder) { ArgumentNullException.ThrowIfNull(builder); @@ -485,12 +485,12 @@ protected override void ConfigureWebHost(IWebHostBuilder builder) } } -public class ResolvedTenantFixture : WebApplicationFactory +public class ResolvedTenantFixture : ConstructionOnlyApiFactory { public static readonly Guid TenantId = Guid.Parse("018f4d40-0000-7000-8000-0000000000aa"); public static readonly Guid OrganizationId = Guid.Parse("018f4d40-0000-7000-8000-0000000000bb"); - protected override void ConfigureWebHost(IWebHostBuilder builder) + protected override void ConfigureTestHost(IWebHostBuilder builder) { ArgumentNullException.ThrowIfNull(builder); builder.UseEnvironment(Environments.Development); diff --git a/backend/tests/LearnStack.Tests.Integration/VersionedRouteEnforcementTests.cs b/backend/tests/LearnStack.Tests.Integration/VersionedRouteEnforcementTests.cs index e333a181..92b97776 100644 --- a/backend/tests/LearnStack.Tests.Integration/VersionedRouteEnforcementTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/VersionedRouteEnforcementTests.cs @@ -211,7 +211,7 @@ public void Apply(ApplicationModel application) /// The production host, with no test controllers added — its endpoint set is /// exactly what a deployed instance serves. /// -public sealed class ProductionHostFixture : WebApplicationFactory +public sealed class ProductionHostFixture : ConstructionOnlyApiFactory { private IReadOnlyList? _patterns; @@ -223,7 +223,7 @@ public sealed class ProductionHostFixture : WebApplicationFactory .Select(endpoint => (endpoint.RoutePattern.RawText ?? string.Empty).TrimStart('/')) .Where(pattern => pattern.Length > 0)]; - protected override void ConfigureWebHost(IWebHostBuilder builder) + protected override void ConfigureTestHost(IWebHostBuilder builder) { ArgumentNullException.ThrowIfNull(builder); builder.UseEnvironment(Environments.Development); @@ -231,9 +231,9 @@ protected override void ConfigureWebHost(IWebHostBuilder builder) } /// Host that admits exactly one probe controller, to prove a guard fires. -public sealed class ProbeHostFixture(Type controller) : WebApplicationFactory +public sealed class ProbeHostFixture(Type controller) : ConstructionOnlyApiFactory { - protected override void ConfigureWebHost(IWebHostBuilder builder) + protected override void ConfigureTestHost(IWebHostBuilder builder) { ArgumentNullException.ThrowIfNull(builder); builder.UseEnvironment(Environments.Development); diff --git a/backend/tests/LearnStack.Tests.Unit/Api/Composition/OutboxDispatcherDataSourceGuardTests.cs b/backend/tests/LearnStack.Tests.Unit/Api/Composition/OutboxDispatcherDataSourceGuardTests.cs new file mode 100644 index 00000000..907868f7 --- /dev/null +++ b/backend/tests/LearnStack.Tests.Unit/Api/Composition/OutboxDispatcherDataSourceGuardTests.cs @@ -0,0 +1,98 @@ +using FluentAssertions; +using LearnStack.Api.Composition; +using LearnStack.Infrastructure.Messaging; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Npgsql; +using Xunit; + +namespace LearnStack.Tests.Unit.Api.Composition; + +public sealed class OutboxDispatcherDataSourceGuardTests +{ + private const string Valid = + "Host=localhost;Port=1;Database=learnstack;Username=learnstack_outbox_admin;Password=synthetic-secret-canary"; // leakwatch:ignore + + [Theory] + [InlineData(null)] + [InlineData("")] + [InlineData(" ")] + [InlineData("Host=localhost;Username=synthetic-username-canary;Password=synthetic-secret-canary")] // leakwatch:ignore + [InlineData("Host=localhost;Username=LEARNSTACK_OUTBOX_ADMIN;Password=synthetic-secret-canary")] // leakwatch:ignore + [InlineData("Host=localhost;Password=synthetic-secret-canary")] // leakwatch:ignore + [InlineData("Host=localhost;Port=synthetic-secret-canary;Username=synthetic-username-canary")] + [InlineData("postgres://synthetic-username-canary:synthetic-secret-canary@localhost/database")] // leakwatch:ignore + [InlineData("synthetic-secret-canary=value;Username=synthetic-username-canary")] + public void Validate_and_build_refuse_invalid_credentials_without_echo_or_inner_exception(string? value) + { + AssertSafeRefusal(() => OutboxDispatcherDataSource.Validate(value)); + AssertSafeRefusal(() => OutboxDispatcherDataSource.Build(value)); + } + + [Fact] + public void A_valid_credential_validates_and_builds_without_connecting() + { + OutboxDispatcherDataSource.Validate(Valid); + using var source = OutboxDispatcherDataSource.Build(Valid); + source.Should().NotBeNull(); + } + + [Theory] + [InlineData("Host=localhost;Username=synthetic-username-canary;Password=synthetic-secret-canary")] // leakwatch:ignore + [InlineData("postgres://synthetic-username-canary:synthetic-secret-canary@localhost/database")] // leakwatch:ignore + [InlineData("Host=localhost;Port=synthetic-secret-canary;Username=learnstack_outbox_admin")] + public void Present_invalid_configuration_is_refused_during_registration(string value) + { + var services = new ServiceCollection(); + AssertSafeRefusal(() => services.AddLearnStackOutboxDispatcher(Configuration(value))); + } + + [Theory] + [InlineData(null)] + [InlineData("")] + [InlineData(" ")] + public void Absent_configuration_is_deferred_until_the_keyed_source_is_resolved(string? value) + { + var services = new ServiceCollection(); + services.AddLearnStackOutboxDispatcher(Configuration(value)); + using var provider = services.BuildServiceProvider(); + var lazy = provider.GetRequiredKeyedService>(OutboxDispatcherDataSource.Key); + + lazy.IsValueCreated.Should().BeFalse(); + AssertSafeRefusal(() => _ = lazy.Value); + } + + [Fact] + public void Registration_is_keyed_lazy_and_owns_one_shared_source() + { + var services = new ServiceCollection(); + services.AddLearnStackOutboxDispatcher(Configuration(Valid)); + services.AddLearnStackOutboxDispatcher(Configuration(Valid)); + using var provider = services.BuildServiceProvider(); + var lazy = provider.GetRequiredKeyedService>(OutboxDispatcherDataSource.Key); + + provider.GetService().Should().BeNull(); + lazy.IsValueCreated.Should().BeFalse(); + var source = lazy.Value; + source.Should().BeSameAs(provider.GetRequiredKeyedService(OutboxDispatcherDataSource.Key)); + source.Should().BeSameAs(lazy.Value); + services.Count(descriptor => descriptor.IsKeyedService + && descriptor.ServiceType == typeof(NpgsqlDataSource) + && Equals(descriptor.ServiceKey, OutboxDispatcherDataSource.Key)).Should().Be(1); + } + + private static IConfiguration Configuration(string? value) => new ConfigurationBuilder() + .AddInMemoryCollection(new Dictionary + { + [$"ConnectionStrings:{OutboxDispatcherDataSource.ConnectionName}"] = value, + }) + .Build(); + + private static void AssertSafeRefusal(Action action) + { + var failure = action.Should().Throw().Which; + failure.Message.Should().Contain("ConnectionStrings:OutboxDispatcher"); + failure.InnerException.Should().BeNull(); + failure.ToString().Should().NotContain("synthetic-secret-canary").And.NotContain("synthetic-username-canary"); + } +} diff --git a/backend/tests/LearnStack.Tests.Unit/Api/Composition/OutboxProductionCompositionTests.cs b/backend/tests/LearnStack.Tests.Unit/Api/Composition/OutboxProductionCompositionTests.cs new file mode 100644 index 00000000..3ddb427c --- /dev/null +++ b/backend/tests/LearnStack.Tests.Unit/Api/Composition/OutboxProductionCompositionTests.cs @@ -0,0 +1,58 @@ +using FluentAssertions; +using LearnStack.Api.Composition; +using LearnStack.Api.Messaging; +using LearnStack.Infrastructure.Messaging; +using LearnStack.Modules.Tenancy.Application.Contracts.IntegrationEvents; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Diagnostics.HealthChecks; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Options; +using Xunit; + +namespace LearnStack.Tests.Unit.Api.Composition; + +public sealed class OutboxProductionCompositionTests +{ + [Fact] + public void Production_registers_exactly_one_mandatory_worker_and_the_unmapped_outbox_health_check() + { + var services = new ServiceCollection(); + var configuration = new ConfigurationBuilder().Build(); + services.AddLearnStackOutboxDispatcher(configuration); + services.AddLearnStackOutboxDispatcher(configuration); + + services.Where(descriptor => descriptor.ServiceType == typeof(IHostedService) + && descriptor.ImplementationType == typeof(OutboxProcessor)) + .Should().ContainSingle("mandatory execution must not depend on a configuration toggle"); + using var provider = services.BuildServiceProvider(); + provider.GetRequiredService>().Value.Registrations + .Where(registration => registration.Name == OutboxHealthCheck.Name).Should().ContainSingle(); + } + + [Fact] + public void Real_provisioning_contract_remains_pending_for_both_distinct_default_consumers() + { + var handlers = IntegrationEventHandlerRegistry.Discover( + typeof(LearnStack.Modules.Tenancy.Application.AssemblyMarker).Assembly, + typeof(LearnStack.Modules.Customization.Application.AssemblyMarker).Assembly, + typeof(LearnStack.Modules.Education.Application.AssemblyMarker).Assembly); + var services = new ServiceCollection(); + services.AddSingleton(handlers); + services.AddLearnStackPersistence(new ConfigurationBuilder().Build()); + using var provider = services.BuildServiceProvider(); + var registry = provider.GetRequiredService(); + var identity = typeof(TenantCreatedIntegrationEventV1).AssemblyQualifiedName!; + + registry.PendingIdentities.Should().ContainSingle().Which.Should().Be(identity); + var resolution = registry.Resolve(identity); + resolution.EventType.Should().Be(); + resolution.Disposition.Should().Be(OutboxEventDisposition.Pending); + resolution.Subscriptions.Select(subscription => subscription.Id).Should().BeEquivalentTo( + ["customization.initialize-content-type-default.v1", "customization.initialize-taxonomy-default.v1"]); + resolution.Subscriptions.Should().OnlyContain(subscription => + subscription.Readiness == OutboxSubscriptionReadiness.Pending && subscription.HandlerType == null); + handlers.For(typeof(TenantCreatedIntegrationEventV1)).Should().BeEmpty( + "P02b-3 activates real consumers; P02b-2 must preserve their facts"); + } +} diff --git a/backend/tests/LearnStack.Tests.Unit/Api/Messaging/OutboxObservationTests.cs b/backend/tests/LearnStack.Tests.Unit/Api/Messaging/OutboxObservationTests.cs new file mode 100644 index 00000000..5a84e558 --- /dev/null +++ b/backend/tests/LearnStack.Tests.Unit/Api/Messaging/OutboxObservationTests.cs @@ -0,0 +1,172 @@ +using System.Diagnostics; +using System.Diagnostics.Metrics; +using FluentAssertions; +using LearnStack.Api.Messaging; +using LearnStack.Infrastructure.Messaging; +using LearnStack.SharedKernel.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Diagnostics.HealthChecks; +using Xunit; + +namespace LearnStack.Tests.Unit.Api.Messaging; + +public sealed class OutboxObservationTests +{ + [Fact] + public async Task Health_requires_progress_and_fresh_snapshot_on_the_monotonic_clock() + { + var time = new ManualTimeProvider(); + using var observation = Create(time); + var check = new OutboxHealthCheck(observation); + observation.IsHealthy.Should().BeFalse(); + observation.Started(); + observation.Progress(); + observation.IsHealthy.Should().BeFalse("there is no successful queue observation yet"); + observation.Refreshed(new(9, 4, 20)); + (await check.CheckHealthAsync(new HealthCheckContext())).Status.Should().Be(HealthStatus.Healthy); + + time.Advance(TimeSpan.FromSeconds(44)); + observation.IsHealthy.Should().BeTrue(); + observation.HeartbeatAge.Should().Be(TimeSpan.FromSeconds(44)); + time.UtcNow = time.UtcNow.AddDays(-30); + observation.IsHealthy.Should().BeTrue("wall-clock adjustments do not reset elapsed progress"); + time.Advance(TimeSpan.FromSeconds(1)); + observation.IsHealthy.Should().BeFalse("45 seconds is the stale boundary"); + observation.Progress(); + observation.IsHealthy.Should().BeFalse("fresh progress cannot conceal a stale SQL snapshot"); + observation.Refreshed(new(8, 3, 25)); + observation.IsHealthy.Should().BeTrue(); + observation.Stopped(); + (await check.CheckHealthAsync(new HealthCheckContext())).Status.Should().Be(HealthStatus.Unhealthy); + } + + [Fact] + public void Failed_refresh_retains_immutable_snapshot_and_terminal_failure_survives_empty_polls() + { + using var observation = Create(new ManualTimeProvider()); + observation.Started(); + observation.Progress(); + observation.Refreshed(new(7, 5, 13)); + var before = observation.Snapshot; + observation.RefreshFailed(); + observation.Snapshot.Queue.Should().BeSameAs(before.Queue); + before.RefreshFailed.Should().BeFalse("published snapshots are immutable"); + observation.IsHealthy.Should().BeFalse(); + observation.Refreshed(new(6, 5, 13)); + observation.IsHealthy.Should().BeTrue(); + + observation.TerminalAuditFailed(); + observation.Progress(); + observation.Refreshed(new(0, 0, 0)); + observation.IsHealthy.Should().BeFalse("a successful empty poll is not terminal audit recovery"); + observation.TerminalAuditSucceeded(); + observation.IsHealthy.Should().BeTrue(); + observation.Outstanding(1); + observation.IsHealthy.Should().BeFalse(); + observation.Outstanding(0); + observation.IsHealthy.Should().BeTrue(); + observation.LoopFailed(); + observation.IsHealthy.Should().BeFalse(); + observation.Progress(); + observation.IsHealthy.Should().BeTrue(); + } + + [Fact] + public void Exact_metric_inventory_has_memory_only_gauges_and_finite_event_labels() + { + var instruments = new List(); + var measurements = new List<(Instrument Instrument, double Value, KeyValuePair[] Tags)>(); + using var listener = new MeterListener(); + listener.InstrumentPublished = (instrument, owner) => + { + if (instrument.Meter.Name != OutboxObservation.InstrumentationName) return; + lock (instruments) instruments.Add(instrument); + owner.EnableMeasurementEvents(instrument); + }; + listener.SetMeasurementEventCallback((instrument, value, tags, _) => + { + lock (measurements) measurements.Add((instrument, value, tags.ToArray())); + }); + listener.SetMeasurementEventCallback((instrument, value, tags, _) => + { + lock (measurements) measurements.Add((instrument, value, tags.ToArray())); + }); + listener.Start(); + var time = new ManualTimeProvider(); + using var observation = Create(time); + observation.Started(); + observation.Progress(); + observation.Refreshed(new(17, 11, 23)); + observation.Duration(nameof(TestEvent), TimeSpan.FromSeconds(3.5)); + Meter ownMeter; + lock (measurements) ownMeter = measurements.Single(value => value.Value == 3.5).Instrument.Meter; + observation.Failed("private-stored-type-and-tenant"); + observation.Deadletter("retired"); + observation.LeaseLost(); + time.Advance(TimeSpan.FromSeconds(8)); + listener.RecordObservableInstruments(); + + Instrument[] ownInstruments; + lock (instruments) ownInstruments = instruments.Where(value => ReferenceEquals(value.Meter, ownMeter)).ToArray(); + ownInstruments.Select(value => value.Name).Should().BeEquivalentTo( + [ + "learnstack_outbox_pending_count", "learnstack_outbox_dispatch_duration_seconds", + "learnstack_outbox_dispatch_failed_total", "learnstack_outbox_deadletter_total", + "learnstack_outbox_lease_lost_total", "learnstack_outbox_oldest_eligible_age_seconds", + "learnstack_outbox_deferred_count", "learnstack_outbox_heartbeat_age_seconds", + ]); + lock (measurements) + { + var own = measurements.Where(value => ReferenceEquals(value.Instrument.Meter, ownMeter)).ToArray(); + own.Single(value => value.Instrument.Name == "learnstack_outbox_pending_count").Value.Should().Be(17); + own.Single(value => value.Instrument.Name == "learnstack_outbox_deferred_count").Value.Should().Be(11); + own.Single(value => value.Instrument.Name == "learnstack_outbox_oldest_eligible_age_seconds").Value.Should().Be(23); + own.Single(value => value.Instrument.Name == "learnstack_outbox_heartbeat_age_seconds").Value.Should().Be(8); + own.Single(value => value.Instrument.Name == "learnstack_outbox_dispatch_failed_total").Tags + .Should().Equal([new KeyValuePair("event_type", "invalid")]); + own.Where(value => value.Instrument.IsObservable || value.Instrument.Name == "learnstack_outbox_lease_lost_total") + .Should().OnlyContain(value => value.Tags.Length == 0); + own.SelectMany(value => value.Tags).Should().OnlyContain(value => value.Key == "event_type"); + } + } + + [Theory] + [InlineData("00-0123456789abcdef0123456789abcdef-0123456789abcdef-01")] + [InlineData("01-0123456789abcdef0123456789abcdef-0123456789abcdef-01-private-extension")] + [InlineData("private-invalid-correlation")] + public void Attempts_always_get_fresh_valid_trace_identity_without_listeners(string correlation) + { + using var observation = Create(new ManualTimeProvider()); + using var first = observation.BeginAttempt(correlation, "private-undeclared-type"); + ActivityContext.TryParse(first.Id, null, out var firstContext).Should().BeTrue(); + using var second = observation.BeginAttempt(correlation, "unknown"); + ActivityContext.TryParse(second.Id, null, out var secondContext).Should().BeTrue(); + firstContext.SpanId.Should().NotBe(secondContext.SpanId); + first.GetTagItem("event_type").Should().Be("invalid"); + if (!ActivityContext.TryParse(correlation, null, out _)) + first.GetTagItem("correlation.id").Should().BeNull(); + else + first.GetTagItem("correlation.id").Should().Be( + "00-0123456789abcdef0123456789abcdef-0123456789abcdef-01"); + } + + private static OutboxObservation Create(TimeProvider time) => new(time, new OutboxDispatchOptions(), + OutboxEventRegistry.Create([OutboxEventRegistration.IntentionallyUnsubscribed(typeof(TestEvent))], [], + IntegrationEventHandlerRegistry.FromServiceDescriptors(new ServiceCollection()), [typeof(TestEvent).Assembly])); + + private sealed record TestEvent : IntegrationEventBase + { + public override string Topic => "learnstack.test.observation"; + public override string PartitionKey => TenantId.ToString("D"); + } + + private sealed class ManualTimeProvider : TimeProvider + { + private long _timestamp; + public DateTimeOffset UtcNow { get; set; } = new(2026, 10, 12, 0, 0, 0, TimeSpan.Zero); + public override long TimestampFrequency => TimeSpan.TicksPerSecond; + public override long GetTimestamp() => _timestamp; + public override DateTimeOffset GetUtcNow() => UtcNow; + public void Advance(TimeSpan duration) => _timestamp += duration.Ticks; + } +} diff --git a/backend/tests/LearnStack.Tests.Unit/Api/Messaging/OutboxPublishAttemptTests.cs b/backend/tests/LearnStack.Tests.Unit/Api/Messaging/OutboxPublishAttemptTests.cs new file mode 100644 index 00000000..d6b98f03 --- /dev/null +++ b/backend/tests/LearnStack.Tests.Unit/Api/Messaging/OutboxPublishAttemptTests.cs @@ -0,0 +1,200 @@ +using FluentAssertions; +using LearnStack.Api.Messaging; +using LearnStack.SharedKernel.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Xunit; + +namespace LearnStack.Tests.Unit.Api.Messaging; + +public sealed class OutboxPublishAttemptTests +{ + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Completion_reports_publication_outcome_after_disposing_the_owned_scope(bool throws) + { + var disposed = false; + CancellationToken captured = default; + IntegrationEventEnvelope? observed = null; + var envelope = Envelope(); + var services = new ServiceCollection(); + services.AddScoped(_ => new CallbackBus((received, token) => + { + observed = received; + captured = token; + return throws ? Task.FromException(new InvalidOperationException("private-provider-error")) + : Task.CompletedTask; + }, () => { disposed = true; return ValueTask.CompletedTask; })); + await using var provider = services.BuildServiceProvider(); + var attempt = new OutboxPublishAttempt(provider.GetRequiredService(), envelope); + + (await attempt.Completion.WaitAsync(TimeSpan.FromSeconds(5))).Should().Be(!throws); + + disposed.Should().BeTrue(); + observed.Should().BeSameAs(envelope); + var accessDisposedToken = () => captured.WaitHandle; + accessDisposedToken.Should().Throw(); + attempt.RequestCancellation(); // Completed attempts safely ignore late cancellation. + } + + [Fact] + public async Task Synchronous_construction_and_publish_prefix_run_inside_the_owned_task() + { + using var constructRelease = new ManualResetEventSlim(); + using var publishRelease = new ManualResetEventSlim(); + var constructed = Signal(); + var published = Signal(); + var disposed = Signal(); + var services = new ServiceCollection(); + services.AddScoped(_ => + { + constructed.TrySetResult(); + constructRelease.Wait(); // Intentionally hostile synchronous construction. + return new CallbackBus((_, _) => + { + published.TrySetResult(); + publishRelease.Wait(CancellationToken.None); // Intentionally hostile synchronous PublishAsync prefix. + return Task.CompletedTask; + }, () => { disposed.TrySetResult(); return ValueTask.CompletedTask; }); + }); + await using var provider = services.BuildServiceProvider(); + var attempt = new OutboxPublishAttempt(provider.GetRequiredService(), Envelope()); + try + { + await constructed.Task.WaitAsync(TimeSpan.FromSeconds(5)); + attempt.Completion.IsCompleted.Should().BeFalse(); + constructRelease.Set(); + await published.Task.WaitAsync(TimeSpan.FromSeconds(5)); + attempt.Completion.IsCompleted.Should().BeFalse(); + disposed.Task.IsCompleted.Should().BeFalse(); + attempt.RequestCancellation(); + } + finally + { + constructRelease.Set(); + publishRelease.Set(); + await attempt.Completion.WaitAsync(TimeSpan.FromSeconds(5)); + } + + (await attempt.Completion).Should().BeFalse("cancelled late success cannot become a successful attempt"); + disposed.Task.IsCompleted.Should().BeTrue(); + } + + [Fact] + public async Task Ignored_cancellation_and_async_cleanup_retain_scope_and_token_until_actual_completion() + { + var entered = Signal(); + var releaseCallback = Signal(); + var cleanupEntered = Signal(); + var releaseCleanup = Signal(); + CancellationToken captured = default; + var disposeCount = 0; + var services = new ServiceCollection(); + services.AddScoped(_ => new CallbackBus(async (_, token) => + { + captured = token; + entered.TrySetResult(); + await releaseCallback.Task; + }, async () => + { + Interlocked.Increment(ref disposeCount); + cleanupEntered.TrySetResult(); + await releaseCleanup.Task; + })); + await using var provider = services.BuildServiceProvider(); + var attempt = new OutboxPublishAttempt(provider.GetRequiredService(), Envelope()); + try + { + await entered.Task.WaitAsync(TimeSpan.FromSeconds(5)); + attempt.RequestCancellation(); + captured.IsCancellationRequested.Should().BeTrue(); + captured.WaitHandle.Should().NotBeNull(); + disposeCount.Should().Be(0); + attempt.Completion.IsCompleted.Should().BeFalse(); + releaseCallback.TrySetResult(); + await cleanupEntered.Task.WaitAsync(TimeSpan.FromSeconds(5)); + captured.WaitHandle.Should().NotBeNull("asynchronous cleanup still owns the cancellation source"); + attempt.Completion.IsCompleted.Should().BeFalse(); + } + finally + { + releaseCallback.TrySetResult(); + releaseCleanup.TrySetResult(); + await attempt.Completion.WaitAsync(TimeSpan.FromSeconds(5)); + } + + disposeCount.Should().Be(1); + var accessDisposedToken = () => captured.WaitHandle; + accessDisposedToken.Should().Throw(); + } + + [Fact] + public async Task Blocking_cancellation_registration_is_retained_and_observed_before_scope_disposal() + { + var entered = Signal(); + var releaseCallback = Signal(); + var cancellationEntered = Signal(); + using var releaseCancellation = new ManualResetEventSlim(); + var disposed = Signal(); + var cancellationCount = 0; + var services = new ServiceCollection(); + services.AddScoped(_ => new CallbackBus(async (_, token) => + { + token.Register(() => + { + Interlocked.Increment(ref cancellationCount); + cancellationEntered.TrySetResult(); + releaseCancellation.Wait(); // Exercise a cancellation callback that blocks and faults. + throw new InvalidOperationException("private-provider-cancellation-error"); + }); + entered.TrySetResult(); + await releaseCallback.Task; + }, () => { disposed.TrySetResult(); return ValueTask.CompletedTask; })); + await using var provider = services.BuildServiceProvider(); + var attempt = new OutboxPublishAttempt(provider.GetRequiredService(), Envelope()); + try + { + await entered.Task.WaitAsync(TimeSpan.FromSeconds(5)); + attempt.RequestCancellation(); + attempt.RequestCancellation(); + await cancellationEntered.Task.WaitAsync(TimeSpan.FromSeconds(5)); + releaseCallback.TrySetResult(); + attempt.Completion.IsCompleted.Should().BeFalse(); + disposed.Task.IsCompleted.Should().BeFalse(); + } + finally + { + releaseCallback.TrySetResult(); + releaseCancellation.Set(); + await attempt.Completion.WaitAsync(TimeSpan.FromSeconds(5)); + } + + (await attempt.Completion).Should().BeFalse(); + disposed.Task.IsCompleted.Should().BeTrue(); + cancellationCount.Should().Be(1); + attempt.RequestCancellation(); + } + + private static TaskCompletionSource Signal() => new(TaskCreationOptions.RunContinuationsAsynchronously); + private static IntegrationEventEnvelope Envelope() => new(new TestEvent + { + EventId = Guid.NewGuid(), + TenantId = Guid.NewGuid(), + OccurredAt = DateTimeOffset.UtcNow, + }, "00-0123456789abcdef0123456789abcdef-0123456789abcdef-01"); + + private sealed record TestEvent : IntegrationEventBase + { + public override string Topic => "learnstack.test.attempt"; + public override string PartitionKey => TenantId.ToString("D"); + } + + private sealed class CallbackBus( + Func publish, + Func dispose) : IEventBus, IAsyncDisposable + { + public Task PublishAsync(IntegrationEventEnvelope envelope, CancellationToken cancellationToken = default) => + publish(envelope, cancellationToken); + public ValueTask DisposeAsync() => dispose(); + } +} diff --git a/backend/tests/LearnStack.Tests.Unit/Api/Tenancy/AuditingTenantAssertionRecorderTests.cs b/backend/tests/LearnStack.Tests.Unit/Api/Tenancy/AuditingTenantAssertionRecorderTests.cs index 80ea90d6..fb4fd589 100644 --- a/backend/tests/LearnStack.Tests.Unit/Api/Tenancy/AuditingTenantAssertionRecorderTests.cs +++ b/backend/tests/LearnStack.Tests.Unit/Api/Tenancy/AuditingTenantAssertionRecorderTests.cs @@ -259,7 +259,8 @@ public async Task The_real_store_reports_a_failure_once_and_the_recorder_does_no () => throw new InvalidOperationException("ConnectionStrings:Default is absent.")), storeLogger, meterFactory, - health); + health, + new AuditCatalog([])); var counted = 0L; using var listener = new MeterListener(); @@ -551,6 +552,14 @@ public Task WriteBestEffortAsync( AuditEntryDraft entry, CancellationToken cancellationToken = default) => throw new NotSupportedException("both slugs are MUST"); + public Task WriteEventTerminalAsync( + AuditEntryDraft entry, + EventTerminalTransition transition, + System.Data.Common.DbConnection connection, + System.Data.Common.DbTransaction transaction, + CancellationToken cancellationToken = default) => + throw new NotSupportedException("The terminal-event path has its own cases."); + public Task WritePlatformScopeAsync( AuditEntryDraft entry, System.Data.Common.DbConnection connection, diff --git a/backend/tests/LearnStack.Tests.Unit/Application/Pipeline/AuditPipelineHarness.cs b/backend/tests/LearnStack.Tests.Unit/Application/Pipeline/AuditPipelineHarness.cs index 5b57ea22..da0cd551 100644 --- a/backend/tests/LearnStack.Tests.Unit/Application/Pipeline/AuditPipelineHarness.cs +++ b/backend/tests/LearnStack.Tests.Unit/Application/Pipeline/AuditPipelineHarness.cs @@ -167,6 +167,14 @@ public Task WriteBestEffortAsync(AuditEntryDraft entry, CancellationToken cancel return Task.CompletedTask; } + public Task WriteEventTerminalAsync( + AuditEntryDraft entry, + EventTerminalTransition transition, + System.Data.Common.DbConnection connection, + System.Data.Common.DbTransaction transaction, + CancellationToken cancellationToken = default) => + throw new NotSupportedException("The terminal-event path has its own cases."); + public Task WritePlatformScopeAsync( AuditEntryDraft entry, System.Data.Common.DbConnection connection, diff --git a/backend/tests/LearnStack.Tests.Unit/Infrastructure/Messaging/OutboxDispatchOptionsTests.cs b/backend/tests/LearnStack.Tests.Unit/Infrastructure/Messaging/OutboxDispatchOptionsTests.cs new file mode 100644 index 00000000..7fe00b49 --- /dev/null +++ b/backend/tests/LearnStack.Tests.Unit/Infrastructure/Messaging/OutboxDispatchOptionsTests.cs @@ -0,0 +1,44 @@ +using FluentAssertions; +using LearnStack.Infrastructure.Messaging; +using Xunit; + +namespace LearnStack.Tests.Unit.Infrastructure.Messaging; + +public sealed class OutboxDispatchOptionsTests +{ + [Fact] + public void Approved_defaults_have_a_finite_publish_and_transition_margin() + { + var options = new OutboxDispatchOptions(); + options.Validate(); + options.LeaseDuration.Should().BeGreaterThan(options.PublishTimeout + options.SqlTimeout * 2); + options.MaxOutstandingPublishes.Should().Be(4); + } + + [Theory] + [InlineData("sql")] + [InlineData("publish")] + [InlineData("lease")] + [InlineData("poll")] + [InlineData("progress")] + [InlineData("snapshot")] + [InlineData("shutdown")] + [InlineData("capacity")] + public void Invalid_bounds_are_refused_before_runtime(string setting) + { + var options = new OutboxDispatchOptions(); + options = setting switch + { + "sql" => options with { SqlTimeout = TimeSpan.FromSeconds(6) }, + "publish" => options with { PublishTimeout = TimeSpan.Zero }, + "lease" => options with { LeaseDuration = TimeSpan.FromSeconds(20) }, + "poll" => options with { IdlePoll = Timeout.InfiniteTimeSpan }, + "progress" => options with { ProgressTimeout = TimeSpan.FromSeconds(46) }, + "snapshot" => options with { SnapshotInterval = TimeSpan.Zero }, + "shutdown" => options with { ShutdownTimeout = TimeSpan.FromSeconds(31) }, + "capacity" => options with { MaxOutstandingPublishes = 5 }, + _ => throw new ArgumentOutOfRangeException(nameof(setting)), + }; + options.Invoking(value => value.Validate()).Should().Throw(); + } +} diff --git a/backend/tests/LearnStack.Tests.Unit/Infrastructure/Messaging/OutboxEnvelopeReaderTests.cs b/backend/tests/LearnStack.Tests.Unit/Infrastructure/Messaging/OutboxEnvelopeReaderTests.cs new file mode 100644 index 00000000..2f1767b4 --- /dev/null +++ b/backend/tests/LearnStack.Tests.Unit/Infrastructure/Messaging/OutboxEnvelopeReaderTests.cs @@ -0,0 +1,164 @@ +using System.Text.Json.Nodes; +using FluentAssertions; +using LearnStack.Infrastructure.Messaging; +using LearnStack.SharedKernel.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Xunit; + +namespace LearnStack.Tests.Unit.Infrastructure.Messaging; + +public sealed class OutboxEnvelopeReaderTests +{ + private const string Trace = "00-0123456789abcdef0123456789abcdef-0123456789abcdef-01"; + private const string Alias = "test.previous.fact"; + + [Fact] + public void Intentional_absence_and_alias_validate_the_full_envelope_before_acknowledgement() + { + var fact = Fact(); + var claim = Claim(fact) with { Type = Alias, OrganizationId = Guid.NewGuid() }; + var result = Reader().Read(claim); + result.Disposition.Should().Be(OutboxEventDisposition.IntentionallyUnsubscribed); + result.TerminalReason.Should().BeNull(); + result.Envelope!.Event.Should().BeEquivalentTo(fact); + result.Envelope.OrganizationId.Should().Be(claim.OrganizationId); + + Reader().Read(claim with { CorrelationId = "not-a-trace" }).TerminalReason + .Should().Be(OutboxTerminalReason.InvalidEnvelope); + } + + [Fact] + public void Pending_precedes_even_malformed_payload_and_envelope_validation() + { + var reader = Reader(pending: true); + var result = reader.Read(Claim(Fact()) with { Payload = "{", CorrelationId = "invalid", Type = Alias }); + result.Disposition.Should().Be(OutboxEventDisposition.Pending); + result.Envelope.Should().BeNull(); + result.TerminalReason.Should().BeNull(); + } + + [Fact] + public void Unknown_and_explicitly_retired_identities_have_closed_distinct_reasons() + { + var claim = Claim(Fact()) with { Payload = "{" }; + Reader().Read(claim with { Type = "System.String, System.Private.CoreLib" }).TerminalReason + .Should().Be(OutboxTerminalReason.UnknownType); + Reader().Read(claim with { Type = "test.retired" }).TerminalReason + .Should().Be(OutboxTerminalReason.RetiredType); + } + + [Theory] + [InlineData("id")] + [InlineData("tenant")] + [InlineData("topic")] + [InlineData("partition")] + [InlineData("timestamp")] + [InlineData("organization")] + [InlineData("causation")] + [InlineData("actor")] + [InlineData("metadata")] + [InlineData("payload")] + public void Every_row_envelope_mismatch_refuses_without_parser_diagnostics(string mismatch) + { + var claim = Claim(Fact()); + claim = mismatch switch + { + "id" => claim with { Id = Guid.NewGuid() }, + "tenant" => claim with { TenantId = Guid.NewGuid() }, + "topic" => claim with { Topic = "learnstack.test.other" }, + "partition" => claim with { PartitionKey = Guid.NewGuid().ToString("D") }, + "timestamp" => claim with { OccurredAt = claim.OccurredAt.AddTicks(10) }, + "organization" => claim with { OrganizationId = Guid.Empty }, + "causation" => claim with { CausationId = Guid.Empty }, + "actor" => claim with { ActorUserId = Guid.Empty }, + "metadata" => claim with { MetadataIsNull = false }, + "payload" => claim with { Payload = "{\"private\":\"do-not-log\"" }, + _ => throw new ArgumentOutOfRangeException(nameof(mismatch)), + }; + var result = Reader().Read(claim); + result.TerminalReason.Should().Be(OutboxTerminalReason.InvalidEnvelope); + result.EventTypeLabel.Should().Be("invalid"); + result.Envelope.Should().BeNull(); + } + + [Theory] + [InlineData("Topic", "missing")] + [InlineData("Topic", "changed")] + [InlineData("Topic", "null")] + [InlineData("Topic", "number")] + [InlineData("Topic", "object")] + [InlineData("PartitionKey", "missing")] + [InlineData("PartitionKey", "changed")] + [InlineData("PartitionKey", "null")] + [InlineData("PartitionKey", "number")] + [InlineData("PartitionKey", "object")] + public void Stored_get_only_routing_fields_must_exist_and_match_the_row(string property, string mutation) + { + var claim = Claim(Fact()); + var root = JsonNode.Parse(claim.Payload)!.AsObject(); + if (mutation == "missing") root.Remove(property); + else root[property] = mutation switch + { + "changed" => JsonValue.Create("private-invalid-routing"), + "null" => null, + "number" => JsonValue.Create(42), + "object" => new JsonObject(), + _ => throw new ArgumentOutOfRangeException(nameof(mutation)), + }; + + var result = Reader().Read(claim with { Payload = root.ToJsonString() }); + + result.TerminalReason.Should().Be(OutboxTerminalReason.InvalidEnvelope); + result.EventTypeLabel.Should().Be("invalid"); + result.Envelope.Should().BeNull(); + Reader().Read(claim).Envelope.Should().NotBeNull("the unchanged control must still be admitted"); + } + + [Fact] + public void PostgreSQL_microsecond_truncation_preserves_valid_occurrence_identity() + { + var fact = Fact() with + { + OccurredAt = DateTimeOffset.Parse("2026-10-12T12:00:00.1234567Z", System.Globalization.CultureInfo.InvariantCulture), + }; + var claim = Claim(fact) with { OccurredAt = fact.OccurredAt.AddTicks(-7) }; + Reader().Read(claim).Envelope.Should().NotBeNull(); + } + + [Fact] + public void Jsonb_whitespace_is_not_the_canonical_producer_byte_budget() + { + var claim = Claim(Fact()); + var root = JsonNode.Parse(claim.Payload)!; + claim = claim with { Payload = root.ToJsonString(new() { WriteIndented = true }) + new string(' ', 262_144) }; + Reader().Read(claim).Envelope.Should().NotBeNull(); + } + + private static OutboxEnvelopeReader Reader(bool pending = false) + { + var registration = pending + ? OutboxEventRegistration.Required(typeof(TestFact), [OutboxSubscriptionManifest.Pending("test.fact.v1")], Alias) + : OutboxEventRegistration.IntentionallyUnsubscribed(typeof(TestFact), Alias); + return new(OutboxEventRegistry.Create([registration], ["test.retired"], + IntegrationEventHandlerRegistry.FromServiceDescriptors(new ServiceCollection()), [typeof(TestFact).Assembly])); + } + + private static TestFact Fact() => new() + { + EventId = Guid.NewGuid(), + TenantId = Guid.NewGuid(), + OccurredAt = DateTimeOffset.UtcNow, + }; + + private static OutboxDispatchClaim Claim(TestFact fact) => new(fact.EventId, fact.TenantId, null, + Guid.NewGuid().ToString("D"), 1, DateTimeOffset.UtcNow.AddSeconds(30), OutboxClaimKind.Normal, + typeof(TestFact).AssemblyQualifiedName!, fact.Topic, fact.PartitionKey, fact.OccurredAt, + Trace, null, null, fact.ToPayloadJson(), true); + + [IntegrationEventPartition(nameof(TenantId), tenantWide: true)] + public sealed record TestFact : IntegrationEventBase + { + public override string Topic => "learnstack.test.fact"; + public override string PartitionKey => TenantId.ToString("D"); + } +} diff --git a/backend/tests/LearnStack.Tests.Unit/Infrastructure/Messaging/OutboxEventRegistryTests.cs b/backend/tests/LearnStack.Tests.Unit/Infrastructure/Messaging/OutboxEventRegistryTests.cs new file mode 100644 index 00000000..10edf9ac --- /dev/null +++ b/backend/tests/LearnStack.Tests.Unit/Infrastructure/Messaging/OutboxEventRegistryTests.cs @@ -0,0 +1,262 @@ +using FluentAssertions; +using LearnStack.Infrastructure.Messaging; +using LearnStack.SharedKernel.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Xunit; + +namespace LearnStack.Tests.Unit.Infrastructure.Messaging; + +public sealed class OutboxEventRegistryTests +{ + private const string ContentTypeSubscription = "customization.initialize-content-type-default.v1"; + private const string TaxonomySubscription = "customization.initialize-taxonomy-default.v1"; + private const string EarlierIdentity = "LearnStack.Previous.TenantCreatedIntegrationEventV1, LearnStack.Previous.Contracts"; + + [Fact] + public void Pending_holds_the_canonical_identity_and_every_alias_without_constructing_handlers() + { + ThrowingHandler.Constructions = 0; + var handlers = Handlers(services => + services.AddScoped, ThrowingHandler>()); + + // A separate Active declaration covers the discovered handler. Merely building + // the manifest must not resolve it, including on the Pending lookup path. + var registry = Create( + [ + OutboxEventRegistration.Required(typeof(SampleEvent), + [ + OutboxSubscriptionManifest.Pending(ContentTypeSubscription), + OutboxSubscriptionManifest.Pending(TaxonomySubscription), + ], EarlierIdentity), + OutboxEventRegistration.Required(typeof(OtherEvent), + [OutboxSubscriptionManifest.Active("test.other.v1", typeof(ThrowingHandler))]), + ], handlers: handlers); + + var canonical = typeof(SampleEvent).AssemblyQualifiedName!; + registry.PendingIdentities.Should().BeEquivalentTo([canonical, EarlierIdentity]); + registry.Resolve(canonical).Disposition.Should().Be(OutboxEventDisposition.Pending); + registry.Resolve(EarlierIdentity).EventType.Should().Be(); + registry.Resolve(EarlierIdentity).Subscriptions.Select(subscription => subscription.Id) + .Should().BeEquivalentTo([ContentTypeSubscription, TaxonomySubscription]); + ThrowingHandler.Constructions.Should().Be(0); + } + + [Fact] + public void Active_and_intentional_absence_have_distinct_exact_outcomes() + { + var registry = Create( + [ + OutboxEventRegistration.Required(typeof(SampleEvent), + [OutboxSubscriptionManifest.Active("test.sample.v1", typeof(SampleHandler))]), + OutboxEventRegistration.IntentionallyUnsubscribed(typeof(OtherEvent), EarlierIdentity), + ], + handlers: Handlers(services => + services.AddScoped, SampleHandler>())); + + registry.Resolve(typeof(SampleEvent).AssemblyQualifiedName).Disposition + .Should().Be(OutboxEventDisposition.Active); + registry.Resolve(EarlierIdentity).Disposition + .Should().Be(OutboxEventDisposition.IntentionallyUnsubscribed); + registry.PendingIdentities.Should().BeEmpty(); + registry.Resolve(typeof(OtherEvent).AssemblyQualifiedName).Subscriptions.Should().BeEmpty(); + } + + [Fact] + public void Unknown_and_retired_names_never_resolve_a_type_from_stored_text() + { + var registry = Create( + [OutboxEventRegistration.IntentionallyUnsubscribed(typeof(SampleEvent))], + retired: [EarlierIdentity]); + + var retired = registry.Resolve(EarlierIdentity); + retired.Disposition.Should().Be(OutboxEventDisposition.Retired); + retired.EventType.Should().BeNull(); + retired.EventTypeLabel.Should().Be("retired"); + + // This is a loadable CLR name in the test assembly, but it was never declared. + // Resolving it would make a database row an assembly-loading instruction. + var undeclared = registry.Resolve(typeof(OtherEvent).AssemblyQualifiedName); + undeclared.Disposition.Should().Be(OutboxEventDisposition.Unknown); + undeclared.EventType.Should().BeNull(); + undeclared.EventTypeLabel.Should().Be("unknown"); + } + + [Theory] + [InlineData("test.sample.v0")] + [InlineData("test.sample.v01")] + [InlineData("Test.sample.v1")] + [InlineData("test.Sample.v1")] + [InlineData("test.sample.v1 ")] + [InlineData("test..v1")] + [InlineData("test.sample.v1.extra")] + [InlineData("test-scope.sample.v1")] + public void Subscription_ids_follow_the_closed_stable_grammar(string id) + { + var build = () => Create( + [OutboxEventRegistration.Required(typeof(SampleEvent), + [OutboxSubscriptionManifest.Pending(id)])]); + + build.Should().Throw(); + } + + [Fact] + public void Alias_and_retired_collisions_refuse_composition() + { + var canonical = typeof(SampleEvent).AssemblyQualifiedName!; + Action duplicateAlias = () => Create( + [OutboxEventRegistration.IntentionallyUnsubscribed(typeof(SampleEvent), + EarlierIdentity, EarlierIdentity)]); + Action canonicalAlias = () => Create( + [OutboxEventRegistration.IntentionallyUnsubscribed(typeof(SampleEvent), canonical)]); + Action retiredAlias = () => Create( + [OutboxEventRegistration.IntentionallyUnsubscribed(typeof(SampleEvent), EarlierIdentity)], + retired: [EarlierIdentity]); + Action duplicateEvent = () => Create( + [ + OutboxEventRegistration.IntentionallyUnsubscribed(typeof(SampleEvent)), + OutboxEventRegistration.IntentionallyUnsubscribed(typeof(SampleEvent)), + ]); + + duplicateAlias.Should().Throw(); + canonicalAlias.Should().Throw(); + retiredAlias.Should().Throw(); + duplicateEvent.Should().Throw(); + } + + [Fact] + public void A_subscription_id_cannot_name_two_independent_consumers() + { + var build = () => Create( + [ + OutboxEventRegistration.Required(typeof(SampleEvent), + [OutboxSubscriptionManifest.Pending(ContentTypeSubscription)]), + OutboxEventRegistration.Required(typeof(OtherEvent), + [OutboxSubscriptionManifest.Pending(ContentTypeSubscription)]), + ]); + + build.Should().Throw(); + } + + [Fact] + public void Active_manifest_and_discovered_handlers_must_match_in_both_directions() + { + Action missingHandler = () => Create( + [OutboxEventRegistration.Required(typeof(SampleEvent), + [OutboxSubscriptionManifest.Active("test.sample.v1", typeof(SampleHandler))])]); + Action undeclaredHandler = () => Create( + [OutboxEventRegistration.IntentionallyUnsubscribed(typeof(SampleEvent))], + handlers: Handlers(services => + services.AddScoped, SampleHandler>())); + Action pendingHandler = () => Create( + [OutboxEventRegistration.Required(typeof(SampleEvent), + [OutboxSubscriptionManifest.Pending(ContentTypeSubscription)])], + handlers: Handlers(services => + services.AddScoped, SampleHandler>())); + + missingHandler.Should().Throw(); + undeclaredHandler.Should().Throw(); + pendingHandler.Should().Throw(); + } + + [Fact] + public void Active_handlers_must_be_closed_constructible_types() + { + var open = typeof(GenericHandler<>); + var discovered = IntegrationEventHandlerRegistry.Discover(typeof(SampleEvent).Assembly); + discovered.All.Should().Contain(subscription => + subscription.EventType == typeof(SampleEvent) && subscription.HandlerType == open, + "the discovery census alone does not reject an open handler"); + + var openHandlers = IntegrationEventHandlerRegistry.FromServiceDescriptors( + [ServiceDescriptor.Scoped(typeof(IIntegrationEventHandler), open)]); + var build = () => Create( + [OutboxEventRegistration.Required(typeof(SampleEvent), + [OutboxSubscriptionManifest.Active("test.sample.v1", open)])], + handlers: openHandlers); + build.Should().Throw().WithMessage("*concrete handler*"); + + var closed = typeof(GenericHandler); + var closedHandlers = IntegrationEventHandlerRegistry.FromServiceDescriptors( + [ServiceDescriptor.Scoped(typeof(IIntegrationEventHandler), closed)]); + var control = Create( + [OutboxEventRegistration.Required(typeof(SampleEvent), + [OutboxSubscriptionManifest.Active("test.sample.v1", closed)])], + handlers: closedHandlers); + control.Resolve(typeof(SampleEvent).AssemblyQualifiedName).Disposition + .Should().Be(OutboxEventDisposition.Active); + } + + [Fact] + public void A_contract_outside_the_explicit_assembly_allowlist_is_rejected() + { + var build = () => OutboxEventRegistry.Create( + [OutboxEventRegistration.IntentionallyUnsubscribed(typeof(SampleEvent))], + [], Handlers(), [typeof(IntegrationEventBase).Assembly]); + + build.Should().Throw(); + } + + [Fact] + public void An_empty_contract_census_cannot_make_every_row_unknown() + { + var build = () => Create([]); + + build.Should().Throw(); + } + + private static OutboxEventRegistry Create( + IEnumerable registrations, + IEnumerable? retired = null, + IntegrationEventHandlerRegistry? handlers = null) => + OutboxEventRegistry.Create( + registrations, + retired ?? [], + handlers ?? Handlers(), + [typeof(SampleEvent).Assembly]); + + private static IntegrationEventHandlerRegistry Handlers( + Action? register = null) + { + var services = new ServiceCollection(); + register?.Invoke(services); + return IntegrationEventHandlerRegistry.FromServiceDescriptors(services); + } + + private sealed record SampleEvent : IntegrationEventBase + { + public override string Topic => "learnstack.test.sample"; + public override string PartitionKey => TenantId.ToString("D"); + } + + private sealed record OtherEvent : IntegrationEventBase + { + public override string Topic => "learnstack.test.other"; + public override string PartitionKey => TenantId.ToString("D"); + } + + private sealed class SampleHandler : IIntegrationEventHandler + { + public Task HandleAsync(SampleEvent integrationEvent, CancellationToken cancellationToken = default) => + Task.CompletedTask; + } + + private sealed class GenericHandler : IIntegrationEventHandler + { + public Task HandleAsync(SampleEvent integrationEvent, CancellationToken cancellationToken = default) => + Task.CompletedTask; + } + + private sealed class ThrowingHandler : IIntegrationEventHandler + { + public static int Constructions; + + public ThrowingHandler() + { + Constructions++; + throw new InvalidOperationException("Handler construction was reached."); + } + + public Task HandleAsync(OtherEvent integrationEvent, CancellationToken cancellationToken = default) => + Task.CompletedTask; + } +} diff --git a/docs/architecture/04-technical-architecture.md b/docs/architecture/04-technical-architecture.md index ec3183aa..14df334d 100644 --- a/docs/architecture/04-technical-architecture.md +++ b/docs/architecture/04-technical-architecture.md @@ -40,9 +40,9 @@ flowchart LR end subgraph backend["LearnStack Backend"] - api[ASP.NET Core API
Modular Monolith] + api[ASP.NET Core API
Modular Monolith + OutboxProcessor] daprd1[Dapr sidecar] - workers[Hangfire Workers
jobs + outbox dispatcher] + workers[Hangfire Workers
background jobs] end subgraph hubplane["LearnStack Hub (separate codebase)"] @@ -173,13 +173,13 @@ See [Database Standards](../standards/05-database.md). flowchart LR trans[Module transaction] outbox[(outbox table)] - dispatcher[Outbox dispatcher
Hangfire recurring job] + dispatcher[API OutboxProcessor
BackgroundService] bus[In-process event bus] handlers[Integration event handlers] externals[External calls
email / search index / etc.] trans -- domain change + integration event row --> outbox - dispatcher -- pull batch --> outbox + dispatcher -- claim one admitted row --> outbox dispatcher -- dispatch --> bus bus --> handlers bus --> externals @@ -187,7 +187,10 @@ flowchart LR ``` - All side-effects beyond the database transaction are deferred to **integration events** written to a transactional `outbox` table in the same transaction as the domain change. -- A dispatcher polls the outbox and dispatches events to in-process handlers and external systems. +- The API-host `OutboxProcessor` polls the outbox and publishes admitted events + through `IEventBus`. P02b-2 Step 3 implements hosted execution, with + both independent review rounds complete; real required subscriptions remain Pending + until P02b-3 supplies durable consumer execution. - Idempotency keys plus consumer-side deduplication guarantee at-least-once with effective once-only semantics. Full details: [Events & Outbox](15-event-and-outbox.md). diff --git a/docs/architecture/09-tenant-isolation.md b/docs/architecture/09-tenant-isolation.md index fe05bbde..bc2136e4 100644 --- a/docs/architecture/09-tenant-isolation.md +++ b/docs/architecture/09-tenant-isolation.md @@ -161,8 +161,8 @@ announcement; the snapshot cannot retarget the transaction. The closed writer set, host factory ceiling and canonical RLS placement remain unchanged under [ADR-0057](../decisions/0057-transactional-event-production-and-completion.md). P02b-1 Steps 2–3 deliver immediate outbox enqueue, successful-save collection -and owner-only completion. Dispatch and consumer delivery remain P02b-2/P02b-3 -work. +and owner-only completion. P02b-2 Step 3 implements confined hosted dispatch; +both independent review rounds are complete. Consumer delivery remains P02b-3. ## Default org semantics diff --git a/docs/architecture/10-cross-module-contracts.md b/docs/architecture/10-cross-module-contracts.md index 7e67abcb..4623064a 100644 --- a/docs/architecture/10-cross-module-contracts.md +++ b/docs/architecture/10-cross-module-contracts.md @@ -13,6 +13,14 @@ Modules collaborate only through explicit contracts. This keeps the modular mono > (Mechanism #2), and read-model projections (Mechanism #4) are unchanged. See > [15-event-and-outbox.md](15-event-and-outbox.md) for the full producer/consumer flow. +**2026-10-12 delivery decision.** +[ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md) supersedes the +2026-08-26 note's per-module inbox placement: P02b-3 owns a shared transport-claimed +inbox keyed by tenant, stable subscription and event. Claiming precedes handler +construction; handlers do not call the guard. The four communication mechanisms +and demand-gated broker trigger remain unchanged. This records the accepted target, +not delivered subscriber machinery. + ## Allowed Contract Types ```mermaid diff --git a/docs/architecture/15-event-and-outbox.md b/docs/architecture/15-event-and-outbox.md index 075a0630..d2d09a61 100644 --- a/docs/architecture/15-event-and-outbox.md +++ b/docs/architecture/15-event-and-outbox.md @@ -24,10 +24,11 @@ not land together. | Piece | Owning phase | State | |---|---|---| -| `outbox_messages` table, its schema, and its LearnStack ownership | [Phase 02a Packet 6](../roadmap/phase-02a-kernel-tenancy.md) | Ships even though nothing dispatches from it yet — the schema and its ownership are a one-way door | +| `outbox_messages` table, its schema, and its LearnStack ownership | [Phase 02a Packet 6](../roadmap/phase-02a-kernel-tenancy.md) | Schema and ownership shipped in Packet 6; producer dispatch is implemented in P02b-2 Step 3 | | `IEventBus` port + `InProcessEventBus` | [Phase 02a Packet 5](../roadmap/phase-02a-kernel-tenancy.md) | The only registered implementation | | `IOutbox` and `TenantCreatedIntegrationEventV1` | [P02b-1](../roadmap/phase-02b-events-auth.md#p02b-1-decision-package-2026-10-12) | Implemented immediate transactional production; no delivery yet | -| `OutboxProcessor`, `IInboxGuard` and the claim protocol | [P02b-2/P02b-3](../roadmap/phase-02b-events-auth.md) | Pending durable dispatch and consumer transactions | +| `OutboxProcessor` and the producer claim protocol | [P02b-2](../roadmap/phase-02b-events-auth.md#p02b-2-step-3--hosted-lifecycle-and-observability-2026-10-12) | Producer runtime and enforcement locally complete; all four steps completed both independent review rounds; CI/merge status: [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) | +| `IInboxGuard` and durable consumer transactions | [P02b-3](../roadmap/phase-02b-events-auth.md) | Pending | | Domain events collected and dispatched in-process, inside the ambient transaction | [P02b-1](../roadmap/phase-02b-events-auth.md#p02b-1-step-3--domain-completion-and-provisioning-publisher-2026-10-12) | Successful-save collection and owner-only completion implemented | | `DaprEventBus` → Dapr pub/sub → Kafka | [Phase 11](../roadmap/phase-11-production-hardening.md) | Demand-gated; trigger: a second process needs to consume an integration event, or event volume / replay / cross-process ordering is required ([ADR-0035](../decisions/0035-demand-gated-infrastructure.md)) | @@ -38,8 +39,8 @@ or persist retries. `IInboxGuard` is a documented seam, not a declared type. [Phase 02b's entry verification](../roadmap/phase-02b-events-auth.md#p02b-0-entry-verification-2026-10-12) records that baseline. The producer/consumer diagrams and examples below describe the durable target: P02b-1 supplies enqueue, P02b-2 dispatch and P02b-3 consumer -transactions, inbox and recovery. G1 settles the inbox's ownership and caller; -this entry audit does not replace the Accepted per-module contract. +transactions, inbox and recovery. Accepted ADR-0046 now assigns inbox ownership +and stable subscriber identity; implementation remains P02b-3. The [P02b-1 decision package](../roadmap/phase-02b-events-auth.md#p02b-1-decision-package-2026-10-12) and [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md) @@ -48,8 +49,24 @@ protection. P02b-1 Steps 1–3 implement shared execution, immediate enqueue, successful-save collection and the completed-provisioning publisher. Step 4 implements production-census guards and catalogue enforcement; its [delivery record](../roadmap/phase-02b-events-auth.md#p02b-1-step-4--producer-enforcement-and-closeout-2026-10-12) -owns final verification and review status. Dispatcher/inbox -choices remain P02b-2/P02b-3. +owns final verification and review status. ADR-0046 accepts dispatcher/inbox +contracts; producer dispatch is implemented through P02b-2 Step 3, while +inbox/consumer implementation remains P02b-3. + +The [P02b-2 decision package](../roadmap/phase-02b-events-auth.md#p02b-2-decision-package-2026-10-12) +and [Accepted ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md) +authorize fenced producer recovery, atomic terminal audit and the coupled shared +inbox/subscriber contract. Steps 1–2 implement schema/composition, fenced +transitions and atomic terminal audit; each completed both review rounds. +Step 3 implements hosted execution and observation, with both independent +review rounds complete in its +[delivery record](../roadmap/phase-02b-events-auth.md#p02b-2-step-3--hosted-lifecycle-and-observability-2026-10-12). +Step 4 enforcement and local closeout are complete; all four steps completed both +independent review rounds. The [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) +owns CI/merge status. +P02b-3 inbox/consumer delivery remains pending. Numerical +runtime bounds live in the package, canonical DDL/grants in Standards 05 and +metric names in Standards 10. [ADR-0038](../decisions/0038-cross-cutting-port-and-event-contracts.md) stands as the decision that Dapr is the @@ -69,44 +86,31 @@ decides when it arrives, and the answer is "when a second process exists". ```mermaid sequenceDiagram - participant Module as Owning Module + participant Producer as Owning module participant DB as PostgreSQL - participant Outbox as outbox_messages - participant Processor as OutboxProcessor (BackgroundService) + participant Worker as OutboxProcessor participant Bus as IEventBus - participant Transport as Transport (in-process dispatcher, or Dapr to Kafka) - participant Consumer as Consuming Module - participant Inbox as inbox_messages - - Module->>DB: Aggregate state change - Module->>Outbox: INSERT outbox row (same transaction) - DB->>DB: COMMIT (atomic: aggregate + outbox row) - - loop Polling (every 200ms; configurable) - Processor->>Outbox: CLAIM — UPDATE SET locked_by, locked_until
WHERE id IN (SELECT ... FOR UPDATE SKIP LOCKED)
RETURNING *; the claim survives the COMMIT - Processor->>Bus: PublishAsync(envelope)
topic + correlation from the row - Bus->>Transport: deliver - Processor->>Outbox: UPDATE processed_at = now()
WHERE id = @id AND locked_by = @me - end - - Transport->>Consumer: deliver to subscribed handler - Consumer->>Consumer: Restore tenant context from event.TenantId - Consumer->>Inbox: IsAlreadyProcessedAsync(event.EventId) - alt Already processed - Inbox-->>Consumer: skip - else New event - Consumer->>Consumer: Handle business logic - Consumer->>Inbox: MarkAsProcessed(event.EventId, event type name) - Consumer->>DB: SaveChangesAsync (business write + inbox marker, atomic) - end + participant Executor as P02b-3 delivery executor + Producer->>DB: State + outbox + MUST audit on one transaction + Producer->>DB: COMMIT + Worker->>DB: Claim one eligible row, fresh token + lease + attempt + Worker->>Bus: Publish validated envelope outside claim transaction + Bus->>DB: P02b-3 durable acceptance of required subscriptions + Worker->>DB: Fenced processed transition + Executor->>DB: App-role transaction + tenant/org announcement + Executor->>DB: Shared inbox claim before handler construction + Executor->>DB: Business + inbox + outbox + MUST audit + fenced completion + Executor->>DB: COMMIT ``` -Read as text: the producer writes the aggregate change and the outbox row in one -transaction. A processor **claims** a batch of unprocessed rows by stamping a lease on -them, and the claim outlives the claiming transaction. It publishes each claimed row and -marks it processed only if it still holds the lease. The transport delivers to the -consumer, which restores tenant context, deduplicates through its inbox, and commits the -business write and the inbox marker together. +Text fallback: the producer commits state, outbox and audit together. The worker +claims one admitted row immediately before publishing and fences its acknowledgement. +Required Pending subscriptions hold rows before acquisition. P02b-3 durably accepts +required subscriptions, then executes each on its own app-role transaction. Shared +inbox completion and business/audit/ledger completion commit together. Producer +production and the API worker are implemented; both Step 3 review rounds are +complete. Durable consumer execution remains P02b-3, and real required +subscriptions remain Pending. ## Outbox table @@ -118,81 +122,59 @@ third document; the last time this table's policy was duplicated, the copy carri superseded two-permissive-policy shape that [ADR-0003 Amendment 3](../decisions/0003-tenant-isolation-defense-in-depth.md) corrects. -Three columns exist for reasons this document owns rather than the schema. They -support partition routing and the pending P02b-2 dispatch/lease decision; the -partition column alone establishes no durable publish-order guarantee: - -| Column | Why it exists | Lands with | -|---|---|---| -| `partition_key text NOT NULL` | The event's declared routing key; durable publish order remains P02b-2 G3. **The event declares it** — `PartitionKey` is abstract on `IntegrationEventBase`, so no event inherits a default — and `IOutbox.EnqueueAsync` copies it onto the row. Nothing resolves or re-derives it. | The table itself, in [Phase 02a Packet 6](../roadmap/phase-02a-kernel-tenancy.md) — a producer-side column is cheaper to ship with the table than to backfill | -| `locked_by text NULL` + `locked_until timestamptz NULL` | The dispatch **lease**. A processor stamps them to claim a row, and the claim survives the claiming transaction's commit. See the claim protocol below. | The dispatcher, in [Phase 02b](../roadmap/phase-02b-events-auth.md) | -| `available_after timestamptz NOT NULL` | Retry backoff. A failed dispatch pushes this forward instead of blocking the batch. | Already in the canonical DDL | - -Both additions go into the canonical DDL in Standards 05 when their phase ships; this -document does not carry a second copy of the table. - -The pending index therefore covers `(available_after)` filtered on -`processed_at IS NULL`, and the claim predicate additionally reads -`locked_until IS NULL OR locked_until < now()`. +The routing key is declared by the event and copied to persistence. It does not +establish durable publish order. Accepted P02b-2 adds paired `locked_by` / +`locked_until` lease fields, `terminal_reason` and `dead_lettered_at`; the former +is a fresh acquisition token, never stable worker identity. `available_after` +controls due time and retry backoff. Canonical DDL/indexes are in Standards 05, +implemented by P02b-2 Step 1 with both review rounds complete. Step 3 implements +dispatcher execution; both independent review rounds are complete. ### Why the `learnstack_outbox_admin` role bypasses RLS — and what bounds the bypass -The OutboxProcessor must dispatch events for **every** tenant from a single -BackgroundService instance; setting `app.tenant_id` per row would defeat the -`FOR UPDATE SKIP LOCKED` batching pattern. The role bypasses RLS by design, and -the bypass is bounded by: - -- **Scope by grant, and grant is the only bound.** `BYPASSRLS` bypasses *policies*, not - `GRANT`s: a role holding the attribute with no privilege on a table gets - `permission denied for table`. The role holds `SELECT` on `outbox_messages` plus a - **column-level** `UPDATE (processed_at, attempts, last_error, available_after)` — and - nothing anywhere else. It has no `DELETE` (purging processed rows is a - `learnstack_platform` operation) and cannot touch `payload`, `tenant_id`, `topic` or - `type`. `SELECT … FOR UPDATE SKIP LOCKED` works with a column-level `UPDATE` grant, so - the claim protocol needs no table-wide `UPDATE`. When `locked_by` and `locked_until` - land in [Phase 02b](../roadmap/phase-02b-events-auth.md), that migration extends the - column list; a column added without extending it fails at runtime with - `permission denied for table`, which is the intended loud failure. Because grant scope - is the *only* thing bounding the bypass, no privilege is ever granted to `PUBLIC` and - no `ALTER DEFAULT PRIVILEGES` grant exists — either would silently un-bound this role - the next time a table is created. -- **No connection sharing.** The role has its own login credential and its own - connection string (`ConnectionStrings:OutboxDispatcher`), present only in the worker - host that runs `OutboxProcessor`. It is not reached by `SET ROLE` from - `learnstack_app`, which is not a member of it. No MediatR handler or API endpoint ever - runs under this role (architecture test - `LearnStack_OutboxAdmin_Role_OnlyUsedBy_OutboxProcessor` enforces this). -- **Audited on use**. Every dispatch attempt produces a structured log entry - with `event_id`, `tenant_id`, `event_type`, and outcome. Bypass-as-such is - not separately audited (the dispatch outcome is); a misuse pattern would - surface as out-of-band tenants in the dispatch log. -- **Permission rationale anchored in ADR-0006 Amendment 1**. The outbox-as- - durable-buffer + Dapr-as-transport pattern requires a single batch-fetcher; - the RLS bypass is the minimum privilege escalation that makes that pattern - work. It is not an exception to ADR-0003 — it is the only legitimate path. - -`learnstack_outbox_admin` is one of the **four** roles in the database model fixed by -[ADR-0003 Amendment 3](../decisions/0003-tenant-isolation-defense-in-depth.md) — -`learnstack_migration` (owner, `NOBYPASSRLS`), `learnstack_app` (runtime, -`NOBYPASSRLS`), `learnstack_platform` (audited cross-tenant admin, `BYPASSRLS`), and -this one. Cross-tenant audit reads use `learnstack_platform` through the audited -`EnterPlatformAdminScope(reason)` path; there is no separate audit role. Adding a fifth -role requires an ADR — every `BYPASSRLS` role is a hole in the isolation model that has -to earn its existence by grant scope, DI-scope isolation, and log discoverability. - -## Inbox table (per consumer module) - -```sql -CREATE TABLE inbox_messages ( - event_id uuid PRIMARY KEY, - event_type text NOT NULL, - received_at timestamptz NOT NULL DEFAULT now(), - processed_at timestamptz NULL -); -``` - -Each module has its own inbox table; deduplication is per-module (a single event can be -processed by N consumers independently). +One API-host BackgroundService schedules every tenant. The existing bypass role +uses its own named keyed, lazy data source; no Seeder/web forwarding or app-role +membership is allowed. Configuration validates the exact username without +connecting or disclosing parse input. Step 3 implements mandatory worker-startup +resolution and connectivity checks; both independent review rounds are complete. +Every new physical connection verifies exact session/current role, bounded +attributes and no privilege-escalating membership. + +[Database Standards](../standards/05-database.md#grant-matrix) owns the exact grant +budget: named outbox status UPDATEs and audit INSERT-only in P02b-2, with no audit +read/update/delete or business access. P02b-3 may add only enumerated delivery +identity/routing/lease/status SELECT and status UPDATE, never payload SELECT. +It executes business work separately as `learnstack_app` under tenant RLS. + +Routine confined poll/claim/mark writes bounded structured outcomes and metrics, +under ADR-0046's explicit audit exemption. Terminal state and MUST/SecurityEvent +audit commit atomically; a restricted witness binds the audit to the actual fenced +UPDATE result, because GUCs and INSERT-only cannot tenant-bind a bypass credential. +The platform-admin gate and pre-entry audit remain unchanged. Four roles and +separate credentials remain the closed model; the code-path and role rules are +implemented for transition-witness construction and terminal audit in Step 2; +both independent review rounds passed. Step 3 implements hosted execution and +observation, with both independent review rounds complete. Step 4 implements +credential/publication censuses and planted controls; all four steps completed +both independent review rounds and local closeout. The [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) +owns CI/merge status. + + + +## Inbox table (shared, transport-owned) + +**Accepted ADR-0046 target; implementation P02b-3.** One tenant-owned +`inbox_messages` deduplicates `(tenant_id, subscription_id, event_id)`. The +transport claims before handler construction on the same announced app-role +transaction as business work. New claims roll back on failure; only successful +completion persists a completed row. Completed rows are never reset by replay. + +The separate delivery ledger retains validated envelopes and owns independent +attempts, availability, leases and terminal/replay state. P02b-3 chooses envelope +copies/references and retention dependencies before its canonical DDL, then +P02b-8 inventories all retained stores for erasure. DDL, class/RLS and exact grants +belong in [Database Standards § Inbox](../standards/05-database.md#inbox); this +document carries no second schema. ## Producer pattern @@ -279,7 +261,7 @@ Step 7's `OutboxFlushBehavior` is an enrollment/invariant guard. The reusable audit coordinator gives a controlled non-MediatR owner the existing classification, MUST-write, boundary-reporting and outer reconciliation duties. -P02b-3 owns production delivery/inbox wiring after G1; the P02b-1 fixture proves +P02b-3 owns delivery/inbox wiring under ADR-0046; the P02b-1 fixture proves that entry without claiming delivered subscribers. ### First useful flow @@ -287,8 +269,9 @@ that entry without claiming delivered subscribers. Completed provisioning produces `TenantCreatedIntegrationEventV1` on `learnstack.tenancy.tenant`, partitioned by TenantId, with no extra payload and null organization. P02b-1 persists pending rows after tenant/default organization/ -back-reference completion. P02b-3, after G1, implements independent Customization -subscriptions for untouched `card@1` and `plain@1`; each owns one root, generation, +back-reference completion. P02b-3 implements independent Customization +subscriptions for the untouched `card@1` content type (`BuiltInCustomizations.Card`) +and `plain@1` taxonomy (`BuiltInCustomizations.Plain`); each owns one root, generation, inbox outcome and MUST audit. Authored/lifecycle/deletion decisions survive. P02b-3 owns semantic key/version Seeder verification and fresh seed/delivery convergence. Typed settings are uncached, definitions probe fresh generations, @@ -307,35 +290,27 @@ everything. ## Consumer pattern -```csharp -public sealed class QueueWelcomeNotificationOnEnrollmentCreated( - IInboxGuard inboxGuard, - NotificationsDbContext db, - ILogger logger) - : IIntegrationEventHandler -{ - public async Task HandleAsync(EnrollmentCreatedIntegrationEventV1 @event, CancellationToken ct) - { - if (await inboxGuard.IsAlreadyProcessedAsync(@event.EventId, ct)) - { - logger.LogDebug("Duplicate event {EventId} skipped", @event.EventId); - return; - } - - // Business logic: the consumer writes its OWN module's aggregate, through its - // own DbContext. Not another module's tables, and never audit_log — see below. - var dispatch = NotificationDispatch.QueueWelcome(@event.LearnerId, @event.EnrollmentId); - db.NotificationDispatches.Add(dispatch); - - inboxGuard.MarkAsProcessed(@event.EventId, @event.GetType().Name); - await db.SaveChangesAsync(ct); // business write + inbox marker, atomic - } -} -``` - -The handler's effective identity is `UserId.SystemActor` (see [Rules](#rules)), and -[Audit Coverage Standards](../standards/18-audit-coverage.md) treats a state-mutating -handler as a `system` actor for audit purposes. The handler composes no audit row itself. +**Accepted target, unimplemented until P02b-3.** Each subscription has an explicit +stable `{module}.{subscription}.v{N}` ID, independent of handler class/assembly. +ADR-0046 owns its closed grammar and version/migration rules. The transport opens +an app-role transaction, restores validated tenant and organization, announces RLS +context, and claims the shared inbox before constructing the handler. Absence of +the organization marker never erases a present envelope organization; a marked +event requires one. Effective actor is `UserId.SystemActor`; human actor remains +causal metadata. + +The common transaction/audit owner classifies the handler and refuses undeclared +mutating work. A handler changes only its own root and saves; it calls neither +`IInboxGuard` nor `IEventBus` nor `IAuditStore`. The owner drains produced facts, +writes MUST audit and completes inbox/ledger atomically. Before COMMIT, ledger +completion checks the current token and database-time expiry, plus a sticky gate +that refuses timed-out/canceled execution. Zero rows or a closed gate poison and +roll back the whole transaction. Cancellation after physical COMMIT starts is +Indeterminate; recover from atomic durable state, not a rollback assumption. + +External effects require provider idempotency and reconciliation. Privileged +Audit erasure belongs to its separate Phase 03 execution contract; P3's ordinary +executor refuses it. An app-role inbox cannot make two connections atomic. ### Audit rides this outbox for fan-out, never for the audit row @@ -367,6 +342,8 @@ is the rule that enforces it, in force since Phase 02a Packet 10. ### The claim problem +The former batch sketch exposed the released-lock defect described here; the +single-row leased protocol below supersedes it under ADR-0046. `SELECT ... FOR UPDATE SKIP LOCKED` holds its row locks **only until the transaction that took them ends**. A processor that selects a batch, commits to release the locks, and *then* starts publishing has released every row it is about to work on. A second @@ -375,136 +352,71 @@ not blocked by any lock, and dispatches the same events. The first processor the them processed, and nothing in the system records that they went out twice. That is why this document no longer claims "no double dispatch". The delivery contract is -**at-least-once**, and consumer-side idempotency through `IInboxGuard` is what makes it -safe — not the row lock. What the claim protocol buys is that duplicates come from crash -recovery and lease expiry, which are rare and bounded, rather than from every concurrent -poll, which is neither. +**at-least-once**. P02b-3's transport-owned shared inbox supplies consumer-side +idempotency; handlers do not call `IInboxGuard`. The claim protocol prevents +concurrent live owners; crash recovery and lease expiry can still redeliver. ### The claim protocol -The row lock has to survive the transaction that takes it, which means the claim must be -**written to the row**, not held in the lock table: - -```csharp -public sealed class OutboxProcessor : BackgroundService -{ - private const int BatchSize = 100; - private static readonly TimeSpan PollInterval = TimeSpan.FromMilliseconds(200); - private static readonly TimeSpan LeaseDuration = TimeSpan.FromSeconds(60); - - private readonly string _processorId = $"{Environment.MachineName}:{Guid.NewGuid():N}"; - - protected override async Task ExecuteAsync(CancellationToken ct) - { - while (!ct.IsCancellationRequested) - { - using var scope = _serviceProvider.CreateAsyncScope(); - var processed = await ProcessBatchAsync(scope.ServiceProvider, ct); - - if (processed == 0) - await Task.Delay(PollInterval, ct); - } - } - - private async Task ProcessBatchAsync(IServiceProvider services, CancellationToken ct) - { - var db = services.GetRequiredService(); - var eventBus = services.GetRequiredService(); - // Connection runs as learnstack_outbox_admin: BYPASSRLS skips the policy, - // and the GRANT is what bounds what it can reach (see § the bypass bounds). - - // 1. CLAIM. One statement, its own short transaction. The lease is written to the - // row, so it outlives the commit that releases the physical locks. - var batch = await db.Set() - .FromSqlInterpolated($@" - UPDATE outbox_messages - SET locked_by = {_processorId}, - locked_until = now() + {LeaseDuration}, - attempts = attempts + 1 - WHERE id IN ( - SELECT id FROM outbox_messages - WHERE processed_at IS NULL - AND available_after <= now() - AND (locked_until IS NULL OR locked_until < now()) - ORDER BY occurred_at - LIMIT {BatchSize} - FOR UPDATE SKIP LOCKED) - RETURNING *") - .ToListAsync(ct); - - if (batch.Count == 0) return 0; - - // 2. PUBLISH. One short transaction per message. A single poisoned message does - // not roll back the batch. - foreach (var msg in batch) - { - await using var perMessageTx = await db.Database.BeginTransactionAsync(ct); - try - { - var eventInstance = JsonSerializer.Deserialize(msg.Payload, Type.GetType(msg.Type)!); - await eventBus.PublishAsync( - new IntegrationEventEnvelope( - (IIntegrationEvent)eventInstance!, - msg.CorrelationId, - msg.OrganizationId, - msg.CausationId, - msg.ActorUserId is { } actor ? UserId.From(actor) : null), - ct); - msg.MarkProcessed(_clock.UtcNow, _processorId); // no-op if the lease was lost - } - catch (Exception ex) - { - msg.RecordFailure(ex.Message, GetBackoffDuration(msg.Attempts)); - _logger.LogWarning(ex, "Outbox dispatch failed for {EventId} (attempt {Attempt})", - msg.Id, msg.Attempts); - } - await db.SaveChangesAsync(ct); - await perMessageTx.CommitAsync(ct); - } - return batch.Count; - } -} -``` - -`MarkProcessed` writes `processed_at` **and** clears the lease under the predicate -`locked_by = @me`, so a processor whose lease expired mid-batch — because the transport -hung past `LeaseDuration` — cannot overwrite the state of whichever processor took the -row next. A lost lease is logged at warning level and surfaces on the -`learnstack_outbox_lease_lost_total` counter; a non-zero rate means `LeaseDuration` is -shorter than the transport's tail latency and must be raised. +**ADR-0046 producer runtime and enforcement locally complete in P02b-2; all four +steps completed both independent review rounds.** The [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) +owns CI/merge status. The processor admits a construction-free +manifest policy before acquisition. Any required Pending subscription holds the +contract and all aliases unchanged, even if stored payload is malformed. Active +requires declared handler registration. Only explicit intentional absence may +acknowledge without publication; zero handler discovery is not that policy. + +One short transaction claims one eligible row immediately before invocation, +assigns a fresh canonical UUID token to `locked_by`, stamps `locked_until` and +increments `attempts` once. Stable worker identity is diagnostic context only. +Normal eligibility excludes processed, dead-lettered, terminal-requested and +exhausted rows; it requires due availability, no unrecovered lease and manifest/ +local-partition admission. Selection is `(available_after, occurred_at, id)`. +Database time governs eligibility and expiry. + +For admitted records, resolve only declared concrete contracts and explicit +compatible aliases/retirements. Never load arbitrary assemblies or CLR types from +stored text. Deserialize with frozen options; validate row/envelope identity, +tenant, topic, partition, occurrence and canonical UTF-8 payload bound. Timestamp +comparison accounts for PostgreSQL microsecond precision. Invalid/unknown/retired +records request terminalization with a closed reason. + +Every success, failure, recovery and terminal transition checks row ID, current +token and unexpired lease. Recovery atomically replaces an expired token first. +Zero rows means lost ownership and permits no late acknowledgement, error change, +lease clearance or audit. Expired claims 1–5 enter backoff; claim six can only +terminalize. Six normal claims total use 1 s, 5 s, 30 s, 5 min and 1 h delays. +A committed claim counts even if a crash makes publication unknown. + +A terminal request is not completed dead-letter state. Terminalization acquires +without another attempt or publication, then atomically writes terminal state +and MUST audit using the row-bound transition witness. Audit failure retains the +request for retry. Success/retry/terminal transitions clear leases in the same +short transaction. Uncertain COMMIT is reconciled from durable state before retry. + +Publish deadlines bound waiting, not arbitrary callback lifetime. Retain and +observe a cancellation-ignoring task's scope until it finishes; block its local +partition and fence its late result. Unrelated keys progress within the approved +four-task limit; any overrun is unhealthy and saturation stops new claims. +Shutdown waits at most the package's finite bound, without disposing live resources. ### The simpler alternative, and its cost -Holding one transaction open across the whole batch — `BEGIN`, `SELECT ... FOR UPDATE -SKIP LOCKED`, publish each, `UPDATE processed_at`, `COMMIT` — is also correct, and is an -acceptable implementation of the same contract. Its costs are real and must be accepted -knowingly: - -- The transaction stays open for the duration of every network publish in the batch, - pinning the `VACUUM` horizon and the oldest transaction id for that long. -- One poisoned message rolls back the batch unless every publish is wrapped in a - savepoint, which reintroduces most of the complexity the approach was meant to avoid. -- A crash mid-batch republishes everything already published in it. - -The lease is the specified design. The batch-held transaction is the acceptable fallback -if a deployment cannot add the two columns. Whichever ships, the choice is made once, in -[Phase 02b](../roadmap/phase-02b-events-auth.md), and written into the dispatcher — not -left to each reader of this document. +The former batch-held transaction fallback is superseded by ADR-0046. P02b-2 uses +one just-in-time leased claim and short fenced transitions; it holds no transaction +across publication. A deployment-specific alternative requires a new decision. ### What the processor guarantees -- **At-least-once delivery.** Duplicates are expected; consumers deduplicate through - `IInboxGuard`. This is the contract, not a defect. -- **At most one active claimant per row.** Two processors cannot both hold a live lease - on the same row, so concurrent polling does not multiply dispatches. Duplicates arise - only from lease expiry and crash recovery. -- **Batch independence.** Each message gets its own transaction; one failure does not - roll back the batch. -- **Bounded retry.** Failed messages retry with exponential backoff (1s, 5s, 30s, 5min, - 1h). On reaching `MaxAttempts` the row moves to the producer-side dead-letter state - described below. -- **Horizontal scalability.** `SKIP LOCKED` plus the lease predicate lets N processors - across N pods drain the same table without coordination. +- At most one live owner per row under token/expiry fencing. Crash or lease expiry + can redeliver; transport publication is at-least-once. +- Deterministic selection for one processor without failure, with no enqueue, + commit, retry or cross-process order promise. +- Ordinary failure and bounded overruns permit unrelated eligible work to progress. +- Pending real subscriptions retain useful facts until P02b-3 activates durable + acceptance. No placeholder consumer or global worker-disable switch is used. +- Terminal state commits with MUST audit; it does not prove that a timed-out + transport never accepted the event. P02b-3's inbox is still required. ## `IEventBus` and `IntegrationEventEnvelope` @@ -586,7 +498,8 @@ public sealed class DaprEventBus(DaprClient daprClient) : IEventBus `InProcessEventBus` serializes invocation per partition key within its process: concurrent across keys, sequential within one. This does not establish durable -publish order across commits and retries; P02b-2 G3 owns that decision. +publish order across commits and retries; ADR-0046 accepts only deterministic +eligible selection within one processor. Topic naming convention: `learnstack.{module}.{aggregate}`. Examples: @@ -601,119 +514,34 @@ Topic naming convention: `learnstack.{module}.{aggregate}`. Examples: ## `InProcessEventBus` -Until the Dapr adapter's trigger fires, `InProcessEventBus` is the only registered -`IEventBus`. The table below states its target obligations alongside the durable -path. Handler scopes and accessor restoration ship today; database announcement -and inbox deduplication remain P02b-3 work, under G1/G6. - -| Obligation | Why it cannot be skipped in development | -|---|---| -| Same handler interface — `IIntegrationEventHandler`, never a bare `INotificationHandler` | Two interfaces means two implementations per consumer, and the one that runs in CI is not the one that runs in production | -| Same `IInboxGuard` deduplication | Deduplication bugs are the most common integration-event defect; a transport that never delivers a duplicate never surfaces them | -| Same tenant-context restoration from `@event.TenantId` before the handler runs | A dev path that skips it is a dev path where Row Level Security and the query filters are never exercised on the consumer side | -| Process-local per-key invocation serialization; durable ordering remains P02b-2 G3 | A serialized development invocation does not prove publish order across concurrent commits and retries | - -```csharp -public sealed partial class InProcessEventBus( - IServiceScopeFactory scopeFactory, - ITenantContextAccessor tenantAccessor, - IPartitionSerializer partitions, - IntegrationEventHandlerRegistry handlers, - ILogger logger) : IEventBus -{ - public Task PublishAsync( - IntegrationEventEnvelope envelope, CancellationToken ct = default) - { - ArgumentNullException.ThrowIfNull(envelope); - - // Returned as a cancelled task rather than thrown inline: a - // fire-and-forget call site must not crash its caller synchronously. - if (ct.IsCancellationRequested) return Task.FromCanceled(ct); - - return partitions.RunSequentiallyFor( - envelope.PartitionKey, () => DispatchAsync(envelope, ct)); - } - - private async Task DispatchAsync(IntegrationEventEnvelope envelope, CancellationToken ct) - { - // The outer save/restore covers the FULL dispatch. Context is set before - // subscription lookup and therefore before any handler can be resolved; - // constructors observe the event tenant rather than the publisher tenant. - var previous = tenantAccessor.Current; - tenantAccessor.Current = EventTenantContext.FromEnvelope(envelope); - - try - { - var subscriptions = handlers.For(envelope.Event.GetType()); - List? failures = null; - - foreach (var subscription in subscriptions) - { - ct.ThrowIfCancellationRequested(); - try - { - // DeliverAsync creates ONE ASYNC SCOPE and resolves exactly - // this subscription's concrete handler. A constructor or - // disposal failure cannot deny a healthy sibling. It also - // starts the subscription's consumer activity and supplies - // its module name through EventTenantContext. - await DeliverAsync(subscription, envelope, ct); - } - catch (OperationCanceledException) when (ct.IsCancellationRequested) - { - throw; // shutdown: do not start later subscriptions - } - catch (Exception ex) - { - // Poison-message containment remains per subscription. - (failures ??= []).Add(ex); - } - } - - // One failure rethrown as itself; several as an AggregateException. - } - finally - { - tenantAccessor.Current = previous; - } - } // Inbox ownership and its call site remain Phase 02b G1. -} -``` - -The listing is abridged — the shipped class also logs each failure with event, tenant -and partition key, refuses a `null` Task from a handler by name, and wraps an -`OperationCanceledException` raised by a token other than the publish token, so an -outbox processor cannot read "the handler gave up" as "we are shutting down". See -`backend/src/LearnStack.Infrastructure/Messaging/InProcessEventBus.cs`. - -**`PartitionSerializer` refuses reentrant work for the key it is already inside.** -Queuing work for a key from within that key's own work is a deadlock by construction: -the new unit chains behind a tail that cannot complete until the current one returns. -An earlier attempt ran such a call inline, reasoning the caller *is* the sequence — which -is unsound, because an `AsyncLocal` marker flows into every task started inside a unit, -so a fire-and-forget spawn inherited it and ran concurrently with the unit it should have -queued behind. Detection is the same either way; only the action differs. A false -positive that throws is loud and diagnosable; one that runs inline is a silent -concurrency violation. The caller that hits it is publishing from inside a handler, -which [Standards 20](../standards/20-infrastructure-stack.md) already forbids — a -handler writes to the outbox. - -What the in-process transport genuinely does **not** provide, and what therefore -constitutes the trigger for the Dapr adapter: delivery to a second process, broker-side -replay, and durability of the in-flight message if the process dies between publish and -handle. Once P02b-1–3 deliver durable enqueue and recovery, the outbox row covers -the third: an unhandled event has not been marked processed and is redelivered. -The existing table alone does not provide that recovery. - -The composition root selects the implementation by `DeploymentMode`; modules never see -the choice. +`InProcessEventBus` remains the only registered transport in every deployment +mode until ADR-0035's Phase 11 trigger fires. The shipped baseline constructs one +async scope per discovered subscription, restores ambient tenant/organization, +serializes invocation per key and collects constructor/handler/disposal failures. +It opens no business transaction and has no durable inbox or subscriber ledger. +Its current exception logs/spans need P02b-3's bounded diagnostic reconciliation +before Active production consumers; P02b-2 minimizes its own dispatcher path. + +P02b-3 changes publish success to durable acceptance of every required delivery. +Duplicate acceptance is idempotent; partial or unknown acceptance can be retried. +The independent scheduler then invokes one internal delivery executor for each +subscription, with the shared inbox and app-role owner described above. Subscriber +business failure spends neither sibling nor producer attempts after acceptance. +The executor never rebroadcasts and receives no publication-rule exception. + +`PartitionSerializer` refuses reentrant work for its current key. Queuing behind +itself would deadlock; running inline would let inherited AsyncLocal state permit +concurrent work. Handlers produce outbox rows rather than publish directly. +Process-local serialization does not strengthen durable ordering. Extraction to +a second process remains the named Phase 11 trigger, with the same accepted +identity, envelope, acceptance and independent execution contracts. ## Rules - Integration events are **versioned**. Breaking changes ship a new versioned record (`UserCreatedIntegrationEventV2`); old version remains supported during a migration window. -- Consumers are **idempotent** via `IInboxGuard` (per-module inbox table). +- Consumers are **idempotent** through the shared transport-owned inbox (P02b-3). - Mandatory on every integration event: `EventId`, `TenantId`, `OccurredAt`, `PartitionKey`. The first three are `required`; the fourth is abstract. - Declared by the event and read by the envelope: `Topic`, the channel @@ -740,8 +568,9 @@ the choice. ## Ordering P02b-1 accepts partition-source derivation under ADR-0057. It accepts no durable -publish-order guarantee. G3's ordering half remains P02b-2; the inherited per-key -transport target must be reconciled there against concurrent commits and retries. +publish-order guarantee. ADR-0046 selects eligible rows in total order +`(available_after, occurred_at, id)` for one processor without failure. This is no +enqueue/commit/retry/cross-process promise; strict head-of-line blocking is rejected. `InProcessEventBus` serializes invocation per key within its process. The rule, therefore, is that **every outbox row carries a non-null `partition_key`**: @@ -769,8 +598,8 @@ domain; deriving it does not itself establish durable publish order. Consequences worth stating plainly: -- Same-aggregate events share a declared key. Durable order across commits/retries - remains G3's P02b-2 decision. +- Same-aggregate events share a declared key. Retry overtaking and reverse commits + can change arrival order; equal occurrence times use ID only as a selection tie. - Two events about **different** aggregates may arrive in any relative order, even inside one topic. A consumer that needs a cross-aggregate order needs a different design — usually a read model, not an ordering assumption. @@ -784,48 +613,32 @@ Consequences worth stating plainly: ## Dead-letter: two sides, two failure domains -The distinction the earlier draft of this document missed: a producer-side dispatch -success means "the transport accepted the bytes". It says nothing about whether any -consumer processed them. The two sides fail independently and need separate handling. +Producer acceptance and subscriber business completion are independent outcomes. +A timeout can leave acceptance unknown, so producer terminal state does not imply +that no subscription accepted the fact. ### Producer side — dispatch never succeeded -The outbox row reaches `MaxAttempts` without a successful `PublishAsync`. - -- The row stays in the table with `processed_at IS NULL`, `attempts >= MaxAttempts`, and - the final `last_error`. -- `available_after` is pushed beyond the retry horizon so the poller stops picking it up. -- It surfaces on the `OutboxStatusEndpoints` admin API and on the - `learnstack_outbox_deadletter_total{event_type}` counter; the alert is on the counter's - rate, not on the table's size. -- Recovery is a manual replay: an operator clears the lease, resets `attempts`, and - resets `available_after`. +Invalid/unknown/retired records or exhausted claims request `terminal_reason`. +Normal publication excludes that row permanently. The terminalizer writes +`dead_lettered_at` and Tenancy-owned `platform.outbox.dead_letter` MUST/SecurityEvent +on one fenced transaction, retaining the envelope for investigation. Audit failure +retains terminal work for retry without publication; completed terminal rows leave +pending gauges. A far-future availability time is never a terminal marker. +Phase 03 must decide producer reset/fencing before any operator replay endpoint. ### Subscriber side — dispatch succeeded, handling did not -A consumer's handler throws on every attempt: a malformed payload, a schema version it -cannot read, a permanently failing downstream dependency. The producer's row is long -since `processed_at`-stamped, and no amount of producer-side retry will help. - -- The transport retries the handler with backoff. On exhausting its retry policy it - routes the message to a **dead-letter topic**, `learnstack.dlq.{module}` — one per - subscribing module, three segments, satisfying - `Dapr_PubSub_TopicNames_FollowConvention`. Under Dapr this is the subscription's - `deadLetterTopic`; under `InProcessEventBus` the equivalent is a row in - `dead_letter_messages` carrying the same envelope. -- **The inbox row is not marked processed.** A dead-lettered message has not been - handled, and recording it as handled would make a later replay a silent no-op. -- Every module subscribes to its own dead-letter topic with a handler that does nothing - but persist the envelope, emit `learnstack_inbox_deadletter_total{module, event_type}`, - and raise an operational alert. A dead-letter topic nobody consumes is a queue that - grows until the broker's retention silently deletes the evidence. -- Recovery is an operator-initiated replay from the dead-letter store back onto the - original topic. Because the inbox never marked the event processed, the replayed - message is handled normally; because `IInboxGuard` is still in the path, a replay that - races a late success is still idempotent. -- Poison-message containment is per subscription. One module's dead-lettered event does - not block another module's consumption of the same topic — each subscription has its - own offset and its own dead-letter path. +**P02b-3 target.** Each stable subscription has its own durable ledger budget, +retained envelope and terminal/replay state. Terminal failure leaves no completed +inbox row and writes Tenancy-owned `platform.delivery.dead_letter` atomically with +terminal state. No `learnstack.dlq.*` topic or duplicate dead-letter table is needed. + +Targeted internal replay resets only that subscription with fresh ownership fencing +and invokes its executor, never `PublishAsync`. Completed siblings remain inbox +deduplicated; a completed inbox is never reset. Phase 03 owns operator permission, +HTTP routes and the separately decided producer replay contract. P02b-8 owns erasure +inventory, Phase 03 privileged redaction and Phase 11 retention purge. ## Service extraction readiness @@ -850,13 +663,13 @@ separate service later: - `Integration_Events_Inherit_From_IntegrationEventBase` — every type implementing `IIntegrationEvent` extends `IntegrationEventBase` (which carries `EventId`, `OccurredAt`, `TenantId` and declares `Topic` / `PartitionKey`). -- `Integration_Event_Handlers_Use_InboxGuard` — every `IIntegrationEventHandler` - implementation invokes `IInboxGuard.IsAlreadyProcessedAsync` before processing. +- `Integration_Event_Handlers_Use_InboxGuard` — Registered for P02b-3's shared + transport-owned claim before construction and atomic fenced completion. - `Dapr_PubSub_TopicNames_FollowConvention` — Phase 11 checks that Dapr component bindings agree with the already-validated topics declared by events. It is not a current `[Topic]`-attribute scan; no such attribute exists. - `OutboxProcessor_NeverBlocks_OnSingleMessageFailure` — integration test asserts one - poisoned message doesn't prevent others in the batch from processing. + failed attempt or bounded overrun permits unrelated eligible work to progress. - `Integration_Event_Handler_Restores_Tenant_Context` — catalogued in [Architecture Tests Catalogue](../standards/21-architecture-tests-catalogue.md); runs against whichever transport is registered, so `InProcessEventBus` is held to the same @@ -870,24 +683,25 @@ separate service later: ## Observability -- Outbox lag: `learnstack_outbox_pending_count{tenant_id}` (gauge). -- Dispatch duration: `learnstack_outbox_dispatch_duration_seconds{event_type}` (histogram). -- Dispatch failures: `learnstack_outbox_dispatch_failed_total{event_type}` (counter). -- Producer-side dead letter: `learnstack_outbox_deadletter_total{event_type}` (counter). -- Lost leases: `learnstack_outbox_lease_lost_total` (counter) — non-zero means the lease - is shorter than the transport's tail latency. -- Inbox dedup: `learnstack_inbox_dedup_total{module, event_type}` (counter). -- Subscriber-side dead letter: `learnstack_inbox_deadletter_total{module, event_type}` - (counter). - -The two dead-letter counters are **separate signals on separate dashboards**. A -producer-side spike means the transport is down; a subscriber-side spike means a handler -or a payload is broken. Collapsing them into one "events failed" number destroys the only -information that distinguishes the two incidents. Once the Dapr adapter lands, both sit -alongside Dapr's own `dapr_component_pubsub_*` metrics. +[Observability Standards § Required Metrics](../standards/10-observability.md#required-metrics) +owns names, labels and queue semantics. P02b-2 emits unlabelled pending, deferred, +oldest-eligible-age and heartbeat gauges from immutable snapshots, plus bounded +attempt duration/failure/terminal/lease-loss observations. Gauges perform no SQL; +refresh failure preserves stale data and makes health unhealthy. Per-attempt spans +and closed diagnostics contain no raw exception/type/payload/connection data. + +P02b-3 adds dedup/subscriber-terminal counters with bounded module/event-type labels. +Producer and subscriber terminal outcomes remain separate observations: acceptance +failure and business execution failure have different retry owners. Metrics are +process observations, not an exactly-once durable event ledger. `outbox` is +registered and not publicly mapped; `/healthz` remains liveness-only, with Phase 11 +owning readiness exposure and dashboards. ## References +- [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md) — accepted + bounded delivery supersessions, readiness, terminal audit and shared inbox contract. + - ADR-0006 (Amendment 1) — Events and Outbox; dispatch transport. - ADR-0010 (Amendment 1) — Cross-Module Communication; outbox dispatch target. - [ADR-0038](../decisions/0038-cross-cutting-port-and-event-contracts.md) — Dapr choice, diff --git a/docs/architecture/29-dapr-integration.md b/docs/architecture/29-dapr-integration.md index 31a45a39..0857b48c 100644 --- a/docs/architecture/29-dapr-integration.md +++ b/docs/architecture/29-dapr-integration.md @@ -2,7 +2,8 @@ **Derives from:** [ADR-0038](../decisions/0038-cross-cutting-port-and-event-contracts.md), [ADR-0006](../decisions/0006-events-and-outbox.md), -[ADR-0010](../decisions/0010-cross-module-communication.md). +[ADR-0010](../decisions/0010-cross-module-communication.md), +[ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md). > **Read this first.** This document describes Dapr in the present tense as the **target > design**. Per [ADR-0035](../decisions/0035-demand-gated-infrastructure.md) no Dapr @@ -210,12 +211,13 @@ developer starts the gated sidecar for inspection. It is a **transport, not a st construction-free subscription metadata, gives each subscription one consumer activity and one async DI scope, restores tenant and module context before resolving exactly one concrete `IIntegrationEventHandler`, and restores the publisher's context after the -full dispatch. It leaves `IInboxGuard` deduplication to the handler exactly as the -durable path does and preserves per-partition-key ordering. A default path that skipped -those is a path where the isolation code is never exercised — see -[15-event-and-outbox.md](15-event-and-outbox.md), which owns the implementation, and -[ADR-0035](../decisions/0035-demand-gated-infrastructure.md), which makes the four -obligations a condition of the gating. +full dispatch. It serializes invocation per partition key within its process; +it opens no business transaction and implements no durable inbox or subscriber ledger. +[ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md) assigns shared +`IInboxGuard` claim/completion to the transport before handler construction, with +implementation in P02b-3. Handlers do not call the guard. Phase 11's Dapr adapter +preserves that accepted contract. See [15-event-and-outbox.md](15-event-and-outbox.md) +for the shipped baseline and the P02b-3 durable acceptance/execution target. ## 4. Cache implementation (DaprCacheService) diff --git a/docs/architecture/31-audit-subsystem.md b/docs/architecture/31-audit-subsystem.md index 8250c25a..5deb611f 100644 --- a/docs/architecture/31-audit-subsystem.md +++ b/docs/architecture/31-audit-subsystem.md @@ -3,6 +3,7 @@ **Derives from:** [ADR-0033](../decisions/0033-audit-durability-model.md) (supersedes [ADR-0016](../decisions/0016-audit-log-subsystem.md)), [ADR-0044 (The Audit Write Path)](../decisions/0044-audit-write-path.md), +[ADR-0046 (Durable Event Delivery and Recovery)](../decisions/0046-durable-event-delivery-and-recovery.md), [ADR-0057 (Transactional Event Production and Completion)](../decisions/0057-transactional-event-production-and-completion.md), [ADR-0017 (Tenant + Organization)](../decisions/0017-tenant-organization-hierarchy.md), [18-audit-coverage.md](../standards/18-audit-coverage.md). @@ -77,6 +78,45 @@ Redaction, projection and external fan-out happen after the commit, reading the row; none of them updates it. ADR-0016's "audit never blocks business logic" is preserved for that second stage and withdrawn for the first. +### Confined event-terminal audit + +**ADR-0046 fifth seam implemented in P02b-2 Step 2; both review rounds complete.** +Existing four delivered writes +retain their contracts. The new narrow method is: + +```csharp +IAuditStore.WriteEventTerminalAsync( + AuditEntryDraft draft, + EventTerminalTransition transition, + DbConnection connection, + DbTransaction transaction, + CancellationToken cancellationToken); +``` + +A successful fenced terminal `UPDATE ... RETURNING` alone mints the immutable +restricted witness. It binds actual stored tenant/org/event, acquisition, +producer/subscriber kind, closed reason, attempts, optional subscription ID and +live connection/transaction. The store checks declared MUST/SecurityEvent +operation, draft identity and transaction before INSERT, opens/commits nothing +and remains the sole audit SQL writer. Mismatch rolls back both state and audit. +GUCs cannot bind a bypass INSERT; real tenant/org/event/transaction mismatch proofs +and a matching organization-scoped positive control are required. + +P02b-2 supplies `platform.outbox.dead_letter`; P02b-3 supplies reserved +`platform.delivery.dead_letter`, both off-path under Tenancy. Register each only +when its writer ships. Outcome success describes terminal state, not publication; +invalid causal fields are omitted in favor of a fresh validated attempt trace. +Closed terminal values use bounded metadata; `Reason` stays for platform entry. +Audit failure retains terminal work for retry without delivery; uncertain COMMIT +is reconciled before retry and cannot duplicate a completed terminal audit. + +The existing outbox role gains audit INSERT-only in the Audit chain, with exact +grants owned by [Database Standards](../standards/05-database.md#grant-matrix). +Routine confined poll/claim/mark uses the accepted bounded audit exemption; +platform-admin pre-entry audit and other bypass operations are unchanged. +P02b-3 adds inbox/delivery machinery to capture exclusions with reasons, because +explicit intents audit business effects; these exclusions remain pending P3. + ## 2. Pipeline overview ```mermaid @@ -631,8 +671,9 @@ then constructs and invokes the handler after announcement. Direct one-root work declares its own intents; nested commands retain their own, without an extra business row. Step 1 supplies the common entry and producer ancestry groundwork; Steps 2–3 deliver immediate outbox enqueue, successful-save collection/promotion -and owner-only domain drain. Dispatch remains P02b-2; production consumer -delivery/inbox wiring remains P02b-3 after G1. +and owner-only domain drain. P02b-2 Step 3 implements hosted producer dispatch, +with both independent review rounds complete; production consumer delivery/inbox wiring +remains P02b-3 under Accepted ADR-0046. ### The two catalogue faces, and the three columns nothing fills yet @@ -843,7 +884,8 @@ uses `CancellationToken.None` and cannot replace the original failure. The share execution boundary creates a W3C Activity even without a telemetry listener and restores ancestry/Activity on disposal. P02b-1 Steps 2–3 deliver immediate outbox enqueue, successful-save collection/promotion and owner-only domain drain before -MUST audit/commit. Dispatch and consumer delivery/inbox remain P02b-2/P02b-3. +MUST audit/commit. P02b-2 Step 3 implements hosted producer dispatch, with +both independent review rounds complete; consumer delivery/inbox remains P02b-3. `IUnitOfWork` remains the seam through which every module context and `IAuditStore` reach the one ambient connection/transaction. Packet 6 shipped that seam and the @@ -1467,128 +1509,21 @@ retention purge, § 9). It is not an exception carved out of the append-only rul convention — it is the shape the `audit_log_append_only_guard` trigger in § 7 was written to permit, and nothing else. -```csharp -// LearnStack.Modules.Audit.Infrastructure.IntegrationEvents -public sealed class UserGdprDeletedIntegrationEventHandler( - AuditDbContext db, - IPlatformAdminScope platformScope, - IAuditStore auditStore, - IInboxGuard inboxGuard, - IClock clock, // the draft's Timestamp; the column's DEFAULT is a backstop - IGuidFactory guidFactory, // AuditEntryId has no New() (ADR-0023 Amendment 9) - IEnumerable userReferenceLocators) - : IIntegrationEventHandler -{ - public async Task HandleAsync(UserGdprDeletedIntegrationEvent @event, CancellationToken ct) - { - // 1. Idempotent: inbox guard. - if (await inboxGuard.IsAlreadyProcessedAsync(@event.EventId, ct)) return; - - // 2. learnstack_app holds no UPDATE privilege on audit_log, so the redaction runs - // as learnstack_platform — on the HANDLE's connection. The injected - // AuditDbContext is bound to IUnitOfWork.Connection (ADR-0040) and stays on - // the request's learnstack_app connection whatever scope surrounds it, so - // issuing the UPDATE through `db` would raise 42501 rather than redact - // anything. Entering the scope writes its own SecurityEvent row, committed - // before the handle is returned (ADR-0044 § 10). - await using var handle = await platformScope.EnterAsync( - reason: $"gdpr-redaction:{@event.UserId}", ct); - - // 3. Actor PII only. The payload columns are NOT touched here. A blanket - // jsonb_set('{redacted}', 'true') would (a) not redact anything — it adds a - // flag and leaves the PII in place — and (b) raise - // 'cannot set path in scalar' on any snapshot that is a JSON scalar or - // array. Payload redaction belongs to the per-module locator below, which - // knows which JSON paths in its own snapshots reference a user. - await using (var redact = handle.Connection.CreateCommand()) - { - redact.Transaction = handle.Transaction; - redact.CommandText = @" - UPDATE audit_log - SET actor_email = '***REDACTED***', -- SensitiveTokenCatalog.RedactedValue - ip_address = NULL, - user_agent = '***REDACTED***' - WHERE actor_user_id = @actor - AND tenant_id = @tenant"; - redact.Parameters.Add(new NpgsqlParameter("actor", @event.UserId)); - redact.Parameters.Add(new NpgsqlParameter("tenant", @event.TenantId)); - await redact.ExecuteNonQueryAsync(ct); - } - - // 4. Payload references, per module. Each locator issues column-restricted - // UPDATEs against before_state / after_state / changes only — on the same - // handle, because they need the same privilege for the same reason. - foreach (var locator in userReferenceLocators) - await locator.RedactReferencesAsync(handle, @event.UserId, @event.TenantId, ct); - - // 5. The result row — ON THE HANDLE'S TRANSACTION, before its commit. The redaction - // is a MUST-class security event, and the row recording it commits with the - // redaction or not at all (Audit Coverage: a MUST row is never written after the - // transaction it describes has committed). An earlier version of this listing - // committed the handle first and wrote the row standalone afterwards: a process - // that stopped between the two left rows permanently redacted and no record of - // it. The scope-entry row step 2 committed proves the scope was entered, not what - // was done inside it — two events, two rows. - // - // THE PORT DOES NOT SANCTION THIS CALL YET. WritePlatformScopeAsync writes on a - // caller's platform transaction, and its contract admits one caller: the scope's - // own entry row. Phase 03 extends it to this second, bounded caller — or gives the - // result row a method of its own — by amending ADR-0044 § 10 before the handler - // lands. The ORDER shown here is what that decision may not change. - // - // Constructed explicitly: AuditEntryDraft has no factory, AuditEntryId has no - // New() (ADR-0023 Amendment 9), and Timestamp comes from IClock. Every member is - // `required`, so the compiler refuses a construction that omits one and every - // value arrives under its own name — ELEVEN of the twenty-two fields are string?, - // six of them consecutive, and a transposition lands in the one table whose rows - // nothing can correct. The metadata carries the event id: see step 6. - await auditStore.WritePlatformScopeAsync( - new AuditEntryDraft - { - Id = AuditEntryId.From(guidFactory.NewUuidV7()), - TenantId = TenantId.From(@event.TenantId), - OrganizationId = null, - ActorUserId = null, - ActorEmail = null, - ModuleName = "audit", - Operation = "audit.redaction.apply", - OperationType = OperationType.SecurityEvent, - OperationClass = OperationClass.Must, - EntityType = null, - EntityId = null, - Outcome = AuditOutcome.Success, - ErrorKey = null, - Reason = $"gdpr-redaction:{@event.UserId}", - BeforeState = null, - AfterState = null, - Changes = null, - CorrelationId = null, - IpAddress = null, - UserAgent = null, - Timestamp = clock.UtcNow, - Metadata = JsonSerializer.Serialize( - new { subjectUserId = @event.UserId, eventId = @event.EventId }), - }, - handle.Connection, - handle.Transaction, - ct); - - // The redaction, the locators' updates and the row recording them: one commit. - await handle.CommitAsync(ct); - - // 6. The inbox mark. It cannot join that commit — it is learnstack_app work on the - // ambient connection, a different connection from the handle's — so a process - // that stops between the two leaves the event unmarked, and it is delivered - // again. The redelivery must therefore be harmless, and Phase 03 makes it so with - // the port decision above: the redaction is idempotent by construction (the same - // columns set to the same sentinel), and before redacting, the handler looks on - // the handle for a result row whose metadata carries this event id — finding one, - // it marks the inbox and stops, so one erasure never records two results. - inboxGuard.MarkAsProcessed(@event.EventId, @event.GetType().Name); - await db.SaveChangesAsync(ct); - } -} -``` +**Phase 03 target; no privileged consumer is shipped.** ADR-0046 requires a +separate privileged execution contract before this consumer can become Active. +P02b-3's ordinary app-role executor refuses privileged subscriptions. Its shared +inbox cannot atomically complete a platform-role transaction on another connection. +The former handler-owned inbox sketch is superseded as the current target. + +Phase 03 uses audited `EnterPlatformAdminScope(reason)` and specifies bounded +result-audit access before the handler ships. Scope entry commits its own audit +before work; the redaction and its MUST result audit then share the platform +working transaction. Per-module `IUserReferenceLocator` implementations redact +only declared payload references through that same handle. Idempotent progress +and durable event reconciliation prevent duplicate result audits after a crash; +no two-connection atomicity or external exactly-once effect is promised. The +ordinary event-terminal fifth seam authorizes no privileged redaction caller. +Phase 11 owns retention purge, not redaction. `actor_user_id`, `module`, `operation`, `operation_type`, `operation_class`, `outcome`, `correlation_id` and `timestamp` are **never** redacted; the trigger rejects an `UPDATE` @@ -1689,14 +1624,15 @@ update it constrains. `(planned)` row whose command has since shipped **fails**, which is what stops the marker from becoming an escape. - *Off the request path.* Operations that are not MediatR requests at all carry - `(off-path)` and are outside both directions. Seven rows carry it — Tenancy's - `platform.admin_scope.enter`, `tenancy.killswitch.toggle`, + `(off-path)` and are outside both directions. Nine rows carry it — Tenancy's + `platform.admin_scope.enter`, `platform.outbox.dead_letter`, + `platform.delivery.dead_letter`, `tenancy.killswitch.toggle`, `tenancy.entitlement.refresh` and [ADR-0036](../decisions/0036-tenant-resolution-trusted-inputs.md)'s `tenancy.tenant_assertion.reject` and `tenancy.tenant_assertion.anonymous_burst`, and Audit's `audit.redaction.apply` and `audit.purge.apply`. They register by slug: the - three whose writer ships through `DeclareOffPath`, the other four — also `(planned)` — - when their writer lands. + four whose writer ships through `DeclareOffPath`, the other five — also + `(planned)` — when their writer lands. - *An `Off` registration carries no slug*, so it raises no matrix row and is outside both directions too (§ 5). 2. `AuditEntry_Inherits_Entity_Not_AuditableEntity` — append-only by construction: an @@ -1782,7 +1718,9 @@ for MUST-class would lock in exactly the defect ADR-0033 removes. | Phase | Deliverable | |-------|-------------| | [02a Packet 9](../roadmap/phase-02a-kernel-tenancy.md) | `AuditChangeTrackerInterceptor` with the broad capture predicate, the `[PiiSensitive]` redaction gate and the size cap; `IAuditStateCapture` + impl, holding an **ordered list** of intents; `AuditLogBehavior` lit up per ADR-0033 as amended; `IAuditStore` + `PostgresAuditStore` with **four** write methods, the fourth serving `EnterPlatformAdminScope`; the classification catalogue as a **pair** — the module-owned `IAuditCatalogSource` / `IAuditCatalogBuilder` and the `IAuditCatalog` the composition root merges them into once at startup — with the six value types beside the five ports in `LearnStack.SharedKernel.Audit`; `TenantId.PlatformSentinel` and the **four** guards that enforce its invariant — `SetProvisioningTenantContextAsync`, `TenantOwnership.EnsureRealTenant`, `EventTenantContext.FromEnvelope` and, deliberately redundant at the one site every announcement passes, `SetTenantContextAsync`; `AuditEntry` aggregate and `AuditConfig` with the fail-closed MUST floor; `audit_log` as a **single plain table** with the composite primary key, org-scoped, in a fourth migration chain, with its append-only grants and its two triggers; and the named `audit` health check §§ 1, 5 and 7 read, registered through `AddHealthChecks()` at the `LearnStack.Api` composition root. | -| P02b-1 Steps 1–3 | Common audit/transaction coordinators, capture-owned audit frames, producer ancestry/receipts and no-listener trace groundwork; Steps 2–3 deliver immediate outbox enqueue, successful-save collection/promotion and owner-only domain drain. Dispatch remains P02b-2; consumer delivery/inbox remains P02b-3. | +| P02b-1 Steps 1–3 | Common audit/transaction coordinators, capture-owned audit frames, producer ancestry/receipts and no-listener trace groundwork; Steps 2–3 deliver immediate outbox enqueue, successful-save collection/promotion and owner-only domain drain. Hosted dispatch is implemented in P02b-2 Step 3; consumer delivery/inbox remains P02b-3. | +| P02b-2 Steps 1–4 | Schema/grants and confined producer transitions with MUST audit through the row-bound fifth seam; real-role atomicity/identity proofs. Steps 1–2 completed both review rounds. Step 3 implements hosted dispatch and observation, with both independent review rounds complete; Step 4 enforcement and local closeout are complete; all four steps completed both independent review rounds. The [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) owns CI/merge status. | +| P02b-3 (Accepted contract; pending) | Shared transport-owned inbox and independent delivery ledger; per-subscription audit frames and Tenancy-owned terminal audit. Ordinary executor refuses privileged subscriptions. | | [03](../roadmap/phase-03-identity-admin.md) | Admin API endpoints over the audit stream; `UserGdprDeletedIntegrationEventHandler` + per-module `IUserReferenceLocator`. | | [06](../roadmap/phase-06-renderer-admin-studio.md) | Admin Studio audit UI: timeline view, filters, diff viewer, CSV / JSON export. | | [09](../roadmap/phase-09-billing-integrations-analytics.md) | Hub-side `hub_audit_log` + cross-stream correlation query. | diff --git a/docs/decisions/0003-tenant-isolation-defense-in-depth.md b/docs/decisions/0003-tenant-isolation-defense-in-depth.md index 96821572..8a325b40 100644 --- a/docs/decisions/0003-tenant-isolation-defense-in-depth.md +++ b/docs/decisions/0003-tenant-isolation-defense-in-depth.md @@ -473,3 +473,19 @@ visible to the application's separate connection and is not a valid test setup. - [PostgreSQL 18: Constraints](https://www.postgresql.org/docs/18/ddl-constraints.html) - [PostgreSQL 18: Generated columns](https://www.postgresql.org/docs/18/ddl-generated-columns.html) - [Education module spec](../modules/education/README.md) + +## Amendment 7 — Event-delivery grant-budget extension (2026-10-12) + +**Status: Accepted.** [ADR-0046](0046-durable-event-delivery-and-recovery.md#atomic-terminal-audit-and-privilege-boundary) +retains Amendment 3's four roles, separate credentials and RLS postures. Its +bounded grant-budget extension is canonical in +[Database Standards](../standards/05-database.md#grant-matrix): P02b-2 adds +named outbox status-column UPDATEs and INSERT-only on `audit_log` for confined, +atomic terminal auditing. P02b-3 adds only its separately enumerated delivery +scheduling grants, without envelope/payload SELECT or business-table access. + +This is an additive navigation disclosure, not an ADR-0041 correction or a claim +that this ADR contained an outbox-only grant clause. Original acceptance metadata, +body and prior amendments remain intact. Acceptance authorizes implementation; +[P02b-2's delivery record](../roadmap/phase-02b-events-auth.md#p02b-2-step-1--decision-schema-and-confined-composition-2026-10-12) +owns evidence. The platform-admin gate and pre-entry audit remain unchanged. diff --git a/docs/decisions/0006-events-and-outbox.md b/docs/decisions/0006-events-and-outbox.md index eb68b94d..9ced01a0 100644 --- a/docs/decisions/0006-events-and-outbox.md +++ b/docs/decisions/0006-events-and-outbox.md @@ -103,3 +103,23 @@ nothing dispatches from it until [Phase 02b](../roadmap/phase-02b-events-auth.md `learnstack_outbox_admin`'s column-scoped `UPDATE` grant in the same migration — a column added without extending it fails at runtime with `permission denied for table`. + +## Amendment 3 — Bounded delivery supersession by ADR-0046 (2026-10-12) + +**Status: Accepted.** [ADR-0046](0046-durable-event-delivery-and-recovery.md#bounded-supersession-and-additive-seams) +supersedes Amendment 1's consumer-side handler-called guard and own-module inbox. +P02b-3 implements a shared transport-owned inbox keyed by tenant, stable +subscription ID and event ID; its separate delivery ledger owns independent +retries, terminal state and targeted replay. Publish success then means durable +acceptance of every required delivery, not completed business effects. + +P02b-2 implements fresh-token leases, claim-counted attempts, explicit terminal +requests and atomic terminal audit under ADR-0046. Required Pending subscriptions +preserve rows before acquisition and validation; only declared intentional +absence acknowledges without publication. The canonical DDL/grants remain in +[Database Standards](../standards/05-database.md#outbox). + +These are bounded new decisions, not ADR-0041 corrections. Original bodies, +acceptance blocks and prior amendments remain intact. Envelope contracts and +at-least-once recovery survive; demand-gated Dapr/Kafka remain Phase 11. +Acceptance claims no dispatcher or inbox delivery. diff --git a/docs/decisions/0010-cross-module-communication.md b/docs/decisions/0010-cross-module-communication.md index 48ab0d1a..164091fe 100644 --- a/docs/decisions/0010-cross-module-communication.md +++ b/docs/decisions/0010-cross-module-communication.md @@ -154,3 +154,18 @@ The optional fourth segment is reserved for **Hub-side event-name suffixes only* LearnStack-core topics stay 3-segment by convention (`learnstack.{module}.{aggregate}`); the test does not reject 3-segment topics in either direction. This amendment clarifies the regex; no other rule changes. + +## Amendment 3 — Bounded inbox supersession by ADR-0046 (2026-10-12) + +**Status: Accepted.** [ADR-0046](0046-durable-event-delivery-and-recovery.md#subscriber-acceptance-and-inbox-ownership) +supersedes Amendment 1's own-module inbox, handler-called guard and corresponding +`Integration_Event_Handlers_Use_InboxGuard` assertion. The +[catalogue rule](../standards/21-architecture-tests-catalogue.md#integration_event_handlers_use_inboxguard) +now requires transport-owned claim/completion before handler construction and +atomic business completion. Two same-module subscriptions retain distinct stable +identities. P02b-3 implements and proves this contract; the rule stays Registered +until those production subjects exist. + +The four communication mechanisms remain unchanged. This is a bounded +supersession, not an ADR-0041 correction. Original acceptance metadata, body and +earlier amendments remain intact; acceptance claims no inbox delivery. diff --git a/docs/decisions/0033-audit-durability-model.md b/docs/decisions/0033-audit-durability-model.md index cca9c53e..40aaaf12 100644 --- a/docs/decisions/0033-audit-durability-model.md +++ b/docs/decisions/0033-audit-durability-model.md @@ -762,6 +762,23 @@ audit. P02b-1 proves a controlled non-MediatR owner; production delivery/inbox atomicity remains P02b-3 after G1. This is a dated clarification, not an ADR-0041 correction. All earlier body text and acceptance metadata remain intact. +## Amendment 9 — Same-transaction event terminal audit (2026-10-12) + +**Status: Accepted.** [ADR-0046](0046-durable-event-delivery-and-recovery.md#atomic-terminal-audit-and-privilege-boundary) +applies this ADR's existing MUST same-transaction guarantee to a new confined +caller. A fenced event-terminal UPDATE and its complete MUST security-event +INSERT commit on one caller-owned transaction, or both roll back. The narrow +fifth `IAuditStore` write receives a restricted witness from the successful +UPDATE result, validates its row identity and live transaction, and opens or +commits no transaction itself. An uncertain COMMIT is reconciled from durable +state before retry; it does not justify a second terminal audit. + +P02b-2 supplies the producer caller; P02b-3 supplies the subscriber caller. +Existing four write paths, classification, refusal posture and append-only rows +retain their contracts. This is a clarification of the guarantee's new caller, +not an ADR-0041 correction. Original acceptance metadata, body and earlier +amendments remain intact. Acceptance authorizes implementation, not delivery. + ## References - [ADR-0016 Audit Log Subsystem](0016-audit-log-subsystem.md) (superseded by this ADR) diff --git a/docs/decisions/0038-cross-cutting-port-and-event-contracts.md b/docs/decisions/0038-cross-cutting-port-and-event-contracts.md index 856a8537..3122c8d7 100644 --- a/docs/decisions/0038-cross-cutting-port-and-event-contracts.md +++ b/docs/decisions/0038-cross-cutting-port-and-event-contracts.md @@ -227,6 +227,26 @@ to this ADR's original envelope, transport, serializer or effective/causal actor contracts. Identifiers minimize copied data; they are not a claim of anonymity. G18's failure descriptors and erasure inventory remain P02b-2 and P02b-8. +### Amendment 3 — Bounded publish-outcome supersession by ADR-0046 (2026-10-12) + +**Status: Accepted.** [ADR-0046](0046-durable-event-delivery-and-recovery.md#subscriber-acceptance-and-inbox-ownership) +supersedes only § Event contract's immediate constructor/invocation/disposal +failure outcome and the requirement that future adapters preserve that outcome. +When P02b-3 installs durable acceptance, `PublishAsync` succeeds after every +required subscription is durably accepted. Unknown/failed acceptance remains a +producer retry; independent subscription execution owns later handler failures. +Retries invoke the internal delivery executor, never another broadcast. + +The non-generic envelope signature, frozen serializer, effective SystemActor, +causal identity, organization restoration, per-key serialization and outbox-only +publication survive. Pending readiness holds in P02b-2 preserve the real +TenantCreated facts until both P02b-3 consumers are Active. Diagnostic +minimization of current transport/consumer exception telemetry is also P02b-3. + +This is a bounded new decision, not an ADR-0041 correction. Original acceptance +metadata, body and earlier amendments remain intact. Acceptance claims no durable +acceptance or consumer delivery. + ## References - [ADR-0006 — Events and Outbox](0006-events-and-outbox.md) diff --git a/docs/decisions/0044-audit-write-path.md b/docs/decisions/0044-audit-write-path.md index 05596eef..bda44db1 100644 --- a/docs/decisions/0044-audit-write-path.md +++ b/docs/decisions/0044-audit-write-path.md @@ -1105,6 +1105,54 @@ audit. P02b-1 proves a controlled non-MediatR owner; production delivery/inbox atomicity remains P02b-3 after G1. This is a dated clarification, not an ADR-0041 correction. All earlier body text and acceptance metadata remain intact. +## Amendment 9 — Bounded terminal-audit grant replacement (2026-10-12) + +**Status: Accepted.** [ADR-0046](0046-durable-event-delivery-and-recovery.md#bounded-supersession-and-additive-seams) +supersedes only § 9's `audit_log` grant cell denying every permission to +`learnstack_outbox_admin`. P02b-2 grants INSERT-only for confined atomic terminal +audit, through a new Audit-chain migration after the table exists. No audit +SELECT/UPDATE/DELETE, business-table grant or privileged membership is added. +[Database Standards](../standards/05-database.md#grant-matrix) owns the exact grants. + +This is a bounded new decision, not an ADR-0041 correction. Original acceptance +metadata, body and prior amendments remain intact. The four-role model, RLS and +append-only controls survive. Acceptance claims no runtime grant delivery. + +## Amendment 10 — Additive event-terminal write seam (2026-10-12) + +**Status: Accepted.** [ADR-0046](0046-durable-event-delivery-and-recovery.md#atomic-terminal-audit-and-privilege-boundary) +adds the fifth write method: + +```csharp +IAuditStore.WriteEventTerminalAsync( + AuditEntryDraft draft, + EventTerminalTransition transition, + DbConnection connection, + DbTransaction transaction, + CancellationToken cancellationToken); +``` + +Only a successful fenced terminal `UPDATE ... RETURNING` mints its immutable, +restricted transition witness. The witness binds the actual stored tenant/org, +event, acquisition, producer/subscriber kind, reason, attempts, optional +subscription ID and live connection/transaction. The seam checks operation, +class/type and every corresponding draft identity before INSERT; mismatches +require rollback of both writes. It opens/commits no transaction, and +`PostgresAuditStore` remains the sole audit SQL writer. Existing four writes +retain their contracts. + +P02b-2 admits only the producer terminal writer and Tenancy-owned off-path +`platform.outbox.dead_letter` as MUST/SecurityEvent. P02b-3 reserves the subscriber +writer and `platform.delivery.dead_letter` under that same owner/class/type; +register each operation only when its writer ships. P02b-3 adds inbox/delivery +machinery to § 7's capture exclusions with reasons, because explicit intents +audit business effects and retry state is machinery. Those exclusions remain +unimplemented until P02b-3. + +This additive seam is separate from Amendment 9's grant replacement and does not +change ADR-0033's durability contract. Original acceptance metadata, body and +prior amendments remain intact. Acceptance authorizes implementation, not delivery. + ## References - [ADR-0033 Audit Durability Model](0033-audit-durability-model.md) — the durability diff --git a/docs/decisions/0046-durable-event-delivery-and-recovery.md b/docs/decisions/0046-durable-event-delivery-and-recovery.md new file mode 100644 index 00000000..5a9ed9d2 --- /dev/null +++ b/docs/decisions/0046-durable-event-delivery-and-recovery.md @@ -0,0 +1,513 @@ +# ADR-0046: Durable Event Delivery and Recovery + +## Status + +Accepted — 2026-10-12. Implementation is authorized; delivery remains pending. + +**Date:** 2026-10-12 +**Deciders:** @cemil + +Delivery updates: [Amendment 1](#amendment-1--producer-runtime-delivery-2026-10-12) +and [Amendment 2](#amendment-2--producer-enforcement-and-local-closeout-2026-10-12). +The Status above records acceptance-time state; Amendment 2 records local producer +closeout with PR-head CI and merge pending. +Current CI/review/merge navigation (2026-10-12): the +[PR handoff](../roadmap/phase-02b-events-auth.md#p02b-2-pr-handoff-2026-10-12) +links the live PR; the delivery notes retain their as-of status. + +The [P02b-2 decision package](../roadmap/phase-02b-events-auth.md#p02b-2-decision-package-2026-10-12) +owns implementation sequencing and numerical runtime bounds. This reserved record +decides producer dispatch and the coupled P02b-3 subscriber state machine before +either is implemented. Acceptance does not claim delivery. + +## Decision Drivers + +- P02b-1 already commits real `TenantCreatedIntegrationEventV1` rows. The current + bus acknowledges zero subscribers; its two default consumers arrive in P02b-3. + Starting dispatch without an explicit admission policy would discard their work. +- Producer retries must not repeatedly execute a sibling subscription that already + committed. Two consumers in the same module require distinct durable identities. +- The existing outbox role cannot insert audit rows. A terminal transition and its + MUST audit must commit together without granting business-table access. +- Leases, unknown outcomes and cancellation require at-least-once recovery, not an + exactly-once transport claim. A timeout cannot terminate arbitrary managed code. +- Four database roles, the ambient application transaction, tenant/organization + isolation and demand-gated cross-process transport remain the baseline. + +## Considered Options + +1. **Grant-bounded dispatcher and durable per-subscription delivery** (recommended). + Preserve required but unready work; separate producer and subscriber outcomes. +2. **Handler-owned, per-module inbox and repeated broadcast** (rejected). It + cannot distinguish two same-module subscriptions or isolate their retry budgets. +3. **Disable all production dispatch until P02b-3** (rejected). It avoids premature + acknowledgement by leaving P02b-2's production runtime unexercised. +4. **Audit terminalization on a separate connection** (rejected). A state change + can commit without its MUST row, or a success row without the state change. +5. **Strict head-of-line ordering per partition** (rejected). Poison and delayed + work would block later eligible facts; no current consumer requires that cost. +6. **Dapr/Kafka now** (rejected). No second consumer process triggers ADR-0035. + A broker would not decide inbox ownership, independent retries or audit policy. + +## Decision + +LearnStack uses a leased, grant-bounded outbox dispatcher with explicit +subscription readiness, fenced retries and atomic audited terminalization. +Durable subscriber acceptance and a transport-owned shared inbox separate each +subscription's business outcome from producer dispatch. The following contract +is normative; packet delivery records own implementation evidence. + +### Producer dispatch and recovery + +- `id = EventId` and the assembly-qualified stored type identity survive ADR-0057. + Resolve only explicitly registered concrete contracts from known contract + assemblies, with explicit compatible aliases and retired identities. Never + load an assembly or resolve an arbitrary CLR type from stored text. Duplicate + identities, alias collisions and invalid registrations refuse composition. +- For identities not held by the Pending policy below, deserialize using frozen + `PayloadJsonOptions`. Check payload identity, tenant, + topic, partition and occurrence against the row; timestamp comparison accounts + for PostgreSQL microsecond precision. Validate the complete envelope and the + canonical reserialized payload's existing UTF-8 cap; `jsonb`'s text formatting + is not the producer byte-budget representation. Unknown, retired, malformed or + mismatched records request terminalization immediately with a closed reason. +- Construction-free manifests distinguish **Pending**, **Active** and explicitly + **IntentionallyUnsubscribed** contracts. Missing policy is not intentional + absence. Any required Pending subscription excludes that contract and its + aliases before acquisition; polling changes none of its payload, attempts, + lease, processed or terminal state, even if its stored payload is malformed. + This hold precedes payload validation/terminalization; activation later admits + those checks. Active declarations without their required + handler registration refuse composition. Intentional absence may acknowledge + without publication; it is a separately tested policy, not a zero-handler guess. +- `TenantCreatedIntegrationEventV1` declares two required Pending subscriptions: + `customization.initialize-content-type-default.v1` initializes the `card@1` + content type (`BuiltInCustomizations.Card`); + `customization.initialize-taxonomy-default.v1` initializes the `plain@1` level + taxonomy (`BuiltInCustomizations.Plain`). These are ADR-0057's existing two + effects, not two content types. P02b-3 activates them together with durable + acceptance. No placeholder consumer or global worker-disable flag substitutes + for this policy. +- Claim one eligible row just before invocation. A short transaction assigns a + fresh acquisition token to `locked_by`, stamps `locked_until` and increments + `attempts` once. Stable worker identity is diagnostic, not an ownership token. + Eligibility and expiry use database time. Lease fields are both null or both + present; processed and dead-lettered states are mutually exclusive. +- Normal publication claims require `processed_at IS NULL`, + `dead_lettered_at IS NULL`, `terminal_reason IS NULL`, `attempts < 6`, due + availability and no live lease, and must pass manifest/local-partition admission. + Expired prior claims go through recovery before becoming publishable again. + Requested or exhausted rows enter terminalization only, never normal publication. +- Every success, failure, recovery and terminal transition requires row identity, + the current token and an unexpired lease. Recovery first acquires its own fresh + token by atomically replacing an expired lease. A zero-row transition is lost + ownership: no late acknowledgement, error overwrite, lease clearance or audit. +- Six total attempts mean the initial claim and five retries, delayed by 1 s, + 5 s, 30 s, 5 min and 1 h. A claim committed before shutdown/crash counts even + when publication's outcome is unknown. Expired claims 1–5 enter backoff before + another normal claim; an expired sixth claim never invokes a seventh publish. +- An exhausted or invalid row records a closed `terminal_reason` request. A + terminalizer acquires it without incrementing attempts or publishing, then + atomically writes `dead_lettered_at` and its MUST audit. Audit failure retains + the request for terminalization retry; it cannot cause another delivery. + `terminal_reason` alone is not completed dead-letter state. Successful, retry + and terminal transitions clear their lease in the same short transaction. + Terminal state does not prove that no subscriber accepted an event: a timed-out + publication may already have committed durable acceptance. +- Dead-lettered rows are permanently excluded from dispatch and pending gauges. + A far-future `available_after` is not a terminal marker. Producer terminal rows + retain their envelope for investigation; operator replay is Phase 03, with a + separately specified reset/fencing contract before any endpoint ships. +- Eligible selection has a deterministic total order + `(available_after, occurred_at, id)`. This promises selection order for one + processor without failure, not enqueue/commit order, retry order or cross-process + order. Per-key in-process serialization does not strengthen that promise. + The normal pending partial index uses that same three-column order and excludes + processed, terminal-requested and dead-lettered rows; recovery/terminalization + receive separate matching indexes in the canonical DDL and migration. + +### Subscriber acceptance and inbox ownership + +P02b-3 implements this coupled contract; P02b-2 does not claim an inbox or +effectively-once business effects. + +- Give each subscription an explicit stable ID, independent of handler class or + assembly name. Its closed ASCII grammar is `{module}.{subscription}.v{N}`: + lowercase letter/digit module, lowercase letter/digit/hyphen subscription, + both starting with a letter, and positive decimal N without leading zeros. + N versions the subscription's deduplication semantics, not the event or built-in + revision. Renaming creates a new identity and needs an explicit migration/replay + decision; a handler refactor preserves its ID. Duplicate IDs refuse composition. + A shared tenant-owned `inbox_messages` uses + `(tenant_id, subscription_id, event_id)` as its deduplication identity. +- The transport owns the inbox call. Inside an announced application-role + transaction, a uniqueness-backed claim serializes duplicate attempts **before + handler construction**. Business state, completed inbox marker, produced outbox, + MUST audit and fenced delivery-ledger completion commit together through + ADR-0040/0057's common owner lifecycle. Before owner commit, the ledger UPDATE + must match its token and database-time unexpired lease. A sticky completion gate + also refuses a timed-out/canceled attempt even if its token is unchanged. Zero + rows or a closed gate poison and roll back the whole transaction before COMMIT. + A failed transaction leaves neither completed inbox nor business effect. + Cancellation after physical COMMIT starts is Indeterminate under ADR-0033, + not guaranteed rollback; recover from the atomic ledger/inbox state. +- The inbox claim is an uncommitted row on that same business transaction; only + successful completion persists it with a completed marker. Rollback removes a + new claim. A committed completed row is never reset by targeted replay; the + delivery ledger owns incomplete/terminal/replay state, not a failed inbox row. +- A separate tenant-owned delivery ledger durably retains the validated envelope, + attempts, availability, lease and terminal/replay state. A terminal subscriber + failure never marks its inbox processed. P02b-3 owns its canonical DDL, RLS, + exact grants and failure/terminal audit; no second dead-letter topic or duplicate + dead-letter table is required. Its envelope must survive producer-row purge. + P02b-3 must choose a per-subscription copy or shared tenant/event envelope before + DDL, name the resulting copies/references and retention dependency, and add all + retained envelope stores to P02b-8's erasure inventory. No new business payload + or arbitrary metadata is introduced by retaining the producer envelope. +- `IEventBus.PublishAsync` keeps its non-generic envelope signature. Once P02b-3 + installs durable acceptance, success means every required delivery was durably + accepted, not that every handler succeeded. Duplicate acceptance is idempotent; + partial acceptance can safely be retried. Unknown acceptance leaves the producer + retryable. A subscriber scheduler retries one `(tenant, subscription, event)` + through an internal delivery executor, never through another broadcast. + The executor invokes that subscription, not `PublishAsync`; the outbox processor + remains the only production publisher and receives no new publication exception. +- The existing outbox role schedules cross-tenant delivery machinery using only + the column-level scheduling-identity/routing/lease/status SELECT and status-UPDATE + grants P02b-3 enumerates on that ledger; no envelope/payload SELECT is granted. + The app role receives only tenant-RLS-protected acceptance and completion + permissions, including the named completion-column UPDATE; it cannot claim, + reset retry counters or schedule another tenant's work. Business execution + always opens a separate `learnstack_app` scope. It restores + the validated envelope tenant and organization before resolution, announces the + transaction, reads and validates its envelope under RLS by claimed identity/token, + then constructs the handler and executes as `UserId.SystemActor`; causal actor + stays metadata. + Absence of the organization marker does not erase a present envelope org. A + truly tenant-wide fact deliberately carries null; the marker requires an org. +- After durable acceptance, a subscription's business failure cannot spend a + sibling's or producer's retry budget. Failed/unknown durable acceptance still + consumes producer attempts; it is not a subscriber execution outcome. + Targeted replay resets only that subscription with fresh ownership fencing; + completed siblings remain deduplicated. P02b-3 supplies the internal replay + operation; Phase 03 supplies operator permission and HTTP surface. +- External effects need provider idempotency keys and reconciliation; an inbox + transaction cannot guarantee exactly once outside its database. Audit's + platform-scope erasure consumer requires its own privileged execution contract + in Phase 03, not a disguised ordinary application-role delivery. Its privileged + effect uses the audited `EnterPlatformAdminScope` path and idempotent progress/ + reconciliation; an app-role inbox cannot make two connections atomic. P3's + ordinary executor refuses privileged subscriptions until that explicit owner + supplies their execution contract. Phase 11 owns retention purge, not redaction. + +### Atomic terminal audit and privilege boundary + +- Keep the four-role model. Extend `learnstack_outbox_admin` only with the new + outbox status-column UPDATE grants and **INSERT-only** on `audit_log`; no audit + SELECT/UPDATE/DELETE, business-table access or privileged role memberships. + The audit grant belongs in a new Audit-chain migration, after `audit_log` exists. +- Add the fifth, narrow audit write seam + `IAuditStore.WriteEventTerminalAsync(AuditEntryDraft, EventTerminalTransition, + DbConnection, DbTransaction, CancellationToken)`. The immutable transition + witness is minted only from the successful fenced terminal `UPDATE ... RETURNING` + in that live transaction, never from caller-supplied expected identity. Restricted + construction and a planted architecture census confine that minting path. + The witness binds connection/transaction, acquisition, actual stored tenant/org, + event, producer/subscriber kind, reason, attempts and, for a subscriber, its ID. + Before INSERT, the seam validates the declared operation/class/type, live caller + transaction and every corresponding draft identity against that witness; a + mismatch throws and requires rollback of both writes. It inserts only and + opens/commits no transaction itself. + P2 admits only the dispatcher terminal writer and the producer operation below. + P3 reserves the subscriber terminal writer and `platform.delivery.dead_letter`, + as off-path MUST/SecurityEvent in the same Tenancy catalogue/matrix, with stable + subscription/event identity and the same-transaction proof; register that + operation only when its writer ships. Reject other callers, undeclared + operations and wrong operation class/type. `PostgresAuditStore` remains the only + audit SQL writer. Existing four writes retain their contracts. +- Declare `platform.outbox.dead_letter` off-path as `OperationClass.Must` and + `OperationType.SecurityEvent` in the Tenancy catalogue/matrix, beside the + existing infrastructure-owned `platform.admin_scope.enter` precedent. It needs + no synthetic business aggregate. SecurityEvent records the consequential + BYPASSRLS terminal state change; routine polling/claims retain the separate + bounded exemption below. The row records the stored real tenant/org, + SystemActor, event ID, closed reason, attempts and validated causal metadata. + Outcome success describes the committed terminal transition, not publication. + If stored correlation/causation/actor metadata is invalid, use a fresh validated + attempt trace and omit invalid causal values; never copy malformed metadata or + require a valid original envelope in order to audit its terminal refusal. + Closed terminal reason/attempts/subscription values live in bounded metadata; + `AuditEntryDraft.Reason` remains reserved for platform-scope entry reasons. +- The fenced terminal UPDATE and audit INSERT share one transaction. Failure of + either rolls back both; a stale claimant writes neither. An uncertain COMMIT is + reconciled from durable state before retrying; it cannot append a second audit + to an already terminal row. Metrics remain process observations, not an + exactly-once durable event ledger. +- The audit INSERT's tenant binding is a witness/call-path invariant, not RLS: + the outbox role bypasses that policy. Neither announcing GUCs nor INSERT-only + grants can bind it to one tenant. Tests must substitute another tenant, org or + event against a real fenced transition and prove both writes roll back, alongside + a matching organization-scoped success. This confines sanctioned code; it does + not sandbox arbitrary SQL using a compromised bypass credential. +- Routine confined dispatcher bypass does not write one security-event audit per + poll/claim/mark. The exemption requires grant confinement, structured bounded + attempt outcomes, metrics/health and MUST-audited terminal transitions. It + changes no `EnterPlatformAdminScope` pre-entry audit or other bypass operation. + +### Diagnostics and execution bounds + +- Persist at most 256 UTF-8 bytes in `last_error`, enforced by a CHECK. Use closed + stage/reason/category codes, an optional allowlisted SQLSTATE and canonical + validated attempt traceparent. Never store exception messages, stacks, provider + details, SQL, connection strings, payloads or raw stored type text. Replace + existing free-form descriptors with a fixed legacy descriptor before the CHECK; + do not truncate potentially sensitive text. Preserve fact identity and state. +- New identity CHECKs refuse empty event/tenant IDs, the platform tenant sentinel + and empty optional organization IDs. Invalid legacy identities fail migration + for explicit repair; they are not silently rewritten or discarded to make an + audit insert possible. +- A named keyed, lazy dispatcher data source is API-host-owned, never Seeder- or + web-owned. Configuration validates the exact username without connecting or + exposing parse input. Boot verifies connectivity and every new physical + connection verifies exact session/current role, BYPASSRLS, NOSUPERUSER, + NOCREATEDB, NOCREATEROLE, NOREPLICATION and no privilege-escalating membership. +- Publish deadlines bound the dispatcher's wait, not arbitrary callback lifetime. + Cancel the callback, retain and observe its task-owned scope until it actually + finishes, and fence all late transitions. Do not dispose live resources or + accumulate unbounded abandoned tasks. Bounded overruns may let unrelated keys + progress; block their own partitions, mark health unhealthy and stop acquisition + when the approved outstanding-task bound is reached. Shutdown has a finite + wait; process exit and leases provide recovery, not forced managed-task killing. + P02b-3 must additionally fence its business transaction against late completion. +- Metrics prohibit `tenant_id`, `event_id`, `partition_key`, trace identifiers and + other unbounded identity labels. `event_type` is allowed only from the finite + declared registry plus fixed unknown/retired/invalid categories. + Gauge callbacks read immutable memory snapshots, never perform SQL. Failed + refreshes retain their prior snapshot and become unhealthy rather than report + a fabricated empty queue. The outbox health check is registered, not publicly + mapped; `/healthz` stays liveness-only and Phase 11 owns readiness exposure. + +### Bounded supersession and additive seams + +On acceptance, the following replacements take effect only in their named scope. +They are new decisions, **not ADR-0041 corrections**. Original Accepted bodies +remain intact, with dated disclosure links naming this record and activation. + +| Authority | Replaced clause | Replacement / activation | +|---|---|---| +| [ADR-0006 Amendment 1](0006-events-and-outbox.md#amendment-1--dapr-pubsub-as-dispatch-transport-2026-05-18) | Consumer-side handler calls and own-module inbox/deduplication | Transport-owned shared inbox, stable subscription key; P02b-3 | +| [ADR-0010 Amendment 1](0010-cross-module-communication.md#amendment-1--dapr-pubsub-as-outbox-dispatch-target-2026-05-18) | Idempotency's own-module inbox and handler-called guard, including its assertion | Shared transport-owned guard and corresponding catalogue rule; P02b-3 | +| [ADR-0038 Event contract](0038-cross-cutting-port-and-event-contracts.md#event-contract) | Publish reports immediate subscriber construction/invocation/disposal failures; future adapters preserve that outcome | Publish reports durable acceptance; separate subscription execution owns these failures; P02b-3 | +| [Database Standards GRANT matrix](../standards/05-database.md#grant-matrix) and [Outbox grants](../standards/05-database.md#outbox) | Outbox-only SELECT and four-column status UPDATE; no audit privilege | Named status writes plus audit INSERT in P02b-2; only named delivery machinery scheduling grants in P02b-3 | +| [ADR-0044 § 9](0044-audit-write-path.md#9-audit_log-and-audit_config-are-tenant-owned-org-scoped) | Outbox role has no `audit_log` permission | INSERT-only for confined atomic terminal audit; P02b-2 | +| [Audit Coverage](../standards/18-audit-coverage.md) | Every platform-bypass invocation receives a security-event row | Confined routine dispatcher exception above; P02b-2 | + +The fifth audit method is an **additive ADR-0044 seam amendment**, distinct from +the bounded grant-cell replacement above. Neither changes ADR-0033's +same-transaction guarantee. ADR-0003 Amendment 3's four roles and RLS postures +remain unchanged; its dated disclosure points to the actual grant-budget extension +in Database Standards, rather than claiming an outbox-only clause it never stated. +ADR-0033 receives a clarification of +that guarantee's new caller. P02b-3 adds inbox/delivery machinery to ADR-0044's +capture exclusions because explicit intents audit business effects; retry state +is not a second business aggregate. ADR-0023's inbox minting-list correction and +canonical Inbox DDL remain P02b-3 obligations, verified against acceptance history. +ADR-0038's envelope, frozen serializer, effective SystemActor, organization +restoration, per-key serialization and producer-only publication rules survive. + +## Context + +The current `PostgresOutbox` writes on the producer transaction, but no dispatcher +exists. `InProcessEventBus` constructs a scope per subscriber and collects their +failures. Treating that broadcast result as a durable per-subscriber outcome would +re-execute successful siblings. Treating zero handlers as unconditional success +would discard P1 provisioning events before their real subscribers exist. + +The current schema has neither leases nor explicit terminal state. The outbox +role can update four status columns and cannot write audit. Existing standalone +audit writes open another transaction; platform-scope writes have a different +credential and gate. Neither preserves atomic terminal-state auditing here. + +The queue and audit are first-party correctness machinery inside the existing +modular monolith. This selects no broker, fifth cross-module mechanism, new worker +host, operator endpoint or generic job runtime. Cross-process adapters still need +ADR-0035's trigger. G9 and ADR-0047 remain the next job-runtime decision, after P3. + +### What would change this decision + +A consumer requiring strict partition sequence, a second consumer process or +provider side effects without idempotent reconciliation requires a new bounded +decision. Schema aliases cannot silently turn an old fact into a breaking V2. + +## Consequences + +### Positive + +- P2 can activate its real production worker without losing P3's useful events. +- Subscriber deduplication and retries follow stable independent identities. +- Terminal state has a durable atomic audit and a privilege boundary tests can pin. +- Crash recovery, ordering and timeouts have falsifiable rather than absolute claims. + +### Negative + +- Pending contracts intentionally retain backlog until P3 activates their consumers. +- Terminalization adds a small explicit state machine and a narrow audit privilege. +- In-process callbacks that ignore cancellation can exhaust the bounded overrun + allowance. Health becomes unhealthy; an operator may need to restart the API. +- At-least-once recovery can repeat transport delivery; P3's inbox remains essential. + +### Neutral + +- No public contract changes; dead-letter operator routes belong to Phase 03. +- Erasure inventory belongs to P02b-8, privileged redaction to Phase 03 and + retention purge to Phase 11, matching the Audit module's existing ownership. +- The companion Hub roadmap's per-module-inbox dependency needs a coordinated + pointer update before P02b-3 closeout; this does not change Hub HTTPS contracts + or authorize Hub runtime work. + +## Implementation Notes + +P02b-2 implements producer claims/recovery/terminal audit, registry/manifests, +credential confinement, API lifecycle, diagnostics and health. P02b-3 implements +durable acceptance, inbox/delivery ledger, independent subscriber retries/audit, +targeted replay, both Customization defaults and Seeder semantic-ID handoff. +P3 must fence timed-out business transactions before enabling Active consumers. +It also reconciles existing bus/consumer exception logging and span capture with +its bounded failure policy; P2's minimized diagnostics cover the dispatcher-owned +path and do not claim existing transport telemetry is already minimized. + +At acceptance, reconcile Standards 05/09/10/11/18/20/21, architecture 15/31, glossary, +the phase register, Tenancy/Audit specs and applicable skills with the named +scope. Register tests before code; mark Implemented only with passing evidence. +Update `.env.example`, API/public-runtime/demo credential propagation and fixtures +with runtime delivery. An isolated test-owned worker replacement must not become +a production disabled-by-default switch. + +The acceptance reconciliation explicitly replaces these non-ADR carriers, without +leaving competing current designs. Target P3 text stays labelled unimplemented: + +| Carrier | Text to reconcile | Replacement / activation | +|---|---|---| +| Architecture 15: claim protocol, simpler alternative and processor guarantees | BatchSize 100, processor-ID lease, batch-held fallback and lease clearing | One just-in-time claim, fresh acquisition token and only the fenced transitions above; P02b-2 | +| Architecture 15: Inbox table and consumer pattern; glossary `IInboxGuard` | Per-module table and handler-called guard | Shared transport-owned claim/completion contract; decision at acceptance, implementation P02b-3 | +| Architecture 15: Dead-letter, both sides | Future-availability terminal hack, `learnstack.dlq.{module}`, `dead_letter_messages`, broadcast replay and `OutboxStatusEndpoints` | Explicit producer terminal state and subscriber delivery-ledger terminal state; targeted replay internal P02b-3, operator routes Phase 03, broker adapter still demand-gated | +| Glossary `OutboxProcessor` and `locked_by` descriptions | Batch claim and stable processor identity | Single-row acquisition tokens; stable worker identity is bounded diagnostic context, not persisted ownership; P02b-2 | +| Standards 09: Outbox | Catch handler failures, then increment producer attempts and log exceptions | Claim-counted producer attempts, bounded dispatcher diagnostics, and separate post-acceptance subscriber outcomes in P02b-3 | +| Standards 10: Spans and Required Metrics | Dispatcher batch spans and tenant-labelled pending count | Per-attempt spans, unlabelled pending count and the package's bounded metric inventory; P02b-2 | +| Standards 18 join grammar and Standards 21 audit-matrix rule | Seven off-path rows | Reconcile actual/planned row enumeration with producer registration in P02b-2 and subscriber registration in P02b-3 | +| Standards 21 `Outbox_Claim_IsHeld_Until_Dispatch_Completes` | Unconditional exactly-once dispatch, batches and two processes | At most one live owner under two independent processor instances, plus expiry/crash redelivery; P02b-2 | +| Standards 21 `OutboxProcessor_NeverBlocks_OnSingleMessageFailure` | Rest-of-batch premise | Continued unrelated eligible work with ordinary failure and bounded overruns; P02b-2 | +| Phase 02b: packet targets, Durable outbox dispatch and Completion Criteria | Unconditional zero-subscriber acknowledgement, batch/terminal sketches and dispatcher-connection replay as a settled operator contract | Explicit intentional absence versus Pending hold, the claim/terminal rules above, and separately decided Phase 03 operator replay; reconcile at acceptance, preserve dated P02b-1 history | + +The superseded consumer subscription sketch does not introduce +`IModule.ConfigureEventHandlers`: no such repository type exists. P02b-3 uses +the declared manifest and the actual composition APIs. Historical P02b-1 "Next" +and "reserved and undrafted" statements are as-of records; the current P02b-2 +package and Active index own navigation to this Accepted record. + +## Architecture Tests + +Existing rules to backfill or reconcile without weakening their intent: + +- `LearnStack_OutboxAdmin_Role_OnlyUsedBy_OutboxProcessor` — exact credential + confinement, with separate platform-key controls. +- `OutboxProcessor_NeverBlocks_OnSingleMessageFailure` — ordinary failure and + bounded cancellation-ignoring overrun, including safe scope ownership. +- `Outbox_Claim_IsHeld_Until_Dispatch_Completes` — two independent processors with + live leases; expiry, same-worker reacquisition and stale success/failure controls. +- `Integration_Event_Handlers_Use_InboxGuard` — P3 transport-owned invocation; + two same-module subscriptions and concurrent duplicates are distinct cases. + +Additional canonical rules to register on acceptance: + +- `Outbox_Retry_And_Terminal_Transitions_Are_Fenced`. +- `Outbox_Terminal_Transition_And_Must_Audit_Are_Atomic`. +- `Outbox_Terminal_Audit_Identity_Is_Row_Bound` — restricted witness construction + census and mismatched tenant/org/event/transaction rollback controls. +- `Outbox_Type_Resolution_Is_Declared_And_Bounded`. +- `Outbox_Pending_Subscriptions_Preserve_Events`. +- `Outbox_Health_And_Metrics_Are_Bounded`. + +Behavioural rules live in integration tests; structural rules live in architecture +tests. Every structural census includes production subjects and planted dirty and +clean controls. P02b-2's production subjects are the registered worker, resolver, +claim/transition store and audit writer, not nonexistent Active handlers. Real +Pending rows prove holds; test-owned Active subscriptions exercise the same shipped +publish/mark path with real PostgreSQL. Handler/durable-acceptance censuses remain +Registered until P02b-3 and must not pass on an empty production-handler set. +The package names the decisive migration, deadline, contention, recovery and +privilege proofs. + +## Amendments + +### Amendment 1 — Producer runtime delivery (2026-10-12) + +P02b-2 Steps 1–3 deliver the producer runtime authorized above: schema/grants and +confined composition, fenced claims/recovery/transitions with row-bound atomic +terminal MUST audit, and the mandatory API-host processor with bounded +publication ownership, observation and unmapped health. All three steps completed +both independent review rounds. The +[Step 3 delivery record](../roadmap/phase-02b-events-auth.md#p02b-2-step-3--hosted-lifecycle-and-observability-2026-10-12) +owns validation, reviewed corrections and the runtime proof scope. + +[Step 4 enforcement and closeout](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) +implements the full production censuses and planted controls; the focused 21-case +census/publication run passes with zero skips and the remaining role/health +catalogue rules are Implemented. Final regression, independent reviews and +exact-head CI remain in progress. This +amendment records delivered producer runtime, not completed P02b-2 packet closeout +or a merge, and changes no decision, rationale, grant budget or subscriber policy. + +P02b-3 remains unimplemented: durable acceptance, shared inbox/delivery ledger, +independent subscriber retries and terminalization, both real default +subscriptions and Seeder handoff stay with that packet. Required production +subscriptions remain Pending; test-owned Active subscriptions supply producer +runtime proof without claiming durable consumer delivery. + +### Amendment 2 — Producer enforcement and local closeout (2026-10-12) + +P02b-2 producer runtime and enforcement are locally complete. All four +implementation steps completed both independent review rounds. The +[Step 4 delivery record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) +records nonempty production credential/publication and observation censuses, +planted controls, review findings and verified corrections. Test-only round-2 +fixes at `606362b` pin exact source-reader method signatures, the actual +`TimeProvider.System` singleton and DI health binding to singleton observation. +All 26 focused/corpus cases and both fresh fix reviews pass. The post-fix Release +build has zero warnings/errors, all 286 architecture cases pass with zero +failures/skips, and solution-wide format verification passes. The full 3,393-case +backend regression predates those test-only guard fixes; the delivery record +states its scope and applicable earlier launcher/frontend evidence. + +The catalogue remains 193 architecture test methods and 192 Implemented rules: +118 in that assembly and 74 elsewhere. This amendment changes no accepted +decision, grant budget or consumer policy. PR-head CI and merge remain pending. +P02b-3 is unimplemented: durable acceptance, shared inbox/delivery ledger, +independent subscriber retries/terminalization, both real default subscriptions +and Seeder handoff remain its obligations. Required production subscriptions +remain Pending; test-owned Active contracts prove producer runtime only. + +## References + +- [ADR-0006](0006-events-and-outbox.md), [ADR-0010](0010-cross-module-communication.md) + and [ADR-0038](0038-cross-cutting-port-and-event-contracts.md). +- [ADR-0003](0003-tenant-isolation-defense-in-depth.md), + [ADR-0017](0017-tenant-organization-hierarchy.md), + [ADR-0036](0036-tenant-resolution-trusted-inputs.md), + [ADR-0040](0040-ambient-unit-of-work.md) and + [ADR-0057](0057-transactional-event-production-and-completion.md). +- [ADR-0023](0023-strongly-typed-id-source-generator.md), + [ADR-0035](0035-demand-gated-infrastructure.md), + [ADR-0041](0041-correcting-false-statements-in-accepted-adrs.md) and + [ADR-0042](0042-tenant-provisioning-cross-aggregate-transaction.md). + [ADR-0047's reservation](README.md#open-adr-drafts) is the later job-runtime + decision, not an existing ADR file. +- [ADR-0033](0033-audit-durability-model.md), + [ADR-0044](0044-audit-write-path.md), + [Database Standards](../standards/05-database.md), + [Observability Standards](../standards/10-observability.md) and + [Events and Outbox](../architecture/15-event-and-outbox.md). +- [PostgreSQL 18 SELECT](https://www.postgresql.org/docs/18/sql-select.html) + (`SKIP LOCKED` queue use and locking privileges). +- [.NET hosted services](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-10.0) + (cooperative cancellation and bounded host shutdown). diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 69211ef6..1f2a03f5 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -23,6 +23,11 @@ supersessions of named ADR-0032 and ADR-0040 clauses. Their original Accepted records remain intact; the linked disclosures below identify the replaced clauses and preserve the unaffected authority. +[ADR-0046](0046-durable-event-delivery-and-recovery.md#bounded-supersession-and-additive-seams) +likewise replaces only the named delivery, inbox and audit-grant clauses. Dated +disclosures in ADR-0003/0006/0010/0033/0038/0044 preserve the earlier records and +name P02b-2/P02b-3 activation; acceptance claims no runtime delivery. + ## Active ADRs | # | Title | Topic | @@ -70,6 +75,7 @@ clauses and preserve the unaffected authority. | 0043 | [What LearnStack Does With a Tenant-Authored Payload](0043-customization-payload-validation.md) | JSON Schema draft 2020-12, evaluated by `JsonSchema.Net` **pinned at 8.0.5** (the last MIT-expression release) behind an `IJsonSchemaValidator` port. A tenant document passes **four ordered gates** — JSON, the LearnStack profile, the 2020-12 meta-schema, then the build — because the library's own defaults keep none of the rules the corpus states: its default dialect is not 2020-12, a tenant's `$schema` line overrides the pinned one, a tenant's `$id` becomes process-global state (first-writer-wins, a cross-tenant denial of service), and `true` / `false` / `{}` are all valid schemas that switch validation off. `pattern` is refused — the built `Regex` carries an infinite `MatchTimeout` and `maxLength` does not short-circuit it (measured, 30.2 s on one property). **No compiled-validator cache**: compiling is measured *cheaper* than evaluating (0.5× at the 100-property ceiling), so § 8.2's cache is deleted rather than re-keyed. Rule bodies are opaque `text` with a `dialect` discriminator naming the language of their `condition` expressions. Amendments 1–2 (2026-09-04/05) refuse **every** `$ref` cycle and bound the reference graph; Amendment 3 (2026-09-06) seals the port's exception surface and makes `ValidateInstance` enforce § 8.4's entry cap; **Amendment 4 (2026-09-07) refuses what the `jsonb` column refuses** — `U+0000`, an unpaired surrogate, a number outside `numeric`; **Amendment 5 (2026-10-02) adds ADR-0051’s optional ordered text-card profile** | | 0044 | [The Audit Write Path](0044-audit-write-path.md) | **Amendments 1–5 (2026-09-08), 6–7 (2026-09-11).** The eleven questions ADR-0033 left open, decided before the first audit file: the reserved **non-nil** platform sentinel tenant id (the nil uuid is refused by three shipped guards that read all-zero as *no tenant*), and a `tenants` CHECK that keeps it unprovisionable; the row's `tenant_id` is the tenant the **transaction announced**, so a provisioning command audits under the tenant it creates and `make seed` still commits; **N intents per request**, one per audited resource, because two shipped commands already write two — and only the **owning** unit-of-work frame writes them or reports the commit boundary, which is what stops a nested `Send` from claiming durability nothing committed; `outcome` closes at four values (`indeterminate` included) and the writer always supplies `timestamp`, so ADR-0033's deliberate duplicate-id pair is legal under `(id, timestamp)` rather than a `23505`; the catalogue is **code** keyed by request type and the matrix is prose, with a test asserting they agree, and every request must be classified — there is no `RequestKind.Other`; the interceptor captures **every** tracked entity minus a named exclusion list, because the `AuditableEntity<>` predicate was blind to `platform_host_to_tenant`; `[PiiSensitive]` and an elision record that preserves the column's JSON type; `audit_log` is org-scoped, carries **no** FK to `tenants`, and both standalone writers announce **two** GUCs — measured: an org-scoped `denied` row is refused without the second. Amendment 1 makes `audit_config` tenant-**wide**; Amendment 2 corrects the redaction sentinel to the shipped constant and the reason `FORCE` needs a trigger (the policy binds the owner by tenant, not by immutability — measured); Amendment 3 gives the matrix↔catalogue join two directions with two domains, because 23 of 30 shipped matrix slugs have no request type and the rule as written was red on arrival, and moves the audit value types into `SharedKernel.Audit` to break the same project cycle `AuditEntryId` had; Amendment 4 makes `[PiiSensitive]` property-granular, `jsonb` included, ships `audit_config` unwritten and binds the matrix rule to modules with code; Amendment 5 guards the sentinel where it is announced and fills `entity_type` / `entity_id` from the declared aggregate; Amendment 6 — from an external review that reproduced four defects against real PostgreSQL — has a handler designate the subject when it writes two instances of its aggregate, carries the actor and correlation id on the intent, keeps each intent's own request's result, captures a contained entity with its aggregate, and instance-qualifies `changes` pointers; Amendment 7 corrects § 7's count of plain entities — seven today, five of them MUST — and names the Tenancy matrix alone as the one that singles out the host mapping. [Amendment 8 (2026-10-12)](0044-audit-write-path.md#amendment-8--common-audit-ownership-outside-mediatr-2026-10-12) clarifies common audit ownership outside MediatR under ADR-0057 without adding an audit writer. | | 0045 | [The Entitlement and Feature-Flag Socket](0045-entitlement-and-feature-flag-socket.md) | **Amendment 1 (2026-09-08)** reads it against the Hub's merged code: the limit vocabulary is the Hub's, `expires_at`/`valid_until` are nullable, the generation guard admits the equal case, and `platform_killswitches` ships **unwritten** because `DenyAllPlatformAdminGate` makes a writer unreachable until Phase 03. **Amendment 2 (2026-09-11):** a registry's membership is the vocabulary the contract names — fourteen Hub features, nine `limits.*`, two tenant flags, three killswitches — and enforcement, not membership, waits for a consumer. **Amendment 3 (2026-09-11):** a key's fail-open/fail-closed class decides only when no projection exists; one past its grace window is read-only, as ADR-0021 decides. **Amendment 4 (2026-09-11):** of the projection's names, `PlanCode` differs from the wire (`tier`) and `ExpiresAt` from the column (`valid_until`) — § 1 had called both wire differences. Declares the port twenty documents name and none define. `IEntitlementProvider.GetAsync(TenantId)` + `RefreshAsync(projection)`, the projection carrying every field `entitlement-v1.schema.json` requires — **`compliance` and `generation` included**, both already `NOT NULL` columns — and the refresh **generation-guarded inside the write statement**, so a stale push cannot resurrect a revoked plan. `IFeatureFlags` is the only module-facing read and **composes over the provider** rather than querying `platform_entitlement_cache`, which is the only reading under which the Phase 02a criterion — swapping the provider changes the answer — can be true. Limits: **`-1` unlimited, `0` denied**, `long` not `long?`; the inverse reading would have made the degraded read-only mode grant unlimited. `NullEntitlementProvider` is registered in **every** deployment mode, per ADR-0035's default-implementation gate rather than ADR-0020's Development-only switch. Killswitches leave `tenant_feature_flags` for a platform-scoped `platform_killswitches`: a foreign key to `tenants` is a constraint no role or `BYPASSRLS` moves, so the sentinel could never have satisfied it. Push-primary, **not** push-only — ADR-0034's `license/verify` fallback stands; the no-Hub absolute belongs to host resolution | +| 0046 | [Durable Event Delivery and Recovery](0046-durable-event-delivery-and-recovery.md) | Accepted 2026-10-12: fenced producer leases/retries, Pending readiness holds and atomic row-bound terminal audit; P02b-2 Steps 1–2 implemented with both review rounds complete; Step 3 hosted dispatch/observation implemented with both independent review rounds complete; Step 4 enforcement and local closeout complete; all four steps completed both independent review rounds; the [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) owns CI/merge status. [Amendment 2](0046-durable-event-delivery-and-recovery.md#amendment-2--producer-enforcement-and-local-closeout-2026-10-12) records verified local closeout. [Amendment 1](0046-durable-event-delivery-and-recovery.md#amendment-1--producer-runtime-delivery-2026-10-12) records producer runtime delivery and its review evidence. Shared inbox/durable subscription acceptance and independent retries remain P02b-3. Named bounded supersessions preserve earlier Accepted records. | | 0050 | [Publication and Course Content Access](0050-publication-and-course-content-access.md) | Accepted 2026-10-02: publication/access separation, restricted backfill and anonymous denial; supersedes ADR-0048; P02d-2 policy, migration, writers and seed delivered ([Amendment 1](0050-publication-and-course-content-access.md#amendment-1--p02d-2-implementation-delivery-2026-10-02)); P02d-4 public-read enforcement delivered, grants Phase 07 | | 0051 | [Ordered Text Card Presentation](0051-ordered-text-card-presentation.md) | Accepted 2026-10-02: optional root `x-fields`, localized ordered text cards; compatible extension linked by ADR-0043 Amendment 5; P02d-2 profile parsing, resolution and seed verification delivered ([Amendment 1](0051-ordered-text-card-presentation.md#amendment-1--p02d-2-implementation-delivery-2026-10-02)); P02d-4 public projection delivered; P02d-6 completes ordered plain-text rendering, automated product proofs and manual accessibility smoke | | 0052 | [Anonymous Public Read Boundary](0052-anonymous-public-read-boundary.md) | Accepted 2026-10-03: host-resolved bootstrap, immutable host ceiling, mandatory marked dispatch and read-only ambient frames; ADR-0036/0040 Amendment 8; P02d-4 Steps 1–3 deliver host provenance, physical read-only frames and all institution GET/HEAD reads; both Step 3 review rounds pass; Step 4 delivers OpenAPI/SDK/CI controls; both review rounds and focused fix verification pass; six real required CI jobs and live sixth-check activation verified; P02d-4 complete and merged through PR #25 on 2026-10-08 ([Amendment 13](0052-anonymous-public-read-boundary.md#amendment-13--merge-closeout-2026-10-08)) | @@ -77,7 +83,7 @@ clauses and preserve the unaffected authority. | 0054 | [Bounded Public Renderer Admission](0054-bounded-public-renderer-admission.md) | Accepted 2026-10-09: coordinated anonymous accounting, native GET/HEAD/production-upgrade admission and equivalent redirect-query encoding are delivered. All five remediation steps are complete and merged through PR #26; the [merge closeout](../roadmap/phase-02d-walking-skeleton.md#p02d-5-merge-and-closeout-2026-10-09) owns final verification ([Amendment 6](0054-bounded-public-renderer-admission.md#amendment-6--merge-closeout-2026-10-09)) | | 0055 | [Handle Public Renderer Bootstrap Failures in Middleware](0055-public-renderer-bootstrap-failures.md) | Accepted 2026-10-10: one live middleware bootstrap and an immutable native same-request snapshot; exact bootstrap 404/429/503, bounded renderer Retry-After and two/one/three-call accounting implemented. Production replacement proofs pass; all three steps completed both review rounds and focused fix verification. [Step 3](../roadmap/phase-02d-walking-skeleton.md#adr-0055-step-3--production-admission-proof-and-closeout) owns delivery evidence, completed reviews and verified fixes; five new catalogue rules are Implemented | | 0056 | [Public Demo Browser and Lighthouse Enforcement](0056-public-demo-browser-and-lighthouse-enforcement.md) | Accepted 2026-10-11: pinned Lighthouse/Playwright Chromium, seed-derived nine-page browser proof, 27 serial audits, actual-page admission, normalized gzip/CLS/accessibility hard checks and advisory timings delivered; four proof rules Implemented, real PR CI/artifacts pass and seventh required context activated ([Amendment 1](0056-public-demo-browser-and-lighthouse-enforcement.md#amendment-1--p02d-7-enforcement-delivery-2026-10-11)); complete and merged through PR #28 on 2026-10-11 ([merge closeout](../roadmap/phase-02d-walking-skeleton.md#p02d-7-merge-and-closeout-2026-10-11)) | -| 0057 | [Transactional Event Production and Completion](0057-transactional-event-production-and-completion.md) | Accepted 2026-10-12: immediate ambient enqueue, owner-only domain completion, sticky produced-work refusal protection, identifier-only bounded payloads and completed-provisioning publisher; eight producer rules Implemented; Steps 1–3 implement shared execution, immediate enqueue, successful-save collection and the publisher; Step 4 adds production censuses and catalogue enforcement. Amendment 1 links delivery verification and review status. P02b-3 owns default-initialization subscriptions and Seeder handoff. [Amendment 2 (2026-10-12)](0057-transactional-event-production-and-completion.md#amendment-2--correct-adr-0032-clause-locators-2026-10-12) corrects the ADR-0032 clause locators through inline errata; the Decision and bounded supersession are unchanged. | +| 0057 | [Transactional Event Production and Completion](0057-transactional-event-production-and-completion.md) | Accepted 2026-10-12: immediate ambient enqueue, owner-only domain completion, sticky produced-work refusal protection, identifier-only bounded payloads and completed-provisioning publisher; eight producer rules Implemented; Steps 1–3 implement shared execution, immediate enqueue, successful-save collection and the publisher; Step 4 adds production censuses and catalogue enforcement. P02b-1 is complete and merged through PR #29 on 2026-10-12; the [merge closeout](../roadmap/phase-02b-events-auth.md#p02b-1-merge-and-closeout-2026-10-12) owns final verification. Amendment 1 links delivery verification and review status. P02b-3 owns default-initialization subscriptions and Seeder handoff. [Amendment 2 (2026-10-12)](0057-transactional-event-production-and-completion.md#amendment-2--correct-adr-0032-clause-locators-2026-10-12) corrects the ADR-0032 clause locators through inline errata; the Decision and bounded supersession are unchanged. | ## Proposed ADRs @@ -146,7 +152,6 @@ table records the phase commitment so reviewers can flag late drafts. |---|---|---|---| | 0025 | Scoring + completion DSL sandbox engine (CEL vs restricted Lua vs custom; sandbox boundary; allowed function set) | **Phase 05** — `TenantCompletionRule` runtime evaluator lights up here; Phase 08a's assessment scoring depends on it | [ADR-0018](0018-tenant-driven-customization-model.md), [phase-05-education-learning-content.md](../roadmap/phase-05-education-learning-content.md), [phase-08a-assessment-notifications.md](../roadmap/phase-08a-assessment-notifications.md) | | 0026 | Release-tag scheme (`vYYYY.MM.DD.` vs SemVer; SaaS continuous-deploy reconciliation; Self-Hosted release cadence) | **Phase 11** — production hardening checklist owns this | [14-git-workflow.md § Tagging and Releases](../standards/14-git-workflow.md) | -| 0046 | Event delivery, the inbox, and the subscriber failure contract (inbox table identity and deduplication key; the dead-letter store; who claims a delivery; per-subscription attempts, terminal state and replay as one state machine) | **P02b-2 decision pass** — before dispatcher code; P02b-3 consumes the same state machine for its inbox and subscriber implementation | [phase-02b-events-auth.md](../roadmap/phase-02b-events-auth.md), [15-event-and-outbox.md](../architecture/15-event-and-outbox.md), [ADR-0006](0006-events-and-outbox.md) | | 0047 | Background job runtime (storage under the closed four-role model; the single tenant-context writer; the job frame's transaction owner; the enqueue-site rule; the retry and dead-letter contract; the ADR-0035 classification; the queue grammar; package pins and licence verdict) | **P02b-4 decision pass**, after P02b-3 — before job-runtime code; no compliant first boot exists today | [phase-02b-events-auth.md](../roadmap/phase-02b-events-auth.md), [20-infrastructure-stack.md § Background Jobs and Hangfire](../standards/20-infrastructure-stack.md), [09-tenant-isolation.md](../architecture/09-tenant-isolation.md) | **Reservation rule:** the numbers above are *reserved but not yet drafted*. When a diff --git a/docs/glossary.md b/docs/glossary.md index 6b350580..8b719f59 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -324,11 +324,11 @@ These entries distinguish P02d-2 contracts from P02d-3 internal read contracts: | **Demand-Gated Building Block** | An infrastructure choice that ships as a **port plus a working default implementation** now, and as a **vendor adapter later**, in a named phase, when a written trigger fires. It qualifies only when all four exist: the port, the default implementation, the owning phase, and the trigger condition. A block missing any of the four is not demand-gated — it is simply missing. Distinguished from a one-way door by the test below. The gated set and each trigger are tabulated in [ADR-0035](decisions/0035-demand-gated-infrastructure.md), which also names its own two exceptions — `audit_log` partitioning, which is schema-internal and so has no port, and LiveKit, which has no default because its absence is a missing product feature rather than a missing implementation. | | **One-Way Door** | A decision that gets more expensive with every week of delay, tested by the question: *if I add this six months from now, will I have to touch code that is already written?* **Yes → one-way door; ship it now** (tenant and organization isolation, the `outbox_messages` table and its ownership, strongly-typed identifiers, the localization schema — each touches every query, migration, or job payload ever written). **No → additive; ship the port now and the adapter on demand.** The test is mechanical, not a matter of taste: tenant isolation's cost grows with the codebase, a Dapr adapter's cost does not. A corollary rule: **a deployment mode or customer segment without a signed contract cannot be the deciding factor in a technical choice** — it may break a tie between otherwise-equal options, nothing more. Per [ADR-0035](decisions/0035-demand-gated-infrastructure.md) and [Engineering Principles](standards/00-principles.md). | | **Dapr Building Blocks** | The three Dapr abstractions LearnStack uses **when it uses Dapr**: pub/sub (Kafka), state (Valkey), secrets (Vault) per [ADR-0038](decisions/0038-cross-cutting-port-and-event-contracts.md). Service invocation, workflow, bindings, and actors are out of scope. Demand-gated to [Phase 11](roadmap/phase-11-production-hardening.md); ADR-0038 decides *what*, [ADR-0035](decisions/0035-demand-gated-infrastructure.md) decides *when*. | -| **`IEventBus`** | Interface for publishing integration events, taking an `IntegrationEventEnvelope`. `InProcessEventBus` is the only registered implementation until the Dapr adapter's trigger fires. It supplies `IIntegrationEventHandler`, scoped subscriptions, ambient tenant-context restoration and process-local per-key invocation serialization. Database announcement and `IInboxGuard` deduplication remain P02b-3 targets; durable publish ordering remains P02b-2 G3. The `OutboxProcessor` is the only sanctioned production caller. | +| **`IEventBus`** | Non-generic envelope publication port. `InProcessEventBus` is the only registered implementation until the Phase 11 trigger. P02b-3 changes success to durable acceptance by every required subscription, then independent execution owns business outcomes; failed/unknown acceptance remains producer retry. The outbox processor remains the only production publisher; subscriber retries invoke an internal executor. Current ambient restoration/per-key invocation exists; durable acceptance/inbox do not. See [ADR-0046](decisions/0046-durable-event-delivery-and-recovery.md). | | **`IOutbox`** | Immediate integration-event INSERT through the enrolled, announced, writable ambient transaction. `PostgresOutbox` is registered in API and Seeder by P02b-1 Step 2; it opens no connection, commits nothing and calls no transport. Identity is `EventId`; duplicates fail. Concrete payloads use the frozen serializer, identifiers only, and the inclusive 262,144-byte UTF-8 `OutboxPayloadLimits.MaxBytes`. `metadata` is SQL NULL. See [ADR-0057](decisions/0057-transactional-event-production-and-completion.md). | -| **`IntegrationEventPartitionAttribute`** | The concrete event's explicit partition-source declaration: a non-nullable aggregate identifier, or `TenantId` with `tenantWide: true`. Runtime admission compares the actual canonical lowercase `D` GUID value. Inherited declarations and implicit tenant partitioning are refused; durable publish ordering stays with P02b-2 G3. | +| **`IntegrationEventPartitionAttribute`** | The concrete event's explicit partition-source declaration: a non-nullable aggregate identifier, or `TenantId` with `tenantWide: true`. Runtime admission compares the actual canonical lowercase `D` GUID value. Inherited declarations and implicit tenant partitioning are refused. [ADR-0046](decisions/0046-durable-event-delivery-and-recovery.md#producer-dispatch-and-recovery) defines eligible-selection order and its limits; the declaration supplies no durable publish-order guarantee. | | **`IntegrationEventEnvelope`** | One integration event plus the dispatch metadata the outbox row carries and the event does not: W3C `CorrelationId`, optional `OrganizationId`, `CausationId`, and causal `ActorUserId`. Its `Topic` and `PartitionKey` are the event's own — both are properties of the event *type* rather than of one delivery, so neither can hold a second answer ([ADR-0038](decisions/0038-cross-cutting-port-and-event-contracts.md)). An event implementing `IOrganizationScopedIntegrationEvent` requires a non-empty organization in its envelope. Metadata describes the *delivery*; the event describes the *fact*. | -| **`IPartitionSerializer`** | Runs work sequentially within one partition key and concurrently across different ones in the current process. This invocation serialization establishes no durable publish-order guarantee; P02b-2 G3 owns that decision. Queuing work for the key you are already inside is refused rather than deadlocked: the caller that does it is publishing from inside a handler, which [Standards 20](standards/20-infrastructure-stack.md) forbids. | +| **`IPartitionSerializer`** | Runs work sequentially within one partition key and concurrently across different ones in the current process. This invocation serialization establishes no durable publish-order guarantee; [ADR-0046](decisions/0046-durable-event-delivery-and-recovery.md#producer-dispatch-and-recovery) defines eligible-selection order and its limits. Queuing work for the key you are already inside is refused rather than deadlocked: the caller that does it is publishing from inside a handler, which [Standards 20](standards/20-infrastructure-stack.md) forbids. | | **`EventTenantContext`** | The `ITenantContext` a consumer runs under, rebuilt from the envelope by the transport before handler discovery or resolution. A consumer executes outside the request that produced the fact, so there is no ambient context to inherit. The effective principal is `UserId.SystemActor`; an envelope human remains available separately as `CausalActorUserId`. Restoring the tenant and optional organization is what makes query filters and RLS policies evaluate against the right scope. | | **`TenantId`** | The strongly-typed tenant identifier, in `LearnStack.SharedKernel.Identifiers` per [ADR-0023 Amendment 2](decisions/0023-strongly-typed-id-source-generator.md)'s cross-cutting placement rule — it rides on `ITenantContext`, on every `[TenantOwned]` entity, in cache keys, job payloads and event envelopes, so a Tenancy-owned type would make each of those a reference to that module. It has **no `New()`**: a tenant id is assigned by the registry that owns the `Tenant` aggregate, because a handler minting its own could not satisfy the self-keyed policy's `WITH CHECK` ([Database Standards § Table classes](standards/05-database.md)). | | **`TenantId.PlatformSentinel`** | The reserved platform sentinel tenant id `00000000-0000-7000-8000-000000000002` — a UUIDv7-shaped constant on the precedent `UserId.SystemActor` sets, and **not** the nil UUID, which three shipped mechanisms read as *the absence of a tenant*. `tenants` carries a `CHECK` refusing it, so no tenant is ever provisioned under it and its rows are invisible to every tenant policy. Exactly one class of row carries it: a platform-scope operation with no resolvable tenant, written standalone — entry into `EnterPlatformAdminScope(reason)` and the operations inside it, of which `tenancy.killswitch.toggle` is the first — a row the Tenancy matrix carries as `(planned)`, because Packet 9 ships the killswitch table with no writer and [Phase 03](roadmap/phase-03-identity-admin.md) owns the toggle command. Never announced by `SetTenantContextAsync`, never written by a tenant request path, and never carrying a rejected tenant assertion — and that invariant has an **enforcer** rather than a hope: `SetProvisioningTenantContextAsync`, the one announcement path that takes a caller-supplied id, refuses the sentinel exactly as it already refuses `Guid.Empty`, and `Tenant.Create` refuses it in the factory. The `CHECK` is the **backstop, not the control** — a constraint on `tenants` cannot stop a GUC from being announced. Per [ADR-0044 § 1 and Amendment 3](decisions/0044-audit-write-path.md). | @@ -341,9 +341,16 @@ These entries distinguish P02d-2 contracts from P02d-3 internal read contracts: | **`IFeatureFlags`** | The **only** module-facing entitlement read: `IsEnabledAsync(FeatureKey)` and `GetLimitAsync(LimitKey)` returning `long`. It resolves the tenant from `ITenantContext` and throws when there is none, so module code can never name another tenant. It **composes** rather than queries: a plan-projected key resolves through `IEntitlementProvider.GetAsync` — not by reading `platform_entitlement_cache` — a tenant-flag key reads `tenant_feature_flags`, and the killswitch overlay is applied last and wins over both. Its implementation lives in `LearnStack.Modules.Tenancy.Infrastructure`, because `tenant_feature_flags` is that module's table, and is registered at the composition root. `IFeatureFlagService` and `IUsageLimitService` are **withdrawn**, not renamed ([ADR-0021](decisions/0021-feature-based-entitlement.md)). Per [ADR-0045 § 2](decisions/0045-entitlement-and-feature-flag-socket.md). | | **`IHostToTenantResolver` / `HostResolution`** | Interface for host → `(tenant_id, organization_id?)` resolution, returning the `HostResolution(TenantId, OrganizationId?)` record or null. Reads `platform_host_to_tenant` and **nothing else** — never the Hub, because an anonymous page load must not depend on a control plane being reachable ([ADR-0034](decisions/0034-hub-contract-surface-invariant.md)). | | **APISIX** | The gateway in standalone YAML-reload mode per [ADR-0015](decisions/0015-api-gateway-apisix.md). The intended tenant-facing ingress, demand-gated to [Phase 11](roadmap/phase-11-production-hardening.md); until then ASP.NET middleware carries the same responsibilities in-process. `/api/internal/*` is never proxied by it — that listener is mTLS-only inside the pod. | -| **`OutboxProcessor`** | The BackgroundService that **claims** batches of `outbox_messages` by writing a lease (`locked_by` / `locked_until`) under `FOR UPDATE SKIP LOCKED`, dispatches each through `IEventBus`, and handles retry / dead-letter. The lease is written to the row rather than held in the transaction, because `FOR UPDATE` locks end when the transaction ends. Delivery is **at-least-once**; consumer-side `IInboxGuard` is what makes duplicates safe. See [15-event-and-outbox.md](architecture/15-event-and-outbox.md). | +| **`OutboxProcessor`** | The API-host BackgroundService that claims one admitted outbox row immediately before invocation, with a fresh acquisition token in `locked_by`, a database-time lease and one counted attempt. Token/expiry-fenced transitions own retry and explicit audited terminal state; stable worker identity is diagnostic only. Accepted [ADR-0046](decisions/0046-durable-event-delivery-and-recovery.md) governs P02b-2 implementation: Steps 1–2 completed both review rounds, and Step 3 implements hosted execution/observation with both independent review rounds complete. Step 4 enforcement and local closeout are complete; all four steps completed both independent review rounds. The [Step 4 record](roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) owns CI/merge status; at-least-once recovery still needs P02b-3 inbox deduplication. | | **`IUnitOfWork`** | The ambient unit of work: **one database connection per scope**, and the transaction on it ([ADR-0040](decisions/0040-ambient-unit-of-work.md)). Every module `DbContext` resolved in the scope is built on that connection and enlisted in that transaction, and `IAuditStore` and `IOutbox` reach it through the same seam. The connection count is a correctness property, not a performance one: `SET LOCAL app.tenant_id` is connection- **and** transaction-local, so a context on its own connection reads zero rows from every tenant-owned table — silently. It exposes no `SaveChangesAsync`; contexts save themselves and it owns only the transaction boundary. [ADR-0040 Amendment 8](decisions/0040-ambient-unit-of-work.md#amendment-8--read-only-public-frames-2026-10-03) accepts explicit read-only/read-write intent, defaulting to writable: same-mode joins preserve ownership, mixed-mode joins poison the unit, and successful read-only completion permits a later writer. P02d-4 Step 1 delivers these modes and lifecycle controls; poison still requires a fresh scope. | -| **`IInboxGuard`** | The per-module inbox-deduplication helper. Every integration-event handler must call `IsAlreadyProcessedAsync` before business logic and `MarkAsProcessed` inside the same **transaction** as the business write — the ambient one the transport opens per delivery through `IUnitOfWork` ([ADR-0033](decisions/0033-audit-durability-model.md), [ADR-0040](decisions/0040-ambient-unit-of-work.md)). "The same `SaveChanges`" was the earlier formulation and ADR-0033 withdrew it. | +| **`IInboxGuard`** | The Accepted shared, transport-owned inbox claim/completion seam, pending P02b-3. Deduplication is `(tenant_id, subscription_id, event_id)`, claimed before handler construction on the announced app-role transaction. Business state, completed inbox, produced outbox, MUST audit and fenced ledger completion commit together; rollback removes a new claim. Handlers do not call the guard, and targeted replay never resets a completed inbox. See [ADR-0046](decisions/0046-durable-event-delivery-and-recovery.md#subscriber-acceptance-and-inbox-ownership). | +| **Inbox** | Shared tenant-owned completion record for one stable subscription/event pair. A new claim is uncommitted business-transaction work; only successful completion persists it. The delivery ledger owns failed/terminal/replay state. Canonical DDL remains P02b-3 in [Database Standards](standards/05-database.md#inbox). | +| **Integration-event subscription** | One explicitly identified consumer reaction, independent of handler class or assembly. Its stable `{module}.{subscription}.v{N}` ID versions deduplication semantics, not event schema or built-in revision. Renaming needs a migration/replay decision; a handler refactor preserves identity. [ADR-0046](decisions/0046-durable-event-delivery-and-recovery.md#subscriber-acceptance-and-inbox-ownership) owns the exact grammar. | +| **Subscription readiness** | Construction-free Pending, Active or IntentionallyUnsubscribed declaration in `OutboxEventRegistry`, the closed resolver of declared concrete contracts, compatible aliases and retired identities. Any required Pending subscription holds its contract/aliases before acquisition and validation. Active requires a real registered handler; intentional absence is explicit, never inferred from zero discovered handlers. Accepted P02b-2 policy under [ADR-0046](decisions/0046-durable-event-delivery-and-recovery.md). | +| **Outbox lease** | Paired `locked_by` / `locked_until` fields; the former holds a fresh canonical UUID acquisition token, the latter a database-time expiry. Every transition checks both current token and unexpired lease. Recovery replaces expired ownership before another transition. Stable worker identity confers no ownership. | +| **Dead letter** | Explicit completed terminal state after invalid data or exhausted attempts, distinct from a terminal request or delayed availability. On the producer row, `terminal_reason` requests terminalization and excludes normal publication; only `dead_lettered_at` records completed terminal state. Producer terminal state retains its envelope and commits with MUST audit; subscriber terminal state belongs to its independent delivery ledger and never completes its inbox. Operator reset/replay is separately owned by Phase 03. See [ADR-0046](decisions/0046-durable-event-delivery-and-recovery.md#producer-dispatch-and-recovery). | +| **`EventTerminalTransition`** | Restricted immutable witness minted only from the successful fenced terminal UPDATE result in its live transaction. It binds actual row identity, acquisition and terminal values to `WriteEventTerminalAsync`; substituting draft tenant/org/event or transaction requires rollback of state and audit. Fifth seam implemented in P02b-2 Step 2; both independent review rounds complete. | + | **`DeploymentMode`** | Enum (`Development \| SaaS \| Dedicated \| SelfHostedOnline \| SelfHostedAirGapped`) read at the composition root to select provider implementations. Modules never read this enum. | ## Hub & Licensing diff --git a/docs/modules/audit/README.md b/docs/modules/audit/README.md index b0faf188..141e184c 100644 --- a/docs/modules/audit/README.md +++ b/docs/modules/audit/README.md @@ -3,7 +3,7 @@ **Status:** Design stable, write path implemented. Shipped today: the two tables and their isolation, the three append-only layers, the SharedKernel ports and value types, `AuditDbContext`, and — in `LearnStack.Infrastructure.Audit` — `AuditStateCapture`, -`AuditChangeTrackerInterceptor`, `PostgresAuditStore` with all four write methods, the +`AuditChangeTrackerInterceptor`, `PostgresAuditStore` with all five write methods, the merged `AuditCatalog` and `AuditConfigService`, registered at both composition roots and attached to every module's `DbContext`. The pipeline is lit: `AuditLogBehavior` classifies and parks intents at step 3 and reconciles in its `finally`, @@ -28,6 +28,15 @@ module's `IUserReferenceLocator` retention purge land in [Phase 11](../../roadmap/phase-11-production-hardening.md) against written triggers ([ADR-0035](../../decisions/0035-demand-gated-infrastructure.md)). +**ADR-0046 fifth writer implemented in P02b-2 Step 2 — 2026-10-12; both review +rounds complete.** The existing +four writes remain implemented. `WriteEventTerminalAsync` adds INSERT-only access +for the confined outbox role and validates the audit draft against a restricted +witness from the actual fenced terminal UPDATE on the caller's transaction. +Terminal state and MUST audit are atomic; this is not a standalone or platform- +scope write. Tenancy owns the producer/subscriber operation declarations in +[its matrix](../tenancy/audit.md), with subscriber implementation in P02b-3. + The third module spec in the repository, per [Documentation Standards § Per-Module Specifications](../../standards/13-documentation.md). @@ -345,7 +354,9 @@ its mirror consumer. Existing source MUST rows remain on the source transaction. The reusable execution coordinator supplies non-MediatR classification/entry, owner write/boundary reporting and outermost reconciliation under ADR-0044/0033. P02b-1 proves a controlled direct owner; P02b-3 owns production delivery and its -per-effect audit after G1. No new audit writer or durability model is introduced. +per-effect audit under ADR-0046's accepted G1 contract. Business audit retains +the existing coordinator and durability model; explicit terminal audit uses the +separate fifth seam described above. ## Permission matrix diff --git a/docs/modules/audit/audit.md b/docs/modules/audit/audit.md index fe1e83da..78607d8a 100644 --- a/docs/modules/audit/audit.md +++ b/docs/modules/audit/audit.md @@ -3,6 +3,14 @@ Per [Audit Coverage](../../standards/18-audit-coverage.md), which names this file. Part of the [module spec](README.md). +Accepted [ADR-0046](../../decisions/0046-durable-event-delivery-and-recovery.md) +adds the confined terminal-audit seam in P02b-2, not an Audit-module operation. +`platform.outbox.dead_letter` and P02b-3's `platform.delivery.dead_letter` belong +to [Tenancy's off-path matrix](../tenancy/audit.md). P02b-2 Step 2 implements +the producer writer, with both review rounds complete; the subscriber writer +remains P02b-3. Phase 03 owns privileged erasure execution and its bounded +caller contract; Phase 11 owns retention purge. + **The module that owns the log is audited like any other.** Reading someone's audit trail is itself a sensitive read; exporting it produces a copy that outlives the request; and redacting or purging a row is the one act in the system that removes diff --git a/docs/modules/customization/README.md b/docs/modules/customization/README.md index 5abefb79..bd1a6724 100644 --- a/docs/modules/customization/README.md +++ b/docs/modules/customization/README.md @@ -181,9 +181,10 @@ an Active winner from another revision is never retired or adopted by seed. ### Primary integration-event flow: provisioning defaults -**Accepted flow, runtime subscriptions pending P02b-3 after G1.** +**Accepted ADR-0046/0057 flow; runtime subscriptions pending P02b-3.** [ADR-0057](../../decisions/0057-transactional-event-production-and-completion.md#first-useful-production-flow) -authorizes independent `card@1` and `plain@1` initialization on completed tenant +authorizes independent `card@1` content-type and `plain@1` taxonomy initialization +on completed tenant provisioning: ```mermaid @@ -196,7 +197,7 @@ flowchart LR P --> PT[One root + generation + inbox + MUST audit] ``` -Text fallback: Tenancy persists the completed-provisioning fact; after G1, +Text fallback: Tenancy persists the completed-provisioning fact; in P02b-3, independent subscriptions each create one untouched built-in concept on their own delivery transaction. One may finish before its sibling; retry completes an incomplete pair without overwriting tenant decisions. @@ -293,15 +294,17 @@ keeps the library's types inside it. ## Integration-event catalogue **No publisher or subscriber implemented yet.** Accepted ADR-0057 fixes the -P02b-3 subscriptions below; G1 must settle distinct deduplication identities and -delivery scope first. Each effect owns one root, generation change, inbox outcome +useful effects; ADR-0046 fixes the stable identities and shared transport-owned +inbox. P02b-2 manifests both required subscriptions Pending, holding real rows and +aliases before acquisition/validation; P02b-3 activates both with durable +acceptance. Each effect owns one root, generation change, inbox outcome and MUST audit. Customization mints UUIDv7 IDs and resolves semantic key/version identity; Tenancy carries neither schemas nor Customization IDs. -| Consumed topic | Event | Independent effect | Owner | -|---|---|---|---| -| `learnstack.tenancy.tenant` | `TenantCreatedIntegrationEventV1` | Create untouched `card@1` | P02b-3 after G1 | -| `learnstack.tenancy.tenant` | `TenantCreatedIntegrationEventV1` | Create untouched `plain@1` | P02b-3 after G1 | +| Consumed topic | Event | Stable subscription | Independent effect | Owner | +|---|---|---|---|---| +| `learnstack.tenancy.tenant` | `TenantCreatedIntegrationEventV1` | `customization.initialize-content-type-default.v1` | Create untouched `card@1` content type (`BuiltInCustomizations.Card`) | P02b-3; Pending until then | +| `learnstack.tenancy.tenant` | `TenantCreatedIntegrationEventV1` | `customization.initialize-taxonomy-default.v1` | Create untouched `plain@1` taxonomy (`BuiltInCustomizations.Plain`) | P02b-3; Pending until then | Initialization preserves Draft, Active, Deprecated, successor and deliberately deleted definitions; it never overwrites, republishes or resurrects them. P02b-3 diff --git a/docs/modules/tenancy/README.md b/docs/modules/tenancy/README.md index 813ac087..2c520ae9 100644 --- a/docs/modules/tenancy/README.md +++ b/docs/modules/tenancy/README.md @@ -494,10 +494,18 @@ back-reference completion; partition source is `TenantId`, payload has no extra fields and envelope organization is null. Enqueue reads the transaction's announcement while provisioning ambient context remains unresolved. State, event and the existing two MUST audit rows share one transaction. P02b-1 persists -pending rows; P02b-2/P02b-3 own dispatch/delivery. +pending rows; P02b-2 implements hosted dispatch and production-census enforcement. +All four steps completed both independent review rounds and local closeout. +The [Step 4 record](../../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) +owns CI/merge status. P02b-3 owns consumer delivery. The subscriptions below are accepted flow contracts, not delivered handlers. -G1 must settle distinct subscription deduplication/delivery scope first. +[ADR-0046](../../decisions/0046-durable-event-delivery-and-recovery.md) accepts G1's +shared transport-owned inbox and separate delivery ledger. P02b-2 holds this +contract Pending, including aliases, before claim or payload validation. P02b-3 +activates `customization.initialize-content-type-default.v1` for the `card@1` +content type and `customization.initialize-taxonomy-default.v1` for the `plain@1` +taxonomy together with durable acceptance; there are no placeholder handlers. | Topic | Payload | Publisher | Consumers | Phase | |---|---|---|---|---| diff --git a/docs/modules/tenancy/audit.md b/docs/modules/tenancy/audit.md index ad78a2d7..1d1100a8 100644 --- a/docs/modules/tenancy/audit.md +++ b/docs/modules/tenancy/audit.md @@ -8,7 +8,19 @@ adds one pending outbox row before the existing two MUST rows and commit. Its domain translator adds no independent intent or audit operation and writes no aggregate. A failed MUST write rolls back tenant, organization and outbox together. The [integration-event catalogue](README.md#integration-event-catalogue) owns the flow; -production dispatch and subscribers remain P02b-2/P02b-3. +P02b-2 implements hosted producer dispatch and production-census enforcement. +All four steps completed both independent review rounds and local closeout. +The [Step 4 record](../../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) +owns CI/merge status. Subscribers remain P02b-3. + +**ADR-0046 producer terminal writer implemented — 2026-10-12; both review rounds +complete.** +The infrastructure operations below belong to this catalogue/matrix, without a +synthetic aggregate. P02b-2 Step 2 registers the delivered producer terminal +operation; P02b-3 registers subscriber terminalization when its writer ships. +Both use the restricted fifth audit seam on the fenced state-transition transaction. +Routine confined dispatch has the explicit ADR-0046 exemption; platform-scope +entry auditing is unchanged. **P02d-4 Step 2 implemented — 2026-10-03.** Both review rounds and focused fix verification pass. Site bootstrap @@ -36,8 +48,9 @@ separate allowlist. Verification request types are Off. Packet 7 ships four audited root writes through three commands: Tenant and default Organization provisioning, a separate Organization create, and host mapping creation. P02d-2 adds two locale commands (SHOULD over Tenant) and one branding command (MUST over -TenantSetting). Thirteen rows remain `(planned)`; five operations have no MediatR -request and carry `(off-path)`. +TenantSetting). P02b-2 Step 2 adds the producer terminal MUST security-event writer. +Fourteen rows remain `(planned)`; seven operations have no MediatR request and +carry `(off-path)`. All four original writes are **MUST**. Their records commit with the business state, under [ADR-0033](../../decisions/0033-audit-durability-model.md). The branding writer @@ -102,7 +115,7 @@ already the permission key, and the first two segments of the two keys have to a so the spelling changes in both files or in neither. **A cell carries its slug and, where the row is not joined by request type, every marker -that applies** — a row can carry both, and two do +that applies** — a row can carry both ([ADR-0044 Amendment 3](../../decisions/0044-audit-write-path.md)). `(planned)` is a row classified ahead of the command that will raise it — the classification [Audit Coverage](../../standards/18-audit-coverage.md) asks for *before* the command @@ -113,6 +126,8 @@ unmarked row is one a shipped command already raises. | Resource | Operation | Class | Why | |---|---|---|---| +| `outbox_messages` | `platform.outbox.dead_letter` `(off-path)` | **MUST** (`security-event`) | P02b-2 Step 2's consequential BYPASSRLS terminal transition; state and row commit together through the row-bound fifth audit seam, with stored tenant/org/event identity and SystemActor ([ADR-0046](../../decisions/0046-durable-event-delivery-and-recovery.md)) | +| delivery ledger | `platform.delivery.dead_letter` `(off-path)` `(planned)` | **MUST** (`security-event`) | P02b-3 owns subscriber terminalization, stable subscription/event identity and the same atomic transition/audit guarantee; no writer or operation registration ships at acceptance | | `Tenant` | `tenancy.tenant.create` | **MUST** | The root of a customer's data; its existence is a contractual fact | | `Tenant` | `tenancy.tenant.change_status` `(planned)` | **MUST** | Suspension withdraws access; an operator must be able to say who and when | | `Tenant` | `tenancy.tenant.rename` `(planned)` | SHOULD | Presentational | @@ -153,24 +168,30 @@ in both directions ([ADR-0044 § 6](../../decisions/0044-audit-write-path.md)). ([ADR-0044 Amendment 3](../../decisions/0044-audit-write-path.md)). Catalogue → matrix is total: every entry this module's `IAuditCatalogSource` registers has a row here carrying the same slug, and one that does not fails. Matrix → catalogue binds only to a -slug whose request type **exists**, which is why the thirteen `(planned)` rows are +slug whose request type **exists**, which is why the remaining `(planned)` rows are classification and not drift — and a `(planned)` row whose command has since shipped fails, so the marker is a claim the rule re-checks on every run rather than an exemption. -Five rows sit outside that join in both directions, each marked `(off-path)` for the +Seven rows sit outside that join in both directions, each marked `(off-path)` for the reason its own cell gives: `platform.admin_scope.enter`, written on the fourth write path ([ADR-0044 § 10](../../decisions/0044-audit-write-path.md)); `tenancy.killswitch.toggle`, written inside `EnterPlatformAdminScope(reason)`; `tenancy.entitlement.refresh`, whose writer is `IEntitlementProvider.RefreshAsync` and -whose refreshing implementation lands in Phase 02c; and the two +whose refreshing implementation lands in Phase 02c; the two `tenancy.tenant_assertion.*` keys, written by a non-MediatR caller reaching `IAuditStore.WriteStandaloneAsync` -([ADR-0033 Amendment 2](../../decisions/0033-audit-durability-model.md)). Three are +([ADR-0033 Amendment 2](../../decisions/0033-audit-durability-model.md)); +`platform.outbox.dead_letter`, written with its fenced producer terminal UPDATE +through `IAuditStore.WriteEventTerminalAsync`; and `platform.delivery.dead_letter`, +whose subscriber writer belongs to P02b-3. Four are registered by slug through `DeclareOffPath` because their writer ships — -`platform.admin_scope.enter` and the two `tenancy.tenant_assertion.*` keys. -`tenancy.killswitch.toggle` and `tenancy.entitlement.refresh` also carry `(planned)` and -register by slug when Phase 03 and Phase 02c land their writers. Every other slug above is +`platform.admin_scope.enter`, `platform.outbox.dead_letter` and the two +`tenancy.tenant_assertion.*` keys. +`tenancy.killswitch.toggle`, `tenancy.entitlement.refresh`, +and `platform.delivery.dead_letter` also carry +`(planned)` and register by slug when Phase 03, Phase 02c and P02b-3, +respectively, land their writers. Every other slug above is joined in both directions the day its command lands. ## P02d-2 contextual verification reads diff --git a/docs/modules/tenancy/permissions.md b/docs/modules/tenancy/permissions.md index 1649298f..dff3c612 100644 --- a/docs/modules/tenancy/permissions.md +++ b/docs/modules/tenancy/permissions.md @@ -9,6 +9,14 @@ permission, and inserts a pending outbox row under the provisioning announcement Existing command reachability is unchanged. The [integration-event catalogue](README.md#integration-event-catalogue) owns the flow. +**ADR-0046 P02b-2 implementation locally complete — 2026-10-12.** All four steps +completed both independent review rounds. The [Step 4 record](../../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) +owns CI/merge status. +The API-owned dispatcher and terminal audit add no +permission key or HTTP endpoint. P02b-3's +targeted replay is internal; Phase 03 separately decides operator permission and +routes. Neither grants a tenant administrator bypass-role access. + **P02d-4 Step 2 implemented — 2026-10-03.** Both review rounds and focused fix verification pass. Site bootstrap is a marked GET/HEAD request, audit Off and without a permission key or grant. It diff --git a/docs/roadmap/README.md b/docs/roadmap/README.md index 92a520e1..1a7c5c1d 100644 --- a/docs/roadmap/README.md +++ b/docs/roadmap/README.md @@ -93,14 +93,21 @@ not deferred to the showcase phase. [merge closeout](phase-02d-walking-skeleton.md#p02d-7-merge-and-closeout-2026-10-11) owns final verification and the completed phase exit. - [Phase 02b: Events, Background Jobs, Identity, and Session](phase-02b-events-auth.md) - — **P02b-0 and P02b-1 complete; P02b-1 PR review/merge pending**. Its + — **P02b-0 and P02b-1 complete; P02b-1 merged through PR #29 on 2026-10-12**. Its [entry audit](phase-02b-events-auth.md#p02b-0-entry-verification-2026-10-12) verifies the inherited baseline; Accepted ADR-0057 and the [producer package](phase-02b-events-auth.md#p02b-1-decision-package-2026-10-12) authorize the implemented lifecycle, enqueue and provisioning publisher. The - [P02b-1 closeout](phase-02b-events-auth.md#p02b-1-packet-closeout-2026-10-12) - records full backend verification and two completed review rounds per step. - Next is P02b-2's decision pass; dispatcher/consumer work remains P02b-2/P02b-3. + [P02b-1 merge closeout](phase-02b-events-auth.md#p02b-1-merge-and-closeout-2026-10-12) + records final CI verification and the completed delivery/review history. + P02b-2's revised package and ADR-0046 are Accepted. Steps 1–2 completed both + review rounds; Step 3 implements hosted dispatch and observation, with + both independent review rounds complete in its + [delivery record](phase-02b-events-auth.md#p02b-2-step-3--hosted-lifecycle-and-observability-2026-10-12). + Step 4 enforcement and local closeout are complete; all four steps completed + both independent review rounds. CI/merge status lives in its + [delivery record](phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12); + consumer delivery remains P02b-3. - [Phase 03: Identity Domain, Authorization, and Admin Foundation](phase-03-identity-admin.md) - [Phase 04: Headless CMS, Page Builder, and Media Library](phase-04-cms-media-pages.md) - [Phase 05: Education Catalog and Learning Content](phase-05-education-learning-content.md) diff --git a/docs/roadmap/phase-02b-events-auth.md b/docs/roadmap/phase-02b-events-auth.md index f56c648c..34a58ade 100644 --- a/docs/roadmap/phase-02b-events-auth.md +++ b/docs/roadmap/phase-02b-events-auth.md @@ -18,9 +18,24 @@ its [record](#p02b-1-step-3--domain-completion-and-provisioning-publisher-2026-1 records passing verification and both completed review rounds. Step 4 adds production-census guards and catalogue enforcement; its [record](#p02b-1-step-4--producer-enforcement-and-closeout-2026-10-12) records full -backend verification and both completed review rounds. **P02b-1 is complete; -PR review and merge remain pending.** The -[packet closeout](#p02b-1-packet-closeout-2026-10-12) owns the next P02b-2 handoff. +backend verification and both completed review rounds. **P02b-1 is complete and +merged** through [PR #29](https://github.com/HodeTech/LearnStack/pull/29) on +2026-10-12. The [merge closeout](#p02b-1-merge-and-closeout-2026-10-12) owns final +verification. The Accepted P02b-2 package below owns the current dispatcher work. + +**P02b-2 decision pass — 2026-10-12.** The +[decision package](#p02b-2-decision-package-2026-10-12) and +[ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md) are Accepted. +The four implementation steps are authorized on development. Step 1 is complete; +its [record](#p02b-2-step-1--decision-schema-and-confined-composition-2026-10-12) owns +delivery evidence and both completed review rounds. Step 2 implements durable +transitions and terminal audit; both independent review rounds pass. Step 3 +implements hosted lifecycle and observation; its +[record](#p02b-2-step-3--hosted-lifecycle-and-observability-2026-10-12) owns +verification and both completed review rounds. Step 4 enforcement and local +closeout are complete; all four steps completed both independent review rounds. +The [PR handoff](#p02b-2-pr-handoff-2026-10-12) owns live CI/review/merge state. +P02b-3 implementation remains pending. **Phase 02d handoff — 2026-10-11.** P02b-0 re-verifies the delivered `/api/v1` public contracts, native HTTPS on retained `*.learnstack.local` seed hosts, @@ -69,20 +84,12 @@ keeps the phase from re-deciding settled questions or repairing repaired defects `InProcessEventBus` serialising handler invocation per key — shipped in [Packet 5](phase-02a-kernel-tenancy.md#delivery-record-packet-5) under [ADR-0038](../decisions/0038-cross-cutting-port-and-event-contracts.md). -- **The claim protocol.** - [Events and Outbox § The claim protocol](../architecture/15-event-and-outbox.md#the-claim-protocol) - specifies the lease, and - [§ The simpler alternative, and its cost](../architecture/15-event-and-outbox.md#the-simpler-alternative-and-its-cost) - demotes the batch-held transaction to a fallback for a deployment that cannot add the - two columns. [ADR-0006 Amendment 2](../decisions/0006-events-and-outbox.md) already - assigns `locked_by` / `locked_until` and the matching `learnstack_outbox_admin` - column-grant extension to this phase's migration. This phase **implements** that - design; it does not choose between candidates. -- **The delivery contract.** At-least-once dispatch with consumer-side idempotency. - [§ The claim problem](../architecture/15-event-and-outbox.md#the-claim-problem) - already withdrew the "no double dispatch" claim, and - [§ What the processor guarantees](../architecture/15-event-and-outbox.md#what-the-processor-guarantees) - bounds duplicates to crash recovery and lease expiry. +- **The claim and delivery protocol.** Accepted + [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md) supersedes the + batch-held fallback and unbounded ordering/exactly-once sketches. P02b-2 uses + fresh acquisition tokens, short claim transactions, six normal claims and + explicit terminal state. Dispatch stays at least once; P02b-3's shared inbox + provides per-subscription database idempotency. - **The ambient unit of work**, and the delivery shape [ADR-0040 § Consumers do not have a request](../decisions/0040-ambient-unit-of-work.md#consumers-do-not-have-a-request-and-still-need-all-of-this) assigns to this phase by name: `BeginTransactionAsync` → `SetTenantContextAsync` → @@ -97,21 +104,20 @@ keeps the phase from re-deciding settled questions or repairing repaired defects At the [P02b-0 entry audit](#p02b-0-entry-verification-2026-10-12), `IOutbox` and step 7's implementation were absent. P02b-1 Step 2 now supplies immediate -enqueue and the enrollment/invariant guard. Still absent are `IInboxGuard`, -`inbox_messages`, the lease columns, `OutboxProcessor`, any `BackgroundService` -or `IHostedService`, Hangfire and its package pins, and the realm's protocol -mappers. Packet 5 shipped the `IInboxGuard` **seam**, -which is the name in three comments — two XML doc comments in `SharedKernel.Messaging` -and the composition root's transport `TODO` — and in no declaration; the guard, its -table and its semantics are this phase's. +enqueue and the enrollment/invariant guard. P02b-2 Step 3 implements the +API-host `OutboxProcessor` BackgroundService, with both independent review +rounds complete. Still absent are `IInboxGuard`, `inbox_messages`, Hangfire and its +package pins, and the realm's protocol mappers. Packet 5 documented a planned +`IInboxGuard` seam in comments, without an interface declaration. P02b-3 owns its +shared transport implementation and table under ADR-0046. ### What this phase is not **It is not the phase where a transport is chosen.** `IEventBus` and `InProcessEventBus` shipped in [Phase 02a Packet 5](phase-02a-kernel-tenancy.md) as a **first-class -transport** — the same `IIntegrationEventHandler`, the same `IInboxGuard` seam, the -same tenant-context restoration as any durable path. Phase 02b adds *durability and -redelivery* on top of that transport; it does not add a second one. +transport** with the shared `IIntegrationEventHandler` contract and +tenant-context restoration. P02b-3 adds shared inbox durability and independent +subscription redelivery on top of that transport; it does not add a second one. The Dapr pub/sub and Kafka adapters are **not in this phase**. Both are demand-gated to [Phase 11](phase-11-production-hardening.md) by @@ -136,16 +142,14 @@ composition-root registration. Each item below is named somewhere as adjacent to this phase's work. Each has an owner. -- **An operator-facing recovery API.** This phase ships replay as a dispatcher-level - operation with its tests (§ The consumer side). The route space, the Platform-scope - permission key, the cross-tenant read and the runbook land in - [Phase 03](phase-03-identity-admin.md) beside the killswitch toggle, on the - [ADR-0045 Amendment 1](../decisions/0045-entitlement-and-feature-flag-socket.md) - precedent — `EnterPlatformAdminScope`'s registered gate is `DenyAllPlatformAdminGate`, - which refuses everyone until then. Whether a platform URL space exists at all is a - question that - [ADR-0036 § The platform-admin override is not a resolution source](../decisions/0036-tenant-resolution-trusted-inputs.md#the-platform-admin-override-is-not-a-resolution-source) - says needs its own decision record. +- **An operator-facing recovery API.** P02b-3 supplies internal targeted + subscriber replay through its delivery ledger and executor. Producer reset, + reason/permission, operator fencing and the runbook remain + [Phase 03](phase-03-identity-admin.md) work. No operator outbox reset is settled + here. `EnterPlatformAdminScope` still denies everyone until Phase 03; the URL + space needs the record + [ADR-0036](../decisions/0036-tenant-resolution-trusted-inputs.md#the-platform-admin-override-is-not-a-resolution-source) + requires. - **The Hangfire dashboard.** No route is mapped in any deployment mode here. [Phase 03](phase-03-identity-admin.md) maps it behind the Platform-scope permission and carries the [ADR-0015](../decisions/0015-api-gateway-apisix.md) amendment its @@ -236,10 +240,10 @@ Decisions made or referenced in this phase: - [ADR-0044 Audit Write Path](../decisions/0044-audit-write-path.md) - [ADR-0045 Entitlement and Feature-Flag Socket](../decisions/0045-entitlement-and-feature-flag-socket.md) (the killswitch precedent for a mechanism shipped without its operator surface) -- Two reserved, undrafted records this phase must Accept before its dependent packets - start: **ADR-0046** (event delivery, the inbox and the subscriber failure contract) - and **ADR-0047** (the background-job runtime), both registered in - [decisions/README § Open ADR Drafts](../decisions/README.md#open-adr-drafts). +- [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md) is Accepted + with its bounded delivery supersessions and separate additive audit seam. + ADR-0047 (job runtime) remains undrafted in + [Open ADR Drafts](../decisions/README.md#open-adr-drafts). ## Scope @@ -253,9 +257,9 @@ Accepted, and its catalogue rows registered, before its first line of code. | Packet | Contents | Cannot start until | |---|---|---| | **P02b-0** | Phase entry: every premise the register below cites re-verified against `HEAD` — the code, the corpus, the Hub repository — and the register corrected where Phase 02d moved it. It drafts no record: each gate closes in the decision pass of the first packet it shapes, and each glossary headword this phase needs — `inbox`, outbox lease, dead letter, audit frame, job activator, integration-event consumer — is written in the pass whose gate fixes its meaning | [Phase 02d](phase-02d-walking-skeleton.md) exits | -| **P02b-1** | Producer: `IOutbox`, the `OutboxFlushBehavior` body, the domain-event collector and its single dispatch site, and the first event with its enqueue path | G2, G3 (the key's derivation), G5 (the stored identities), G6, G18 (the payload rule), G19 Accepted | -| **P02b-2** | Dispatcher: the lease migration and grant extension, claim / publish / mark, backoff, the producer terminal state, the credential and its boot guard, the metrics and the health check | G1, G3, G4, G5, G7, G8, G17, G18 (`last_error`) Accepted; P02b-1 | -| **P02b-3** | Consumer: the inbox DDL, `IInboxGuard`, the per-delivery transaction, the sample flow's consumer, per-subscription attempts, the dead-letter store and its audit row, replay as a dispatcher operation | G1, G6, G7 Accepted; P02b-2 | +| **P02b-1** | **Complete and merged — [PR #29 closeout](#p02b-1-merge-and-closeout-2026-10-12).** Producer: `IOutbox`, the `OutboxFlushBehavior` invariant guard, successful-save domain-event collection and owner-only completion, and the first event with its enqueue path | G2, G3 (the key's derivation), G5 (the stored identities), G6, G18 (the payload rule), G19 Accepted | +| **P02b-2** | **Decision Accepted; all four implementation steps and both independent review rounds complete; local closeout complete; [PR handoff](#p02b-2-pr-handoff-2026-10-12) owns CI/review/merge state.** Dispatcher: the lease migration and grant extension, claim / publish / mark, backoff, the producer terminal state, the credential and its boot guard, the metrics and the health check | G1, G3, G4, G5, G7, G8, G17, G18 (`last_error`) Accepted; P02b-1 | +| **P02b-3** | Consumer: durable acceptance, shared inbox/ledger DDL and `IInboxGuard`, the common per-delivery transaction, both real defaults subscriptions and Seeder handoff, independent attempts and audited terminal state, targeted internal ledger replay | G1, G6, G7 Accepted; P02b-2 | | **P02b-4** | Background jobs: the runner, its storage, the tenant-context hook, the enqueue guard, no job definition | G9 Accepted; P02b-3 | | **P02b-5** | API authentication: JwtBearer wiring and its insertion order, claim population, the host/JWT cross-check through the existing recorder, token-keyed rate limits, the 401 rows, one authenticated endpoint | G10, G11, G12 (the BFF's client), G14, G20 Accepted | | **P02b-6** | Realm and seed: the shared client scope, the mappers, the user-profile declaration, UUID attributes agreeing with `SeedData`, the warm-realm reconciler, the Keycloak test fixture | G12, G13 (the access-token lifetime's authority) Accepted; P02b-5 | @@ -283,25 +287,25 @@ premise a row cites is re-verified at that pass rather than trusted. | # | Question | Leaning | Vehicle | Blocks | Status | |---|---|---|---|---|---| -| G1 | What is the subscriber's fate when its handler exhausts retries, and what is the inbox table — one shared or per module, keyed `event_id` or `(tenant_id, consumer, event_id)`, claimed by the handler or by the transport? Three arms travel with it: the class of consumer whose effect cannot share the inbox transaction (an external call, the platform-scope erasure handler the Audit spec already carries) and what "effectively once" means for it; whether an event without the organization marker is delivered tenant-wide regardless of the producing request's organization; and whether the inbox entity joins [ADR-0044 § 7](../decisions/0044-audit-write-path.md)'s named audit-capture exclusion list once a delivery declares intents | One shared tenant-owned `inbox_messages`, transport-claimed; the terminal state in its own store so "the inbox row is not marked processed" stays true | **ADR-0046** + a dated **ADR-0006 Amendment** (or an explicit supersession note in ADR-0046 naming Amendment 1's consumer-side paragraph, which fixes the per-module inbox) + a dated **ADR-0010 Amendment**, because its § Idempotency makes two things mandatory that the shared-table leaning changes — the inbox is "the module's **own** inbox table", and the guard is called by the consumer — + the `Integration_Event_Handlers_Use_InboxGuard` assertion re-written in the same record to name whoever ends up owning the call, since a transport-claimed delivery fails a structural rule that looks for the call inside the handler + Database Standards § Inbox + an ADR-0041 erratum beside ADR-0023's minting list | P02b-2 (the record and its amendments: the dispatcher's retry state and a failed delivery's next attempt are one state machine), P02b-3 (Database Standards § Inbox and the ADR-0023 erratum) | Open | -| G2 | Does `IOutbox.EnqueueAsync` write the row immediately on the ambient transaction, or buffer for `OutboxFlushBehavior` at step 7 — and what is the flush point for the two paths that have no step 7? | Immediate write, with the behavior asserting no buffered message survives the handler | [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md) bounded supersession of ADR-0032 Sub-decision 2's step-7 completion role, with a dated supersession disclosure in ADR-0032 at acceptance; phase-doc + [Events and Outbox § Producer pattern](../architecture/15-event-and-outbox.md#producer-pattern) | P02b-1 | Accepted — immediate enqueue/owner completion | -| G3 | What replaces the unconditional same-key publish-order promise — a scoped bound, or strict per-key head-of-line blocking — and is the key's derivation checkable structurally, given that [§ Ordering](../architecture/15-event-and-outbox.md#ordering) admits an aggregate id or a deliberately declared tenant-wide key and nothing else? | Scope it: no column in `outbox_messages` carries an unconditional per-key guarantee, so the honest bound is one processor, no failure, enqueue order | Derivation: [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md). Ordering: scoping is a phase-doc plus [§ Ordering](../architecture/15-event-and-outbox.md#ordering) edit; blocking is a dated ADR-0006 Amendment plus an index | P02b-1 (whether the key's derivation is checked structurally: the first event declares a key), P02b-2 (the ordering bound) | Derivation Accepted; ordering open (P02b-2) | -| G4 | What counts as an attempt, what is `MaxAttempts` bound to, and what marker keeps a terminal row terminal as the clock advances? | Six attempts (one immediate plus the five documented delays) and an explicit terminal column, because a pushed-out `available_after` stops being terminal and never leaves the pending gauge | Dated ADR-0006 Amendment + the Database Standards § Outbox DDL, index and grant edits riding it | P02b-2 | Open | -| G5 | Does the stored event identity stay the assembly-qualified type name with declared compatibility mappings, and what happens to an unresolvable type, a retired type and an event with no subscriber? And what is the **row's** identity — a surrogate `id` plus an `event_id` column, or `EventId` as the key — given that two documents list an `event_id` the canonical DDL does not have? | Keep the type identity, add a bounded resolver reading through `PayloadJsonOptions` and declared mappings; dead-letter a retired type at once rather than spending its attempt budget; reconcile the row's column list in one direction | Stored identities: [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md) bounded supersession of ADR-0032 Sub-decision 12 and the Phase 02b deliverables table in Implementation Notes, with a dated supersession disclosure at acceptance; Database Standards alignment. Resolver outcomes remain a P02b-2 decision; replacing the type identity requires its own ADR-0006 vehicle | P02b-1 (the stored type identity and the row's identity, both written at enqueue), P02b-2 (unresolvable, retired and unsubscribed types) | Stored identities Accepted; resolution outcomes open (P02b-2) | -| G6 | When the unit-of-work owner is not an `AuditLogBehavior`, who performs the four owner-side audit acts and enters the outermost audit frame — and does a state-mutating delivery declare its own intents, keyed by handler type? | The transport opens the outermost frame beside its transaction; a delivery declares its own intents and an unregistered handler is refused as an unregistered request is | [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md): bounded ADR-0040 nested-failure supersession; separately enumerated additive UoW read-seam amendment; clarifying ADR-0044/0033 common-owner amendments. G1-dependent production delivery/inbox implementation remains P02b-3 | P02b-1, P02b-3 | Producer/common owner Accepted; production delivery/inbox open (P02b-3 after G1) | -| G7 | For a dead-letter transition: what class, what `OperationType`, what slug, on which transaction, in what order against the state change, with what failure posture — and may a second non-MediatR caller reach `IAuditStore.WriteStandaloneAsync`? | class MUST, `OperationType` `security-event`, the slug open under the § The join grammar `{module}.{resource}.{verb}`; the row written before the transition, so a failed write leaves the transition unmade and the next poll retries | Dated ADR-0033 Amendment + `DeclareOffPath` registrations + the Baseline row in [Audit Coverage Standards](../standards/18-audit-coverage.md#baseline-coverage-learnstack-core-modules). If the chosen ordering persists the terminal transition and its MUST row on **separate** transactions, that changes what ADR-0033's Decision guarantees rather than clarifying it, and the vehicle is a superseding record, not an amendment | P02b-2 (the producer dead-letter row its criteria assert), P02b-3 | Open | -| G8 | Does the dispatcher's `BYPASSRLS` use carry a per-invocation `security-event` row, or is it exempt — and on what written terms? | Exempt, on three terms: the bypass is grant-bounded and code-path-confined, dispatch attempts are logged rather than audited, and the dead-letter transitions are the audited events | Dated ADR-0006 or ADR-0003 Amendment; an architecture document cannot grant it | P02b-2 | Open | +| G1 | What is the subscriber's fate when its handler exhausts retries, and what is the inbox table — one shared or per module, keyed `event_id` or `(tenant_id, consumer, event_id)`, claimed by the handler or by the transport? Three arms travel with it: the class of consumer whose effect cannot share the inbox transaction (an external call, the platform-scope erasure handler the Audit spec already carries) and what "effectively once" means for it; whether an event without the organization marker is delivered tenant-wide regardless of the producing request's organization; and whether the inbox entity joins [ADR-0044 § 7](../decisions/0044-audit-write-path.md)'s named audit-capture exclusion list once a delivery declares intents | One shared tenant-owned `inbox_messages` keyed by tenant/stable subscription/event, transport-claimed before construction; a separate delivery ledger owns retries, terminal state and targeted replay; completion shares the business/audit/outbox transaction | **ADR-0046** + a dated **ADR-0006 Amendment** (or an explicit supersession note in ADR-0046 naming Amendment 1's consumer-side paragraph, which fixes the per-module inbox) + a dated **ADR-0010 Amendment**, because its § Idempotency makes two things mandatory that the shared-table leaning changes — the inbox is "the module's **own** inbox table", and the guard is called by the consumer — + the `Integration_Event_Handlers_Use_InboxGuard` assertion re-written in the same record to name whoever ends up owning the call, since a transport-claimed delivery fails a structural rule that looks for the call inside the handler + Database Standards § Inbox + an ADR-0041 erratum beside ADR-0023's minting list | P02b-2 (the record and its amendments: the dispatcher's retry state and a failed delivery's next attempt are one state machine), P02b-3 (Database Standards § Inbox and the ADR-0023 erratum) | Accepted — 2026-10-12; decision/amendments delivered; subscriber implementation pending P02b-3 | +| G2 | Does `IOutbox.EnqueueAsync` write the row immediately on the ambient transaction, or buffer for `OutboxFlushBehavior` at step 7 — and what is the flush point for the two paths that have no step 7? | Immediate write, with the behavior asserting no buffered message survives the handler | [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md) bounded supersession of ADR-0032 Sub-decision 2's step-7 completion role, with a dated supersession disclosure in ADR-0032 under this acceptance; phase-doc + [Events and Outbox § Producer pattern](../architecture/15-event-and-outbox.md#producer-pattern) | P02b-1 | Accepted — immediate enqueue/owner completion | +| G3 | What replaces the unconditional same-key publish-order promise — a scoped bound, or strict per-key head-of-line blocking — and is the key's derivation checkable structurally, given that [§ Ordering](../architecture/15-event-and-outbox.md#ordering) admits an aggregate id or a deliberately declared tenant-wide key and nothing else? | Scope it: no column in `outbox_messages` carries an unconditional per-key guarantee, so the honest bound is one processor, no failure, deterministic eligible selection; no enqueue/commit-order guarantee | Derivation: [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md). Ordering: Accepted ADR-0046 and the P02b-2 package, followed by [§ Ordering](../architecture/15-event-and-outbox.md#ordering) alignment under this acceptance | P02b-1 (whether the key's derivation is checked structurally: the first event declares a key), P02b-2 (the ordering bound) | Derivation and ordering Accepted; scoped ordering runtime [implemented in P02b-2](#p02b-2-step-3--hosted-lifecycle-and-observability-2026-10-12); local closeout complete; [PR handoff](#p02b-2-pr-handoff-2026-10-12) owns CI/review/merge state | +| G4 | What counts as an attempt, what is `MaxAttempts` bound to, and what marker keeps a terminal row terminal as the clock advances? | Six attempts (one immediate plus the five documented delays) and an explicit terminal column, because a pushed-out `available_after` stops being terminal and never leaves the pending gauge | Accepted ADR-0046 + dated ADR-0006 disclosure under this acceptance; Database Standards § Outbox DDL, index and grant edits | P02b-2 | Accepted — 2026-10-12; producer attempts/terminal state [implemented](#p02b-2-step-2--durable-transitions-and-terminal-audit-2026-10-12) | +| G5 | Does the stored event identity stay the assembly-qualified type name with declared compatibility mappings, and what happens to an unresolvable type, a retired type and an event with no subscriber? And what is the **row's** identity — a surrogate `id` plus an `event_id` column, or `EventId` as the key — given that two documents list an `event_id` the canonical DDL does not have? | Keep the type identity, add a bounded resolver reading through `PayloadJsonOptions` and declared mappings; dead-letter a retired type at once rather than spending its attempt budget; reconcile the row's column list in one direction | Stored identities: [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md) bounded supersession of ADR-0032 Sub-decision 12 and the Phase 02b deliverables table in Implementation Notes, with a dated supersession disclosure under this acceptance; Database Standards alignment. ADR-0046 accepts declared resolution, Pending holds and intentional absence; dispatch runtime remains P02b-2 | P02b-1 (the stored type identity and the row's identity, both written at enqueue), P02b-2 (unresolvable, retired and unsubscribed types) | Stored identities and resolution outcomes Accepted; resolver/readiness runtime [implemented](#p02b-2-step-3--hosted-lifecycle-and-observability-2026-10-12); Active durable subscriptions pending P02b-3 | +| G6 | When the unit-of-work owner is not an `AuditLogBehavior`, who performs the four owner-side audit acts and enters the outermost audit frame — and does a state-mutating delivery declare its own intents, keyed by handler type? | The transport opens the outermost frame beside its transaction; a delivery declares its own intents and an unregistered handler is refused as an unregistered request is | [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md): bounded ADR-0040 nested-failure supersession; separately enumerated additive UoW read-seam amendment; clarifying ADR-0044/0033 common-owner amendments. G1-dependent production delivery/inbox implementation remains P02b-3 | P02b-1, P02b-3 | Producer/common owner delivered; delivery/inbox contract Accepted, implementation pending P02b-3 | +| G7 | For a dead-letter transition: what class, what `OperationType`, what slug, on which transaction, in what order against the state change, with what failure posture — and may a second non-MediatR caller reach `IAuditStore.WriteStandaloneAsync`? | MUST `security-event`; producer `platform.outbox.dead_letter` and reserved P3 `platform.delivery.dead_letter`; fenced terminal transition and audit INSERT on the same short transaction through a narrow fifth write seam; failed audit retries terminalization without publication | Accepted ADR-0046 + bounded ADR-0044 audit-grant replacement, separate additive fifth-writer seam amendment and ADR-0033 same-transaction clarification under this acceptance; `DeclareOffPath` registration and Audit Coverage baseline | P02b-2 (the producer dead-letter row its criteria assert), P02b-3 | Accepted — 2026-10-12; producer terminal audit [implemented](#p02b-2-step-2--durable-transitions-and-terminal-audit-2026-10-12); subscriber terminal audit pending P02b-3 | +| G8 | Does the dispatcher's `BYPASSRLS` use carry a per-invocation `security-event` row, or is it exempt — and on what written terms? | Exempt, on three terms: the bypass is grant-bounded and code-path-confined, dispatch attempts are logged rather than audited, and the dead-letter transitions are the audited events | Accepted ADR-0046 explicitly replaces the routine-dispatch coverage rule and bounds the ADR-0003 grant extension; dated disclosures under this acceptance, not an ADR-0041 correction | P02b-2 | Accepted — 2026-10-12; confined dispatcher exception/census [implemented](#p02b-2-step-4--enforcement-and-closeout-2026-10-12); local closeout complete; [PR handoff](#p02b-2-pr-handoff-2026-10-12) owns CI/review/merge state | | G9 | Which record decides the background-job runtime — its storage under the closed four-role model, the single tenant-context writer, the job frame's transaction owner, the enqueue-site rule *and who may enqueue whose job type*, the retry contract (whose class, slug and posture adopt **G7**'s), the tenantless platform-job class a purge needs, the ADR-0035 classification, the queue grammar, and the package pins with their licence verdict? | One record answering all of it; a job reaches the database only through `ISender`, which leaves ADR-0040's closed setter set at eight | **ADR-0047** + a dated ADR-0003 Amendment for the storage + a dated **ADR-0036 Amendment** (or an erratum) if the job-path writer is not the `JobActivator` its § Rules names in the closed four-caller set, then Standards 05 and Standards 20 | P02b-4 | Open | | G10 | Does the active tenant travel as a scalar `tenant_id` claim, a `memberships` array, or both — and what value does `UserId` hold? | The scalar claim, whose 02b source is an admin-set user attribute; a second UUID-valued `user_id` attribute for the actor, with `sub` staying Keycloak's subject | Phase-doc: [ADR-0036 § What is out of scope](../decisions/0036-tenant-resolution-trusted-inputs.md#what-is-out-of-scope-and-what-is-not) hands the shape to this phase by name. Reconcile `architecture/13`'s array sketch in the same change | P02b-5 | Open | | G11 | Which [ADR-0036 matrix](../decisions/0036-tenant-resolution-trusted-inputs.md#the-reconciliation-matrix) rows resolve in this phase, given that row 10 consults a reader that denies everyone — and what row covers a signature-valid token carrying no `tenant_id` claim on a tenant host? | Either mint `organization_id` scoped to the host's organization, which makes row 10 need no membership read, or declare the organization host's authenticated 404 and test it as the expected outcome | A dated **ADR-0036** amendment plus errata beside § Staging across packets and the 2026-09-02 erratum, both of which assign row 10 to this phase without noting that its `M covers (T, O)` term is denied until Phase 03 — which groups it with rows 7 and 14 rather than with 6 and 9 | P02b-5 | Open | | G12 | Which Keycloak client does the BFF exchange the code with, where does sign-in happen across two tenant hosts, and how do the client scope, the mappers and the seed users reach a Keycloak database that has already consumed the import? Native HTTPS on retained `*.learnstack.local` hosts, with manual hosts/trust setup, is delivered by Phase 02d and re-verified by P02b-0 | A confidential BFF client with one redirect URI per seed host, retaining native HTTPS and `Secure` cookies; and an idempotent reconciler inside `make seed` — a mapper added only to the import reaches no existing workstation | Phase-doc + the realm JSON + `scripts/seed.sh` + `infra/keycloak/README.md`; a new ADR only if the reconciler becomes ADR-0004's `IIdentityProvider` | P02b-0 re-verifies [Phase 02d G32](phase-02d-walking-skeleton.md#the-decision-register), ADR-0053/0054 and ADR-0055; P02b-5 (which client the BFF redeems the code with: the API pins `azp` against it), P02b-6 | Open | | G13 | What is the BFF session — its custody of the refresh token, its store, its idle and absolute lifetimes, its refresh serialisation under concurrent requests, its terminal `invalid_grant` behaviour, its CSRF control and its logout — and which document owns the access-token lifetime? | The `HttpOnly` cookie Security Standards already describes, encrypted by the session adapter, read only by server code, with single-flight refresh and one authority for the TTL | Phase-doc + one reconciled [Security Standards § Authentication](../standards/11-security.md#authentication); a **new ADR** only for a server-side session store | P02b-6 (which document owns the access-token lifetime: the realm export and its reconciler write it), P02b-7 | Open | | G14 | What bounds the request types reachable over HTTP while `AuthorizationBehavior` is `return next()`, and what is the shape of the token-keyed rate-limit stage behind a shared BFF connection? | An enumerated read-only routable set held by a catalogue-registered rule, and a limiter stage after `UseAuthentication` keyed on the validated subject. The early anonymous stage already runs before authentication: P02d-5 delivers 60/min per canonical direct or authenticated-hop visitor IP plus 600/min peer acquisitions/allocations under ADR-0053 and ADR-0054. Exhausted known visitors refuse before peer debit; unknown visitor allocation remains peer-gated. These remain IP/peer quotas, not token quotas; a request they reject never reaches the token stage. Re-verify the [P5 remediation decision](phase-02d-walking-skeleton.md#p02d-5-external-review-remediation-2026-10-09) before choosing authenticated accounting, preserving pre-classification protection against unknown-host abuse | Phase-doc + catalogue registration; a dated ADR-0032 Amendment for a deny-by-default step instead | P02b-5 (the token-keyed stage and the early stage's treatment of validated callers); G34's anonymous-hop policy and ADR-0054's bounded accounting replacement are Accepted and delivered in P02d-5 | Open | -| G15 | What initiates a replay in this phase? | A dispatcher-level operation on the dispatcher's own connection, with the operator surface deferred | Phase-doc; the URL space is its own decision record, per ADR-0036 | — (answered in § The consumer side; recorded so the question travels with the deferral) | Answered in consumer scope; implementation pending | +| G15 | What initiates a replay in this phase? | Targeted internal subscriber replay in P02b-3; producer/operator reset and fencing require a separate Phase 03 contract | Phase-doc; the URL space is its own decision record, per ADR-0036 | — (answered in § The consumer side; recorded so the question travels with the deferral) | Accepted — internal subscriber replay P02b-3; operator contract Phase 03 | | G16 | What is `LS0002`'s rule name, which project trees does it run over, what does it inspect, and what escalates it to Error? | The name and scope recorded the way Amendment 1 recorded `LS0001`, with the scope reaching the assemblies that actually handle tokens | A dated **ADR-0032** amendment + a catalogue entry + an `AnalyzerReleases` row | P02b-8 | Open | -| G17 | What are the label sets for the event and job metrics, is the tenant axis permitted, and is it spelled `tenant` or `tenant_id`? | One note in the document that owns metric names, no raw tenant label on a per-event series, and an oldest-eligible-row age metric separate from the pending count | [Observability Standards § Required Metrics](../standards/10-observability.md#required-metrics) edit | P02b-2 (the first series ships there, and a label added afterwards touches every emitter already written) | Open | -| G18 | May an integration-event payload or a job argument carry personal data, what may `last_error` contain, and do the four new durable stores enter the erasure scope? | Identifiers only, a sanitized and length-bounded failure descriptor, and the stores named in the erasure inventory | [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md) decides the new producer payload/NULL JSON metadata policy and application byte cap; a dated ADR-0038 cross-reference at acceptance. [Data Protection § Right to Erasure](../architecture/23-data-protection.md#right-to-erasure-right-to-be-forgotten), P02b-8 and Phase 11 retain erasure ownership | P02b-1 (the payload rule: the first payload ships there), P02b-2 (the `last_error` rule and its DDL bound), P02b-8 (the erasure inventory) | Payload Accepted; failure descriptors and erasure open (P02b-2/P02b-8) | -| G19 | Where do the sample flow's publisher and consumer live — a fixture pair in the integration assembly, or a real event a shipped module consumes? | A real event between two shipped domain modules, because a test-assembly-only consumer leaves every module-scoped sweep with zero subjects | [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md) and the P02b-1 package; reconcile Tenancy/Customization/Audit specs at acceptance. G1 must settle distinct subscription deduplication and delivery scope before P02b-3 implements the two effects and Seeder handoff. The provisioning publisher must prove enqueue from its transaction announcement while ambient context remains unresolved | P02b-1 | Flow Accepted; subscriptions/Seeder handoff pending P02b-3 after G1 | +| G17 | What are the label sets for the event and job metrics, is the tenant axis permitted, and is it spelled `tenant` or `tenant_id`? | One note in the document that owns metric names, no raw tenant label on a per-event series, and an oldest-eligible-row age metric separate from the pending count | [Observability Standards § Required Metrics](../standards/10-observability.md#required-metrics) edit | P02b-2 (the first series ships there, and a label added afterwards touches every emitter already written) | Event metrics Accepted — 2026-10-12; runtime/guards [implemented](#p02b-2-step-4--enforcement-and-closeout-2026-10-12); local closeout complete; [PR handoff](#p02b-2-pr-handoff-2026-10-12) owns CI/review/merge state; job metrics pending P02b-4 | +| G18 | May an integration-event payload or a job argument carry personal data, what may `last_error` contain, and do the four new durable stores enter the erasure scope? | Identifiers only, a sanitized and length-bounded failure descriptor, and the stores named in the erasure inventory | [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md) decides the new producer payload/NULL JSON metadata policy and application byte cap; a dated ADR-0038 cross-reference under this acceptance. [Data Protection § Right to Erasure](../architecture/23-data-protection.md#right-to-erasure-right-to-be-forgotten), P02b-8 and Phase 11 retain erasure ownership | P02b-1 (the payload rule: the first payload ships there), P02b-2 (the `last_error` rule and its DDL bound), P02b-8 (the erasure inventory) | Payload and failure descriptors Accepted and implemented; erasure inventory open P02b-8 | +| G19 | Where do the sample flow's publisher and consumer live — a fixture pair in the integration assembly, or a real event a shipped module consumes? | A real event between two shipped domain modules, because a test-assembly-only consumer leaves every module-scoped sweep with zero subjects | [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md) and the P02b-1 package; reconcile Tenancy/Customization/Audit specs under this acceptance. ADR-0046 accepts distinct subscription deduplication and envelope delivery scope; P02b-3 implements the two effects and Seeder handoff. The provisioning publisher must prove enqueue from its transaction announcement while ambient context remains unresolved | P02b-1 | Flow and delivery contract Accepted; subscriptions/Seeder handoff pending P02b-3 | | G20 | What does an anonymous request to a **routed non-public** request type on a live tenant host answer — and does the catalogued `Backend_RequiresJwt_OnAllAuthenticatedRoutes` narrow to a surface outside tenant resolution? A second half rides with it: the shipped assertion middleware says that from this phase the refusal code differs by caller, `tenant_mismatch` for an authenticated one and `not_found` for an anonymous one, written by the middleware itself | The matrix wins on a tenant host — [row 2](../decisions/0036-tenant-resolution-trusted-inputs.md#the-reconciliation-matrix) answers 404, byte-identical to an unknown host's, and the 401 rule narrows to the surface where a 401 discloses nothing | The dated **ADR-0036** amendment G11 needs + a catalogue edit to that rule's Asserts line | P02b-5 | Open | ### P02b-0 entry verification (2026-10-12) @@ -1108,11 +1112,968 @@ format, seven ADR lifecycle checks, the changed-document link/anchor sweep and changes. The [PR checks](https://github.com/HodeTech/LearnStack/pull/29/checks) own final-head CI evidence; PR review and merge remain pending. +### P02b-1 merge and closeout (2026-10-12) + +The maintainer merged [PR #29](https://github.com/HodeTech/LearnStack/pull/29) at +2026-10-12 13:47:58 UTC. GitHub records final PR source +`8a97cd2b7c19bf54a1b0b1f1ad75175256417c9c` and squash merge +`b1f6f21cab556ee7ba497dfc0d64e51b8cf1dad8`, whose parent is the previous main +`d37f46ccf0ebbf909f02c81a59d9019af3e9a1ca`. Source and squash merge have the +identical tree `ba6f9b552c17598ec219ec9d2fc04ef5f235eec0`. Development incorporates +the main merge without switching branches or rewriting published history. + +All seven required checks pass in the +[final PR run](https://github.com/HodeTech/LearnStack/actions/runs/38255804801) +for that source; CodeRabbit also reports success. The tested synthetic merge is +`50fe60560b7966a23c14ab359531aadfbf59a442`. Verified downloaded artifacts record: + +- **3,189 backend passes**, zero failures/skips: 1,766 unit, 266 architecture, + four contract, 180 Docker-free integration and 973 Docker integration cases. +- **1,550 guarded frontend passes**: 1,490 web and 60 SDK, zero + failures/skips/todos; frontend lint, typecheck and builds also pass. +- Exactly 27 matching Lighthouse JSON/HTML report pairs across nine URLs, nine + screenshots and matching source/checkout identities and report digests. + Automated accessibility is 1 throughout, CLS is 0 and maximum normalized + initial JS gzip is 106,552 bytes. Owned cleanup passes; timing medians remain + advisory. These are final PR-run results, not a claim about a post-merge run. + +**P02b-1 is closed.** All four steps and verified review fixes are delivered; +their two independent review rounds remain recorded in the +[packet closeout and follow-ups](#p02b-1-packet-closeout-2026-10-12). Earlier dated +delivery/review records retain their historical pre-merge wording. Accepted ADRs +and P02b-0's entry record are unchanged. Phase 02b remains in progress. + +**Next: P02b-2 decision pass.** Reverify G1, G3 ordering, G4, G5 resolver policy, +G7/G8, G17 and G18 failure descriptors before dispatcher implementation. +ADR-0046 remains reserved and undrafted; the decision package and any new ADR +require maintainer approval before implementation. P02b-2 owns the lease/grant +migration, existing-row handling, dispatcher credentials/boot guard, +claim/publish/mark, retry/terminal policy and metrics/health. P02b-3 owns inbox +delivery, the two Customization defaults subscriptions and the Seeder handoff. +Producer rows are durable but do not yet claim delivered consumption. Background +jobs, API authentication, realm reconciliation, BFF sessions and phase-exit +enforcement remain with P02b-4–8 in the [packet table](#packets-and-decision-gates). + +**Documentation-only closeout validation.** All 266 local Release architecture +cases and seven ADR lifecycle/workflow controls pass, zero failures/skips. The +five changed Markdown files resolve 585 relative links and 186 fragments without +errors; added prose wrapping and whitespace checks pass. Prior dated delivery +records and all Accepted ADR files are byte-preserved. Full runtime suites are +not rerun locally for this documentation-only change; the final PR artifacts +above supply their evidence. + +### P02b-2 decision package (2026-10-12) + +**Accepted — 2026-10-12, @cemil.** The revised decision package and ADR-0046 +are approved, including all four implementation steps and their review loop. +The documentation/code audit used `development` +at `b1f6df34b1d0db9dff77b355743d7bf8885947b3`, the P02b-1 merge closeout. +Three independent read-only audits covered delivery, privilege/audit and runtime/ +observability. They ran no tests and changed no files. The accepted package uses +[ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md); earlier +Accepted ADR bodies and prior dated delivery records remain intact. The dated +acceptance record below distinguishes approval from implementation evidence. + +The dated P02b-1 closeouts' "Next" and "reserved and undrafted" descriptions +record the state before this package. This current package and the Active index +replace those historical navigation instructions; the old records remain intact. + +#### Review findings and accepted answers + +The current bus's zero-handler success is unsafe for P1's real provisioning rows: +their consumers do not exist until P3. Separately, the outbox role cannot write +the MUST audit required by terminalization, and cancellation alone cannot bound +a callback that ignores its token. ADR-0046 explicitly resolves these gaps. + +| Gate / concern | Accepted answer | Delivery owner | +|---|---|---| +| G1 subscriber identity and isolation | Shared transport-owned inbox keyed by tenant, stable subscription ID and event ID; separate delivery ledger owns attempts, terminal state and targeted replay | Contract now; implementation P02b-3 | +| G3 ordering | Deterministic eligible selection by availability, occurrence and ID within one processor; no enqueue, commit, retry or cross-process ordering promise | P02b-2 | +| G4 attempts and terminal state | Six committed normal claims; five published retry delays; fresh-token crash recovery; explicit terminal request and audited terminal timestamp; no seventh publish | P02b-2 | +| G5 type and zero-subscriber policy | Declared concrete types/compatible aliases only; invalid/retired/unknown records terminalize; Pending required subscriptions preserve rows; only intentional absence acknowledges without publication | P02b-2; Active durable subscribers P02b-3 | +| G7 terminal audit | Fenced state transition plus MUST security-event on one transaction through `WriteEventTerminalAsync`, with a restricted transition witness binding audit tenant/org/event to the actual UPDATE result; `platform.outbox.dead_letter` now, reserved Tenancy-owned `platform.delivery.dead_letter` when P3's writer ships; failed audit retries terminalization without delivery | P02b-2 producer; P02b-3 subscriber | +| G8 bypass | Explicit routine-dispatch exception, confined credential and exact grants; existing platform-scope entry audit unchanged | P02b-2 | +| G17 metric dimensions | No tenant labels; registry-bounded event types; separate backlog, eligible age, deferred backlog and heartbeat; immutable gauge snapshots | P02b-2 | +| G18 failure storage | Closed descriptor, at most 256 UTF-8 bytes with CHECK; replace free-form legacy descriptors without changing fact/state | P02b-2; erasure inventory P02b-8 | +| Credential topology | API-owned keyed lazy source; exact physical outbox-role checks; boot connectivity; no Seeder/web credential propagation | P02b-2 | +| G6/G19 dependency | Preserve common owner, envelope organization and effective SystemActor; activate both real defaults subscriptions only with durable acceptance and Seeder handoff | P02b-3 | + +The exact normative replacements and additive audit seam are enumerated in +[ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md#bounded-supersession-and-additive-seams). +They are not ADR-0041 errata. At acceptance, append dated disclosures to +ADR-0006/0010/0038, ADR-0003's unchanged role model and extended grant-budget +navigation, the bounded audit-grant replacement disclosure and +separate fifth-writer seam amendment to ADR-0044, and the same-transaction +clarification to ADR-0033. Keep original acceptance blocks, +decisions and earlier amendments byte-preserved. G1's rule changes are recorded +now but their implementation remains Registered for P3. + +#### Runtime bounds and observation + +These approved bounds are not implemented defaults yet. Options reject invalid +or unbounded values at composition; tests can inject time and tighter positive +bounds. Lease ownership always uses PostgreSQL time, never a caller's clock. + +| Setting / policy | Approved initial bound | +|---|---| +| Worker and claim capacity | One worker; one just-in-time normal claim | +| Idle poll | 200 ms | +| SQL operation / complete terminal transaction deadline | 5 s | +| Publish wait deadline | 10 s | +| Lease | 30 s; validate margin for publish plus fenced transition | +| Progress health bound | 45 s without actual completed progress | +| Snapshot refresh | After completed iterations; idle target 5 s; stale at the 45 s health bound | +| Cancellation-ignoring callbacks | At most four retained tasks; block their partitions; unhealthy while any remain; stop new claims at four | +| Shutdown wait | At most 30 s; retain live task-owned resources until completion or process exit | +| Attempts and backoff | Six normal claims total; 1 s, 5 s, 30 s, 5 min, 1 h | +| Durable failure descriptor | Closed vocabulary, at most 256 UTF-8 bytes | + +A publish timeout cancels the callback and persists its fenced failure, or leaves +an unknown outcome for expired-lease recovery if persistence fails. The callback +keeps ownership of its scope until actual completion; its late result cannot +acknowledge the row. Exclude its partition from new local claims until it finishes. +Unrelated partitions can progress within the four-task allowance. Saturation +stops acquisition and reports unhealthy; no unbounded detached-task replacement +or forced managed-task termination is claimed. P3 must also fence its business +transaction before enabling Active production subscribers. + +Its completion gate must reject a canceled/timed-out attempt before COMMIT even +if failure persistence did not replace its token. Cancellation after physical +COMMIT starts remains Indeterminate under ADR-0033; P3 recovers from atomic +business/inbox/ledger state rather than promise rollback at that boundary. + +Pending count includes all unprocessed, nonterminal rows, including delayed, +leased, terminal-requested and manifest-deferred work. Eligible age uses the exact +normal-publication claim predicate: it excludes terminal requests, exhausted +attempts, future availability, live or unrecovered expired claims, and held/locally +blocked partitions. It is zero when none qualify and never negative. Deferred +count exposes manifest-held work. +Heartbeat advances only after actual bounded loop work, including an empty poll, +not from an independent timer. Failed refresh retains the last snapshot and +marks observation stale. Terminal-audit failure is unhealthy, never a fabricated +successful empty queue. `/healthz` remains liveness-only. + +Standards 10 owns the final metric inventory at acceptance. Preserve existing +outbox duration/failure/dead-letter/lease-loss names with registry-bounded +`event_type` where applicable. Add unlabelled +`learnstack_outbox_oldest_eligible_age_seconds`, +`learnstack_outbox_deferred_count` and +`learnstack_outbox_heartbeat_age_seconds`; pending count becomes unlabelled. +Inbox counters remain reserved for P3, and job metrics for P4. Gauge callbacks +perform no SQL. Dispatcher-owned logs, traces and failure descriptors use validated +correlation and bounded outcomes, without exception text, payload, raw stored type +or connection data. Current bus/consumer exception telemetry is a separate P3 +reconciliation obligation before Active production subscriptions; this packet +does not claim that existing transport diagnostics are already minimized. + +#### Four approved implementation steps + +Each step implements and commits on `development`, then runs two fresh independent +review rounds. Verify findings against current code, fix valid findings, validate +and commit fixes before the next round/step. A clean review needs no empty commit. +The loop continues automatically unless a blocker or new maintainer decision +requires input. No branch switch or history rewrite is part of this package. + +1. **Decision reconciliation, schema and confined composition.** Accept ADR-0046 + and the gate parts above before code. Register catalogue obligations; update + canonical Outbox DDL/indexes/grants, metric labels, audit policy, architecture + and glossary. Add Platform-chain lease/terminal/descriptor/identity migration + and Audit-chain INSERT-grant migration. Declare registry/aliases/readiness, + keyed credential validation and physical-role guard. Adapt test-owned host + composition explicitly; production registration stays enabled. Validate exact + privilege sets, forward/down/reapply and invalid legacy input. No handler or + arbitrary type is constructed to discover subscription metadata. +2. **Durable transitions and terminal audit.** Implement parameterized claim, + recovery, success, backoff, terminal-request and atomic terminal-audit paths. + Add the confined fifth store method, off-path catalogue/matrix row and narrow + call-site/transition-witness guards. Prove token/expiry fences, same-worker + reclaim, six-claim crash, + no seventh publish, uncertain completion and audit rollback on real PostgreSQL + using actual outbox/application roles. Intentional absence and Pending holds + must have distinct non-vacuous controls. +3. **Hosted lifecycle and observability.** Activate the API BackgroundService; + implement bounded waits/overruns/shutdown, progress health, safe traces and + snapshot metrics. Adapt the API launcher/demo environment allowlist and its + proofs without forwarding the credential to web/Seeder. Prove production + registration, startup rejection/valid controls, poison-row progress, ignored + cancellation, safe disposal and continued unrelated work within the bound. +4. **Enforcement and closeout.** Complete production-census and planted controls, + full Release regression/format, mandatory Docker proofs and applicable demo/ + frontend tooling checks. Reconcile current-state docs, actual test counts, + catalogue status and seven existing CI checks at the exact PR head. Create the + PR for maintainer review only after every step/review is complete; claim no + merge or P3 delivery from P2 closeout. + +#### Decisive proof obligations + +- Two independent processors contend under live leases; a release-on-read mutant + fails. Expiry allows redelivery; both stale success and stale failure lose the + token fence, including reacquisition by the same worker identity. This is not + a claim of two separate processes or exactly-once publication. +- Retry boundaries use controllable time/database availability, not wall-clock + sleeps. Crash recovery after claim six and audit refusal cannot cause a seventh + publish. Reconcile lost/uncertain terminal COMMIT before retrying so a completed + terminal audit is not duplicated. Terminal, delayed, leased, deferred and due + rows distinguish all queue gauges. +- Audit refusal on a terminal request with fewer than six attempts never makes + that row publishable again. Its pending contribution remains; its contribution + to oldest publish-eligible age is excluded. +- Known V1 payloads remain readable after declared deployment aliases. Reverse + commits, equal occurrence times and retry overtaking falsify stronger ordering. + Unknown/retired/malformed/mismatched rows do not trigger arbitrary type loading. +- Real TenantCreated rows survive repeated Pending polls without attempts or + constructions. P3's activation proof must later deliver them to both stable IDs. +- Missing/malformed/wrong-user credentials refuse without secret text; valid + configuration is lazy; real physical guards reject elevated/transitive roles. + Audit INSERT is allowed; audit read/update/delete and business access are not. +- Against the real bypass role, replace an audit draft's tenant, organization or + event while retaining the successful fenced transition's restricted witness. + Each mismatch must leave neither terminal state nor audit row committed. Prove + a matching organization-scoped success and rejection of stale/wrong-transaction + witnesses; GUC/RLS assertions cannot substitute for these controls. +- Cancellation-ignoring operations retain safe resource ownership, cannot write + late acknowledgements and cannot grow beyond the approved bound. A blocked or + saturated loop becomes unhealthy while `/healthz` still returns 200. +- Before P3 activates subscribers, its proof must include failed timeout + persistence followed by late business completion and timeout during physical + COMMIT on the final subscriber claim; atomic ledger/inbox state decides recovery. +- Structural censuses include production subjects and planted clean/dirty cases + for each key's resolution, publication, type loading and write confinement. + Behavioural controls mutate lease/terminal/audit safeguards to prove sensitivity. + +The P02b-2 publication proof uses the shipped BackgroundService, registry and +transition/audit stores on real PostgreSQL and actual roles. Keep real TenantCreated +subscriptions Pending. Only isolated test composition supplies an Active synthetic +contract and controlled transport/handler to exercise publish/mark, failures, +recovery and deadlines; intentional absence is a separate control. Assert nonempty +production machinery, without inventing a production subscriber. P3 owns handler +and durable-acceptance censuses when those production subjects exist. + +#### Scope retained by later owners + +P3 owns shared Inbox/delivery DDL and RLS, per-subscription business/audit frames, +retries, terminal audit and targeted replay, both Customization defaults and +Seeder identity handoff. That boundary is required by the packet dependency, +not an unowned deferral. The companion Hub roadmap's per-module inbox reference +needs a coordinated dependency-pointer update before P3 closeout; Hub remains +roadmap owner and this package adds no Hub endpoint or runtime work. P4 owns jobs; +Phase 03 owns operator routes/permissions; P02b-8 owns erasure inventory; Phase 11 +owns brokers, cross-process evidence, purge and readiness exposure. + +#### Proposal review and validation (2026-10-12) + +At proposal commit `cc5de26`, two independent review rounds completed before +maintainer approval. Round 1 used +separate GPT-6.1-sol xhigh delivery and security/corpus reviewers. Verified fixes +separate the audit-grant replacement from the additive fifth seam, allow safe +terminal auditing of malformed causal metadata, scope diagnostic minimization to +the dispatcher, and require atomic subscriber ledger completion plus a sticky +timeout gate. Cancellation after physical COMMIT starts remains Indeterminate. +Both reviewers approved the corrected proposal. + +Round 2 used fresh GPT-6.1-sol xhigh delivery and GPT-6-astra high corpus/privilege +reviewers. Both approved without actionable findings. Reviews were read-only; +they did not run runtime suites or claim implementation evidence. + +Documentation-only validation passes: 266 local Release architecture cases, +seven ADR lifecycle/workflow controls, 329 local links and 119 fragments across +the four changed Markdown files, prose wrapping and `git diff --check`. Prior +P02b-0/P02b-1 dated delivery records and all Accepted ADR files are byte-preserved. +Full runtime/Docker suites remain implementation-step obligations; no dispatcher, +inbox or gate delivery is claimed by this proposal. + +#### External proposal review reconciliation (2026-10-12) + +Two external reports were checked against the current draft, accepted contracts, +canonical DDL and production code. Their reported line numbers are not authority. +This revision remains Proposed and changes no runtime or Accepted ADR body. + +| Finding | Verified disposition | +|---|---| +| Erasure assigned to Phase 11 | Already corrected in `cc5de26`: privileged redaction and its bounded caller contract belong to Phase 03; only retention purge belongs to Phase 11 | +| Two subscription IDs conflict with `card@1` / `plain@1` | Rejected: `BuiltInCustomizations.Card` is a `BuiltInContentType`; `Plain` is a `BuiltInTaxonomy`. ADR-0057 freezes those two independent effects, not two content types. The ADR now maps each ID explicitly | +| Bypass audit INSERT lacks row-bound identity proof | Valid: the seam now receives a restricted witness from the actual fenced UPDATE result; draft tenant/org/event/subscription must match it. Forged-identity and matching-success proofs are explicit | +| Terminal-request normal eligibility and undefined release | Explicit predicates now exclude terminal requests/exhausted attempts; recovery is separate. Pending holds precede payload validation, even for malformed payloads. Removed the unexplained release transition; success/retry/terminal paths clear leases | +| Inbox completion, identity grammar and replay ambiguity | Clarified uncommitted claim and atomic completion/rollback; completed inbox rows never reset; subscription IDs version deduplication semantics | +| Grant locator, legacy batch/exactly-once/DLQ designs and metric/off-path carriers | Named exact authorities and acceptance replacements. ADR-0003's role model remains unchanged; Database Standards owns actual grants. No competing current dead-letter design remains after acceptance | +| Envelope storage, narrow P3 SELECT, subscriber audit owner and P2 proof subjects | P3 must name durable envelope copies/references before DDL and inventory them for erasure; scheduler cannot read payload; subscriber audit is Tenancy-owned. P2 exercises shipped machinery with isolated test contracts while real subscriptions stay Pending | +| Final acceptance/handler failure and roadmap target checks | Failed/unknown durable acceptance consumes producer attempts; only accepted deliveries isolate business retry budgets. The phase's own undated zero-subscriber/replay sketches are now named acceptance replacements | +| Missing references and historical Next navigation | Added applicable ADR links and the ADR-0047 reservation link; added current navigation without rewriting dated P1 records. ADR-0045 is not used by this ADR and needs no artificial reference | + +The internal executor performs delivery, not publication; it receives no exception +to the outbox-only publisher rule. SecurityEvent records a consequential bypass +terminal transition, not routine polling. The selected order receives matching +normal-claim and recovery/terminal indexes. These are contract clarifications, +not claims that the proposed mechanisms have shipped. + +The revision received read-only GPT-6-astra high audit/privilege verification and +a fresh GPT-6-sol xhigh delivery/corpus review. The latter identified the Pending +validation priority, acceptance-failure budget and phase-carrier gaps recorded +above. Both reviewers approved after the verified fixes; neither claimed runtime +execution evidence. + +Revision validation passes: 266 Release architecture cases, seven ADR workflow +controls, 337 local links and 124 fragments across the four proposal Markdown +files, prose wrapping and `git diff --check`. Accepted ADR files and dated +P02b-0/P02b-1 delivery records remain byte-preserved. Only this package and the +Proposed ADR changed in the revision. Maintainer approval still precedes runtime +implementation and acceptance-carrier edits. + +### P02b-2 acceptance (2026-10-12) + +**Accepted by @cemil.** The maintainer approves the revised ADR-0046 and this +decision package, and authorizes all four implementation steps on development +with two fresh independent review rounds per step. The accepted decision is +committed before runtime edits. G1, G3's ordering, G4, G5's resolver/readiness, +G7, G8, G17's event metrics and G18's failure-descriptor parts are Accepted. +P02b-3 still owns durable acceptance, inbox/ledger DDL, independent subscriber +execution/retries, both real default subscriptions and Seeder handoff; job and +erasure gates retain their later owners. + +Acceptance appends bounded supersession disclosures and additive seams to the +named Accepted ADRs, reconciles the current standards/architecture/glossary/skill +carriers, and registers six catalogue obligations. Original ADR acceptance +metadata, decisions and prior amendments remain intact. Dated P02b-0/P02b-1 +records remain byte-preserved. Registered rules and planned audit operations +claim no runtime enforcement; delivery evidence belongs to each step below. + +### P02b-2 Step 1 — Decision, schema and confined composition (2026-10-12) + +**Implemented; both independent review rounds and fresh fix review complete.** +Decision commit +`192810b` precedes runtime edits. Step 1 adds the Platform-chain migration +`20261012180355_add_outbox_dispatch_state` and Audit-chain migration +`20261012180433_grant_outbox_terminal_audit`. Existing migration bodies remain +unchanged. New state/identity/UTF-8 CHECKs and eligibility/recovery/terminal +indexes accompany exactly eight outbox status-column UPDATE privileges and +INSERT-only audit access. Legacy diagnostic replacement spans both tenants; +invalid legacy identities reject the migration and roll back its owner-only +backfill. Disposable-schema down/reapply preserves event facts and retains +redaction; live rollback requires a forward fix. + +The API registers a keyed, lazy, DI-owned outbox source with exact configured +username checks and physical session/current-role, attribute and membership +checks. Missing configuration fails upon resolution; Step 3's mandatory worker +startup will resolve it and prove connectivity. Seeder/web receive no new source. +The immutable contract/alias/readiness registry constructs no handlers. Real +TenantCreated has its two distinct Pending subscriptions; Active, intentional +absence, unknown and retired identities have separate controls. The two existing +connection-resolution censuses explicitly admit the approved dispatcher builder +and composition root, without wildcard exemptions. + +**Verification before review:** `CI=true` Release build has zero warnings/errors. +TRX counters and process exit codes record zero failed and zero skipped cases: + +| Suite | Passed | Scope | +|---|---:|---| +| Unit | 1,800 | Complete unit suite, including registry/composition/configuration controls | +| Architecture | 266 | Complete non-skippable suite | +| Contract | 4 | Served OpenAPI/public SDK contract suite | +| Integration, no Docker | 180 | Complete HTTP suite | +| Integration, Docker | 128 | Migration/role/schema/grant/rollback classes affected by Step 1 | + +The Docker slice includes `OutboxDispatchMigrationTests`, +`OutboxDispatcherRoleGuardTests`, `PlatformSchemaTests`, `TenancySchemaTests`, +`AuditSchemaTests` and `MigrationRollbackTests`. Role-mutation proofs own a +separate container; destructive schema probes own disposable databases. These +counts describe this step's completed checks, not a full Docker regression. +The six new catalogue rules remain Registered: transition/audit/worker proof +lands in later steps. No dispatcher execution or P02b-3 delivery is claimed. + +#### Step 1 review round 1 + +Fresh read-only GPT-6-astra xhigh security/schema and GPT-6.1-sol xhigh +corpus/test reviewers examined `192810b..d9bc77b`. Security review found no +actionable defect. Corpus review found a stale handler-owned inbox instruction +in the code-review skill and an unqualified historical per-module inbox note in +Architecture 10. Both were verified and reconciled to ADR-0046's transport-owned +P02b-3 target. Original dated notes remain intact. No runtime change was needed; +the existing test evidence above remains applicable. The correction passes local +file/fragment checks, prose wrapping and `git diff --check`. + +#### Step 1 review round 2 + +Fresh GPT-6.1-sol xhigh security/composition and GPT-6-astra high corpus/proof +reviewers examined `192810b..6a13455`. Corpus review approved without findings. +The composition review verified that an open generic Active handler could match +the discovery census despite being unconstructible. Active admission now requires +a closed type. A regression proves discovery sees the open generic offender, +refuses its matching manifest and accepts a closed generic control. The complete +Release unit suite passes 1,801 cases; the complete architecture suite passes +266, both with zero failures/skips. A fresh focused review checks this correction +before Step 2 begins. Removing the closed-type check makes the new regression +fail; restoring it makes all 17 registry cases pass. No migration, grant or +Pending policy changed. + +Fresh GPT-6.1-sol high focused review of `6a13455..b6fc714` approved the correction +without findings and independently checked its accepting/rejecting controls and +saved TRX counters. The completed Step 1 commits are `d9bc77b`, `6a13455` and +`b6fc714`; the next authorized implementation step is durable transitions and +terminal audit. + +### P02b-2 Step 2 — Durable transitions and terminal audit (2026-10-12) + +**Implemented; both independent review rounds complete.** `OutboxDispatchStore` uses +short parameterized database-time transitions. Normal acquisition commits one +fresh token and one attempt before returning; expired ownership requires a fresh +recovery token without another attempt. Success, failure, terminal request and +terminalization require the current token and an unexpired lease. Five backoffs +follow six counted normal claims; exhausted or requested work never reenters +normal publication. Pending identities and aliases are excluded before payload +validation in every acquisition path. Local blocked partitions receive the same +exclusion. + +`OutboxEnvelopeReader` uses only the registry's concrete type and frozen options. +It compares persisted fact/routing/time, tolerates PostgreSQL microsecond +truncation and checks the canonical reserialization byte cap, rather than jsonb +whitespace. Complete envelope admission precedes intentional acknowledgement; +Pending, unknown, retired and invalid outcomes remain distinct. + +The fifth `IAuditStore` method preserves the existing four writes. Only +`OutboxDispatchStore.TryTerminalizeAsync` derives the minting capability and calls +it. The private immutable witness reads all identity and causal values from the +actual fenced UPDATE result, including the old acquisition token. The real audit +store requires the declared producer MUST/SecurityEvent, matching draft identity +and the same live connection/transaction. Both writes commit together. Invalid +causal values are omitted; invalid correlation uses a validated attempt trace. +No audit SQL is added outside `PostgresAuditStore`. Tenancy registers the producer +operation; the subscriber operation remains planned for P02b-3. + +Terminal SQL plus audit share one five-second deadline. A lost COMMIT response +triggers a separate bounded durable-state read before returning/retrying; it +cannot insert a second audit into a completed terminal row. A test-owned TCP +proxy drops the real PostgreSQL COMMIT acknowledgement after physical commit, +proves reconnection and checks one durable terminal/audit pair. Production has no +fault hook. Failed audit retains the terminal request, including below attempt +six, and never permits another publication. + +Tests use actual `learnstack_app` inserts with announced tenant/org and the +guarded `learnstack_outbox_admin` source for transitions. Disposable databases +isolate grant and state mutations. Nonempty whole-production Cecil censuses and +clean/dirty controls cover witness shape, minting/fifth-write callers, async and +delegate helpers, required catalogue injection and named dynamic type-loading +APIs. This is a bounded IL capability check, not arbitrary reflection/dataflow +analysis. SharedKernel has no friend assembly. + +Four behavioural mutation controls each fail as intended: removing lease expiry, +admitting terminal requests for publication, omitting MUST audit and discarding +successful uncertain-COMMIT reconciliation. Every edit is restored before the +clean validation run. Six-attempt tests start at zero and commit all six claims; +they cover both an explicit sixth failure and an expired sixth claim. Snapshot +proofs distinguish due, future, leased, expired, requested, exhausted, deferred, +locally blocked, processed and completed-terminal rows. + +The four transition/audit/type catalogue rules are Implemented. Hosted lifecycle, +publication/overrun proofs, credential caller enforcement and observation remain +Steps 3–4. Real default subscriptions stay Pending; no P02b-3 business effect, +inbox or delivery ledger is claimed. + +**Verification before review:** Release solution build has zero warnings/errors; +formatting covers all 21 changed C# files. TRX counters and process exit codes +record zero failures/skips: complete unit 1,825, complete architecture 273, +contract 4, complete Docker-free integration 180 and affected Docker integration +110. The Docker slice covers `OutboxDispatchStoreTests`, `AuditStoreTests`, +`PlatformAdminScopeTests`, `TenantAssertionAuditTests` and +`PublicReadTransactionTests`; it is not a full Docker-suite claim. All four +behavioural mutants fail their named tests, then the restored code passes. +Local file/fragment, historical-record/Accepted-ADR preservation, prose-width and +`git diff --check` controls pass. + +#### Step 2 review round 1 + +Fresh read-only GPT-6-astra xhigh security/concurrency and GPT-6.1-sol xhigh +proof/corpus reviewers examined `c6ceaa9..7976113`. The proof review approved; +its independent rerun passed 273 architecture and 24 focused unit cases. The +security review reproduced a stored-routing gap: deserialization regenerates +get-only Topic/PartitionKey values, masking changed or absent payload members. +The reader now checks those required string members in the original JSON before +using the declared CLR contract, then compares the reconstructed routing with +the row. Ten unit mutation cases and four real-database cases cover changed, +missing and malformed routing, terminal-only progression and minimized audit. +The unchanged payload remains an admitted control. Removing the original-field +check makes all ten new unit cases fail. After restoration, the complete unit +suite passes 1,835 cases, architecture passes 273 and the four new Docker cases +pass, all with zero failures/skips. Round 2 follows the verified correction; +hosted lifecycle remains Step 3. + +#### Step 2 review round 2 + +Fresh read-only GPT-6.1-sol xhigh security/concurrency and GPT-6-astra high +proof/corpus reviewers examined `c6ceaa9..ffce05f`. Both approved without verified +findings. Independent Release no-build reruns passed 34 envelope/options unit +cases, 19 dispatch/audit architecture cases and all 53 dispatch-store Docker +cases, with zero failures/skips. The proof reviewer independently passed all seven +dispatch capability facts and checked the real COMMIT-response-loss result in +the restored database receipt. Both inspected the actual row-bound audit and +routing correction rather than accepting review notes as evidence. Step 2 is +closed; Step 3 may activate the mandatory worker and observation. No P3 delivery +or inbox effect is claimed. + +### P02b-2 Step 3 — Hosted lifecycle and observability (2026-10-12) + +**Complete; both independent review rounds passed.** The API owns +its mandatory `OutboxProcessor` BackgroundService, guarded dispatcher-source +startup resolution/connectivity, bounded publication lifecycle and retained +overrun ownership. Observation supplies immutable gauge snapshots, bounded +producer diagnostics and the registered, unmapped `outbox` health check. +`/healthz` remains liveness-only. Paired API launchers supply application and +confined dispatcher credentials; every Seeder invocation uses an explicit +application-only environment, and web receives neither database credential. + +**Verification before review:** the Release solution build reports zero +warnings/errors. TRX counters record zero failures/skips in the complete unit +suite (1,845), contract suite (4), Docker-free integration suite (182), restored +architecture suite (273), and the actual-host PostgreSQL `OutboxProcessorTests` +slice (16). Worker helper/observation unit cases are included in the unit total. +The initial architecture run exposed the newly registered worker in the global +keyed-resolution allowlist; the exact outbox-key allowance is corrected and the +restored 273-case run passes. This is a focused Docker-runtime receipt, not a +full Docker-suite claim. All three actual API-launcher cases also pass after +correcting the environment reader to accept the explicit dispatcher credential. +The two affected frontend files pass 116 tests; web lint and typecheck pass. +Whitespace verification passes for all 39 changed C# files; affected TypeScript +and runtime-script Prettier checks, script syntax, diff whitespace and 2,172 +relative links/396 fragments pass. Historical producer delivery records and +Accepted ADR bodies remain byte-preserved. + +`OutboxProcessorTests` covers actual registration/guarded boot, invalid credential +refusal, real provisioning Pending holds, poison/ordinary failure progression, +explicit intentional absence, retained callback/cleanup ownership, four-task +saturation and finite shutdown. Observation tests cover monotonic stale health, +failed-refresh retention, terminal-audit failure, finite metric labels and fresh +trace identity, with bounded canonical correlation reconstructed from parsed +identity rather than optional traceparent extensions. Two independent actual API +workers prove no second publication/claim while the first live lease is held. +Never-blocks, live-ownership and Pending catalogue rules are Implemented. The full +health production census/planted controls remain Step 4; that rule is Registered. + +Two behavioural mutations fail in the intended executed test: removing local +partition admission permits a sibling claim before retained cleanup completes; +removing live-lease admission permits a competing worker to claim the held row. +Both source files are byte-restored before the clean build/unit/worker rerun. +Step 3 completed both independent rounds and the verified corrections below. +Steps 1–2 completed both rounds in their +records above. Step 4 owns full credential/publication censuses, catalogue +enforcement and packet closeout. P02b-3 still owns durable acceptance, shared +inbox/ledger, independent subscriber retries, both real default subscriptions and +Seeder handoff. Real required subscriptions remain Pending; no consumer business +effect is claimed. + +#### Step 3 review round 1 + +Fresh read-only GPT-6-astra xhigh security/lifecycle and GPT-6.1-sol xhigh +proof/corpus reviewers examined `d4c06a46..756c8523`. Security approved without +verified findings. The proof reviewer found that the demo's pre-existing +`--validate-demo-config` invocation forwarded all four credentials to Seeder, +contradicting the new confinement claim. Validation now runs inside the launcher +with a closed local connection grammar; the obsolete Seeder mode is removed. +Manifest and convergence modes share the same explicit child allowlist, and +standalone `make seed` uses it too, excluding ambient aliases and role passwords. +Stale role/password/database/host/port and malformed configuration controls move +with their owner instead of weakening the check. The complete Docker integration +regression at the implementation commit passes 1,083 cases with zero failures or +skips. This supplements the focused pre-review receipt above; final packet +verification will rerun the changed tooling and actual PR-head CI. +The correction's Release build has zero warnings/errors; the Docker-free +integration run passes 175 cases after seven obsolete parser cases moved to the +launcher suite. The guarded complete frontend run passes 1,608 cases (1,548 web +and 60 SDK), with zero skips/todos; 91 demo lifecycle cases include all four-role +configuration and actual shell/child-environment controls. Typecheck, affected +formatting, script syntax, diff whitespace and the delivery-record link check +pass. These are correction receipts, not a new full-Docker rerun. + +#### Step 3 review round 2 + +Fresh read-only GPT-6.1-sol xhigh security/lifecycle and GPT-6-astra xhigh +proof/corpus reviewers examined `d4c06a46..da637e35`. Neither found a Blocker or +Major. Security identified one operability gap: the safe startup refusal omitted +the configuration key/form required by the accepted completion criterion. The +fixed message now names `ConnectionStrings:OutboxDispatcher`, Npgsql key/value +syntax and `learnstack_outbox_admin`, without provider or supplied-value text. +The real-host negative theory asserts that guidance for missing configuration; +all four credential cases pass after the correction. Independent review reruns +passed 91 demo lifecycle cases, 11 focused outbox unit cases and two construction +host cases; the security reviewer separately passed nine helper/observation cases. +Both inspected the prior receipts and production paths. They did not claim a new +full build or Docker rerun. Step 3 is closed; Step 4 may add the complete +credential/publication/health censuses and close the packet. + +### P02b-2 Step 4 — Enforcement and closeout (2026-10-12) + +**Implementation and local closeout complete; both independent review rounds +passed.** Current CI/review/merge state belongs to the +[PR handoff](#p02b-2-pr-handoff-2026-10-12). +Steps 1–3 completed both independent review rounds; Step 3 is closed at `02540d0`. +Step 4 adds nonempty whole-production credential/publication censuses, exact +keyed factories and consumer bindings, and planted helper/delegate/async controls. +Only the processor constructs the task-owned publication attempt; its private +callback is the sole event-bus publication path. The existing canonical module +bus rule now also enforces that complete production fence. + +The five-subject observation graph follows async and production-owned +polymorphic contracts, rejects persistence/service-locator capabilities, and +inspects actual gauge callbacks. Actual DI controls reject missing/duplicate +workers/checks and wrong factories. Round 2 fixes at `606362b` pin exact credential +reader signatures, the external System clock and actual health/observation binding. +All 26 focused/corpus cases and both fresh fix reviews pass. +The literal-SQL census admits only exact producer enqueue, dispatcher +status-transition and migration backfill methods; quoted/schema/case/CTE and +missing/extra-writer controls prove its reach. These are bounded IL/literal-SQL +checks, not arbitrary reflection or dynamic-SQL dataflow analysis. + +**Verification before review:** the full Release solution build reports zero +warnings/errors, and solution-wide `dotnet format --verify-no-changes --no-restore` +passes. All 21 focused census/publication cases pass with zero skips, +including compiled actual production subjects and every planted control. Role +and Health catalogue rules are Implemented; Retry also names its canonical +architecture method. The source/catalogue recount is 193 architecture test +methods, 192 Implemented rules, 118 in that assembly and 74 elsewhere. + +The full Release backend regression passes 3,393 cases with zero failures/skips: +1,845 unit, 286 architecture, four contract, 175 Docker-free integration and +1,083 Docker integration. TRX counters were inspected directly; the nonempty-run +guard passes. The three additional case-insensitive credential-alias controls +also pass in the focused seven-case run. Step 3's corrected launcher/frontend +receipt remains the applicable local evidence: 91 lifecycle cases and 1,608 +frontend cases. These checks are not claimed as new Step 4 frontend reruns. +The full PR-range documentation check passes 2,555 relative links/459 fragments, +and all seven ADR-status script tests pass. Historical records and Accepted ADR +text remain preserved. All four implementation steps and both independent review +rounds are locally complete; post-fix verification is recorded below. The +[PR handoff](#p02b-2-pr-handoff-2026-10-12) owns current CI/review/merge state. + +P02b-3 still owns durable acceptance, shared inbox/ledger, independent subscriber +retries, both real default subscriptions and Seeder handoff. Required production +subscriptions remain Pending; test-owned Active contracts prove producer runtime +without claiming consumer business effects. + +#### Step 4 review round 1 + +Fresh read-only GPT-6-astra xhigh boundary and GPT-6.1-sol xhigh proof/corpus +reviewers examined `02540d05..d2531f2`. They reproduced three structural test gaps, +without finding a current production dispatcher defect. Captured key/provider +fields could be reassigned after their original capture; valid comment-separated +SQL and a later `TRUNCATE` target escaped the literal matcher; an additional +factory-based hosted worker escaped the descriptor type filter. + +The fixes close all three: captured-field mutations are restricted to the exact +factory parameter assignment, SQL recognizes those literal token separators and +all listed truncate targets, and the actual hosted descriptor set includes only +the singleton dispatcher and framework health publisher. Added dirty controls +cover key/provider writes in the factory, callback and helper, field/argument +address escapes and argument reassignment, SQL variants, and factory/instance +worker duplicates. No test method or catalogue count is added. All 21 focused +cases pass after the three corrections; the final seven boundary cases pass +again after the adjacent argument-mutation controls, with zero failures/skips. +The full regression above predates these test-only changes. The proof reviewer +independently passed 26 existing focused/corpus cases, with zero failures/skips; +neither reviewer claimed a new full suite. Round 2 remains pending. + +#### Step 4 review round 2 + +Fresh read-only GPT-6.1-sol xhigh boundary and GPT-6-astra xhigh proof reviewers +examined `02540d05..2f8693e`. They verified three test-guard gaps without finding +a current production defect. Named worker/store source-reader overloads could +self-authorize; an external `TimeProvider` subclass was invisible to the +production-owned contract graph and DI checks; a same-type health factory detached +from the singleton observation passed the returned-type check. + +The fixes are delivered at `606362b`: pin unique literal source-reader method +signatures, visibility and staticness with overload controls; require the actual +`TimeProvider.System` singleton instance with missing/duplicate/alternate-instance +and factory controls; and exercise the actual DI-registered health check through +Healthy → Unhealthy → Healthy against the singleton observation, with a detached +factory dirty control. The polymorphic graph covers production-owned contracts; +the external clock is admitted only through the explicit System-instance pin. +No test method, catalogue rule or published count is added. + +Post-round-1 verification at `2f8693e` passes the full 286-case architecture suite +and five Corpus cases; the Release build has zero warnings/errors. After the +round-2 fixes at `606362b`, all 26 focused/corpus cases pass with zero +failures/skips. Fresh GPT-6.1-sol xhigh boundary and GPT-6-astra xhigh observation +fix reviewers both approve with no remaining verified findings. The boundary +reviewer independently rejects both named-method overload plants (two offenders +each), observes zero clean/restored offenders and passes three focused Release +cases. The observation reviewer independently detects the SQL-capable external +clock and detached same-type health factory, observes zero restored graph +offenders and passes two focused Release cases. Neither claims a new full suite. + +The full post-fix Release build has zero warnings/errors, all 286 architecture +cases pass with zero failures/skips, and solution-wide format verification +passes. The final local documentation check passes 2,575 relative links and +479 fragments with zero errors. The 3,393-case regression above predates these +test-only guard fixes; its +TRX counters and the 26-case focused/corpus counters were checked directly. No +test method, catalogue rule or published count changes. All four P02b-2 +implementation steps and both independent review rounds are locally complete. +PR-head CI and merge remain pending; this closes producer dispatch/enforcement +without claiming any P02b-3 consumer delivery. + +### P02b-2 PR handoff (2026-10-12) + +[PR #30](https://github.com/HodeTech/LearnStack/pull/30) opens development → main +from local closeout `7bf14a0`. All four implementation steps and both independent +review rounds per step are complete. The PR's checks and current-head evidence +in its description own live CI state; its review and merge state remain on the +same PR. Earlier pending statements in dated local receipts are as-of records, +not the current check verdict. No merge is claimed by opening the PR. + +The seven existing required contexts and strict up-to-date protection were read +back from GitHub unchanged. Final verification must use the PR's actual current +head, not an earlier passing run. The PR description records run URLs and suite +counters after that verification. Five final local corpus checks pass with zero +failures/skips; this is not a CI result. + +P02b-3 is next after P02b-2 review/merge and its own decision pass: shared inbox +and delivery ledger, durable acceptance, independent subscriber retries/audit, +both real defaults subscriptions and Seeder handoff. No consumer delivery, +background-job or authentication completion is claimed here. + +### P02b-2 PR review follow-up (2026-10-13) + +All seven required jobs pass on PR head `a80e06d`, including the complete +3,393-case backend regression and 1,608 frontend cases, with zero failures/skips. +[Run 38306262165](https://github.com/HodeTech/LearnStack/actions/runs/38306262165) +also supplies 27 matching Lighthouse JSON/HTML report pairs and nine screenshots. +All reports have zero CLS, accessibility score 1 and normalized gzip JavaScript +between 106,359 and 106,553 bytes. Proof finishes in 646.454 seconds; owned process, +Compose, private-file, environment and lock cleanup all pass. These are receipts +for that head, not evidence for a later correction. + +Automated PR review identified a local-launcher validation defect: omitting both +a role's connection-string password and its selected password variable compared +two absent values as equal. Independent reproduction accepts that invalid state +for all four roles. Four new regression cases fail against the previous validator +and pass after requiring a nonempty configured string password. Existing quoted +password, whitespace and alias controls remain green; diagnostics remain fixed +and contain no supplied configuration. No runtime dispatcher or API contract +changes. + +Local verification passes 95 lifecycle cases and the guarded frontend suite's +1,612 cases (1,552 web and 60 SDK), with zero failures/skips/todos; lifecycle cases +are a subset, not an additional suite count. Workspace lint/typecheck pass. +Fresh sequential GPT-6.1-sol high and GPT-6-sol high correction reviewers both +approve `a80e06d..a851b7c` with no findings. Each independently reproduces the four +parent/current differences and passes all 95 lifecycle cases. Five Corpus cases +and seven ADR-status script tests also pass; the final link check covers 2,586 +relative links and 490 fragments with zero errors. Final correction-head CI +remains pending in this local receipt. +The [PR handoff](#p02b-2-pr-handoff-2026-10-12) continues to own live PR state. + +### P02b-2 external review remediation (2026-10-13) + +Both external reports against `44089707` were treated as review evidence and +checked against current code and Accepted ADR-0046. No new decision, consumer +delivery or grant change is part of this correction. Existing migrations and +Accepted ADR bodies remain unchanged. + +- Recovery now states its index predicate explicitly, without relying on + PostgreSQL to infer the lease CHECK. A new Platform migration replaces only + the recovery index: both expired leases and unleased exhausted rows retain + `(available_after, occurred_at, id)` order. Real migrated-database plans with + 50,000 Pending rows distinguish the predecessor's scan from indexed recovery; + admission exclusions, ordering and down/reapply are asserted. +- A failed durability probe after uncertain terminal COMMIT preserves both + exceptions, in COMMIT/probe order. The real response-dropping proxy exercises + that failure and independently verifies one durable terminal/audit pair and + no duplicate on retry. The unused persisted shutdown outcome was removed; + shutdown continues to leave ownership for expiry recovery. +- The launcher exposes only the exact fixed local database diagnostic, with + secret-bearing prefix/unknown-message controls. Absent configuration uses the + same diagnostic. Root runtime-script formatting is restored. The audit marker + count, transport-owned inbox wording, historical batch discussion, catalogue + proof kind, glossary and later-phase navigation now match implementation. +- The generic iteration diagnostic no longer mislabels every exception as a + storage failure. Normal dispatch avoids its unused registry lookup. Contention + warms both independent hosts before its live-lease deadline. Shutdown advances + exactly one controlled deadline with four retained callbacks; an isolated + second-budget mutant fails, and the rebuilt clean control passes. + +The remaining report suggestions were checked rather than adopted as new policy: + +| Concern | Disposition | +|---|---| +| Final `44089707` CI remains outstanding | Already resolved: all seven required jobs pass in [run 38309098697](https://github.com/HodeTech/LearnStack/actions/runs/38309098697). The earlier local receipt is dated evidence, not live state; this correction needs its own head verification | +| Failure metric precedes a successful fence; gauges may straddle a refresh | Metrics observe attempts, not durable transitions. Every gauge reads an immutable snapshot; no scrape-wide atomicity is promised. The phase criterion below now names that exact invariant | +| Disabled test worker leaves outbox health unhealthy | Intentional and now documented in factory remarks. The private check reports the absent worker; static `/healthz` remains liveness-only | +| Invalid options are reported as credential failures | Not on the shipped DI path: the store validates the shared options before worker construction | +| Alternate worker registration, duplicate short type labels, permissive terminal-request kind | No current production violation: composition census pins the sole worker; the registry has one production contract; the worker requests terminal state only from Normal claims. Recovery is deliberately used by a store fixture | +| Witness acquisition is not compared with audit metadata | SQL fences the actual token and returns it into the transaction-bound witness. The audit draft has no acquisition field; adding one is not required by ADR-0046 | +| Invalid causal metadata changes the original correlation | Matches ADR-0046: invalid correlation/causation/actor uses a validated attempt trace and omits invalid causal values | +| Role guard should reject other principals inheriting dispatcher rights | The guard validates the dispatcher's effective role and escalating memberships. Reverse grant auditing is outside that connection contract; changing it would not sandbox compromised privileged SQL | +| Ambient local credentials, migration-child environment, quoted passwords | Ambient overrides are explicit existing loader policy and tested. The administrative migration child is distinct from the confined web/Seeder children. Single/double-quote parsing is the declared closed local grammar, with existing controls; no provider mismatch was demonstrated | +| Audit seam needs duplicate tests in another file; sub-microsecond and census optimization suggestions | The real transition tests already exercise the fifth seam and rollback substitutions. Quantization has explicit envelope controls. Duplicate tests or census caching do not repair a demonstrated contract violation | +| ADR navigation date, test placement and optional trace descriptor | Historical ADR navigation is preserved; local-loader tests sit with that loader. Standard 09 and ADR-0046 allow optional trace data without requiring its persistence | + +Local validation passes 1,845 unit, 286 architecture, four contract and 175 +Docker-free integration cases, plus 97 focused Docker cases (54 store, 27 +migration, 16 worker), with zero failures/skips. Guarded frontend passes 1,616 +cases (1,556 web, 60 SDK), with zero failures/skips/todos; lint/typecheck pass. +Release build reports zero warnings/errors; format, seven ADR-status script +controls, 2,587 relative links and 491 fragments pass. Planner predecessor and +isolated second-budget mutant controls fail; clean counterparts pass. Both fresh +independent correction review +rounds and complete correction-head CI remain pending in this local receipt; +the PR description owns live head/check evidence. + +#### External correction review round 1 + +Fresh GPT-6-astra xhigh store/security and GPT-6.1-sol xhigh proof/corpus +reviewers approve `44089707..13cd632` without introduced findings. Independent +verification passes seven focused PostgreSQL cases and 99 lifecycle cases. +The PostgreSQL rerun uses the existing Release binaries after a whitespace-only +source correction; a fresh committed-source build and CI remain separate checks. + +The corpus reviewer finds a pre-existing adjacent source-comment contradiction: +`IIntegrationEventHandler` still requires handler-owned per-module inboxes. +Its remarks now name ADR-0046's transport-owned shared inbox and P02b-3 boundary. +The adjacent `IIntegrationEvent` remarks also stop promising unconditional +partition ordering or describing the implemented contract guard as Registered. +These are documentation corrections, with no interface or runtime change. + +#### External correction review round 2 and local closeout + +Fresh GPT-6-astra xhigh runtime/security and GPT-6.1-sol xhigh proof/corpus +reviewers independently approve `44089707..41ac2924`, with zero findings across +all 29 changed files. Both read the complete correction and the first round's +source-comment fix. The runtime reviewer independently inspects the completed +TRX counters; neither repeats the root's complete suites or mutation experiment. +Both correction review rounds are complete. No Accepted ADR text, grant or +consumer-delivery contract changes in this remediation. + +Complete local backend verification passes 3,396 cases: 1,845 unit, 286 +architecture, four contract, 175 Docker-free integration and 1,086 Docker +integration, with zero failures/skips. The 97 focused Docker cases above are a +subset, not additional cases. Guarded frontend passes 1,616 cases (1,556 web, +60 SDK), with zero failures/skips/todos; lint/typecheck pass. A fresh Release +build of `41ac2924` reports zero warnings/errors after the source-comment and +whitespace corrections. Five corpus controls, seven ADR-status script controls, +2,693 relative links and 500 fragments pass; historical P02b-0/P02b-1 records +and Accepted ADR text remain preserved. + +Correction-head PR CI and its real demo artifacts still require independent +verification after publication. This local receipt does not reuse the earlier +green head as evidence; the PR description owns live head/check evidence. + +### P02b-2 review of 3531a2eb (2026-10-13) + +The next external report was checked against `3531a2eb`, not adopted as an +instruction source. That baseline's seven required jobs and complete demo +artifacts pass in [run 38318773974](https://github.com/HodeTech/LearnStack/actions/runs/38318773974). +This follow-up requires its own review and CI evidence. No Accepted ADR body, +migration, grant, consumer-delivery contract or dependency changes here. + +- The earlier reverse-membership disposition is reconsidered using a real + direct/transitive `SET ROLE` control. The app guard already rejects elevated + memberships on new app connections; an administrator's later grant can still + elevate an existing app session. The dispatcher now detects incoming grants + on its next physical connection, enforcing the existing four-role boundary. + Superusers are excluded from that predicate because PostgreSQL implicitly + treats them as members of every role. This is grant-drift detection, not + continuous revocation. Four sync/async direct/transitive cases prove actual + escalation, refusal and healthy reconnection after restoration. An isolated + PostgreSQL comparison distinguishes the predecessor and corrected predicates. +- Credential census now rejects explicit configuration indexers with a + same-body `ConnectionStrings:` key/prefix, including compiled concatenation, + interpolation, formatting and static-readonly names. Unrelated configuration, + dictionary indexers and diagnostic text remain clean. This bounded check + does not claim arbitrary cross-method key dataflow. A separate planted control + closes `Assembly.CreateInstance`; `Activator.CreateInstanceFrom` was already + covered. Both omissions were reproduced against the predecessor detector. +- The constraint/RLS statement, shared-inbox and fifth-writer statuses, settled + ordering pointers and catalogue proof references now match current code. + Skills use a shipped public assembly marker in explicitly abridged examples. + The architecture method count is 194; Implemented rule count remains 192. +- Publication tests now cover successful/throwing completion, cleanup and + repeated/late cancellation. Wrong-kind store calls must leave the actual row + and audit unchanged. Test shutdown and publish timeouts use distinct values. + A source comment explains exact Npgsql microsecond truncation; the local API + environment comment names all three forwarded credentials. + +The remaining claims were checked against their actual scope: + +| Concern | Disposition | +|---|---| +| Startup options failure is reported as a credential error | Not on the shipped DI path: `OutboxDispatchStore` validates the same options before worker construction or source resolution | +| Broaden COMMIT catch to cure sticky terminal health | No demonstrated shipped exception escapes the existing provider/network/cancellation set. The dispatcher configures no custom driver logger. Failed reconciliation remains an unknown outcome even with catch-all; conservative health clears after successful terminalization | +| Non-admitted Normal claim hot-loops | Immutable registry and SQL exclude Pending identities before acquisition; unknown/retired/invalid rows already request terminalization. No current path reaches that invariant arm | +| Reject every unmapped JSON member | ADR-0046 explicitly uses the frozen payload options and compares declared facts; producer contracts enforce identifier-only fields. Unknown fields are not passed to the handler. A new stored-wire rejection policy is not inferred from this suggestion | +| Producer witness kind/subscription should anticipate subscriber minting | The current closed minting census admits only the producer store. P02b-3 owns a distinct subscriber capability and its identity controls; no subscriber factory is introduced early | +| Legacy-row diagnostics, blocking migration and attempt constants | Invalid facts deliberately refuse migration. Existing migrations remain immutable; checks/index creation need an owner maintenance window, with no online-migration claim. Tests pin six claims and the full backoff/DDL state; copying a new constant into historical migrations would not eliminate drift | +| UTF-8 SQL literals escape the literal census | Npgsql's command entrypoints accept strings. Decoding bytes is dynamic construction/dataflow outside this explicitly bounded literal census; raw byte data alone is not an executable SQL call | +| Platform credential rule is no longer independently sufficient | The whole-production credential/factory/consumer census independently pins both exact roots and their distinct sources. Repeating it inside the companion source rule adds no uncovered current boundary | +| COMMIT/probe stack assertions are redundant | Both failures are `NpgsqlException`; the stage-specific stack assertions distinguish their order. They supplement the physical outcome/no-duplicate proof and currently pass with the pinned driver | +| Ambient public-api overrides and seed validation should cover four credentials | Explicit ambient API overrides are an existing tested local-loader policy. Standalone seed consumes only Default and Migration; other ambient credentials never reach its child. Shared demo validates all four before startup | +| Forward proxy/cache/home environment into Seeder | The narrow child allowlist deliberately excludes ambient credentials/diagnostics; proxy values can themselves contain credentials. No failure on the documented restored-dependency path was demonstrated; the API's separate allowlist intentionally excludes `DOTNET_CLI_HOME` | +| Print the failed connection-string suffix | The fixed diagnostic is now safely visible. More detailed refusal messages are optional; values remain excluded and no further operational regression was reproduced | +| Ingress env tests depend on file order | Each loader test writes its own selected file before reading it; they are sequential, not concurrent. The later configuration test uses an explicit object, not that file | +| Duplicated construction-only factory helper | The remarks explain the intentionally absent worker/unhealthy private check. Sharing helpers across separate test assemblies would add a dependency without correcting behaviour | +| Skill index claims handler-owned inbox | Not reproduced: its generic inventory names an inbox guard but assigns no handler ownership; detailed skills now identify the shipped/P02b-3 boundary | +| Persist traceparent or measure recovery/terminal as publication duration | Trace persistence is optional. The duration histogram measures publication; recovery and terminal transitions do not publish and retain their own spans/outcomes | +| Retained partition also delays terminal acquisition | Deliberate local ownership: no further acquisition for that partition until callback and scope cleanup actually finish, including terminal work; ADR-0046's four-overrun saturation applies | +| Assembly-qualified identities, closed generics and invalid disposition label | Exact assembly-qualified identity plus explicit compatible aliases is Accepted. A declared closed generic is a concrete known type, not arbitrary loading. Invalid-envelope reason and the fixed `invalid` label remain distinct from unknown-type reason | +| Legacy non-NULL metadata should be backfilled | ADR-0057 requires new producer metadata to be SQL NULL; ADR-0046 deliberately terminalizes invalid legacy envelopes rather than silently rewriting their facts | +| Alternate health-registration dedupe and audit trigger | The sole production composition is pinned. A new trigger/privilege design is Phase 11 work, not an unreviewed replacement for the Accepted row-bound terminal seam | +| Squash migrations or mechanically gate published fragment counts | Existing migrations/history are preserved. Link/fragment totals are explicitly local checks, not CI enforcement claims; the prior and current source scans remain separate evidence | + +Local Release build reports zero warnings/errors; format verification passes. +Unit passes 1,847, architecture 287, contract four and Docker-free integration +175 cases, all with zero failures/skips. Twenty role controls and six focused +claim-kind/shutdown cases pass. The new compiled-probe helper's initial deferred +Cecil-body failure was corrected before the successful full architecture rerun; +the guard expectation was preserved. Seven ADR-status controls, 2,711 relative +links and 505 fragments pass. Historical delivery records and Accepted ADR text +remain unchanged. Complete Docker validation, both fresh independent review +rounds and correction-head CI remain pending in this initial local receipt; +the PR description owns final head/check evidence. + +#### Review of 3531a2eb: verification closeout + +Both fresh independent review rounds approve `3531a2eb..a5b1e723` with zero +findings across all 17 changed files. Each round uses a fresh GPT-6-astra xhigh +security/runtime reviewer and GPT-6.1-sol xhigh proof/corpus reviewer. All review +lenses were covered. Reviewers inspect source and TRX evidence without repeating +the root's full suites; the second proof reviewer independently confirms 194 +architecture methods, 192 Implemented rules and all 13 explicit method links. +No further runtime correction was required after the implementation commit. + +Complete local backend verification passes 3,408 cases: 1,847 unit, 287 +architecture, four contract, 175 Docker-free integration and 1,095 Docker +integration, with zero failures/skips. The 20 role and six focused claim-kind/ +shutdown cases are subsets of Docker integration, not additional cases. Release +build, format, seven ADR-status controls and the 2,711-link/505-fragment local +check pass. Historical delivery records and Accepted ADR text remain preserved. +No frontend source or dependency changed in this correction; the next real CI +run re-verifies frontend and demo artifacts along with the corrected backend. + +Both correction rounds and local validation are complete. Exact correction-head +CI remains pending in this local receipt; the PR description owns live checks +and artifact-backed evidence after publication. + +#### Snapshot interval correction (2026-10-13) + +The dispatch loop no longer forces a full queue snapshot after every successful +claim. It checks `SnapshotInterval` after completed iterations, while `didWork` +still controls idle polling. A controlled monotonic-clock integration case +dispatches three real rows: the initial snapshot survives successful work and +the last tick before the interval; the exact boundary refreshes it after the +next completed dispatch. Release build reports zero warnings/errors; format, +1,847 unit, 287 architecture and all 17 worker integration cases pass, with zero +failures/skips. Two fresh sequential independent reviews of `3620b421..3baf3a22` +approve with no findings. In an isolated tracked-source copy, reverting only the +production call to forced refresh makes the new case fail its retained-snapshot +assertion (one executed, one failed); the corrected case passes. New-head CI +remains pending in this local receipt; the PR description owns live checks and +artifact-backed evidence after publication. + ### Durable outbox dispatch P02b-1 Steps 2/3 implement `IOutbox`, replace Packet 3's step-7 shell with an enrollment/invariant guard, and supply owner-side domain collection and the -first publisher. Dispatch and consumer delivery remain P02b-2/P02b-3. +first publisher. P02b-2 delivers hosted producer dispatch and production-census +enforcement; all four steps and both independent review rounds are locally +complete. The [PR handoff](#p02b-2-pr-handoff-2026-10-12) owns current CI/review/merge +state; consumer delivery remains P02b-3. The table shipped in Packet 6. A handler calls `IOutbox.EnqueueAsync` and the row commits on the same **transaction** @@ -1131,7 +2092,7 @@ cannot enqueue. Production delivery wiring remains P02b-3. ADR-0006 Amendment 1, ADR-0010 and the glossary all specify, with a 200 ms poll no job schedule expresses. This phase hosts it in `LearnStack.Api`, the only web host. `LearnStack.Tools.Seeder` is a separate one-shot executable, not a worker host. -Nothing in `backend/src` implements `IHostedService` today. The corpus's gateway +P02b-2 Step 3 registers the mandatory hosted processor. The corpus's gateway route table puts `/admin/hangfire*` on that host ([API Gateway](../architecture/30-api-gateway.md), route id 3, upstream `learnstack-api`), and the production image is @@ -1144,8 +2105,10 @@ Dapr trigger verbatim — so it stays with Phase 11. - `ConnectionStrings:OutboxDispatcher` carries the `learnstack_outbox_admin` credential, registered as a keyed data source behind a `Lazy<>` with a boot-time role guard, on the pattern `Platform_DataSource_Resolved_Only_By_PlatformAdminScope` already holds. - `LearnStack_OutboxAdmin_Role_OnlyUsedBy_OutboxProcessor` is **Awaiting backfill** in - the catalogue and is carried here or nowhere. + `LearnStack_OutboxAdmin_Role_OnlyUsedBy_OutboxProcessor` is **Implemented** + with Step 4's nonempty credential caller census and passing planted controls; + all four steps and both independent review rounds are locally complete. + The [PR handoff](#p02b-2-pr-handoff-2026-10-12) owns CI/review/merge state. - The three carriers that say the credential lives in "the worker host" — [Database Standards § Database roles](../standards/05-database.md#database-roles), `architecture/15` and `.env.example` — are derived cells, not a topology decision; @@ -1154,42 +2117,37 @@ Dapr trigger verbatim — so it stays with Phase 11. #### The dispatcher -- **The lease.** The claim is a durable `UPDATE` stamping `locked_by` / `locked_until` - inside a short transaction; dispatch runs outside it; the claim predicate itself - reclaims an expired lease, so there is no sweeper. Every state transition carries the - ownership fence, failure included — a stale processor must not overwrite a new - claimant's success or clear its lease. -- **The migration** adds both columns *and* extends `learnstack_outbox_admin`'s - column-scoped `UPDATE` grant in the same change. ADR-0006 Amendment 2 requires that - pairing and Database Standards § Outbox already predicts the runtime failure of - omitting it: `permission denied for table`. The integration case - `PlatformSchemaTests.TheDispatcherHoldsExactlyTheFourColumnUpdateGrant` pins the - current four columns by exact equality, and is re-pinned — and renamed — here. -- **Retry** is driven by `available_after` with `attempts` and `last_error` on the row, - on the schedule - [Infrastructure Stack Standards § Outbox and Inbox](../standards/20-infrastructure-stack.md#outbox-and-inbox) - publishes. That document publishes the schedule and a maximum of five retries; **G4** - fixes what counts as an attempt against that number and what marks a row terminal, and - **G18** fixes what `last_error` may contain — both Accepted before this packet writes - the canonical DDL, which is a canonical artifact. -- **The SQL form.** The claim, the lease, the mark and the terminal transition are - parameterised commands on the dispatcher's own data source, never EF's set-based APIs - — which CLAUDE.md and `No_Set_Based_Write_Bypasses_The_Audit_Capture` forbid, and - whose `FromSqlInterpolated` cousin that rule cannot see is the other wrong answer. The - platform tables already follow this shape: their `DbContext` maps no entity types. -- **A per-delivery deadline**, so one hanging handler cannot stall dispatch for every - tenant, and so a timeout is not misread as shutdown. +- **The lease.** A committed normal claim uses a fresh canonical UUID token in + `locked_by`, database-time expiry and one just-in-time row. Every acknowledgement, + failure and terminal transition checks row ID, token and unexpired lease. + Recovery first replaces an expired token with a fresh one; it never publishes + or increments attempts. Stale callbacks cannot clear a later lease. +- **The migration** adds lease/terminal columns and exact status-column grants + together. [Database Standards § Outbox](../standards/05-database.md#outbox) owns + canonical DDL/checks/indexes. Audit-chain INSERT-only privilege enables the + separate fifth audit writer on the same confined transaction. +- **Retry** counts six committed normal claims, including claim-before-crash. + The five delays are 1 s, 5 s, 30 s, 5 min and 1 h. Failure clears the lease; + exhausted or invalid rows request terminalization without another publication. + A failed terminal MUST audit rolls back terminal state and retries only + terminalization. Stored descriptors are closed codes capped at 256 UTF-8 bytes. +- **The SQL form.** Parameterized commands use only the confined data source; + no EF materialization or set-based API owns dispatcher status. Required Pending + subscriptions and their aliases are excluded before claim/payload validation; + intentional absence is the only known-type zero-subscriber acknowledgement. +- **Deadlines.** The accepted bounds are in the P02b-2 package. Ignored + cancellation retains and observes the task, blocks its local partition and + makes health unhealthy; four retained overruns stop new claims. Late work + cannot acknowledge the row. There is no forced task termination promise. #### Ordering -`partition_key` is on the row and `InProcessEventBus` serialises per key within a -process. What a key guarantees across commits, retries and processes is **G3**: no -column obtainable from `outbox_messages` carries an unconditional per-key publish-order -guarantee — `occurred_at` is the transaction's start time and is identical for every row -one transaction writes, the `uuidv7()` id is minted at `INSERT`, a failed row's -`available_after` moves past its own successor, and an unordered `RETURNING` has no -defined row order. The completion criterion follows the gate, and -[§ Ordering](../architecture/15-event-and-outbox.md#ordering) is edited to match it. +ADR-0046 accepts deterministic eligible selection by +`(available_after, occurred_at, id)` within one processor. It promises no enqueue, +commit, retry, cross-process or unconditional same-key publish order. Interleaved +transactions and retry deferral are proof subjects. Partition-key derivation +remains the separately delivered ADR-0057 contract; a key alone proves no durable +sequence. See [Ordering](../architecture/15-event-and-outbox.md#ordering). #### Domain events collected and dispatched @@ -1219,78 +2177,45 @@ every terminal path clears transaction-scoped source/collector state exactly onc #### The consumer side -- **`inbox_messages` and `IInboxGuard`**, built from the canonical DDL that lands in - Database Standards § Inbox — beside § Outbox, on ADR-0006 Amendment 2's precedent, - with its table class, its policy taken by link to - [§ Table classes](../standards/05-database.md#table-classes) rather than copied, and - its grants. Today the only inbox DDL in the corpus is four columns keyed - `event_id uuid PRIMARY KEY` in an architecture document, Database Standards has no - inbox section, and ADR-0023 lists the table as DB-side-minting while Database - Standards says its key is the producing envelope's `EventId`. **G1** settles the shape - and the erratum lands with it. -- **One unit of work per `(event, consumer)` delivery**, opened by the transport: - `BeginTransactionAsync`, then `SetTenantContextAsync`, then the handler, then commit - or roll back. Both statements run **before** the handler is resolved, because a module - `DbContext` built outside the ambient transaction is refused — and the shipped - transport resolves the handler first, so this is a change to it, not a description of - it. A construction failure therefore happens inside a transaction that must roll back, - and counts as one failed attempt for that pair. -- **Tenant and organization context restored from the envelope** in every handler scope. - A consumer that runs without tenant context writes rows Row Level Security rejects, or - worse, does not. `EventTenantContext` restores from the payload's `TenantId`; the - row's `tenant_id` is the value Row Level Security checked on insert, and the two are - required to agree before dispatch. -- **The subscriber failure contract**, under **G1**. Two committed designs currently - contradict each other: this phase's per-`(event, consumer)` attempts with a terminal - state inside `inbox_messages`, and - [§ Dead-letter](../architecture/15-event-and-outbox.md#dead-letter-two-sides-two-failure-domains)'s - dead-letter destination with the inbox row deliberately left unmarked. They are - mutually exclusive, the four-column inbox cannot hold attempts or a consumer - dimension, and the requirement both are trying to meet is the same: one poisoned - consumer must not dead-letter an event that four other consumers handled cleanly. The - losing text is deleted, not left beside the winner. -- **A named owner for every unhandled delivery's next attempt.** The shipped - `InProcessEventBus` rethrows a handler failure out of `PublishAsync`, and - [Error Handling Standards § Outbox](../standards/09-error-handling.md#outbox) charges - any `PublishAsync` exception to the outbox row — so today one poisoned consumer - increments the *producer's* attempts and redelivers to every subscription. ADR-0038's - one-method port also cannot address a single subscription. Whatever G1 decides, the - record names the component that redelivers and where the per-subscription attempt - count survives the delivery transaction's rollback. -- **Dead-lettering emits its counter and writes a MUST-class audit row.** A silently - dropped integration event is a data-integrity event, not a log line. No slug, class, - catalogue source or matrix row exists for it today, and Audit Coverage Standards makes - catalogue → matrix total with no exemption, so **G7** is what makes it buildable. -- **Replay is a dispatcher operation** here: reset the dead-lettered pair and redeliver - to that one subscription through the guard, with integration tests. It runs on the - dispatcher's own connection, because no API-reachable role may `UPDATE` an outbox row - — `learnstack_platform` holds `SELECT, DELETE` — and the operator surface is Phase - 03's (§ Explicitly not in this phase). +**Accepted contract; implementation pending P02b-3.** Construction-free manifests +carry stable versioned subscription IDs, compatible aliases and required +Pending/Active readiness. Active publication durably accepts every matching +delivery without running business handlers. The independent executor owns retry +budgets, leases and terminal state in a separate delivery ledger. + +One shared `inbox_messages` is keyed by `(tenant_id, subscription_id, event_id)`. +The transport opens the app-role unit of work, announces envelope tenant and +organization, then claims before handler construction. Completed duplicates skip +construction. Business state, completed inbox, produced outbox, MUST audit and +fenced ledger completion share one physical transaction. Rollback removes a new +claim; completed inbox rows never reset. Ordinary delivery uses SystemActor and +refuses privileged work; Phase 03 owns erasure's separate audited boundary. + +Timeout poisons the business owner, preventing late success. An uncertain COMMIT +is Indeterminate, including the final claim; atomic ledger/inbox state decides +recovery. The bypass scheduler sees only enumerated identity/routing/status +columns, never envelope payload. P02b-3 supplies canonical DDL, exact RLS/grants, +capture exclusions and the ADR-0023 minting correction before runtime. + +A poisoned subscription leaves siblings untouched. Its fenced terminal state and +Tenancy-owned `platform.delivery.dead_letter` MUST security-event commit together. +Internal replay resets only that ledger pair and invokes its executor, never +`PublishAsync`. Phase 03 retains producer reset and operator permission/reason, +fencing and surface contracts. Phase 11 retains purge intervals and broker +binding. External effects need subscription/event idempotency or an outbox intent. #### Observability [Observability Standards § Required Metrics](../standards/10-observability.md#required-metrics) -owns metric names and -[Events and Outbox § Observability](../architecture/15-event-and-outbox.md#observability) -specifies the seven this subsystem needs. This phase emits all seven in those spellings -and registers the four Observability Standards does not yet carry — the producer-side -dead-letter counter, the lost-lease counter, the inbox dedup counter and the -subscriber-side dead-letter counter. It re-spells none of the three that document -already holds, and the only names it adds beyond the seven are the two liveness signals -below, registered in the same place under **G17**. - -- The two dead-letter counters stay separate signals on separate dashboards, for the - reason `architecture/15` gives: a producer-side spike means the transport is down, a - subscriber-side spike means a handler or a payload is broken. -- A pending **count** is not a **lag**. A terminal row keeps `processed_at IS NULL`, so - a naive count includes rows that will never be scheduled again, and a small stuck - queue is worse than a large one draining. An oldest-eligible-row age metric lands with - the dispatcher, and the alerting row that reads a raw count is corrected with it. -- Dispatcher liveness is an `outbox` health check plus a heartbeat gauge, following - `audit`'s registered-not-mapped precedent — not the job scheduler's surface. -- Label sets are settled by **G17** before the first series ships, in the decision pass - of `P02b-2`: a label added afterwards touches every emitter, and a series shipped with - a guessed label cannot be renamed once a dashboard and an alert read it. +is the sole inventory of names and dimensions. P02b-2 supplies producer duration, +failure, terminal and lease-loss diagnostics, immutable backlog/eligible-age/ +deferred/heartbeat snapshots and registered `outbox` health. No metric has a +tenant label; event-type labels come only from the bounded registry. Required +Pending rows, completed and terminal rows are excluded from normal eligibility. +P02b-3 owns independent subscriber terminal/dedup signals and minimizes the old +transport diagnostics. Separate producer/subscriber failure signals describe +separate budgets. Health is registered, not mapped; `/healthz` stays liveness and +Phase 11 owns `/readyz` and production alert windows. ### Background jobs @@ -1534,8 +2459,9 @@ on `learnstack.tenancy.tenant`, partitioned by `TenantId`, with no extra payload fields and null envelope organization. P02b-1 emits it only after tenant/default organization/back-reference completion and persists pending rows. -After G1, P02b-3 supplies two independent Customization subscriptions for `card@1` -and `plain@1`. Each create-only effect owns one root, generation update and MUST +Under ADR-0046, P02b-3 supplies independent Customization subscriptions for the +`card@1` content type and `plain@1` taxonomy. Each create-only effect owns one root, +generation update and MUST audit on its inbox transaction, preserves authored/lifecycle/deletion decisions, and resolves incomplete pairs through retry. P02b-3 owns semantic key/version Seeder verification, preserved warm IDs and fresh seed/delivery convergence. @@ -1561,7 +2487,7 @@ points it does not reach, and both of Packet 9's hand-offs land here. **No row would be written and nothing would fail.** That is the window [ADR-0033](../decisions/0033-audit-durability-model.md) says must not exist. **G6's common-owner contract is Accepted under ADR-0057**; P02b-3 implements - it in production delivery after G1. Its proof is a MUST-classified direct-write + it in production delivery under ADR-0046. Its proof is a MUST-classified direct-write consumer, not only a consumer that sends a command. - **The audit frame's owner on a non-pipeline entry point.** What is *already* decided @@ -1589,20 +2515,18 @@ points it does not reach, and both of Packet 9's hand-offs land here. absorbed by a committing delivery from a produced-work refusal that poisons it, a faulted `COMMIT` producing exactly one `indeterminate` row, and `Clear()` running exactly once per delivery. -- **The dead-letter transitions**, under **G7**. "Through `IAuditStore`" names neither a - method nor a permitted caller, and no single role can both make the terminal outbox - transition and insert the row on one transaction: `learnstack_app` has no `UPDATE` on - `outbox_messages` and `learnstack_outbox_admin` has nothing at all on `audit_log`. The - record names the class, the slug, the transaction, the ordering against the state - change and the failure posture; the slugs are registered off-path and carry their - matrix rows in the module that writes them. Poisoned-job handling inherits the same - shape. -- **The dispatcher's `BYPASSRLS` posture**, under **G8**. Audit Coverage Standards - requires a `security-event` row on every platform-bypass invocation and lists RLS - bypass as MUST; the only exemption is a sentence in an architecture document, which - cannot grant it. A per-invocation row on a 200 ms polling loop is a different - table-growth curve from a per-request one, and this is the first `BYPASSRLS` - credential that actually runs. +- **Dead-letter transitions — Accepted ADR-0046.** Producer terminal state and + Tenancy-owned `platform.outbox.dead_letter` MUST security-event commit on one + short confined-role transaction. `WriteEventTerminalAsync` validates its + restricted `EventTerminalTransition` witness against the actual fenced UPDATE + and draft identity. Audit failure rolls back state and retries terminalization, + without another publication. The corresponding subscriber operation remains + P02b-3. Register each planned off-path slug only with its writer. +- **Routine dispatcher bypass — Accepted ADR-0046.** The confined credential, + exact grants, bounded diagnostics and atomic terminal audit explicitly exempt + routine polls/claims/acknowledgements/retries from per-invocation security audit. + Existing `EnterPlatformAdminScope` entry audit and all other bypass obligations + remain unchanged. This is no general background-job exemption. - **The rejected-assertion row names its actor once there is one.** The recorder writes a null actor on the authenticated tier today, because no principal exists in the process and `IsAuthenticated` is a tier rather than an identity. ADR-0036 makes the @@ -1658,11 +2582,12 @@ Its own tests assert the positive case, a clean case staying clean, a sensitive outside a `throw` staying clean, no `AD0001` crash — the failure mode ADR-0032 Amendment 1 records for `LS0001` — and that the analyzer's token set equals the catalogue's. -The same change bounds what the durable sinks may hold, under **G18**: a sanitized -descriptor — the exception type, the error code and the attempt's `traceparent` — with a -length bound in the canonical DDL. P02b-2 owns that migration beside the lease -columns and must account for existing P02b-1 producer rows; its decision pass -settles descriptor handling before applying the bound. +P02b-2 bounds the durable failure descriptor under **G18** to closed stage, +reason and category codes, with only allowlisted SQLSTATE where applicable; +exception text/type, trace identity and payload are not persisted there. The +canonical DDL owns the byte bound. Step 1 implements the migration beside the +lease columns, including redaction of legacy descriptors without rewriting +event facts/state; Steps 2–3 implement transitions and bounded diagnostics. ### Architecture tests @@ -1690,9 +2615,9 @@ What this phase owes the catalogue beyond implementing those rows: unresolved provisioning. Its partition declaration rule combines independent stored-source IL checks, varied identifier probes and real persisted keys; an uninitialized instance is not a value proof. Publish ordering stays P02b-2. - `Outbox_Claim_IsHeld_Until_Dispatch_Completes` says "exactly once" unconditionally, - which contradicts the at-least-once contract; it narrows to live-lease contention and - gains a redelivery companion for crash and expiry. + `Outbox_Claim_IsHeld_Until_Dispatch_Completes` replaces the former unconditional + exactly-once assertion with live-lease contention, expiry/crash redelivery and + stale-transition refusal; Step 3 implements the actual two-worker proof. `Hangfire_Job_Payloads_Include_TenantId` gains the organization, actor and traceparent fields. `OutboxProcessor_NeverBlocks_OnSingleMessageFailure` gains the hanging-handler case. Two rules the catalogue stamps **02a** also @@ -1734,16 +2659,18 @@ command. The phase retains zero skips, per assembly. ## Deliverables -- **ADR-0046 Accepted** — event delivery, the inbox table's canonical shape, and the - subscriber failure contract as one state machine: subscription identity, attempts, - backoff, cancellation, terminal transitions and replay, with the losing design - reconciled out of its carriers. +- **ADR-0046 Accepted** — event delivery, the shared inbox contract and separate + delivery + ledger: stable subscription identity, readiness, durable acceptance, attempts, + cancellation, terminal transitions and targeted replay. Canonical inbox/ledger + DDL remains P02b-3 work; acceptance claims no consumer runtime. - **ADR-0047 Accepted** — the background-job runtime, with the dated ADR-0003 Amendment its storage decision owes. - **The dated amendments the register names**, each Accepted before the code it governs: - ADR-0044 / ADR-0033 / ADR-0040 for the delivery and domain-event frames (G6), ADR-0033 - for the dead-letter writer (G7), ADR-0006 or ADR-0003 for the dispatcher's bypass - posture (G8), ADR-0006 for the retry contract (G4), ADR-0036 for the matrix staging + ADR-0044 / ADR-0033 / ADR-0040 for the delivery and domain-event frames (G6), + ADR-0046 with ADR-0044/0033 disclosures and fifth-writer seam + for terminal audit (G7), ADR-0046 with ADR-0003/0006 disclosures for bounded + bypass/retry (G8/G4), ADR-0036 for the matrix staging (G11), ADR-0032 for `LS0002` (G16), ADR-0038 for the payload-content rule (G18). - **Database Standards § Inbox** — the canonical inbox DDL beside § Outbox, with its table class, its policy taken by link and its grants; plus the erratum ADR-0023's @@ -1752,26 +2679,28 @@ command. The phase retains zero skips, per assembly. [Phase 11](phase-11-production-hardening.md) picks the purge interval that honours it. - `IOutbox` and the `OutboxFlushBehavior` invariant guard, delivered by P02b-1 under Accepted ADR-0057; immediate INSERT uses the announced transaction. -- `TenantCreatedIntegrationEventV1` from completed provisioning; after G1, two - independent Customization subscriptions install untouched `card@1`/`plain@1` - defaults with one-root effects and the P02b-3 Seeder identity handoff. +- `TenantCreatedIntegrationEventV1` from completed provisioning; P02b-3 + independent Customization subscriptions install untouched `card@1` content-type + and `plain@1` taxonomy defaults with one-root effects and the P02b-3 Seeder + identity handoff. - A durable `OutboxProcessor` implementing the specified lease: a fenced claim, a per-delivery deadline, bounded retry, a terminal producer state, and the migration that adds `locked_by` / `locked_until` **and** extends `learnstack_outbox_admin`'s column-scoped `UPDATE` grant in the same change. - The dispatcher's credential registered as a keyed data source behind a boot-checked - `Lazy<>`, with `LearnStack_OutboxAdmin_Role_OnlyUsedBy_OutboxProcessor` moved from - Awaiting backfill to Implemented and a companion proving the sweep can fail. + `Lazy<>`, with `LearnStack_OutboxAdmin_Role_OnlyUsedBy_OutboxProcessor` promoted + from Registered only after the full census and planted controls pass. - `partition_key` persisted onto the row and rebuilt into the envelope on dispatch. `PartitionKey` abstract on `IntegrationEventBase` and per-key serialisation in `InProcessEventBus` shipped in Packet 5; the durable hop is this phase's. - `inbox_messages` and `IInboxGuard` over the canonical DDL, the per-`(event, consumer)` delivery transaction opened before handler construction, and tenant and organization context restored in every handler scope from the envelope. -- A subscriber-side dead-letter path in the shape G1 settles: a terminal outcome per +- A subscriber-side ledger and terminal path accepted by ADR-0046: a terminal + outcome per subscription, a sanitized failure descriptor, its counter, its MUST-class audit row, - and a replay that re-enters the inbox-guarded path — as a dispatcher operation, with - the operator surface deferred to Phase 03. + and targeted internal replay through the executor/inbox boundary. Producer + reset and operator permission/reason/fencing remain Phase 03 contracts. - A worked sample flow — enqueue → outbox → dispatch → idempotent consumption — placed where G19 settles, between modules that hold domain code, so the module-scoped sweeps have a subject. @@ -1785,12 +2714,10 @@ command. The phase retains zero skips, per assembly. - The background-job runner, its storage on the path ADR-0047 names, its tenant-context hook, its enqueue-time guard over four fields, its package pins — and no job definition and no dashboard route. -- The `outbox` health check registered and not mapped, plus six rows in - [Observability Standards § Required Metrics](../standards/10-observability.md#required-metrics) - — the four event counters in the spellings - [Events and Outbox](../architecture/15-event-and-outbox.md#observability) already - uses, and the dispatcher heartbeat gauge and the oldest-eligible-row age metric under - **G17**. +- The registered, unmapped `outbox` health check and accepted producer event + metrics in + [Observability Standards](../standards/10-observability.md#required-metrics), + including bounded immutable snapshots. Subscriber signals remain P02b-3. - `UseAuthentication` and `UseAuthorization` at their two pinned positions, the claim signal populated at the resolver's reserved site, one JWT authority for `/api/v1/*` with its full validation contract written into @@ -1867,6 +2794,12 @@ shape depends on an open gate say so, and are written when that gate is Accepted **Producer and dispatcher** +P02b-2-owned dispatcher criteria below are met by the +[Step 4 local closeout](#p02b-2-step-4--enforcement-and-closeout-2026-10-12). +The combined HTTP/event-delivery tracing criterion still requires P02b-3 +consumer execution; the whole phase is not complete. The +[PR handoff](#p02b-2-pr-handoff-2026-10-12) owns CI/review/merge state. + - An aggregate change and its outbox row commit together and roll back together, asserted as `learnstack_app`. An enqueue whose event `TenantId` differs from the successfully announced transaction tenant is refused, including unresolved @@ -1877,80 +2810,64 @@ shape depends on an open gate say so, and are written when that gate is Accepted activity's trace, whose `tenant_id` and `organization_id` equal the announced context, and whose `partition_key` equals the event's — and the assertion fails on a null `partition_key`. -- Two `OutboxProcessor` instances draining one pending batch, **both holding unexpired - leases**, publish each message once; the claim `UPDATE` succeeds as - `learnstack_outbox_admin` and fails with `permission denied for table` against the - pre-extension grant. A companion of the same harness against a release-on-read claim - sees duplicates. -- A lease that expires mid-dispatch is reclaimed, the first processor's success **and** - failure writes are both no-ops under the ownership fence, the lost-lease counter - increments, and the consumer sees one business effect. -- A producer dead-letter writes exactly one MUST-class row under the slug **G7** names, - and a forced failure of that write produces the behaviour G7 chose — the transition - unmade and retried, or the duplicate that ordering accepts. The subscriber side - already has both halves; the producer side needs them more, because its transition and - its audit row are made by two roles. -- A failed dispatch retries on the published backoff with its lease cleared — so the - first retry runs at its stated delay rather than at the lease duration — reaches the - terminal state G4 names at the stated attempt, increments the producer dead-letter - counter, and leaves the pending gauge. +- Two coordinators on one PostgreSQL store force an interleaving with at most + one live owner. Claims use the exact confined-role grant and fresh acquisition + tokens; this proves neither two OS processes nor exactly-once publication. +- Expiry/recovery replaces the token, and stale success/failure/terminal writes + affect zero rows. The six-claim crash case never publishes a seventh time. +- Producer terminal state and exactly one `platform.outbox.dead_letter` MUST + security-event commit atomically. Audit refusal rolls back terminal state and + permits only terminalization retry; no second publication consumes a budget. +- Retry uses the five accepted delays, clears the lease and stores only closed + bounded descriptors. Terminal requests and exhausted rows leave normal + eligibility; successful terminalization leaves backlog snapshots. - A blocked dispatcher loop turns the `outbox` health check unhealthy within a bounded number of poll intervals, while `/healthz` still answers 200. - With one terminal row, one row whose `available_after` is in the future and one due row in the table, the age metric reports the **due** row's age — the case that separates it from `min(occurred_at) WHERE processed_at IS NULL`. - A `ConnectionStrings:OutboxDispatcher` naming any role other than - `learnstack_outbox_admin` refuses the boot at registration; an absent or malformed - value names its key and the expected form; a valid one registers without connecting; - and no message carries the password. The four cases already exist for the application - data source and are copied rather than re-derived. -- A payload written by a shipped event's V1 is read by V2-deployment code without loss; - a row naming a retired type dead-letters at once under its named reason; a dispatched - event with no registered subscription marks its row processed and increments no - failure counter. -- The same-key ordering criterion is written when **G3** is Accepted. Whatever it - asserts, it runs interleaved producing transactions — the first to begin commits last - — because a same-key pair written with a wide clock gap by one processor is green - against a dispatcher that reorders whenever two producing transactions overlap. - -**Consumer side** - -- One envelope delivered twice, sequentially and then concurrently, yields one business - row and one processed inbox record, and the dedup counter increments once; the same - case with the guard check removed fails. -- A handler that marks the inbox and then throws commits neither, and the next - redelivery writes exactly once. -- A fan-out where consumer 1 commits and consumer 2 throws leaves consumer 1's write - committed, consumer 2's rolled back, consumer 2's failure recorded durably **after** - that rollback, and exactly one of the two retry budgets advanced. The test names the - component that owns consumer 2's next attempt. -- A handler that throws on every delivery reaches the terminal state for **that consumer - only**, leaves the other consumers of the same event unaffected, accumulates no - attempts on the producer's row, increments the subscriber dead-letter counter, and - produces exactly one MUST-class audit row under the slug G7 names. -- A reset `(event, consumer)` pair is redelivered to that subscription and to no other, - and a late success racing the reset is absorbed by the guard. -- The same event delivered to two handlers in one module runs both, and neither one's - inbox record makes the other skip. -- An event of tenant B published during tenant A's request is handled with tenant B - resolved **and** `app.tenant_id` set to B, before the handler is constructed; the - handler's write is visible to B and invisible to A; and a transport variant that skips - the context statement fails the same case. No shipped case asserts `app.tenant_id`; - `InProcessEventBusTests` proves the accessor and the scoped context in-process only. -- A consumer whose constructor takes a module write store resolves — true only if the - delivery transaction is open before the handler is resolved. -- Each module-assembly event sweep reports a non-zero subject count as its own leg, so a - sample flow that lives only in a test assembly fails the phase rather than passing - three rules over nothing. -- A tenant-wide event published from an organization-scoped request is consumed with the - organization scope **G1** settles, and its consumer can write the rows that scope - permits rather than failing obscurely. -- An inbox record written under tenant A is invisible to a `learnstack_app` session - announcing tenant B, and all module migration chains still apply together on an empty - database. -- A dispatch failure whose exception message embeds a value the sensitive-token - catalogue matches leaves no trace of it in any persisted failure field, and the length - bound refuses a longer value. + `learnstack_outbox_admin` refuses API startup upon mandatory worker resolution; + an absent or malformed value names its key and the expected form. Lazy registration + constructs no data source and performs no connection; mandatory worker startup + resolves the source and verifies connectivity. No message carries the password. +- Declared compatible aliases deserialize through frozen payload options; + unknown/retired/invalid rows terminalize under closed reasons. Pending required + subscriptions, including aliases, preserve rows byte-exact before claim and + validation. Only intentional absence acknowledges without publication. +- Deterministic eligible selection is tested with interleaved commits and retry + deferral; no unconditional same-key publish-order assertion is retained. +- Overrun tasks are retained/observed, block local partitions and make health + unhealthy. Unrelated work progresses until the cap of four stops new claims; + late completion never acknowledges the row. + +**Consumer side — P02b-3** + +- Sequential/concurrent duplicates yield one completed inbox/business effect per + stable subscription/event. Distinct sibling subscriptions both execute. A + completed duplicate skips handler construction; removing transport guard + acquisition makes the proof fail. +- Faulted business work commits no business/inbox/outbox success. A poisoned + subscriber advances only its own durable budget after rollback, leaves producer + and healthy siblings unaffected and terminalizes atomically with its MUST audit. +- Targeted internal replay invokes only its executor; completed inbox state is + never reset. Late work cannot defeat the new fence. +- Envelope tenant and organization are announced before resolving handlers, + stores or module contexts. SystemActor is effective identity; a human actor is + causal metadata only. Non-marked event types preserve envelope organization. +- Construction failure and cancellation count under the subscriber policy. A + timeout poisons the owner; uncertain physical COMMIT, including on the final + claim, resolves from atomic ledger/inbox completion without repeating success. +- Every production module-assembly sweep reports its nonzero subjects. Both real + Customization defaults subscriptions, one-root audits, authored/lifecycle + preservation and semantic key/version Seeder handoff pass through the actual + delivery boundary. +- Inbox/ledger RLS isolates tenants; scheduler column-only grants cannot read + payload or perform business writes. Migration chains apply together. Ordinary + app-role execution refuses privileged work and external effects require their + own idempotency or outbox intent. +- Persisted diagnostics reject raw exception/payload/identity text and values over + 256 UTF-8 bytes; no sensitive token leaks through retained failure descriptors. **Audit** @@ -1971,9 +2888,10 @@ shape depends on an open gate say so, and are written when that gate is Accepted outbox row uncommitted. Planted handlers using writes, nested sends, notification publication, raises or providers fail the complete module/helper/interface sweep. Rollback, refusal, cancellation, fault and same-scope reuse cannot replay sources. -- A handler that ignores its cancellation token does not stall the rest: the - per-delivery deadline elapses, other keys keep dispatching, and the abandoned delivery - commits nothing afterwards. +- Ignored cancellation poisons the owner and blocks late success. Retained tasks + remain observed and hold their local partition; unrelated work progresses + within the accepted cap. An uncertain physical COMMIT is Indeterminate and + recovers from atomic durable state rather than assuming rollback. - A frame whose captures name an aggregate root that no intent in the frame declares produces the behaviour G6 records. - A consumer handling an event caused by user U writes a row whose metadata names U and @@ -2087,12 +3005,11 @@ shape depends on an open gate say so, and are written when that gate is Accepted analyzer. - The phase's authenticated endpoint appears in the generated OpenAPI document and in the regenerated SDK surface. -- Every metric this phase emits is spelled as its owner spells it — the seven names - [Events and Outbox § Observability](../architecture/15-event-and-outbox.md#observability) - already carries, and the two liveness names **G17** registers in - [Observability Standards § Required Metrics](../standards/10-observability.md#required-metrics) - — asserted against literal strings taken from that table rather than against the - emitting type's own constant, which is true by construction. +- Every event metric uses the literal name and bounded dimensions in + [Observability Standards](../standards/10-observability.md#required-metrics). + Refresh publishes one complete immutable queue snapshot; each gauge reads an + immutable snapshot. Pending, deferred, terminal and normal eligibility are + tested as distinct states. Job dimensions remain G9/G17 work. - **No row in the [catalogue](../standards/21-architecture-tests-catalogue.md) whose Phase names 02b is still Registered or Awaiting backfill.** Structural rows are green in the `backend` check and behavioural rows in the `backend integration` check, which @@ -2123,21 +3040,16 @@ shape depends on an open gate say so, and are written when that gate is Accepted reproduces the failure, and a single-processor run of the concurrency test passes against the broken protocol. The mitigation is two processors with a forced interleaving, plus the lease-expiry companion, landing **with** the fix. -- **The partition key is persisted but meaningless.** - [§ Ordering](../architecture/15-event-and-outbox.md#ordering) admits two shapes: the - aggregate id for an event about an instance, and `TenantId` chosen deliberately for a - tenant-wide fact. The failure modes are the other two — a key derived from `EventId`, - which is a partition of one, and a per-type default, which is one partition for every - tenant. The catalogued rule asserts non-blankness and detects neither, so **G3** asks - whether the derivation is checkable structurally, and the ordering criterion it - settles is what observes order. -- **Dead-lettering becomes invisible.** A terminal state that appears only in a database - column is silent data loss. The counter and the audit row are part of the deliverable, - and the audit row has no sanctioned writer until G7 closes — which is why G7 blocks - the packet rather than trailing it. -- **The two dead-letter designs are both implemented.** They contradict each other in - the two carriers that describe them, so an implementer who reads one and a reviewer - who reads the other both pass. G1 picks one and deletes the other. +- **Partition identity is mistaken for durable order.** ADR-0057 enforces the + declared aggregate/tenant key. ADR-0046 separately bounds deterministic eligible + selection; interleaved commits and deferred retries must remain proof subjects. +- **Terminal audit is detached from state.** The witness must come from the + actual fenced transition and bind draft tenant/organization/event. A failed + MUST insert rolls back terminal state and retries no business publication. +- **Consumer delivery leaks into the producer budget.** P02b-3 must durably accept + before acknowledgement and use its separate executor/ledger. Shared inbox + deduplication uses stable subscription identity, preserving healthy siblings + and completed outcomes during targeted replay. - **The Keycloak mapper regresses, or never reaches a warm workstation.** Realm JSON is regenerated by hand and by the admin console, and a mapper is easy to lose; worse, the import is consumed once, so a fresh-container test is green while every existing @@ -2192,9 +3104,9 @@ shape depends on an open gate say so, and are written when that gate is Accepted - The host-vs-JWT cross-check returns 404 and writes one row under Packet 9's existing operation key with a non-null actor and `metadata.assertionSource = jwt-claim`. - The sample integration event flows enqueue → outbox → dispatch → idempotent - consumption; two dispatchers holding live leases publish each message once; and a - **forced redelivery** produces one business effect — the check that exercises the - guard, which "no duplicate handling" alone does not. + consumption. Forced interleaving proves one live producer owner; publication + stays at least once. **Forced redelivery** produces one committed business + effect per subscription through the actual shared inbox boundary. - A poisoned consumer dead-letters visibly instead of looping: the terminal state for that consumer only, its counter, and its audit row. - A MUST-classified direct-write consumer produces exactly one row for its delivery, and diff --git a/docs/roadmap/phase-11-production-hardening.md b/docs/roadmap/phase-11-production-hardening.md index 96a290d7..68037b31 100644 --- a/docs/roadmap/phase-11-production-hardening.md +++ b/docs/roadmap/phase-11-production-hardening.md @@ -322,10 +322,11 @@ bounds request *cost* once a request is inside. Neither substitutes for the othe audit row — ships in [Phase 02b](phase-02b-events-auth.md). - Outbox dispatcher reliability under multi-pod load. The claim protocol is already specified — [Events and Outbox § The claim protocol](../architecture/15-event-and-outbox.md#the-claim-protocol) - writes the lease, [ADR-0006 Amendment 2](../decisions/0006-events-and-outbox.md) - assigns its columns and grant to Phase 02b's migration, and - [§ The simpler alternative, and its cost](../architecture/15-event-and-outbox.md#the-simpler-alternative-and-its-cost) - demotes the batch-held transaction to a fallback. What this phase adds is the + describes [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md)'s + single-row acquisition tokens and fenced transitions. P02b-2 implements that + producer protocol; P02b-3 owns durable subscriber execution. The + [simpler-alternative section](../architecture/15-event-and-outbox.md#the-simpler-alternative-and-its-cost) + records the supersession of the batch-held transaction fallback. This phase adds the multi-instance evidence the single-instance phase cannot produce: two real dispatcher processes across pods, lease-duration tuning against measured tail latency, and the lost-lease alert. @@ -559,7 +560,7 @@ same `ILiveClassProvider`. and by the glossary, and a cross-process transport does not change it. The retry and dead-letter contract being exercised here is [Phase 02b](phase-02b-events-auth.md)'s, unchanged. -- Two dispatcher processes in separate pods drain one pending batch with no duplicate +- Two dispatcher processes in separate pods drain the same queue with no duplicate business effect; a lease that expires mid-dispatch is reclaimed, the stale processor's writes are no-ops under the ownership fence, and `learnstack_outbox_lease_lost_total` both increments and pages. The lease duration is set against measured tail latency diff --git a/docs/standards/02-backend-coding.md b/docs/standards/02-backend-coding.md index e6701326..88513222 100644 --- a/docs/standards/02-backend-coding.md +++ b/docs/standards/02-backend-coding.md @@ -410,7 +410,8 @@ decides what happens *inside* the steps ADR-0033 named; the order is untouched: immediately through the announced ambient connection/transaction; step 7 neither buffers nor flushes messages nor dispatches domain events. The owner completes those events before MUST audit/commit. P02b-1 implements this - contract; P02b-2 owns later dispatch through `IEventBus`. + contract; P02b-2 Step 3 implements hosted dispatch through `IEventBus`, with + both independent review rounds complete. 8. **Handler** — domain logic; returns `Result`. **No** `throw new DomainException` for expected business-rule violations — use `Result.Fail(business_rule_violation, ...)`. The diff --git a/docs/standards/05-database.md b/docs/standards/05-database.md index 736a8b03..9cbb2dc6 100644 --- a/docs/standards/05-database.md +++ b/docs/standards/05-database.md @@ -28,6 +28,8 @@ Public-read additions derive from [ADR-0052](../decisions/0052-anonymous-public-read-boundary.md). Producer additions derive from [ADR-0057](../decisions/0057-transactional-event-production-and-completion.md). +Delivery/recovery additions derive from +[ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md). PostgreSQL schema, EF Core, and migration conventions. @@ -216,12 +218,13 @@ CREATE INDEX ix_courses_tenant_id_organization_id ON courses (tenant_id, organiz PostgreSQL evaluates referential integrity at **runtime** as a security-restricted operation on behalf of the table owner, and those RI triggers are **not subject to Row -Level Security**. (The DDL path is the opposite and matters in migrations: the scan -`ALTER TABLE … ADD CONSTRAINT` / `VALIDATE CONSTRAINT` performs runs as the -issuing role -under its policies, so a constraint added to a populated tenant-owned table validates -against the rows that role can see. § Data Migrations carries the consequence.) A -single-column `lessons.course_id → courses.id` therefore lets a row in tenant A +Level Security** ([PostgreSQL row-security documentation](https://www.postgresql.org/docs/18/ddl-rowsecurity.html)). +Constraint validation also checks existing rows regardless of their RLS visibility: +`ALTER TABLE … ADD CONSTRAINT` checks all existing rows unless `NOT VALID` skips +that scan; `VALIDATE CONSTRAINT` performs the deferred check +([PostgreSQL ALTER TABLE documentation](https://www.postgresql.org/docs/18/sql-altertable.html)). +An ordinary `UPDATE` backfill remains subject to RLS; see [Data Migrations](#data-migrations). +A single-column `lessons.course_id → courses.id` therefore lets a row in tenant A reference a row in tenant B: the child's own `WITH CHECK` passes because its `tenant_id` is A's, and the FK check passes because it can see B's row. The result is a permanent cross-tenant reference in the one system whose top-line threat is @@ -886,7 +889,7 @@ closed; a fifth role requires an ADR. | `learnstack_migration` | `ConnectionStrings:Migration` | the deploy job / `make migrate` only — **never** in API or worker runtime configuration | `dotnet ef database update`; **owns** every table | `NOBYPASSRLS`; `FORCE ROW LEVEL SECURITY` denies ownership any bypass | | `learnstack_app` | `ConnectionStrings:Default` | API host, worker host | every request-path and job-path `DbContext` | `NOBYPASSRLS` | | `learnstack_platform` | `ConnectionStrings:PlatformAdmin` | API host, worker host | `EnterPlatformAdminScope(reason)` and nothing else | `BYPASSRLS` | -| `learnstack_outbox_admin` | `ConnectionStrings:OutboxDispatcher` | the worker host that runs `OutboxProcessor` | `OutboxProcessor` and nothing else | `BYPASSRLS` | +| `learnstack_outbox_admin` | `ConnectionStrings:OutboxDispatcher` | API host only; never Seeder or web | confined dispatcher infrastructure only | `BYPASSRLS` | **Passwords arrive from the environment, through `\getenv`.** The four `:'name'` placeholders below are **psql client variables**, and the @@ -992,18 +995,28 @@ privileges implicitly. | `platform_host_to_tenant` | `SELECT, INSERT, UPDATE, DELETE` | `SELECT, INSERT, UPDATE, DELETE` | — | | `platform_killswitches` | `SELECT` | `SELECT, INSERT, UPDATE, DELETE` | — | | `idempotency_keys` | `SELECT, INSERT, UPDATE` | `SELECT, DELETE` | — | -| `outbox_messages` | `SELECT, INSERT` | `SELECT, DELETE` | `SELECT`, `UPDATE (processed_at, attempts, last_error, available_after)` | +| `outbox_messages` | `SELECT, INSERT` | `SELECT, DELETE` | `SELECT`, `UPDATE (processed_at, attempts, last_error, available_after, locked_by, locked_until, terminal_reason, dead_lettered_at)` (P02b-2 Step 1) | | `tenant_content_types` | `SELECT, INSERT, UPDATE, DELETE` | `SELECT` | — | | `tenant_level_taxonomies` | `SELECT, INSERT, UPDATE, DELETE` | `SELECT` | — | | `tenant_level_taxonomy_items` | `SELECT, INSERT, UPDATE, DELETE` | `SELECT` | — | | `customization_generations` | `SELECT, INSERT, UPDATE` | `SELECT` | — | -| `audit_log` | `SELECT, INSERT` | `SELECT, INSERT, DELETE`, `UPDATE (actor_email, ip_address, user_agent, before_state, after_state, changes)` | — | +| `audit_log` | `SELECT, INSERT` | `SELECT, INSERT, DELETE`, `UPDATE (actor_email, ip_address, user_agent, before_state, after_state, changes)` | `INSERT` only (P02b-2 Step 1) | | `audit_config` | `SELECT` | `SELECT` | — | | `courses` | `SELECT, INSERT, UPDATE` | `SELECT` | — | | `lessons` | `SELECT, INSERT, UPDATE` | `SELECT` | — | | `course_translations` | `SELECT, INSERT` | `SELECT` | — | | `lesson_translations` | `SELECT, INSERT` | `SELECT` | — | +ADR-0046 accepts the two P02b-2 cells above. Step 1 implements their migrations +and exact-grant proofs; both independent review rounds are complete in the +[delivery record](../roadmap/phase-02b-events-auth.md#p02b-2-step-1--decision-schema-and-confined-composition-2026-10-12). +Outbox status grants land in the Platform chain. The audit +INSERT grant lands in a new Audit-chain migration after `audit_log` exists, never +in initdb. No audit SELECT/UPDATE/DELETE, business access or role membership is +added. P02b-3 separately enumerates column-level scheduling grants on its delivery +ledger; it grants no envelope/payload SELECT to the bypass scheduler and only +RLS-protected acceptance/completion permissions to `learnstack_app`. + The four Education grants are **implemented in P02d-1 Step 2** and checked against the applied schema. Root UPDATE supports publication and concurrency; satellite UPDATE/DELETE and root DELETE have no Phase 02d command and are not @@ -1380,6 +1393,26 @@ Public read models consumed by other modules: ## Outbox +**ADR-0046 schema implemented in P02b-2 Step 1; both review rounds complete.** +The canonical +DDL below includes lease/terminal/identity/descriptor changes. Step 2 implements +fenced transitions and atomic terminal auditing; both review rounds passed. +Step 3 implements hosted execution and metrics; both independent review rounds +are complete. +Step 4 enforcement and local closeout are complete; all four steps completed both +independent review rounds. The [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) +owns CI/merge status. Existing +migrations remain immutable; a new Platform-chain migration replaces free-form +legacy `last_error` with `migration:legacy:redacted` before its byte CHECK. +This specific ADR-0046 Platform migration temporarily uses owner-only `NO FORCE` +for that platform-wide diagnostic backfill, then restores `FORCE` before commit. +The migration is atomic: any failure rolls back both the data change and the +security setting. It widens no runtime policy, role or grant and does not change +the per-tenant data-migration rule above. Invalid legacy identities fail the whole +migration for explicit repair; facts/state are not silently rewritten. Existing +attempts above six remain representable and route +to terminalization, never another normal publication. + ```sql CREATE TABLE outbox_messages ( id uuid PRIMARY KEY DEFAULT uuidv7(), -- producer supplies EventId @@ -1397,15 +1430,47 @@ CREATE TABLE outbox_messages ( processed_at timestamptz NULL, attempts int NOT NULL DEFAULT 0, last_error text NULL, - available_after timestamptz NOT NULL DEFAULT now() + available_after timestamptz NOT NULL DEFAULT now(), + locked_by text NULL, -- fresh canonical UUID acquisition token + locked_until timestamptz NULL, + terminal_reason text NULL, + dead_lettered_at timestamptz NULL, + CONSTRAINT ck_outbox_messages_identity CHECK ( + id <> '00000000-0000-0000-0000-000000000000'::uuid + AND tenant_id NOT IN ('00000000-0000-0000-0000-000000000000'::uuid, + '00000000-0000-7000-8000-000000000002'::uuid) + AND (organization_id IS NULL OR + organization_id <> '00000000-0000-0000-0000-000000000000'::uuid)), + CONSTRAINT ck_outbox_messages_attempts CHECK (attempts >= 0), + CONSTRAINT ck_outbox_messages_last_error_size CHECK ( + last_error IS NULL OR octet_length(last_error) <= 256), + CONSTRAINT ck_outbox_messages_lease CHECK ( + (locked_by IS NULL AND locked_until IS NULL) OR + (locked_by IS NOT NULL AND length(locked_by) > 0 AND locked_until IS NOT NULL)), + CONSTRAINT ck_outbox_messages_terminal_reason CHECK ( + terminal_reason IS NULL OR terminal_reason IN ( + 'unknown_type', 'retired_type', 'invalid_envelope', 'retry_exhausted')), + CONSTRAINT ck_outbox_messages_state CHECK ( + (processed_at IS NULL OR + (dead_lettered_at IS NULL AND terminal_reason IS NULL AND locked_by IS NULL)) + AND (dead_lettered_at IS NULL OR + (processed_at IS NULL AND terminal_reason IS NOT NULL AND locked_by IS NULL))) ); CREATE INDEX ix_outbox_messages_pending - ON outbox_messages (available_after) - WHERE processed_at IS NULL; + ON outbox_messages (available_after, occurred_at, id) + WHERE processed_at IS NULL AND dead_lettered_at IS NULL + AND terminal_reason IS NULL AND attempts < 6 AND locked_by IS NULL; CREATE INDEX ix_outbox_messages_tenant_pending ON outbox_messages (tenant_id, available_after) - WHERE processed_at IS NULL; + WHERE processed_at IS NULL AND dead_lettered_at IS NULL; +CREATE INDEX ix_outbox_messages_recovery + ON outbox_messages (available_after, occurred_at, id) + WHERE processed_at IS NULL AND dead_lettered_at IS NULL + AND terminal_reason IS NULL AND (locked_by IS NOT NULL OR attempts >= 6); +CREATE INDEX ix_outbox_messages_terminal + ON outbox_messages (available_after, occurred_at, id) + WHERE processed_at IS NULL AND dead_lettered_at IS NULL AND terminal_reason IS NOT NULL; ALTER TABLE outbox_messages ENABLE ROW LEVEL SECURITY; ALTER TABLE outbox_messages FORCE ROW LEVEL SECURITY; @@ -1425,7 +1490,8 @@ CREATE POLICY outbox_messages_isolation ON outbox_messages GRANT SELECT, INSERT ON outbox_messages TO learnstack_app; GRANT SELECT, DELETE ON outbox_messages TO learnstack_platform; GRANT SELECT ON outbox_messages TO learnstack_outbox_admin; -GRANT UPDATE (processed_at, attempts, last_error, available_after) +GRANT UPDATE (processed_at, attempts, last_error, available_after, + locked_by, locked_until, terminal_reason, dead_lettered_at) ON outbox_messages TO learnstack_outbox_admin; -- The OutboxProcessor connects as learnstack_outbox_admin, which holds BYPASSRLS so it @@ -1434,10 +1500,9 @@ GRANT UPDATE (processed_at, attempts, last_error, available_after) -- learnstack_app, inside the business transaction, and WITH CHECK pins the row to the -- caller's tenant. BYPASSRLS bypasses policies, not GRANTs — the column list above is -- what actually bounds the dispatcher, and SELECT ... FOR UPDATE SKIP LOCKED works with --- a column-level UPDATE grant, so no table-wide UPDATE is needed. When locked_by and --- locked_until land in Phase 02b, that migration extends this grant; a column added --- without extending it fails at runtime with `permission denied for table`. --- See ADR-0006 Amendment 1 and 15-event-and-outbox.md. +-- a column-level UPDATE grant, so no table-wide UPDATE is needed. The P02b-2 migration +-- adds the status columns and their grants together. INSERT-only on audit_log belongs +-- in the Audit chain, not this DDL. See ADR-0046 and 15-event-and-outbox.md. ``` [ADR-0057](../decisions/0057-transactional-event-production-and-completion.md) @@ -1454,8 +1519,10 @@ Additional payload fields are identifiers only; the complete canonical UTF-8 payload is capped at 262,144 bytes inclusive before INSERT. `metadata` is SQL NULL with no caller extension channel. This is application admission, not a new DDL CHECK or a claim about stored `jsonb` byte length. P02b-1 implements this -producer boundary; type-resolution outcomes, leases and inbox stay P02b-2/P02b-3 -work. +producer boundary. ADR-0046 accepts producer recovery and the shared inbox +contract; P02b-2 Step 2 implements producer transitions and terminal audit, +Step 3 implements hosted dispatch with both independent review rounds complete, and +P02b-3 owns inbox/delivery runtime. `organization_id` is `uuid NULL` and mirrors the organization context of the transaction that enqueued the row — @@ -1485,6 +1552,36 @@ on developer discipline. The earlier justification for nullability — that APIS so its contract cannot rest on a component that is not there yet. The architecture deep dive lives in [15-event-and-outbox.md](../architecture/15-event-and-outbox.md). + +Normal publication eligibility requires no processed/dead-lettered/terminal-request +state, fewer than six attempts, due availability, no lease awaiting recovery and +manifest/local-partition admission. Required Pending contracts and aliases are +excluded before acquisition and payload validation. Every transition checks ID, +current token and database-time unexpired lease. Recovery first replaces an +expired token; attempts count committed normal claims only. Terminalization +claims without incrementing attempts or publishing, then writes terminal state +and its MUST audit atomically. See +[ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md#producer-dispatch-and-recovery). + +## Inbox + +**Accepted contract; DDL/grants pending P02b-3.** One shared tenant-owned +`inbox_messages` deduplicates by `(tenant_id, subscription_id, event_id)`. +The transport's uncommitted claim precedes handler construction; completed inbox, +business state, produced outbox, MUST audit and fenced ledger completion share +one announced app-role transaction. Rollback removes a new claim; completed rows +are never reset by targeted replay. A separate delivery ledger owns attempts, +leases, terminal/replay state and retained validated envelopes. + +P02b-3 supplies canonical DDL, table-class assignments, exact RLS/grants and +ADR-0023's minting-list correction before runtime. It first chooses durable +envelope copies/references and their retention dependency; P02b-8 inventories +every retained store for erasure. The bypass scheduler receives only enumerated +identity/routing/lease/status column SELECT and status UPDATE, never payload +SELECT; the app role receives tenant-RLS-protected acceptance/completion, +including the named completion-column UPDATE, never cross-tenant scheduling. +The canonical RLS templates remain in [Table classes](#table-classes). + ## Idempotency `idempotency_keys` is the schema the durable `IIdempotencyStore` will read and diff --git a/docs/standards/09-error-handling.md b/docs/standards/09-error-handling.md index e2546573..7fa7ff6e 100644 --- a/docs/standards/09-error-handling.md +++ b/docs/standards/09-error-handling.md @@ -8,6 +8,8 @@ Public UI message mapping derives from [ADR-0027](../decisions/0027-frontend-i18 and the [Accepted P02d-6 package](../roadmap/phase-02d-walking-skeleton.md#p02d-6-decision-package-2026-10-09). Bootstrap response ownership and renderer Retry-After qualification derive from [ADR-0055](../decisions/0055-public-renderer-bootstrap-failures.md). +Delivery/recovery additions derive from +[ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md). How LearnStack represents, propagates, surfaces, and recovers from failures. @@ -562,9 +564,31 @@ exception types (`LiveKit.NET.LiveKitException`, `Stripe.StripeException`, ## Outbox -- Outbox dispatcher catches all exceptions from a handler, logs them, increments `attempts`, sets `available_after = now() + backoff`. -- Poison messages (max attempts reached) are dead-lettered. -- Dashboard alert when dead-letter count > 0. +**Accepted ADR-0046; P02b-2 producer runtime and enforcement locally complete.** +All four steps completed both independent review rounds. The [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) +owns CI/merge status. Subscriber delivery/retries and inbox +remain P02b-3. + +- Each committed normal claim increments producer `attempts` once, including an + unknown crash outcome. Six total claims use retry delays 1 s, 5 s, 30 s, 5 min + and 1 h. Expired claims recover with fresh tokens; claim six cannot publish again. +- Required Pending subscriptions preserve rows before acquisition/validation. + Only declared intentional absence may acknowledge without publication. +- Unknown/retired/invalid records request explicit terminalization. Terminal + requests never publish; terminal UPDATE and MUST audit share one transaction. + Audit refusal retains terminal work for retry and makes health unhealthy. +- Every success/failure/recovery/terminal transition is token- and expiry-fenced. + A timeout bounds the wait; cancel and retain the task-owned scope until actual + completion. Late results cannot acknowledge or overwrite a new owner. +- Persist only closed stage/reason/category codes, optional allowlisted SQLSTATE + and validated attempt traceparent in `last_error`, at most 256 UTF-8 bytes. + Dispatcher logs/traces likewise exclude exception text/stacks, payloads, raw + stored type and connection data. Existing transport exception telemetry is a + P02b-3 reconciliation obligation before Active production consumers. +- After P02b-3 durable acceptance, subscriber business failure spends only its + independent ledger budget. Failed/unknown acceptance still spends producer + attempts. Subscriber retry invokes one internal delivery executor, never a + broadcast; operator recovery routes and producer reset/fencing are Phase 03. ## Webhooks diff --git a/docs/standards/10-observability.md b/docs/standards/10-observability.md index c1883972..40826599 100644 --- a/docs/standards/10-observability.md +++ b/docs/standards/10-observability.md @@ -4,6 +4,8 @@ **Derives from:** [ADR 0002 — Initial Architecture](../decisions/0002-initial-architecture.md), [ADR 0006 — Events and Outbox](../decisions/0006-events-and-outbox.md) (outbox + tenant-context propagation across async boundaries), [ADR 0032 — Exception Handling, Logging, and Observability Architecture](../decisions/0032-exception-handling-logging-and-observability.md) (Sentry / OTel boundary, Serilog wiring, span attribute propagation, deployment-mode branching). Producer additions derive from [ADR-0057](../decisions/0057-transactional-event-production-and-completion.md). +Delivery/recovery additions derive from +[ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md). Three signals — logs, traces, metrics — bound by a single correlation id. Everything we ship is observable from day one. @@ -151,7 +153,7 @@ Auto-instrument: - Entity Framework Core queries. - MediatR commands and queries. - Hangfire job invocations. -- Outbox dispatcher batches. +- Outbox dispatcher attempts (P02b-2 Step 3; no batch span). - Valkey client calls. - SeaweedFS/S3 SDK calls. - LiveKit provider calls. @@ -223,9 +225,16 @@ backend treat business rejections as system failures. | `learnstack_http_request_duration_seconds` | histogram | route, method, status | | `learnstack_http_request_total` | counter | route, method, status | | `learnstack_db_query_duration_seconds` | histogram | module, operation | -| `learnstack_outbox_pending_count` | gauge | tenant | +| `learnstack_outbox_pending_count` | gauge | — | | `learnstack_outbox_dispatch_duration_seconds` | histogram | event_type | | `learnstack_outbox_dispatch_failed_total` | counter | event_type | +| `learnstack_outbox_deadletter_total` | counter | event_type | +| `learnstack_outbox_lease_lost_total` | counter | — | +| `learnstack_outbox_oldest_eligible_age_seconds` | gauge | — | +| `learnstack_outbox_deferred_count` | gauge | — | +| `learnstack_outbox_heartbeat_age_seconds` | gauge | — | +| `learnstack_inbox_dedup_total` | counter | module, event_type (P02b-3 reserved) | +| `learnstack_inbox_deadletter_total` | counter | module, event_type (P02b-3 reserved) | | `learnstack_job_duration_seconds` | histogram | job_name | | `learnstack_job_failed_total` | counter | job_name | | `learnstack_provider_request_duration_seconds` | histogram | provider, operation, outcome | @@ -244,6 +253,29 @@ backend treat business rejections as system failures. | `learnstack_entitlement_source_total` | counter | `source` | | `learnstack_entitlement_unresolved_total` | counter | — | +Outbox metrics above are implemented in P02b-2 Step 3 under ADR-0046; +both independent review rounds are complete. Inbox counters remain P02b-3 and job +metrics P02b-4. `event_type` comes only from the +finite declared registry plus fixed unknown/retired/invalid categories. Never +label event metrics by tenant, event, partition, subscription or trace identity. + +Pending count includes all unprocessed, non-dead-lettered rows, including delayed, +leased, terminal-requested and manifest-deferred work. Oldest eligible age uses +the exact normal claim predicate, excluding future, exhausted, terminal-requested, +live/unrecovered-expired leases and manifest/local partition holds. It is zero +when no row qualifies and never negative. Deferred count isolates manifest holds. +Heartbeat advances after actual bounded loop work, including empty polls. + +Gauges read immutable memory snapshots and perform no SQL. Refresh follows +completed iterations, targets 5 s idle and becomes stale at the approved 45 s +health bound. Failure retains the prior snapshot and reports unhealthy. The +[P02b-2 bounds](../roadmap/phase-02b-events-auth.md#runtime-bounds-and-observation) +own deadlines and task limits. Dispatcher-owned logs/spans use validated +correlation and bounded outcomes, without exception text/stacks, payload, raw +stored type or connection data. P02b-3 reconciles existing transport/consumer +exception telemetry before Active consumers; no current transport/consumer +minimization is claimed. + Cache `cache.name` is a governed, low-cardinality family from the Standards 20 inventory (`hub:host-map`, `hub:entitlement`, `identity:permissions`, `tenancy:feature-flags`, `tenancy:settings`, `audit:config`, or @@ -279,8 +311,11 @@ touched. | Check | Registered as | Unhealthy when | |---|---|---| | Audit write path | `audit` | The most recent MUST-class standalone audit write failed and no later one has succeeded | +| Outbox dispatcher (P02b-2 Step 3) | `outbox` | No actual progress or observation refresh for 45 s, any retained callback overrun, or terminal-audit failure | -The `audit` check is registered in Phase 02a Packet 9 and **not mapped**: `/healthz` stays +The `audit` check is registered in Phase 02a Packet 9; P02b-2 Step 3 registers +`outbox`, with both independent review rounds complete. +Both are **not mapped**: `/healthz` stays a liveness probe, because a process that cannot write audit rows is still worth leaving alive. The readiness surface that reads the check, and the deployment-level backstop that stops serving past a configured unhealthy window, are demand-gated to @@ -347,7 +382,7 @@ The first set of dashboards (Phase 11 deliverable): |-------|-----------|----------| | API 5xx rate | > 1% for 5 min | page | | API latency p95 | > 2× budget for 10 min | page | -| Outbox pending count | > 10k for 10 min | page | +| Outbox oldest eligible age / heartbeat | > 45 s; validate production window in Phase 11 | page | | Background job failures | > 5% for 10 min | page | | Classroom join failure | > 5% for 5 min | page | | Disk usage (DB) | > 80% | warn | diff --git a/docs/standards/11-security.md b/docs/standards/11-security.md index 70e76745..6c84817a 100644 --- a/docs/standards/11-security.md +++ b/docs/standards/11-security.md @@ -24,6 +24,8 @@ Public-read additions derive from ingress/caller and anonymous visitor budgets derive from [ADR-0053](../decisions/0053-trusted-public-server-rendering.md); coordinated accounting, native admission and redirect/query rules derive from [ADR-0054](../decisions/0054-bounded-public-renderer-admission.md). +Delivery/recovery additions derive from +[ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md). Producer announcement and common execution ownership derive from [ADR-0057](../decisions/0057-transactional-event-production-and-completion.md). @@ -239,6 +241,36 @@ for the full strategy. Standards-side: `SET LOCAL` inside the ambient transaction — see § Tenant Context immediately below, which is the single authority for that placement. +### Confined event dispatcher + +ADR-0046 accepts this P02b-2 boundary. Step 1 implements source/role guards; +both review rounds are complete. Step 3 implements mandatory worker startup +resolution and connectivity checks; both independent review rounds are complete. +Step 4 credential/publication enforcement and local closeout are complete; all +four steps completed both independent review rounds. The [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) +owns CI/merge status. The named keyed, lazy +dispatcher source belongs only to the API host. Configuration validates the exact +username without connecting or +echoing parse input. Worker startup must verify connectivity. Every new physical +connection verifies exact session/current +`learnstack_outbox_admin`, BYPASSRLS, NOSUPERUSER, NOCREATEDB, NOCREATEROLE, +NOREPLICATION and no outgoing role membership or incoming membership by a +non-superuser, directly or transitively. Superusers inherently reach every role; +this check detects grant drift on a new physical connection, not continuous +revocation of an existing session. No Seeder/web forwarding or shared app +credential is allowed. [Database Standards](05-database.md#grant-matrix) +owns exact status grants and audit INSERT-only; no business access or audit read, +update or delete is admitted. + +Routine confined poll/claim/mark receives ADR-0046's bounded audit exception; +terminal state is MUST/SecurityEvent and atomic with its row-bound witness audit. +INSERT-only and GUCs cannot tenant-bind a bypass credential. Real mismatch controls +must roll back terminal state and audit together. Existing platform-admin entry +audit and gate stay unchanged. P02b-3 opens separate app-role business scopes, +restores validated tenant/organization and announces RLS context before reading +the envelope and constructing a handler. The ordinary executor refuses privileged +subscriptions until their Phase 03 execution contract exists. + ## Tenant Context **This section is the single authority for RLS session-variable placement.** Every other @@ -671,7 +703,8 @@ and the edge has no way to know it. ## Audit Log -Every privileged operation writes an audit log entry per +Every privileged operation writes an audit log entry, subject to the confined +routine dispatcher exception above, per [ADR-0033 Audit Durability Model](../decisions/0033-audit-durability-model.md) (supersedes ADR-0016), [ADR-0044 The Audit Write Path](../decisions/0044-audit-write-path.md) and the diff --git a/docs/standards/18-audit-coverage.md b/docs/standards/18-audit-coverage.md index 21d5cfe1..10fdceb0 100644 --- a/docs/standards/18-audit-coverage.md +++ b/docs/standards/18-audit-coverage.md @@ -10,6 +10,8 @@ subsystem's context), [01-architecture-standards.md](01-architecture-standards.md). Producer additions derive from [ADR-0057](../decisions/0057-transactional-event-production-and-completion.md). +Delivery/recovery additions derive from +[ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md). This standard defines **which operations must be audited**, **what an audit entry contains**, **how long entries are retained**, and how each module signs up for its @@ -151,18 +153,20 @@ will raise them — which this standard requires it to do. - **`(planned)` is re-checked, not an escape.** A row still marked `(planned)` whose command has since shipped **fails**. The marker is a claim the rule tests on every run, which is what stops the scoping from becoming a hole. -- **`(off-path)` is for operations that are not MediatR requests at all.** Seven rows - carry it: Tenancy's `platform.admin_scope.enter`, `tenancy.killswitch.toggle`, +- **`(off-path)` is for operations that are not MediatR requests at all.** Nine rows + carry it: Tenancy's `platform.admin_scope.enter`, `platform.outbox.dead_letter`, + `platform.delivery.dead_letter`, `tenancy.killswitch.toggle`, `tenancy.entitlement.refresh` and the two tenant-assertion keys of § Baseline Coverage, and Audit's `audit.redaction.apply` and `audit.purge.apply`. They sit outside the request-type join in **both** directions and register by slug rather than by type — the - three whose writer ships through `DeclareOffPath` today, the other four when their - writer lands. + four whose writer ships through `DeclareOffPath` today, the other five when their + writer lands. P02b-2 Step 2 registers the shipped producer terminal writer; + subscriber terminal registration remains planned for P02b-3. - **One operation, one row.** A slug is classified in exactly one row across every matrix. A copy is a second answer that passes every comparison the moment it agrees, so the matrix → catalogue direction refuses it, and the catalogue → matrix direction compares every row that carries a slug rather than the first. -- **A row can carry both markers**, and four do. `tenancy.killswitch.toggle` is +- **A row can carry both markers**, and five do. `tenancy.killswitch.toggle` is `(off-path)` because every write runs inside `EnterPlatformAdminScope(reason)` rather than through the pipeline, and `(planned)` because Packet 9 ships the table, the policies and the read path but no writer — @@ -383,7 +387,7 @@ Rules: and calls `IAuditStateCapture.Clear()`. Accepted ADR-0057 separates audit ownership from per-invocation producer receipts. A controlled direct owner proves common entry in P02b-1; production delivery - opens it before handler construction in P02b-3 after G1, refuses an + opens it before handler construction in P02b-3 under ADR-0046, refuses an unregistered mutating handler and declares direct one-root intents. Nested commands keep their own intents without a duplicate business row. Modules see none of this plumbing; the MUST floor and standalone refusal posture survive. @@ -492,7 +496,26 @@ table above is the policy those deliverables implement, not a description of tod - Failed `denied` outcomes are audited even though no state changed. - Background jobs that mutate MUST-audit resources receive the `actor` via job payload (operator id, or the seed of a system actor with a stable id) and write the entry under that identity. - Integration event handlers that mutate state are treated as actors of type `system` and audited accordingly. -- Platform-bypass code paths (`IgnoreQueryFilters()` etc.) write a `security-event` entry on every invocation. +- Platform-bypass code paths (`IgnoreQueryFilters()` etc.) write a `security-event` + entry on every invocation, with ADR-0046's confined routine dispatcher exception: + polling/claim/mark uses exact grant confinement, bounded structured attempt + outcomes, metrics/health and MUST-audited terminal transitions. This exception + changes no platform-admin pre-entry audit or other bypass operation. +- `WriteEventTerminalAsync(AuditEntryDraft, EventTerminalTransition, DbConnection, + DbTransaction, CancellationToken)` is the narrow fifth write, implemented in + P02b-2 Step 2; both independent review rounds passed. + Only a successful fenced UPDATE result mints the restricted witness binding + actual tenant/org/event/acquisition/kind/reason/attempts/subscription and live + transaction. Validate declared MUST/SecurityEvent operation and matching draft + before INSERT; mismatch rolls back both writes. No transaction is opened or + committed by the seam. `PostgresAuditStore` remains the sole audit SQL writer. + Tenancy owns off-path `platform.outbox.dead_letter` (P02b-2) and reserved + `platform.delivery.dead_letter` (P02b-3); register only with each writer. + Outcome success records terminal state, not publication. Invalid causal metadata + is omitted and a fresh validated attempt trace is used. Closed terminal values + live in bounded metadata; `AuditEntryDraft.Reason` remains for platform entry. + Audit refusal retries terminalization without delivery. Because the bypass + role ignores RLS, witness/call-path proof binds identity, not announced GUCs. - Entering `EnterPlatformAdminScope(reason)` writes that row through `IAuditStore.WritePlatformScopeAsync` — a fourth **write** method, not an update method — on the scope's own platform-role connection and **before** the operation @@ -503,7 +526,8 @@ table above is the policy those deliverables implement, not a description of tod disposed without `CommitAsync` rolls back, taking the record of the bypass with it. That is the row [Database Standards § How `EnterPlatformAdminScope(reason)` reaches `learnstack_platform`](05-database.md) calls "committed on its own". -- **The row's tenant is the tenant the ambient transaction announced**, never a value off +- **An ordinary audit intent's tenant is the tenant the ambient transaction + announced**, never a value off the request payload: `ITenantContext.TenantId` under a resolved context, `IProvisionsTenant.ProvisioningTenantId` for provisioning under an unresolved one, and `TenantId.PlatformSentinel` — the reserved `00000000-0000-7000-8000-000000000002`, not diff --git a/docs/standards/20-infrastructure-stack.md b/docs/standards/20-infrastructure-stack.md index 25c76716..a10105e7 100644 --- a/docs/standards/20-infrastructure-stack.md +++ b/docs/standards/20-infrastructure-stack.md @@ -17,6 +17,8 @@ Public-read additions derive from [ADR-0052](../decisions/0052-anonymous-public-read-boundary.md). Producer additions derive from [ADR-0057](../decisions/0057-transactional-event-production-and-completion.md). +Delivery/recovery additions derive from +[ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md). Three audit decisions are referenced here but **not owned** here, and the split matters because they are easy to conflate: @@ -27,8 +29,9 @@ intent multiplicity, capture, classification, and which assembly each type lives [ADR-0028](../decisions/0028-audit-log-partition-management.md) owns the `audit_log` **partition lifecycle**. This standard records that partitioning is demand-gated, where the audit types live, and that modules never write `audit_log` directly. Assembly -placement and the four-method write path in § Audit Plumbing are ADR-0044's, which is -why it — alone of the three — is also named in the header above. +placement and the delivered four-method write path in § Audit Plumbing are ADR-0044's; +ADR-0046 adds the confined fifth seam implemented in P02b-2; both write-path +authorities are named in the header above. This standard defines how application code uses the foundation infrastructure introduced in the 2026-05-18 redesign: Dapr building blocks, the APISIX gateway, the Hub HTTPS @@ -78,10 +81,11 @@ Rules that follow from this: written against the port and holds for the default implementation too. If a rule only makes sense for the vendor adapter, it is a rule about the adapter and belongs in that adapter's section, not in module guidance. -- **`InProcessEventBus` is a first-class transport, not a stub.** Same - `IIntegrationEventHandler`, same `IInboxGuard`, same tenant-context restoration as - the durable path. A development path that skips those never exercises the isolation - code, and every consumer would end up needing two implementations. +- **`InProcessEventBus` is a first-class transport, not a stub.** It uses + `IIntegrationEventHandler` and restores tenant context before handler resolution. + Durable acceptance and the shared transport-owned `IInboxGuard` remain P02b-3 + work under [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md#subscriber-acceptance-and-inbox-ownership). + Phase 11's adapter must preserve that consumer contract. - **A demand-gated block is not "deferred".** It qualifies only with all four of: a port, a working default implementation, an owning phase, and a written trigger. If a trigger fires earlier than its phase, the item moves to the phase where it fired and @@ -181,10 +185,15 @@ adapter replaces `InProcessEventBus`. [21-architecture-tests-catalogue.md](21-architecture-tests-catalogue.md). - Hub-side topics use the same `learnstack.hub.*` prefix (`learnstack.hub.entitlement`, `learnstack.hub.custom-domain.activated`). -- Consumers implement `IIntegrationEventHandler` and **must** invoke - `IInboxGuard.IsAlreadyProcessedAsync` before any business logic. Phase 02b adds the - architecture test `Integration_Event_Handlers_Use_InboxGuard` with the first real - consumers; it is registered in the catalogue today. +- Consumers implement `IIntegrationEventHandler`. Accepted ADR-0046 assigns + shared inbox claim/completion to the transport, before handler construction. + `Integration_Event_Handlers_Use_InboxGuard` is Registered for P02b-3's nonempty + production census and real atomicity/duplicate proofs; handlers do not own the guard. +- Construction-free readiness declares Pending, Active or intentional absence. + Any required Pending subscription holds its contract/aliases before acquisition + and validation; Active declarations require actual handler registration. + P02b-3 publish success means every required subscription was durably accepted; + later execution failures belong to independent subscription ledgers. - Cross-instance L1-cache invalidation rides on `learnstack.cache.invalidation` (a small payload of `(tenant_id, cache_key)`). Modules that maintain L1 caches subscribe here. @@ -430,16 +439,28 @@ Rules: permits no recursive mutation. Produced-work receipts propagate to ancestors; a returning `Result.Fail` poisons the shared unit before frame resolution. Joiners neither drain nor clear the owner; terminal paths discard state. -- The shared outbox table is RLS-protected; `OutboxProcessor` connects with the - `learnstack_outbox_admin` role that bypasses RLS. -- `OutboxProcessor` uses `FOR UPDATE SKIP LOCKED` to allow horizontal scaling. Each - message has its own transaction; one failure does not roll back the batch. -- Retry backoff: 1s, 5s, 30s, 5min, 1h. After max retries (5), the message is - dead-lettered and surfaces via the `OutboxStatusEndpoints` admin API; manual - intervention is required. -- Consumers are **idempotent** via per-module `inbox_messages` tables. The - `IInboxGuard.MarkAsProcessed(eventId, eventTypeName)` write enrolls in the consumer's - business `DbContext` so the inbox marker and the business write commit atomically. +- ADR-0046 accepts a single just-in-time normal claim with a fresh acquisition + token and database-time lease. Every transition is token/expiry fenced; expired + claims recover before publication. Six claims total use delays 1 s, 5 s, 30 s, + 5 min and 1 h. Terminal requests never publish or increment attempts; explicit + terminal state commits atomically with MUST audit through the confined fifth seam. +- P02b-2's API-owned dispatcher source uses the existing bypass role with exact + status grants and audit INSERT-only. [Database Standards](05-database.md#grant-matrix) + owns grants and canonical schema. No batch-held fallback, EF set-based write, + arbitrary type loading or free-form persisted exception descriptor is permitted. +- P02b-3 implements the shared transport-owned inbox keyed by tenant/subscription/ + event. Its claim precedes handler construction; business state, inbox completion, + produced outbox, MUST audit and fenced ledger completion share the app-role owner. + Independent delivery ledgers retain validated envelopes and own subscriber + retries/terminal/replay. Targeted replay invokes one internal executor, never + broadcast; the outbox processor remains the only production publisher. +- Operator routes, permissions and producer reset/fencing belong to Phase 03. + Brokers and retention purge remain demand-gated Phase 11. P02b-2 Step 3 + implements hosted producer dispatch, with both independent review rounds complete; + Step 4 enforcement and local closeout are complete; all four steps completed + both independent review rounds. The [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) + owns CI/merge status. Inbox and + effectively-once consumer delivery remain P02b-3. Full deep dive: [15-event-and-outbox.md](../architecture/15-event-and-outbox.md). @@ -627,9 +648,9 @@ are the Hub's public API, governed by the Hub repository. classification per [18-audit-coverage.md](18-audit-coverage.md)) and the MediatR `AuditLogBehavior`; there is no per-module audit code. - `IAuditStore` is the only sanctioned write path; the architecture test - `Modules_Do_Not_Write_AuditLog_Directly` is the rule that enforces it. The store - carries **four** write - methods and no update method: `WritePendingAsync` on the ambient transaction, the two + `Modules_Do_Not_Write_AuditLog_Directly` is the rule that enforces it. The + original four write methods retain their contracts, with no update method: + `WritePendingAsync` on the ambient transaction, the two standalone writers (`WriteStandaloneAsync`, `WriteBestEffortAsync`), and `WritePlatformScopeAsync`, whose only caller is `EnterPlatformAdminScope(reason)` — it writes its security-event row on its own platform-role connection before the @@ -638,6 +659,18 @@ are the Hub's public API, governed by the Hub repository. ([ADR-0044 § 10](../decisions/0044-audit-write-path.md), [05-database.md § How `EnterPlatformAdminScope(reason)` reaches `learnstack_platform`](05-database.md)). +Accepted ADR-0046 adds the confined fifth method +`WriteEventTerminalAsync(AuditEntryDraft, EventTerminalTransition, DbConnection, +DbTransaction, CancellationToken)`, implemented in P02b-2 Step 2 with both +independent review rounds complete. The delivered store has five write methods. +Its restricted witness comes only from the actual successful fenced terminal UPDATE. +The store validates declared operation/class/type, row identity and live caller +transaction, then INSERTs without opening/committing a transaction. Existing four +writes retain their contracts. The terminal UPDATE and audit commit together, or +both roll back. Tenancy owns producer terminal audit in P02b-2 and subscriber +terminal audit in P02b-3. See +[Audit Coverage](18-audit-coverage.md#required-behaviours). + ## Background Jobs and Hangfire - Long-running domain work (cohort enrollments, recording transcoding) goes on diff --git a/docs/standards/21-architecture-tests-catalogue.md b/docs/standards/21-architecture-tests-catalogue.md index a0dcfb7f..87e1e4de 100644 --- a/docs/standards/21-architecture-tests-catalogue.md +++ b/docs/standards/21-architecture-tests-catalogue.md @@ -12,7 +12,12 @@ [ADR-0056 Public Demo Browser and Lighthouse Enforcement](../decisions/0056-public-demo-browser-and-lighthouse-enforcement.md) (accepted 2026-10-11; browser/audit replacement proofs implemented), [ADR-0057 Transactional Event Production and Completion](../decisions/0057-transactional-event-production-and-completion.md) -(producer lifecycle, payload and production-census proofs implemented). The +(producer lifecycle, payload and production-census proofs implemented), +[ADR-0046 Durable Event Delivery and Recovery](../decisions/0046-durable-event-delivery-and-recovery.md) +(P02b-2 transition, terminal-witness and hosted runtime proofs implemented; +all four steps completed both independent review rounds and local closeout; +the [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) +owns CI/merge status; P02b-3 consumer proofs remain pending). The catalogue grows as subsequent ADRs and phases land their tests; per-test ownership stays with the originating ADR / standard. @@ -105,11 +110,12 @@ not implemented is the failure mode this column exists to prevent. ### Implemented today -**173 test methods run in +**194 test methods run in [`backend/tests/LearnStack.Tests.Architecture`](../../backend/tests/LearnStack.Tests.Architecture),** shipped by [Phase 01](../roadmap/phase-01-repository-tooling.md), [Phase 02a Packets 2–3](../roadmap/phase-02a-kernel-tenancy.md), Packet 4, Packet 6, Packet 7, -Packet 8, Packet 9, Packet 10, P02d-1, P02d-2, P02d-3, P02d-4 Steps 1–3 and P02b-1. +Packet 8, Packet 9, Packet 10, P02d-1, P02d-2, P02d-3, P02d-4 Steps 1–3, P02b-1 +and P02b-2 Steps 2–4. Methods are not rows: a `[Theory]` is one row and many cases, and most rows pair a rule with the companion assertion that stops it passing vacuously. @@ -136,8 +142,8 @@ two fifths of its subject is the defect this section is about. It also refuses a test class that exists nowhere, because otherwise a renamed or deleted file drops its entries out of the subject instead of failing. -**183 rules in this catalogue are Implemented, and 113 of them are in that assembly.** -The other 70 are no less binding, and most could not live there. The table says where +**192 rules in this catalogue are Implemented, and 118 of them are in that assembly.** +The other 74 are no less binding, and most could not live there. The table says where and why, and deliberately carries no per-row count: those are the numbers nothing recomputes, and the first version of this table claimed "three rules" for a suite that holds ten. @@ -1787,14 +1793,34 @@ diagnostic above row security, not the boundary, as its own note says. #### `LearnStack_OutboxAdmin_Role_OnlyUsedBy_OutboxProcessor` -- **Asserts:** `ConnectionStrings:OutboxDispatcher` is resolved by `OutboxProcessor` - and nothing else. A `GRANT` names a role, not a code path — every handler in the - API process runs as the same role — so code-path confinement of a `BYPASSRLS` - credential is carried here or nowhere. -- **Source:** [05-database.md § GRANT matrix](05-database.md); ADR-0003 Amendment 3. -- **Type:** NetArchTest + DI registration inspection. **Kind:** structural. -- **Status:** **Awaiting backfill** — cited by the standard, no dispatcher yet. -- **Phase:** 02b. +- **Asserts:** only the confined dispatcher infrastructure resolves the named keyed, + lazy `ConnectionStrings:OutboxDispatcher` source in the API host. Seeder/web and + arbitrary keyed/service-provider resolution are refused. Physical connections + verify the exact outbox role, bounded attributes and memberships, including + direct/transitive incoming membership from other non-superuser roles. A nonempty + production census has planted clean/dirty resolution and registration controls. + Named worker/store source readers require unique literal method signatures, + visibility and staticness; a same-name overload grants no authority. Real + role/grant probes include valid and elevated/transitive negative controls. +- **Source:** [05-database.md § GRANT matrix](05-database.md#grant-matrix); + [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md). +- **Type:** architecture/DI census and real PostgreSQL role probes. + **Kind:** structural + startup + behavioural. +- **Status:** **Implemented** (`OutboxDispatchConventionTests`, + `OutboxDispatcherRoleGuardTests`, `OutboxProcessorTests`): exact nonempty + whole-production credential/factory/consumer census, keyed registration and + constructor bindings, planted clean/dirty helper/delegate/async controls, and + real PostgreSQL role/grant/startup refusal proofs. Exact unique literal + source-reader signatures, visibility and staticness reject named-method overloads; + those focused controls and both fresh fix reviews passed at `606362b`. + [OutboxDispatchConventionTests.Credential_census_rejects_compiled_connection_string_indexers_and_composed_keys](../../backend/tests/LearnStack.Tests.Architecture/OutboxDispatchBoundaryTests.cs) + plants actual compiled concatenation, interpolation, formatting and configuration + indexer calls, with clean configuration/dictionary/diagnostic controls. + [OutboxDispatcherRoleGuardTests.App_membership_in_the_dispatcher_is_refused_directly_and_transitively_then_restoration_connects](../../backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatcherRoleGuardTests.cs) + proves an existing app session can escalate through the planted grant, then + verifies refusal on a new dispatcher physical connection and successful + connection after grant removal, for both direct and transitive membership. +- **Phase:** 02b, P02b-2. #### `CoreInfrastructure_DoesNotDependOn_AnyModule` @@ -2631,16 +2657,18 @@ which decides identity, multiplicity, capture and classification; the reverse, which also carries the anti-rot check and a guard against sweeping no rows at all. - **Off the request path.** Some audited operations are not MediatR requests at all. - Seven rows carry `(off-path)`: Tenancy's `platform.admin_scope.enter`, + Nine rows carry `(off-path)`: Tenancy's `platform.admin_scope.enter`, + `platform.outbox.dead_letter`, `platform.delivery.dead_letter`, `tenancy.killswitch.toggle`, `tenancy.entitlement.refresh`, `tenancy.tenant_assertion.reject` and `tenancy.tenant_assertion.anonymous_burst` — snake_case within each segment since [ADR-0036 Amendment 7](../decisions/0036-tenant-resolution-trusted-inputs.md), so that one parser reads every audit slug and every permission key — and Audit's `audit.redaction.apply` and `audit.purge.apply`. They sit outside the request-type join - in both directions and register by slug: the three whose writer ships through - `DeclareOffPath` today, the other four — which also carry `(planned)` — when their - writer lands. + in both directions and register by slug: the four whose writer ships through + `DeclareOffPath` today, the other five — which also carry `(planned)` — when their + writer lands. ADR-0046 assigns the producer terminal writer to P02b-2 and the + subscriber terminal writer to P02b-3; acceptance alone registers neither slug. Stated as one unconditional join, the rule is red on its first run: the two shipped matrices classify many more operations than `backend/src/Modules` has request types, and @@ -2878,7 +2906,23 @@ which decides identity, multiplicity, capture and classification; - **Source:** ADR-0033 (carried from ADR-0016); [31-audit-subsystem.md § 10](../architecture/31-audit-subsystem.md). - **Type:** xUnit + source / migration scan. **Kind:** structural. -- **Status:** **Implemented** (`AuditConventionTests`, Packet 9 step 8), in three parts: a source sweep over `backend/src` for an `UPDATE` or `DELETE` targeting `audit_log` — with or without `ONLY`, a schema qualifier or identifier quotes, which the first pattern missed until the fourth review of Packet 9 measured it; a companion, `The_AppendOnly_Sweep_Can_Actually_Fail`, that checks the pattern against the shapes it must catch and the shapes it must not, because with no offending statement anywhere the sweep passes whether it works or matches nothing; and a reflection check, `IAuditStore_Exposes_No_Update_Method`, that `IAuditStore` exposes exactly four write methods and no update. None of the three sanctioned redaction sites exists yet — they land in Phase 03 and Phase 11 — so the exemption set ships with the rule **empty**, as exact paths: the first site to land adds its own path, so it is exempted by name rather than the pattern widened. (A substring predicate stood there first, and exempted any Audit-module file whose path contained "Redaction".) Since the fifth review of Packet 9 the sweep also asserts its premise — it must have read `PostgresAuditStore`'s own `INSERT INTO audit_log` — because with no offender anywhere, a sweep that read nothing passed as well. +- **Status:** **Implemented** (`AuditConventionTests`, Packet 9 step 8), in three parts: + a source sweep over `backend/src` for an `UPDATE` or `DELETE` targeting `audit_log` — + with or without `ONLY`, a schema qualifier or identifier quotes, which the first + pattern missed until the fourth review of Packet 9 measured it; a companion, + `The_AppendOnly_Sweep_Can_Actually_Fail`, that checks the pattern against the shapes + it must catch and the shapes it must not, because with no offending statement anywhere + the sweep passes whether it works or matches nothing; and a reflection check, + `IAuditStore_Exposes_No_Update_Method`, that `IAuditStore` exposes exactly five write + methods and no update (the additive fifth terminal seam ships in P02b-2 Step 2). None + of the three sanctioned redaction sites exists yet — they land in Phase 03 and Phase + 11 — so the exemption set ships with the rule **empty**, as exact paths: the first + site to land adds its own path, so it is exempted by name rather than the pattern + widened. (A substring predicate stood there first, and exempted any Audit-module file + whose path contained "Redaction".) Since the fifth review of Packet 9 the sweep also + asserts its premise — it must have read `PostgresAuditStore`'s own `INSERT INTO + audit_log` — because with no offender anywhere, a sweep that read nothing passed as + well. - **Phase:** 02a (Packet 9). #### `Every_PII_Module_RegistersUserReferenceLocator` @@ -3165,13 +3209,20 @@ Introduced by [Phase 02b](../roadmap/phase-02b-events-auth.md). #### `Integration_Event_Handlers_Use_InboxGuard` -- **Asserts:** every `IIntegrationEventHandler` calls - `IInboxGuard.IsAlreadyProcessedAsync` before any business logic. -- **Source:** [20-infrastructure-stack.md § `IEventBus`](20-infrastructure-stack.md); - [ADR-0006](../decisions/0006-events-and-outbox.md). -- **Type:** xUnit + IL / source scan. **Kind:** structural. -- **Status:** **Registered.** -- **Phase:** 02b. +- **Asserts:** the transport claims the shared inbox before constructing each + declared subscription's handler. Business state, completed inbox, produced + outbox, MUST audit and fenced ledger completion share the announced app-role + transaction. Concurrent duplicates serialize; two same-module subscriptions + have distinct identities. Stale tokens, expired leases and a closed timeout + gate roll back all business work. Completed inbox rows survive targeted replay. + A nonempty production census and planted dirty/clean controls prevent an empty + handler set from appearing enforced. +- **Source:** [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md#subscriber-acceptance-and-inbox-ownership), + superseding ADR-0006/0010's handler-called guard assertion. +- **Type:** architecture call-path census and integration tests (Testcontainers). + **Kind:** structural + behavioural. +- **Status:** **Registered** — transport-owned inbox/delivery remains unimplemented. +- **Phase:** 02b, P02b-3. #### `Integration_Events_Inherit_From_IntegrationEventBase` @@ -3196,15 +3247,16 @@ Introduced by [Phase 02b](../roadmap/phase-02b-events-auth.md). unrelated-ID and missing-declaration offenders plus clean controls. Behavioral cases compare actual source/key values. The envelope reads the event's key; it cannot hold a second answer. This establishes derivation, not durable order; - G3's publish-order half remains P02b-2. + ADR-0046 defines eligible-selection order and its limits. - **Source:** [ADR-0057](../decisions/0057-transactional-event-production-and-completion.md); + [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md#producer-dispatch-and-recovery); [15-event-and-outbox.md](../architecture/15-event-and-outbox.md#ordering). - **Type:** xUnit + reflection/IL/source scan and value probes. **Kind:** structural + behavioural. - **Status:** **Implemented** (`IntegrationEventContractTests`, `OutboxProducerTests`); stored-source IL grammar, independently varied identifiers, dirty/clean controls and persisted keys. Unsupported derivations fail loudly. -- **Phase:** 02b, P02b-1 (derivation); P02b-2 (ordering). +- **Phase:** 02b, P02b-1 (derivation); P02b-2 (selection-order bounds). #### `Outbox_Enqueue_Uses_The_Announced_Transaction` @@ -3296,26 +3348,164 @@ Introduced by [Phase 02b](../roadmap/phase-02b-events-auth.md). #### `OutboxProcessor_NeverBlocks_OnSingleMessageFailure` -- **Asserts:** one poisoned message does not prevent the rest of its batch from being - dispatched. +- **Asserts:** ordinary publication failure permits unrelated eligible work to + progress. A callback ignoring cancellation retains its live task-owned scope, + blocks its own partition and makes health unhealthy; unrelated keys progress + within the approved four-task bound. Saturation stops acquisition, late results + cannot acknowledge, and shutdown waits finitely without disposing live resources. - **Source:** [15-event-and-outbox.md § Architecture tests](../architecture/15-event-and-outbox.md); Phase 02b deliverable. - **Type:** **integration** test (Testcontainers). **Kind:** behavioural. -- **Status:** **Registered.** +- **Status:** **Implemented** (`OutboxProcessorTests`, `OutboxPublishAttemptTests`): + actual hosted ordinary-failure, retained-scope/cleanup, partition hold, four-task + saturation and finite-shutdown proofs; hostile synchronous construction and + cancellation callbacks retain ownership through completion. - **Phase:** 02b. #### `Outbox_Claim_IsHeld_Until_Dispatch_Completes` -- **Asserts:** two concurrent `OutboxProcessor` instances draining one pending batch - dispatch each message **exactly once** — the claim is held for the duration of the - dispatch, not released when the row is read. Requires **two** processors: a - single-processor test passes against the broken protocol. +- **Asserts:** two independent processor instances have at most one live owner of + a row. A claim commits before publication and is fenced by a fresh acquisition + token and database-time lease. Release-on-read controls fail. Expiry/crash permits + redelivery; same-worker reacquisition rejects both stale success and stale failure. + This proves live ownership and at-least-once recovery, not exactly-once publication. - **Source:** [Phase 02b](../roadmap/phase-02b-events-auth.md); [15-event-and-outbox.md](../architecture/15-event-and-outbox.md). -- **Type:** **integration** test (Testcontainers, two processes). **Kind:** behavioural. -- **Status:** **Registered.** +- **Type:** **integration** test (Testcontainers, two independent instances). + **Kind:** behavioural. +- **Status:** **Implemented** (`OutboxProcessorTests`, `OutboxDispatchStoreTests`): + two actual API workers contend while a live publication is held; committed + leases, repeat reads, expiry/reclaim and stale transitions have real-role + controls. Removing live-lease admission makes the hosted contention proof fail. - **Phase:** 02b. +#### `Outbox_Retry_And_Terminal_Transitions_Are_Fenced` + +- **Asserts:** Normal claims count once, stop at six and use the five approved delays. + Expired claims require fresh-token recovery; a sixth expired claim never publishes a + seventh time. Every transition checks identity, token and database-time expiry. + Terminal requests never reenter normal publication, including after audit refusal. +- **Source:** [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md); + [P02b-2 proof obligations](../roadmap/phase-02b-events-auth.md#decisive-proof-obligations). +- **Type:** production architecture census with planted controls where structural; + real PostgreSQL/runtime integration proofs where behavioural. + **Kind:** structural + behavioural. +- **Status:** **Implemented** (`OutboxDispatchConventionTests`, + `OutboxDispatchStoreTests`): exact production literal-SQL writer census with + quoted/schema/case/CTE and missing/extra-writer controls; committed claim/backoff, + expiry/reclaim, sixth-claim recovery and terminal-request refusal proofs. The + SQL census does not claim arbitrary dynamic-SQL dataflow analysis. +- **Phase:** 02b, P02b-2. + +#### `Outbox_Terminal_Transition_And_Must_Audit_Are_Atomic` + +- **Asserts:** The fenced terminal UPDATE and MUST/SecurityEvent INSERT commit + together. Failed audit or stale ownership commits neither; terminalization retries + never publish. Unknown COMMIT reconciles durable state before retry and cannot + duplicate the terminal audit. Exact real-role grants allow only the declared writes. +- **Source:** [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md); + [P02b-2 proof obligations](../roadmap/phase-02b-events-auth.md#decisive-proof-obligations). +- **Type:** integration tests (Testcontainers), including physical COMMIT-response + loss and real-role grant controls. + **Kind:** behavioural. +- **Status:** **Implemented** (`OutboxDispatchStoreTests`, + `OutboxDispatchMigrationTests`), with these proofs: + [OutboxDispatchStoreTests.Refused_audit_rolls_back_the_terminal_update_and_retains_the_request_for_audited_retry](../../backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatchStoreTests.cs), + [OutboxDispatchStoreTests.Expiry_alone_refuses_every_transition_without_changing_state](../../backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatchStoreTests.cs), + [OutboxDispatchStoreTests.A_lost_physical_COMMIT_response_reconciles_durable_terminal_state_without_duplicate_audit](../../backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatchStoreTests.cs), + [OutboxDispatchStoreTests.A_lost_COMMIT_response_and_failed_reconciliation_preserve_both_failures_without_duplicate_audit](../../backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatchStoreTests.cs), + and [OutboxDispatchMigrationTests.Outbox_role_has_no_audit_read_mutation_or_business_access](../../backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatchMigrationTests.cs) + prove audit-grant refusal, atomic rollback/retry, stale ownership, durable + reconciliation and completed-row replay without duplicate audit. +- **Phase:** 02b, P02b-2. + +#### `Outbox_Terminal_Audit_Identity_Is_Row_Bound` + +- **Asserts:** Only the successful fenced UPDATE result mints a restricted + EventTerminalTransition witness. A nonempty production mint/call-site census has + planted clean/dirty controls. Real bypass-role proofs substitute tenant, + organization, event and transaction against that witness and roll back both writes; + matching organization-scoped identity succeeds. +- **Source:** [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md); + [P02b-2 proof obligations](../roadmap/phase-02b-events-auth.md#decisive-proof-obligations). +- **Type:** production architecture census with planted controls where structural; + real PostgreSQL/runtime integration proofs where behavioural. + **Kind:** structural + behavioural. +- **Status:** **Implemented** (`OutboxDispatchConventionTests`, + `OutboxDispatchStoreTests`): exact nonempty production mint/write censuses with + planted controls and real-row identity/transaction refusal proofs. +- **Phase:** 02b, P02b-2. + +#### `Outbox_Type_Resolution_Is_Declared_And_Bounded` + +- **Asserts:** Only declared concrete contract identities and explicit compatible + aliases resolve; collisions refuse composition. Unknown/retired/malformed/mismatched + data terminalizes without arbitrary type loading. Frozen options, canonical UTF-8 + admission, microsecond timestamp comparison and full row/envelope agreement have + positive and negative controls. +- **Source:** [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md); + [P02b-2 proof obligations](../roadmap/phase-02b-events-auth.md#decisive-proof-obligations). +- **Type:** production architecture census with planted controls where structural; + real PostgreSQL/runtime integration proofs where behavioural. + **Kind:** structural + startup + behavioural. +- **Status:** **Implemented** (`OutboxDispatchConventionTests`, + `OutboxEnvelopeReaderTests`, `OutboxEventRegistryTests`): declared resolver and + transitive production helper census, planted loading calls and envelope probes. +- **Phase:** 02b, P02b-2. + +#### `Outbox_Pending_Subscriptions_Preserve_Events` + +- **Asserts:** Real TenantCreated rows and aliases remain unchanged through repeated + Pending polls, even with malformed stored payloads: no acquisition, attempts, + validation or handler construction. Declared intentional absence is a distinct + tested acknowledgement policy. Test-owned Active contracts exercise the shipped + publish/mark path; real subscriptions stay Pending until P02b-3. +- **Source:** [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md); + [P02b-2 proof obligations](../roadmap/phase-02b-events-auth.md#decisive-proof-obligations). +- **Type:** integration tests (Testcontainers) and unit tests of production + composition and manifest refusal. + **Kind:** startup + behavioural. +- **Status:** **Implemented** (`OutboxProcessorTests`, `OutboxDispatchStoreTests`, + `OutboxProductionCompositionTests`, `OutboxEventRegistryTests`), with these proofs: + [OutboxProcessorTests.Real_provisioning_rows_remain_Pending_across_hosted_polls_without_construction_or_mutation](../../backend/tests/LearnStack.Tests.Integration/Database/OutboxProcessorTests.cs), + [OutboxDispatchStoreTests.Real_TenantCreated_contracts_and_aliases_are_held_before_any_payload_validation](../../backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatchStoreTests.cs), + [OutboxProductionCompositionTests.Real_provisioning_contract_remains_pending_for_both_distinct_default_consumers](../../backend/tests/LearnStack.Tests.Unit/Api/Composition/OutboxProductionCompositionTests.cs), + and [OutboxEventRegistryTests.Active_manifest_and_discovered_handlers_must_match_in_both_directions](../../backend/tests/LearnStack.Tests.Unit/Infrastructure/Messaging/OutboxEventRegistryTests.cs) + prove real Pending holds and composition refusal. Test-owned Active publication + runs in [OutboxProcessorTests.A_poison_first_row_terminalizes_with_MUST_audit_and_the_following_real_handler_progresses](../../backend/tests/LearnStack.Tests.Integration/Database/OutboxProcessorTests.cs); + [OutboxProcessorTests.Explicit_intentional_absence_acknowledges_without_constructing_a_handler](../../backend/tests/LearnStack.Tests.Integration/Database/OutboxProcessorTests.cs) + proves the distinct intentional-absence policy. +- **Phase:** 02b, P02b-2. + +#### `Outbox_Health_And_Metrics_Are_Bounded` + +- **Asserts:** Production worker registration, bounded deadlines/overruns/shutdown and + real progress health have nonempty subjects and planted controls. Gauge callbacks + read immutable snapshots without SQL; failed refresh retains stale data. + Due/delayed/leased/deferred/terminal-requested/terminal rows distinguish gauges. + Labels and diagnostics prohibit unbounded identities or sensitive + exception/type/payload/connection data; /healthz remains liveness-only. + Polymorphic helper traversal covers production-owned contracts. The external + clock is pinned to the actual singleton `TimeProvider.System` instance; actual + DI health must follow Healthy → Unhealthy → Healthy on its singleton observation, + refusing a same-type detached factory. +- **Source:** [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md); + [P02b-2 proof obligations](../roadmap/phase-02b-events-auth.md#decisive-proof-obligations). +- **Type:** production architecture census with planted controls where structural; + real PostgreSQL/runtime integration proofs where behavioural. + **Kind:** structural + behavioural. +- **Status:** **Implemented** (`OutboxDispatchConventionTests`, + `OutboxProcessorTests`, `OutboxObservationTests`, + `OutboxProductionCompositionTests`): nonempty five-subject observation graph, + actual gauge callbacks, async/production-owned polymorphic helpers and + persistence/locator capability controls; actual DI missing/duplicate/wrong-factory + controls plus bounded runtime, immutable snapshot, metric-label and liveness + proofs. The external clock is pinned to the actual `TimeProvider.System` singleton, + and actual DI health follows Healthy → Unhealthy → Healthy against the singleton + observation; alternate clock and detached same-type factory controls are rejected. + Focused verification and both fresh fix reviews pass at `606362b`. +- **Phase:** 02b, P02b-2. + ### Demand-gated: lands with its adapter These rules are agreed but cannot be written until the technology they constrain is @@ -3343,11 +3533,15 @@ registered, which is the shape of gap this catalogue exists to close. It is ther - **Asserts:** no type in a module assembly takes, returns or stores `IEventBus`, and no module takes or stores `IServiceProvider` as a service-locator escape hatch. Constructor - and method parameters, return types, fields and properties are checked. The only - sanctioned publisher is the `OutboxProcessor`; modules write to the outbox. + and method parameters, return types, fields and properties are checked. A nonempty + whole-production publication census admits only the private task-owned publish + callback and the attempt constructed by `OutboxProcessor.DispatchNormalAsync`; + interface/concrete/delegate/async publication and unowned attempts are refused. + Modules write to the outbox. - **Source:** [20-infrastructure-stack.md § `IEventBus`](20-infrastructure-stack.md); [ADR-0010](../decisions/0010-cross-module-communication.md). -- **Type:** xUnit + reflection over module assemblies. **Kind:** structural. +- **Type:** xUnit + reflection over module assemblies and whole-production IL + publication census. **Kind:** structural. - **Status:** **Implemented** (`CrossCuttingFoundationTests`). A module holding the bus gets a synchronous cross-module call with no durability and no transactional atomicity — a fifth cross-module mechanism in everything but name, and one that looks @@ -3356,8 +3550,11 @@ registered, which is the shape of gap this catalogue exists to close. It is ther `LearnStack.SharedKernel.Messaging` for `IIntegrationEvent` and `IIntegrationEventHandler`. The module sweep has had a subject since Packet 7 — four modules ship code — and the checker is also pointed at direct-injection, - method-injection and service-locator deliberate offenders in the test assembly, so a - clean sweep is evidence rather than an absence. + method-injection and service-locator deliberate offenders in the test assembly. + P02b-2 Step 4 adds the complete production publication fence through the same + canonical test; missing subjects, extra implementations/callers, delegate/async + escapes and unowned attempts have dirty/clean controls. This is bounded IL + capability inspection, not arbitrary reflection/dataflow analysis. - **Phase:** 02a (Packet 5). #### `Integration_Event_TopicNames_FollowConvention` diff --git a/frontend/apps/web/src/server/ingress.test.ts b/frontend/apps/web/src/server/ingress.test.ts index 8ab994bd..064587c9 100644 --- a/frontend/apps/web/src/server/ingress.test.ts +++ b/frontend/apps/web/src/server/ingress.test.ts @@ -204,6 +204,17 @@ describe('private local configuration', () => { ).toThrow('Conflicting public server configuration'); }); + it('loads the API dispatcher credential from the selected file or explicit environment', () => { + writeFileSync(join(directory, '.env'), 'ConnectionStrings__OutboxDispatcher=selected'); + expect(loadLocalEnvironment(directory, {}).ConnectionStrings__OutboxDispatcher).toBe( + 'selected', + ); + expect( + loadLocalEnvironment(directory, { ConnectionStrings__OutboxDispatcher: 'explicit' }) + .ConnectionStrings__OutboxDispatcher, + ).toBe('explicit'); + }); + const configuration = { LEARNSTACK_PUBLIC_API_ORIGIN: 'http://127.0.0.1:5080', LEARNSTACK_PUBLIC_HOP_SECRET: secret, diff --git a/frontend/apps/web/src/server/ingress.ts b/frontend/apps/web/src/server/ingress.ts index 141c685d..4fe577cf 100644 --- a/frontend/apps/web/src/server/ingress.ts +++ b/frontend/apps/web/src/server/ingress.ts @@ -275,7 +275,11 @@ export function loadLocalEnvironment( if (new Set(values).size > 1) throw new Error('Conflicting public server configuration'); if (values[0] !== undefined) result[key] = values[0]; } - for (const key of ['ConnectionStrings__Default', 'ConnectionStrings__PlatformAdmin']) { + for (const key of [ + 'ConnectionStrings__Default', + 'ConnectionStrings__PlatformAdmin', + 'ConnectionStrings__OutboxDispatcher', + ]) { if (environment[key] !== undefined) result[key] = environment[key]; } return result; diff --git a/frontend/apps/web/src/test/demo-lifecycle.test.ts b/frontend/apps/web/src/test/demo-lifecycle.test.ts index b7e68af7..8fd79b64 100644 --- a/frontend/apps/web/src/test/demo-lifecycle.test.ts +++ b/frontend/apps/web/src/test/demo-lifecycle.test.ts @@ -1,5 +1,6 @@ // @vitest-environment node import { execFileSync, spawnSync } from 'node:child_process'; +import { randomBytes } from 'node:crypto'; import { once } from 'node:events'; import { mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, writeFileSync } from 'node:fs'; import { createServer as createHttpsServer } from 'node:https'; @@ -10,6 +11,10 @@ import { join } from 'node:path'; import { afterEach, describe, expect, it } from 'vitest'; import { composeSelection } from '../../../../../scripts/compose.mjs'; +import { + publicSeederEnvironment, + validateDemoDatabaseConfiguration, +} from '../../../../../scripts/demo-database.mjs'; import { boundedBody, demoRequest, vacant } from '../../../../../scripts/demo-http.mjs'; import { apiCompletionObserver, @@ -17,6 +22,7 @@ import { } from '../../../../../scripts/demo-observation.mjs'; import { DemoProcesses } from '../../../../../scripts/demo-process.mjs'; import { + demoFailureMessage, disposeDemoResources, isApiListenerEvent, validateCertificate, @@ -45,6 +51,173 @@ function owner(signal = new AbortController().signal) { return processes; } +describe('launcher-owned local database validation', () => { + const roles = [ + ['Migration', 'learnstack_migration', 'LEARNSTACK_MIGRATION_PW'], + ['Default', 'learnstack_app', 'LEARNSTACK_APP_PW'], + ['PlatformAdmin', 'learnstack_platform', 'LEARNSTACK_PLATFORM_PW'], + ['OutboxDispatcher', 'learnstack_outbox_admin', 'LEARNSTACK_OUTBOX_PW'], + ] as const; + const databasePassword = randomBytes(16).toString('hex'); + function databaseValues(password = databasePassword) { + const values: Record = { POSTGRES_DB: 'learnstack' }; + for (const [suffix, role, passwordKey] of roles) { + values[passwordKey] = password; + values[`ConnectionStrings__${suffix}`] = + `Host=localhost;Port=5432;Database=learnstack;Username=${role};Password="${password.replaceAll('"', '""')}"`; + } + return values; + } + it.each([null, undefined])( + 'refuses absent configuration (%s) with the fixed diagnostic', + (values) => { + expect(() => validateDemoDatabaseConfiguration(values)).toThrowError( + /^Local demo database configuration is invalid$/, + ); + }, + ); + it('reports the real database validation failure through the launcher diagnostic boundary', () => { + const values = databaseValues(); + delete values.LEARNSTACK_OUTBOX_PW; + let failure: unknown; + try { + validateDemoDatabaseConfiguration(values); + } catch (error) { + failure = error; + } + expect(failure).toBeInstanceOf(Error); + expect(demoFailureMessage(failure)).toBe('Local demo database configuration is invalid'); + const secret = randomBytes(16).toString('hex'); + for (const message of [ + `Provider failure: ${secret}`, + `Local demo database configuration is invalid: ${secret}`, + `Local demo database configuration is invalid\n${secret}`, + ]) { + expect(demoFailureMessage(new Error(message))).toBe( + 'demo: startup/proof failed; inspect prerequisites, TLS, configuration and selected stack', + ); + } + }); + it.each([ + 'simple', + 'semi;colon', + ' leading and trailing ', + ' ', + 'a"double"quote', + "a'single'quote", + ])('accepts all four local roles with quoted password %s', (password) => { + expect(() => validateDemoDatabaseConfiguration(databaseValues(password))).not.toThrow(); + }); + it('accepts supported key aliases, single quoting, loopback and the default port', () => { + const values = databaseValues("semi;single'quote"); + for (const [suffix, role] of roles) + values[`ConnectionStrings__${suffix}`] = + ` Server = 127.0.0.1 ; Database = learnstack ; User ID = ${role} ; Pwd = 'semi;single''quote' ;`; + expect(() => validateDemoDatabaseConfiguration(values)).not.toThrow(); + }); + it.each( + roles.flatMap(([suffix]) => + [ + ['role', (value: string) => value.replace(/Username=[^;]+/, 'Username=foreign')], + ['password', (value: string) => value.replace(/Password=.*/, 'Password=foreign')], + ['database', (value: string) => value.replace('Database=learnstack', 'Database=foreign')], + ['host', (value: string) => value.replace('Host=localhost', 'Host=foreign')], + ['port', (value: string) => value.replace('Port=5432', 'Port=5433')], + ['malformed', (value: string) => value + ';Password="unclosed'], + ].map(([field, mutate]) => ({ suffix, field, mutate })), + ), + )('refuses $suffix $field without exposing configuration', ({ suffix, mutate }) => { + const values = databaseValues(); + const key = `ConnectionStrings__${suffix}`; + if (typeof mutate !== 'function') throw new Error('Missing mutation'); + values[key] = mutate(values[key] ?? ''); + expect(() => validateDemoDatabaseConfiguration(values)).toThrowError( + /^Local demo database configuration is invalid$/, + ); + }); + const applicationConnection = databaseValues().ConnectionStrings__Default!; + it.each([ + applicationConnection.replace(/Password=.*/, 'Password="unclosed'), + applicationConnection.replace(/Password=.*/, 'Password="closed"junk'), + applicationConnection.replace(/Password=.*/, 'Password=plain"quote'), + applicationConnection + ';UID=learnstack_platform', + applicationConnection + ';Unknown=value', + applicationConnection.replace('Host', 'H o s t'), + applicationConnection.replace('Port=5432', 'Port=nope'), + applicationConnection.replace('Port=5432', 'Port=5432.0'), + applicationConnection.replace(/;Password=.*/, ''), + applicationConnection.replace(';Username=learnstack_app', ''), + applicationConnection.replace(';Database=learnstack', ''), + applicationConnection.replace(/^[^;]+;/, ''), + applicationConnection.replace(/Password=.*/, 'Password='), + ['postgres://learnstack_app', databasePassword + '@localhost/learnstack'].join(':'), + '', + ])('refuses unsupported or malformed local connection syntax %#', (connection) => { + const values = databaseValues(); + values.ConnectionStrings__Default = connection; + expect(() => validateDemoDatabaseConfiguration(values)).toThrowError( + /^Local demo database configuration is invalid$/, + ); + }); + it.each([ + ...roles.map(([suffix]) => `ConnectionStrings__${suffix}`), + ...roles.map(([, , key]) => key), + 'POSTGRES_DB', + ])('requires selected configuration %s', (missing) => { + const values = databaseValues(); + delete values[missing]; + expect(() => validateDemoDatabaseConfiguration(values)).toThrowError( + /^Local demo database configuration is invalid$/, + ); + }); + it.each(roles)( + 'refuses %s when both password sources are missing', + (suffix, role, passwordKey) => { + const values = databaseValues(); + delete values[passwordKey]; + values[`ConnectionStrings__${suffix}`] = + `Host=localhost;Port=5432;Database=learnstack;Username=${role}`; + expect(() => validateDemoDatabaseConfiguration(values)).toThrowError( + /^Local demo database configuration is invalid$/, + ); + expect(() => validateDemoDatabaseConfiguration(databaseValues())).not.toThrow(); + }, + ); + it('keeps every Seeder mode free of inherited credentials and diagnostics', () => { + const base = { + PATH: '/bin', + HOME: '/home', + DOTNET_ROOT: '/dotnet', + TMPDIR: '/tmp', + APPDATA: '/profile', + ConnectionStrings__Default: 'ambient-application', + ConnectionStrings__Migration: 'owner', + ConnectionStrings__OutboxDispatcher: 'dispatcher', + CONNECTIONSTRINGS__OUTBOXDISPATCHER: 'uppercase-dispatcher', + 'ConnectionStrings:OutboxDispatcher': 'colon-dispatcher', + LEARNSTACK_OUTBOX_PW: 'dispatcher-password', + DEBUG: '*', + }; + const prerequisites = { + PATH: '/bin', + HOME: '/home', + DOTNET_ROOT: '/dotnet', + TMPDIR: '/tmp', + APPDATA: '/profile', + }; + expect(publicSeederEnvironment(undefined, base)).toEqual(prerequisites); + expect(publicSeederEnvironment('selected-application', base)).toEqual({ + ...prerequisites, + ConnectionStrings__Default: 'selected-application', + }); + const source = readFileSync( + new URL('../../../../../scripts/demo.mjs', import.meta.url), + 'utf8', + ); + expect(source).not.toContain('--validate-demo-config'); + }); +}); + describe('shared explicit demo stack selection', () => { it.each([ 'COMPOSE_PROFILES=gated', @@ -92,23 +265,23 @@ describe('shared explicit demo stack selection', () => { expect(result.stderr).not.toContain('foreign'); } }); - it('launches the argument-free Seeder from the documented shell path', () => { + it('launches the argument-free Seeder without selected or ambient dispatcher credentials', () => { const root = temporary(); for (const directory of ['scripts', 'backend', 'bin']) mkdirSync(join(root, directory)); - for (const file of ['seed.sh', 'connection-string.awk']) + for (const file of ['seed.sh', 'connection-string.awk', 'demo-database.mjs']) writeFileSync( join(root, 'scripts', file), readFileSync(new URL(`../../../../../scripts/${file}`, import.meta.url)), ); writeFileSync( join(root, '.env'), - 'ConnectionStrings__Migration=Username=learnstack_migration\nConnectionStrings__Default=Username=learnstack_app\n', + 'ConnectionStrings__Migration=Username=learnstack_migration\nConnectionStrings__Default=Username=learnstack_app\nConnectionStrings__OutboxDispatcher=Username=learnstack_outbox_admin\n', ); for (const [name, body] of Object.entries({ docker: '[ "$*" = "compose version" ]', - node: '[ "$*" = "scripts/wait-compose.mjs" ]', + node: 'if [ "$*" = "scripts/wait-compose.mjs" ]; then exit 0; fi\nexec "$REAL_NODE" "$@"', dotnet: - '[ "$#" = 3 ] && [ "$1" = run ] && [ "$2" = --project ] && [ "$3" = src/LearnStack.Tools.Seeder ] && [ "$PWD" = "$TEST_ROOT/backend" ] && [ "$ConnectionStrings__Default" = Username=learnstack_app ] && printf launched > "$TEST_ROOT/launched"', + '[ "$#" = 3 ] && [ "$1" = run ] && [ "$2" = --project ] && [ "$3" = src/LearnStack.Tools.Seeder ] && [ "$(basename "$PWD")" = backend ] && [ "$ConnectionStrings__Default" = Username=learnstack_app ] && [ "$HOME" = /seed-home ] && [ -n "$PATH" ] && [ -z "${ConnectionStrings__OutboxDispatcher+x}" ] && [ -z "${CONNECTIONSTRINGS__OUTBOXDISPATCHER+x}" ] && [ -z "${LEARNSTACK_OUTBOX_PW+x}" ] && ! env | /usr/bin/grep -q "^ConnectionStrings:OutboxDispatcher=" && printf launched > ../launched', })) writeFileSync(join(root, 'bin', name), `#!/bin/sh\n${body}\n`, { mode: 0o700 }); const launch = () => @@ -117,7 +290,13 @@ describe('shared explicit demo stack selection', () => { env: { PATH: `${join(root, 'bin')}:${process.env.PATH}`, TEST_ROOT: root, + REAL_NODE: process.execPath, + HOME: '/seed-home', NODE_ENV: 'test', + ConnectionStrings__OutboxDispatcher: 'ambient-dispatcher', + CONNECTIONSTRINGS__OUTBOXDISPATCHER: 'ambient-alias', + 'ConnectionStrings:OutboxDispatcher': 'ambient-colon-alias', + LEARNSTACK_OUTBOX_PW: 'ambient-password', }, encoding: 'utf8', timeout: 5000, @@ -128,8 +307,8 @@ describe('shared explicit demo stack selection', () => { writeFileSync( script, readFileSync(script, 'utf8').replace( - 'dotnet run --project src/LearnStack.Tools.Seeder', - 'dotnet run --project src/LearnStack.Tools.Seeder --nologo', + "['run', '--project', 'src/LearnStack.Tools.Seeder']", + "['run', '--project', 'src/LearnStack.Tools.Seeder', '--nologo']", ), ); const refused = launch(); @@ -255,7 +434,10 @@ describe('demo runtime boundaries', () => { it('isolates the Development API user-secret profile within the invocation', () => { const root = temporary(); const env = publicDemoApiEnvironment( - { ConnectionStrings__Default: 'application' }, + { + ConnectionStrings__Default: 'application', + ConnectionStrings__OutboxDispatcher: 'dispatcher', + }, { apiOrigin: 'http://127.0.0.1:5080', secret: 'private' }, root, { @@ -270,6 +452,7 @@ describe('demo runtime boundaries', () => { expect(env.TMPDIR).toBe(realpathSync(root)); expect(env).not.toHaveProperty('DOTNET_CLI_HOME'); expect(env.ConnectionStrings__Default).toBe('application'); + expect(env.ConnectionStrings__OutboxDispatcher).toBe('dispatcher'); expect(env.ASPNETCORE_ENVIRONMENT).toBe('Development'); }); it('gives native web addons an owned cache without ambient credentials or diagnostics', () => { @@ -280,9 +463,15 @@ describe('demo runtime boundaries', () => { LEARNSTACK_PUBLIC_TLS_CERT: '/owned/cert', LEARNSTACK_PUBLIC_TLS_KEY: '/owned/key', ConnectionStrings__Migration: 'owner', + ConnectionStrings__OutboxDispatcher: 'dispatcher', }; expect( - publicWebEnvironment(values, root, { PATH: '/bin', DEBUG: '*', HOME: '/foreign' }), + publicWebEnvironment(values, root, { + PATH: '/bin', + DEBUG: '*', + HOME: '/foreign', + ConnectionStrings__OutboxDispatcher: 'ambient-dispatcher', + }), ).toEqual({ PATH: '/bin', TMPDIR: root, @@ -295,9 +484,12 @@ describe('demo runtime boundaries', () => { ), }); }); - it('passes only request runtime credentials and explicit hop configuration', () => { + it('passes only explicit API credentials and hop configuration', () => { const env = publicApiEnvironment( - { ConnectionStrings__Default: 'application' }, + { + ConnectionStrings__Default: 'application', + ConnectionStrings__OutboxDispatcher: 'dispatcher', + }, { apiOrigin: 'http://127.0.0.1:5080', secret: 'private' }, { PATH: '/bin', @@ -311,12 +503,34 @@ describe('demo runtime boundaries', () => { expect(env).toEqual({ PATH: '/bin', ConnectionStrings__Default: 'application', + ConnectionStrings__OutboxDispatcher: 'dispatcher', ASPNETCORE_ENVIRONMENT: 'Development', ASPNETCORE_URLS: 'http://127.0.0.1:5080', Tenancy__TrustedHop__Networks__0: '127.0.0.1/32', Tenancy__TrustedHop__Secrets__0: 'private', }); }); + it.each(['ConnectionStrings__Default', 'ConnectionStrings__OutboxDispatcher'])( + 'requires explicit %s even when the ambient environment has it', + (missing) => { + const values: Record = { + ConnectionStrings__Default: 'application', + ConnectionStrings__OutboxDispatcher: 'dispatcher', + }; + const configuration = { apiOrigin: 'http://127.0.0.1:5080', secret: 'private' }; + const ambient = { ...values }; + delete values[missing]; + expect(() => publicApiEnvironment(values, configuration, {})).toThrow( + 'credential is missing', + ); + expect(() => publicApiEnvironment(values, configuration, ambient)).toThrow( + 'credential is missing', + ); + expect(() => publicDemoApiEnvironment(values, configuration, temporary(), ambient)).toThrow( + 'credential is missing', + ); + }, + ); it('refuses a foreign listener without closing it', async () => { const server = createServer(); await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)); diff --git a/infra/dapr/README.md b/infra/dapr/README.md index b6a78085..b4a0fe92 100644 --- a/infra/dapr/README.md +++ b/infra/dapr/README.md @@ -28,11 +28,11 @@ Phase ownership (per [phase-02a](../../docs/roadmap/phase-02a-kernel-tenancy.md) familiarity only — no backend code path reaches them, and `backend/Directory.Packages.props` declares no client library for any of the four. -- **Phase 02b** wires the `OutboxProcessor`, which is the only sanctioned - caller of `IEventBus.PublishAsync` per ADR-0010 Amendment 1. Modules - therefore *consume* `ICacheService` and `ISecretProvider` from 02a but - do not *write* to `IEventBus` directly until the outbox path exists in - 02b. +- **P02b-2 Step 3** implements the API-host `OutboxProcessor`, with + both independent review rounds complete. It is the only sanctioned production caller + of `IEventBus.PublishAsync` per ADR-0010 Amendment 1. Modules consume + `ICacheService` and `ISecretProvider` from 02a and enqueue through `IOutbox`; + they never publish through `IEventBus` directly. Service invocation, workflow, bindings, **actors**, configuration, and distributed lock are **not adopted**; if a future need appears the gate is a @@ -138,7 +138,7 @@ YAML metadata field. the interfaces and default adapters shipped in Phase 02a; Dapr-backed adapters belong to Phase 11 (see § Phase ownership above). - Outbox dispatcher (`OutboxProcessor` polling + dispatch) — Phase 02b. -- Per-module `inbox_messages` table + `IInboxGuard` — Phase 02b. +- Shared transport-owned `inbox_messages` table + `IInboxGuard` — P02b-3. - Production Vault setup (HA mode, auto-unseal, AppRole policies) — Phase 11. - Additional sidecars for the Hub overlay (separate app id, separate consumer group) — Phase 02c. diff --git a/scripts/demo-database.d.mts b/scripts/demo-database.d.mts new file mode 100644 index 00000000..f30ac117 --- /dev/null +++ b/scripts/demo-database.d.mts @@ -0,0 +1,5 @@ +export function validateDemoDatabaseConfiguration(values: Record | null | undefined): void; +export function publicSeederEnvironment( + applicationConnectionString?: string, + base?: Partial, +): NodeJS.ProcessEnv; diff --git a/scripts/demo-database.mjs b/scripts/demo-database.mjs new file mode 100644 index 00000000..49bca807 --- /dev/null +++ b/scripts/demo-database.mjs @@ -0,0 +1,110 @@ +const invalidConfiguration = () => new Error('Local demo database configuration is invalid'); +const aliases = new Map([ + ['host', 'host'], + ['server', 'host'], + ['port', 'port'], + ['database', 'database'], + ['username', 'username'], + ['user name', 'username'], + ['userid', 'username'], + ['user id', 'username'], + ['uid', 'username'], + ['password', 'password'], + ['pwd', 'password'], + ['psw', 'password'], +]); + +// The local demo accepts only these five fields, not the general Npgsql grammar. +// Quoted values preserve semicolons/whitespace and use doubled quote escaping. +function localConnectionFields(source) { + if (typeof source !== 'string' || /[\u0000-\u0008\u000a-\u001f\u007f]/.test(source)) + throw invalidConfiguration(); + const fields = new Map(); + let index = 0; + while (index < source.length) { + while (/[; \t]/.test(source[index] ?? '') && index < source.length) index++; + if (index === source.length) break; + const start = index; + while (index < source.length && source[index] !== '=') { + if (!/[a-z \t]/i.test(source[index])) throw invalidConfiguration(); + index++; + } + if (index === source.length) throw invalidConfiguration(); + const key = aliases.get(source.slice(start, index).trim().toLowerCase()); + if (!key || fields.has(key)) throw invalidConfiguration(); + index++; + while (index < source.length && /[ \t]/.test(source[index])) index++; + let value = ''; + const quote = source[index]; + if (quote === '"' || quote === "'") { + index++; + let closed = false; + while (index < source.length) { + const character = source[index++]; + if (character !== quote) value += character; + else if (source[index] === quote) { + value += quote; + index++; + } else { + closed = true; + break; + } + } + if (!closed) throw invalidConfiguration(); + while (index < source.length && /[ \t]/.test(source[index])) index++; + if (index < source.length && source[index] !== ';') throw invalidConfiguration(); + } else { + const start = index; + while (index < source.length && source[index] !== ';') { + if (source[index] === '"' || source[index] === "'") throw invalidConfiguration(); + index++; + } + value = source.slice(start, index).trim(); + } + if (!value) throw invalidConfiguration(); + fields.set(key, value); + } + return fields; +} + +/** Reject stale local credentials in the launcher without giving them to a child process. */ +export function validateDemoDatabaseConfiguration(values) { + if ( + values === null || + values === undefined || + typeof values.POSTGRES_DB !== 'string' || + !values.POSTGRES_DB + ) + throw invalidConfiguration(); + for (const [suffix, role, passwordKey] of [ + ['Migration', 'learnstack_migration', 'LEARNSTACK_MIGRATION_PW'], + ['Default', 'learnstack_app', 'LEARNSTACK_APP_PW'], + ['PlatformAdmin', 'learnstack_platform', 'LEARNSTACK_PLATFORM_PW'], + ['OutboxDispatcher', 'learnstack_outbox_admin', 'LEARNSTACK_OUTBOX_PW'], + ]) { + const password = values[passwordKey]; + const fields = localConnectionFields(values[`ConnectionStrings__${suffix}`]); + const port = fields.get('port') ?? '5432'; + if ( + typeof password !== 'string' || + !password || + fields.get('username') !== role || + fields.get('password') !== password || + fields.get('database') !== values.POSTGRES_DB || + !['localhost', '127.0.0.1'].includes(fields.get('host')) || + !/^\d+$/.test(port) || + Number(port) !== 5432 + ) + throw invalidConfiguration(); + } +} + +/** Every Seeder mode receives only its own credential and process prerequisites. */ +export function publicSeederEnvironment(applicationConnectionString, base = process.env) { + const environment = {}; + for (const key of ['PATH', 'HOME', 'DOTNET_ROOT', 'TMPDIR', 'APPDATA']) + if (base[key] !== undefined) environment[key] = base[key]; + if (applicationConnectionString !== undefined) + environment.ConnectionStrings__Default = applicationConnectionString; + return environment; +} diff --git a/scripts/demo.d.mts b/scripts/demo.d.mts index 1ffc8e2b..5392fc16 100644 --- a/scripts/demo.d.mts +++ b/scripts/demo.d.mts @@ -1,4 +1,5 @@ export function isApiListenerEvent(line: string, origin: string): boolean; +export function demoFailureMessage(error: unknown): string; export function validateCertificate( certificate: Buffer, privateKey: Buffer, diff --git a/scripts/demo.mjs b/scripts/demo.mjs index 3b59ae2b..a6bb7abb 100644 --- a/scripts/demo.mjs +++ b/scripts/demo.mjs @@ -20,6 +20,7 @@ import { demoRequest, vacant } from './demo-http.mjs'; import { defaultServices, waitCompose } from './wait-compose.mjs'; import { publicDemoApiEnvironment, publicWebEnvironment } from './public-runtime.mjs'; import { apiCompletionObserver } from './demo-observation.mjs'; +import { publicSeederEnvironment, validateDemoDatabaseConfiguration } from './demo-database.mjs'; const appRoot = join(repositoryRoot, 'frontend/apps/web'); const backendRoot = join(repositoryRoot, 'backend'); @@ -71,6 +72,18 @@ export async function disposeDemoResources({ processes, finish, teardown, remove await removePrivate(); } +/** Only fixed local diagnostics may leave the launcher boundary. */ +export function demoFailureMessage(error) { + return error instanceof DemoCommandError || + (error instanceof Error && + (error.message === 'Local demo database configuration is invalid' || + /^(?:Use make demo|Port \d+ is occupied|Another demo owns|TLS leaf|Seed hosts|Environment disagrees|CI requires)/.test( + error.message, + ))) + ? error.message + : 'demo: startup/proof failed; inspect prerequisites, TLS, configuration and selected stack'; +} + /** Start the same seeded, production-built topology for foreground use and CI proof. */ export async function startDemo(mode) { if (!['local', 'ci'].includes(mode)) throw new Error('Use make demo DEMO_MODE=local or ci'); @@ -242,6 +255,7 @@ export async function startDemo(mode) { await run('dotnet', [seederDll, '--demo-manifest'], 'Seed-derived manifest', { capture: true, cwd: backendRoot, + env: publicSeederEnvironment(), }), ); const hosts = [...new Set(manifest.map((lane) => lane.host))]; @@ -313,12 +327,7 @@ export async function startDemo(mode) { const values = loadLocalEnvironment(repositoryRoot, process.env); const configuration = publicServerConfiguration(values, repositoryRoot); await vacant(Number(new URL(configuration.apiOrigin).port || '80')); - await run( - 'dotnet', - [seederDll, '--validate-demo-config'], - 'Local database configuration validation', - { cwd: backendRoot, env: { ...process.env, ...rawValues } }, - ); + validateDemoDatabaseConfiguration(rawValues); const certificate = readFileSync(configuration.certificate); const leaf = validateCertificate(certificate, readFileSync(configuration.privateKey), [ ...hosts, @@ -391,11 +400,7 @@ export async function startDemo(mode) { await waitCompose(selected, processes, controller.signal); await run('dotnet', [seederDll], 'Contextual seed convergence', { cwd: backendRoot, - env: { - PATH: process.env.PATH, - HOME: process.env.HOME, - ConnectionStrings__Default: rawValues.ConnectionStrings__Default, - }, + env: publicSeederEnvironment(rawValues.ConnectionStrings__Default), timeout: 180000, }); if (mode === 'ci') { @@ -539,15 +544,7 @@ if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.ur } catch (error) { if (runtime?.mode !== 'local' || !runtime.signalRequested()) { // Labels are fixed locally; never echo provider, configuration or build output. - console.error( - error instanceof DemoCommandError || - (error instanceof Error && - /^(?:Use make demo|Port \d+ is occupied|Another demo owns|TLS leaf|Seed hosts|Environment disagrees|CI requires)/.test( - error.message, - )) - ? error.message - : 'demo: startup/proof failed; inspect prerequisites, TLS, configuration and selected stack', - ); + console.error(demoFailureMessage(error)); process.exitCode = 1; } } finally { diff --git a/scripts/public-runtime.mjs b/scripts/public-runtime.mjs index 69fc7a53..86be368a 100644 --- a/scripts/public-runtime.mjs +++ b/scripts/public-runtime.mjs @@ -1,13 +1,16 @@ import { realpathSync } from 'node:fs'; import { join } from 'node:path'; -/** Keep privileged setup/worker credentials and arbitrary configuration out of runtime. */ +/** Give the API its explicit application/dispatcher credentials and bounded configuration. */ export function publicApiEnvironment(values, configuration, base = process.env) { const environment = {}; for (const name of ['PATH', 'HOME', 'DOTNET_ROOT', 'TMPDIR', 'APPDATA']) if (base[name] !== undefined) environment[name] = base[name]; if (!values.ConnectionStrings__Default) throw new Error('Application credential is missing'); + if (!values.ConnectionStrings__OutboxDispatcher) + throw new Error('Outbox dispatcher credential is missing'); environment.ConnectionStrings__Default = values.ConnectionStrings__Default; + environment.ConnectionStrings__OutboxDispatcher = values.ConnectionStrings__OutboxDispatcher; if (values.ConnectionStrings__PlatformAdmin) environment.ConnectionStrings__PlatformAdmin = values.ConnectionStrings__PlatformAdmin; return { diff --git a/scripts/seed.sh b/scripts/seed.sh index 60b892fd..d2ca7150 100755 --- a/scripts/seed.sh +++ b/scripts/seed.sh @@ -126,8 +126,19 @@ fi # Run from `backend/`, where the SDK pin lives (`backend/global.json`): from the # repository root no global.json applies and whichever SDK is newest answers, which is # not the one CI and `make migrate` use. -if ! (cd backend && ConnectionStrings__Default="$seed_cs" \ - dotnet run --project src/LearnStack.Tools.Seeder); then +# The API's dispatcher credential and its aliases never enter the Seeder process. +# Node is already required above and shares the demo's exact child allowlist. +if ! ConnectionStrings__Default="$seed_cs" node --input-type=module <<'NODE' +import { spawnSync } from 'node:child_process'; +import { publicSeederEnvironment } from './scripts/demo-database.mjs'; +const result = spawnSync('dotnet', ['run', '--project', 'src/LearnStack.Tools.Seeder'], { + cwd: 'backend', + env: publicSeederEnvironment(process.env.ConnectionStrings__Default), + stdio: 'inherit', +}); +process.exit(result.status ?? 1); +NODE +then red "seed: tenant seeding failed." red " Has the schema been applied? → make migrate" exit 1