diff --git a/.claude/skills/add-integration-event/SKILL.md b/.claude/skills/add-integration-event/SKILL.md index 4c2d8185..eed91c00 100644 --- a/.claude/skills/add-integration-event/SKILL.md +++ b/.claude/skills/add-integration-event/SKILL.md @@ -34,8 +34,11 @@ production censuses and catalogue enforcement; the owns final verification and review status. Accepted [ADR-0046](../../../docs/decisions/0046-durable-event-delivery-and-recovery.md) owns dispatcher recovery, subscription readiness and the shared inbox contract. -P02b-2 implements producer dispatch; P02b-3 still owns durable acceptance, -independent execution, canonical inbox/ledger DDL and real subscriptions. +P02b-2 implements producer dispatch; P02b-3 Steps 1–4 implement durable +acceptance, shared inbox/ledger storage, independent execution and scheduling. +Step 5 supplies the real default subscriptions and semantic Seeder handoff; the +[delivery record](../../../docs/roadmap/phase-02b-events-auth.md#p02b-3-step-5--useful-defaults-and-semantic-seed-handoff-2026-10-13) +owns validation/review status. ## When to use @@ -193,7 +196,7 @@ fails the build. ### Step 4: Consumer — transport-owned inbox boundary -**Accepted target, pending P02b-3 implementation.** The transport owns the shared +**P02b-3 owner implemented.** The transport owns the shared `inbox_messages` guard; business handlers do not inject or call `IInboxGuard`. Deduplication is `(tenant_id, subscription_id, event_id)`, with a stable versioned subscription ID independent of handler type names. The separate delivery ledger @@ -221,9 +224,9 @@ outbox intent; database rollback alone cannot undo them. The first real flow has two independent Customization subscriptions: `customization.initialize-content-type-default.v1` installs `card@1` content type; -`customization.initialize-taxonomy-default.v1` installs `plain@1` taxonomy. Both -remain Pending until P02b-3 supplies durable acceptance, create-only preservation, -per-effect MUST audit and the semantic key/version Seeder handoff. Neither may +`customization.initialize-taxonomy-default.v1` installs `plain@1` taxonomy. +P02b-3 activates both together with create-only preservation, classified nested +initialization commands and the semantic key/version Seeder handoff. Neither may suppress its sibling or overwrite authored, retired or deleted definitions. ### Step 5: Subscription registration @@ -239,17 +242,21 @@ builder.AddLearnStackCrossCuttingFoundation( typeof(LearnStack.Modules.Customization.Application.AssemblyMarker).Assembly); ``` -Today `InProcessEventBus` resolves the concrete -`IIntegrationEventHandler` directly from DI in that subscription's async -scope. MediatR is not involved. There is no shipped +The independent `DeliveryExecutor` resolves the concrete +`IIntegrationEventHandler` from its owned async scope after validation, inbox +claim and classification. A coordinating handler may send one classified nested +command through MediatR. Check that command's `Result`; a refusal must fail the +consumer with a bounded exception rather than silently complete its inbox. The +two default coordinators are explicitly Off; their commands own the MUST intents. +There is no shipped `AddDaprSubscription` helper; do not invent one. P02b-2's construction-free manifest implements stable subscription IDs, compatible type aliases and required Pending/Active readiness. Pending holds the producer row before acquisition or payload validation. Only explicit intentional absence may acknowledge a known type without publication. P02b-3 makes Active publication mean durable acceptance of all matching deliveries, independently of business handling. -Do not invent a registration API before its implementation ships. Phase 11 must -preserve these contracts when it adds Dapr. +Use the shipped `OutboxSubscriptionManifest` and handler-registry registration +in the API composition root. Phase 11 must preserve these contracts when it adds Dapr. ### Step 6: Tests @@ -273,8 +280,8 @@ byte boundaries, SQL NULL metadata and terminal reuse. For delivered flows: - `dotnet build` and `dotnet test` pass. - `LearnStack.Tests.Architecture` is green — `Integration_Event_TopicNames_FollowConvention` and `Integration_Events_Inherit_From_IntegrationEventBase` run today. - `Integration_Event_Handlers_Use_InboxGuard` remains Registered for P02b-3's - transport-owned consumer delivery. Check + `Integration_Event_Handlers_Use_InboxGuard` is Implemented with P02b-3's + nonempty all-layer production census and real transport-owned delivery proofs. Check [the catalogue](../../../docs/standards/21-architecture-tests-catalogue.md) for their status rather than assuming a net is under you. - P02b-2 proves producer recovery and terminal audit independently of Pending @@ -288,8 +295,9 @@ byte boundaries, SQL NULL metadata and terminal reuse. For delivered flows: that owns the aggregate. The moment another module needs to react, it must be an integration event. - **Moving the inbox guard into a handler.** The transport must claim before - construction. The rule remains Registered until P02b-3; its name alone proves - no enforcement. Redelivery must preserve independent sibling effects. + construction. P02b-3 enforces typed invocation boundaries and actual transaction + atomicity; extend both controls for a new subscription. Redelivery must preserve + independent sibling effects. - **Writing the outbox row in a separate transaction.** Use `IOutbox.EnqueueAsync` through the enrolled ambient `IUnitOfWork`; never `new TransactionScope`. - **Supplying a second topic at enqueue or subscription time.** The concrete diff --git a/.claude/skills/add-integration-test/SKILL.md b/.claude/skills/add-integration-test/SKILL.md index 8beaa950..065f200b 100644 --- a/.claude/skills/add-integration-test/SKILL.md +++ b/.claude/skills/add-integration-test/SKILL.md @@ -70,7 +70,7 @@ architecture test) plus any other invariant the change touches. See > what almost every test wants: **every** migration chain applied — through > `MigrationChains.ApplyAllAsync`, never a list a fixture keeps itself — and every > table seeded for two tenants, with a second organization under tenant A. There -> are five chains and 21 data tables (26 including migration histories). Discover +> are five chains and 23 data tables (28 including migration histories). Discover > current chains and history names through > [MigrationChains](../../../backend/tests/LearnStack.Tests.Integration/Database/MigrationChains.cs); > do not copy that inventory into a fixture. Share the schema with @@ -241,8 +241,10 @@ those shipped examples for Education rather than inventing an authenticated rout > `IOutbox`; follow `OutboxProducerTests` and `TenantCreatedProducerTests` for > shipped producer fixtures. P02b-2 Step 3 implements the API-host dispatcher; > both independent review rounds are complete, and `OutboxProcessorTests` owns its -> runtime proofs. Real subscriptions remain Pending; consumer/inbox assertions -> belong to P02b-3. `audit_log` shipped in Packet 9: for a real audit assertion, +> runtime proofs. P02b-3 Steps 1–5 implement shared inbox, per-subscription +> execution and both Active production defaults. Follow `DeliveryExecutionTests` +> for ownership/atomicity and `CustomizationDefaultDeliveryTests` for the actual +> API-worker/Seeder flow. `audit_log` shipped in Packet 9: for a real audit assertion, > follow `AuditWorkflowTests`, which drives commands through > the seeder's composition root on `SchemaFixture` as `learnstack_app` and reads the > persisted columns. The durable `IIdempotencyStore` ships on the trigger @@ -272,8 +274,9 @@ public async Task CreateEnrollment_publishes_EnrollmentCreated_via_outbox() } ``` -The future P02b-3 consumer-side assertion below is illustrative; the shipped -worker has no `_fx.RunOutboxProcessorOnceAsync()` fixture API: +The consumer-side assertion below remains illustrative; the shipped P02b-3 +fixtures above exercise actual workers and have no +`_fx.RunOutboxProcessorOnceAsync()` fixture API: ```csharp await _fx.RunOutboxProcessorOnceAsync(); diff --git a/.claude/skills/code-review/SKILL.md b/.claude/skills/code-review/SKILL.md index df60c1f7..a0cfad51 100644 --- a/.claude/skills/code-review/SKILL.md +++ b/.claude/skills/code-review/SKILL.md @@ -295,11 +295,14 @@ Author intent: causation/causal actor; transaction announcement supplies tenant/organization. No caller header/snapshot establishes authority. Common non-MediatR audit entry refuses unregistered direct mutation and reconciles only at the outer - frame; producer receipts are distinct. P02b-3 delivery/inbox remains open. + frame; producer receipts are distinct. P02b-3 Steps 1–6 deliver durable + acceptance, transport-owned inbox/business completion and useful defaults; + check the packet record for current enforcement and review evidence. - **Real flow and evidence.** Completed provisioning emits - `TenantCreatedIntegrationEventV1`; P02b-1 claims pending rows only. P02b-3 owns + `TenantCreatedIntegrationEventV1`; P02b-1 claims pending rows only. P02b-3 delivers separate create-only `card@1`/`plain@1` subscriptions and Seeder identity - handoff after G1; no settings invalidation or Audit mirror is the sample. + handoff under Accepted G1; both subscriptions are Active. No settings + invalidation or Audit mirror is the sample. Read current producer/consumer statuses in the catalogue; require production censuses and planted offenders rather than zero-subject passes. - **No direct write to `audit_log` / `outbox_messages` / diff --git a/.claude/skills/seed-tenant/SKILL.md b/.claude/skills/seed-tenant/SKILL.md index c3955494..16bb6e95 100644 --- a/.claude/skills/seed-tenant/SKILL.md +++ b/.claude/skills/seed-tenant/SKILL.md @@ -16,7 +16,11 @@ description: > The [P02d-2 accepted package](../../../docs/roadmap/phase-02d-walking-skeleton.md#p02d-2-decision-package-2026-10-02) records the decisions. Step 4 delivers the complete inventory and contextual verification; both implementation review rounds and final verification passed. -P02d-2 is merged, and P02d-4 supplies public API reads. Server ingress/transport +P02d-2 is merged, and P02d-4 supplies public API reads. The accepted +[P02b-3 package](../../../docs/roadmap/phase-02b-events-auth.md#p02b-3-decision-package-2026-10-13) +adds shared create-only defaults and the semantic built-in handoff; its +[Step 5 record](../../../docs/roadmap/phase-02b-events-auth.md#p02b-3-step-5--useful-defaults-and-semantic-seed-handoff-2026-10-13) +owns current validation/review status. Server ingress/transport belongs to P02d-5; P02d-6 pages and P02d-7's shared foreground/CI demo are delivered. Step 3 supplies the narrow browser/Lighthouse proof; its [delivery record](../../../docs/roadmap/phase-02d-walking-skeleton.md#p02d-7-step-3--browser-and-audit-proof-2026-10-11) @@ -74,7 +78,9 @@ that file. There is no `make seed-tenant`, `infra/seed/`, or seed-reset command. ### 2. Read the declaration -The existing fixed tenant, organization, host and built-in IDs are preserved. +Fixed tenant, organization, host and authored curriculum IDs are preserved. +P02b-3 discovers built-ins by semantic key/version and verifies their complete +Active shape while retaining the stored ID; new built-ins mint owner UUIDv7 IDs. English's host remains tenant-wide; Yoga's host maps to Studio One. The exact [accepted inventory](../../../docs/roadmap/phase-02d-walking-skeleton.md#seed-inventory-and-ownership) and `SeedData` own the choices; this skill does not keep a second literal list. @@ -94,7 +100,9 @@ URLs or logo are seeded. Restricted publication grants no anonymous lesson acces and host mapping. The tenant starts Trial. Provisioning alone writes the sanctioned Tenant/Organization pair ([ADR-0042](../../../docs/decisions/0042-tenant-provisioning-cross-aggregate-transaction.md)). 2. Enabled locales, with the declared default first. -3. Built-in and tenant-specific definitions, each registered then published. +3. The two built-ins through separate shared initialization commands; each new + root is Active at one save, generation bump and MUST audit. Tenant-authored + definitions retain separate exact-ID registration and publication. 4. Complete branding, in tenant-wide scope. 5. Draft Course/Lesson creation and each translation, followed by each selected publication. Course and Lesson publication remain independent. @@ -116,7 +124,11 @@ IDs; contract-local IDs cross the module boundary as Guid under ADR-0023. - Missing acts write through the ordinary command. - Completed acts skip before invoking a writer, so the second completed run changes no root, timestamp, version, generation or audit row. -- Creation accepts only the declared Draft intermediate or intended final state +- Built-ins require exact Active version/shape but allow matching warm/generated + IDs. Any Draft, deprecated, deleted or successor history is preserved; a failed + demo postcondition refuses without overwrite. Initialization and registration + share the transaction-owned semantic concept lock. +- Authored creation accepts only the declared Draft intermediate or intended final state with matching immutable data. Definition creation accepts Draft/Active, with a separate publication act verifying Active. - Existing translations are checked before draft-only insertion. JSON object @@ -134,7 +146,8 @@ IDs; contract-local IDs cross the module boundary as Guid under ADR-0023. The process entry point exits 2 for missing connection configuration, 1 for a refused credential or run failure, and 0 for a completed run. Completed acts remain durable; a later explicit run resumes the remaining acts. If a different revision -becomes Active between registration's pre-read and post-read, the newly committed +becomes Active between authored registration's pre-read and post-read, the newly +committed Draft, generation increment and audit remain durable; the run refuses and requires explicit operator reconciliation. It does not retire either revision or undo an already committed act. diff --git a/.claude/skills/standards-check/SKILL.md b/.claude/skills/standards-check/SKILL.md index aa03371b..46925f6d 100644 --- a/.claude/skills/standards-check/SKILL.md +++ b/.claude/skills/standards-check/SKILL.md @@ -377,7 +377,8 @@ domain the diff doesn't touch. endpoint carries its ADR and lands in both repositories. - [ ] Producer INSERT uses its announced transaction. ADR-0046 dispatcher transitions use fresh tokens and atomic terminal audit; transport-owned shared - inbox acquisition precedes handler construction (P02b-3 pending). + inbox acquisition precedes handler construction (P02b-3 Steps 2–3 delivered; + Step 5 activates both real defaults). - [ ] Manifest readiness preserves required Pending rows. Active publication means durable acceptance; subscriber retry/terminal/replay state is independent of the producer budget. Check catalogue status before claiming enforcement. diff --git a/.claude/skills/wire-dapr-pubsub/SKILL.md b/.claude/skills/wire-dapr-pubsub/SKILL.md index 1b3e9724..99db8d53 100644 --- a/.claude/skills/wire-dapr-pubsub/SKILL.md +++ b/.claude/skills/wire-dapr-pubsub/SKILL.md @@ -24,8 +24,9 @@ that contract. The wiring contract is in this skill is the **mechanical** check-list. Accepted [ADR-0046](../../../docs/decisions/0046-durable-event-delivery-and-recovery.md) adds construction-free stable subscription identities, Pending/Active readiness -and durable acceptance. Those consumer components remain P02b-3 work; Phase 11 -must preserve their contracts rather than reintroduce handler-owned deduplication. +and durable acceptance, implemented by P02b-3 with independent execution/retries +and both real default subscribers. Phase 11 must preserve these contracts rather +than reintroduce handler-owned deduplication. ## When to use @@ -266,12 +267,12 @@ obligations to the adapter or infer delivery from a registered metric name. - Current architecture tests `Integration_Event_TopicNames_FollowConvention` and `Modules_Do_Not_Inject_IEventBus_Directly` pass. -- When doing Phase 02b consumer work, add/keep its registered inheritance and +- When doing Phase 02b consumer work, extend its implemented inheritance and inbox-guard rules. When doing Phase 11 adapter work, add/keep the Dapr binding and direct-provider boundary rules owned by that phase. -- The current baseline has no durable inbox. P02b-3 must prove acceptance plus - atomic inbox/business completion, concurrent duplicates, independent sibling - retries, sticky timeout and uncertain-COMMIT recovery on the real delivery path. +- P02b-3 implements durable acceptance and the shared inbox. Extend the real + delivery proofs for atomic inbox/business completion, concurrent duplicates, + independent sibling retries, sticky timeout and uncertain-COMMIT recovery. - When Phase 11's adapter exists, its Dapr/Testcontainers binding test and transport metrics pass as additional validation. diff --git a/CLAUDE.md b/CLAUDE.md index 4acb64d5..b68a9c2c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -192,14 +192,45 @@ Step 3 implements hosted execution and observation; its [delivery record](docs/roadmap/phase-02b-events-auth.md#p02b-2-step-3--hosted-lifecycle-and-observability-2026-10-12) owns verification and both completed review rounds. Step 4 enforcement and local closeout are complete; all four steps completed both independent review rounds. -The -[delivery record](docs/roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) -owns local closeout evidence and CI/merge status. +**P02b-2 is complete and merged** through +[PR #30](https://github.com/HodeTech/LearnStack/pull/30) on 2026-10-13. Its +[merge closeout](docs/roadmap/phase-02b-events-auth.md#p02b-2-merge-and-closeout-2026-10-13) +owns final source/merge verification and CI evidence. Acceptance registers six new catalogue rules; the Step 4 record owns their -delivery evidence. P02b-2 claims no inbox or subscriber delivery. P02b-3 still owns -durable acceptance, shared inbox/ledger, independent retries, both real default -subscriptions and Seeder handoff. -Phase 06 retains the full renderer/Studio expansion. +delivery evidence. P02b-2 claims no inbox or subscriber delivery. P02b-3's +[decision package](docs/roadmap/phase-02b-events-auth.md#p02b-3-decision-package-2026-10-13) +is Accepted — 2026-10-13. Six implementation steps, each with commit and two +independent review rounds, deliver durable acceptance, shared inbox/ledger, +independent retries, both real defaults and Seeder handoff. Implementation is +locally complete; Step 1 implements shared storage and privilege proofs, and both +independent review rounds are complete. Its +[delivery record](docs/roadmap/phase-02b-events-auth.md#p02b-3-step-1--shared-storage-and-privileges-2026-10-13) +owns verification and review status. Step 2 implements durable acceptance and +inbox primitives; its +[record](docs/roadmap/phase-02b-events-auth.md#p02b-3-step-2--durable-acceptance-and-inbox-primitives-2026-10-13) +owns validation and both completed review rounds. Step 3 implements the closed +business owner and completion gate; its +[record](docs/roadmap/phase-02b-events-auth.md#p02b-3-step-3--business-owner-and-completion-gate-2026-10-13) +owns validation and both completed review rounds. Step 4 adds independent +subscriber scheduling, terminal audit, replay and bounded observation; its +[record](docs/roadmap/phase-02b-events-auth.md#p02b-3-step-4--subscriber-scheduling-terminalization-and-replay-2026-10-13) +owns validation and both completed review rounds. Step 5 implements useful +defaults and semantic seed handoff; its +[record](docs/roadmap/phase-02b-events-auth.md#p02b-3-step-5--useful-defaults-and-semantic-seed-handoff-2026-10-13) +owns validation and both completed independent review rounds. Both real default +subscriptions are Active. Step 6 implements production-census enforcement and +full packet closeout; its +[record](docs/roadmap/phase-02b-events-auth.md#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) +owns completed validation/review receipts and PR/CI handoff. All six steps +completed both independent review rounds; P02b-3 implementation is locally +complete. The +[external review follow-up](docs/roadmap/phase-02b-events-auth.md#p02b-3-external-review-follow-up-2026-10-15) +records verified corrections and two further independent review rounds on the +same unmerged branch; the +[second follow-up](docs/roadmap/phase-02b-events-auth.md#p02b-3-second-external-review-follow-up-2026-10-15) +owns subsequent remediation status. PR review and merge remain pending; +P02b-4's background-job decision pass is next. Phase 06 retains the full +renderer/Studio expansion. **Phase 01** shipped the .NET 10 solution scaffold under `backend/` (core + 7 modules × 4 projects + 4 test projects including the @@ -404,8 +435,11 @@ Then read the phase that is live and the one it stands on: Accepted. Steps 1–2 completed both review rounds; Step 3 implements hosted dispatch and observation, with both independent review rounds complete. Step 4 enforcement and local closeout are complete; all four steps completed both - independent review rounds; the Step 4 record above owns CI/merge status. - P02b-3 owns consumer/inbox delivery; each later packet accepts its gates first. + independent review rounds. P02b-2 is complete and merged through PR #30 on + 2026-10-13; its merge closeout above owns final verification. P02b-3 Steps 1–6 + completed both review rounds; Step 6 owns the PR/CI handoff. P02b-4's + background-job decision pass follows maintainer review of the current packet. + Each later packet accepts its gates first. - [docs/roadmap/phase-02d-walking-skeleton.md](docs/roadmap/phase-02d-walking-skeleton.md) — complete and merged; the delivered public-site baseline and P02b-0 handoff. `02d` sorts after `02b`/`02c` but runs **before** them; the roadmap dependency @@ -656,6 +690,9 @@ rules: parameterized SQL; its terminal UPDATE and narrow witness-bound audit INSERT share one transaction. P02b-2 Step 2 implements that accepted exception; both independent review rounds passed. +- Write `inbox_messages` or `event_deliveries` from a module or Seeder. ADR-0046's + transport-owned acceptance, inbox/business owner and confined scheduler alone + write their granted columns. - Write rows through EF Core's set-based APIs — `ExecuteUpdate`, `ExecuteDelete`, `ExecuteSql*`. The audit capture sees only what the `ChangeTracker` holds ([ADR-0044 § 7](docs/decisions/0044-audit-write-path.md)), so such a write commits with diff --git a/README.md b/README.md index e840ca06..d514c24a 100644 --- a/README.md +++ b/README.md @@ -71,14 +71,16 @@ is complete. **P02b-1 is complete and merged** through [PR #29](https://github.com/HodeTech/LearnStack/pull/29) on 2026-10-12. Its [merge closeout](docs/roadmap/phase-02b-events-auth.md#p02b-1-merge-and-closeout-2026-10-12) records transactional event production, enforcement, completed reviews and final -CI verification under Accepted ADR-0057. P02b-2's revised package and ADR-0046 -are Accepted. Steps 1–2 completed both review rounds; Step 3 implements hosted -dispatch and observation, with both independent review rounds complete. Its -[delivery record](docs/roadmap/phase-02b-events-auth.md#p02b-2-step-3--hosted-lifecycle-and-observability-2026-10-12) -owns its runtime evidence. Step 4 enforcement and local closeout are complete; -all four steps completed both independent review rounds. CI and merge status -live in the -[delivery record](docs/roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12). +CI verification under Accepted ADR-0057. **P02b-2 is complete and merged** through +[PR #30](https://github.com/HodeTech/LearnStack/pull/30) on 2026-10-13. Its +[merge closeout](docs/roadmap/phase-02b-events-auth.md#p02b-2-merge-and-closeout-2026-10-13) +records hosted dispatch, terminal audit, observation/enforcement, completed +reviews and final CI verification under Accepted ADR-0046. P02b-3 implements +durable subscriber acceptance, shared inbox/delivery ledger, both real default +subscriptions and Seeder handoff. Its +[Step 6 record](docs/roadmap/phase-02b-events-auth.md#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) +owns enforcement and current review/PR status. P02b-4 background jobs follows; +Phase 02b remains in progress. [P02d-1](docs/roadmap/phase-02d-walking-skeleton.md#merge-and-closeout-2026-09-14) is **complete and merged**: Education domain, schema and isolation proofs. **P02d-2's decision package and ADR-0050/0051 are Accepted** as of 2026-10-02, @@ -128,10 +130,14 @@ proof pass real CI; the seventh required context is active. The owns CI enforcement, reviews and the positive phase-exit verdict. [PR #28](https://github.com/HodeTech/LearnStack/pull/28) is merged; the [merge closeout](docs/roadmap/phase-02d-walking-skeleton.md#p02d-7-merge-and-closeout-2026-10-11) -owns final verification. P02b-0 and P02b-1 are complete; P02b-2 producer dispatch -and enforcement are locally complete, with both independent review rounds complete -for all four steps; the Step 4 record above owns CI/merge status. Consumer -delivery remains P02b-3. The +owns final verification. P02b-0 through P02b-2 are complete; the +[P02b-2 merge closeout](docs/roadmap/phase-02b-events-auth.md#p02b-2-merge-and-closeout-2026-10-13) +owns final source/merge verification. P02b-3 Steps 1–5 implement durable acceptance, +shared inbox/ledger, independent subscriber scheduling and both useful defaults +with semantic Seeder handoff. Its +[Step 6 record](docs/roadmap/phase-02b-events-auth.md#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) +owns completed enforcement/regression and both review rounds for all six steps; +PR review and merge remain pending. The [P02d-4 decision package](docs/roadmap/phase-02d-walking-skeleton.md#p02d-4-decision-package-2026-10-03) is Accepted — 2026-10-03. All four implementation steps and their review loops are complete. P02d-5 supplies trusted server rendering; P02d-6 supplies public @@ -140,10 +146,10 @@ page consumers and P02d-7 the two-host browser demo. | Area | Delivered now | Next milestone | |---|---|---| | **Tenancy** | Tenant provisioning, organizations, locales, typed settings/branding reads, host resolution and database isolation | User membership and permissions in [Phase 03](docs/roadmap/phase-03-identity-admin.md) | -| **Customization** | Content types, level taxonomies, exact-definition and generation-cached batched display readers, text-card metadata validation and tenant-authored seeds | Remaining authoring capabilities across [Phases 04–08a](docs/roadmap/README.md) | +| **Customization** | Content types, level taxonomies, exact-definition and generation-cached batched display readers, text-card metadata validation, create-only provisioning defaults and semantic built-in seed verification | Remaining authoring capabilities across [Phases 04–08a](docs/roadmap/README.md) | | **Audit** | Classified write path and transactional durability for business changes | Operational hardening in [Phase 11](docs/roadmap/phase-11-production-hardening.md) | | **Education** | Course and Lesson aggregates, translations, protected-content policy, scoped authoring commands, complete demo seeds and isolated public GET/HEAD reads | Full versioned curriculum in [Phase 05](docs/roadmap/phase-05-education-learning-content.md) | -| **API foundation** | Error contracts, validation, tenancy, concurrency, observability, transactional event production and hosted producer dispatch with production-census enforcement (all four P02b-2 steps and both review rounds locally complete) | Authentication and durable event consumption in [Phase 02b](docs/roadmap/phase-02b-events-auth.md) | +| **API foundation** | Error contracts, validation, tenancy, concurrency, observability, transactional event production, hosted producer dispatch and independent durable subscriber delivery | Background jobs and authentication in [Phase 02b](docs/roadmap/phase-02b-events-auth.md) | | **Frontend** | Native TLS ingress, live locale entry, trusted server caller, localized catalog/course/lesson pages, safe themes, accessibility controls/manual smoke and generated typed SDK | Full renderer/Studio and Playwright Test/axe in [Phase 06](docs/roadmap/phase-06-renderer-admin-studio.md) | **Four modules contain domain implementations:** Tenancy, Customization, Audit and diff --git a/backend/src/LearnStack.Api/Composition/CrossCuttingFoundationExtensions.cs b/backend/src/LearnStack.Api/Composition/CrossCuttingFoundationExtensions.cs index 45f596c5..51238a20 100644 --- a/backend/src/LearnStack.Api/Composition/CrossCuttingFoundationExtensions.cs +++ b/backend/src/LearnStack.Api/Composition/CrossCuttingFoundationExtensions.cs @@ -1,3 +1,6 @@ +using LearnStack.Infrastructure.Messaging; +using LearnStack.Modules.Tenancy.Application.Contracts.IntegrationEvents; +using LearnStack.Modules.Customization.Application.IntegrationEvents; using LearnStack.Api.Common; using LearnStack.Application.Pipeline; using LearnStack.Infrastructure.ErrorTracking; @@ -137,6 +140,15 @@ public static WebApplicationBuilder AddLearnStackCrossCuttingFoundation( } builder.Services.TryAddSingleton(integrationEventHandlers); + builder.Services.TryAddSingleton(provider => OutboxEventRegistry.Create( + [OutboxEventRegistration.Required( + typeof(TenantCreatedIntegrationEventV1), + [OutboxSubscriptionManifest.Active("customization.initialize-content-type-default.v1", + typeof(InitializeContentTypeDefaultHandler)), + OutboxSubscriptionManifest.Active("customization.initialize-taxonomy-default.v1", + typeof(InitializeTaxonomyDefaultHandler))])], + [], provider.GetRequiredService(), + [typeof(TenantCreatedIntegrationEventV1).Assembly])); builder.Services.TryAddSingleton(SelectEventBus); builder.Services.AddProblemDetails(); @@ -304,11 +316,7 @@ private static LearnStack.SharedKernel.Messaging.IEventBus SelectEventBus( services.GetRequiredService(), services.GetRequiredService(), services.GetRequiredService(), - services.GetRequiredService< - LearnStack.Infrastructure.Messaging.IntegrationEventHandlerRegistry>(), - services.GetRequiredService< - Microsoft.Extensions.Logging.ILogger< - LearnStack.Infrastructure.Messaging.InProcessEventBus>>()); + services.GetRequiredService()); } /// diff --git a/backend/src/LearnStack.Api/Composition/OutboxCompositionExtensions.cs b/backend/src/LearnStack.Api/Composition/OutboxCompositionExtensions.cs index 4ab9d192..da8ddb13 100644 --- a/backend/src/LearnStack.Api/Composition/OutboxCompositionExtensions.cs +++ b/backend/src/LearnStack.Api/Composition/OutboxCompositionExtensions.cs @@ -7,7 +7,7 @@ namespace LearnStack.Api.Composition; -/// Owns the dispatcher credential in the API composition root. +/// Registers producer/subscriber dispatch and its API-owned dispatcher credential. public static class OutboxCompositionExtensions { public static IServiceCollection AddLearnStackOutboxDispatcher( @@ -50,6 +50,22 @@ public static IServiceCollection AddLearnStackOutboxDispatcher( services.AddHostedService(); services.AddHealthChecks().AddCheck(OutboxHealthCheck.Name); } + services.TryAddSingleton(new DeliveryDispatchOptions()); + services.TryAddSingleton(); + services.TryAddSingleton(); + services.TryAddSingleton(provider => new DeliveryDispatchStore( + provider.GetRequiredKeyedService>(OutboxDispatcherDataSource.Key), + provider.GetRequiredService(), + provider.GetRequiredService(), + provider.GetRequiredService(), + provider.GetRequiredService(), + provider.GetRequiredService())); + if (!services.Any(descriptor => descriptor.ServiceType == typeof(IHostedService) + && descriptor.ImplementationType == typeof(DeliveryProcessor))) + { + services.AddHostedService(); + services.AddHealthChecks().AddCheck(DeliveryHealthCheck.Name); + } return services; } } diff --git a/backend/src/LearnStack.Api/Composition/PersistenceCompositionExtensions.cs b/backend/src/LearnStack.Api/Composition/PersistenceCompositionExtensions.cs index ac3aca77..4661ac0a 100644 --- a/backend/src/LearnStack.Api/Composition/PersistenceCompositionExtensions.cs +++ b/backend/src/LearnStack.Api/Composition/PersistenceCompositionExtensions.cs @@ -2,7 +2,6 @@ using LearnStack.Modules.Customization.Application.Contracts.Definitions; using LearnStack.Modules.Tenancy.Application.Contracts.Locales; using LearnStack.Modules.Tenancy.Application.Contracts.PublicReads; -using LearnStack.Modules.Tenancy.Application.Contracts.IntegrationEvents; using LearnStack.Modules.Education.Application.Audit; using LearnStack.Infrastructure.Audit; using LearnStack.Modules.Customization.Application.Audit; @@ -103,13 +102,6 @@ public static IServiceCollection AddLearnStackPersistence( ArgumentNullException.ThrowIfNull(configuration); services.AddLearnStackOutboxDispatcher(configuration); - services.TryAddSingleton(provider => OutboxEventRegistry.Create( - [OutboxEventRegistration.Required( - typeof(TenantCreatedIntegrationEventV1), - [OutboxSubscriptionManifest.Pending("customization.initialize-content-type-default.v1"), - OutboxSubscriptionManifest.Pending("customization.initialize-taxonomy-default.v1")])], - [], provider.GetRequiredService(), - [typeof(TenantCreatedIntegrationEventV1).Assembly])); var connectionString = configuration.GetConnectionString(DefaultConnectionName); @@ -279,6 +271,8 @@ public static IServiceCollection AddLearnStackPersistence( services.TryAddScoped(); services.TryAddScoped(); + services.TryAddScoped(); + services.TryAddScoped(); // The catalogue, merged once from every module's source. A singleton: it is built // at composition time and read on every request, and rebuilding it per scope would diff --git a/backend/src/LearnStack.Api/Messaging/DeliveryAttempt.cs b/backend/src/LearnStack.Api/Messaging/DeliveryAttempt.cs new file mode 100644 index 00000000..b1d6200b --- /dev/null +++ b/backend/src/LearnStack.Api/Messaging/DeliveryAttempt.cs @@ -0,0 +1,27 @@ +using LearnStack.Infrastructure.Messaging; + +namespace LearnStack.Api.Messaging; + +/// The retained task owns construction, callbacks and the actual application scope. +internal sealed class DeliveryAttempt +{ + private readonly DeliveryExecutionControl _control = new(); + + public DeliveryAttempt(DeliveryExecutor executor, DeliveryClaim claim) + { + ArgumentNullException.ThrowIfNull(executor); + ArgumentNullException.ThrowIfNull(claim); + PartitionKey = claim.PartitionKey; + Completion = Task.Run(() => executor.ExecuteControlledAsync(claim, _control), CancellationToken.None); + // Observe even an unexpected fault outside the executor's owned-scope + // boundary after the scheduler has already stopped waiting. + _ = Completion.ContinueWith(static completed => { _ = completed.Exception; }, + CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default); + } + + public string PartitionKey { get; } + public bool CommitAcknowledged => _control.Gate.CommitAcknowledged; + public bool CommitStarted => _control.Gate.CommitStarted; + public Task Completion { get; } + public void RequestCancellation() => _control.RequestCancellation(); +} diff --git a/backend/src/LearnStack.Api/Messaging/DeliveryHealthCheck.cs b/backend/src/LearnStack.Api/Messaging/DeliveryHealthCheck.cs new file mode 100644 index 00000000..006d5404 --- /dev/null +++ b/backend/src/LearnStack.Api/Messaging/DeliveryHealthCheck.cs @@ -0,0 +1,14 @@ +using Microsoft.Extensions.Diagnostics.HealthChecks; + +namespace LearnStack.Api.Messaging; + +/// Private readiness observation; the public liveness route does not select this check. +public sealed class DeliveryHealthCheck(DeliveryObservation observation) : IHealthCheck +{ + public const string Name = "delivery"; + + public Task CheckHealthAsync(HealthCheckContext context, + CancellationToken cancellationToken = default) => Task.FromResult(observation.IsHealthy + ? HealthCheckResult.Healthy() + : HealthCheckResult.Unhealthy("Delivery dispatch is not healthy.")); +} diff --git a/backend/src/LearnStack.Api/Messaging/DeliveryObservation.cs b/backend/src/LearnStack.Api/Messaging/DeliveryObservation.cs new file mode 100644 index 00000000..3ededa06 --- /dev/null +++ b/backend/src/LearnStack.Api/Messaging/DeliveryObservation.cs @@ -0,0 +1,133 @@ +using System.Collections.Frozen; +using System.Diagnostics; +using System.Diagnostics.Metrics; +using LearnStack.Infrastructure.Messaging; + +namespace LearnStack.Api.Messaging; + +/// Immutable process observation; timestamps use TimeProvider's monotonic clock. +public sealed record DeliveryObservationSnapshot( + DeliveryQueueSnapshot Queue, + bool Started = false, + bool Stopped = false, + long? LastProgressTimestamp = null, + long? LastSnapshotTimestamp = null, + int OutstandingExecutions = 0, + bool RefreshFailed = false, + bool LoopFailed = false, + bool TerminalAuditFailed = false); + +/// Bounded dispatch telemetry; observable callbacks only read memory. +public sealed class DeliveryObservation : IDisposable +{ + public const string InstrumentationName = "learnstack.delivery"; + private readonly object _gate = new(); + private readonly TimeProvider _time; + private readonly DeliveryDispatchOptions _options; + private readonly FrozenSet _labels; + private readonly FrozenSet _subscriptions; + private readonly Meter _meter = new(InstrumentationName); + private readonly ActivitySource _activities = new(InstrumentationName); + private readonly Histogram _duration; + private readonly Counter _failed; + private readonly Counter _deadletter; + private readonly Counter _dedup; + private readonly Counter _leaseLost; + private DeliveryObservationSnapshot _snapshot = new(new DeliveryQueueSnapshot(0, 0, 0)); + + public DeliveryObservation(TimeProvider time, DeliveryDispatchOptions options, OutboxEventRegistry registry) + { + ArgumentNullException.ThrowIfNull(time); + ArgumentNullException.ThrowIfNull(options); + ArgumentNullException.ThrowIfNull(registry); + _time = time; + _options = options; + _labels = registry.EventTypeLabels.Concat(["unknown", "retired", "invalid"]) + .ToFrozenSet(StringComparer.Ordinal); + _subscriptions = registry.SubscriptionIds.ToFrozenSet(StringComparer.Ordinal); + _dedup = _meter.CreateCounter("learnstack_inbox_dedup_total"); + _duration = _meter.CreateHistogram("learnstack_delivery_execute_duration_seconds", "s"); + _failed = _meter.CreateCounter("learnstack_delivery_failed_total"); + _deadletter = _meter.CreateCounter("learnstack_inbox_deadletter_total"); + _leaseLost = _meter.CreateCounter("learnstack_delivery_lease_lost_total"); + _meter.CreateObservableGauge("learnstack_delivery_pending_count", () => Snapshot.Queue.PendingCount); + _meter.CreateObservableGauge("learnstack_delivery_oldest_eligible_age_seconds", + () => Snapshot.Queue.OldestEligibleAgeSeconds, "s"); + _meter.CreateObservableGauge("learnstack_delivery_deferred_count", () => Snapshot.Queue.DeferredCount); + _meter.CreateObservableGauge("learnstack_delivery_heartbeat_age_seconds", () => HeartbeatAge.TotalSeconds, "s"); + } + + public DeliveryObservationSnapshot Snapshot => Volatile.Read(ref _snapshot); + + public TimeSpan HeartbeatAge => Age(Snapshot.LastProgressTimestamp); + + public bool IsHealthy + { + get + { + var snapshot = Snapshot; + return snapshot.Started && !snapshot.Stopped && !snapshot.RefreshFailed + && !snapshot.LoopFailed && !snapshot.TerminalAuditFailed + && snapshot.OutstandingExecutions == 0 + && Age(snapshot.LastProgressTimestamp) < _options.ProgressTimeout + && Age(snapshot.LastSnapshotTimestamp) < _options.ProgressTimeout; + } + } + + internal void Started() => Update(value => value with { Started = true, Stopped = false }); + internal void Stopped() => Update(value => value with { Stopped = true }); + internal void Progress() => Update(value => value with + { + LastProgressTimestamp = _time.GetTimestamp(), + LoopFailed = false, + }); + internal void Refreshed(DeliveryQueueSnapshot queue) => Update(value => value with + { + Queue = queue, + LastSnapshotTimestamp = _time.GetTimestamp(), + RefreshFailed = false, + }); + internal void RefreshFailed() => Update(value => value with { RefreshFailed = true }); + internal void LoopFailed() => Update(value => value with { LoopFailed = true }); + internal void TerminalAuditFailed() => Update(value => value with { TerminalAuditFailed = true }); + internal void TerminalAuditSucceeded() => Update(value => value with { TerminalAuditFailed = false }); + internal void Outstanding(int count) => Update(value => value with { OutstandingExecutions = count }); + internal void LeaseLost() => _leaseLost.Add(1); + internal void Failed(string label, string subscription) => _failed.Add(1, Labels(label, subscription)); + internal void Deadletter(string label, string subscription) => _deadletter.Add(1, Labels(label, subscription)); + internal void Deduplicated(string label, string subscription) => _dedup.Add(1, Labels(label, subscription)); + internal void Duration(string label, string subscription, TimeSpan duration) => + _duration.Record(Math.Max(0, duration.TotalSeconds), Labels(label, subscription)); + + internal Activity BeginAttempt(string label, string subscription) + { + // The scheduler has no causal-envelope SELECT grant. Every attempt starts + // its own canonical trace; the application executor restores original causality. + var parent = new ActivityContext(ActivityTraceId.CreateRandom(), ActivitySpanId.CreateRandom(), ActivityTraceFlags.None); + var activity = _activities.StartActivity("delivery.dispatch", ActivityKind.Internal, + parent) + ?? new Activity("delivery.dispatch").SetIdFormat(ActivityIdFormat.W3C) + .SetParentId(parent.TraceId, parent.SpanId, parent.TraceFlags).Start(); + activity.SetTag("event_type", SafeLabel(label)); + activity.SetTag("subscription", SafeSubscription(subscription)); + return activity; + } + + private KeyValuePair[] Labels(string label, string subscription) => + [new("event_type", SafeLabel(label)), new("subscription", SafeSubscription(subscription))]; + private string SafeLabel(string label) => _labels.Contains(label) ? label : "invalid"; + private string SafeSubscription(string subscription) => _subscriptions.Contains(subscription) ? subscription : "invalid"; + private TimeSpan Age(long? timestamp) => timestamp is { } value + ? _time.GetElapsedTime(value, _time.GetTimestamp()) : _options.ProgressTimeout; + + private void Update(Func change) + { + lock (_gate) Volatile.Write(ref _snapshot, change(_snapshot)); + } + + public void Dispose() + { + _activities.Dispose(); + _meter.Dispose(); + } +} diff --git a/backend/src/LearnStack.Api/Messaging/DeliveryProcessor.cs b/backend/src/LearnStack.Api/Messaging/DeliveryProcessor.cs new file mode 100644 index 00000000..568fac24 --- /dev/null +++ b/backend/src/LearnStack.Api/Messaging/DeliveryProcessor.cs @@ -0,0 +1,380 @@ +using System.Diagnostics; +using LearnStack.Infrastructure.Messaging; +using Npgsql; + +namespace LearnStack.Api.Messaging; + +/// One just-in-time claimant with bounded, task-owned execution overruns. +public sealed partial class DeliveryProcessor : BackgroundService +{ + private readonly Lazy _source; + private readonly DeliveryDispatchStore _store; + private readonly DeliveryExecutor _executor; + private readonly OutboxEventRegistry _registry; + private readonly DeliveryDispatchOptions _options; + private readonly TimeProvider _time; + private readonly DeliveryObservation _observation; + private readonly ILogger _logger; + private readonly object _attemptGate = new(); + private readonly List _attempts = []; + private DeliveryClaimKind _nextClaimKind = DeliveryClaimKind.Terminal; + + public DeliveryProcessor( + [FromKeyedServices(OutboxDispatcherDataSource.Key)] Lazy source, + DeliveryDispatchStore store, DeliveryExecutor executor, OutboxEventRegistry registry, + DeliveryDispatchOptions options, TimeProvider time, + DeliveryObservation observation, ILogger logger) + { + ArgumentNullException.ThrowIfNull(source); + ArgumentNullException.ThrowIfNull(store); + ArgumentNullException.ThrowIfNull(executor); + ArgumentNullException.ThrowIfNull(registry); + ArgumentNullException.ThrowIfNull(options); + ArgumentNullException.ThrowIfNull(time); + ArgumentNullException.ThrowIfNull(observation); + ArgumentNullException.ThrowIfNull(logger); + _source = source; + _store = store; + _executor = executor; + _registry = registry; + _options = options; + _time = time; + _observation = observation; + _logger = logger; + } + + public override async Task StartAsync(CancellationToken cancellationToken) + { + // .NET 10 runs ExecuteAsync in the background. Startup refusal therefore + // belongs here, before the host can report that it has started. + _options.Validate(); + try + { + using var timeout = new CancellationTokenSource(_options.SqlTimeout, _time); + using var deadline = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, timeout.Token); + await using var connection = await _source.Value.OpenConnectionAsync(deadline.Token).ConfigureAwait(false); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw new OperationCanceledException(cancellationToken); + } + catch (Exception) + { + // No connection data, provider exception or inner exception escapes boot. + throw new InvalidOperationException("Delivery dispatcher startup was refused. " + + "Configure ConnectionStrings:OutboxDispatcher in Npgsql key/value form " + + "for learnstack_outbox_admin and verify database connectivity."); + } + + _observation.Started(); + await base.StartAsync(cancellationToken).ConfigureAwait(false); + } + + protected override async Task ExecuteAsync(CancellationToken stoppingToken) + { + try + { + while (!stoppingToken.IsCancellationRequested) + { + var blocked = RetainedPartitions(); + var didWork = false; + try + { + if (OutstandingCount < _options.MaxOutstandingExecutions) + { + var claim = await ClaimNextAsync(blocked, stoppingToken).ConfigureAwait(false); + if (claim is not null) + { + didWork = true; + await DispatchAsync(claim, stoppingToken).ConfigureAwait(false); + } + + // Even an empty completed poll demonstrates real bounded work. + _observation.Progress(); + } + + await RefreshAsync(stoppingToken).ConfigureAwait(false); + } + catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested) + { + break; + } + catch (Exception) + { + _observation.LoopFailed(); + LogIterationFailure(_logger); + didWork = false; + } + + if (!didWork) + await Task.Delay(_options.IdlePoll, _time, stoppingToken).ConfigureAwait(false); + } + } + catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested) + { + // Normal hosted-service cancellation; live attempts own their cleanup. + } + finally + { + CancelAttempts(); + _observation.Stopped(); + } + } + + private async Task ClaimNextAsync(string[] blocked, CancellationToken cancellationToken) + { + // Service every eligible class within three claims, even when terminal + // audit keeps failing. Empty classes fall through without claiming work. + for (var checkedKinds = 0; checkedKinds < 3; checkedKinds++) + { + var kind = _nextClaimKind; + // Advance before acquisition/dispatch: a fault must not pin its class. + _nextClaimKind = kind switch + { + DeliveryClaimKind.Terminal => DeliveryClaimKind.Recovery, + DeliveryClaimKind.Recovery => DeliveryClaimKind.Normal, + _ => DeliveryClaimKind.Terminal, + }; + var claim = kind switch + { + DeliveryClaimKind.Terminal => await _store.TryClaimTerminalAsync(blocked, cancellationToken).ConfigureAwait(false), + DeliveryClaimKind.Recovery => await _store.TryAcquireRecoveryAsync(blocked, cancellationToken).ConfigureAwait(false), + _ => await _store.TryClaimAsync(blocked, cancellationToken).ConfigureAwait(false), + }; + if (claim is not null) return claim; + } + return null; + } + + private async Task DispatchAsync(DeliveryClaim claim, CancellationToken stoppingToken) + { + if (claim.Kind == DeliveryClaimKind.Normal) + { + await DispatchNormalAsync(claim, stoppingToken).ConfigureAwait(false); + return; + } + + var label = _registry.Resolve(claim.Type).EventTypeLabel; + using var activity = _observation.BeginAttempt(label, claim.SubscriptionId); + if (claim.Kind == DeliveryClaimKind.Recovery) + { + Fence(await _store.FailAsync(claim, DeliveryFailureReason.UnknownOutcome, stoppingToken).ConfigureAwait(false)); + activity.SetTag("outcome", "recovered"); + LogAttempt(activity, label); + return; + } + + try + { + // BeginAttempt always starts a W3C Activity, including without listeners. + var terminalized = await _store.TryTerminalizeAsync(claim, activity.Id!, stoppingToken).ConfigureAwait(false); + Fence(terminalized); + if (terminalized) + { + _observation.TerminalAuditSucceeded(); + _observation.Deadletter(label, claim.SubscriptionId); + } + activity.SetTag("outcome", terminalized ? "deadlettered" : "lease_lost"); + } + catch (Exception) + { + _observation.TerminalAuditFailed(); + activity.SetTag("outcome", "terminal_audit_failed"); + throw; + } + finally + { + LogAttempt(activity, label); + } + } + + private async Task DispatchNormalAsync(DeliveryClaim claim, CancellationToken stoppingToken) + { + var resolution = _registry.Resolve(claim.Type); + var label = resolution.EventTypeLabel; + using var activity = _observation.BeginAttempt(label, claim.SubscriptionId); + var started = _time.GetTimestamp(); + try + { + var terminal = resolution.Disposition switch + { + OutboxEventDisposition.Unknown => DeliveryTerminalReason.UnknownType, + OutboxEventDisposition.Retired => DeliveryTerminalReason.RetiredType, + _ => _registry.IsActiveOrdinarySubscription(claim.Type, claim.SubscriptionId) + ? (DeliveryTerminalReason?)null : DeliveryTerminalReason.InvalidSubscription, + }; + if (terminal is { } reason) + { + Fence(await _store.RequestTerminalAsync(claim, reason, stoppingToken).ConfigureAwait(false)); + activity.SetTag("outcome", "terminal_requested"); + return; + } + + var attempt = new DeliveryAttempt(_executor, claim); + lock (_attemptGate) _attempts.Add(attempt); + DeliveryExecutionResult result; + try + { + result = await attempt.Completion.WaitAsync(_options.ExecutionTimeout, _time, stoppingToken) + .ConfigureAwait(false); + } + catch (TimeoutException) + { + attempt.RequestCancellation(); + RetainedPartitions(); + // Acknowledged business durability survives a hung cleanup. + if (attempt.CommitAcknowledged) + { + activity.SetTag("outcome", "completed_cleanup_timeout"); + return; + } + _observation.Failed(label, claim.SubscriptionId); + activity.SetTag("outcome", "execution_timeout"); + Fence(await _store.FailAsync(claim, attempt.CommitStarted + ? DeliveryFailureReason.UnknownOutcome : DeliveryFailureReason.ExecutionTimeout, + stoppingToken).ConfigureAwait(false)); + return; + } + catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested) + { + attempt.RequestCancellation(); + RetainedPartitions(); + activity.SetTag("outcome", "shutdown"); + throw; + } + + RetainedPartitions(); + stoppingToken.ThrowIfCancellationRequested(); + if (result.CommitAcknowledged) + { + if (result.Outcome == DeliveryExecutionOutcome.Duplicate) + _observation.Deduplicated(label, claim.SubscriptionId); + activity.SetTag("outcome", result.CleanupFailed ? "completed_cleanup_failed" + : result.Outcome == DeliveryExecutionOutcome.Duplicate ? "duplicate" : "completed"); + return; + } + if (result.Outcome == DeliveryExecutionOutcome.LostClaim) + { + _observation.LeaseLost(); + activity.SetTag("outcome", "lease_lost"); + return; + } + if (result.Outcome is DeliveryExecutionOutcome.InvalidEnvelope or DeliveryExecutionOutcome.InvalidSubscription) + { + Fence(await _store.RequestTerminalAsync(claim, + result.Outcome == DeliveryExecutionOutcome.InvalidEnvelope + ? DeliveryTerminalReason.InvalidEnvelope : DeliveryTerminalReason.InvalidSubscription, + stoppingToken).ConfigureAwait(false)); + activity.SetTag("outcome", "terminal_requested"); + return; + } + _observation.Failed(label, claim.SubscriptionId); + Fence(await _store.FailAsync(claim, result.Outcome switch + { + DeliveryExecutionOutcome.Indeterminate => DeliveryFailureReason.UnknownOutcome, + DeliveryExecutionOutcome.Refused => DeliveryFailureReason.ExecutionRefused, + _ => DeliveryFailureReason.ExecutionFailed, + }, + stoppingToken).ConfigureAwait(false)); + activity.SetTag("outcome", result.Outcome switch + { + DeliveryExecutionOutcome.Indeterminate => "unknown_outcome", + DeliveryExecutionOutcome.Refused => "catalogue_refused", + _ => "execution_failed", + }); + } + finally + { + _observation.Duration(label, claim.SubscriptionId, _time.GetElapsedTime(started, _time.GetTimestamp())); + LogAttempt(activity, label); + } + } + + private void LogAttempt(Activity activity, string label) + { + if (_logger.IsEnabled(LogLevel.Information)) + { + var outcome = activity.GetTagItem("outcome") as string ?? "storage_failure"; + var subscription = activity.GetTagItem("subscription") as string ?? "invalid"; + LogAttemptFinished(_logger, activity.Id, label, subscription, outcome); + } + } + + [LoggerMessage(Level = LogLevel.Warning, Message = "Delivery dispatch iteration failed with outcome iteration_failed.")] + private static partial void LogIterationFailure(ILogger logger); + + [LoggerMessage(Level = LogLevel.Information, + Message = "Delivery dispatch attempt {CorrelationId} finished with event type {EventType}, subscription {Subscription} and outcome {Outcome}.")] + private static partial void LogAttemptFinished(ILogger logger, string? correlationId, string eventType, string subscription, string outcome); + + private async Task RefreshAsync(CancellationToken cancellationToken) + { + var last = _observation.Snapshot.LastSnapshotTimestamp; + if (last is { } timestamp + && _time.GetElapsedTime(timestamp, _time.GetTimestamp()) < _options.SnapshotInterval) return; + try + { + var queue = await _store.ReadSnapshotAsync(RetainedPartitions(), cancellationToken).ConfigureAwait(false); + _observation.Refreshed(queue); + } + catch (Exception) + { + _observation.RefreshFailed(); + throw; + } + } + + private void Fence(bool changed) + { + if (!changed) _observation.LeaseLost(); + } + + private int OutstandingCount + { + get { lock (_attemptGate) return _attempts.Count; } + } + + private string[] RetainedPartitions() + { + lock (_attemptGate) + { + // Completion includes cancellation callbacks and asynchronous scope + // disposal; only then may this partition be acquired locally again. + _attempts.RemoveAll(attempt => attempt.Completion.IsCompleted); + _observation.Outstanding(_attempts.Count); + return _attempts.Select(attempt => attempt.PartitionKey).Distinct(StringComparer.Ordinal).ToArray(); + } + } + + private DeliveryAttempt[] CancelAttempts() + { + DeliveryAttempt[] attempts; + lock (_attemptGate) attempts = _attempts.ToArray(); + foreach (var attempt in attempts) attempt.RequestCancellation(); + return attempts; + } + + public override async Task StopAsync(CancellationToken cancellationToken) + { + // The base service and callback drain share one overall budget. Never add + // a second shutdown wait after the host has already exhausted the first. + using var timeout = new CancellationTokenSource(_options.ShutdownTimeout, _time); + using var deadline = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, timeout.Token); + try + { + await base.StopAsync(deadline.Token).ConfigureAwait(false); + var attempts = CancelAttempts(); + await Task.WhenAll(attempts.Select(attempt => attempt.Completion)).WaitAsync(deadline.Token) + .ConfigureAwait(false); + } + catch (OperationCanceledException) when (deadline.IsCancellationRequested) + { + CancelAttempts(); + } + finally + { + RetainedPartitions(); + _observation.Stopped(); + } + } +} diff --git a/backend/src/LearnStack.Api/Messaging/OutboxProcessor.cs b/backend/src/LearnStack.Api/Messaging/OutboxProcessor.cs index 1e42450c..e29e9af3 100644 --- a/backend/src/LearnStack.Api/Messaging/OutboxProcessor.cs +++ b/backend/src/LearnStack.Api/Messaging/OutboxProcessor.cs @@ -18,6 +18,7 @@ public sealed partial class OutboxProcessor : BackgroundService private readonly ILogger _logger; private readonly object _attemptGate = new(); private readonly List _attempts = []; + private OutboxClaimKind _nextClaimKind = OutboxClaimKind.Terminal; public OutboxProcessor( [FromKeyedServices(OutboxDispatcherDataSource.Key)] Lazy source, @@ -84,11 +85,7 @@ protected override async Task ExecuteAsync(CancellationToken stoppingToken) { if (OutstandingCount < _options.MaxOutstandingPublishes) { - // Terminal requests and expired acquisitions never spend - // another publication attempt, and take priority over new work. - var claim = await _store.TryClaimTerminalAsync(blocked, stoppingToken).ConfigureAwait(false) - ?? await _store.TryAcquireRecoveryAsync(blocked, stoppingToken).ConfigureAwait(false) - ?? await _store.TryClaimAsync(blocked, stoppingToken).ConfigureAwait(false); + var claim = await ClaimNextAsync(blocked, stoppingToken).ConfigureAwait(false); if (claim is not null) { didWork = true; @@ -127,6 +124,31 @@ protected override async Task ExecuteAsync(CancellationToken stoppingToken) } } + private async Task ClaimNextAsync(string[] blocked, CancellationToken cancellationToken) + { + // Service every eligible class within three claims, even when terminal + // audit keeps failing. Empty classes fall through without claiming work. + for (var checkedKinds = 0; checkedKinds < 3; checkedKinds++) + { + var kind = _nextClaimKind; + // Advance before acquisition/dispatch: a fault must not pin its class. + _nextClaimKind = kind switch + { + OutboxClaimKind.Terminal => OutboxClaimKind.Recovery, + OutboxClaimKind.Recovery => OutboxClaimKind.Normal, + _ => OutboxClaimKind.Terminal, + }; + var claim = kind switch + { + OutboxClaimKind.Terminal => await _store.TryClaimTerminalAsync(blocked, cancellationToken).ConfigureAwait(false), + OutboxClaimKind.Recovery => await _store.TryAcquireRecoveryAsync(blocked, cancellationToken).ConfigureAwait(false), + _ => await _store.TryClaimAsync(blocked, cancellationToken).ConfigureAwait(false), + }; + if (claim is not null) return claim; + } + return null; + } + private async Task DispatchAsync(OutboxDispatchClaim claim, CancellationToken stoppingToken) { if (claim.Kind == OutboxClaimKind.Normal) diff --git a/backend/src/LearnStack.Application/Audit/AuditExecutionCoordinator.cs b/backend/src/LearnStack.Application/Audit/AuditExecutionCoordinator.cs index e6630801..9c5f4fe7 100644 --- a/backend/src/LearnStack.Application/Audit/AuditExecutionCoordinator.cs +++ b/backend/src/LearnStack.Application/Audit/AuditExecutionCoordinator.cs @@ -58,7 +58,8 @@ public async Task ExecuteAsync( // silent — which is why TryGet and not a count of entries tells them apart. if (!catalog.TryGet(operationType, out var registration)) { - LogUnclassified(logger, operationType.FullName ?? operationType.Name, null); + LogUnclassified(logger, tenantContext is EventTenantContext + ? "delivery" : operationType.FullName ?? operationType.Name, null); return Result.FailFor(AuditErrors.UnclassifiedOperation); } @@ -206,7 +207,7 @@ private async Task ReconcileAsync() // aggregate and none designated — is a programmer error that nothing else // reports, and it must not cost the next intent its attempt nor replace the // caller's outcome. - LogReconcileRowRefused(logger, intent.Operation, refused); + LogReconcileRowRefused(logger, intent.Operation, DiagnosticException(refused)); continue; } @@ -242,7 +243,7 @@ private async Task ReconcileAsync() { // A draft the store refused before any write — the platform sentinel — is a // caller error the store does not report (IAuditStore), so this is its alert. - LogReconcileRowRefused(logger, intent.Operation, refused); + LogReconcileRowRefused(logger, intent.Operation, DiagnosticException(refused)); } catch (Exception lost) { @@ -254,12 +255,18 @@ private async Task ReconcileAsync() // changes the response only when the operation would otherwise have // SUCCEEDED, and everything reaching here is already failing, refused or in // doubt, so the caller keeps its status. - LogReconcileRowLost(logger, intent.Operation, lost); + LogReconcileRowLost(logger, intent.Operation, DiagnosticException(lost)); } #pragma warning restore CA1031 } } + // Delivery diagnostics use the declared operation and fixed event code only. + // A nested command shares this context, so its failure cannot leak payloads + // through a provider/composer exception before the transport sees the fault. + private Exception? DiagnosticException(Exception failure) => + tenantContext is EventTenantContext ? null : failure; + /// /// Parks one intent per audited (resource, operation). /// @@ -341,7 +348,11 @@ private async Task DeclareAsync( // declared one and the intent and the catalogue cannot disagree. entry.OperationClass, entry.EntityType, - clock.UtcNow)); + clock.UtcNow) + { + CausationId = tenantContext is EventTenantContext delivery ? delivery.EventId : null, + CausalActorUserId = tenantContext is EventTenantContext ? tenantContext.CausalActorUserId : null, + }); } } diff --git a/backend/src/LearnStack.Application/Messaging/DeliveryCommitGate.cs b/backend/src/LearnStack.Application/Messaging/DeliveryCommitGate.cs new file mode 100644 index 00000000..68d67145 --- /dev/null +++ b/backend/src/LearnStack.Application/Messaging/DeliveryCommitGate.cs @@ -0,0 +1,31 @@ +namespace LearnStack.Application.Messaging; + +/// Sticky arbitration between delivery cancellation and physical commit admission. +public sealed class DeliveryCommitGate +{ + private const int Open = 0; + private const int Closed = 1; + private const int Committing = 2; + private int _state; + private int _commitAcknowledged; + private int _executionStarted; + + /// Definitive durability, independent of later owned resource cleanup. + public bool CommitAcknowledged => Volatile.Read(ref _commitAcknowledged) != 0; + + /// Admission has won; an unacknowledged commit may require durable reconciliation. + public bool CommitStarted => Volatile.Read(ref _state) == Committing; + + /// Cancellation won admission; a foreign cancellation exception alone does not close it. + public bool IsClosed => Volatile.Read(ref _state) == Closed; + + /// Binds this receipt to one executor invocation, including attempts canceled before entry. + public bool TryStartExecution() => Interlocked.CompareExchange(ref _executionStarted, 1, 0) == 0; + + /// Cancellation closes an open gate; it never operates on the transaction. + public void Close() => Interlocked.CompareExchange(ref _state, Closed, Open); + + internal bool TryBeginCommit() => Interlocked.CompareExchange(ref _state, Committing, Open) == Open; + + internal void RecordCommitted() => Volatile.Write(ref _commitAcknowledged, 1); +} diff --git a/backend/src/LearnStack.Application/Messaging/DeliveryCompletionParticipant.cs b/backend/src/LearnStack.Application/Messaging/DeliveryCompletionParticipant.cs new file mode 100644 index 00000000..03937733 --- /dev/null +++ b/backend/src/LearnStack.Application/Messaging/DeliveryCompletionParticipant.cs @@ -0,0 +1,37 @@ +using LearnStack.SharedKernel.Persistence; + +namespace LearnStack.Application.Messaging; + +/// +/// Closed delivery completion seam. The production subtype and all installation/call +/// sites are enumerated by architecture tests; this is not a general precommit hook. +/// +public abstract class DeliveryCompletionParticipant +{ + private readonly IUnitOfWork _unitOfWork; + + protected DeliveryCompletionParticipant(IUnitOfWork unitOfWork, DeliveryCommitGate gate) + { + _unitOfWork = unitOfWork ?? throw new ArgumentNullException(nameof(unitOfWork)); + Gate = gate ?? throw new ArgumentNullException(nameof(gate)); + } + + internal DeliveryCommitGate Gate { get; } + + internal void RequireUnit(IUnitOfWork unitOfWork) + { + if (!ReferenceEquals(_unitOfWork, unitOfWork)) + { + throw new InvalidOperationException("Delivery completion must share its prepared owner's unit."); + } + } + + internal Task FenceOwnerAsync(IUnitOfWork unitOfWork, CancellationToken cancellationToken) + { + RequireUnit(unitOfWork); + return FenceAsync(cancellationToken); + } + + /// Updates only this delivery's completion column on its bound live transaction. + protected abstract Task FenceAsync(CancellationToken cancellationToken); +} diff --git a/backend/src/LearnStack.Application/Messaging/ProducerExecution.cs b/backend/src/LearnStack.Application/Messaging/ProducerExecution.cs index 3f69e420..2bf7e5b0 100644 --- a/backend/src/LearnStack.Application/Messaging/ProducerExecution.cs +++ b/backend/src/LearnStack.Application/Messaging/ProducerExecution.cs @@ -21,7 +21,8 @@ public sealed class ProducerExecution(IUnitOfWork unitOfWork) /// Enrolls after announcement. An unresolved, event-free invocation is legal; /// actual production must pass . /// - internal Invocation BeginInvocation(bool isTransactionOwner, UserId? causalActor = null) + internal Invocation BeginInvocation(bool isTransactionOwner, UserId? causalActor = null, + Guid? incomingEventId = null) { if (_dispatch is not null) { @@ -52,7 +53,8 @@ internal Invocation BeginInvocation(bool isTransactionOwner, UserId? causalActor } var invocation = new Invocation(this, _current, announcement, hasOwner, isTransactionOwner, ownedActivity, - _current is null ? causalActor : _current.CausalActor); + _current is null ? causalActor : _current.CausalActor, + _current is null ? incomingEventId : _current.IncomingEventId); _current = invocation; return invocation; } @@ -110,7 +112,7 @@ public IntegrationEventEnvelope CreateEnvelope(IIntegrationEvent integrationEven } return new IntegrationEventEnvelope(integrationEvent, traceparent, - announcement.OrganizationId?.Value, CausationId: _dispatch?.EventId, + announcement.OrganizationId?.Value, CausationId: _dispatch?.EventId ?? _current!.IncomingEventId, ActorUserId: _dispatch is null ? _current!.CausalActor : _dispatch.Origin.CausalActor); } @@ -257,7 +259,7 @@ public sealed class Invocation : IDisposable internal Invocation( ProducerExecution execution, Invocation? parent, TenantContextAnnouncement? announcement, bool hasOwner, bool isTransactionOwner, - Activity? ownedActivity, UserId? causalActor) + Activity? ownedActivity, UserId? causalActor, Guid? incomingEventId) { _execution = execution; Parent = parent; @@ -266,6 +268,7 @@ internal Invocation( IsTransactionOwner = isTransactionOwner; _ownedActivity = ownedActivity; CausalActor = causalActor; + IncomingEventId = incomingEventId; } internal Invocation? Parent { get; } @@ -273,6 +276,7 @@ internal Invocation( internal bool HasOwner { get; } internal bool IsTransactionOwner { get; } internal UserId? CausalActor { get; } + internal Guid? IncomingEventId { get; } /// Successful production by this invocation or any descendant. public bool HasProducedWork { get; internal set; } diff --git a/backend/src/LearnStack.Application/Persistence/TransactionExecutionCoordinator.cs b/backend/src/LearnStack.Application/Persistence/TransactionExecutionCoordinator.cs index ff716e56..7da1eef4 100644 --- a/backend/src/LearnStack.Application/Persistence/TransactionExecutionCoordinator.cs +++ b/backend/src/LearnStack.Application/Persistence/TransactionExecutionCoordinator.cs @@ -1,3 +1,4 @@ +using System.Data.Common; using LearnStack.Application.Messaging; using LearnStack.SharedKernel.Audit; using LearnStack.SharedKernel.Identifiers; @@ -44,18 +45,81 @@ public async Task ExecuteAsync( } catch { - if (openingOwner) collector.Clear(); + if (openingOwner) + { + try + { + collector.Clear(); + } + catch + { + // Retain the setup failure; the DI scope still owns cleanup. + } + } + throw; } - await using var transactionScope = scope; + return await ExecuteFrameAsync(scope, provisioning, operationName, next, + null, null, null, cancellationToken); + } + + /// + /// Opens only the delivery's routing transaction, before full envelope metadata or + /// module services exist. The private token retains the actual owner frame. + /// + public static Task PrepareDeliveryOwnerAsync( + IUnitOfWork unitOfWork, ITenantContext routingContext, + CancellationToken cancellationToken = default) => + PreparedDeliveryOwner.CreateAsync(unitOfWork, routingContext, cancellationToken); + + /// Consumes the prepared owner without joining or announcing a second frame. + public Task ExecutePreparedDeliveryAsync( + PreparedDeliveryOwner owner, + DeliveryCompletionParticipant completion, + Guid incomingEventId, + Func> next, + CancellationToken cancellationToken = default) + where TResponse : IResultBase + { + ArgumentNullException.ThrowIfNull(owner); + ArgumentNullException.ThrowIfNull(completion); + ArgumentNullException.ThrowIfNull(next); + if (incomingEventId == Guid.Empty) + { + throw new ArgumentException("Delivery production requires its incoming event identity.", nameof(incomingEventId)); + } + + completion.RequireUnit(unitOfWork); + var frame = owner.Consume(unitOfWork, tenantContext); + return ExecuteFrameAsync(frame, null, "delivery", next, owner, completion, + incomingEventId, cancellationToken); + } + + private async Task ExecuteFrameAsync( + IUnitOfWorkScope scope, + IProvisionsTenant? provisioning, + string operationName, + Func> next, + PreparedDeliveryOwner? preparedOwner, + DeliveryCompletionParticipant? completion, + Guid? incomingEventId, + CancellationToken cancellationToken) + where TResponse : IResultBase + { var committing = false; + var failed = false; try { // Provisioning changes SQL authority only; the ambient accessor stays // unresolved. A resolved caller cannot announce another tenant here. - if (!tenantContext.IsResolved && provisioning is not null) + if (preparedOwner is not null) + { + preparedOwner.RequireLive(unitOfWork, tenantContext); + collector.RefreshTrackedSources(); + } + else if (!tenantContext.IsResolved && provisioning is not null) { await unitOfWork.SetProvisioningTenantContextAsync( provisioning.ProvisioningTenantId, cancellationToken); @@ -71,7 +135,7 @@ await unitOfWork.SetProvisioningTenantContextAsync( ? tenantContext.CausalActorUserId ?? (tenantContext.UserId == UserId.SystemActor ? null : tenantContext.UserId) : null; - using var invocation = producer.BeginInvocation(scope.IsOwner, causalActor); + using var invocation = producer.BeginInvocation(scope.IsOwner, causalActor, incomingEventId); var response = await next(); if (response.IsFailure) { @@ -99,6 +163,18 @@ await unitOfWork.SetProvisioningTenantContextAsync( await auditStore.WritePendingAsync(unitOfWork, cancellationToken); } + if (completion is not null) + { + preparedOwner!.RequireLive(unitOfWork, tenantContext); + await completion.FenceOwnerAsync(unitOfWork, cancellationToken); + preparedOwner.RequireLive(unitOfWork, tenantContext); + if (!completion.Gate.TryBeginCommit()) + { + throw new OperationCanceledException("Delivery commit admission is closed.", cancellationToken); + } + } + + // No await between the atomic admission above and the owned Complete call. committing = true; try { @@ -123,11 +199,16 @@ await unitOfWork.SetProvisioningTenantContextAsync( throw; } - if (scope.IsOwner) capture.MarkCommitted(); + if (scope.IsOwner) + { + completion?.Gate.RecordCommitted(); + capture.MarkCommitted(); + } return response; } catch when (!committing) { + failed = true; unitOfWork.MarkRollbackOnly(); // Cleanup must neither be abandoned by cancellation nor replace the // original dispatch/audit failure with a bookkeeping exception. @@ -137,7 +218,8 @@ await unitOfWork.SetProvisioningTenantContextAsync( } catch (Exception rollbackFailure) { - LogRollbackFailure(logger, operationName, rollbackFailure); + LogRollbackFailure(logger, operationName, + tenantContext is EventTenantContext ? null : rollbackFailure); } if (scope.IsOwner && capture.State != AuditIntentState.Indeterminate) @@ -147,12 +229,164 @@ await unitOfWork.SetProvisioningTenantContextAsync( throw; } + catch + { + failed = true; + throw; + } finally { - // Announcement/setup failures also reach here. Child completion never - // drains or clears sources belonging to the still-open physical owner. + await CleanupFrameAsync(scope, failed); + } + } + + private async ValueTask CleanupFrameAsync(IUnitOfWorkScope scope, bool failed) + { + // Announcement/setup failures also reach here. Child completion never + // drains or clears sources belonging to the still-open physical owner. + try + { if (scope.IsOwner) collector.Clear(); } + catch when (failed) + { + // A source's cleanup cannot replace the primary failure. + } + catch + { + failed = true; + throw; + } + finally + { + try + { + await scope.DisposeAsync(); + } + catch when (failed) + { + // Preserve the original dispatch/COMMIT/cleanup exception. + // Commit acknowledgment predates cleanup and remains definitive. + } + } + } + + /// + /// A single-use delivery owner, minted only by the closed preparation entry. An + /// unconsumed token rolls back, including failures before audit classification. + /// +#pragma warning disable CA1034 // Nesting keeps prepared owner construction private without friend assemblies. + public sealed class PreparedDeliveryOwner : IAsyncDisposable +#pragma warning restore CA1034 + { + private readonly IUnitOfWork _unit; + private readonly IUnitOfWorkScope _frame; + private readonly DbTransaction _transaction; + private readonly TenantContextAnnouncement _announcement; + private bool _consumed; + private bool _disposed; + + private PreparedDeliveryOwner(IUnitOfWork unit, IUnitOfWorkScope frame, + DbTransaction transaction, TenantContextAnnouncement announcement) + { + _unit = unit; + _frame = frame; + _transaction = transaction; + _announcement = announcement; + } + + internal static async Task CreateAsync( + IUnitOfWork unit, ITenantContext routingContext, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(unit); + ArgumentNullException.ThrowIfNull(routingContext); + cancellationToken.ThrowIfCancellationRequested(); + if (unit.HasActiveTransaction || unit.IsRollbackOnly || !routingContext.IsResolved + || !routingContext.TenantId.IsInitialized() + || routingContext.TenantId.Value == Guid.Empty + || routingContext.TenantId == TenantId.PlatformSentinel + || (routingContext.OrganizationId is { } organization + && (!organization.IsInitialized() || organization.Value == Guid.Empty))) + { + throw new InvalidOperationException("Delivery preparation requires a fresh unit and valid routing scope."); + } + + var frame = await unit.BeginTransactionAsync(TransactionMode.ReadWrite, cancellationToken); + try + { + if (!frame.IsOwner) + { + throw new InvalidOperationException("Delivery preparation must own its transaction."); + } + + await unit.SetTenantContextAsync(routingContext, cancellationToken); + var transaction = unit.Transaction + ?? throw new InvalidOperationException("Delivery preparation lost its transaction."); + var announcement = unit.GetTenantContextAnnouncement(transaction) + ?? throw new InvalidOperationException("Delivery preparation requires an announced transaction."); + var owner = new PreparedDeliveryOwner(unit, frame, transaction, announcement); + owner.RequireLive(unit, routingContext); + return owner; + } + catch + { + unit.MarkRollbackOnly(); + try + { + await frame.FailAsync(CancellationToken.None); + } + catch + { + // Preserve preparation failure, including cancellation. + } + + try + { + await frame.DisposeAsync(); + } + catch + { + // The fresh DI scope still owns connection cleanup. + } + throw; + } + } + + internal void RequireLive(IUnitOfWork unit, ITenantContext context) + { + if (_disposed || !ReferenceEquals(_unit, unit) || !_frame.IsOwner + || !unit.HasActiveTransaction || unit.IsRollbackOnly + || unit.Mode != TransactionMode.ReadWrite + || !ReferenceEquals(unit.Transaction, _transaction) + || _announcement.Generation <= 0 + || unit.GetTenantContextAnnouncement(_transaction) != _announcement + || !context.IsResolved || context.TenantId != _announcement.TenantId + || context.OrganizationId != _announcement.OrganizationId) + { + throw new InvalidOperationException("The prepared delivery owner no longer matches its live transaction and context."); + } + } + + internal IUnitOfWorkScope Consume(IUnitOfWork unit, ITenantContext context) + { + RequireLive(unit, context); + if (_consumed) + { + throw new InvalidOperationException("A prepared delivery owner can be consumed only once."); + } + + _consumed = true; + return _frame; + } + + public async ValueTask DisposeAsync() + { + if (_disposed) return; + _disposed = true; + // Consumption transfers cleanup to the common owner body. Before that, + // the original frame handle protects against later generations/units. + if (!_consumed) await _frame.DisposeAsync(); + } } private static readonly Action LogRollbackFailure = diff --git a/backend/src/LearnStack.Infrastructure.Audit/AuditConfigService.cs b/backend/src/LearnStack.Infrastructure.Audit/AuditConfigService.cs index 63d2b229..1f783d4d 100644 --- a/backend/src/LearnStack.Infrastructure.Audit/AuditConfigService.cs +++ b/backend/src/LearnStack.Infrastructure.Audit/AuditConfigService.cs @@ -1,6 +1,7 @@ using LearnStack.SharedKernel.Audit; using LearnStack.SharedKernel.Caching; using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Tenancy; using Microsoft.Extensions.Logging; using Npgsql; @@ -48,10 +49,12 @@ namespace LearnStack.Infrastructure.Audit; /// /// /// Records an override read that failed and fell back. +/// Selects bounded delivery diagnostics; never supplies override authority. public sealed class AuditConfigService( ICacheService cache, Lazy dataSource, - ILogger logger) + ILogger logger, + ITenantContextAccessor? tenantAccessor = null) : IAuditConfigService { /// The module segment of this projection's cache key. @@ -115,7 +118,8 @@ public async Task ClassifyAsync( // catalogue carries the same MUST floor, so nothing proceeds unaudited — and // rejecting every request platform-wide because a cache is unavailable is a // worse compliance outcome than losing one tenant's narrowing. - LogOverrideReadFailed(logger, tenantId.Value.Value, failure); + LogOverrideReadFailed(logger, tenantId.Value.Value, + tenantAccessor?.Current is EventTenantContext ? null : failure); return declared; } diff --git a/backend/src/LearnStack.Infrastructure.Audit/Capture/AuditChangeTrackerInterceptor.cs b/backend/src/LearnStack.Infrastructure.Audit/Capture/AuditChangeTrackerInterceptor.cs index 2fc24a0a..4b81bda6 100644 --- a/backend/src/LearnStack.Infrastructure.Audit/Capture/AuditChangeTrackerInterceptor.cs +++ b/backend/src/LearnStack.Infrastructure.Audit/Capture/AuditChangeTrackerInterceptor.cs @@ -57,7 +57,10 @@ public sealed class AuditChangeTrackerInterceptor(IAuditStateCapture capture) /// /// /// By name and with a reason each. OutboxMessage: its payload is the audited - /// event and the row is machinery. IdempotencyKey: request plumbing. + /// event and the row is machinery. InboxMessage: transport-owned completion + /// identity. EventDelivery: retained envelope and retry/terminal machinery + /// (ADR-0044 Amendment 11). Explicit handler and nested-command business intents + /// remain captured and audited. IdempotencyKey: request plumbing. /// AuditEntry / AuditConfig: the audit tables themselves — which cannot /// arise anyway, because the store writes parameterised SQL and never a /// DbContext, so there is no re-entrancy path. They are listed regardless, @@ -67,6 +70,8 @@ public sealed class AuditChangeTrackerInterceptor(IAuditStateCapture capture) private static readonly HashSet Excluded = new(StringComparer.Ordinal) { "OutboxMessage", + "InboxMessage", + "EventDelivery", "IdempotencyKey", "AuditEntry", "AuditConfig", diff --git a/backend/src/LearnStack.Infrastructure.Audit/PostgresAuditStore.cs b/backend/src/LearnStack.Infrastructure.Audit/PostgresAuditStore.cs index f3d51356..204054e9 100644 --- a/backend/src/LearnStack.Infrastructure.Audit/PostgresAuditStore.cs +++ b/backend/src/LearnStack.Infrastructure.Audit/PostgresAuditStore.cs @@ -4,6 +4,7 @@ using LearnStack.SharedKernel.Audit; using LearnStack.SharedKernel.Identifiers; using LearnStack.SharedKernel.Persistence; +using LearnStack.SharedKernel.Tenancy; using System.Diagnostics.Metrics; using Microsoft.Extensions.Logging; using Npgsql; @@ -75,6 +76,7 @@ public sealed class PostgresAuditStore : IAuditStore private readonly ILogger _logger; private readonly IAuditHealth _health; private readonly IAuditCatalog _catalog; + private readonly ITenantContextAccessor _tenantAccessor; private readonly Counter _durableDuplicates; private readonly Counter _standaloneWriteFailures; @@ -84,7 +86,8 @@ public PostgresAuditStore( ILogger logger, IMeterFactory meterFactory, IAuditHealth health, - IAuditCatalog catalog) + IAuditCatalog catalog, + ITenantContextAccessor tenantAccessor) { ArgumentNullException.ThrowIfNull(meterFactory); @@ -93,6 +96,9 @@ public PostgresAuditStore( _logger = logger; _health = health ?? throw new ArgumentNullException(nameof(health)); _catalog = catalog ?? throw new ArgumentNullException(nameof(catalog)); + // Context controls diagnostics only. Row authority still comes exclusively + // from the caller's classified draft/intent, including unresolved provisioning. + _tenantAccessor = tenantAccessor ?? throw new ArgumentNullException(nameof(tenantAccessor)); var meter = meterFactory.Create(MeterName); _durableDuplicates = meter.CreateCounter(DurableDuplicateCounterName); @@ -216,7 +222,7 @@ public async Task WriteStandaloneAsync( // (ADR-0044 § 5). // Logged AND counted, which is what the two ADRs say and what the comment // above used to claim on its own. - LogDurableDuplicate(_logger, entry.Operation, duplicate); + LogDurableDuplicate(_logger, entry.Operation, DiagnosticException(duplicate)); _durableDuplicates.Add(1, new KeyValuePair("operation", entry.Operation)); // And healthy: the row is durable, which is the opposite of the state the @@ -305,7 +311,7 @@ private static void EnsureNotPlatformScope(AuditEntryDraft entry) /// private void ReportFailure(string operation, Exception failure) { - LogStandaloneWriteFailed(_logger, operation, failure); + LogStandaloneWriteFailed(_logger, operation, DiagnosticException(failure)); _standaloneWriteFailures.Add(1, new KeyValuePair("operation", operation)); _health.ReportStandaloneWriteFailed(operation); } @@ -340,10 +346,13 @@ public async Task WriteBestEffortAsync( // carrying the platform sentinel — refused before this try, as the caller error // it is, so that swallowing it here cannot hide the one misuse the guard exists // to surface. - LogBestEffortLost(_logger, entry.Operation, failure); + LogBestEffortLost(_logger, entry.Operation, DiagnosticException(failure)); } } + private Exception? DiagnosticException(Exception failure) => + _tenantAccessor.Current is EventTenantContext ? null : failure; + /// public async Task WritePlatformScopeAsync( AuditEntryDraft entry, @@ -390,16 +399,20 @@ public async Task WriteEventTerminalAsync( ArgumentNullException.ThrowIfNull(transition); transition.EnsureMatches(entry, connection, transaction); - const string operation = "platform.outbox.dead_letter"; + var operation = (transition.Kind, transition.SubscriptionId) switch + { + ("producer", null) => "platform.outbox.dead_letter", + ("subscriber", not null) => "platform.delivery.dead_letter", + _ => throw new ArgumentException("The terminal transition kind is not supported.", nameof(transition)), + }; if (!_catalog.TryGetOffPath(operation, out var declared) || declared.Operation != operation || declared.ModuleName != "platform" || declared.OperationClass != OperationClass.Must || declared.OperationType != OperationType.SecurityEvent || declared.EntityType is not null || entry.Operation != declared.Operation || entry.ModuleName != declared.ModuleName - || entry.OperationClass != declared.OperationClass || entry.OperationType != declared.OperationType - || transition.Kind != "producer" || transition.SubscriptionId is not null) + || entry.OperationClass != declared.OperationClass || entry.OperationType != declared.OperationType) { - throw new ArgumentException("The terminal audit requires its declared producer MUST security event.", nameof(entry)); + throw new ArgumentException("The terminal audit requires its declared MUST security event.", nameof(entry)); } try diff --git a/backend/src/LearnStack.Infrastructure/Messaging/DeliveryDispatchOptions.cs b/backend/src/LearnStack.Infrastructure/Messaging/DeliveryDispatchOptions.cs new file mode 100644 index 00000000..cf158d0a --- /dev/null +++ b/backend/src/LearnStack.Infrastructure/Messaging/DeliveryDispatchOptions.cs @@ -0,0 +1,31 @@ +namespace LearnStack.Infrastructure.Messaging; + +/// Finite execution bounds for ADR-0046's independent subscriber worker. +public sealed record DeliveryDispatchOptions +{ + public TimeSpan SqlTimeout { get; init; } = TimeSpan.FromSeconds(5); + public TimeSpan ExecutionTimeout { get; init; } = TimeSpan.FromSeconds(10); + public TimeSpan LeaseDuration { get; init; } = TimeSpan.FromSeconds(30); + public TimeSpan IdlePoll { get; init; } = TimeSpan.FromMilliseconds(200); + public TimeSpan ProgressTimeout { get; init; } = TimeSpan.FromSeconds(45); + public TimeSpan SnapshotInterval { get; init; } = TimeSpan.FromSeconds(5); + public TimeSpan ShutdownTimeout { get; init; } = TimeSpan.FromSeconds(30); + /// Bounds retained execution/cleanup tasks; normal execution is awaited serially. + public int MaxOutstandingExecutions { get; init; } = 4; + + public void Validate() + { + if (SqlTimeout <= TimeSpan.Zero || SqlTimeout > TimeSpan.FromSeconds(5) + || ExecutionTimeout <= TimeSpan.Zero || ExecutionTimeout > TimeSpan.FromSeconds(10) + || LeaseDuration <= ExecutionTimeout + SqlTimeout + SqlTimeout + || LeaseDuration > TimeSpan.FromSeconds(30) + || IdlePoll <= TimeSpan.Zero || IdlePoll > TimeSpan.FromSeconds(1) + || SnapshotInterval <= TimeSpan.Zero || SnapshotInterval > TimeSpan.FromSeconds(5) + || ProgressTimeout <= LeaseDuration || ProgressTimeout > TimeSpan.FromSeconds(45) + || ShutdownTimeout <= TimeSpan.Zero || ShutdownTimeout > TimeSpan.FromSeconds(30) + || MaxOutstandingExecutions is < 1 or > 4) + { + throw new ArgumentException("Delivery execution bounds are invalid."); + } + } +} diff --git a/backend/src/LearnStack.Infrastructure/Messaging/DeliveryDispatchStore.cs b/backend/src/LearnStack.Infrastructure/Messaging/DeliveryDispatchStore.cs new file mode 100644 index 00000000..fb358d49 --- /dev/null +++ b/backend/src/LearnStack.Infrastructure/Messaging/DeliveryDispatchStore.cs @@ -0,0 +1,357 @@ +using System.Data.Common; +using LearnStack.SharedKernel.Audit; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Time; +using Microsoft.Extensions.DependencyInjection; +using Npgsql; +using NpgsqlTypes; + +namespace LearnStack.Infrastructure.Messaging; + +/// Short, database-time fenced dispatcher transitions on its confined role. +public sealed class DeliveryDispatchStore : EventTerminalTransition.MintingSource +{ + private const string Incomplete = "processed_at IS NULL AND dead_lettered_at IS NULL"; + private const string Normal = Incomplete + " AND terminal_reason IS NULL AND attempts < 6" + + " AND available_after <= clock_timestamp() AND locked_by IS NULL"; + private const string Recovery = Incomplete + " AND terminal_reason IS NULL" + + " AND (locked_by IS NOT NULL OR attempts >= 6)" + + " AND ((locked_by IS NOT NULL AND locked_until <= clock_timestamp())" + + " OR (locked_by IS NULL AND attempts >= 6))"; + private const string Terminal = Incomplete + " AND terminal_reason IS NOT NULL" + + " AND available_after <= clock_timestamp()" + + " AND (locked_until IS NULL OR locked_until <= clock_timestamp())"; + private const string Held = "EXISTS (SELECT 1 FROM unnest(@held_types, @held_subscriptions)" + + " AS held(type, subscription_id) WHERE held.type = event_deliveries.type" + + " AND held.subscription_id = event_deliveries.subscription_id)"; + private const string Admitted = " AND NOT (" + Held + ") AND NOT (partition_key = ANY(@blocked))"; + private const string ClaimColumns = "tenant_id, subscription_id, event_id, organization_id, locked_by," + + " attempts, locked_until, type, topic, partition_key, occurred_at"; + private const string Identity = "tenant_id = @tenant AND subscription_id = @subscription AND event_id = @event"; + private const string Owned = Identity + " AND locked_by = @token AND " + Incomplete; + private const string Fence = Owned + " AND locked_until > clock_timestamp()"; + + private readonly Lazy _source; + private readonly OutboxEventRegistry _registry; + private readonly DeliveryDispatchOptions _options; + private readonly IGuidFactory _ids; + private readonly IClock _clock; + private readonly IServiceScopeFactory _scopes; + + public DeliveryDispatchStore(Lazy source, OutboxEventRegistry registry, + DeliveryDispatchOptions options, IGuidFactory ids, IClock clock, IServiceScopeFactory scopes) + { + ArgumentNullException.ThrowIfNull(source); + ArgumentNullException.ThrowIfNull(registry); + ArgumentNullException.ThrowIfNull(options); + ArgumentNullException.ThrowIfNull(ids); + ArgumentNullException.ThrowIfNull(clock); + ArgumentNullException.ThrowIfNull(scopes); + options.Validate(); + _source = source; + _registry = registry; + _options = options; + _ids = ids; + _clock = clock; + _scopes = scopes; + } + + public Task TryClaimAsync(IReadOnlyCollection blockedPartitions, + CancellationToken cancellationToken = default) => + AcquireAsync(DeliveryClaimKind.Normal, Normal, blockedPartitions, cancellationToken); + + public Task TryAcquireRecoveryAsync(IReadOnlyCollection blockedPartitions, + CancellationToken cancellationToken = default) => + AcquireAsync(DeliveryClaimKind.Recovery, Recovery, blockedPartitions, cancellationToken); + + public Task TryClaimTerminalAsync(IReadOnlyCollection blockedPartitions, + CancellationToken cancellationToken = default) => + AcquireAsync(DeliveryClaimKind.Terminal, Terminal, blockedPartitions, cancellationToken); + + private async Task AcquireAsync(DeliveryClaimKind kind, string eligibility, + IReadOnlyCollection blocked, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(blocked); + using var deadline = Deadline(cancellationToken); + await using var connection = await _source.Value.OpenConnectionAsync(deadline.Token).ConfigureAwait(false); + await using var transaction = await connection.BeginTransactionAsync(deadline.Token).ConfigureAwait(false); + await using var command = new NpgsqlCommand($""" + WITH candidate AS ( + SELECT tenant_id, subscription_id, event_id FROM event_deliveries WHERE {eligibility}{Admitted} + ORDER BY available_after, occurred_at, event_id, tenant_id, subscription_id + FOR UPDATE SKIP LOCKED LIMIT 1 + ) + UPDATE event_deliveries SET locked_by = @token, + locked_until = clock_timestamp() + @lease, + attempts = attempts + @increment + WHERE (tenant_id, subscription_id, event_id) = + (SELECT tenant_id, subscription_id, event_id FROM candidate) + RETURNING {ClaimColumns} + """, connection, transaction); + Configure(command); + AddAdmission(command, blocked); + command.Parameters.AddWithValue("token", _ids.NewUuidV4().ToString("D")); + command.Parameters.AddWithValue("lease", _options.LeaseDuration); + command.Parameters.AddWithValue("increment", kind == DeliveryClaimKind.Normal ? 1 : 0); + DeliveryClaim? result; + await using (var reader = await command.ExecuteReaderAsync(deadline.Token).ConfigureAwait(false)) + { + result = await reader.ReadAsync(deadline.Token).ConfigureAwait(false) ? ReadClaim(reader, kind) : null; + } + + await transaction.CommitAsync(deadline.Token).ConfigureAwait(false); + return result; + } + + public Task FailAsync(DeliveryClaim claim, DeliveryFailureReason reason, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(claim); + if (claim.Kind == DeliveryClaimKind.Terminal) + throw new ArgumentException("Terminal requests cannot spend an execution retry.", nameof(claim)); + var descriptor = reason switch + { + DeliveryFailureReason.ExecutionFailed => "execute:failed:handler", + DeliveryFailureReason.ExecutionRefused => "execute:refused:catalogue", + DeliveryFailureReason.ExecutionTimeout => "execute:timeout:deadline", + DeliveryFailureReason.UnknownOutcome => claim.Kind == DeliveryClaimKind.Recovery + ? "recovery:unknown:expired" : "execute:unknown:commit", + _ => throw new ArgumentOutOfRangeException(nameof(reason)), + }; + return UpdateAsync(claim, """ + terminal_reason = CASE WHEN attempts >= 6 THEN 'retry_exhausted' ELSE NULL END, + available_after = clock_timestamp() + CASE attempts + WHEN 1 THEN interval '1 second' WHEN 2 THEN interval '5 seconds' + WHEN 3 THEN interval '30 seconds' WHEN 4 THEN interval '5 minutes' + WHEN 5 THEN interval '1 hour' ELSE interval '0 seconds' END, + last_error = @descriptor, locked_by = NULL, locked_until = NULL + """, " AND terminal_reason IS NULL", descriptor, cancellationToken); + } + + public Task RequestTerminalAsync(DeliveryClaim claim, DeliveryTerminalReason reason, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(claim); + var code = TerminalCode(reason); + return UpdateAsync(claim, "terminal_reason = @reason, available_after = clock_timestamp()," + + " last_error = @descriptor, locked_by = NULL, locked_until = NULL", + " AND terminal_reason IS NULL", "terminal:" + code + ":closed", cancellationToken, code); + } + + private async Task UpdateAsync(DeliveryClaim claim, string assignments, string extra, + string? descriptor, CancellationToken cancellationToken, string? reason = null) + { + using var deadline = Deadline(cancellationToken); + await using var connection = await _source.Value.OpenConnectionAsync(deadline.Token).ConfigureAwait(false); + await using var transaction = await connection.BeginTransactionAsync(deadline.Token).ConfigureAwait(false); + if (!await LockOwnedAsync(connection, transaction, claim, extra, deadline.Token).ConfigureAwait(false)) return false; + await using var command = new NpgsqlCommand( + $"UPDATE event_deliveries SET {assignments} WHERE {Fence}{extra}", connection, transaction); + Configure(command); + AddFence(command, claim); + if (descriptor is not null) command.Parameters.AddWithValue("descriptor", descriptor); + if (reason is not null) command.Parameters.AddWithValue("reason", reason); + var updated = await command.ExecuteNonQueryAsync(deadline.Token).ConfigureAwait(false) == 1; + await transaction.CommitAsync(deadline.Token).ConfigureAwait(false); + return updated; + } + + private async Task LockOwnedAsync(NpgsqlConnection connection, NpgsqlTransaction transaction, + DeliveryClaim claim, string extra, CancellationToken cancellationToken) + { + // A lock wait on an unchanged tuple does not re-evaluate an earlier clock + // predicate. Acquire without time first; the following UPDATE reads fresh time. + await using var command = new NpgsqlCommand( + $"SELECT TRUE FROM event_deliveries WHERE {Owned}{extra} FOR UPDATE", connection, transaction); + Configure(command); + AddFence(command, claim); + return await command.ExecuteScalarAsync(cancellationToken).ConfigureAwait(false) is true; + } + + /// Fences, mints the actual returned-row witness, inserts MUST audit, then commits both. + public async Task TryTerminalizeAsync(DeliveryClaim claim, string validatedAttemptTraceparent, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(claim); + if (claim.Kind != DeliveryClaimKind.Terminal) + throw new ArgumentException("Terminalization requires its own acquisition.", nameof(claim)); + using var deadline = Deadline(cancellationToken); + await using var connection = await _source.Value.OpenConnectionAsync(deadline.Token).ConfigureAwait(false); + await using var transaction = await connection.BeginTransactionAsync(deadline.Token).ConfigureAwait(false); + if (!await LockOwnedAsync(connection, transaction, claim, " AND terminal_reason IS NOT NULL", + deadline.Token).ConfigureAwait(false)) return false; + await using var command = new NpgsqlCommand($""" + UPDATE event_deliveries SET dead_lettered_at = clock_timestamp(), + locked_by = NULL, locked_until = NULL + WHERE {Fence} AND terminal_reason IS NOT NULL + RETURNING event_id, tenant_id, organization_id, terminal_reason, attempts, + old.locked_by AS acquisition_token, subscription_id + """, connection, transaction); + Configure(command); + AddFence(command, claim); + EventTerminalTransition witness; + await using (var reader = await command.ExecuteReaderAsync(deadline.Token).ConfigureAwait(false)) + { + if (!await reader.ReadAsync(deadline.Token).ConfigureAwait(false)) return false; + witness = FromSubscriberUpdate(connection, transaction, reader, validatedAttemptTraceparent); + } + + await using var scope = _scopes.CreateAsyncScope(); + var audit = scope.ServiceProvider.GetRequiredService(); + var draft = new AuditEntryDraft + { + Id = AuditEntryId.From(_ids.NewUuidV7()), + TenantId = TenantId.From(witness.TenantId), + OrganizationId = witness.OrganizationId is { } organization ? OrganizationId.From(organization) : null, + ActorUserId = UserId.SystemActor, + ActorEmail = null, + ModuleName = "platform", + Operation = "platform.delivery.dead_letter", + OperationType = OperationType.SecurityEvent, + OperationClass = OperationClass.Must, + EntityType = null, + EntityId = null, + Outcome = AuditOutcome.Success, + ErrorKey = null, + Reason = null, + BeforeState = null, + AfterState = null, + Changes = null, + CorrelationId = witness.CorrelationId, + IpAddress = null, + UserAgent = null, + Timestamp = _clock.UtcNow, + Metadata = witness.Metadata, + }; + await audit.WriteEventTerminalAsync(draft, witness, connection, transaction, deadline.Token).ConfigureAwait(false); + try + { + await transaction.CommitAsync(deadline.Token).ConfigureAwait(false); + return true; + } + catch (Exception exception) when (exception is NpgsqlException or OperationCanceledException or IOException) + { + // COMMIT may have reached PostgreSQL. Reconcile on a fresh bounded connection; + // a durable terminal row already includes its audit and must never be inserted again. + try + { + if (await IsDurablyTerminalAsync(claim).ConfigureAwait(false)) return true; + } + catch (Exception reconciliationFailure) + { + throw new AggregateException("The delivery terminal commit outcome remains unknown.", + exception, reconciliationFailure); + } + throw; + } + } + + private async Task IsDurablyTerminalAsync(DeliveryClaim claim) + { + using var deadline = Deadline(CancellationToken.None); + await using var connection = await _source.Value.OpenConnectionAsync(deadline.Token).ConfigureAwait(false); + await using var command = new NpgsqlCommand( + $"SELECT EXISTS (SELECT 1 FROM event_deliveries WHERE {Identity} AND dead_lettered_at IS NOT NULL)", connection); + Configure(command); + AddIdentity(command, claim.TenantId, claim.SubscriptionId, claim.EventId); + return (bool)(await command.ExecuteScalarAsync(deadline.Token).ConfigureAwait(false))!; + } + + public async Task ReadSnapshotAsync(IReadOnlyCollection blockedPartitions, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(blockedPartitions); + using var deadline = Deadline(cancellationToken); + await using var connection = await _source.Value.OpenConnectionAsync(deadline.Token).ConfigureAwait(false); + await using var command = new NpgsqlCommand($""" + SELECT count(*), count(*) FILTER (WHERE {Held}), + COALESCE(GREATEST(0, extract(epoch FROM clock_timestamp() - min(occurred_at) + FILTER (WHERE {Normal}{Admitted}))), 0)::double precision + FROM event_deliveries WHERE {Incomplete} + """, connection); + Configure(command); + AddAdmission(command, blockedPartitions); + await using var reader = await command.ExecuteReaderAsync(deadline.Token).ConfigureAwait(false); + await reader.ReadAsync(deadline.Token).ConfigureAwait(false); + return new(reader.GetInt64(0), reader.GetInt64(1), reader.GetDouble(2)); + } + + /// Resets exactly one completed terminal Active ordinary subscription; inbox identity is never reset. + public async Task ReplayAsync(Guid tenant, string subscription, Guid eventId, + CancellationToken cancellationToken = default) + { + if (tenant == Guid.Empty || tenant == TenantId.PlatformSentinel.Value || eventId == Guid.Empty + || !OutboxEventRegistry.ValidSubscriptionId(subscription)) return false; + using var deadline = Deadline(cancellationToken); + await using var connection = await _source.Value.OpenConnectionAsync(deadline.Token).ConfigureAwait(false); + await using var transaction = await connection.BeginTransactionAsync(deadline.Token).ConfigureAwait(false); + const string replayable = " AND processed_at IS NULL AND dead_lettered_at IS NOT NULL" + + " AND terminal_reason IS NOT NULL AND locked_by IS NULL AND locked_until IS NULL"; + await using var rowLock = new NpgsqlCommand( + $"SELECT type FROM event_deliveries WHERE {Identity}{replayable} FOR UPDATE", connection, transaction); + Configure(rowLock); + AddIdentity(rowLock, tenant, subscription, eventId); + if (await rowLock.ExecuteScalarAsync(deadline.Token).ConfigureAwait(false) is not string type + || !_registry.IsActiveOrdinarySubscription(type, subscription)) return false; + await using var reset = new NpgsqlCommand($""" + UPDATE event_deliveries SET terminal_reason = NULL, dead_lettered_at = NULL, + attempts = 0, last_error = NULL, locked_by = NULL, locked_until = NULL, + available_after = clock_timestamp(), replay_count = replay_count + 1, + replayed_at = clock_timestamp() + WHERE {Identity}{replayable} AND type = @type + """, connection, transaction); + Configure(reset); + AddIdentity(reset, tenant, subscription, eventId); + reset.Parameters.AddWithValue("type", type); + var updated = await reset.ExecuteNonQueryAsync(deadline.Token).ConfigureAwait(false) == 1; + await transaction.CommitAsync(deadline.Token).ConfigureAwait(false); + return updated; + } + + private static DeliveryClaim ReadClaim(DbDataReader row, DeliveryClaimKind kind) => + new(row.GetGuid(0), row.GetString(1), row.GetGuid(2), row.IsDBNull(3) ? null : row.GetGuid(3), + row.GetString(4), row.GetInt32(5), row.GetFieldValue(6), kind, + row.GetString(7), row.GetString(8), row.GetString(9), row.GetFieldValue(10)); + + private CancellationTokenSource Deadline(CancellationToken cancellationToken) + { + var source = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + source.CancelAfter(_options.SqlTimeout); + return source; + } + + private void Configure(NpgsqlCommand command) => + command.CommandTimeout = Math.Max(1, (int)Math.Ceiling(_options.SqlTimeout.TotalSeconds)); + + private void AddAdmission(NpgsqlCommand command, IReadOnlyCollection blocked) + { + command.Parameters.AddWithValue("held_types", NpgsqlDbType.Array | NpgsqlDbType.Text, + _registry.HeldDeliverySubscriptions.Select(pair => pair.Type).ToArray()); + command.Parameters.AddWithValue("held_subscriptions", NpgsqlDbType.Array | NpgsqlDbType.Text, + _registry.HeldDeliverySubscriptions.Select(pair => pair.SubscriptionId).ToArray()); + command.Parameters.AddWithValue("blocked", NpgsqlDbType.Array | NpgsqlDbType.Text, blocked.ToArray()); + } + + private static void AddFence(NpgsqlCommand command, DeliveryClaim claim) + { + AddIdentity(command, claim.TenantId, claim.SubscriptionId, claim.EventId); + command.Parameters.AddWithValue("token", claim.Token); + } + + private static void AddIdentity(NpgsqlCommand command, Guid tenant, string subscription, Guid eventId) + { + command.Parameters.AddWithValue("tenant", tenant); + command.Parameters.AddWithValue("subscription", subscription); + command.Parameters.AddWithValue("event", eventId); + } + + private static string TerminalCode(DeliveryTerminalReason reason) => reason switch + { + DeliveryTerminalReason.UnknownType => "unknown_type", + DeliveryTerminalReason.RetiredType => "retired_type", + DeliveryTerminalReason.InvalidEnvelope => "invalid_envelope", + DeliveryTerminalReason.InvalidSubscription => "invalid_subscription", + DeliveryTerminalReason.RetryExhausted => "retry_exhausted", + _ => throw new ArgumentOutOfRangeException(nameof(reason)), + }; +} diff --git a/backend/src/LearnStack.Infrastructure/Messaging/DeliveryExecutionControl.cs b/backend/src/LearnStack.Infrastructure/Messaging/DeliveryExecutionControl.cs new file mode 100644 index 00000000..44ccc24e --- /dev/null +++ b/backend/src/LearnStack.Infrastructure/Messaging/DeliveryExecutionControl.cs @@ -0,0 +1,44 @@ +using System.Diagnostics.CodeAnalysis; +using LearnStack.Application.Messaging; + +namespace LearnStack.Infrastructure.Messaging; + +/// One execution's sticky admission gate and task-owned cancellation lifetime. +[SuppressMessage("Design", "CA1001:Types that own disposable fields should be disposable", + Justification = "Only the executor may dispose cancellation after the actual callback and scope cleanup finish.")] +public sealed class DeliveryExecutionControl +{ + private readonly object _sync = new(); + private readonly CancellationTokenSource _cancellation = new(); + private Task? _callbacks; + private bool _finishing; + + public DeliveryCommitGate Gate { get; } = new(); + internal CancellationToken Token => _cancellation.Token; + + public void RequestCancellation() + { + // The decision cannot wait behind a callback or operate on a live unit. + Gate.Close(); + lock (_sync) + { + if (_finishing || _callbacks is not null) return; + _callbacks = _cancellation.CancelAsync(); + } + } + + internal async Task FinishCallbacksAsync() + { + Task? callbacks; + lock (_sync) + { + _finishing = true; + callbacks = _callbacks; + } + if (callbacks is null) return true; + try { await callbacks.ConfigureAwait(false); return true; } + catch (Exception) { return false; } + } + + internal void DisposeCancellation() => _cancellation.Dispose(); +} diff --git a/backend/src/LearnStack.Infrastructure/Messaging/DeliveryExecutionModels.cs b/backend/src/LearnStack.Infrastructure/Messaging/DeliveryExecutionModels.cs new file mode 100644 index 00000000..c4ec3c0b --- /dev/null +++ b/backend/src/LearnStack.Infrastructure/Messaging/DeliveryExecutionModels.cs @@ -0,0 +1,25 @@ +namespace LearnStack.Infrastructure.Messaging; + +public enum DeliveryClaimKind { Normal, Recovery, Terminal } + +/// A scheduler claim carries routing and lease metadata, never the causal envelope or payload. +public sealed record DeliveryClaim( + Guid TenantId, string SubscriptionId, Guid EventId, Guid? OrganizationId, + string Token, int Attempts, DateTimeOffset LockedUntil, DeliveryClaimKind Kind, + string Type, string Topic, string PartitionKey, DateTimeOffset OccurredAt); + +public enum DeliveryExecutionOutcome +{ + Completed, Duplicate, LostClaim, InvalidEnvelope, InvalidSubscription, + Refused, Failed, Canceled, Indeterminate, +} + +/// Bounded execution evidence. Acknowledged commit remains definitive despite cleanup failure. +public sealed record DeliveryExecutionResult( + DeliveryExecutionOutcome Outcome, bool CommitAcknowledged, bool CleanupFailed); + +public enum DeliveryFailureReason { ExecutionFailed, ExecutionTimeout, UnknownOutcome, ExecutionRefused } + +public enum DeliveryTerminalReason { UnknownType, RetiredType, InvalidEnvelope, InvalidSubscription, RetryExhausted } + +public sealed record DeliveryQueueSnapshot(long PendingCount, long DeferredCount, double OldestEligibleAgeSeconds); diff --git a/backend/src/LearnStack.Infrastructure/Messaging/DeliveryExecutor.cs b/backend/src/LearnStack.Infrastructure/Messaging/DeliveryExecutor.cs new file mode 100644 index 00000000..503812a9 --- /dev/null +++ b/backend/src/LearnStack.Infrastructure/Messaging/DeliveryExecutor.cs @@ -0,0 +1,285 @@ +using System.Diagnostics; +using System.Reflection; +using System.Runtime.ExceptionServices; +using LearnStack.Application.Audit; +using LearnStack.Application.Messaging; +using LearnStack.Application.Persistence; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Messaging; +using LearnStack.SharedKernel.Persistence; +using LearnStack.SharedKernel.Results; +using LearnStack.SharedKernel.Tenancy; +using Microsoft.Extensions.DependencyInjection; + +namespace LearnStack.Infrastructure.Messaging; + +/// Owns one application scope through actual delivery execution and cleanup. +public sealed class DeliveryExecutor( + IServiceScopeFactory scopeFactory, + ITenantContextAccessor tenantAccessor, + IPartitionSerializer partitions, + OutboxEventRegistry registry, + IntegrationEventHandlerRegistry handlers) +{ + private static readonly ActivitySource Activities = new(InProcessEventBus.ActivitySourceName); + + /// + /// The caller may stop waiting, close the gate and cancel the callback, but + /// must retain this task until completion: it still owns the live scope. + /// + public Task ExecuteAsync( + DeliveryClaim claim, DeliveryCommitGate gate, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(claim); + ArgumentNullException.ThrowIfNull(gate); + if (!gate.TryStartExecution()) + throw new InvalidOperationException("A delivery commit gate belongs to exactly one execution."); + return ExecuteCoreAsync(claim, gate, null, cancellationToken); + } + + /// The production entry retains the actual application scope through cancellation callbacks. + public async Task ExecuteControlledAsync( + DeliveryClaim claim, DeliveryExecutionControl control) + { + ArgumentNullException.ThrowIfNull(claim); + ArgumentNullException.ThrowIfNull(control); + // Reuse rejection must not finish or dispose a different live invocation. + if (!control.Gate.TryStartExecution()) + throw new InvalidOperationException("A delivery commit gate belongs to exactly one execution."); + try + { + return await ExecuteCoreAsync(claim, control.Gate, control, control.Token).ConfigureAwait(false); + } + finally + { + await control.FinishCallbacksAsync().ConfigureAwait(false); + control.DisposeCancellation(); + } + } + + private async Task ExecuteCoreAsync( + DeliveryClaim claim, DeliveryCommitGate gate, DeliveryExecutionControl? control, + CancellationToken cancellationToken) + { + if (!ValidClaim(claim)) + return new(DeliveryExecutionOutcome.InvalidEnvelope, false, false); + + // The registration touches only the atomic gate, never a live connection. + using var registration = cancellationToken.Register(static state => ((DeliveryCommitGate)state!).Close(), gate); + DeliveryExecutionResult result = new(DeliveryExecutionOutcome.Failed, false, false); + // Acceptance must not queue behind a subscriber's live callback or cleanup. + // Validated event partitions are canonical UUIDs; this internal lane cannot + // collide with the bus's raw acceptance lane. Consumer ordering stays local. + await partitions.RunSequentiallyFor("delivery:" + claim.PartitionKey, async () => + { + result = await ExecuteOwnedAsync(claim, gate, control, cancellationToken).ConfigureAwait(false); + }).ConfigureAwait(false); + return result; + } + + private async Task ExecuteOwnedAsync( + DeliveryClaim claim, DeliveryCommitGate gate, DeliveryExecutionControl? control, CancellationToken cancellationToken) + { + var outcome = DeliveryExecutionOutcome.Failed; + var cleanupFailed = false; + var previous = tenantAccessor.Current; + AsyncServiceScope? scope = null; + TransactionExecutionCoordinator.PreparedDeliveryOwner? prepared = null; + IUnitOfWork? unit = null; + Activity? activity = null; + try + { + // Routing validation/read owns no inherited caller identity. Module + // services see only the fully validated event context installed below. + tenantAccessor.Current = null; + cancellationToken.ThrowIfCancellationRequested(); + // Before retained-envelope validation resolve only the bare unit. + // The routing context is exclusively a SQL setter argument; it is + // never installed in the accessor or captured by a module service. + scope = scopeFactory.CreateAsyncScope(); + unit = scope.Value.ServiceProvider.GetRequiredService(); + prepared = await TransactionExecutionCoordinator.PrepareDeliveryOwnerAsync( + unit, new DeliveryRoutingContext(claim), cancellationToken).ConfigureAwait(false); + var completion = new PostgresDeliveryCompletion(unit, gate, claim); + var stored = await completion.ReadEnvelopeAsync(cancellationToken).ConfigureAwait(false); + if (stored is null) + { + outcome = DeliveryExecutionOutcome.LostClaim; + } + else if (!MatchesClaim(stored, claim)) + { + outcome = DeliveryExecutionOutcome.InvalidEnvelope; + } + else + { + var resolution = registry.Resolve(stored.Type); + var envelope = StoredEventEnvelopeReader.Read(stored, resolution); + var subscription = SelectSubscription(resolution, claim.SubscriptionId); + if (envelope is null) + { + outcome = DeliveryExecutionOutcome.InvalidEnvelope; + } + else if (subscription is null) + { + outcome = DeliveryExecutionOutcome.InvalidSubscription; + } + else + { + // Install in this async method and retain through owned + // disposal. AsyncLocal writes in an async helper would not + // propagate back to this scope's subsequent resolutions. + tenantAccessor.Current = EventTenantContext.FromEnvelope(envelope, subscription.ModuleName); + activity = BeginConsumerActivity(envelope); + var services = scope.Value.ServiceProvider; + var inbox = services.GetRequiredService(); + var claimed = await inbox.TryClaimAsync( + TenantId.From(claim.TenantId), claim.SubscriptionId, claim.EventId, + cancellationToken).ConfigureAwait(false); + var transaction = services.GetRequiredService(); + outcome = claimed ? DeliveryExecutionOutcome.Completed : DeliveryExecutionOutcome.Duplicate; + if (claimed) + { + var audit = services.GetRequiredService(); + var response = await audit.ExecuteAsync(subscription.HandlerType, null, + () => transaction.ExecutePreparedDeliveryAsync(prepared, completion, claim.EventId, + () => InvokeAsync(services, subscription, envelope.Event, cancellationToken), + cancellationToken), cancellationToken).ConfigureAwait(false); + if (response.IsFailure) outcome = DeliveryExecutionOutcome.Refused; + } + else + { + // A durable inbox duplicate bypasses classification and + // construction, but still completes this claim under its fence. + await transaction.ExecutePreparedDeliveryAsync(prepared, completion, claim.EventId, + () => Task.FromResult(Result.Ok(None.Value)), cancellationToken).ConfigureAwait(false); + } + } + } + } + catch (Exception exception) + { + if (gate.CommitAcknowledged) + { + cleanupFailed = true; + } + else + { + outcome = gate.CommitStarted && unit?.IsRollbackOnly != true + ? DeliveryExecutionOutcome.Indeterminate + : exception is OperationCanceledException && (cancellationToken.IsCancellationRequested || gate.IsClosed) + ? DeliveryExecutionOutcome.Canceled : DeliveryExecutionOutcome.Failed; + } + // No exception messages, raw types, stacks or payloads cross this boundary. + } + finally + { + if (control is not null && !await control.FinishCallbacksAsync().ConfigureAwait(false)) + cleanupFailed = true; + try + { + if (prepared is not null) await prepared.DisposeAsync().ConfigureAwait(false); + } + catch + { + cleanupFailed = true; + } + + try + { + if (scope is { } ownedScope) await ownedScope.DisposeAsync().ConfigureAwait(false); + } + catch + { + cleanupFailed = true; + } + finally + { + try + { + activity?.Dispose(); + } + catch + { + cleanupFailed = true; + } + finally + { + tenantAccessor.Current = previous; + } + } + } + + return new(outcome, gate.CommitAcknowledged, cleanupFailed); + } + + private IntegrationEventSubscription? SelectSubscription(OutboxEventResolution resolution, string id) + { + var selected = resolution.Subscriptions.SingleOrDefault(subscription => subscription.Id == id); + if (selected is not + { + Readiness: OutboxSubscriptionReadiness.Active, + ExecutionScope: OutboxSubscriptionExecutionScope.OrdinaryTenant, HandlerType: not null + } + || resolution.EventType is null) + return null; + // A Pending sibling holds producer acquisition, not an already accepted + // Active delivery. Exact manifest and handler identity remain mandatory. + return handlers.For(resolution.EventType).SingleOrDefault( + subscription => subscription.HandlerType == selected.HandlerType); + } + + private static async Task> InvokeAsync( + IServiceProvider services, IntegrationEventSubscription subscription, + IIntegrationEvent fact, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + var handler = services.GetRequiredService(subscription.HandlerType); + Task task; + try + { + task = subscription.Handle.Invoke(handler, [fact, cancellationToken]) as Task + ?? throw new InvalidOperationException("An integration handler must return its execution task."); + } + catch (TargetInvocationException exception) when (exception.InnerException is { } inner) + { + ExceptionDispatchInfo.Capture(inner).Throw(); + throw; + } + await task.ConfigureAwait(false); + return Result.Ok(None.Value); + } + + private static Activity BeginConsumerActivity(IntegrationEventEnvelope envelope) + { + _ = ActivityContext.TryParse(envelope.CorrelationId, null, isRemote: true, out var parent); + return Activities.StartActivity("delivery.execute", ActivityKind.Consumer, parent) + ?? new Activity("delivery.execute").SetIdFormat(ActivityIdFormat.W3C) + .SetParentId(parent.TraceId, parent.SpanId, parent.TraceFlags).Start(); + } + + private static bool MatchesClaim(StoredEventEnvelope stored, DeliveryClaim claim) => + stored.EventId == claim.EventId && stored.TenantId == claim.TenantId + && stored.OrganizationId == claim.OrganizationId && stored.Type == claim.Type + && stored.Topic == claim.Topic && stored.PartitionKey == claim.PartitionKey + && stored.OccurredAt.UtcTicks / 10 == claim.OccurredAt.UtcTicks / 10; + + private static bool ValidClaim(DeliveryClaim claim) => + claim.TenantId != Guid.Empty && claim.TenantId != TenantId.PlatformSentinel.Value + && claim.EventId != Guid.Empty && claim.OrganizationId != Guid.Empty + && OutboxEventRegistry.ValidSubscriptionId(claim.SubscriptionId) + && Guid.TryParseExact(claim.Token, "D", out var token) && token != Guid.Empty + && claim.Token == token.ToString("D") && !string.IsNullOrWhiteSpace(claim.PartitionKey) + && claim.Attempts is >= 1 and <= 6 && claim.Kind == DeliveryClaimKind.Normal; + + private sealed class DeliveryRoutingContext(DeliveryClaim claim) : ITenantContext + { + public bool IsResolved => true; + public TenantId TenantId { get; } = TenantId.From(claim.TenantId); + public OrganizationId? OrganizationId { get; } = claim.OrganizationId is { } id + ? LearnStack.SharedKernel.Identifiers.OrganizationId.From(id) : null; + public UserId? UserId => null; + public string? CorrelationId => null; + public string? ModuleName => null; + + } +} diff --git a/backend/src/LearnStack.Infrastructure/Messaging/InProcessEventBus.cs b/backend/src/LearnStack.Infrastructure/Messaging/InProcessEventBus.cs index aed831dc..8260b56a 100644 --- a/backend/src/LearnStack.Infrastructure/Messaging/InProcessEventBus.cs +++ b/backend/src/LearnStack.Infrastructure/Messaging/InProcessEventBus.cs @@ -1,176 +1,85 @@ -using System.Diagnostics; -using System.Reflection; -using System.Runtime.ExceptionServices; using LearnStack.SharedKernel.Messaging; using LearnStack.SharedKernel.Tenancy; using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Logging; namespace LearnStack.Infrastructure.Messaging; -/// The process-local transport. +/// The process-local durable acceptance transport. /// -/// It supplies the shared handler contract, ambient tenant restoration, -/// process-local per-key invocation serialization and subscription failure isolation. Each -/// subscription is constructed once in its own async scope. Dapr becomes the -/// adapter when ADR-0035's multi-process/replay trigger is met. +/// Publication persists every required subscription in one fresh application-role +/// transaction. Business construction and execution belong to the delivery executor. /// -public sealed partial class InProcessEventBus( +public sealed class InProcessEventBus( IServiceScopeFactory scopeFactory, ITenantContextAccessor tenantAccessor, IPartitionSerializer partitions, - IntegrationEventHandlerRegistry handlers, - ILogger logger) : IEventBus + OutboxEventRegistry registry) : IEventBus { public const string ActivitySourceName = "learnstack.messaging"; - private const string HandleMethodName = - nameof(IIntegrationEventHandler.HandleAsync); - - private static readonly ActivitySource ActivitySource = new(ActivitySourceName); - public Task PublishAsync( IntegrationEventEnvelope envelope, CancellationToken cancellationToken = default) { ArgumentNullException.ThrowIfNull(envelope); - - if (cancellationToken.IsCancellationRequested) + cancellationToken.ThrowIfCancellationRequested(); + var serialized = OutboxEventContract.Serialize(envelope.Event); + var resolution = registry.Resolve(serialized.TypeName); + if (resolution.EventType != envelope.Event.GetType() + || resolution.Disposition is not (OutboxEventDisposition.Active + or OutboxEventDisposition.IntentionallyUnsubscribed)) { - return Task.FromCanceled(cancellationToken); + throw new InvalidOperationException("Publication requires an Active or intentionally unsubscribed contract."); } - return partitions.RunSequentiallyFor( - envelope.PartitionKey, - () => DispatchAsync(envelope, cancellationToken)); + // Validate the complete delivery scope even for intentional absence. + var context = EventTenantContext.FromEnvelope(envelope); + return partitions.RunSequentiallyFor(serialized.PartitionKey, + () => AcceptAsync(envelope, serialized, resolution, context, cancellationToken)); } - private async Task DispatchAsync( + private async Task AcceptAsync( IntegrationEventEnvelope envelope, + OutboxEventContract.SerializedEvent serialized, + OutboxEventResolution resolution, + EventTenantContext context, CancellationToken cancellationToken) { + cancellationToken.ThrowIfCancellationRequested(); var previous = tenantAccessor.Current; - - // Established before even subscription lookup. Any future registry or - // resolution work that observes ITenantContext therefore sees the event, - // never the publisher that happened to invoke this transport. - tenantAccessor.Current = EventTenantContext.FromEnvelope(envelope); - + tenantAccessor.Current = context; try { - var subscriptions = handlers.For(envelope.Event.GetType()); - if (subscriptions.Count == 0) + if (resolution.Disposition == OutboxEventDisposition.IntentionallyUnsubscribed) { - ReachedNoHandler(logger, envelope.Event.GetType().Name, envelope.Event.EventId); return; } - _ = ActivityContext.TryParse( - envelope.CorrelationId, - traceState: null, - out var parentContext); - - List? failures = null; - - for (var index = 0; index < subscriptions.Count; index++) - { - cancellationToken.ThrowIfCancellationRequested(); - var subscription = subscriptions[index]; - - try - { - await DeliverAsync( - subscription, - envelope, - parentContext, - cancellationToken) - .ConfigureAwait(false); - } - catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) - { - // Publish cancellation is control flow, not a poison - // subscription. Stop immediately so later handlers do not - // start business work during shutdown. - throw; - } - catch (Exception exception) - { - HandlerFailed( - logger, - envelope.Event.GetType().Name, - envelope.Event.EventId, - subscription.HandlerType.Name, - envelope.Event.TenantId, - envelope.PartitionKey, - exception); - (failures ??= []).Add(exception); - } - } - - if (failures is { Count: 1 }) - { - ExceptionDispatchInfo.Capture(failures[0]).Throw(); - } - - if (failures is { Count: > 1 }) - { - throw new AggregateException( - $"{failures.Count} handlers failed for {envelope.Event.GetType().Name}.", - failures); - } - } - finally - { - tenantAccessor.Current = previous; - } - } - - private async Task DeliverAsync( - IntegrationEventSubscription subscription, - IntegrationEventEnvelope envelope, - ActivityContext parentContext, - CancellationToken cancellationToken) - { - var previous = tenantAccessor.Current; - tenantAccessor.Current = EventTenantContext.FromEnvelope(envelope, subscription.ModuleName); - - try - { - using var activity = ActivitySource.StartActivity( - $"{envelope.Topic} process", - ActivityKind.Consumer, - parentContext, - tags: - [ - new("messaging.system", "in-process"), - new("messaging.destination.name", envelope.Topic), - new("messaging.operation.type", "process"), - new("learnstack.module", subscription.ModuleName), - ]); - - // The activity has to cover construction, invocation AND the await: - // a span that only wraps the happy path ends Unset on every failure, - // so a trace backend shows a green consumer span next to the error - // log that describes the same delivery. + // Restore before scope creation or service resolution. No handler is + // resolved, and no unit from the publisher's scope can be joined. + var scope = scopeFactory.CreateAsyncScope(); + var failed = false; try { - await DeliverToHandlerAsync(subscription, envelope, cancellationToken) + await scope.ServiceProvider.GetRequiredService() + .AcceptAsync(envelope, serialized, resolution.Subscriptions, cancellationToken) .ConfigureAwait(false); } - catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + catch { - // Publish-token cancellation is shutdown, not a failed delivery. - // Left Unset for the reason Standards 10 leaves a client - // disconnect Unset: nothing failed that anyone should be paged - // about, and marking it Error would put shutdown noise into the - // 100%-sampled error traces. + failed = true; throw; } - catch (Exception exception) + finally { - activity?.AddException(exception); - activity?.SetStatus(ActivityStatusCode.Error, exception.GetType().Name); - throw; + try + { + await scope.DisposeAsync().ConfigureAwait(false); + } + catch when (failed) + { + // Preserve the acceptance/COMMIT failure for producer retry. + } } } finally @@ -178,89 +87,4 @@ await DeliverToHandlerAsync(subscription, envelope, cancellationToken) tenantAccessor.Current = previous; } } - - private async Task DeliverToHandlerAsync( - IntegrationEventSubscription subscription, - IntegrationEventEnvelope envelope, - CancellationToken cancellationToken) - { - await using var scope = scopeFactory.CreateAsyncScope(); - - object handler; - try - { - handler = scope.ServiceProvider.GetRequiredService(subscription.HandlerType); - } - catch (Exception exception) - { - throw new InvalidOperationException( - $"Integration-event handler {subscription.HandlerType.FullName} " - + "failed to construct.", - exception); - } - - var handle = subscription.Handle; - Task delivery; - - try - { - delivery = (Task)handle.Invoke(handler, [envelope.Event, cancellationToken])!; - } - catch (TargetInvocationException wrapped) - when (wrapped.InnerException is OperationCanceledException cancellation - && !cancellationToken.IsCancellationRequested) - { - throw new InvalidOperationException( - "An integration-event handler was cancelled by a token other than " - + "the publish token.", - cancellation); - } - catch (TargetInvocationException wrapped) when (wrapped.InnerException is not null) - { - ExceptionDispatchInfo.Capture(wrapped.InnerException).Throw(); - throw; - } - - if (delivery is null) - { - throw new InvalidOperationException( - $"{handler.GetType().FullName} returned a null Task from {HandleMethodName}."); - } - - try - { - await delivery.ConfigureAwait(false); - } - catch (OperationCanceledException exception) - when (!cancellationToken.IsCancellationRequested) - { - throw new InvalidOperationException( - "An integration-event handler was cancelled by a token other than " - + "the publish token.", - exception); - } - } - - [LoggerMessage( - EventId = 1, - Level = LogLevel.Error, - Message = "Integration event {EventType} ({IntegrationEventId}) failed in handler " - + "{HandlerType} for tenant {TenantId} on partition {PartitionKey}")] - private static partial void HandlerFailed( - ILogger logger, - string eventType, - Guid integrationEventId, - string handlerType, - Guid tenantId, - string partitionKey, - Exception exception); - - [LoggerMessage( - EventId = 2, - Level = LogLevel.Debug, - Message = "Integration event {EventType} ({IntegrationEventId}) reached no handler")] - private static partial void ReachedNoHandler( - ILogger logger, - string eventType, - Guid integrationEventId); } diff --git a/backend/src/LearnStack.Infrastructure/Messaging/OutboxEnvelopeReader.cs b/backend/src/LearnStack.Infrastructure/Messaging/OutboxEnvelopeReader.cs index 1a460595..18bf9653 100644 --- a/backend/src/LearnStack.Infrastructure/Messaging/OutboxEnvelopeReader.cs +++ b/backend/src/LearnStack.Infrastructure/Messaging/OutboxEnvelopeReader.cs @@ -1,17 +1,14 @@ -using System.Text.Json; -using LearnStack.SharedKernel.Identifiers; -using LearnStack.SharedKernel.Messaging; - namespace LearnStack.Infrastructure.Messaging; -/// Validates a stored fact using only the registry's declared CLR contract. +/// Applies producer readiness before validating its stored envelope. public sealed class OutboxEnvelopeReader(OutboxEventRegistry registry) { public OutboxEnvelopeReadResult Read(OutboxDispatchClaim claim) { ArgumentNullException.ThrowIfNull(claim); var resolution = registry.Resolve(claim.Type); - // A Pending hold precedes validation, including deliberately malformed payloads. + // Producer hold deliberately precedes payload validation. An already + // accepted Active sibling uses the readiness-independent reader directly. if (resolution.Disposition == OutboxEventDisposition.Pending) return new(resolution.Disposition, resolution.EventTypeLabel, null, null); if (resolution.Disposition == OutboxEventDisposition.Unknown) @@ -19,45 +16,14 @@ public OutboxEnvelopeReadResult Read(OutboxDispatchClaim claim) if (resolution.Disposition == OutboxEventDisposition.Retired) return new(resolution.Disposition, "retired", null, OutboxTerminalReason.RetiredType); - try - { - if (!claim.MetadataIsNull || claim.TenantId == Guid.Empty - || claim.TenantId == TenantId.PlatformSentinel.Value || claim.OrganizationId == Guid.Empty) - return Invalid(); - - using var document = JsonDocument.Parse(claim.Payload); - var root = document.RootElement; - // Read the stored routing fields before deserialization: get-only - // properties would otherwise silently regenerate missing or altered values. - if (root.ValueKind != JsonValueKind.Object - || root.GetProperty(nameof(IIntegrationEvent.Topic)).GetString() != claim.Topic - || root.GetProperty(nameof(IIntegrationEvent.PartitionKey)).GetString() != claim.PartitionKey) - return Invalid(); - - var fact = root.Deserialize(resolution.EventType!, - IntegrationEventBase.PayloadJsonOptions) as IntegrationEventBase; - if (fact is null) return Invalid(); - var serialized = OutboxEventContract.Serialize(fact); - // Npgsql writes timestamps at microsecond precision by truncating - // sub-microsecond ticks; compare that stored precision, not a tolerance. - if (serialized.EventId != claim.Id || serialized.TenantId != claim.TenantId - || serialized.Topic != claim.Topic || serialized.PartitionKey != claim.PartitionKey - || serialized.OccurredAt.UtcTicks / 10 != claim.OccurredAt.UtcTicks / 10) - return Invalid(); - - var envelope = new IntegrationEventEnvelope(fact, claim.CorrelationId, - claim.OrganizationId, claim.CausationId, - claim.ActorUserId is { } actor ? UserId.From(actor) : null); - return new(resolution.Disposition, resolution.EventTypeLabel, envelope, null); - } - catch (Exception exception) when (exception is JsonException or ArgumentException - or InvalidOperationException or NotSupportedException or KeyNotFoundException) - { - // Stored data is untrusted; neither its text nor parser diagnostics leave this boundary. - return Invalid(); - } - - static OutboxEnvelopeReadResult Invalid() => - new(OutboxEventDisposition.Unknown, "invalid", null, OutboxTerminalReason.InvalidEnvelope); + var envelope = claim.MetadataIsNull + ? StoredEventEnvelopeReader.Read(new StoredEventEnvelope( + claim.Id, claim.TenantId, claim.OrganizationId, claim.Type, claim.Topic, + claim.PartitionKey, claim.OccurredAt, claim.CorrelationId, claim.CausationId, + claim.ActorUserId, claim.Payload), resolution) + : null; + return envelope is null + ? new(OutboxEventDisposition.Unknown, "invalid", null, OutboxTerminalReason.InvalidEnvelope) + : new(resolution.Disposition, resolution.EventTypeLabel, envelope, null); } } diff --git a/backend/src/LearnStack.Infrastructure/Messaging/OutboxEventRegistry.cs b/backend/src/LearnStack.Infrastructure/Messaging/OutboxEventRegistry.cs index bad98752..afed9573 100644 --- a/backend/src/LearnStack.Infrastructure/Messaging/OutboxEventRegistry.cs +++ b/backend/src/LearnStack.Infrastructure/Messaging/OutboxEventRegistry.cs @@ -22,15 +22,24 @@ public enum OutboxSubscriptionReadiness Active, } +/// The authority required by a subscription; ordinary delivery cannot enter privileged work. +public enum OutboxSubscriptionExecutionScope +{ + OrdinaryTenant, + PrivilegedPlatform, +} + /// Construction-free metadata for one subscription. public sealed class OutboxSubscriptionManifest { private OutboxSubscriptionManifest( - string id, OutboxSubscriptionReadiness readiness, Type? handlerType) + string id, OutboxSubscriptionReadiness readiness, Type? handlerType, + OutboxSubscriptionExecutionScope executionScope = OutboxSubscriptionExecutionScope.OrdinaryTenant) { Id = id; Readiness = readiness; HandlerType = handlerType; + ExecutionScope = executionScope; } public string Id { get; } @@ -40,11 +49,15 @@ private OutboxSubscriptionManifest( /// Only an Active subscription names a concrete handler. public Type? HandlerType { get; } + public OutboxSubscriptionExecutionScope ExecutionScope { get; } + public static OutboxSubscriptionManifest Pending(string id) => new(id, OutboxSubscriptionReadiness.Pending, null); - public static OutboxSubscriptionManifest Active(string id, Type handlerType) => - new(id, OutboxSubscriptionReadiness.Active, handlerType); + public static OutboxSubscriptionManifest Active( + string id, Type handlerType, + OutboxSubscriptionExecutionScope executionScope = OutboxSubscriptionExecutionScope.OrdinaryTenant) => + new(id, OutboxSubscriptionReadiness.Active, handlerType, executionScope); } /// A concrete contract, its declared aliases and its delivery policy. @@ -131,11 +144,41 @@ private OutboxEventRegistry( _retired = retired; PendingIdentities = pendingIdentities; EventTypeLabels = eventTypeLabels; + HeldDeliverySubscriptions = Array.AsReadOnly(known + .SelectMany(pair => pair.Value.Subscriptions + .Where(subscription => subscription.Readiness == OutboxSubscriptionReadiness.Pending + || subscription.ExecutionScope == OutboxSubscriptionExecutionScope.PrivilegedPlatform) + .Select(subscription => (Type: pair.Key, SubscriptionId: subscription.Id))) + .OrderBy(pair => pair.Type, StringComparer.Ordinal) + .ThenBy(pair => pair.SubscriptionId, StringComparer.Ordinal).ToArray()); + SubscriptionIds = Array.AsReadOnly(known.Values.SelectMany(resolution => resolution.Subscriptions) + .Select(subscription => subscription.Id).Distinct(StringComparer.Ordinal) + .Order(StringComparer.Ordinal).ToArray()); } /// Exact canonical and alias identities held before acquisition. public IReadOnlyList PendingIdentities { get; } + /// Declared identities excluded from the ordinary tenant scheduler. + public IReadOnlyList PrivilegedSubscriptionIds => _known.Values + .SelectMany(static resolution => resolution.Subscriptions) + .Where(static subscription => subscription.ExecutionScope == OutboxSubscriptionExecutionScope.PrivilegedPlatform) + .Select(static subscription => subscription.Id).Distinct(StringComparer.Ordinal) + .Order(StringComparer.Ordinal).ToArray(); + + /// Exact declared pairs held by the ordinary subscriber scheduler, including aliases. + public IReadOnlyList<(string Type, string SubscriptionId)> HeldDeliverySubscriptions { get; } + + /// The finite declared subscription labels allowed in subscriber metrics. + public IReadOnlyList SubscriptionIds { get; } + + /// Readiness belongs to this exact subscription, independently of Pending siblings. + public bool IsActiveOrdinarySubscription(string storedIdentity, string subscriptionId) => + _known.TryGetValue(storedIdentity, out var resolution) + && resolution.Subscriptions.Any(subscription => subscription.Id == subscriptionId + && subscription.Readiness == OutboxSubscriptionReadiness.Active + && subscription.ExecutionScope == OutboxSubscriptionExecutionScope.OrdinaryTenant); + /// The finite labels allowed for declared event types in metrics. public IReadOnlyList EventTypeLabels { get; } @@ -200,6 +243,12 @@ public static OutboxEventRegistry Create( "Outbox subscription IDs must be valid and globally unique."); } + if (!Enum.IsDefined(subscription.ExecutionScope)) + { + throw new InvalidOperationException( + "Outbox subscription execution scopes must be declared values."); + } + if (subscription.Readiness == OutboxSubscriptionReadiness.Pending) { if (subscription.HandlerType is not null) @@ -307,7 +356,7 @@ private static bool ValidIdentity([NotNullWhen(true)] string? identity) => !string.IsNullOrWhiteSpace(identity) && string.Equals(identity, identity.Trim(), StringComparison.Ordinal); - private static bool ValidSubscriptionId(string? id) + internal static bool ValidSubscriptionId(string? id) { if (id is null) return false; diff --git a/backend/src/LearnStack.Infrastructure/Messaging/PostgresDeliveryAcceptance.cs b/backend/src/LearnStack.Infrastructure/Messaging/PostgresDeliveryAcceptance.cs new file mode 100644 index 00000000..0bc4200b --- /dev/null +++ b/backend/src/LearnStack.Infrastructure/Messaging/PostgresDeliveryAcceptance.cs @@ -0,0 +1,182 @@ +using LearnStack.SharedKernel.Messaging; +using LearnStack.SharedKernel.Persistence; +using LearnStack.SharedKernel.Tenancy; +using Npgsql; +using NpgsqlTypes; + +namespace LearnStack.Infrastructure.Messaging; + +/// Owns the fresh application transaction that durably accepts all subscriptions. +public sealed class PostgresDeliveryAcceptance( + IUnitOfWork unitOfWork, ITenantContext tenantContext, OutboxEventRegistry registry) +{ + private const string InsertSql = """ + INSERT INTO event_deliveries ( + tenant_id, subscription_id, event_id, organization_id, occurred_at, + correlation_id, causation_id, actor_user_id, type, topic, partition_key, payload) + VALUES (@tenant_id, @subscription_id, @event_id, @organization_id, @occurred_at, + @correlation_id, @causation_id, @actor_user_id, @type, @topic, @partition_key, @payload) + ON CONFLICT (tenant_id, subscription_id, event_id) DO NOTHING + RETURNING TRUE + """; + + // A separate statement gets a fresh READ COMMITTED snapshot after a concurrent + // INSERT finishes. A same-statement CTE could see the conflict but miss its row. + private const string MatchesSql = """ + SELECT organization_id IS NOT DISTINCT FROM @organization_id + AND occurred_at = @occurred_at + AND correlation_id = @correlation_id + AND causation_id IS NOT DISTINCT FROM @causation_id + AND actor_user_id IS NOT DISTINCT FROM @actor_user_id + AND topic = @topic AND partition_key = @partition_key + AND payload = @payload, + type + FROM event_deliveries + WHERE tenant_id = @tenant_id AND subscription_id = @subscription_id AND event_id = @event_id + """; + + internal async Task AcceptAsync( + IntegrationEventEnvelope envelope, + OutboxEventContract.SerializedEvent serialized, + IReadOnlyList subscriptions, + CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + if (subscriptions.Count == 0 + || subscriptions.Any(subscription => subscription.Readiness != OutboxSubscriptionReadiness.Active + || subscription.HandlerType is null || !OutboxEventRegistry.ValidSubscriptionId(subscription.Id))) + { + throw new InvalidOperationException("Acceptance requires declared Active subscriptions."); + } + + if (unitOfWork.HasActiveTransaction || unitOfWork.IsRollbackOnly) + { + throw new InvalidOperationException("Acceptance must own a fresh application transaction."); + } + + var frame = await unitOfWork.BeginTransactionAsync(cancellationToken).ConfigureAwait(false); + var failed = false; + try + { + if (!frame.IsOwner) + { + throw new InvalidOperationException("Acceptance cannot join a publisher transaction."); + } + + await unitOfWork.SetTenantContextAsync(tenantContext, cancellationToken).ConfigureAwait(false); + var announcement = RequireAnnouncement(serialized, envelope); + if (unitOfWork.Transaction is not NpgsqlTransaction transaction + || unitOfWork.Connection is not NpgsqlConnection connection + || !ReferenceEquals(transaction.Connection, connection)) + { + throw new InvalidOperationException("Acceptance requires the ambient Npgsql transaction."); + } + + foreach (var subscription in subscriptions) + { + RequireSameTransaction(announcement, transaction, serialized, envelope); + await using var insert = CreateCommand(InsertSql, connection, transaction, subscription.Id, serialized, envelope); + if (await insert.ExecuteScalarAsync(cancellationToken).ConfigureAwait(false) is true) + { + continue; + } + + RequireSameTransaction(announcement, transaction, serialized, envelope); + await using var matches = CreateCommand(MatchesSql, connection, transaction, subscription.Id, serialized, envelope); + await using var retained = await matches.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false); + // Canonical identity and explicitly compatible aliases name one + // contract. Every other immutable envelope field must still match. + if (!await retained.ReadAsync(cancellationToken).ConfigureAwait(false) + || !retained.GetBoolean(0) + || registry.Resolve(retained.GetString(1)).EventType != envelope.Event.GetType()) + { + // Hidden cross-organization conflicts are refusals too. Do not + // reveal the original envelope or overwrite scheduling state. + throw new InvalidOperationException("The delivery identity already has a different or inaccessible envelope."); + } + } + + RequireSameTransaction(announcement, transaction, serialized, envelope); + await frame.CompleteAsync(cancellationToken).ConfigureAwait(false); + } + catch + { + failed = true; + unitOfWork.MarkRollbackOnly(); + try + { + await frame.FailAsync(CancellationToken.None).ConfigureAwait(false); + } + catch + { + // Cleanup cannot replace the SQL failure or unknown COMMIT outcome. + } + + throw; + } + finally + { + try + { + await frame.DisposeAsync().ConfigureAwait(false); + } + catch when (failed) + { + // The producer retries the original failure, including unknown COMMIT. + } + } + } + + private TenantContextAnnouncement RequireAnnouncement( + OutboxEventContract.SerializedEvent serialized, IntegrationEventEnvelope envelope) + { + var announcement = unitOfWork.GetTenantContextAnnouncement(unitOfWork.Transaction); + if (!unitOfWork.HasActiveTransaction || unitOfWork.IsRollbackOnly + || unitOfWork.Mode != TransactionMode.ReadWrite + || announcement is null || announcement.Generation <= 0 + || announcement.TenantId.Value != serialized.TenantId + || announcement.OrganizationId?.Value != envelope.OrganizationId + || !tenantContext.IsResolved || tenantContext.TenantId != announcement.TenantId + || tenantContext.OrganizationId != announcement.OrganizationId) + { + throw new InvalidOperationException("Acceptance requires its live, matching writable tenant announcement."); + } + + return announcement; + } + + private void RequireSameTransaction( + TenantContextAnnouncement announcement, + NpgsqlTransaction transaction, + OutboxEventContract.SerializedEvent serialized, + IntegrationEventEnvelope envelope) + { + if (RequireAnnouncement(serialized, envelope) != announcement + || !ReferenceEquals(unitOfWork.Transaction, transaction) + || !ReferenceEquals(transaction.Connection, unitOfWork.Connection)) + { + throw new InvalidOperationException("The acceptance transaction changed during publication."); + } + } + + private static NpgsqlCommand CreateCommand( + string sql, NpgsqlConnection connection, NpgsqlTransaction transaction, + string subscriptionId, OutboxEventContract.SerializedEvent serialized, + IntegrationEventEnvelope envelope) + { + var command = new NpgsqlCommand(sql, connection, transaction) { CommandTimeout = 5 }; + command.Parameters.AddWithValue("tenant_id", NpgsqlDbType.Uuid, serialized.TenantId); + command.Parameters.AddWithValue("subscription_id", NpgsqlDbType.Text, subscriptionId); + command.Parameters.AddWithValue("event_id", NpgsqlDbType.Uuid, serialized.EventId); + command.Parameters.AddWithValue("organization_id", NpgsqlDbType.Uuid, (object?)envelope.OrganizationId ?? DBNull.Value); + command.Parameters.AddWithValue("occurred_at", NpgsqlDbType.TimestampTz, serialized.OccurredAt.ToUniversalTime()); + command.Parameters.AddWithValue("correlation_id", NpgsqlDbType.Text, envelope.CorrelationId); + command.Parameters.AddWithValue("causation_id", NpgsqlDbType.Uuid, (object?)envelope.CausationId ?? DBNull.Value); + command.Parameters.AddWithValue("actor_user_id", NpgsqlDbType.Uuid, (object?)envelope.ActorUserId?.Value ?? DBNull.Value); + command.Parameters.AddWithValue("type", NpgsqlDbType.Text, serialized.TypeName); + command.Parameters.AddWithValue("topic", NpgsqlDbType.Text, serialized.Topic); + command.Parameters.AddWithValue("partition_key", NpgsqlDbType.Text, serialized.PartitionKey); + command.Parameters.AddWithValue("payload", NpgsqlDbType.Jsonb, serialized.Payload); + return command; + } +} diff --git a/backend/src/LearnStack.Infrastructure/Messaging/PostgresDeliveryCompletion.cs b/backend/src/LearnStack.Infrastructure/Messaging/PostgresDeliveryCompletion.cs new file mode 100644 index 00000000..6754b9b6 --- /dev/null +++ b/backend/src/LearnStack.Infrastructure/Messaging/PostgresDeliveryCompletion.cs @@ -0,0 +1,105 @@ +using LearnStack.Application.Messaging; +using LearnStack.SharedKernel.Persistence; +using Npgsql; +using NpgsqlTypes; + +namespace LearnStack.Infrastructure.Messaging; + +/// The single application-role delivery completion participant. +internal sealed class PostgresDeliveryCompletion : DeliveryCompletionParticipant +{ + private readonly IUnitOfWork _unit; + private readonly DeliveryClaim _claim; + private readonly TenantContextAnnouncement _announcement; + private readonly NpgsqlConnection _connection; + private readonly NpgsqlTransaction _transaction; + + private const string IdentityPredicate = """ + tenant_id = @tenant_id AND subscription_id = @subscription_id AND event_id = @event_id + AND organization_id IS NOT DISTINCT FROM @organization_id AND locked_by = @token + AND processed_at IS NULL AND dead_lettered_at IS NULL AND terminal_reason IS NULL + """; + + internal PostgresDeliveryCompletion(IUnitOfWork unit, DeliveryCommitGate gate, DeliveryClaim claim) + : base(unit, gate) + { + _unit = unit; + _claim = claim; + _announcement = RequireAnnouncement(); + _transaction = unit.Transaction as NpgsqlTransaction + ?? throw new InvalidOperationException("Delivery requires the application transaction."); + _connection = unit.Connection as NpgsqlConnection + ?? throw new InvalidOperationException("Delivery requires the application connection."); + RequireCurrent(); + } + + internal async Task ReadEnvelopeAsync(CancellationToken cancellationToken) + { + RequireCurrent(); + await using var command = CreateCommand(""" + SELECT event_id, tenant_id, organization_id, type, topic, partition_key, + occurred_at, correlation_id, causation_id, actor_user_id, payload::text + FROM event_deliveries WHERE + """ + " " + IdentityPredicate + " AND locked_until > clock_timestamp()"); + await using var reader = await command.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false); + if (!await reader.ReadAsync(cancellationToken).ConfigureAwait(false)) return null; + return new StoredEventEnvelope( + reader.GetGuid(0), reader.GetGuid(1), reader.IsDBNull(2) ? null : reader.GetGuid(2), + reader.GetString(3), reader.GetString(4), reader.GetString(5), + reader.GetFieldValue(6), reader.GetString(7), + reader.IsDBNull(8) ? null : reader.GetGuid(8), + reader.IsDBNull(9) ? null : reader.GetGuid(9), reader.GetString(10)); + } + + protected override async Task FenceAsync(CancellationToken cancellationToken) + { + RequireCurrent(); + // Acquire the row lock without evaluating time. A lease could expire + // while an unchanged row is locked; evaluate clock_timestamp only in + // the separate UPDATE after this lock has actually been acquired. + await using (var rowLock = CreateCommand( + "SELECT TRUE FROM event_deliveries WHERE " + IdentityPredicate + " FOR UPDATE")) + { + if (await rowLock.ExecuteScalarAsync(cancellationToken).ConfigureAwait(false) is not true) + throw new InvalidOperationException("The delivery completion fence refused ownership."); + } + + RequireCurrent(); + await using var complete = CreateCommand( + "UPDATE event_deliveries SET processed_at = clock_timestamp() WHERE " + + IdentityPredicate + " AND locked_until > clock_timestamp() RETURNING TRUE"); + if (await complete.ExecuteScalarAsync(cancellationToken).ConfigureAwait(false) is not true) + throw new InvalidOperationException("The delivery completion fence refused ownership."); + // Token and expiry remain inert provenance. App grants permit only processed_at. + } + + private TenantContextAnnouncement RequireAnnouncement() + { + var announcement = _unit.GetTenantContextAnnouncement(_unit.Transaction); + if (!_unit.HasActiveTransaction || _unit.IsRollbackOnly || _unit.Mode != TransactionMode.ReadWrite + || announcement is null || announcement.Generation <= 0 + || announcement.TenantId.Value != _claim.TenantId + || announcement.OrganizationId?.Value != _claim.OrganizationId) + throw new InvalidOperationException("Delivery requires its matching writable tenant announcement."); + return announcement; + } + + private void RequireCurrent() + { + if (RequireAnnouncement() != _announcement || !ReferenceEquals(_unit.Transaction, _transaction) + || !ReferenceEquals(_transaction.Connection, _connection) + || !ReferenceEquals(_unit.Connection, _connection)) + throw new InvalidOperationException("The delivery transaction changed during execution."); + } + + private NpgsqlCommand CreateCommand(string sql) + { + var command = new NpgsqlCommand(sql, _connection, _transaction) { CommandTimeout = 5 }; + command.Parameters.AddWithValue("tenant_id", NpgsqlDbType.Uuid, _claim.TenantId); + command.Parameters.AddWithValue("subscription_id", NpgsqlDbType.Text, _claim.SubscriptionId); + command.Parameters.AddWithValue("event_id", NpgsqlDbType.Uuid, _claim.EventId); + command.Parameters.AddWithValue("organization_id", NpgsqlDbType.Uuid, (object?)_claim.OrganizationId ?? DBNull.Value); + command.Parameters.AddWithValue("token", NpgsqlDbType.Text, _claim.Token); + return command; + } +} diff --git a/backend/src/LearnStack.Infrastructure/Messaging/PostgresInboxGuard.cs b/backend/src/LearnStack.Infrastructure/Messaging/PostgresInboxGuard.cs new file mode 100644 index 00000000..dbbe715c --- /dev/null +++ b/backend/src/LearnStack.Infrastructure/Messaging/PostgresInboxGuard.cs @@ -0,0 +1,91 @@ +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Messaging; +using LearnStack.SharedKernel.Persistence; +using LearnStack.SharedKernel.Tenancy; +using Npgsql; +using NpgsqlTypes; + +namespace LearnStack.Infrastructure.Messaging; + +/// Claims an inbox identity on the already announced business transaction. +public sealed class PostgresInboxGuard(IUnitOfWork unitOfWork, ITenantContext tenantContext) : IInboxGuard +{ + private const string InsertSql = """ + INSERT INTO inbox_messages (tenant_id, subscription_id, event_id) + VALUES (@tenant_id, @subscription_id, @event_id) + ON CONFLICT (tenant_id, subscription_id, event_id) DO NOTHING + RETURNING TRUE + """; + + public async Task TryClaimAsync( + TenantId tenantId, + string subscriptionId, + Guid eventId, + CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + if (!tenantId.IsInitialized() || tenantId.Value == Guid.Empty + || tenantId == TenantId.PlatformSentinel) + { + throw new ArgumentException("An inbox claim requires a real tenant.", nameof(tenantId)); + } + + if (!OutboxEventRegistry.ValidSubscriptionId(subscriptionId)) + { + throw new ArgumentException("An inbox claim requires a stable subscription identity.", nameof(subscriptionId)); + } + + if (eventId == Guid.Empty) + { + throw new ArgumentException("An inbox claim requires an event identity.", nameof(eventId)); + } + + // Validate before reading Connection, which otherwise opens it lazily. + var announcement = RequireAnnouncement(tenantId); + if (unitOfWork.Transaction is not NpgsqlTransaction transaction + || unitOfWork.Connection is not NpgsqlConnection connection + || !ReferenceEquals(transaction.Connection, connection)) + { + throw new InvalidOperationException("An inbox claim requires the ambient Npgsql transaction."); + } + + await using var command = new NpgsqlCommand(InsertSql, connection, transaction) { CommandTimeout = 5 }; + command.Parameters.AddWithValue("tenant_id", NpgsqlDbType.Uuid, tenantId.Value); + command.Parameters.AddWithValue("subscription_id", NpgsqlDbType.Text, subscriptionId); + command.Parameters.AddWithValue("event_id", NpgsqlDbType.Uuid, eventId); + if (RequireAnnouncement(tenantId) != announcement + || !ReferenceEquals(unitOfWork.Transaction, transaction) + || !ReferenceEquals(transaction.Connection, unitOfWork.Connection)) + { + throw new InvalidOperationException("The inbox transaction changed before the claim."); + } + + try + { + // PostgreSQL uniqueness waits for a concurrent uncommitted claim. + // Commit returns false; rollback allows this transaction's new claim. + return await command.ExecuteScalarAsync(cancellationToken).ConfigureAwait(false) is true; + } + catch + { + unitOfWork.MarkRollbackOnly(); + throw; + } + } + + private TenantContextAnnouncement RequireAnnouncement(TenantId tenantId) + { + var announcement = unitOfWork.GetTenantContextAnnouncement(unitOfWork.Transaction); + if (!unitOfWork.HasActiveTransaction || unitOfWork.IsRollbackOnly + || unitOfWork.Mode != TransactionMode.ReadWrite + || announcement is null || announcement.Generation <= 0 + || announcement.TenantId != tenantId + || !tenantContext.IsResolved || tenantContext.TenantId != tenantId + || tenantContext.OrganizationId != announcement.OrganizationId) + { + throw new InvalidOperationException("An inbox claim requires its live, matching writable tenant announcement."); + } + + return announcement; + } +} diff --git a/backend/src/LearnStack.Infrastructure/Messaging/StoredEventEnvelope.cs b/backend/src/LearnStack.Infrastructure/Messaging/StoredEventEnvelope.cs new file mode 100644 index 00000000..aa32d493 --- /dev/null +++ b/backend/src/LearnStack.Infrastructure/Messaging/StoredEventEnvelope.cs @@ -0,0 +1,57 @@ +using System.Text.Json; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Messaging; + +namespace LearnStack.Infrastructure.Messaging; + +/// Immutable persisted fact and causal envelope, read under tenant authority. +internal sealed record StoredEventEnvelope( + Guid EventId, Guid TenantId, Guid? OrganizationId, string Type, string Topic, + string PartitionKey, DateTimeOffset OccurredAt, string CorrelationId, + Guid? CausationId, Guid? ActorUserId, string Payload); + +/// Shared envelope validation, independent of producer subscription readiness. +internal static class StoredEventEnvelopeReader +{ + internal static IntegrationEventEnvelope? Read(StoredEventEnvelope stored, OutboxEventResolution resolution) + { + if (resolution.EventType is null) return null; + try + { + if (stored.TenantId == Guid.Empty + || stored.TenantId == TenantId.PlatformSentinel.Value || stored.OrganizationId == Guid.Empty) + return null; + + using var document = JsonDocument.Parse(stored.Payload); + var root = document.RootElement; + // Read the stored routing fields before deserialization: get-only + // properties would otherwise silently regenerate missing or altered values. + if (root.ValueKind != JsonValueKind.Object + || root.GetProperty(nameof(IIntegrationEvent.Topic)).GetString() != stored.Topic + || root.GetProperty(nameof(IIntegrationEvent.PartitionKey)).GetString() != stored.PartitionKey) + return null; + + var fact = root.Deserialize(resolution.EventType, + IntegrationEventBase.PayloadJsonOptions) as IntegrationEventBase; + if (fact is null) return null; + var serialized = OutboxEventContract.Serialize(fact); + // Npgsql writes timestamps at microsecond precision by truncating + // sub-microsecond ticks; compare that stored precision, not a tolerance. + if (serialized.EventId != stored.EventId || serialized.TenantId != stored.TenantId + || serialized.Topic != stored.Topic || serialized.PartitionKey != stored.PartitionKey + || serialized.OccurredAt.UtcTicks / 10 != stored.OccurredAt.UtcTicks / 10) + return null; + + var envelope = new IntegrationEventEnvelope(fact, stored.CorrelationId, + stored.OrganizationId, stored.CausationId, + stored.ActorUserId is { } actor ? UserId.From(actor) : null); + return envelope; + } + catch (Exception exception) when (exception is JsonException or ArgumentException + or InvalidOperationException or NotSupportedException or KeyNotFoundException) + { + // Stored data is untrusted; neither its text nor parser diagnostics leave this boundary. + return null; + } + } +} diff --git a/backend/src/LearnStack.Infrastructure/Persistence/Migrations/20261013083903_add_event_delivery_and_inbox.Designer.cs b/backend/src/LearnStack.Infrastructure/Persistence/Migrations/20261013083903_add_event_delivery_and_inbox.Designer.cs new file mode 100644 index 00000000..7226e773 --- /dev/null +++ b/backend/src/LearnStack.Infrastructure/Persistence/Migrations/20261013083903_add_event_delivery_and_inbox.Designer.cs @@ -0,0 +1,29 @@ +// +using LearnStack.Infrastructure.Persistence; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Migrations; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; +using Npgsql.EntityFrameworkCore.PostgreSQL.Metadata; + +#nullable disable + +namespace LearnStack.Infrastructure.Persistence.Migrations +{ + [DbContext(typeof(PlatformDbContext))] + [Migration("20261013083903_add_event_delivery_and_inbox")] + partial class add_event_delivery_and_inbox + { + /// + protected override void BuildTargetModel(ModelBuilder modelBuilder) + { +#pragma warning disable 612, 618 + modelBuilder + .HasAnnotation("ProductVersion", "10.0.12") + .HasAnnotation("Relational:MaxIdentifierLength", 63); + + NpgsqlModelBuilderExtensions.UseIdentityByDefaultColumns(modelBuilder); +#pragma warning restore 612, 618 + } + } +} diff --git a/backend/src/LearnStack.Infrastructure/Persistence/Migrations/20261013083903_add_event_delivery_and_inbox.cs b/backend/src/LearnStack.Infrastructure/Persistence/Migrations/20261013083903_add_event_delivery_and_inbox.cs new file mode 100644 index 00000000..267cd651 --- /dev/null +++ b/backend/src/LearnStack.Infrastructure/Persistence/Migrations/20261013083903_add_event_delivery_and_inbox.cs @@ -0,0 +1,170 @@ +using Microsoft.EntityFrameworkCore.Migrations; + +#nullable disable + +namespace LearnStack.Infrastructure.Persistence.Migrations +{ + /// + public partial class add_event_delivery_and_inbox : Migration + { + /// + protected override void Up(MigrationBuilder migrationBuilder) + { + // Shared transport infrastructure has no EF entity model. Canonical + // DDL: Standards 05, Inbox. Tenancy owns the shared organization guard + // function and runs first in the sanctioned migration sequence. + migrationBuilder.Sql(""" + CREATE TABLE event_deliveries ( + tenant_id uuid NOT NULL, + subscription_id text NOT NULL, + event_id uuid NOT NULL, + organization_id uuid NULL, + occurred_at timestamptz NOT NULL, + correlation_id text NOT NULL, + causation_id uuid NULL, + actor_user_id uuid NULL, + type text NOT NULL, + topic text NOT NULL, + partition_key text NOT NULL, + payload jsonb NOT NULL, + processed_at timestamptz NULL, + attempts int NOT NULL DEFAULT 0, + last_error text NULL, + available_after timestamptz NOT NULL DEFAULT now(), + locked_by text NULL, + locked_until timestamptz NULL, + terminal_reason text NULL, + dead_lettered_at timestamptz NULL, + replay_count int NOT NULL DEFAULT 0, + replayed_at timestamptz NULL, + CONSTRAINT pk_event_deliveries PRIMARY KEY (tenant_id, subscription_id, event_id), + CONSTRAINT ck_event_deliveries_identity CHECK ( + tenant_id NOT IN ('00000000-0000-0000-0000-000000000000'::uuid, + '00000000-0000-7000-8000-000000000002'::uuid) + AND event_id <> '00000000-0000-0000-0000-000000000000'::uuid + AND (organization_id IS NULL OR + organization_id <> '00000000-0000-0000-0000-000000000000'::uuid) + AND (causation_id IS NULL OR causation_id <> '00000000-0000-0000-0000-000000000000'::uuid) + AND (actor_user_id IS NULL OR actor_user_id <> '00000000-0000-0000-0000-000000000000'::uuid)), + CONSTRAINT ck_event_deliveries_subscription CHECK ( + subscription_id COLLATE "C" ~ '^[a-z][a-z0-9]*\.[a-z][a-z0-9-]*\.v[1-9][0-9]*$'), + CONSTRAINT ck_event_deliveries_attempts CHECK (attempts >= 0), + CONSTRAINT ck_event_deliveries_replay CHECK ( + replay_count >= 0 AND ((replay_count = 0 AND replayed_at IS NULL) + OR (replay_count > 0 AND replayed_at IS NOT NULL))), + CONSTRAINT ck_event_deliveries_last_error_size CHECK ( + last_error IS NULL OR octet_length(last_error) <= 256), + CONSTRAINT ck_event_deliveries_lease CHECK ( + (locked_by IS NULL AND locked_until IS NULL) OR + (locked_by IS NOT NULL AND locked_until IS NOT NULL + AND locked_by COLLATE "C" ~ '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$' + AND locked_by <> '00000000-0000-0000-0000-000000000000')), + CONSTRAINT ck_event_deliveries_terminal_reason CHECK ( + terminal_reason IS NULL OR terminal_reason IN ( + 'unknown_type', 'retired_type', 'invalid_envelope', + 'invalid_subscription', 'retry_exhausted')), + CONSTRAINT ck_event_deliveries_state CHECK ( + (processed_at IS NULL OR (dead_lettered_at IS NULL AND terminal_reason IS NULL)) + AND (dead_lettered_at IS NULL OR + (processed_at IS NULL AND terminal_reason IS NOT NULL AND locked_by IS NULL))) + ); + CREATE INDEX ix_event_deliveries_tenant_id_organization_id + ON event_deliveries (tenant_id, organization_id); + CREATE INDEX ix_event_deliveries_pending + ON event_deliveries (available_after, occurred_at, event_id, tenant_id, subscription_id) + WHERE processed_at IS NULL AND dead_lettered_at IS NULL + AND terminal_reason IS NULL AND attempts < 6 AND locked_by IS NULL; + CREATE INDEX ix_event_deliveries_recovery + ON event_deliveries (available_after, occurred_at, event_id, tenant_id, subscription_id) + WHERE processed_at IS NULL AND dead_lettered_at IS NULL + AND terminal_reason IS NULL AND (locked_by IS NOT NULL OR attempts >= 6); + CREATE INDEX ix_event_deliveries_terminal + ON event_deliveries (available_after, occurred_at, event_id, tenant_id, subscription_id) + WHERE processed_at IS NULL AND dead_lettered_at IS NULL AND terminal_reason IS NOT NULL; + ALTER TABLE event_deliveries ENABLE ROW LEVEL SECURITY; + ALTER TABLE event_deliveries FORCE ROW LEVEL SECURITY; + CREATE POLICY event_deliveries_isolation ON event_deliveries + USING ( + tenant_id = NULLIF(current_setting('app.tenant_id', true), '')::uuid + AND (organization_id IS NULL + OR organization_id = NULLIF(current_setting('app.organization_id', true), '')::uuid + OR current_setting('app.scope', true) = 'tenant')) + WITH CHECK ( + tenant_id = NULLIF(current_setting('app.tenant_id', true), '')::uuid + AND ((organization_id IS NULL + AND NULLIF(current_setting('app.organization_id', true), '') IS NULL) + OR organization_id = NULLIF(current_setting('app.organization_id', true), '')::uuid)); + CREATE POLICY event_deliveries_org_write_guard ON event_deliveries + AS RESTRICTIVE FOR UPDATE + USING ((organization_id IS NULL + AND NULLIF(current_setting('app.organization_id', true), '') IS NULL) + OR organization_id = NULLIF(current_setting('app.organization_id', true), '')::uuid); + CREATE POLICY event_deliveries_org_delete_guard ON event_deliveries + AS RESTRICTIVE FOR DELETE + USING ((organization_id IS NULL + AND NULLIF(current_setting('app.organization_id', true), '') IS NULL) + OR organization_id = NULLIF(current_setting('app.organization_id', true), '')::uuid); + CREATE TRIGGER tg_event_deliveries_organization_id_immutable + BEFORE UPDATE ON event_deliveries + FOR EACH ROW EXECUTE FUNCTION fn_organization_id_immutable(); + CREATE TABLE inbox_messages ( + tenant_id uuid NOT NULL, + subscription_id text NOT NULL, + event_id uuid NOT NULL, + recorded_at timestamptz NOT NULL DEFAULT now(), + CONSTRAINT pk_inbox_messages PRIMARY KEY (tenant_id, subscription_id, event_id), + CONSTRAINT ck_inbox_messages_identity CHECK ( + tenant_id NOT IN ('00000000-0000-0000-0000-000000000000'::uuid, + '00000000-0000-7000-8000-000000000002'::uuid) + AND event_id <> '00000000-0000-0000-0000-000000000000'::uuid), + CONSTRAINT ck_inbox_messages_subscription CHECK ( + subscription_id COLLATE "C" ~ '^[a-z][a-z0-9]*\.[a-z][a-z0-9-]*\.v[1-9][0-9]*$') + ); + ALTER TABLE inbox_messages ENABLE ROW LEVEL SECURITY; + ALTER TABLE inbox_messages FORCE ROW LEVEL SECURITY; + CREATE POLICY inbox_messages_isolation ON inbox_messages + USING (tenant_id = NULLIF(current_setting('app.tenant_id', true), '')::uuid) + WITH CHECK (tenant_id = NULLIF(current_setting('app.tenant_id', true), '')::uuid); + CREATE FUNCTION fn_inbox_messages_immutable() RETURNS trigger + LANGUAGE plpgsql AS $$ + BEGIN + RAISE EXCEPTION 'inbox_messages is immutable (attempted % as %)', TG_OP, current_user + USING ERRCODE = 'insufficient_privilege'; + END; + $$; + CREATE TRIGGER tg_inbox_messages_immutable + BEFORE UPDATE OR DELETE ON inbox_messages + FOR EACH ROW EXECUTE FUNCTION fn_inbox_messages_immutable(); + CREATE TRIGGER tg_inbox_messages_no_truncate + BEFORE TRUNCATE ON inbox_messages + FOR EACH STATEMENT EXECUTE FUNCTION fn_inbox_messages_immutable(); + GRANT SELECT ON event_deliveries, inbox_messages TO learnstack_app; + GRANT INSERT (tenant_id, subscription_id, event_id, organization_id, occurred_at, + correlation_id, causation_id, actor_user_id, type, topic, partition_key, payload) + ON event_deliveries TO learnstack_app; + GRANT UPDATE (processed_at) ON event_deliveries TO learnstack_app; + GRANT INSERT (tenant_id, subscription_id, event_id) ON inbox_messages TO learnstack_app; + GRANT SELECT (tenant_id, subscription_id, event_id, organization_id, occurred_at, + type, topic, partition_key, processed_at, attempts, last_error, + available_after, locked_by, locked_until, terminal_reason, + dead_lettered_at, replay_count, replayed_at) + ON event_deliveries TO learnstack_outbox_admin; + GRANT UPDATE (attempts, last_error, available_after, locked_by, locked_until, + terminal_reason, dead_lettered_at, replay_count, replayed_at) + ON event_deliveries TO learnstack_outbox_admin; + """); + } + + /// + protected override void Down(MigrationBuilder migrationBuilder) + { + // Disposable-schema reversal only; live rollback uses a forward fix. + // Neither table has producer/business foreign keys or cascade retention. + migrationBuilder.Sql(""" + DROP TABLE inbox_messages; + DROP TABLE event_deliveries; + DROP FUNCTION fn_inbox_messages_immutable(); + """); + } + } +} diff --git a/backend/src/LearnStack.Infrastructure/Persistence/NpgsqlUnitOfWork.cs b/backend/src/LearnStack.Infrastructure/Persistence/NpgsqlUnitOfWork.cs index 370f4650..832aa455 100644 --- a/backend/src/LearnStack.Infrastructure/Persistence/NpgsqlUnitOfWork.cs +++ b/backend/src/LearnStack.Infrastructure/Persistence/NpgsqlUnitOfWork.cs @@ -29,9 +29,10 @@ namespace LearnStack.Infrastructure.Persistence; /// guarded one, and it is what TransactionBehavior uses. /// /// -/// Three rules that only look like details. A commit disposes its -/// transaction in a finally, so a faulted COMMIT still leaves a -/// clean unit. A rollback on a unit with nothing to resolve is a no-op, because +/// Three rules that only look like details. Acknowledged COMMIT returns +/// before transaction cleanup; the owning frame, unit disposal or next begin +/// disposes that resource. Faulted COMMIT preserves its original exception. +/// A rollback on a unit with nothing to resolve is a no-op, because /// rollback is cleanup and cleanup must never throw over the exception it is /// cleaning up after — measured, the strict form replaced every commit-time /// exception with "no transaction frame is open". And @@ -51,6 +52,7 @@ public sealed class NpgsqlUnitOfWork( private NpgsqlConnection? _connection; private DbTransaction? _transaction; + private DbTransaction? _completedTransaction; private int _depth; private TransactionMode? _mode; @@ -139,6 +141,7 @@ public async Task BeginTransactionAsync( try { + await DisposeCompletedTransactionAsync(); _connection ??= _dataSource.CreateConnection(); if (_connection.State != System.Data.ConnectionState.Open) @@ -172,9 +175,9 @@ public async Task BeginTransactionAsync( { await RollbackCoreAsync(); } - catch (Exception cleanupFailure) + catch { - LogSetupCleanupFailure(_logger, cleanupFailure); + LogSetupCleanupFailure(_logger, null); } throw; @@ -443,6 +446,8 @@ public async ValueTask DisposeAsync() { await RollbackCoreAsync(); } + + await DisposeCompletedTransactionAsync(); } finally { @@ -502,13 +507,42 @@ private async Task CommitFrameAsync(CancellationToken cancellationToken) { await transaction.CommitAsync(cancellationToken); } - finally + catch { - // In a finally, so a faulted COMMIT still leaves the connection clean. - // Deliberately NOT rolled back: a COMMIT that threw leaves the - // server-side outcome genuinely unknown, and ADR-0033 calls that state - // Indeterminate rather than failed. - await transaction.DisposeAsync(); + // Unknown COMMIT remains the primary failure even if resource cleanup + // also fails. No explicit rollback can establish its server outcome. + try + { + await transaction.DisposeAsync(); + } + catch + { + // Connection cleanup remains owned by this unit. + } + throw; + } + + // The caller can now record definitive durability before awaiting cleanup. + // Only one completed transaction can remain: Begin cleans it before reuse. + _completedTransaction = transaction; + } + + private async ValueTask DisposeCompletedTransactionAsync() + { + var completed = _completedTransaction; + _completedTransaction = null; + if (completed is null) return; + + try + { + await completed.DisposeAsync(); + } + catch + { + // Prior COMMIT remains definitive; only reuse of an unclean unit is + // refused. Delivery's acknowledged receipt is already recorded. + _rollbackOnly = true; + throw; } } @@ -646,14 +680,13 @@ public Task FailAsync(CancellationToken cancellationToken = default) return unitOfWork.RollbackAsync(cancellationToken); } - public ValueTask DisposeAsync() => - // Through FailAsync, not the frame-blind RollbackAsync: a frame that - // ends unresolved has failed, and it has failed in exactly the way - // FailAsync already handles — including the leaked-frames collapse. - // Calling RollbackAsync here instead decremented the shared depth by - // one and left the transaction open, so a frame opened later joined an - // abandoned transaction and reported a success that never committed. - new(FailAsync(CancellationToken.None)); + public async ValueTask DisposeAsync() + { + // Through FailAsync, never frame-blind rollback. A stale frame cannot + // roll back or clean resources belonging to a later generation. + await FailAsync(CancellationToken.None); + if (IsOwner && !IsStale()) await unitOfWork.DisposeCompletedTransactionAsync(); + } private bool AlreadyResolved() => _resolved || unitOfWork._disposed || unitOfWork._depth < depth; diff --git a/backend/src/LearnStack.Infrastructure/Persistence/PlatformDbContext.cs b/backend/src/LearnStack.Infrastructure/Persistence/PlatformDbContext.cs index cc0aeb98..6fd6a3f0 100644 --- a/backend/src/LearnStack.Infrastructure/Persistence/PlatformDbContext.cs +++ b/backend/src/LearnStack.Infrastructure/Persistence/PlatformDbContext.cs @@ -7,7 +7,8 @@ namespace LearnStack.Infrastructure.Persistence; /// /// /// -/// outbox_messages and idempotency_keys belong to no module: they +/// outbox_messages, idempotency_keys, event_deliveries and +/// inbox_messages belong to no module: they /// are reached through SharedKernel ports by any module's handler and read by /// infrastructure that belongs to none of them. IIdempotencyStore ships /// today, behind InMemoryIdempotencyStore until its ADR-0035 trigger @@ -28,8 +29,10 @@ namespace LearnStack.Infrastructure.Persistence; /// registration and never acquire a dispatcher connection. /// /// -/// Both tables are tenant-owned, tenant-wide despite living outside a -/// module: every row carries a tenant_id, and the ordinary policy applies. +/// All four tables are tenant-owned despite living outside a module. The +/// delivery ledger is organization-scoped to protect retained envelopes; +/// the other three are tenant-wide machinery. Every row carries a tenant key +/// and the corresponding canonical policy applies. /// The dispatcher reads across tenants through learnstack_outbox_admin's /// BYPASSRLS, bounded by a column-scoped grant rather than by the policy. /// @@ -40,7 +43,7 @@ protected override void OnModelCreating(ModelBuilder modelBuilder) { ArgumentNullException.ThrowIfNull(modelBuilder); - // No entity types. Both tables are written through their ports with + // No entity types. These tables are written through their ports with // parameterised SQL, not through a change tracker: an outbox row is // enqueued and never updated by application code, and an idempotency // claim is a single INSERT ... ON CONFLICT that decides five outcomes in diff --git a/backend/src/LearnStack.Infrastructure/Properties/AssemblyInfo.cs b/backend/src/LearnStack.Infrastructure/Properties/AssemblyInfo.cs index 44fe94bc..a2eac60d 100644 --- a/backend/src/LearnStack.Infrastructure/Properties/AssemblyInfo.cs +++ b/backend/src/LearnStack.Infrastructure/Properties/AssemblyInfo.cs @@ -1,3 +1,4 @@ using System.Runtime.CompilerServices; [assembly: InternalsVisibleTo("LearnStack.Tests.Unit")] +[assembly: InternalsVisibleTo("LearnStack.Tests.Integration")] diff --git a/backend/src/LearnStack.SharedKernel/Audit/AuditDraftComposer.cs b/backend/src/LearnStack.SharedKernel/Audit/AuditDraftComposer.cs index dc756f80..5b6edbde 100644 --- a/backend/src/LearnStack.SharedKernel/Audit/AuditDraftComposer.cs +++ b/backend/src/LearnStack.SharedKernel/Audit/AuditDraftComposer.cs @@ -156,10 +156,36 @@ private static AuditEntryDraft Compose( IpAddress = null, UserAgent = null, Timestamp = timestamp, - Metadata = null, + Metadata = ComposeMetadata(intent), }; } + private static string? ComposeMetadata(AuditIntent intent) + { + if (intent.CausationId is null && intent.CausalActorUserId is null) return null; + if (intent.CausationId == Guid.Empty + || (intent.CausalActorUserId is { } actor + && (!actor.IsInitialized() || actor.Value == Guid.Empty))) + { + throw new ArgumentException("Delivery audit metadata requires initialized, nonempty identifiers.", nameof(intent)); + } + + // This closed identifier-only shape is shared by both writing paths; it + // grants no arbitrary metadata channel to an intent or business handler. + var fields = new List(2); + if (intent.CausationId is { } causation) + { + fields.Add("\"causation_id\":" + AuditJson.Quote(causation.ToString("D"))); + } + + if (intent.CausalActorUserId is { } causalActor) + { + fields.Add("\"causal_actor_user_id\":" + AuditJson.Quote(causalActor.Value.ToString("D"))); + } + + return "{" + string.Join(',', fields) + "}"; + } + /// /// The changes column: a JSON array of { path, before, after }. /// diff --git a/backend/src/LearnStack.SharedKernel/Audit/AuditIntent.cs b/backend/src/LearnStack.SharedKernel/Audit/AuditIntent.cs index 9e970065..c7ec258c 100644 --- a/backend/src/LearnStack.SharedKernel/Audit/AuditIntent.cs +++ b/backend/src/LearnStack.SharedKernel/Audit/AuditIntent.cs @@ -99,6 +99,12 @@ public sealed record AuditIntent( Type? EntityType, DateTimeOffset DeclaredAt) { + /// The trusted incoming event identity for delivery-originated work. + public Guid? CausationId { get; init; } + + /// The original causal actor, separate from the effective system actor. + public UserId? CausalActorUserId { get; init; } + /// /// The instance the row is about, when the handler designated one through /// ; null otherwise. diff --git a/backend/src/LearnStack.SharedKernel/Audit/EventTerminalTransition.cs b/backend/src/LearnStack.SharedKernel/Audit/EventTerminalTransition.cs index f84db3c0..c03a16ea 100644 --- a/backend/src/LearnStack.SharedKernel/Audit/EventTerminalTransition.cs +++ b/backend/src/LearnStack.SharedKernel/Audit/EventTerminalTransition.cs @@ -5,6 +5,7 @@ using System.Globalization; using System.Text; using System.Text.Json; +using System.Text.RegularExpressions; using LearnStack.SharedKernel.Identifiers; namespace LearnStack.SharedKernel.Audit; @@ -12,8 +13,8 @@ namespace LearnStack.SharedKernel.Audit; /// The immutable result of one fenced terminal UPDATE in its live transaction. /// /// Construction is private; the protected minting seam is confined to the producer -/// transition store by the production architecture census. It grants no friend-assembly -/// access to other SharedKernel authority. Subscriber minting belongs to P02b-3. +/// and subscriber transition stores by the production architecture census. It grants +/// no friend-assembly access to other SharedKernel authority. /// public sealed class EventTerminalTransition { @@ -24,7 +25,7 @@ private EventTerminalTransition( DbConnection connection, DbTransaction transaction, DbDataReader returnedRow, - string validatedAttemptTraceparent) + string validatedAttemptTraceparent, bool subscriber) { EnsureLiveTransaction(connection, transaction); if (!TryCanonicalTraceparent(validatedAttemptTraceparent, out var attempt) @@ -33,7 +34,7 @@ private EventTerminalTransition( throw new ArgumentException("The terminal attempt requires a canonical W3C traceparent."); } - if (returnedRow.FieldCount != 9) + if (returnedRow.FieldCount != (subscriber ? 7 : 9)) { throw new ArgumentException("The terminal UPDATE must return its complete bound row."); } @@ -46,11 +47,17 @@ private EventTerminalTransition( Reason = returnedRow.GetString(3); Attempts = returnedRow.GetInt32(4); AcquisitionToken = returnedRow.GetString(5); + Kind = subscriber ? "subscriber" : "producer"; + SubscriptionId = subscriber ? returnedRow.GetString(6) : null; if (EventId == Guid.Empty || TenantId == Guid.Empty || TenantId == Identifiers.TenantId.PlatformSentinel.Value || OrganizationId == Guid.Empty || Attempts < 0 - || Reason is not ("unknown_type" or "retired_type" or "invalid_envelope" or "retry_exhausted") + || (Reason is not ("unknown_type" or "retired_type" or "invalid_envelope" or "retry_exhausted") + && !(subscriber && Reason == "invalid_subscription")) + || (subscriber && !Regex.IsMatch(SubscriptionId!, + @"\A[a-z][a-z0-9]*\.[a-z][a-z0-9-]*\.v[1-9][0-9]*\z", RegexOptions.CultureInvariant, + TimeSpan.FromMilliseconds(100))) || !Guid.TryParseExact(AcquisitionToken, "D", out var token) || token == Guid.Empty || !string.Equals(AcquisitionToken, token.ToString("D"), StringComparison.Ordinal)) @@ -58,6 +65,13 @@ private EventTerminalTransition( throw new ArgumentException("The terminal UPDATE returned invalid transition identity."); } + if (subscriber) + { + CorrelationId = attempt; + Metadata = ComposeMetadata(null, null); + return; + } + Guid? causationId = returnedRow.IsDBNull(7) ? null : returnedRow.GetGuid(7); Guid? causalActorUserId = returnedRow.IsDBNull(8) ? null : returnedRow.GetGuid(8); CorrelationId = TryCanonicalTraceparent( @@ -75,7 +89,7 @@ private EventTerminalTransition( public string AcquisitionToken { get; } public string CorrelationId { get; } public string Metadata { get; } - public string Kind { get; } = "producer"; + public string Kind { get; } public string? SubscriptionId { get; } /// Refuses a draft or transaction that does not describe this exact transition. @@ -122,7 +136,20 @@ protected static EventTerminalTransition FromProducerUpdate( ArgumentNullException.ThrowIfNull(connection); ArgumentNullException.ThrowIfNull(transaction); ArgumentNullException.ThrowIfNull(returnedRow); - return new EventTerminalTransition(connection, transaction, returnedRow, validatedAttemptTraceparent); + return new EventTerminalTransition(connection, transaction, returnedRow, validatedAttemptTraceparent, subscriber: false); + } + + /// Reads the actual routing-only result of the successful fenced subscriber UPDATE. + protected static EventTerminalTransition FromSubscriberUpdate( + DbConnection connection, + DbTransaction transaction, + DbDataReader returnedRow, + string validatedAttemptTraceparent) + { + ArgumentNullException.ThrowIfNull(connection); + ArgumentNullException.ThrowIfNull(transaction); + ArgumentNullException.ThrowIfNull(returnedRow); + return new EventTerminalTransition(connection, transaction, returnedRow, validatedAttemptTraceparent, subscriber: true); } } @@ -141,6 +168,7 @@ private string ComposeMetadata(Guid? causationId, Guid? causalActorUserId) { writer.WriteStartObject(); writer.WriteString("event_id", EventId); + if (SubscriptionId is not null) writer.WriteString("subscription_id", SubscriptionId); writer.WriteString("reason", Reason); writer.WriteNumber("attempts", Attempts); if (causationId is { } causation && causation != Guid.Empty) diff --git a/backend/src/LearnStack.SharedKernel/Audit/IAuditStore.cs b/backend/src/LearnStack.SharedKernel/Audit/IAuditStore.cs index f8fcb625..be8cd409 100644 --- a/backend/src/LearnStack.SharedKernel/Audit/IAuditStore.cs +++ b/backend/src/LearnStack.SharedKernel/Audit/IAuditStore.cs @@ -131,8 +131,9 @@ Task WritePlatformScopeAsync( /// must be declared off-path as MUST/SecurityEvent, and the draft must match /// the actual UPDATE result carried by . /// Any mismatch or write failure requires the caller to roll back the state - /// transition. This seam opens and commits no transaction. P02b-3 owns the - /// future subscriber writer; the existing four writes retain their contracts. + /// transition. This seam opens and commits no transaction. P02b-3 Step 4 + /// delivers the subscriber transition caller on the same witness-bound seam; + /// the existing four writes retain their contracts. /// Task WriteEventTerminalAsync( AuditEntryDraft entry, diff --git a/backend/src/LearnStack.SharedKernel/Messaging/IEventBus.cs b/backend/src/LearnStack.SharedKernel/Messaging/IEventBus.cs index 16126504..e21f2815 100644 --- a/backend/src/LearnStack.SharedKernel/Messaging/IEventBus.cs +++ b/backend/src/LearnStack.SharedKernel/Messaging/IEventBus.cs @@ -14,7 +14,7 @@ namespace LearnStack.SharedKernel.Messaging; /// transport resolving IIntegrationEventHandler<TEvent> would then /// look for a handler of IIntegrationEvent, which no concrete consumer /// implements. The publish would reach zero handlers and report success. Both -/// transports resolve by the event's runtime type instead. +/// transports resolve the declared policy by the event's runtime type instead. /// /// /// The envelope carries the dispatch metadata the outbox row holds and the event @@ -24,7 +24,12 @@ namespace LearnStack.SharedKernel.Messaging; /// public interface IEventBus { - /// Publishes one envelope, ordered against others sharing its partition key. + /// Durably accepts one envelope, ordered against others sharing its partition key. + /// + /// Success means every required subscription has a durable delivery identity, + /// or the validated contract explicitly declares intentional absence. It does + /// not acknowledge business execution. Failed or unknown acceptance is retryable. + /// Task PublishAsync( IntegrationEventEnvelope envelope, CancellationToken cancellationToken = default); diff --git a/backend/src/LearnStack.SharedKernel/Messaging/IInboxGuard.cs b/backend/src/LearnStack.SharedKernel/Messaging/IInboxGuard.cs new file mode 100644 index 00000000..22a4ff3b --- /dev/null +++ b/backend/src/LearnStack.SharedKernel/Messaging/IInboxGuard.cs @@ -0,0 +1,19 @@ +using LearnStack.SharedKernel.Identifiers; + +namespace LearnStack.SharedKernel.Messaging; + +/// Claims a stable subscription/event identity on the transport's business transaction. +/// +/// Only the transport calls this, before handler construction. A true result is +/// provisional until the owning business transaction commits; rollback removes it. +/// False means the identity already exists. A competing transaction must commit +/// before its claim can produce that result. +/// +public interface IInboxGuard +{ + Task TryClaimAsync( + TenantId tenantId, + string subscriptionId, + Guid eventId, + CancellationToken cancellationToken = default); +} diff --git a/backend/src/LearnStack.SharedKernel/Messaging/IIntegrationEventHandler.cs b/backend/src/LearnStack.SharedKernel/Messaging/IIntegrationEventHandler.cs index 65a6ccb0..3f78f07e 100644 --- a/backend/src/LearnStack.SharedKernel/Messaging/IIntegrationEventHandler.cs +++ b/backend/src/LearnStack.SharedKernel/Messaging/IIntegrationEventHandler.cs @@ -19,8 +19,9 @@ namespace LearnStack.SharedKernel.Messaging; /// handler; handlers do not call it. Business state, inbox completion and fenced /// delivery completion share one transaction. Durable acceptance, the shared /// inbox_messages table and the nonempty production proof for -/// Integration_Event_Handlers_Use_InboxGuard belong to P02b-3. -/// The shipped in-process baseline does not yet provide durable deduplication. +/// Integration_Event_Handlers_Use_InboxGuard are delivered by P02b-3. +/// Publication now durably accepts delivery identities without constructing handlers; +/// P02b-3 also delivers the separate business execution boundary. /// /// /// @@ -28,8 +29,10 @@ namespace LearnStack.SharedKernel.Messaging; /// Invariant on purpose. Declaring in TEvent would promise a /// variance the container does not honour: measured, a handler registered for a /// base event type compiles, registers, and is never invoked, because -/// GetServices matches the closed generic exactly — and "no handler" is -/// not an error here, so the publish reports success having reached nobody. +/// GetServices matches the closed generic exactly — and "no handler" +/// was not an error in the immediate-broadcast baseline, so publication could +/// report success having reached nobody. The durable registry now refuses +/// undeclared contracts and requires exact Active handler declarations. /// A promise the runtime cannot keep is worse than no promise. /// /// diff --git a/backend/src/LearnStack.SharedKernel/Tenancy/EventTenantContext.cs b/backend/src/LearnStack.SharedKernel/Tenancy/EventTenantContext.cs index da9424d6..a2b97357 100644 --- a/backend/src/LearnStack.SharedKernel/Tenancy/EventTenantContext.cs +++ b/backend/src/LearnStack.SharedKernel/Tenancy/EventTenantContext.cs @@ -18,12 +18,14 @@ namespace LearnStack.SharedKernel.Tenancy; public sealed class EventTenantContext : ITenantContext { private EventTenantContext( + Guid eventId, TenantId tenantId, OrganizationId? organizationId, UserId? causalActorUserId, string? correlationId, string? moduleName) { + EventId = eventId; TenantId = tenantId; OrganizationId = organizationId; UserId = Identifiers.UserId.SystemActor; @@ -35,6 +37,9 @@ private EventTenantContext( /// public bool IsResolved => true; + /// The validated incoming event that causes this delivery's work. + public Guid EventId { get; } + /// Matrix row 17: the envelope carried the tenant. /// /// Proof that the factory is not the only producer of a resolved context — it is @@ -122,6 +127,7 @@ public static EventTenantContext FromEnvelope( } return new EventTenantContext( + envelope.Event.EventId, Identifiers.TenantId.From(envelope.Event.TenantId), envelope.OrganizationId is { } organization ? Identifiers.OrganizationId.From(organization) diff --git a/backend/src/LearnStack.Tools.Seeder/DemoIdentities.cs b/backend/src/LearnStack.Tools.Seeder/DemoIdentities.cs index dbd6c10e..50c64873 100644 --- a/backend/src/LearnStack.Tools.Seeder/DemoIdentities.cs +++ b/backend/src/LearnStack.Tools.Seeder/DemoIdentities.cs @@ -72,6 +72,18 @@ void Labels(string kind, IReadOnlyDictionary labels) } } } + // Built-ins retain their semantic vocabulary; their tenant-owned identities + // are generated at initialization and are not demo literals. + Add("content-type-key", BuiltInCustomizations.Card.Key); + Labels("definition-label", BuiltInCustomizations.Card.DisplayName); + foreach (var identity in SchemaValues(BuiltInCustomizations.Card.JsonSchema)) values.Add(identity); + Add("taxonomy-key", BuiltInCustomizations.Plain.Key); + Labels("definition-label", BuiltInCustomizations.Plain.DisplayName); + foreach (var band in BuiltInCustomizations.Plain.Bands) + { + Add("band-key", band.Key); + Labels("band-label", band.DisplayName); + } var platform = new HashSet(StringComparer.Ordinal) { BuiltInCustomizations.Card.Key, BuiltInCustomizations.Card.RendererKey, BuiltInCustomizations.Plain.Key diff --git a/backend/src/LearnStack.Tools.Seeder/SeedData.cs b/backend/src/LearnStack.Tools.Seeder/SeedData.cs index 6c867104..c04a9880 100644 --- a/backend/src/LearnStack.Tools.Seeder/SeedData.cs +++ b/backend/src/LearnStack.Tools.Seeder/SeedData.cs @@ -20,18 +20,17 @@ public static class SeedData public static readonly SeedTenant English = new( TenantId.From(Id("01930000-0000-7000-8000-000000000001")), "demo-english", "English Hero", FirstBranch, SecondBranch, "demo-english.learnstack.local", false, - Id("01930000-0000-7000-8000-0000000000c1"), Id("01930000-0000-7000-8000-0000000000d1"), EnglishCurriculum()); public static readonly SeedTenant Yoga = new( TenantId.From(Id("01930000-0000-7000-8000-000000000002")), "demo-yoga", "Anatolia Yoga", FirstStudio, SecondStudio, "demo-yoga.learnstack.local", true, - Id("01930000-0000-7000-8000-0000000000c2"), Id("01930000-0000-7000-8000-0000000000d2"), YogaCurriculum()); public static readonly IReadOnlyList All = [English, Yoga]; // Provisioning audits both sanctioned roots; follow-on organization and host each add one. + // Each tenant initializes two built-ins once; authored definitions audit registration and publication. public static long ExpectedAuditWrites => Inventory.Tenants * 4L + Inventory.Locales - + 2L * (Inventory.ContentTypes + Inventory.Taxonomies) + Inventory.Themes + + All.Count * 2L + 2L * All.Sum(tenant => ContentTypes(tenant).Count() + Taxonomies(tenant).Count()) + Inventory.Themes + Inventory.Courses + Inventory.Lessons + Inventory.Translations + All.Sum(tenant => tenant.Curriculum?.Courses.Sum(course => (course.Status == "Published" ? 1 : 0) + course.Lessons.Count(lesson => lesson.Status == "Published")) ?? 0); @@ -39,8 +38,8 @@ public static class SeedData public static SeedInventory Inventory => new( All.Count, All.Count * 2, All.Count, All.Sum(tenant => tenant.Curriculum?.Locales.Length ?? 0), - All.Sum(tenant => ContentTypes(tenant).Count()), All.Sum(tenant => Taxonomies(tenant).Count()), - All.Sum(tenant => Taxonomies(tenant).Sum(taxonomy => taxonomy.Bands.Length)), + All.Sum(tenant => 1 + ContentTypes(tenant).Count()), All.Sum(tenant => 1 + Taxonomies(tenant).Count()), + All.Sum(tenant => BuiltInCustomizations.Plain.Bands.Count + Taxonomies(tenant).Sum(taxonomy => taxonomy.Bands.Length)), All.Count(tenant => tenant.Curriculum is not null), All.Sum(tenant => tenant.Curriculum?.Courses.Length ?? 0), All.Sum(tenant => tenant.Curriculum?.Courses.Sum(course => course.Lessons.Length) ?? 0), @@ -49,24 +48,21 @@ public static class SeedData public static IEnumerable ContentTypes(SeedTenant tenant) { - yield return new(tenant.BuiltInContentTypeId, BuiltInCustomizations.Card.Key, - BuiltInCustomizations.SchemaVersion, BuiltInCustomizations.Card.DisplayName, - BuiltInCustomizations.Card.JsonSchema, BuiltInCustomizations.Card.RendererKey); if (tenant.Curriculum is { } curriculum) yield return curriculum.ContentType; } public static IEnumerable Taxonomies(SeedTenant tenant) { - yield return new(tenant.BuiltInTaxonomyId, BuiltInCustomizations.Plain.Key, - BuiltInCustomizations.SchemaVersion, BuiltInCustomizations.Plain.DisplayName, - [.. BuiltInCustomizations.Plain.Bands.Select(band => new SeedBand(band.Key, band.DisplayName, band.Sort))]); if (tenant.Curriculum is { } curriculum) yield return curriculum.Taxonomy; } public static string[] CustomizationProjection(SeedTenant tenant) => - [.. ContentTypes(tenant).Select(type => $"content-type:{type.Key}@{tenant.TenantId}"), + [$"content-type:{BuiltInCustomizations.Card.Key}@{tenant.TenantId}", + $"taxonomy:{BuiltInCustomizations.Plain.Key}@{tenant.TenantId}", + .. BuiltInCustomizations.Plain.Bands.Select(band => $"band:{band.Key}@{tenant.TenantId}"), + .. ContentTypes(tenant).Select(type => $"content-type:{type.Key}@{tenant.TenantId}"), .. Taxonomies(tenant).Select(taxonomy => $"taxonomy:{taxonomy.Key}@{tenant.TenantId}"), .. Taxonomies(tenant).SelectMany(taxonomy => taxonomy.Bands.Select(band => $"band:{band.Key}@{tenant.TenantId}"))]; public static long CustomizationGeneration(SeedTenant tenant) => - 2L * (ContentTypes(tenant).Count() + Taxonomies(tenant).Count()); + 2L + 2L * (ContentTypes(tenant).Count() + Taxonomies(tenant).Count()); private static SeedCurriculum EnglishCurriculum() => new( [new("en", true, true, 0)], @@ -169,7 +165,7 @@ private static ImmutableDictionary Labels(string en, string? tr public sealed record SeedTenant(TenantId TenantId, string Slug, string DisplayName, SeedOrganization DefaultOrganization, SeedOrganization SecondOrganization, string Host, - bool MapHostToDefaultOrganization, Guid BuiltInContentTypeId, Guid BuiltInTaxonomyId, + bool MapHostToDefaultOrganization, SeedCurriculum? Curriculum = null); public sealed record SeedOrganization(OrganizationId OrganizationId, string Slug, string DisplayName); public sealed record SeedCurriculum(ImmutableArray Locales, SeedContentType ContentType, diff --git a/backend/src/LearnStack.Tools.Seeder/SeedRunner.cs b/backend/src/LearnStack.Tools.Seeder/SeedRunner.cs index 1acb8a29..853bea01 100644 --- a/backend/src/LearnStack.Tools.Seeder/SeedRunner.cs +++ b/backend/src/LearnStack.Tools.Seeder/SeedRunner.cs @@ -55,6 +55,7 @@ await ActAsync(tenant, context, "host mapping", foreach (var locale in curriculum.Locales.OrderByDescending(locale => locale.IsDefault).ThenBy(locale => locale.Sort)) await ActAsync(tenant, context, "locale", ReadTenant, row => SeedVerification.Locale(row, locale, tenant), row => new AddTenantLocaleCommand(Version(row), locale.Locale, locale.IsEnabled, locale.IsDefault, locale.Sort), ct); + await SeedBuiltInsAsync(tenant, context, ct); foreach (var type in SeedData.ContentTypes(tenant)) await SeedContentTypeAsync(tenant, context, type, ct); foreach (var taxonomy in SeedData.Taxonomies(tenant)) await SeedTaxonomyAsync(tenant, context, taxonomy, ct); if (tenant.Curriculum is not { } content) return; @@ -66,6 +67,35 @@ await ActAsync(tenant, context, "branding", foreach (var course in content.Courses) await PublishCourseAsync(tenant, course, ct); } + private async Task SeedBuiltInsAsync(SeedTenant tenant, ITenantContext context, CancellationToken ct) + { + async Task ReadContentType() + { + var active = (await ReadAsync(context, new GetActiveContentTypeSeedRevisionQuery(BuiltInCustomizations.Card.Key), ct)).State; + if (active is null) return null; + SeedVerification.Require(active.Id != Guid.Empty, tenant, "built-in content type identity"); + var row = (await ReadAsync(context, new GetContentTypeSeedStateQuery(active.Id), ct)).State; + SeedVerification.Require(row is not null && row.Id == active.Id, tenant, "built-in content type identity"); + return row; + } + await ActAsync(tenant, context, "built-in content type", ReadContentType, + row => SeedVerification.BuiltInContentType(row, tenant), + _ => new InitializeTenantContentTypeDefaultCommand(), ct); + + async Task ReadTaxonomy() + { + var active = (await ReadAsync(context, new GetActiveTaxonomySeedRevisionQuery(BuiltInCustomizations.Plain.Key), ct)).State; + if (active is null) return null; + SeedVerification.Require(active.Id != Guid.Empty, tenant, "built-in level taxonomy identity"); + var row = (await ReadAsync(context, new GetTaxonomySeedStateQuery(active.Id), ct)).State; + SeedVerification.Require(row is not null && row.Id == active.Id, tenant, "built-in level taxonomy identity"); + return row; + } + await ActAsync(tenant, context, "built-in level taxonomy", ReadTaxonomy, + row => SeedVerification.BuiltInTaxonomy(row, tenant), + _ => new InitializeTenantLevelTaxonomyDefaultCommand(), ct); + } + private async Task SeedContentTypeAsync(SeedTenant tenant, ITenantContext context, SeedContentType type, CancellationToken ct) { async Task Read() diff --git a/backend/src/LearnStack.Tools.Seeder/SeedVerification.cs b/backend/src/LearnStack.Tools.Seeder/SeedVerification.cs index 9092bcc3..011b1910 100644 --- a/backend/src/LearnStack.Tools.Seeder/SeedVerification.cs +++ b/backend/src/LearnStack.Tools.Seeder/SeedVerification.cs @@ -1,6 +1,7 @@ using System.Text.Json; using System.Text.Json.Nodes; using LearnStack.Modules.Customization.Application.Contracts.Seeding; +using LearnStack.Modules.Customization.Application.Contracts.Customization; using LearnStack.Modules.Education.Application.Contracts.Seeding; using LearnStack.Modules.Tenancy.Application.Contracts.Seeding; using LearnStack.Modules.Tenancy.Application.Branding; @@ -71,6 +72,30 @@ internal static bool Locale(TenantSeedDto? row, SeedLocale expected, SeedTenant private static bool LocaleMatches(TenantLocaleSeedDto row, SeedLocale expected) => row.Locale == expected.Locale && row.IsEnabled == expected.IsEnabled && row.IsDefault == expected.IsDefault && row.Sort == expected.Sort; + internal static bool BuiltInContentType(ContentTypeSeedDto? row, SeedTenant tenant) + { + if (row is null) return false; + Require(row.Id != Guid.Empty && row.TenantId == tenant.TenantId && row.Key == BuiltInCustomizations.Card.Key + && row.SchemaVersion == BuiltInCustomizations.SchemaVersion && row.Status == "Active" + && JsonEqual(row.DisplayNameJson, JsonSerializer.Serialize(BuiltInCustomizations.Card.DisplayName)) + && JsonEqual(row.JsonSchema, BuiltInCustomizations.Card.JsonSchema) + && row.RendererKey == BuiltInCustomizations.Card.RendererKey, tenant, "built-in content type"); + return true; + } + + internal static bool BuiltInTaxonomy(TaxonomySeedDto? row, SeedTenant tenant) + { + if (row is null) return false; + Require(row.Id != Guid.Empty && row.TenantId == tenant.TenantId && row.Key == BuiltInCustomizations.Plain.Key + && row.SchemaVersion == BuiltInCustomizations.SchemaVersion && row.Status == "Active" + && JsonEqual(row.DisplayNameJson, JsonSerializer.Serialize(BuiltInCustomizations.Plain.DisplayName)) + && row.Items.Length == BuiltInCustomizations.Plain.Bands.Count + && row.Items.All(actual => BuiltInCustomizations.Plain.Bands.Any(band => actual.Key == band.Key && actual.Sort == band.Sort + && JsonEqual(actual.DisplayNameJson, JsonSerializer.Serialize(band.DisplayName)) && actual.Metadata is null)), + tenant, "built-in level taxonomy"); + return true; + } + internal static bool ContentType(ContentTypeSeedDto? row, SeedContentType expected, SeedTenant tenant, bool published) { if (row is null) return false; diff --git a/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application.Contracts/Customization/BuiltInCustomizations.cs b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application.Contracts/Customization/BuiltInCustomizations.cs index 8d64e939..2026488d 100644 --- a/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application.Contracts/Customization/BuiltInCustomizations.cs +++ b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application.Contracts/Customization/BuiltInCustomizations.cs @@ -9,7 +9,7 @@ namespace LearnStack.Modules.Customization.Application.Contracts.Customization; /// Data, not code, and that is the point being demonstrated. /// ADR-0018 /// says a tenant's shapes are rows; these are the first rows, and they go in -/// through the same four commands a tenant admin uses. Nothing here is privileged +/// through the owner-local create-only initialization commands. Nothing here is privileged /// — a tenant may deprecate either one and publish its own successor on day one, /// which is exactly what a language school replacing plain with CEFR does. /// @@ -22,15 +22,15 @@ namespace LearnStack.Modules.Customization.Application.Contracts.Customization; /// Vision § Genericity boundary is where that line is drawn. /// /// -/// Applied by whoever creates the tenant, never by provisioning itself. +/// Applied by independent consumers and the Seeder, never by provisioning itself. /// ADR-0042 /// keeps the cross-aggregate allow-list at one entry, and folding a customization /// write into ProvisionTenantCommand would add a second aggregate to that -/// transaction. Today the caller is LearnStack.Tools.Seeder; the -/// self-service signup that will need the same four commands for a real tenant -/// lands with the Hub-side onboarding in -/// Phase -/// 02c, and reads this same declaration rather than a second copy of it. +/// transaction. The two TenantCreatedIntegrationEventV1 consumers each +/// initialize one root; LearnStack.Tools.Seeder uses the same commands. +/// Any concept history is preserved, including deleted and successor-only rows. +/// Seeder verification discovers the semantic key/version and retains the stored +/// identifier rather than requiring a demo identifier. /// /// public static class BuiltInCustomizations diff --git a/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application.Contracts/Customization/InitializeTenantDefaultsCommands.cs b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application.Contracts/Customization/InitializeTenantDefaultsCommands.cs new file mode 100644 index 00000000..8f224f47 --- /dev/null +++ b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application.Contracts/Customization/InitializeTenantDefaultsCommands.cs @@ -0,0 +1,10 @@ +using LearnStack.SharedKernel.Results; +using MediatR; + +namespace LearnStack.Modules.Customization.Application.Contracts.Customization; + +/// Creates Active card@1 only when the ambient tenant has no card history. +public sealed record InitializeTenantContentTypeDefaultCommand : IRequest>; + +/// Creates Active plain@1 only when the ambient tenant has no plain history. +public sealed record InitializeTenantLevelTaxonomyDefaultCommand : IRequest>; diff --git a/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Abstractions/CustomizationStores.cs b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Abstractions/CustomizationStores.cs index e26ec820..eee39fc9 100644 --- a/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Abstractions/CustomizationStores.cs +++ b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Abstractions/CustomizationStores.cs @@ -34,6 +34,12 @@ namespace LearnStack.Modules.Customization.Application.Abstractions; public interface ITenantContentTypeStore : IAggregateWriteStore { + /// Serializes registration and initialization for this tenant's concept until transaction end. + Task LockConceptAsync(string key, CancellationToken cancellationToken = default); + + /// Whether any revision exists, including Draft, deprecated and soft-deleted history. + Task HasConceptHistoryAsync(string key, CancellationToken cancellationToken = default); + /// The revision with this id, tracked, or nothing. Task FindAsync( TenantContentTypeId id, CancellationToken cancellationToken = default); @@ -61,6 +67,12 @@ public interface ITenantContentTypeStore public interface ITenantLevelTaxonomyStore : IAggregateWriteStore { + /// + Task LockConceptAsync(string key, CancellationToken cancellationToken = default); + + /// + Task HasConceptHistoryAsync(string key, CancellationToken cancellationToken = default); + /// The revision with this id and its bands, tracked, or nothing. Task FindAsync( TenantLevelTaxonomyId id, CancellationToken cancellationToken = default); diff --git a/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Audit/CustomizationAuditCatalogSource.cs b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Audit/CustomizationAuditCatalogSource.cs index f8f29530..386122bb 100644 --- a/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Audit/CustomizationAuditCatalogSource.cs +++ b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Audit/CustomizationAuditCatalogSource.cs @@ -1,5 +1,6 @@ using LearnStack.Modules.Customization.Application.Contracts.Seeding; using LearnStack.Modules.Customization.Application.Contracts.Customization; +using LearnStack.Modules.Customization.Application.IntegrationEvents; using LearnStack.Modules.Customization.Domain; using LearnStack.SharedKernel.Audit; @@ -10,7 +11,7 @@ namespace LearnStack.Modules.Customization.Application.Audit; /// the matrix. /// /// -/// Four commands, four MUST rows. Registration and publication are separate operations +/// Each write command declares its own MUST operation. Registration and publication are separate operations /// because they have different blast radii — a wrong draft constrains nothing, while a /// wrong publication retires the incumbent and changes what every subsequent content write /// is validated against — and the matrix carries one row for each. @@ -29,8 +30,18 @@ public void Describe(IAuditCatalogBuilder builder) builder.Off(); builder.Off(); builder.Off(); + builder.Off(); + builder.Off(); builder + .MustAudit( + "customization.content_type.initialize", + OperationType.Create, + typeof(TenantContentType)) + .MustAudit( + "customization.level_taxonomy.initialize", + OperationType.Create, + typeof(TenantLevelTaxonomy)) .MustAudit( "customization.content_type.register", OperationType.Create, diff --git a/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Customization/InitializeTenantContentTypeDefaultCommandHandler.cs b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Customization/InitializeTenantContentTypeDefaultCommandHandler.cs new file mode 100644 index 00000000..76395ff6 --- /dev/null +++ b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Customization/InitializeTenantContentTypeDefaultCommandHandler.cs @@ -0,0 +1,58 @@ +using LearnStack.Modules.Customization.Application.Abstractions; +using LearnStack.Modules.Customization.Application.Contracts.Customization; +using LearnStack.Modules.Customization.Domain; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Localization; +using LearnStack.SharedKernel.Results; +using LearnStack.SharedKernel.Tenancy; +using LearnStack.SharedKernel.Time; +using LearnStack.SharedKernel.Validation; +using MediatR; + +namespace LearnStack.Modules.Customization.Application.Customization; + +/// Installs the built-in only for a concept the tenant has never touched. +internal sealed class InitializeTenantContentTypeDefaultCommandHandler( + ITenantContentTypeStore contentTypes, + ICustomizationGenerationStore generations, + ITenantLevelTaxonomyCatalog taxonomies, + IJsonSchemaValidator schemas, + ITenantContext tenantContext, + IClock clock, + IGuidFactory guids) + : IRequestHandler> +{ + public async Task> Handle( + InitializeTenantContentTypeDefaultCommand request, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(request); + if (!tenantContext.IsResolved) return CustomizationFailures.Unresolved(); + + var definition = BuiltInCustomizations.Card; + await contentTypes.LockConceptAsync(definition.Key, cancellationToken); + if (await contentTypes.HasConceptHistoryAsync(definition.Key, cancellationToken)) + return Result.Ok(None.Value); + + var admitted = schemas.AdmitSchema(definition.JsonSchema); + if (admitted.IsFailure) return CustomizationFailures.SchemaRefused(admitted.Error!); + + var presentation = TextCardPresentation.Resolve(definition.JsonSchema, definition.RendererKey); + if (presentation.IsFailure) return CustomizationFailures.SchemaRefused(presentation.Error); + + var unresolved = await SchemaExtensionResolution.UnresolvedAsync( + admitted.Value!, taxonomies, cancellationToken, textCardResolved: true); + if (unresolved.Count > 0) return CustomizationFailures.ExtensionsUnresolved(unresolved); + + var actor = tenantContext.UserId ?? UserId.SystemActor; + var root = TenantContentType.Create( + TenantContentTypeId.From(guids.NewUuidV7()), tenantContext.TenantId, + definition.Key, BuiltInCustomizations.SchemaVersion, LocalizedText.From(definition.DisplayName), + definition.JsonSchema, definition.RendererKey, clock, actor); + root.Publish(clock, actor); + + // One save observes the complete Active root and produces one initialization audit. + await contentTypes.AddAsync(root, cancellationToken); + await generations.BumpAsync(tenantContext.TenantId, cancellationToken); + return Result.Ok(None.Value); + } +} diff --git a/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Customization/InitializeTenantLevelTaxonomyDefaultCommandHandler.cs b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Customization/InitializeTenantLevelTaxonomyDefaultCommandHandler.cs new file mode 100644 index 00000000..aa415eae --- /dev/null +++ b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Customization/InitializeTenantLevelTaxonomyDefaultCommandHandler.cs @@ -0,0 +1,46 @@ +using LearnStack.Modules.Customization.Application.Abstractions; +using LearnStack.Modules.Customization.Application.Contracts.Customization; +using LearnStack.Modules.Customization.Domain; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Localization; +using LearnStack.SharedKernel.Results; +using LearnStack.SharedKernel.Tenancy; +using LearnStack.SharedKernel.Time; +using MediatR; + +namespace LearnStack.Modules.Customization.Application.Customization; + +/// Installs the built-in vocabulary without revising any existing history. +internal sealed class InitializeTenantLevelTaxonomyDefaultCommandHandler( + ITenantLevelTaxonomyStore taxonomies, + ICustomizationGenerationStore generations, + ITenantContext tenantContext, + IClock clock, + IGuidFactory guids) + : IRequestHandler> +{ + public async Task> Handle( + InitializeTenantLevelTaxonomyDefaultCommand request, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(request); + if (!tenantContext.IsResolved) return CustomizationFailures.Unresolved(); + + var definition = BuiltInCustomizations.Plain; + await taxonomies.LockConceptAsync(definition.Key, cancellationToken); + if (await taxonomies.HasConceptHistoryAsync(definition.Key, cancellationToken)) + return Result.Ok(None.Value); + + var actor = tenantContext.UserId ?? UserId.SystemActor; + var root = TenantLevelTaxonomy.Create( + TenantLevelTaxonomyId.From(guids.NewUuidV7()), tenantContext.TenantId, + definition.Key, BuiltInCustomizations.SchemaVersion, LocalizedText.From(definition.DisplayName), + clock, actor); + foreach (var band in definition.Bands) + root.AddItem(band.Key, LocalizedText.From(band.DisplayName), band.Sort, null, clock, actor); + root.Publish(clock, actor); + + await taxonomies.AddAsync(root, cancellationToken); + await generations.BumpAsync(tenantContext.TenantId, cancellationToken); + return Result.Ok(None.Value); + } +} diff --git a/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Customization/RegisterTenantContentTypeCommandHandler.cs b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Customization/RegisterTenantContentTypeCommandHandler.cs index 64e0dbb3..a032a618 100644 --- a/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Customization/RegisterTenantContentTypeCommandHandler.cs +++ b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Customization/RegisterTenantContentTypeCommandHandler.cs @@ -96,6 +96,7 @@ public async Task> Handle( try { + await contentTypes.LockConceptAsync(request.Key, cancellationToken); await contentTypes.AddAsync(contentType, cancellationToken); } catch (AggregateConflictException conflict) diff --git a/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Customization/RegisterTenantLevelTaxonomyCommandHandler.cs b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Customization/RegisterTenantLevelTaxonomyCommandHandler.cs index e569c6ad..cf2bb582 100644 --- a/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Customization/RegisterTenantLevelTaxonomyCommandHandler.cs +++ b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/Customization/RegisterTenantLevelTaxonomyCommandHandler.cs @@ -81,6 +81,7 @@ public async Task> Handle( try { + await taxonomies.LockConceptAsync(request.Key, cancellationToken); await taxonomies.AddAsync(taxonomy, cancellationToken); } catch (AggregateConflictException conflict) diff --git a/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/IntegrationEvents/InitializeContentTypeDefaultHandler.cs b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/IntegrationEvents/InitializeContentTypeDefaultHandler.cs new file mode 100644 index 00000000..a21b6148 --- /dev/null +++ b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/IntegrationEvents/InitializeContentTypeDefaultHandler.cs @@ -0,0 +1,19 @@ +using LearnStack.Modules.Customization.Application.Contracts.Customization; +using LearnStack.Modules.Tenancy.Application.Contracts.IntegrationEvents; +using LearnStack.SharedKernel.Messaging; +using MediatR; + +namespace LearnStack.Modules.Customization.Application.IntegrationEvents; + +/// Coordinates the content-type effect inside the transport-owned transaction. +public sealed class InitializeContentTypeDefaultHandler(ISender sender) + : IIntegrationEventHandler +{ + public async Task HandleAsync(TenantCreatedIntegrationEventV1 @event, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(@event); + var result = await sender.Send(new InitializeTenantContentTypeDefaultCommand(), cancellationToken); + if (result.IsFailure) + throw new InvalidOperationException("Content type default initialization was refused."); + } +} diff --git a/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/IntegrationEvents/InitializeTaxonomyDefaultHandler.cs b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/IntegrationEvents/InitializeTaxonomyDefaultHandler.cs new file mode 100644 index 00000000..cac83e35 --- /dev/null +++ b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/IntegrationEvents/InitializeTaxonomyDefaultHandler.cs @@ -0,0 +1,19 @@ +using LearnStack.Modules.Customization.Application.Contracts.Customization; +using LearnStack.Modules.Tenancy.Application.Contracts.IntegrationEvents; +using LearnStack.SharedKernel.Messaging; +using MediatR; + +namespace LearnStack.Modules.Customization.Application.IntegrationEvents; + +/// Coordinates the taxonomy effect inside the transport-owned transaction. +public sealed class InitializeTaxonomyDefaultHandler(ISender sender) + : IIntegrationEventHandler +{ + public async Task HandleAsync(TenantCreatedIntegrationEventV1 @event, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(@event); + var result = await sender.Send(new InitializeTenantLevelTaxonomyDefaultCommand(), cancellationToken); + if (result.IsFailure) + throw new InvalidOperationException("Taxonomy default initialization was refused."); + } +} diff --git a/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/LearnStack.Modules.Customization.Application.csproj b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/LearnStack.Modules.Customization.Application.csproj index b8600772..f10889b1 100644 --- a/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/LearnStack.Modules.Customization.Application.csproj +++ b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Application/LearnStack.Modules.Customization.Application.csproj @@ -8,6 +8,7 @@ + diff --git a/backend/src/Modules/Customization/LearnStack.Modules.Customization.Infrastructure/Persistence/CustomizationConceptHistory.cs b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Infrastructure/Persistence/CustomizationConceptHistory.cs new file mode 100644 index 00000000..f2191510 --- /dev/null +++ b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Infrastructure/Persistence/CustomizationConceptHistory.cs @@ -0,0 +1,57 @@ +using System.Buffers.Binary; +using System.Security.Cryptography; +using System.Text; +using LearnStack.Modules.Customization.Domain; +using LearnStack.SharedKernel.Identifiers; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Storage; +using Npgsql; + +namespace LearnStack.Modules.Customization.Infrastructure.Persistence; + +/// Tenant-protected coordination and history for the two versioned concepts. +internal static class CustomizationConceptHistory +{ + internal static async Task LockAsync( + CustomizationDbContext db, bool taxonomy, string key, CancellationToken cancellationToken) + { + var (tenant, connection, transaction) = RequireContext(db, key); + // A stable namespace includes tenant and kind, but deliberately no revision. + // A hash collision only serializes unrelated writers; it cannot grant access. + var identity = $"customization:{tenant.Value:D}:{(taxonomy ? "taxonomy" : "content-type")}:{key}"; + var digest = SHA256.HashData(Encoding.UTF8.GetBytes(identity)); + var lockKey = BinaryPrimitives.ReadInt64BigEndian(digest); + await using var command = new NpgsqlCommand( + "SELECT pg_advisory_xact_lock(@lock_key)", connection, transaction); + command.Parameters.AddWithValue("lock_key", lockKey); + await command.ExecuteNonQueryAsync(cancellationToken); + } + + internal static async Task ExistsAsync( + CustomizationDbContext db, bool taxonomy, string key, CancellationToken cancellationToken) + { + var (tenant, connection, transaction) = RequireContext(db, key); + // This owner-local query includes tombstones without disabling EF tenant filters. + // RLS remains active, and both tenant and semantic key are explicit parameters. + var sql = taxonomy + ? "SELECT EXISTS (SELECT 1 FROM tenant_level_taxonomies WHERE tenant_id = @tenant AND key = @key)" + : "SELECT EXISTS (SELECT 1 FROM tenant_content_types WHERE tenant_id = @tenant AND key = @key)"; + await using var command = new NpgsqlCommand(sql, connection, transaction); + command.Parameters.AddWithValue("tenant", tenant.Value); + command.Parameters.AddWithValue("key", key); + return (bool)(await command.ExecuteScalarAsync(cancellationToken))!; + } + + private static (TenantId Tenant, NpgsqlConnection Connection, NpgsqlTransaction Transaction) RequireContext( + CustomizationDbContext db, string key) + { + CustomizationKey.EnsureValid(key, nameof(key)); + var tenant = db.CurrentTenantId; + if (!tenant.IsInitialized() || tenant.Value == Guid.Empty || tenant == TenantId.PlatformSentinel) + throw new InvalidOperationException("Customization concept access requires a resolved tenant."); + if (db.Database.CurrentTransaction?.GetDbTransaction() is not NpgsqlTransaction transaction + || transaction.Connection is not { } connection) + throw new InvalidOperationException("Customization concept access requires an active transaction."); + return (tenant, connection, transaction); + } +} diff --git a/backend/src/Modules/Customization/LearnStack.Modules.Customization.Infrastructure/Persistence/CustomizationWriteStores.cs b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Infrastructure/Persistence/CustomizationWriteStores.cs index 80140565..e16ea9aa 100644 --- a/backend/src/Modules/Customization/LearnStack.Modules.Customization.Infrastructure/Persistence/CustomizationWriteStores.cs +++ b/backend/src/Modules/Customization/LearnStack.Modules.Customization.Infrastructure/Persistence/CustomizationWriteStores.cs @@ -13,7 +13,7 @@ namespace LearnStack.Modules.Customization.Infrastructure.Persistence; /// /// /// -/// Each method saves. The two aggregates carry no EF relationship to each +/// Each write method saves. The two aggregates carry no EF relationship to each /// other, so batching into one SaveChanges leaves the order EF sends them /// unspecified — and publishing depends on it: the incumbent's deprecation has to /// reach PostgreSQL before the successor's publication, or the partial index @@ -41,6 +41,12 @@ namespace LearnStack.Modules.Customization.Infrastructure.Persistence; /// public sealed class TenantContentTypeStore(CustomizationDbContext db, CustomizationReadState reads) : ITenantContentTypeStore { + public Task LockConceptAsync(string key, CancellationToken cancellationToken = default) => + CustomizationConceptHistory.LockAsync(db, taxonomy: false, key, cancellationToken); + + public Task HasConceptHistoryAsync(string key, CancellationToken cancellationToken = default) => + CustomizationConceptHistory.ExistsAsync(db, taxonomy: false, key, cancellationToken); + public Task AddAsync( TenantContentType aggregate, CancellationToken cancellationToken = default) { @@ -81,6 +87,12 @@ public Task UpdateAsync( /// public sealed class TenantLevelTaxonomyStore(CustomizationDbContext db, CustomizationReadState reads) : ITenantLevelTaxonomyStore { + public Task LockConceptAsync(string key, CancellationToken cancellationToken = default) => + CustomizationConceptHistory.LockAsync(db, taxonomy: true, key, cancellationToken); + + public Task HasConceptHistoryAsync(string key, CancellationToken cancellationToken = default) => + CustomizationConceptHistory.ExistsAsync(db, taxonomy: true, key, cancellationToken); + public Task AddAsync( TenantLevelTaxonomy aggregate, CancellationToken cancellationToken = default) { diff --git a/backend/src/Modules/Tenancy/LearnStack.Modules.Tenancy.Application/Audit/TenancyAuditCatalogSource.cs b/backend/src/Modules/Tenancy/LearnStack.Modules.Tenancy.Application/Audit/TenancyAuditCatalogSource.cs index 2d0df4f0..6583919f 100644 --- a/backend/src/Modules/Tenancy/LearnStack.Modules.Tenancy.Application/Audit/TenancyAuditCatalogSource.cs +++ b/backend/src/Modules/Tenancy/LearnStack.Modules.Tenancy.Application/Audit/TenancyAuditCatalogSource.cs @@ -80,6 +80,8 @@ public void Describe(IAuditCatalogBuilder builder) "platform.admin_scope.enter", OperationType.SecurityEvent, OperationClass.Must) .DeclareOffPath( "platform.outbox.dead_letter", OperationType.SecurityEvent, OperationClass.Must) + .DeclareOffPath( + "platform.delivery.dead_letter", OperationType.SecurityEvent, OperationClass.Must) .DeclareOffPath( "tenancy.tenant_assertion.reject", OperationType.SecurityEvent, OperationClass.Must) .DeclareOffPath( diff --git a/backend/tests/LearnStack.Tests.Architecture/AggregateWriteTests.cs b/backend/tests/LearnStack.Tests.Architecture/AggregateWriteTests.cs index d084c5da..5e348b59 100644 --- a/backend/tests/LearnStack.Tests.Architecture/AggregateWriteTests.cs +++ b/backend/tests/LearnStack.Tests.Architecture/AggregateWriteTests.cs @@ -1,6 +1,8 @@ using System.Reflection; using FluentAssertions; using LearnStack.SharedKernel.Persistence; +using LearnStack.SharedKernel.Messaging; +using LearnStack.Modules.Tenancy.Application.Contracts.IntegrationEvents; using LearnStack.SharedKernel.Identifiers; using LearnStack.Modules.Education.Application.Abstractions; using LearnStack.Modules.Education.Domain; @@ -22,6 +24,14 @@ namespace LearnStack.Tests.Architecture; /// public sealed class AggregateWriteTests { + private static readonly Type TenantProvisioningHandler = Assembly.Load("LearnStack.Modules.Tenancy.Application") + .GetType("LearnStack.Modules.Tenancy.Application.Tenant.ProvisionTenantCommandHandler", throwOnError: true)!; + + /// + /// ADR-0042 + /// and ADR-0057 + /// permit only provisioning to hold constructor write ports for two roots. + /// [Fact] public void Cross_Aggregate_Writes_Are_Confined_To_Tenant_Provisioning() { @@ -36,16 +46,9 @@ public void Cross_Aggregate_Writes_Are_Confined_To_Tenant_Provisioning() // forbidden, so no handler can name a DbSet at all. A rule at Implemented status // that cannot fire is worse than one at Registered, because the catalogue then // claims coverage it does not have. - var offenders = CrossAggregateWriters(ProductionAssemblies() + AssertOnlyTenantProvisioningWritesAcrossAggregates(ProductionAssemblies() .Select(Assembly.Load) - .SelectMany(assembly => assembly.GetTypes())) - .Select(type => type.Name).Distinct().ToList(); - - offenders.Should().BeEquivalentTo( - ["ProvisionTenantCommandHandler"], - "a handler that can write two aggregate roots writes across an aggregate " - + "boundary in one transaction, which ADR-0042 sanctions for exactly one " - + "operation — a second name here is a decision that needs its own record"); + .SelectMany(assembly => assembly.GetTypes())); } [Fact] @@ -120,6 +123,35 @@ public void Cross_Aggregate_Census_Catches_Fused_Separate_Notification_And_Inter "all message shapes and non-public constructors must count every reachable root; Fix: preserve the shared production census"); } + /// + /// ADR-0057 + /// applies the constructor write-port census to integration handlers, including + /// inherited contracts and non-public constructors. + /// + [Fact] + public void Cross_Aggregate_Census_Catches_Integration_Handlers_And_Inherited_Contracts() + { + Type[] forbidden = [typeof(FusedIntegrationHandler), typeof(TwoPortIntegrationHandler), typeof(InheritedIntegrationHandler), typeof(ProvisionTenantCommandHandler)]; + CrossAggregateWriters(forbidden.Concat([typeof(SameRootIntegrationHandler), typeof(NonHandler)])) + .Should().BeEquivalentTo(forbidden, + "integration handlers count distinct roots through every constructor port; " + + "Fix: retain fused, separate and inherited contracts in the production census"); + + Action sameNameOffender = () => AssertOnlyTenantProvisioningWritesAcrossAggregates( + [TenantProvisioningHandler, typeof(ProvisionTenantCommandHandler)]); + sameNameOffender.Should().Throw( + "the provisioning exception belongs to one exact type, not every handler with its name; Fix: compare Type identities before applying the exception"); + AssertOnlyTenantProvisioningWritesAcrossAggregates([TenantProvisioningHandler]); + } + + private static void AssertOnlyTenantProvisioningWritesAcrossAggregates(IEnumerable types) => + CrossAggregateWriters(types).Should().Equal( + [TenantProvisioningHandler], + "a handler that can write two aggregate roots writes across an aggregate " + + "boundary in one transaction, which ADR-0042 sanctions for exactly one " + + "operation. Fix: split integration effects into one-root subscriptions; " + + "any new cross-aggregate exception needs its own decision record"); + private static IEnumerable CrossAggregateWriters(IEnumerable types) => types .Where(type => type is { IsAbstract: false, IsInterface: false }) .Where(IsMessageHandler) @@ -183,6 +215,33 @@ public SameRootHandler(ICourseWriteStore first, ICourseWriteStore second) { } } private sealed class NonHandler { public NonHandler(IFusedWritePort store) { } } + private sealed class FusedIntegrationHandler : IIntegrationEventHandler + { + internal FusedIntegrationHandler(IFusedWritePort store) { } + public Task HandleAsync(TenantCreatedIntegrationEventV1 @event, CancellationToken cancellationToken = default) => Task.CompletedTask; + } + private sealed class TwoPortIntegrationHandler : IIntegrationEventHandler + { + public TwoPortIntegrationHandler(ICourseWriteStore courses, ILessonWriteStore lessons) { } + public Task HandleAsync(TenantCreatedIntegrationEventV1 @event, CancellationToken cancellationToken = default) => Task.CompletedTask; + } + private sealed class ProvisionTenantCommandHandler : IIntegrationEventHandler + { + public ProvisionTenantCommandHandler(ICourseWriteStore courses, ILessonWriteStore lessons) { } + public Task HandleAsync(TenantCreatedIntegrationEventV1 @event, CancellationToken cancellationToken = default) => Task.CompletedTask; + } + private interface IInheritedIntegrationContract : IIntegrationEventHandler; + private sealed class InheritedIntegrationHandler : IInheritedIntegrationContract + { + private InheritedIntegrationHandler(ICourseWriteStore courses, ILessonWriteStore lessons) { } + public Task HandleAsync(TenantCreatedIntegrationEventV1 @event, CancellationToken cancellationToken = default) => Task.CompletedTask; + } + private sealed class SameRootIntegrationHandler : IIntegrationEventHandler + { + public SameRootIntegrationHandler(ICourseWriteStore first, ICourseWriteStore second) { } + public Task HandleAsync(TenantCreatedIntegrationEventV1 @event, CancellationToken cancellationToken = default) => Task.CompletedTask; + } + private interface IWrappedKeyReader { void Read(IEnumerable ids); } private interface IDirectWriter { void Apply(Course root); } private interface IBulkWriter { void Apply(IEnumerable roots); } @@ -223,7 +282,7 @@ private static IEnumerable WriteStoreConstructions(Type parameterType) => && candidate.GetGenericTypeDefinition() == typeof(IAggregateWriteStore<,>)); /// - /// Whether the type handles a message on the MediatR pipeline. + /// Whether the type handles a MediatR message or an integration event. /// /// /// INotificationHandler is in the set deliberately. An intra-module domain @@ -234,6 +293,12 @@ private static IEnumerable WriteStoreConstructions(Type parameterType) => /// because a domain-event handler does not look like a write boundary. Measured: with /// only IRequestHandler<,> in the set, one dropped into a production /// assembly passed all 76 architecture cases. + /// + /// Integration handlers run in the transport-owned transaction under ADR-0057. + /// This counts constructor write ports; it does not trace arbitrary nested + /// ISender.Send dataflow. The real default coordinators hold only + /// ISender, and Step 5's runtime proofs establish their one-effect behavior. + /// /// private static bool IsMessageHandler(Type type) => type.GetInterfaces().Any(contract => @@ -245,6 +310,7 @@ private static bool IsMessageHandler(Type type) => typeof(IRequestHandler<,>), typeof(IRequestHandler<>), typeof(INotificationHandler<>), + typeof(IIntegrationEventHandler<>), ]; /// Every production assembly, by name, from the project files on disk. diff --git a/backend/tests/LearnStack.Tests.Architecture/AuditCatalogDiscovery.cs b/backend/tests/LearnStack.Tests.Architecture/AuditCatalogDiscovery.cs index c4cd59a1..0a033195 100644 --- a/backend/tests/LearnStack.Tests.Architecture/AuditCatalogDiscovery.cs +++ b/backend/tests/LearnStack.Tests.Architecture/AuditCatalogDiscovery.cs @@ -1,4 +1,3 @@ -using System.Reflection; using LearnStack.Infrastructure.Audit; using LearnStack.SharedKernel.Audit; @@ -33,23 +32,10 @@ public static AuditCatalog Catalogue() => .ToList()); /// Every type every project under backend/src compiles. + /// + /// Assembly or type-load failures fail the census rather than silently shrinking + /// its population to the loadable subset. + /// public static IEnumerable Types() => - Directory - .EnumerateFiles(RepositoryPaths.BackendSrc(), "*.csproj", SearchOption.AllDirectories) - .Where(path => !path.Contains( - $"{Path.DirectorySeparatorChar}obj{Path.DirectorySeparatorChar}", StringComparison.Ordinal)) - .Select(path => Assembly.Load(Path.GetFileNameWithoutExtension(path))) - .SelectMany(LoadableTypes); - - private static IEnumerable LoadableTypes(Assembly assembly) - { - try - { - return assembly.GetTypes(); - } - catch (ReflectionTypeLoadException partial) - { - return partial.Types.OfType(); - } - } + ProductionAssemblies.All().SelectMany(assembly => assembly.GetTypes()); } diff --git a/backend/tests/LearnStack.Tests.Architecture/AuditCoverageIntegrationTests.cs b/backend/tests/LearnStack.Tests.Architecture/AuditCoverageIntegrationTests.cs new file mode 100644 index 00000000..58e39379 --- /dev/null +++ b/backend/tests/LearnStack.Tests.Architecture/AuditCoverageIntegrationTests.cs @@ -0,0 +1,137 @@ +using FluentAssertions; +using LearnStack.Infrastructure.Audit; +using LearnStack.Modules.Tenancy.Application.Contracts.IntegrationEvents; +using LearnStack.SharedKernel.Audit; +using LearnStack.SharedKernel.Messaging; +using MediatR; +using Xunit; + +namespace LearnStack.Tests.Architecture +{ + public sealed partial class AuditCoverageTests + { + /// + /// ADR-0057 + /// classifies each concrete integration handler, even when its event is shared. + /// + [Fact] + public void Integration_Audit_Census_Catches_An_Unregistered_Handler_For_A_Shared_Event() + { + var subjects = AuditSubjects(IntegrationProbeTypes()); + var catalogue = new AuditCatalog([new FirstIntegrationHandlerSource()]); + + subjects.Should().BeEquivalentTo( + [typeof(FirstIntegrationAuditHandler), typeof(InheritedIntegrationAuditHandler), typeof(CreateOrganizationCommandProbe)], + "the census must keep concrete handlers and MediatR requests, including inherited contracts; Fix: classify handlers separately from their events"); + UnregisteredOf(subjects, catalogue).Should().Equal([typeof(InheritedIntegrationAuditHandler)], + "one registered consumer cannot cover another consumer of the same event; Fix: register each concrete handler"); + } + + /// + /// ADR-0057 + /// keys delivery audit classification on handler identity rather than event identity. + /// + [Fact] + public void Integration_Audit_Census_Rejects_Event_Only_Classification() + { + var catalogue = new AuditCatalog([new EventOnlyAuditSource()]); + var subjects = AuditSubjects([typeof(FirstIntegrationAuditHandler), typeof(InheritedIntegrationAuditHandler)]); + + UnregisteredOf(subjects, catalogue).Should().BeEquivalentTo(subjects, + "classifying the event leaves both handlers unclassified; Fix: register concrete handler types in the owning audit source"); + } + + /// + /// ADR-0057 + /// permits explicit Off coordinators whose nested commands declare business intents. + /// + [Fact] + public void Integration_Audit_Census_Accepts_Explicit_Off_For_Both_Handlers() + { + var catalogue = new AuditCatalog([new BothIntegrationHandlersSource()]); + + UnregisteredOf(AuditSubjects(IntegrationProbeTypes()), catalogue).Should().BeEmpty( + "explicit Off is a real classification and nested commands own their MUST intents; Fix: distinguish Off registration from missing registration"); + } + + /// + /// ADR-0044 + /// and ADR-0057 + /// require a matrix even when a module ships only an integration handler. + /// + [Fact] + public void Integration_Only_Module_Requires_An_Audit_Matrix() + { + Type[] types = [typeof(LearnStack.Modules.Ghost.Application.IntegrationOnlyAuditHandler)]; + var owning = ModulesWithCode(AuditSubjects(types), types); + + owning.Should().Equal(["Ghost"], + "a module with an integration handler is visible without an aggregate or MediatR request; Fix: use all audit subjects for module discovery"); + ModulesWithoutMatrix(owning, ModuleSpecs()).Should().Equal(["Ghost"], + "an integration-only module cannot avoid the matrix census; Fix: add its module audit.md"); + } + + private static Type[] IntegrationProbeTypes() => + [ + typeof(FirstIntegrationAuditHandler), typeof(InheritedIntegrationAuditHandler), + typeof(IInheritedIntegrationAuditContract), typeof(AbstractIntegrationAuditHandler), + typeof(TenantCreatedIntegrationEventV1), typeof(IntegrationAuditRequestHandler), + typeof(IntegrationAuditNotificationHandler), typeof(string), + ]; + + private interface IInheritedIntegrationAuditContract : IIntegrationEventHandler; + + private sealed class FirstIntegrationAuditHandler : IIntegrationEventHandler + { + internal FirstIntegrationAuditHandler() { } + public Task HandleAsync(TenantCreatedIntegrationEventV1 @event, CancellationToken cancellationToken = default) => Task.CompletedTask; + } + + private abstract class AbstractIntegrationAuditHandler : IInheritedIntegrationAuditContract + { + public Task HandleAsync(TenantCreatedIntegrationEventV1 @event, CancellationToken cancellationToken = default) => Task.CompletedTask; + } + + private sealed class InheritedIntegrationAuditHandler : AbstractIntegrationAuditHandler; + + private sealed record CreateOrganizationCommandProbe : IRequest; + private sealed class IntegrationAuditRequestHandler : IRequestHandler + { + public Task Handle(CreateOrganizationCommandProbe request, CancellationToken cancellationToken) => Task.CompletedTask; + } + + private sealed record IntegrationAuditNotification : INotification; + private sealed class IntegrationAuditNotificationHandler : INotificationHandler + { + public Task Handle(IntegrationAuditNotification notification, CancellationToken cancellationToken) => Task.CompletedTask; + } + + private sealed class FirstIntegrationHandlerSource : IAuditCatalogSource + { + public string ModuleName => "customization"; + public void Describe(IAuditCatalogBuilder builder) => builder.Off().Off(); + } + + private sealed class EventOnlyAuditSource : IAuditCatalogSource + { + public string ModuleName => "customization"; + public void Describe(IAuditCatalogBuilder builder) => builder.Off(); + } + + private sealed class BothIntegrationHandlersSource : IAuditCatalogSource + { + public string ModuleName => "customization"; + public void Describe(IAuditCatalogBuilder builder) => + builder.Off().Off().Off(); + } + } +} + +namespace LearnStack.Modules.Ghost.Application +{ + /// A handler-only module without a matrix; never part of the production census. + internal sealed class IntegrationOnlyAuditHandler : IIntegrationEventHandler + { + public Task HandleAsync(TenantCreatedIntegrationEventV1 @event, CancellationToken cancellationToken = default) => Task.CompletedTask; + } +} diff --git a/backend/tests/LearnStack.Tests.Architecture/AuditCoverageTests.cs b/backend/tests/LearnStack.Tests.Architecture/AuditCoverageTests.cs index 07bbb2de..27401cb2 100644 --- a/backend/tests/LearnStack.Tests.Architecture/AuditCoverageTests.cs +++ b/backend/tests/LearnStack.Tests.Architecture/AuditCoverageTests.cs @@ -1,6 +1,8 @@ using System.Text.RegularExpressions; using FluentAssertions; using LearnStack.Infrastructure.Audit; +using LearnStack.Modules.Customization.Application.IntegrationEvents; +using LearnStack.SharedKernel.Messaging; using LearnStack.Modules.Tenancy.Application.Contracts.Tenant; using LearnStack.Modules.Tenancy.Domain; using LearnStack.SharedKernel.Audit; @@ -12,7 +14,7 @@ namespace LearnStack.Tests.Architecture; /// /// The catalogue in code and the matrix in prose have to agree — and every request the -/// backend ships has to be in the catalogue at all. +/// backend ships and every concrete integration handler has to be in the catalogue at all. /// /// /// @@ -33,25 +35,32 @@ namespace LearnStack.Tests.Architecture; public sealed partial class AuditCoverageTests { /// - /// Every request type the backend can dispatch — one with a handler — is registered - /// in the catalogue, Off included. + /// Every MediatR request type and concrete integration handler the backend can + /// dispatch is registered, Off included, under + /// ADR-0044 and + /// ADR-0057. /// [Fact] public void Every_Shipped_Request_Is_Registered() { // The rule AuditLogBehavior enforces at run time — an unregistered request is // refused with audit_unclassified_operation — checked where a defect costs a red - // build instead of a 500 per call. Keyed on the REQUEST TYPE, which is what the - // behavior looks up; a slug-level comparison cannot see a request whose slug some - // other request also registers. - var shipped = ShippedRequestTypes(); + // build instead of a 500 per call. MediatR classifies request types and delivery + // classifies concrete handler types before construction. A slug or event-level + // comparison cannot distinguish multiple consumers of the same event. + var shipped = AuditSubjects(AuditCatalogDiscovery.Types()); shipped.Should().Contain(typeof(CreateOrganizationCommand), "a discovery that missed a shipped command would pass while proving nothing"); + shipped.Should().Contain([typeof(InitializeContentTypeDefaultHandler), typeof(InitializeTaxonomyDefaultHandler)], + "the production census must include both actual integration coordinators; Fix: scan concrete integration handlers in every production assembly"); + + // The defaults are legitimately Off here: nested commands declare their MUST + // intents. This census requires classification, not a duplicate business row. UnregisteredOf(shipped, Catalogue()).Should().BeEmpty( - "every request reaching pipeline step 3 must be classified, silence included, " - + "or the running system refuses it"); + "every request and integration handler must be classified, silence included. " + + "Fix: register the concrete handler type, not its event, in the owning audit source"); } [Fact] @@ -305,8 +314,9 @@ public void The_Forward_Sweep_Reads_Only_The_Registering_Module_s_Matrix() } /// - /// A module that ships an aggregate or a request type has a coverage matrix to classify - /// them in. + /// A module shipping an aggregate, a request or an integration handler owes a matrix + /// under ADR-0044 + /// and ADR-0057. /// [Fact] public void Every_Module_With_An_Aggregate_Or_A_Request_Has_A_Matrix() @@ -318,14 +328,15 @@ public void Every_Module_With_An_Aggregate_Or_A_Request_Has_A_Matrix() // which the matrix join has nothing to compare against and passes. Aggregates are // half of it: the review of Packet 9 added one to a scaffold module's Domain, with no // handler and no matrix, and the request-only version of this rule stayed green. - var owning = ModulesWithCode(ShippedRequestTypes(), AuditCatalogDiscovery.Types()); + var productionTypes = AuditCatalogDiscovery.Types().ToArray(); + var owning = ModulesWithCode(AuditSubjects(productionTypes), productionTypes); owning.Should().Contain(["Tenancy", "Customization", "Audit"], "a discovery that found no module would pass while proving nothing — and Audit " + "ships aggregates and no request, which is exactly the half being checked"); ModulesWithoutMatrix(owning, ModuleSpecs()).Should().BeEmpty( - "a module that ships an aggregate or a command owes a matrix that classifies it"); + "a module that ships an aggregate, request or integration handler owes a matrix. Fix: add docs/modules//audit.md"); } [Fact] @@ -339,9 +350,9 @@ public void The_Module_Sweep_Can_Actually_Fail() ModulesWithoutMatrix(["Tenancy", .. owning], ModuleSpecs()).Should().Equal("Ghost"); } - /// The request types among the catalogue does not know. + /// The request or handler subjects among the catalogue does not know. private static List UnregisteredOf(IEnumerable shipped, AuditCatalog catalog) => - [.. shipped.Where(request => !catalog.TryGet(request, out _))]; + [.. shipped.Where(subject => !catalog.TryGet(subject, out _))]; /// The catalogue → matrix direction's findings, one line each. /// @@ -457,9 +468,9 @@ private static IEnumerable RowProblems(AuditCatalogEntry entry, string r /// private const string PlatformSegment = "platform"; - /// The modules that own a request type or an aggregate root, by name. - private static List ModulesWithCode(IEnumerable requests, IEnumerable types) => - [.. requests + /// The modules that own an audit subject or an aggregate root, by name. + private static List ModulesWithCode(IEnumerable subjects, IEnumerable types) => + [.. subjects .Concat(types.Where(type => type is { IsAbstract: false, IsInterface: false } && type.GetInterfaces().Any(contract => contract.IsGenericType && contract.GetGenericTypeDefinition() == typeof(IAggregateRoot<>)))) @@ -536,31 +547,36 @@ [.. shipping.Where(module => !File.Exists(Path.Combine(specs, module.ToLowerInvariant(), "audit.md")))]; /// - /// Every request type with a handler in any backend assembly. + /// The identities the audit coordinator classifies: MediatR request types and + /// concrete integration handler types, including inherited contracts. /// /// - /// A request with a handler is one the pipeline can be asked to run, which is the - /// population AuditLogBehavior classifies. The assemblies are every project under - /// backend/src, loaded by name — so a new module is swept the moment it builds, - /// without a line here. + /// ADR-0057 classifies the handler before construction. Two consumers of the same + /// event therefore remain separate subjects. Notification handlers do not own a + /// business audit frame and are excluded. Production callers pass every type from + /// every backend assembly; planted controls use this same predicate. /// - private static List ShippedRequestTypes() => - [.. AuditCatalogDiscovery.Types() + private static List AuditSubjects(IEnumerable types) => + [.. types .Where(type => type is { IsAbstract: false, IsInterface: false }) - .SelectMany(type => type.GetInterfaces()) - .Where(contract => contract.IsGenericType - && (contract.GetGenericTypeDefinition() == typeof(IRequestHandler<,>) - || contract.GetGenericTypeDefinition() == typeof(IRequestHandler<>))) - .Select(contract => contract.GetGenericArguments()[0]) - .Distinct()]; + .SelectMany(type => type.GetInterfaces() + .Where(contract => contract.IsGenericType + && (contract.GetGenericTypeDefinition() == typeof(IIntegrationEventHandler<>) + || contract.GetGenericTypeDefinition() == typeof(IRequestHandler<,>) + || contract.GetGenericTypeDefinition() == typeof(IRequestHandler<>))) + .Select(contract => contract.GetGenericTypeDefinition() == typeof(IIntegrationEventHandler<>) + ? type + : contract.GetGenericArguments()[0])) + .Distinct() + .OrderBy(type => type.FullName, StringComparer.Ordinal)]; /// The catalogue the composition roots build — see . private static AuditCatalog Catalogue() => AuditCatalogDiscovery.Catalogue(); - /// The module a request type belongs to, from its namespace, or null. - private static string? ModuleOf(Type request) + /// The module a type belongs to, from its namespace, or null. + private static string? ModuleOf(Type type) { - var match = ModuleNamespace().Match(request.Namespace ?? string.Empty); + var match = ModuleNamespace().Match(type.Namespace ?? string.Empty); return match.Success ? match.Groups[1].Value : null; } diff --git a/backend/tests/LearnStack.Tests.Architecture/CorpusConsistencyTests.cs b/backend/tests/LearnStack.Tests.Architecture/CorpusConsistencyTests.cs index 168c3261..80c7a0e8 100644 --- a/backend/tests/LearnStack.Tests.Architecture/CorpusConsistencyTests.cs +++ b/backend/tests/LearnStack.Tests.Architecture/CorpusConsistencyTests.cs @@ -203,6 +203,10 @@ public void The_Corpus_Guards_Can_Actually_Fail() - **Status:** **Implemented** (Packet 6 step 4, `SomeTests`). + ### `A_Rule_Implemented_Twice` + + - **Status:** **Implemented**, twice. Both paths run in `SomeTests`. + ### `A_Rule_That_Does_Not` - **Status:** **Registered.** @@ -211,12 +215,19 @@ public void The_Corpus_Guards_Can_Actually_Fail() - **Status:** **Implemented** (`LearnStack.Tests.Integration`, `OtherTests`). + ### `A_Registered_Rule_With_Implemented_Prose` + + - **Asserts:** a companion is **Implemented** in `SomeTests`. + - **Status:** **Registered.** A future test may become **Implemented**. + - **Phase:** later; the **Implemented** companion is `SomeTests`. + ### `An-Analyzer-Rule` - **Status:** **Implemented** — `SomeTests.cs`. """).Should().Equal([ "A_Rule_That_Runs", "A_Rule_Whose_Entry_Names_The_Class", + "A_Rule_Implemented_Twice", "An-Analyzer-Rule", ], "both third- and fourth-level rule headings count; a hyphen in a rule name " + "is a rule name: excluding it dropped the `LS0001` analyzer's entry from " @@ -286,9 +297,14 @@ [.. StatusRow().Matches(readme) /// Every catalogue entry whose Status reports Implemented, wherever it runs. private static List<(string Rule, string Status)> ImplementedEntries() => - [.. CatalogueEntry().Matches(Catalogue()) - .Where(entry => entry.Groups["status"].Value.Contains("**Implemented**", StringComparison.Ordinal)) - .Select(entry => (entry.Groups["rule"].Value, entry.Groups["status"].Value))]; + ImplementedEntriesIn(Catalogue()); + + private static List<(string Rule, string Status)> ImplementedEntriesIn(string catalogue) => + [.. CatalogueEntry().Matches(catalogue) + .Select(entry => (Rule: entry.Groups["rule"].Value, + Status: CatalogueStatus().Match(entry.Groups["status"].Value))) + .Where(entry => entry.Status.Success && entry.Status.Groups["state"].Value == "Implemented") + .Select(entry => (entry.Rule, entry.Status.Value))]; /// The test classes this assembly actually carries. /// @@ -324,11 +340,10 @@ [.. Directory.EnumerateFiles(TestsRoot, "*Tests.cs", SearchOption.AllDirectories /// /// private static List ImplementedIn(string catalogue, IReadOnlyCollection? classes = null) => - [.. CatalogueEntry().Matches(catalogue) - .Where(entry => entry.Groups["status"].Value.Contains("**Implemented**", StringComparison.Ordinal)) - .Where(entry => NamedClasses(entry.Groups["status"].Value) + [.. ImplementedEntriesIn(catalogue) + .Where(entry => NamedClasses(entry.Status) .Any(name => (classes ?? ["SomeTests"]).Contains(name))) - .Select(entry => entry.Groups["rule"].Value)]; + .Select(entry => entry.Rule)]; /// The test classes a Status line names, with or without the file extension. private static List NamedClasses(string status) => @@ -428,6 +443,12 @@ private static Dictionary> DeclaredMethods() => [GeneratedRegex(@"^#{3,4} `(?[A-Za-z0-9_\-]+)`(?.*?)(?=\n#{2,4} |\z)", RegexOptions.Singleline | RegexOptions.Multiline)] private static partial Regex CatalogueEntry(); + // Read the declaration's first bold state, not an Implemented mention in its + // assertion, history, later fields or the explanation of a Registered status. + // Continuation lines still carry the test-class references the census resolves. + [GeneratedRegex(@"^- \*\*Status:\*\* \*\*(?[^*\r\n]+)\*\*[^\r\n]*(?:\r?\n(?!- \*\*|#{1,6} )[^\r\n]*)*", RegexOptions.Multiline)] + private static partial Regex CatalogueStatus(); + // Three spellings, because the catalogue writes all three: the bare class, the file with // its extension, and `Class.Method` — which is how the entries outside this assembly name // their test. Reading only the first two left three shipped rules with nothing to resolve, diff --git a/backend/tests/LearnStack.Tests.Architecture/DeliveryDispatchBoundaryTests.cs b/backend/tests/LearnStack.Tests.Architecture/DeliveryDispatchBoundaryTests.cs new file mode 100644 index 00000000..2f26b58b --- /dev/null +++ b/backend/tests/LearnStack.Tests.Architecture/DeliveryDispatchBoundaryTests.cs @@ -0,0 +1,515 @@ +using System.Text.RegularExpressions; +using FluentAssertions; +using LearnStack.Api.Messaging; +using LearnStack.Infrastructure.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Diagnostics.HealthChecks; +using Microsoft.Extensions.Options; +using Mono.Cecil; +using Mono.Cecil.Cil; +using Xunit; + +namespace LearnStack.Tests.Architecture; + +public sealed partial class OutboxDispatchConventionTests +{ + private const string DeliveryObserver = "LearnStack.Api.Messaging.DeliveryObservation"; + private const string DeliveryHealth = "LearnStack.Api.Messaging.DeliveryHealthCheck"; + private const string ExecutionControl = "LearnStack.Infrastructure.Messaging.DeliveryExecutionControl"; + private const string ExecutionResultTask = "System.Threading.Tasks.Task`1"; + private static readonly string[] DeliverySubjects = + [ + DeliveryWorker, DeliveryAttempt, DeliveryObserver, DeliveryHealth, + "LearnStack.Api.Messaging.DeliveryObservationSnapshot", + ]; + + /// ADR-0046 and Accepted P02b-3: controlled execution and memory-only snapshot/health graphs. + [Fact] + public void Delivery_Execution_And_Observation_Are_Bounded() + { + using var census = ProductionCensus.Read(); + DeliveryExecutionOffenders(census).Should().BeEmpty( + "Fix: execute only through the task-owned attempt and controlled executor; preserve callback/scope ownership"); + DeliveryObservationOffenders(census).Should().BeEmpty( + "Fix: keep subscriber gauges and health memory-only through concrete, interface, inherited and async helpers"); + DeliveryRegistrationOffenders(ObservationServices()).Should().BeEmpty( + "Fix: retain the mandatory subscriber worker and health check bound to their singleton observation"); + } + + [Fact] + public void Delivery_execution_census_rejects_uncontrolled_direct_delegate_async_and_missing_paths() + { + using var census = ProductionCensus.Read(); + var module = census.Modules.Single(item => item.Assembly.Name.Name == DeliveryApplication); + var executor = census.Types[(InfrastructureAssembly, DeliveryExecutorType)]; + var control = census.Types[(InfrastructureAssembly, ExecutionControl)]; + foreach (var capability in executor.Methods.Where(method => method.Name is "ExecuteAsync" or "ExecuteControlledAsync") + .Concat(control.Methods.Where(method => method.Name is "get_Token" or "FinishCallbacksAsync" or "DisposeCancellation"))) + { + foreach (var opcode in new[] { OpCodes.Call, OpCodes.Callvirt, OpCodes.Ldftn, OpCodes.Ldvirtftn }) + AssertMutation(census, "an extra execution/lifetime capability through " + opcode.Name, + () => AddCallProbe(module, "ExecutionEscape", capability, opcode), DeliveryExecutionOffenders); + AssertMutation(census, "an asynchronous execution/lifetime escape", + () => AddAsyncCallProbe(module, "AsyncExecutionEscape", capability), DeliveryExecutionOffenders); + } + foreach (var subject in DeliverySubjects) + AssertMutation(census, "a missing real execution subject " + subject, () => + { + var type = census.Types[(BoundaryApi, subject)]; + var ownerModule = type.Module; + ownerModule.Types.Remove(type); + return () => ownerModule.Types.Add(type); + }, DeliveryExecutionOffenders); + var probe = AddProbe(module, "CleanExecutionObserver"); + AddCall(probe, "Observe", control.Properties.Single(property => property.Name == "Gate").GetMethod, OpCodes.Callvirt); + try { DeliveryExecutionOffenders(census).Should().BeEmpty("gate observation alone does not own cancellation lifetime"); } + finally { module.Types.Remove(probe); } + } + + [Fact] + public void Delivery_observation_walk_rejects_concrete_interface_inherited_async_and_callback_persistence() + { + using var census = ProductionCensus.Read(); + var module = census.Modules.Single(item => item.Assembly.Name.Name == DeliveryApplication); + foreach (var rootName in new[] { DeliveryObserver, DeliveryHealth }) + foreach (var capability in new[] { typeof(Npgsql.NpgsqlCommand), typeof(System.Data.Common.DbConnection), + typeof(IServiceProvider), typeof(IServiceScopeFactory) }) + AssertMutation(census, "a subscriber observation helper carrying " + capability.Name, () => + { + var root = census.Types[(BoundaryApi, rootName)]; + var helper = AddProbe(module, "DeliveryObservationHelper"); + helper.Fields.Add(new FieldDefinition("Capability", FieldAttributes.Public, module.ImportReference(capability))); + var field = new FieldDefinition("Probe", FieldAttributes.Private, root.Module.ImportReference(helper)); + root.Fields.Add(field); + return () => { root.Fields.Remove(field); module.Types.Remove(helper); }; + }, DeliveryObservationOffenders); + foreach (var isInterface in new[] { true, false }) + AssertMutation(census, "a polymorphic asynchronous subscriber gauge helper", () => + { + var root = census.Types[(BoundaryApi, DeliveryObserver)]; + var contract = AddProbe(module, "DeliveryObservationContract"); + if (isInterface) contract.Attributes |= TypeAttributes.Interface | TypeAttributes.Abstract; + var helper = AddProbe(module, "DeliveryObservationImplementation"); + if (isInterface) helper.Interfaces.Add(new InterfaceImplementation(contract)); + else helper.BaseType = contract; + AddAsyncCall(helper, "Read", module.ImportReference(typeof(IServiceScopeFactory).GetMethod(nameof(IServiceScopeFactory.CreateScope))!)); + var field = new FieldDefinition("Probe", FieldAttributes.Private, root.Module.ImportReference(contract)); + root.Fields.Add(field); + return () => { root.Fields.Remove(field); module.Types.Remove(helper); module.Types.Remove(contract); }; + }, DeliveryObservationOffenders); + foreach (var target in new[] + { + typeof(Type).GetMethod(nameof(Type.GetType), [typeof(string)])!, + typeof(Activator).GetMethod(nameof(Activator.CreateInstance), [typeof(Type)])!, + typeof(System.Reflection.MethodBase).GetMethod(nameof(System.Reflection.MethodBase.Invoke), [typeof(object), typeof(object[])])!, + }) + foreach (var opcode in new[] { OpCodes.Call, OpCodes.Ldftn }) + AssertMutation(census, "reflective construction/invocation in an asynchronous observation helper", () => + { + var root = census.Types[(BoundaryApi, DeliveryObserver)]; + var helper = AddProbe(module, "ReflectiveObservationHelper"); + AddAsyncCall(helper, "Read", module.ImportReference(target)); + AddCall(helper, "DelegateRead", module.ImportReference(target), opcode); + var field = new FieldDefinition("Probe", FieldAttributes.Private, root.Module.ImportReference(helper)); + root.Fields.Add(field); + return () => { root.Fields.Remove(field); module.Types.Remove(helper); }; + }, DeliveryObservationOffenders); + var observation = census.Types[(BoundaryApi, DeliveryObserver)]; + var callbacks = observation.Methods.SelectMany(DelegateTargets).Where(target => SameType(target.DeclaringType, observation)).ToArray(); + callbacks.Should().HaveCountGreaterThanOrEqualTo(4, "the actual production gauge callbacks must be nonempty"); + foreach (var callback in callbacks) + AssertMutation(census, "persistence inside an actual subscriber gauge callback", () => + { + var method = observation.Methods.Single(item => MethodId(item) == MethodId(callback)); + var instruction = Instruction.Create(OpCodes.Newobj, method.Module.ImportReference(typeof(Npgsql.NpgsqlCommand).GetConstructor(Type.EmptyTypes)!)); + method.Body.Instructions.Insert(0, instruction); + return () => method.Body.Instructions.Remove(instruction); + }, DeliveryObservationOffenders); + var clean = AddProbe(module, "CleanDeliveryMemory"); + clean.Fields.Add(new FieldDefinition("Count", FieldAttributes.Public, module.TypeSystem.Int32)); + AddCall(clean, "ReadDeclaredMetadata", module.ImportReference(typeof(Type).GetProperty(nameof(Type.Assembly))!.GetMethod!), OpCodes.Callvirt); + var cleanField = new FieldDefinition("Probe", FieldAttributes.Private, observation.Module.ImportReference(clean)); + observation.Fields.Add(cleanField); + try { DeliveryObservationOffenders(census).Should().BeEmpty("pure memory helper graphs remain admitted"); } + finally { observation.Fields.Remove(cleanField); module.Types.Remove(clean); } + } + + [Fact] + public void Delivery_credentials_reject_extra_store_locators_forwarding_overloads_and_wrong_keys() + { + using var census = ProductionCensus.Read(); + var module = census.Modules.Single(item => item.Assembly.Name.Name == DeliveryApplication); + var store = census.Types[(InfrastructureAssembly, DeliveryStore)]; + foreach (var capability in store.Methods.Where(method => method.Name is ".ctor" or "TryClaimAsync" or "ReplayAsync")) + foreach (var opcode in new[] { OpCodes.Call, OpCodes.Callvirt, OpCodes.Ldftn, OpCodes.Ldvirtftn }) + AssertMutation(census, "a subscriber privileged store escape through " + opcode.Name, + () => AddCallProbe(module, "DeliveryStoreEscape", capability, opcode), CredentialOffenders); + var locator = typeof(ServiceProviderServiceExtensions).GetMethods().Single(method => method.Name == "GetRequiredService" + && method.IsGenericMethodDefinition && method.GetParameters().Length == 1).MakeGenericMethod(typeof(DeliveryDispatchStore)); + AssertMutation(census, "an unkeyed subscriber store locator", () => + AddAsyncCallProbe(module, "DeliveryStoreLocator", module.ImportReference(locator)), CredentialOffenders); + var worker = census.Types[(BoundaryApi, DeliveryWorker)]; + foreach (var (type, name, field) in new[] { (worker, "StartAsync", "_source"), (worker, "DispatchAsync", "_store"), + (worker, "ClaimNextAsync", "_store"), + (store, "ReplayAsync", "_source"), (store, "AcquireAsync", "_source") }) + AssertMutation(census, "an extra privileged forwarding overload", () => AddBoundaryFieldGetter(type, name, field), CredentialOffenders); + var parameter = worker.Methods.Single(method => method.IsConstructor && !method.IsStatic).Parameters + .Single(item => item.CustomAttributes.Any(attribute => attribute.AttributeType.Name == "FromKeyedServicesAttribute")); + var attribute = parameter.CustomAttributes.Single(item => item.AttributeType.Name == "FromKeyedServicesAttribute"); + AssertMutation(census, "subscriber worker crossing to the platform credential", () => + { + var original = attribute.ConstructorArguments[0]; + attribute.ConstructorArguments[0] = new CustomAttributeArgument(original.Type, BoundaryPlatform); + return () => attribute.ConstructorArguments[0] = original; + }, CredentialOffenders); + } + + [Fact] + public void Delivery_SQL_writers_and_dynamic_helpers_are_closed() + { + using var census = ProductionCensus.Read(); + DeliverySqlOffenders(census).Should().BeEmpty( + "Fix: ledger/inbox literal writes belong only to acceptance, completion, inbox claim and exact subscriber scheduler transitions"); + } + + [Fact] + public void Delivery_SQL_census_rejects_quoted_CTE_helper_overload_missing_and_payload_paths() + { + using var census = ProductionCensus.Read(); + var module = census.Modules.Single(item => item.Assembly.Name.Name == DeliveryApplication); + foreach (var sql in new[] { "UPDATE event_deliveries SET attempts=0", "WITH x AS (SELECT 1) UPDATE ONLY public.\"event_deliveries\" SET attempts=0", + "DELETE FROM inbox_messages", "INSERT/* probe */INTO public.\"inbox_messages\" VALUES(1)", + "TRUNCATE another_table, event_deliveries", "MERGE INTO event_deliveries USING x ON true WHEN MATCHED THEN DELETE" }) + AssertMutation(census, "an unauthorized subscriber SQL writer", () => AddSqlProbe(module, sql), DeliverySqlOffenders); + var store = census.Types[(InfrastructureAssembly, DeliveryStore)]; + foreach (var name in new[] { "ReplayAsync", "UnrelatedSqlHelper" }) + AssertMutation(census, "an extra SQL helper or overload", () => + { + var method = AddSqlMethod(store, name, "UPDATE event_deliveries SET attempts=0"); + return () => store.Methods.Remove(method); + }, DeliverySqlOffenders); + foreach (var sql in new[] { "SELECT payload FROM event_deliveries", "SELECT correlation_id FROM event_deliveries", + "SELECT * FROM event_deliveries", "SELECT event_id FROM inbox_messages", "UPDATE event_deliveries SET processed_at = now()" }) + AssertMutation(census, "a scheduler exceeding its granted envelope/completion boundary", () => + { + var method = AddSqlMethod(store, "UnexpectedSql", sql); + return () => store.Methods.Remove(method); + }, DeliverySqlOffenders); + foreach (var helper in new[] { "AcquireAsync", "UpdateAsync", "LockOwnedAsync" }) + AssertMutation(census, "arbitrary SQL passed through a scheduler helper", () => + AddCallProbe(module, "SqlPassover", store.Methods.Single(method => method.Name == helper), OpCodes.Call), DeliverySqlOffenders); + var literal = OwnedBodies(census).Where(item => item.Owner.DeclaringType.FullName == DeliveryStore) + .SelectMany(item => item.Body.Body.Instructions).First(item => item.Operand is string sql && DeliveryMutation.IsMatch(sql)); + AssertMutation(census, "a missing actual subscriber SQL writer", () => ReplaceInstruction(literal, OpCodes.Ldstr, "SELECT 1"), DeliverySqlOffenders); + foreach (var sql in new[] { "SELECT event_id FROM event_deliveries", "UPDATE event_deliveries_archive SET attempts=0", "INSERT INTO another_inbox_messages VALUES(1)" }) + { + var undo = AddSqlProbe(module, sql); + try { DeliverySqlOffenders(census).Should().BeEmpty("unrelated writes and ordinary app-role reads remain admitted"); } + finally { undo(); } + } + } + + [Fact] + public void Delivery_registration_census_rejects_missing_duplicate_replaced_and_detached_subjects() + { + foreach (var change in new[] { "missing", "duplicate" }) + { + var services = ObservationServices(); + var worker = services.Single(item => item.ServiceType == typeof(Microsoft.Extensions.Hosting.IHostedService) + && item.ImplementationType == typeof(DeliveryProcessor)); + if (change == "missing") services.Remove(worker); + else services.Add(worker); + DeliveryRegistrationOffenders(services).Should().NotBeEmpty("mandatory subscriber worker " + change + " must fail"); + } + foreach (var change in new[] { "missing", "duplicate", "wrong-type" }) + { + var services = ObservationServices(); + services.PostConfigure(options => + { + var check = options.Registrations.Single(item => item.Name == DeliveryHealthCheck.Name); + if (change != "duplicate") options.Registrations.Remove(check); + if (change == "duplicate") options.Registrations.Add(check); + if (change == "wrong-type") options.Registrations.Add(new HealthCheckRegistration(DeliveryHealthCheck.Name, + _ => new UnrelatedHealthCheck(), HealthStatus.Unhealthy, [])); + }); + DeliveryRegistrationOffenders(services).Should().NotBeEmpty("subscriber health " + change + " must fail"); + } + var detachedServices = ObservationServices(); + using var provider = detachedServices.BuildServiceProvider(); + using var detached = new DeliveryObservation(TimeProvider.System, new DeliveryDispatchOptions(), + provider.GetRequiredService()); + detachedServices.PostConfigure(options => + { + options.Registrations.Remove(options.Registrations.Single(item => item.Name == DeliveryHealthCheck.Name)); + options.Registrations.Add(new HealthCheckRegistration(DeliveryHealthCheck.Name, + _ => new DeliveryHealthCheck(detached), HealthStatus.Unhealthy, [])); + }); + DeliveryRegistrationOffenders(detachedServices).Should().NotBeEmpty("a detached same-type subscriber health check must fail"); + DeliveryRegistrationOffenders(ObservationServices()).Should().BeEmpty("the exact singleton production binding is the clean control"); + } + + [Fact] + public void Subscriber_terminal_census_rejects_extra_mint_paths_crossed_owners_and_missing_subjects() + { + using var census = ProductionCensus.Read(); + var source = census.Types[(KernelAssembly, MintingSource)]; + var factory = source.Methods.Single(method => method.Name == SubscriberMint); + var module = census.Modules.Single(item => item.Assembly.Name.Name == DeliveryApplication); + foreach (var opcode in new[] { OpCodes.Call, OpCodes.Callvirt, OpCodes.Ldftn, OpCodes.Ldvirtftn }) + AssertTerminalMutation(census, "an extra subscriber mint caller through " + opcode.Name, + () => AddCallProbe(module, "SubscriberMintEscape", factory, opcode)); + AssertTerminalMutation(census, "an asynchronous subscriber mint helper", () => + AddAsyncCallProbe(module, "AsyncSubscriberMintEscape", factory)); + AssertTerminalMutation(census, "a producer using the subscriber witness mint", () => + { + var owner = census.Types[(InfrastructureAssembly, Store)]; + var helper = AddCall(owner, "UnexpectedSubscriberMint", factory, OpCodes.Call); + return () => owner.Methods.Remove(helper); + }); + AssertTerminalMutation(census, "a missing production subscriber store", () => + { + var store = census.Types[(InfrastructureAssembly, DeliveryStore)]; + var ownerModule = store.Module; + ownerModule.Types.Remove(store); + return () => ownerModule.Types.Add(store); + }); + } + + private static readonly Lazy DeliveryMutationPattern = new(() => new Regex( + (OutboxMutation ?? throw new InvalidOperationException("The shared SQL mutation detector must be initialized.")) + .ToString().Replace("outbox_messages", "(?:event_deliveries|inbox_messages)", StringComparison.Ordinal), + RegexOptions.IgnoreCase | RegexOptions.CultureInvariant | RegexOptions.Singleline, TimeSpan.FromSeconds(1))); + private static Regex DeliveryMutation => DeliveryMutationPattern.Value; + private static readonly Regex SchedulerForbiddenSql = new(@"\b(?:payload|correlation_id|causation_id|actor_user_id|inbox_messages)\b|\bSELECT\s+(?:[a-z_][a-z0-9_]*\.)?\*|\bprocessed_at\s*=", + RegexOptions.IgnoreCase | RegexOptions.CultureInvariant, TimeSpan.FromSeconds(1)); + + private static SortedSet DeliverySqlOffenders(ProductionCensus census) + { + var offenders = new SortedSet(StringComparer.Ordinal); + var types = census.Types; + var expected = new List(); + foreach (var (typeName, methodName) in new[] + { + ("LearnStack.Infrastructure.Messaging.PostgresDeliveryAcceptance", "AcceptAsync"), + ("LearnStack.Infrastructure.Messaging.PostgresInboxGuard", "TryClaimAsync"), + (DeliveryCompletionType, "FenceAsync"), + (DeliveryStore, "AcquireAsync"), (DeliveryStore, "UpdateAsync"), + (DeliveryStore, "TryTerminalizeAsync"), (DeliveryStore, "ReplayAsync"), + }) + { + if (!types.TryGetValue((InfrastructureAssembly, typeName), out var type)) + { offenders.Add("Missing subscriber SQL subject: " + typeName); continue; } + var methods = type.Methods.Where(method => method.Name == methodName).ToArray(); + if (methods.Length != 1) offenders.Add("Missing or ambiguous subscriber SQL owner: " + typeName + "." + methodName); + else expected.Add(MethodId(methods[0])); + } + CheckSet(offenders, "subscriber SQL mutation methods", OwnedBodies(census).SelectMany(item => item.Body.Body.Instructions + .Where(instruction => instruction.Operand is string sql && DeliveryMutation.IsMatch(sql)).Select(_ => MethodId(item.Owner))), expected); + foreach (var item in OwnedBodies(census).Where(item => item.Owner.DeclaringType.FullName == DeliveryStore)) + foreach (var instruction in item.Body.Body.Instructions.Where(instruction => instruction.Operand is string)) + if (SchedulerForbiddenSql.IsMatch((string)instruction.Operand)) + offenders.Add(MethodId(item.Owner) + ": scheduler SQL references protected envelope/inbox/completion data"); + if (!types.TryGetValue((InfrastructureAssembly, DeliveryStore), out var store)) return offenders; + var sites = CallSites(census).ToArray(); + foreach (var (helper, owners) in new[] + { + ("AcquireAsync", new[] { "TryClaimAsync", "TryAcquireRecoveryAsync", "TryClaimTerminalAsync" }), + ("UpdateAsync", new[] { "FailAsync", "RequestTerminalAsync" }), + ("LockOwnedAsync", new[] { "UpdateAsync", "TryTerminalizeAsync" }), + }) + CheckSet(offenders, "scheduler SQL helper " + helper, sites.Where(site => site.Target.Name == helper + && SameType(site.Target.DeclaringType, store)).Select(site => site.Owner), + store.Methods.Where(method => owners.Contains(method.Name, StringComparer.Ordinal)).Select(MethodId)); + return offenders; + } + + private static SortedSet DeliveryExecutionOffenders(ProductionCensus census) + { + var offenders = new SortedSet(StringComparer.Ordinal); + var types = census.Types; + foreach (var subject in DeliverySubjects.Select(name => (BoundaryApi, name)).Concat(new[] + { (InfrastructureAssembly, DeliveryExecutorType), (InfrastructureAssembly, ExecutionControl) })) + if (!types.ContainsKey(subject)) offenders.Add("Missing delivery execution subject: " + subject.Item2); + if (offenders.Count != 0) return offenders; + var attempt = types[(BoundaryApi, DeliveryAttempt)]; + var executor = types[(InfrastructureAssembly, DeliveryExecutorType)]; + var control = types[(InfrastructureAssembly, ExecutionControl)]; + var sites = CallSites(census).ToArray(); + if (!attempt.IsSealed || attempt.IsPublic) offenders.Add("DeliveryAttempt must remain internal sealed"); + RequireBoundaryMethod(offenders, attempt, ".ctor", "System.Void", MethodAttributes.Public, + DeliveryExecutorType, DeliveryClaimType); + var constructors = attempt.Methods.Where(method => method.IsConstructor && !method.IsStatic).ToArray(); + var callbacks = constructors.SelectMany(DelegateTargets) + .Where(method => method.ReturnType.FullName == ExecutionResultTask && method.Parameters.Count == 0).ToArray(); + if (callbacks.Length != 1) offenders.Add("DeliveryAttempt requires one task-owned execution callback"); + RequireBoundaryMethod(offenders, executor, "ExecuteControlledAsync", ExecutionResultTask, MethodAttributes.Public, + DeliveryClaimType, ExecutionControl); + CheckSet(offenders, "controlled executor callers", sites.Where(site => site.Target.Name == "ExecuteControlledAsync" + && SameType(site.Target.DeclaringType, executor)).Select(site => site.Owner), callbacks.Select(MethodId)); + CheckSet(offenders, "uncontrolled production executor callers", sites.Where(site => site.Target.Name == "ExecuteAsync" + && SameType(site.Target.DeclaringType, executor)).Select(site => site.Owner), []); + var worker = types[(BoundaryApi, DeliveryWorker)]; + var owner = RequireBoundaryMethod(offenders, worker, "DispatchNormalAsync", BoundaryTask, MethodAttributes.Private, + DeliveryClaimType, BoundaryCancellation); + CheckSet(offenders, "delivery attempt construction sites", sites.Where(site => site.Target.Name == ".ctor" + && SameType(site.Target.DeclaringType, attempt)).Select(site => site.Owner), [owner]); + // Only executor methods may obtain the private cancellation token or end its + // callback lifetime. Caller-side cancellation has its separate public seam. + foreach (var name in new[] { "get_Token", "FinishCallbacksAsync", "DisposeCancellation" }) + { + var declarations = control.Methods.Where(method => method.Name == name).ToArray(); + if (declarations.Length != 1 || declarations.Any(method => method.IsPublic)) + offenders.Add(ExecutionControl + "." + name + ": requires one nonpublic lifecycle capability"); + } + CheckSet(offenders, "execution control token readers", sites.Where(site => site.Target.Name == "get_Token" + && SameType(site.Target.DeclaringType, control)).Select(site => site.Owner), + executor.Methods.Where(method => method.Name == "ExecuteControlledAsync").Select(MethodId)); + CheckSet(offenders, "execution callback finish callers", sites.Where(site => site.Target.Name == "FinishCallbacksAsync" + && SameType(site.Target.DeclaringType, control)).Select(site => site.Owner), + executor.Methods.Where(method => method.Name is "ExecuteControlledAsync" or "ExecuteOwnedAsync").Select(MethodId)); + CheckSet(offenders, "execution cancellation disposal callers", sites.Where(site => site.Target.Name == "DisposeCancellation" + && SameType(site.Target.DeclaringType, control)).Select(site => site.Owner), + executor.Methods.Where(method => method.Name == "ExecuteControlledAsync").Select(MethodId)); + return offenders; + } + + private static SortedSet DeliveryObservationOffenders(ProductionCensus census) + { + var offenders = new SortedSet(StringComparer.Ordinal); + var types = census.Types; + CheckSet(offenders, "delivery observation assemblies", census.Modules.Select(module => module.Assembly.Name.Name), ProductionAssemblies.Names()); + foreach (var subject in DeliverySubjects) + if (!types.ContainsKey((BoundaryApi, subject))) offenders.Add("Missing delivery observation subject: " + subject); + if (offenders.Count != 0) return offenders; + var pending = new Queue(new[] { types[(BoundaryApi, DeliveryObserver)], types[(BoundaryApi, DeliveryHealth)] }); + var visited = new HashSet<(string Assembly, string Type)>(); + while (pending.TryDequeue(out var type)) + { + if (!visited.Add(Key(type))) continue; + foreach (var method in Il.Methods(type).Select(item => item.Definition)) + foreach (var target in MethodOperands(method)) + if (DynamicResolution(target) || target.Name == "Invoke" + && target.DeclaringType.FullName is "System.Reflection.MethodBase" or "System.Reflection.MethodInfo" or "System.Reflection.ConstructorInfo") + offenders.Add(MethodId(method) + ": reflective construction/invocation is not memory observation"); + foreach (var reference in Il.ReferencedTypeReferences(type)) + { + if (reference.Namespace.StartsWith("Npgsql", StringComparison.Ordinal) + || reference.Namespace.StartsWith("System.Data", StringComparison.Ordinal) + || reference.Namespace.StartsWith("Microsoft.EntityFrameworkCore", StringComparison.Ordinal) + || reference.FullName is "System.IServiceProvider" or "Microsoft.Extensions.DependencyInjection.IServiceScopeFactory" + or "Microsoft.Extensions.DependencyInjection.IKeyedServiceProvider") + offenders.Add(TypeId(type) + " -> " + reference.FullName); + if (!types.TryGetValue(Key(reference), out var helper)) continue; + pending.Enqueue(helper); + if (!helper.IsSealed) + foreach (var implementation in types.Values.Where(candidate => !candidate.IsInterface && Derives(candidate, helper, types))) + pending.Enqueue(implementation); + } + } + return offenders; + } + + private static SortedSet DeliveryRegistrationOffenders(IServiceCollection services) + { + var offenders = ObservationRegistrationOffenders(services); + using var provider = services.BuildServiceProvider(); + var checks = provider.GetRequiredService>().Value.Registrations + .Where(check => check.Name == DeliveryHealthCheck.Name).ToArray(); + var check = checks.Length == 1 ? checks[0].Factory(provider) : null; + if (check is not DeliveryHealthCheck) offenders.Add("Private delivery health registration"); + else + { + var observation = provider.GetRequiredService(); + observation.Started(); + observation.Progress(); + observation.Refreshed(new DeliveryQueueSnapshot(0, 0, 0)); + var context = new HealthCheckContext { Registration = checks[0] }; + if (check.CheckHealthAsync(context).GetAwaiter().GetResult().Status != HealthStatus.Healthy) + offenders.Add("Delivery check must follow the healthy singleton observation"); + observation.TerminalAuditFailed(); + if (check.CheckHealthAsync(context).GetAwaiter().GetResult().Status != HealthStatus.Unhealthy) + offenders.Add("Delivery check must follow terminal audit failure"); + observation.TerminalAuditSucceeded(); + if (check.CheckHealthAsync(context).GetAwaiter().GetResult().Status != HealthStatus.Healthy) + offenders.Add("Delivery check must follow terminal audit recovery"); + } + return offenders; + } + + private static string DeliveryWorkerConstructor(SortedSet offenders, TypeDefinition worker) => + RequireBoundaryMethod(offenders, worker, ".ctor", "System.Void", MethodAttributes.Public, + BoundaryLazySource, DeliveryStore, DeliveryExecutorType, + "LearnStack.Infrastructure.Messaging.OutboxEventRegistry", "LearnStack.Infrastructure.Messaging.DeliveryDispatchOptions", + "System.TimeProvider", DeliveryObserver, "Microsoft.Extensions.Logging.ILogger`1<" + DeliveryWorker + ">"); + + private static void CheckDeliveryCredentials(SortedSet offenders, ProductionCensus census, + MethodDefinition composition, MethodDefinition? factory) + { + var types = census.Types; + var worker = types[(BoundaryApi, DeliveryWorker)]; + var store = types[(InfrastructureAssembly, DeliveryStore)]; + var workerConstructor = DeliveryWorkerConstructor(offenders, worker); + CheckSourceFieldAccess(offenders, census, worker, workerConstructor, + [RequireBoundaryMethod(offenders, worker, "StartAsync", BoundaryTask, MethodAttributes.Public, BoundaryCancellation)]); + var storeConstructor = RequireBoundaryMethod(offenders, store, ".ctor", "System.Void", MethodAttributes.Public, + BoundaryLazySource, "LearnStack.Infrastructure.Messaging.OutboxEventRegistry", + "LearnStack.Infrastructure.Messaging.DeliveryDispatchOptions", "LearnStack.SharedKernel.Identifiers.IGuidFactory", + "LearnStack.SharedKernel.Time.IClock", "Microsoft.Extensions.DependencyInjection.IServiceScopeFactory"); + CheckSourceFieldAccess(offenders, census, store, storeConstructor, + [ + RequireBoundaryMethod(offenders, store, "AcquireAsync", "System.Threading.Tasks.Task`1<" + DeliveryClaimType + ">", MethodAttributes.Private, + "LearnStack.Infrastructure.Messaging.DeliveryClaimKind", "System.String", BoundaryStrings, BoundaryCancellation), + RequireBoundaryMethod(offenders, store, "UpdateAsync", BoundaryBooleanTask, MethodAttributes.Private, + DeliveryClaimType, "System.String", "System.String", "System.String", BoundaryCancellation, "System.String"), + RequireBoundaryMethod(offenders, store, "TryTerminalizeAsync", BoundaryBooleanTask, MethodAttributes.Public, + DeliveryClaimType, "System.String", BoundaryCancellation), + RequireBoundaryMethod(offenders, store, "IsDurablyTerminalAsync", BoundaryBooleanTask, MethodAttributes.Private, DeliveryClaimType), + RequireBoundaryMethod(offenders, store, "ReadSnapshotAsync", "System.Threading.Tasks.Task`1", + MethodAttributes.Public, BoundaryStrings, BoundaryCancellation), + RequireBoundaryMethod(offenders, store, "ReplayAsync", BoundaryBooleanTask, MethodAttributes.Public, + "System.Guid", "System.String", "System.Guid", BoundaryCancellation), + ]); + var expected = new[] + { + workerConstructor, + RequireBoundaryMethod(offenders, worker, "ClaimNextAsync", "System.Threading.Tasks.Task`1<" + DeliveryClaimType + ">", + MethodAttributes.Private, "System.String[]", BoundaryCancellation), + RequireBoundaryMethod(offenders, worker, "DispatchAsync", BoundaryTask, MethodAttributes.Private, DeliveryClaimType, BoundaryCancellation), + RequireBoundaryMethod(offenders, worker, "DispatchNormalAsync", BoundaryTask, MethodAttributes.Private, DeliveryClaimType, BoundaryCancellation), + RequireBoundaryMethod(offenders, worker, "RefreshAsync", BoundaryTask, MethodAttributes.Private, BoundaryCancellation), + MethodId(composition), + }.Concat(factory is null ? [] : new[] { MethodId(factory) }); + CheckSet(offenders, "privileged delivery store consumers", OwnedBodies(census) + .Where(item => item.Owner.DeclaringType.FullName != DeliveryStore && ReferencesCapability(item.Body, DeliveryStore)) + .Select(item => MethodId(item.Owner)).Concat(types.Values.SelectMany(type => type.Methods) + .Where(method => !method.HasBody && method.DeclaringType.FullName != DeliveryStore && ReferencesCapability(method, DeliveryStore)) + .Select(MethodId)).Distinct(StringComparer.Ordinal), expected); + var stateMachines = OwnedBodies(census).Where(item => item.Body != item.Owner && item.Owner.DeclaringType.FullName == DeliveryStore) + .Select(item => Key(item.Body.DeclaringType)).ToHashSet(); + var fields = types.Values.SelectMany(type => type.Fields).Where(field => field.FieldType.FullName == DeliveryStore + && !(stateMachines.Contains(Key(field.DeclaringType)) && field.Name == "<>4__this")).ToArray(); + CheckSet(offenders, "privileged delivery store holders", fields.Select(field => TypeId(field.DeclaringType) + ":" + field.Name), + [TypeId(worker) + ":_store"]); + if (fields.Any(field => !field.IsPrivate || !field.IsInitOnly)) offenders.Add("Delivery store holders must be private readonly"); + } + + private static bool ReferencesCapability(MethodDefinition method, string capability) + { + bool Contains(TypeReference type) => type.FullName == capability + || type is GenericInstanceType generic && generic.GenericArguments.Any(Contains) + || type is TypeSpecification specification && Contains(specification.ElementType); + if (Contains(method.ReturnType) || method.Parameters.Any(parameter => Contains(parameter.ParameterType))) return true; + if (!method.HasBody) return false; + return method.Body.Variables.Any(variable => Contains(variable.VariableType)) + || method.Body.Instructions.Any(instruction => instruction.Operand switch + { + GenericInstanceMethod generic => generic.GenericArguments.Any(Contains) || Contains(generic.DeclaringType) + || Contains(generic.ReturnType) || generic.Parameters.Any(parameter => Contains(parameter.ParameterType)), + MethodReference target => Contains(target.DeclaringType) || Contains(target.ReturnType) + || target.Parameters.Any(parameter => Contains(parameter.ParameterType)), + FieldReference field => Contains(field.FieldType), + TypeReference type => Contains(type), + _ => false, + }); + } +} diff --git a/backend/tests/LearnStack.Tests.Architecture/DeliveryOwnerBoundaryTests.cs b/backend/tests/LearnStack.Tests.Architecture/DeliveryOwnerBoundaryTests.cs new file mode 100644 index 00000000..a02051f4 --- /dev/null +++ b/backend/tests/LearnStack.Tests.Architecture/DeliveryOwnerBoundaryTests.cs @@ -0,0 +1,191 @@ +using FluentAssertions; +using Mono.Cecil; +using Mono.Cecil.Cil; +using Xunit; + +namespace LearnStack.Tests.Architecture; + +public sealed partial class OutboxDispatchConventionTests +{ + private const string DeliveryApplication = "LearnStack.Application"; + private const string DeliveryCoordinator = DeliveryApplication + ".Persistence.TransactionExecutionCoordinator"; + private const string DeliveryPrepared = DeliveryCoordinator + "/PreparedDeliveryOwner"; + private const string DeliveryParticipant = DeliveryApplication + ".Messaging.DeliveryCompletionParticipant"; + private const string DeliveryGate = DeliveryApplication + ".Messaging.DeliveryCommitGate"; + private const string DeliveryProducer = DeliveryApplication + ".Messaging.ProducerExecution"; + private const string DeliveryExecutorType = "LearnStack.Infrastructure.Messaging.DeliveryExecutor"; + private const string DeliveryCompletionType = "LearnStack.Infrastructure.Messaging.PostgresDeliveryCompletion"; + private const string DeliveryInbox = "LearnStack.SharedKernel.Messaging.IInboxGuard"; + private const string DeliveryInboxImplementation = "LearnStack.Infrastructure.Messaging.PostgresInboxGuard"; + + [Fact] + public void Delivery_owner_and_inbox_capabilities_are_closed() + { + using var census = ProductionCensus.Read(); + DeliveryOwnerOffenders(census).Should().BeEmpty( + "Fix: delivery preparation, inbox claim, final fence and commit admission stay in the enumerated owner path"); + } + + [Fact] + public void Delivery_owner_census_rejects_extra_direct_delegate_async_and_inherited_capabilities() + { + using var census = ProductionCensus.Read(); + var types = census.Types; + var module = census.Modules.Single(item => item.Assembly.Name.Name == DeliveryApplication); + var capabilities = new[] + { + types[(DeliveryApplication, DeliveryGate)].Methods.Single(method => method.Name == "TryBeginCommit"), + types[(DeliveryApplication, DeliveryGate)].Methods.Single(method => method.Name == "RecordCommitted"), + types[(DeliveryApplication, DeliveryGate)].Methods.Single(method => method.Name == "TryStartExecution"), + types[(DeliveryApplication, DeliveryCoordinator)].Methods.Single(method => method.Name == "PrepareDeliveryOwnerAsync"), + types[(DeliveryApplication, DeliveryPrepared)].Methods.Single(method => method.Name == "CreateAsync"), + types[(DeliveryApplication, DeliveryPrepared)].Methods.Single(method => method.Name == "Consume"), + types[(DeliveryApplication, DeliveryParticipant)].Methods.Single(method => method.Name == "FenceOwnerAsync"), + types[(KernelAssembly, DeliveryInbox)].Methods.Single(method => method.Name == "TryClaimAsync"), + types[(InfrastructureAssembly, DeliveryInboxImplementation)].Methods.Single(method => method.Name == "TryClaimAsync"), + }; + foreach (var capability in capabilities) + { + foreach (var opcode in new[] { OpCodes.Call, OpCodes.Callvirt, OpCodes.Ldftn, OpCodes.Ldvirtftn }) + AssertMutation(census, "an extra delivery capability through " + opcode.Name, + () => AddCallProbe(module, "DeliveryCapabilityProbe", capability, opcode), DeliveryOwnerOffenders); + AssertMutation(census, "an extra asynchronous delivery capability", + () => AddAsyncCallProbe(module, "AsyncDeliveryCapabilityProbe", capability), DeliveryOwnerOffenders); + } + + var participant = types[(DeliveryApplication, DeliveryParticipant)]; + AssertMutation(census, "an additional inherited completion participant", () => + { + var parent = AddProbe(module, "AdditionalCompletionBase"); + parent.Attributes |= TypeAttributes.Abstract; + parent.BaseType = module.ImportReference(participant); + var child = AddProbe(module, "AdditionalCompletionImplementation"); + child.BaseType = parent; + return () => { module.Types.Remove(child); module.Types.Remove(parent); }; + }, DeliveryOwnerOffenders); + var owner = types[(DeliveryApplication, DeliveryPrepared)]; + var constructor = owner.Methods.Single(method => method.IsConstructor && !method.IsStatic); + AssertMutation(census, "a public prepared owner constructor", () => + ChangeAttributes(() => constructor.Attributes, value => constructor.Attributes = value, + (constructor.Attributes & ~MethodAttributes.MemberAccessMask) | MethodAttributes.Public), DeliveryOwnerOffenders); + AssertMutation(census, "a direct prepared owner constructor caller", () => + AddCallProbe(module, "PreparedOwnerProbe", constructor, OpCodes.Newobj), DeliveryOwnerOffenders); + var completion = types[(InfrastructureAssembly, DeliveryCompletionType)]; + var completionConstructor = completion.Methods.Single(method => method.IsConstructor && !method.IsStatic); + AssertMutation(census, "an extra completion constructor caller", () => + AddCallProbe(completion.Module, "CompletionConstructorProbe", completionConstructor, OpCodes.Newobj), + DeliveryOwnerOffenders); + AssertMutation(census, "an extra asynchronous completion constructor caller", () => + AddAsyncCallProbe(completion.Module, "AsyncCompletionConstructorProbe", completionConstructor), + DeliveryOwnerOffenders); + var completionSite = types[(InfrastructureAssembly, DeliveryExecutorType)].NestedTypes + .SelectMany(type => type.Methods).Where(method => method.HasBody) + .SelectMany(method => method.Body.Instructions) + .Single(instruction => instruction.Operand is MethodReference target + && target.Name == ".ctor" && SameType(target.DeclaringType, completion)); + AssertMutation(census, "a missing completion constructor site", () => + { + var originalOpcode = completionSite.OpCode; + var originalOperand = completionSite.Operand; + completionSite.OpCode = OpCodes.Nop; + completionSite.Operand = null; + return () => { completionSite.OpCode = originalOpcode; completionSite.Operand = originalOperand; }; + }, DeliveryOwnerOffenders); + AssertMutation(census, "a missing completion participant subject", () => + { + var completionModule = completion.Module; + completionModule.Types.Remove(completion); + return () => completionModule.Types.Add(completion); + }, DeliveryOwnerOffenders); + + var clean = AddProbe(module, "CleanDeliveryObserver"); + AddCall(clean, "Observe", types[(DeliveryApplication, DeliveryGate)].Properties + .Single(property => property.Name == "CommitAcknowledged").GetMethod, OpCodes.Callvirt); + try + { + DeliveryOwnerOffenders(census).Should().BeEmpty("observation does not confer commit authority"); + } + finally { module.Types.Remove(clean); } + } + + private static SortedSet DeliveryOwnerOffenders(ProductionCensus census) + { + var offenders = new SortedSet(StringComparer.Ordinal); + var types = census.Types; + CheckSet(offenders, "delivery production assemblies", census.Modules.Select(module => module.Assembly.Name.Name), + ProductionAssemblies.Names()); + var subjects = new[] + { + (DeliveryApplication, DeliveryCoordinator), (DeliveryApplication, DeliveryPrepared), + (DeliveryApplication, DeliveryParticipant), (DeliveryApplication, DeliveryGate), + (DeliveryApplication, DeliveryProducer), (InfrastructureAssembly, DeliveryExecutorType), + (InfrastructureAssembly, DeliveryCompletionType), (KernelAssembly, DeliveryInbox), + (InfrastructureAssembly, DeliveryInboxImplementation), + }; + foreach (var subject in subjects) + if (!types.ContainsKey(subject)) offenders.Add("Missing delivery subject: " + subject.Item2); + if (offenders.Count != 0) return offenders; + var coordinator = types[(DeliveryApplication, DeliveryCoordinator)]; + var prepared = types[(DeliveryApplication, DeliveryPrepared)]; + var participant = types[(DeliveryApplication, DeliveryParticipant)]; + var gate = types[(DeliveryApplication, DeliveryGate)]; + var executor = types[(InfrastructureAssembly, DeliveryExecutorType)]; + var completion = types[(InfrastructureAssembly, DeliveryCompletionType)]; + var inbox = types[(KernelAssembly, DeliveryInbox)]; + var inboxImplementation = types[(InfrastructureAssembly, DeliveryInboxImplementation)]; + if (!prepared.IsSealed || prepared.Methods.Count(method => method.IsConstructor && !method.IsStatic) != 1 + || prepared.Methods.Any(method => method.IsConstructor && !method.IsStatic && !method.IsPrivate) + || prepared.Methods.Any(method => method.IsPublic && method.Name is "CompleteAsync" or "CommitAsync") + || prepared.Fields.Count(field => field.IsInitOnly && !field.IsStatic) < 4) + offenders.Add("Prepared owner must retain its private construction and bound immutable frame identity"); + if (!participant.IsAbstract || participant.IsSealed || !completion.IsSealed) + offenders.Add("Completion must use the closed abstract seam and sealed concrete participant"); + if (!gate.IsSealed || gate.Properties.Any(property => property.SetMethod is not null) + || gate.Methods.Any(method => method.IsPublic && method.Name is "TryBeginCommit" or "RecordCommitted")) + offenders.Add("Commit admission/acknowledgement must not be public capabilities"); + CheckSet(offenders, "delivery completion implementations", + types.Values.Where(type => Derives(type, participant, types)).Select(TypeId), [TypeId(completion)]); + CheckSet(offenders, "inbox implementations", + types.Values.Where(type => !type.IsInterface && Derives(type, inbox, types)).Select(TypeId), [TypeId(inboxImplementation)]); + + var sites = CallSites(census).ToArray(); + CheckCapability(completion, ".ctor", executor, "ExecuteOwnedAsync"); + CheckCapability(prepared, ".ctor", prepared, "CreateAsync"); + CheckCapability(prepared, "CreateAsync", coordinator, "PrepareDeliveryOwnerAsync"); + CheckCapability(prepared, "Consume", coordinator, "ExecutePreparedDeliveryAsync"); + CheckCapability(coordinator, "PrepareDeliveryOwnerAsync", executor, "ExecuteOwnedAsync"); + var executionEntries = new[] + { + RequireBoundaryMethod(offenders, executor, "ExecuteAsync", + "System.Threading.Tasks.Task`1", + MethodAttributes.Public, "LearnStack.Infrastructure.Messaging.DeliveryClaim", DeliveryGate, BoundaryCancellation), + RequireBoundaryMethod(offenders, executor, "ExecuteControlledAsync", + "System.Threading.Tasks.Task`1", + MethodAttributes.Public, "LearnStack.Infrastructure.Messaging.DeliveryClaim", + "LearnStack.Infrastructure.Messaging.DeliveryExecutionControl"), + }; + CheckSet(offenders, "delivery execution gate entry sites", sites.Where(site => site.Target.Name == "TryStartExecution" + && SameType(site.Target.DeclaringType, gate)).Select(site => site.Owner), executionEntries); + CheckCapability(gate, "TryBeginCommit", coordinator, "ExecuteFrameAsync"); + CheckCapability(gate, "RecordCommitted", coordinator, "ExecuteFrameAsync"); + CheckCapability(participant, "FenceOwnerAsync", coordinator, "ExecuteFrameAsync"); + CheckCapability(participant, "FenceAsync", participant, "FenceOwnerAsync"); + var inboxSites = sites.Where(site => site.Target.Name == "TryClaimAsync" + && (SameType(site.Target.DeclaringType, inbox) || Derives(site.Target.DeclaringType, inbox, types))); + CheckSet(offenders, "transport inbox claim sites", inboxSites.Select(site => site.Owner), + executor.Methods.Where(method => method.Name == "ExecuteOwnedAsync").Select(MethodId)); + CheckCapability(types[(DeliveryApplication, DeliveryProducer)], "BeginInvocation", coordinator, "ExecuteFrameAsync"); + return offenders; + + void CheckCapability(TypeDefinition targetType, string targetName, TypeDefinition ownerType, string ownerName) + { + var declarations = targetType.Methods.Where(method => method.Name == targetName).ToArray(); + var owners = ownerType.Methods.Where(method => method.Name == ownerName).ToArray(); + if (declarations.Length != 1 || owners.Length != 1) + offenders.Add(targetType.FullName + "." + targetName + ": capability and owner must each have one declaration"); + CheckSet(offenders, targetType.FullName + "." + targetName + " sites", sites + .Where(site => site.Target.Name == targetName && SameType(site.Target.DeclaringType, targetType)) + .Select(site => site.Owner), owners.Select(MethodId)); + } + } +} diff --git a/backend/tests/LearnStack.Tests.Architecture/DomainGenericityTests.DemoIdentities.cs b/backend/tests/LearnStack.Tests.Architecture/DomainGenericityTests.DemoIdentities.cs index 8d2584fa..84ac83e3 100644 --- a/backend/tests/LearnStack.Tests.Architecture/DomainGenericityTests.DemoIdentities.cs +++ b/backend/tests/LearnStack.Tests.Architecture/DomainGenericityTests.DemoIdentities.cs @@ -1,6 +1,7 @@ using System.Text.Json; using FluentAssertions; using LearnStack.Tools.Seeder; +using LearnStack.Modules.Customization.Application.Contracts.Customization; using Microsoft.CodeAnalysis; using Microsoft.CodeAnalysis.CSharp; using Microsoft.CodeAnalysis.CSharp.Syntax; @@ -36,6 +37,15 @@ public void Demo_Identity_Projection_Is_Complete_And_The_Frontend_Snapshot_Match var values = vocabulary.Values; values.Select(row => row.Kind).Distinct().Should().BeEquivalentTo(IdentityKinds); + values.Should().Contain(new DemoIdentity("content-type-key", BuiltInCustomizations.Card.Key)) + .And.Contain(new DemoIdentity("taxonomy-key", BuiltInCustomizations.Plain.Key)); + foreach (var band in BuiltInCustomizations.Plain.Bands) + { + values.Should().Contain(new DemoIdentity("band-key", band.Key)); + foreach (var label in band.DisplayName.Values) + values.Should().Contain(new DemoIdentity("band-label", label)); + } + foreach (var tenant in SeedData.All) { values.Should().Contain(new DemoIdentity("id", tenant.TenantId.Value.ToString("D"))); diff --git a/backend/tests/LearnStack.Tests.Architecture/IntegrationDeliveryBoundaryTests.cs b/backend/tests/LearnStack.Tests.Architecture/IntegrationDeliveryBoundaryTests.cs new file mode 100644 index 00000000..b20c41dd --- /dev/null +++ b/backend/tests/LearnStack.Tests.Architecture/IntegrationDeliveryBoundaryTests.cs @@ -0,0 +1,731 @@ +using FluentAssertions; +using LearnStack.Api.Composition; +using LearnStack.Infrastructure.Messaging; +using LearnStack.Modules.Customization.Application.IntegrationEvents; +using LearnStack.Modules.Tenancy.Application.Contracts.IntegrationEvents; +using LearnStack.SharedKernel.Hosting; +using LearnStack.SharedKernel.Messaging; +using MediatR; +using Microsoft.AspNetCore.Builder; +using Microsoft.Extensions.DependencyInjection; +using Mono.Cecil; +using Mono.Cecil.Cil; +using Xunit; + +namespace LearnStack.Tests.Architecture; + +public sealed partial class OutboxDispatchConventionTests +{ + private const string IntegrationHandlerContract = "LearnStack.SharedKernel.Messaging.IIntegrationEventHandler`1"; + private const string IntegrationSubscription = "LearnStack.Infrastructure.Messaging.IntegrationEventSubscription"; + private const string IntegrationRegistry = "LearnStack.Infrastructure.Messaging.IntegrationEventHandlerRegistry"; + private static readonly Type[] RequiredDeliveryHandlers = + [typeof(InitializeContentTypeDefaultHandler), typeof(InitializeTaxonomyDefaultHandler)]; + private static readonly string[] IntegrationModuleLayers = ["Domain", "Application.Contracts", "Application", "Infrastructure"]; + + /// + /// ADR-0046 and Accepted P02b-3: every production handler has one declared Active + /// subscription and scoped registration; typed invocation and the known reflection + /// capability stay in the delivery owner. PostgreSQL tests prove claim-before- + /// construction, context ordering and atomicity. This is not arbitrary reflection + /// dataflow or control-flow dominance analysis. + /// + [Fact] + public void Integration_Event_Handlers_Use_InboxGuard() + { + using var census = ProductionCensus.Read(); + IntegrationDeliveryOffenders(census, ReadIntegrationDeliveryRegistration()).Should().BeEmpty( + "Fix: declare every concrete integration handler as an Active scoped subscription and invoke it only through the inbox-owned delivery callback"); + DeliveryOwnerOffenders(census).Should().BeEmpty( + "Fix: keep inbox claim, final fence and commit admission in the enumerated delivery owner path"); + IntegrationLifecycleOffenders(census).Should().BeEmpty( + "Fix: let only the approved common owner and transport scopes announce tenants and complete delivery transactions"); + } + + [Fact] + public void Integration_handler_census_rejects_missing_assemblies_layers_and_production_subjects() + { + using var census = ProductionCensus.Read(); + var registration = ReadIntegrationDeliveryRegistration(); + foreach (var module in census.Modules.ToArray()) + AssertMutation(census, "a missing production assembly " + module.Name, () => + { + census.Modules.Remove(module); + return () => census.Modules.Add(module); + }, current => IntegrationCoverageOffenders(current, registration)); + + foreach (var handler in RequiredDeliveryHandlers) + { + var subject = census.Types[(handler.Assembly.GetName().Name!, handler.FullName!)]; + var subjectModule = subject.Module; + AssertMutation(census, "a missing actual handler " + handler.Name, () => + { + subjectModule.Types.Remove(subject); + return () => subjectModule.Types.Add(subject); + }, current => IntegrationCoverageOffenders(current, registration)); + } + + // Each production layer is also a possible newly added handler's home. + // Comparing with the manifest, rather than requiring just two known names, + // makes this extra unregistered subject fail everywhere. + foreach (var module in census.Modules) + AssertMutation(census, "an unregistered integration handler in " + module.Name, () => + { + var probe = AddIntegrationHandlerProbe(module, "UnregisteredIntegrationHandler"); + return () => module.Types.Remove(probe); + }, current => IntegrationCoverageOffenders(current, registration)); + } + + [Fact] + public void Integration_handler_discovery_sees_inherited_generic_and_explicit_contracts() + { + using var census = ProductionCensus.Read(); + using var probes = ModuleDefinition.ReadModule(typeof(OutboxDispatchConventionTests).Assembly.Location); + var types = census.Types; + foreach (var type in probes.GetTypes()) types.Add(Key(type), type); + var bindings = IntegrationHandlerBindings(types).ToArray(); + foreach (var handler in new[] { typeof(IntegrationDeliveryProbeHandler), typeof(IntegrationDeliveryProbeInheritedHandler), + typeof(IntegrationDeliveryProbeExplicitHandler), typeof(IntegrationDeliveryProbePlainInheritedHandler), + typeof(IntegrationDeliveryProbeGenericPlainInheritedHandler), typeof(IntegrationDeliveryProbePlainOverrideHandler) }) + bindings.Where(binding => TypeId(binding.Handler) == IntegrationRuntimeTypeId(handler)) + .Select(binding => TypeId(binding.Event)).Should().Equal([IntegrationRuntimeTypeId(typeof(TenantCreatedIntegrationEventV1))], + "Fix: preserve closed event arguments through inherited and explicit integration contracts"); + bindings.Select(binding => TypeId(binding.Handler)).Should().NotContain( + [IntegrationRuntimeTypeId(typeof(IntegrationDeliveryProbeBase<>)), IntegrationRuntimeTypeId(typeof(IntegrationDeliveryProbeClean))], + "abstract bases and unrelated HandleAsync methods are not concrete subscribers"); + } + + [Fact] + public void Integration_invocation_census_rejects_all_layer_direct_interface_helper_async_and_delegate_bypasses() + { + using var census = ProductionCensus.Read(); + var types = census.Types; + var handler = types[(typeof(InitializeContentTypeDefaultHandler).Assembly.GetName().Name!, typeof(InitializeContentTypeDefaultHandler).FullName!)]; + var handle = handler.Methods.Single(method => method.Name == "HandleAsync"); + var constructor = handler.Methods.Single(method => method.IsConstructor && !method.IsStatic); + var contract = typeof(IIntegrationEventHandler).GetMethod("HandleAsync")!; + var subscription = types[(InfrastructureAssembly, IntegrationSubscription)]; + var capability = subscription.Properties.Single(property => property.Name == "Handle").GetMethod; + var invoke = types[(InfrastructureAssembly, DeliveryExecutorType)].Methods.Single(method => method.Name == "InvokeAsync"); + + foreach (var module in census.Modules) + { + AssertMutation(census, "direct handler invocation in " + module.Name, () => + AddCallProbe(module, "DirectIntegrationInvocation", handle, OpCodes.Call), IntegrationInvocationOffenders); + AssertMutation(census, "interface handler invocation in " + module.Name, () => + AddCallProbe(module, "InterfaceIntegrationInvocation", module.ImportReference(contract), OpCodes.Callvirt), IntegrationInvocationOffenders); + AssertMutation(census, "handler construction in " + module.Name, () => + AddCallProbe(module, "DirectIntegrationConstruction", constructor, OpCodes.Newobj), IntegrationInvocationOffenders); + } + + var application = census.Modules.Single(module => module.Assembly.Name.Name == DeliveryApplication); + foreach (var target in new[] { handle, capability, invoke }) + { + foreach (var opcode in new[] { OpCodes.Call, OpCodes.Callvirt, OpCodes.Ldftn, OpCodes.Ldvirtftn }) + AssertMutation(census, "an extra invocation capability through " + opcode.Name, () => + AddCallProbe(application, "IntegrationInvocationHelper", target, opcode), IntegrationInvocationOffenders); + AssertMutation(census, "an asynchronous invocation helper", () => + AddAsyncCallProbe(application, "AsyncIntegrationInvocationHelper", target), IntegrationInvocationOffenders); + } + + var executor = types[(InfrastructureAssembly, DeliveryExecutorType)]; + AssertMutation(census, "another helper inside the real executor", () => + { + var method = AddCall(executor, "UnexpectedInvocationHelper", invoke, OpCodes.Call); + return () => executor.Methods.Remove(method); + }, IntegrationInvocationOffenders); + var invokeBody = OwnedBodies(census).Single(item => item.Owner == invoke && item.Body.Name == "MoveNext").Body; + var readHandle = invokeBody.Body.Instructions.Single(instruction => instruction.Operand is MethodReference method + && method.Name == "get_Handle" && SameType(method.DeclaringType, subscription)); + AssertMutation(census, "a missing actual reflective invocation capability", () => + ReplaceInstruction(readHandle, OpCodes.Ldstr, "removed invocation capability"), IntegrationInvocationOffenders); + + var clean = AddProbe(application, "CleanIntegrationMetadataObserver"); + AddCall(clean, "Observe", subscription.Properties.Single(property => property.Name == "EventType").GetMethod, OpCodes.Callvirt); + try { IntegrationInvocationOffenders(census).Should().BeEmpty("reading event identity does not invoke a subscriber"); } + finally { application.Types.Remove(clean); } + } + + [Fact] + public void Integration_invocation_compiled_controls_distinguish_helpers_inheritance_async_and_clean_methods() + { + using var census = ProductionCensus.Read(); + using var probes = ModuleDefinition.ReadModule(typeof(OutboxDispatchConventionTests).Assembly.Location); + census.Modules.Add(probes); + try + { + var types = census.Types; + var calls = CallSites(census).Where(site => site.Owner.Contains("/IntegrationDeliveryProbe", StringComparison.Ordinal)).ToArray(); + var offenders = IntegrationTypedInvocationOffenders(types, calls, + probes.GetTypes().Where(type => type.FullName.Contains("/IntegrationDeliveryProbe", StringComparison.Ordinal))); + foreach (var name in new[] { "Direct", "ViaInterface", "ViaInherited", "ViaPlainInherited", "ViaGenericPlainInherited", + "ViaPlainVirtualSlot", "CapturePlainInherited", "AsyncPlainInherited", "Capture", "Async", "Run", + "ResolveOptional", "ResolveRequired", "ResolveAll", "ResolveWrapped", "ResolveKeyed", "ResolveRequiredKeyed", + "ResolveAllKeyed", "Activate", "ResolveOrActivate", "ResolveAsync", "CaptureResolution", + "ResolveForwarded", "ResolveForwardedWrapped", "ResolveFromGenericType", "ResolveRecursive", + "ResolveGrowing", "ResolveForwardedAsync", "CaptureForwardedResolution" }) + offenders.Should().Contain(offender => offender.Contains("::" + name + "(", StringComparison.Ordinal), + "Fix: compiled " + name + " must reach the same typed invocation detector"); + offenders.Should().Contain(offender => offender.Contains("::ResolveConstrained", StringComparison.Ordinal), + "a generic helper constrained to a handler cannot hide typed DI resolution"); + offenders.Should().Contain(offender => offender.Contains("::Construct(", StringComparison.Ordinal)); + offenders.Should().NotContain(offender => offender.Contains("/IntegrationDeliveryProbeClean", StringComparison.Ordinal), + "ordinary HandleAsync names, async helpers and metadata reads are clean"); + } + finally { census.Modules.Remove(probes); } + } + + [Fact] + public void Integration_manifest_census_rejects_missing_rebound_duplicate_and_unscoped_bindings() + { + using var census = ProductionCensus.Read(); + var registration = ReadIntegrationDeliveryRegistration(); + IntegrationCoverageOffenders(census, registration).Should().BeEmpty(); + var first = registration.Handlers[0]; + var manifest = registration.Manifests[0]; + var descriptor = registration.Services.Single(service => service.ServiceType == first.HandlerType); + var mutations = new[] + { + registration with { Handlers = registration.Handlers.Skip(1).ToArray() }, + registration with { Handlers = [.. registration.Handlers, first] }, + registration with { Handlers = [first with { EventType = typeof(IIntegrationEvent) }, .. registration.Handlers.Skip(1)] }, + registration with { Handlers = [first with { Handle = typeof(object).GetMethod(nameof(ToString))! }, .. registration.Handlers.Skip(1)] }, + registration with { Manifests = registration.Manifests.Skip(1).ToArray() }, + registration with { Manifests = [.. registration.Manifests, manifest] }, + registration with { Manifests = [manifest with { Subscription = OutboxSubscriptionManifest.Pending(manifest.Subscription.Id) }, .. registration.Manifests.Skip(1)] }, + registration with { Manifests = [manifest with { Subscription = OutboxSubscriptionManifest.Active(manifest.Subscription.Id, typeof(object)) }, .. registration.Manifests.Skip(1)] }, + registration with { Manifests = [manifest with { Subscription = OutboxSubscriptionManifest.Active(manifest.Subscription.Id, manifest.Subscription.HandlerType!, OutboxSubscriptionExecutionScope.PrivilegedPlatform) }, .. registration.Manifests.Skip(1)] }, + registration with { Manifests = [manifest with { Subscription = OutboxSubscriptionManifest.Active(manifest.Subscription.Id + "-renamed", manifest.Subscription.HandlerType!) }, .. registration.Manifests.Skip(1)] }, + registration with { Services = registration.Services.Where(service => service != descriptor).ToArray() }, + registration with { Services = [.. registration.Services, descriptor] }, + registration with { Services = [.. registration.Services.Where(service => service != descriptor), ServiceDescriptor.Singleton(first.HandlerType, first.HandlerType)] }, + registration with { Services = [.. registration.Services.Where(service => service != descriptor), ServiceDescriptor.Scoped(first.HandlerType, _ => throw new InvalidOperationException("A metadata census must never construct a handler."))] }, + }; + foreach (var mutation in mutations) + IntegrationCoverageOffenders(census, mutation).Should().NotBeEmpty( + "Fix: handler, manifest, reflective method and concrete scoped registration must agree independently"); + IntegrationCoverageOffenders(census, registration).Should().BeEmpty("the actual composition is the clean control"); + } + + private static SortedSet IntegrationDeliveryOffenders(ProductionCensus census, IntegrationDeliveryRegistration registration) + { + var offenders = IntegrationCoverageOffenders(census, registration); + offenders.UnionWith(IntegrationInvocationOffenders(census)); + offenders.UnionWith(IntegrationLifecycleOffenders(census)); + return offenders; + } + + private static SortedSet IntegrationCoverageOffenders(ProductionCensus census, IntegrationDeliveryRegistration registration) + { + var offenders = new SortedSet(StringComparer.Ordinal); + var assemblies = census.Modules.Select(module => module.Assembly.Name.Name).ToArray(); + CheckSet(offenders, "integration-handler production assemblies", assemblies, ProductionAssemblies.Names()); + foreach (var module in assemblies.Where(name => name.StartsWith("LearnStack.Modules.", StringComparison.Ordinal)) + .Select(name => name.Split('.')[2]).Distinct(StringComparer.Ordinal)) + CheckSet(offenders, module + " integration-handler layers", assemblies.Where(name => name.StartsWith("LearnStack.Modules." + module + ".", StringComparison.Ordinal)), + IntegrationModuleLayers.Select(layer => "LearnStack.Modules." + module + "." + layer)); + + var bindings = IntegrationHandlerBindings(census.Types).ToArray(); + if (bindings.Length == 0) offenders.Add("Missing nonempty production integration-handler population"); + foreach (var required in RequiredDeliveryHandlers) + if (!bindings.Any(binding => TypeId(binding.Handler) == IntegrationRuntimeTypeId(required))) + offenders.Add("Missing real default handler: " + required.FullName); + foreach (var binding in bindings.Where(binding => binding.Handler.HasGenericParameters || binding.Event.ContainsGenericParameter)) + offenders.Add("Integration handlers must be closed concrete types: " + TypeId(binding.Handler)); + CheckSet(offenders, "independent handlers and registered metadata", + bindings.Select(binding => TypeId(binding.Event) + " -> " + TypeId(binding.Handler)), + registration.Handlers.Select(handler => IntegrationRuntimeTypeId(handler.EventType) + " -> " + IntegrationRuntimeTypeId(handler.HandlerType))); + CheckSet(offenders, "Active manifests and handler metadata", + registration.Manifests.Where(item => item.Subscription.Readiness == OutboxSubscriptionReadiness.Active) + .Select(item => IntegrationRuntimeTypeId(item.EventType) + " -> " + IntegrationRuntimeTypeId(item.Subscription.HandlerType!)), + registration.Handlers.Select(handler => IntegrationRuntimeTypeId(handler.EventType) + " -> " + IntegrationRuntimeTypeId(handler.HandlerType))); + if (registration.Manifests.Select(item => item.Subscription.Id).Distinct(StringComparer.Ordinal).Count() != registration.Manifests.Length) + offenders.Add("Subscription IDs must be unique"); + foreach (var (id, handler) in new[] + { + ("customization.initialize-content-type-default.v1", typeof(InitializeContentTypeDefaultHandler)), + ("customization.initialize-taxonomy-default.v1", typeof(InitializeTaxonomyDefaultHandler)), + }) + if (registration.Manifests.Count(item => item.EventType == typeof(TenantCreatedIntegrationEventV1) + && item.Subscription.Id == id && item.Subscription.HandlerType == handler + && item.Subscription.Readiness == OutboxSubscriptionReadiness.Active + && item.Subscription.ExecutionScope == OutboxSubscriptionExecutionScope.OrdinaryTenant) != 1) + offenders.Add("Missing exact ordinary Active default subscription: " + id); + + foreach (var handler in registration.Handlers) + { + var contract = typeof(IIntegrationEventHandler<>).MakeGenericType(handler.EventType); + if (handler.ContractType != contract || !handler.HandlerType.GetInterfaces().Contains(contract) + || handler.Handle != contract.GetMethod("HandleAsync")) + offenders.Add("Invalid handler contract/method metadata: " + handler.HandlerType.FullName); + } + foreach (var handler in registration.Handlers.Select(handler => handler.HandlerType).Distinct()) + { + var descriptors = registration.Services.Where(service => service.ServiceType == handler).ToArray(); + if (descriptors.Length != 1 || descriptors[0].IsKeyedService || descriptors[0].Lifetime != ServiceLifetime.Scoped + || descriptors[0].ImplementationType != handler) + offenders.Add("Handler requires exactly one concrete scoped registration: " + handler.FullName); + } + foreach (var root in new[] { typeof(IntegrationEventHandlerRegistry), typeof(OutboxEventRegistry) }) + { + var descriptors = registration.Services.Where(service => service.ServiceType == root).ToArray(); + if (descriptors.Length != 1 || descriptors[0].IsKeyedService || descriptors[0].Lifetime != ServiceLifetime.Singleton) + offenders.Add("One singleton metadata root is required: " + root.FullName); + } + return offenders; + } + + private static SortedSet IntegrationInvocationOffenders(ProductionCensus census) + { + var offenders = new SortedSet(StringComparer.Ordinal); + var types = census.Types; + foreach (var subject in new[] { (InfrastructureAssembly, DeliveryExecutorType), (InfrastructureAssembly, IntegrationSubscription), + (InfrastructureAssembly, IntegrationRegistry), (KernelAssembly, IntegrationHandlerContract) }) + if (!types.ContainsKey(subject)) offenders.Add("Missing integration invocation subject: " + subject.Item2); + if (offenders.Count != 0) return offenders; + var sites = CallSites(census).ToArray(); + offenders.UnionWith(IntegrationTypedInvocationOffenders(types, sites, types.Values)); + var executor = types[(InfrastructureAssembly, DeliveryExecutorType)]; + var subscription = types[(InfrastructureAssembly, IntegrationSubscription)]; + var invokers = executor.Methods.Where(method => method.Name == "InvokeAsync").ToArray(); + var owners = executor.Methods.Where(method => method.Name == "ExecuteOwnedAsync").ToArray(); + if (invokers.Length != 1 || owners.Length != 1 || !invokers[0].IsPrivate || !invokers[0].IsStatic) + { + offenders.Add("DeliveryExecutor requires its private invocation and one owned entry"); + return offenders; + } + var invoke = MethodId(invokers[0]); + var owner = MethodId(owners[0]); + var metadataMembers = subscription.Methods.Where(method => method.CustomAttributes.Any(attribute => + attribute.AttributeType.FullName == "System.Runtime.CompilerServices.CompilerGeneratedAttribute") + && method.Name is "PrintMembers" or "Equals" or "GetHashCode" or "Deconstruct" or "$" or ".ctor") + .Select(MethodId).ToHashSet(StringComparer.Ordinal); + var handleReaders = sites.Where(site => SameType(site.Target.DeclaringType, subscription) && site.Target.Name == "get_Handle").ToArray(); + CheckSet(offenders, "subscription Handle consumers", handleReaders.Where(site => !metadataMembers.Contains(site.Owner)).Select(site => site.Owner), [invoke]); + foreach (var site in sites.Where(site => SameType(site.Target.DeclaringType, subscription) + && (site.Target.Name == "Deconstruct" || site.Target.Name.StartsWith("set_", StringComparison.Ordinal)))) + if (!metadataMembers.Contains(site.Owner)) offenders.Add("Extra subscription metadata extraction/mutation: " + site.Owner); + var metadataFactory = types[(InfrastructureAssembly, IntegrationRegistry)].Methods.Where(method => method.Name == "CreateSubscription").Select(MethodId).ToArray(); + CheckSet(offenders, "subscription construction", sites.Where(site => SameType(site.Target.DeclaringType, subscription) + && site.Target.Name == ".ctor" && !metadataMembers.Contains(site.Owner)).Select(site => site.Owner), metadataFactory); + if (metadataFactory.Length != 1) offenders.Add("Missing sole subscription metadata factory"); + var reflectiveCalls = sites.Where(site => site.Owner == invoke && site.Target.Name == "Invoke" + && site.Target.DeclaringType.FullName is "System.Reflection.MethodBase" or "System.Reflection.MethodInfo").ToArray(); + if (reflectiveCalls.Length != 1) offenders.Add("Missing sole reflective handler invocation in the delivery boundary"); + if (sites.Count(site => site.Owner == invoke && site.Target.Name == "GetRequiredService" + && site.Target.DeclaringType.FullName == "Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions" + && ParametersMatch(site.Target, ["System.IServiceProvider", "System.Type"])) != 1) + offenders.Add("Missing runtime-type handler resolution in the delivery boundary"); + + // Follow actual delegate operands from the attributed owner's bodies. A + // generated name alone confers no authority. These edges identify the known + // callback chain; they do not prove argument flow or execution dominance. + var bodies = OwnedBodies(census).GroupBy(item => MethodId(item.Owner)) + .ToDictionary(group => group.Key, group => group.Select(item => item.Body).ToArray(), StringComparer.Ordinal); + var callbacks = bodies.GetValueOrDefault(owner, []).SelectMany(DelegateTargets).Select(MethodId).ToArray(); + var preparedCallbacks = callbacks.Where(callback => bodies.GetValueOrDefault(callback, []).SelectMany(MethodOperands) + .Any(target => target.Name == "ExecutePreparedDeliveryAsync" && target.DeclaringType.FullName == DeliveryCoordinator)).ToArray(); + if (preparedCallbacks.Length != 1) offenders.Add("Missing sole prepared-delivery callback rooted in ExecuteOwnedAsync"); + var invocationCallbacks = preparedCallbacks.SelectMany(callback => bodies.GetValueOrDefault(callback, []).SelectMany(DelegateTargets)) + .Select(MethodId).ToArray(); + if (invocationCallbacks.Length != 1) offenders.Add("Missing sole handler callback below the prepared delivery callback"); + CheckSet(offenders, "private handler invocation callers", sites.Where(site => site.Target.Name == "InvokeAsync" + && SameType(site.Target.DeclaringType, executor)).Select(site => site.Owner), invocationCallbacks); + CheckSet(offenders, "prepared delivery entries", sites.Where(site => site.Target.Name == "ExecutePreparedDeliveryAsync" + && site.Target.DeclaringType.FullName == DeliveryCoordinator).Select(site => site.Owner), [owner, .. preparedCallbacks]); + if (sites.Count(site => site.Owner == owner && site.Target.Name == "ExecuteAsync" + && site.Target.DeclaringType.FullName == "LearnStack.Application.Audit.AuditExecutionCoordinator") != 1) + offenders.Add("Missing audit classification entry in the delivery owner"); + return offenders; + } + + private static SortedSet IntegrationTypedInvocationOffenders( + Dictionary<(string Assembly, string Type), TypeDefinition> types, + IEnumerable<(string Owner, MethodReference Target)> sites, IEnumerable bodies) + { + var offenders = new SortedSet(StringComparer.Ordinal); + var implementations = IntegrationHandlerImplementationMethods(types).Select(MethodId).ToHashSet(StringComparer.Ordinal); + foreach (var site in sites.Where(site => (site.Target.Name == "HandleAsync" || site.Target.Name.EndsWith(".HandleAsync", StringComparison.Ordinal)) + && (IntegrationHandlerContracts(site.Target.DeclaringType, types).Any() + || ResolveIntegrationMethod(site.Target, types) is { } method && implementations.Contains(MethodId(method))))) + offenders.Add("Typed handler invocation: " + site.Owner + " -> " + site.Target.FullName); + foreach (var site in sites.Where(site => IntegrationResolutionUsesHandler(site.Target, types))) + offenders.Add("Typed handler resolution/construction: " + site.Owner + " -> " + site.Target.FullName); + foreach (var method in bodies.SelectMany(type => type.Methods).Where(method => method.HasBody)) + foreach (var instruction in method.Body.Instructions.Where(instruction => instruction.OpCode == OpCodes.Newobj)) + if (instruction.Operand is MethodReference target && IntegrationHandlerContracts(target.DeclaringType, types).Any()) + offenders.Add("Direct handler construction: " + MethodId(method) + " -> " + target.FullName); + return offenders; + } + + private static IEnumerable IntegrationHandlerImplementationMethods( + Dictionary<(string Assembly, string Type), TypeDefinition> types) + { + foreach (var (handler, eventType) in IntegrationHandlerBindings(types)) + { + // A concrete subscriber may inherit its public implementation from a + // base that does not itself implement the handler contract. Its call + // operand names that base, so contract discovery alone is insufficient. + TypeReference? current = handler; + var visited = new HashSet(StringComparer.Ordinal); + while (current is not null && visited.Add(TypeId(current)) && types.TryGetValue(Key(current), out var definition)) + { + var arguments = current is GenericInstanceType generic ? generic.GenericArguments.ToArray() : []; + var implementation = definition.Methods.FirstOrDefault(method => method is { IsPublic: true, IsStatic: false } + && method.Name == "HandleAsync" && method.ReturnType.FullName == "System.Threading.Tasks.Task" + && method.Parameters.Count == 2 + && TypeId(SubstituteIntegrationArguments(method.Parameters[0].ParameterType, arguments)) == TypeId(eventType) + && method.Parameters[1].ParameterType.FullName == BoundaryCancellation); + if (implementation is not null) + { + yield return implementation; + // An override can also be invoked through its inherited + // virtual slot, whose operand still names the plain base. + if (!implementation.IsVirtual || implementation.IsNewSlot) break; + } + current = definition.BaseType is { } parent ? SubstituteIntegrationArguments(parent, arguments) : null; + } + } + } + + private static MethodDefinition? ResolveIntegrationMethod(MethodReference reference, + Dictionary<(string Assembly, string Type), TypeDefinition> types) + { + if (!types.TryGetValue(Key(reference.DeclaringType), out var declaring)) return null; + var element = reference.GetElementMethod(); + return declaring.Methods.SingleOrDefault(method => method.Name == element.Name + && method.GenericParameters.Count == element.GenericParameters.Count + && IntegrationSignatureType(method.ReturnType) == IntegrationSignatureType(element.ReturnType) + && method.Parameters.Select(parameter => IntegrationSignatureType(parameter.ParameterType)) + .SequenceEqual(element.Parameters.Select(parameter => IntegrationSignatureType(parameter.ParameterType)), StringComparer.Ordinal)); + } + + private static string IntegrationSignatureType(TypeReference type) => type switch + { + GenericParameter parameter => (parameter.Type == GenericParameterType.Type ? "!" : "!!") + parameter.Position, + GenericInstanceType generic => TypeId(generic.ElementType) + "<" + string.Join(",", generic.GenericArguments.Select(IntegrationSignatureType)) + ">", + ArrayType array => IntegrationSignatureType(array.ElementType) + "[" + new string(',', array.Rank - 1) + "]", + ByReferenceType reference => IntegrationSignatureType(reference.ElementType) + "&", + PointerType pointer => IntegrationSignatureType(pointer.ElementType) + "*", + _ => TypeId(type), + }; + + private static bool IntegrationResolutionUsesHandler(MethodReference root, + Dictionary<(string Assembly, string Type), TypeDefinition> types) + { + var pending = new Queue<(MethodDefinition Method, bool[] TypeArguments, bool[] MethodArguments)>(); + var visited = new HashSet(StringComparer.Ordinal); + if (Enqueue(root, [], [])) return true; + while (pending.TryDequeue(out var frame)) + { + // Only handler membership affects the known DI predicate. Substitute + // that bit for every type/method argument: parameter permutations have + // distinct states, while List> growth cannot create an infinite + // specialization walk. Ordinary generic uses do not become violations. + var identity = MethodId(frame.Method) + ":" + string.Join(",", frame.TypeArguments) + + ":" + string.Join(",", frame.MethodArguments); + if (!visited.Add(identity)) continue; + foreach (var target in MethodOperands(frame.Method)) + if (Enqueue(target, frame.TypeArguments, frame.MethodArguments)) return true; + + foreach (var attribute in frame.Method.CustomAttributes.Where(attribute => attribute.AttributeType.FullName is + "System.Runtime.CompilerServices.AsyncStateMachineAttribute" or "System.Runtime.CompilerServices.IteratorStateMachineAttribute" + or "System.Runtime.CompilerServices.AsyncIteratorStateMachineAttribute")) + { + if (attribute.ConstructorArguments.FirstOrDefault().Value is not TypeReference generated + || !types.TryGetValue(Key(generated), out var stateMachine)) continue; + var constructed = frame.Method.Body.Variables.Select(variable => variable.VariableType) + .Concat(MethodOperands(frame.Method).Select(method => method.DeclaringType)) + .FirstOrDefault(type => Key(type) == Key(generated)); + var arguments = constructed is GenericInstanceType generic + ? generic.GenericArguments.Select(argument => ContainsIntegrationHandler(argument, types, frame.TypeArguments, frame.MethodArguments)).ToArray() + : []; + foreach (var moveNext in stateMachine.Methods.Where(method => method.Name == "MoveNext" && method.HasBody)) + pending.Enqueue((moveNext, arguments, [])); + } + } + return false; + + bool Enqueue(MethodReference target, bool[] typeArguments, bool[] methodArguments) + { + var boundTypeArguments = target.DeclaringType is GenericInstanceType declaring + ? declaring.GenericArguments.Select(argument => ContainsIntegrationHandler(argument, types, typeArguments, methodArguments)).ToArray() + : []; + var boundMethodArguments = target is GenericInstanceMethod generic + ? generic.GenericArguments.Select(argument => ContainsIntegrationHandler(argument, types, typeArguments, methodArguments)).ToArray() + : []; + if (IsIntegrationResolutionCapability(target) && boundMethodArguments.Any(value => value)) return true; + if (!boundTypeArguments.Any(value => value) && !boundMethodArguments.Any(value => value)) return false; + // Follow only statically named bodies in the production census. No + // runtime-Type values, reflective construction or branch order enter it. + if (ResolveIntegrationMethod(target, types) is { HasBody: true } method) + pending.Enqueue((method, boundTypeArguments, boundMethodArguments)); + return false; + } + } + + // Only statically typed Microsoft DI capabilities are classified here. The + // production runtime-Type resolver is pinned above; arbitrary Type-valued + // argument provenance and reflective construction are not inferred. + private static bool IsIntegrationResolutionCapability(MethodReference method) => + Key(method.DeclaringType).Assembly == "Microsoft.Extensions.DependencyInjection.Abstractions" + && (method.DeclaringType.FullName switch + { + "Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions" => + method.Name is "GetService" or "GetRequiredService" or "GetServices", + "Microsoft.Extensions.DependencyInjection.ServiceProviderKeyedServiceExtensions" => + method.Name is "GetKeyedService" or "GetRequiredKeyedService" or "GetKeyedServices", + "Microsoft.Extensions.DependencyInjection.ActivatorUtilities" => + method.Name is "CreateInstance" or "GetServiceOrCreateInstance", + _ => false, + }); + + private static bool ContainsIntegrationHandler(TypeReference type, + Dictionary<(string Assembly, string Type), TypeDefinition> types, + bool[]? typeArguments = null, bool[]? methodArguments = null) + { + var pending = new Queue(); + var visited = new HashSet(); + pending.Enqueue(type); + while (pending.TryDequeue(out var current)) + { + if (!visited.Add(current)) continue; + // A parameter has constraints rather than a resolvable declaring type. + if (current is GenericParameter parameter) + { + var arguments = parameter.Type == GenericParameterType.Type ? typeArguments : methodArguments; + if (arguments is not null && parameter.Position < arguments.Length && arguments[parameter.Position]) return true; + foreach (var constraint in parameter.Constraints) pending.Enqueue(constraint.ConstraintType); + continue; + } + if (IntegrationHandlerContracts(current, types).Any()) return true; + if (current is GenericInstanceType generic) + foreach (var argument in generic.GenericArguments) pending.Enqueue(argument); + else if (current is TypeSpecification specification) pending.Enqueue(specification.ElementType); + } + return false; + } + + private static IEnumerable<(TypeDefinition Handler, TypeReference Event)> IntegrationHandlerBindings( + Dictionary<(string Assembly, string Type), TypeDefinition> types) => + types.Values.Where(type => !type.IsInterface && !type.IsAbstract) + .SelectMany(type => IntegrationHandlerContracts(type, types).Select(contract => (type, contract.GenericArguments[0]))); + + private static IEnumerable IntegrationHandlerContracts(TypeReference type, + Dictionary<(string Assembly, string Type), TypeDefinition> types) + { + var pending = new Queue(); + var visited = new HashSet(StringComparer.Ordinal); + pending.Enqueue(type); + while (pending.TryDequeue(out var current)) + { + if (!visited.Add(TypeId(current))) continue; + if (current is GenericInstanceType contract && Key(current) == (KernelAssembly, IntegrationHandlerContract)) + yield return contract; + if (!types.TryGetValue(Key(current), out var definition)) continue; + var arguments = current is GenericInstanceType generic ? generic.GenericArguments.ToArray() : []; + if (definition.BaseType is { } parent) pending.Enqueue(SubstituteIntegrationArguments(parent, arguments)); + foreach (var implemented in definition.Interfaces) + pending.Enqueue(SubstituteIntegrationArguments(implemented.InterfaceType, arguments)); + } + } + + private static TypeReference SubstituteIntegrationArguments(TypeReference reference, TypeReference[] arguments) + { + if (reference is GenericParameter { Type: GenericParameterType.Type } parameter && parameter.Position < arguments.Length) + return arguments[parameter.Position]; + if (reference is not GenericInstanceType generic) return reference; + var result = new GenericInstanceType(generic.ElementType); + foreach (var argument in generic.GenericArguments) result.GenericArguments.Add(SubstituteIntegrationArguments(argument, arguments)); + return result; + } + + private static IntegrationDeliveryRegistration ReadIntegrationDeliveryRegistration() + { + var production = ProductionAssemblies.All().SelectMany(assembly => assembly.GetTypes()).ToArray(); + var handlerAssemblies = production.Where(type => !type.IsAbstract && !type.IsInterface + && type.GetInterfaces().Any(contract => contract.IsGenericType && contract.GetGenericTypeDefinition() == typeof(IIntegrationEventHandler<>))) + .Select(type => type.Assembly).Distinct().ToArray(); + var builder = WebApplication.CreateBuilder([]); + builder.Configuration.Sources.Clear(); + // Use the real registration method with independently discovered assemblies. + // The actual Program root and hosted effects are covered by the real default- + // delivery integration fixture; no business handler is constructed here. + builder.AddLearnStackCrossCuttingFoundation(DeploymentMode.Development, handlerAssemblies); + using var provider = builder.Services.BuildServiceProvider(); + var handlers = provider.GetRequiredService(); + var registry = provider.GetRequiredService(); + var events = production.Where(type => !type.IsAbstract && !type.IsInterface && !type.ContainsGenericParameters + && typeof(IIntegrationEvent).IsAssignableFrom(type)); + return new(handlers.All.ToArray(), events.SelectMany(type => registry.Resolve(type.AssemblyQualifiedName).Subscriptions + .Select(subscription => new IntegrationManifestBinding(type, subscription))).ToArray(), builder.Services.ToArray()); + } + + private static string IntegrationRuntimeTypeId(Type type) => type.Assembly.GetName().Name + ":" + type.FullName!.Replace('+', '/'); + + private static TypeDefinition AddIntegrationHandlerProbe(ModuleDefinition module, string name) + { + var type = AddProbe(module, name); + var contract = new GenericInstanceType(module.ImportReference(typeof(IIntegrationEventHandler<>))); + contract.GenericArguments.Add(module.ImportReference(typeof(TenantCreatedIntegrationEventV1))); + type.Interfaces.Add(new InterfaceImplementation(contract)); + return type; + } + + private sealed record IntegrationManifestBinding(Type EventType, OutboxSubscriptionManifest Subscription); + private sealed record IntegrationDeliveryRegistration(IntegrationEventSubscription[] Handlers, + IntegrationManifestBinding[] Manifests, ServiceDescriptor[] Services); + + private sealed class IntegrationDeliveryProbeHandler : IIntegrationEventHandler + { + public Task HandleAsync(TenantCreatedIntegrationEventV1 @event, CancellationToken cancellationToken = default) => Task.CompletedTask; + } + private abstract class IntegrationDeliveryProbeBase : IIntegrationEventHandler where T : IIntegrationEvent + { + public Task HandleAsync(T @event, CancellationToken cancellationToken = default) => Task.CompletedTask; + } + private sealed class IntegrationDeliveryProbeInheritedHandler : IntegrationDeliveryProbeBase; + private abstract class IntegrationDeliveryProbePlainBase + { + private readonly Task _completion = Task.CompletedTask; + public Task HandleAsync(TenantCreatedIntegrationEventV1 fact, CancellationToken cancellationToken = default) => _completion; + } + private sealed class IntegrationDeliveryProbePlainInheritedHandler : IntegrationDeliveryProbePlainBase, IIntegrationEventHandler; + private abstract class IntegrationDeliveryProbeGenericPlainBase + { + private readonly Task _completion = Task.CompletedTask; + public Task HandleAsync(T fact, CancellationToken cancellationToken = default) => _completion; + } + private sealed class IntegrationDeliveryProbeGenericPlainInheritedHandler : IntegrationDeliveryProbeGenericPlainBase, IIntegrationEventHandler; + private abstract class IntegrationDeliveryProbePlainVirtualBase + { + public virtual Task HandleAsync(TenantCreatedIntegrationEventV1 fact, CancellationToken cancellationToken = default) => Task.CompletedTask; + } + private sealed class IntegrationDeliveryProbePlainOverrideHandler : IntegrationDeliveryProbePlainVirtualBase, IIntegrationEventHandler + { + public override Task HandleAsync(TenantCreatedIntegrationEventV1 fact, CancellationToken cancellationToken = default) => Task.CompletedTask; + } + private sealed class IntegrationDeliveryProbeExplicitHandler : IIntegrationEventHandler + { + Task IIntegrationEventHandler.HandleAsync(TenantCreatedIntegrationEventV1 @event, CancellationToken cancellationToken) => Task.CompletedTask; + } + private interface IIntegrationDeliveryProbeHelper + { + Task Run(IIntegrationEventHandler handler, TenantCreatedIntegrationEventV1 fact); + } + private sealed class IntegrationDeliveryProbeHelper : IIntegrationDeliveryProbeHelper + { + public Task Run(IIntegrationEventHandler handler, TenantCreatedIntegrationEventV1 fact) => handler.HandleAsync(fact); + } + private interface IIntegrationDeliveryProbeCleanHelper + { + Task Run(TenantCreatedIntegrationEventV1 fact); + } + private sealed class IntegrationDeliveryProbeCleanHelper : IIntegrationDeliveryProbeCleanHelper + { + public Task Run(TenantCreatedIntegrationEventV1 fact) => IntegrationDeliveryProbeClean.HandleAsync(fact); + } + private static class IntegrationDeliveryProbeGenericHelpers + { + public static T Resolve(IServiceProvider services) where T : notnull => services.GetRequiredService(); + public static T Forward(IServiceProvider services) where T : notnull => Resolve(services); + public static object Recursive(IServiceProvider services, bool stop) where T : notnull where TOther : notnull => + stop ? services.GetRequiredService() : Recursive(services, true); + public static object Growing(IServiceProvider services, bool stop) where T : notnull => + stop ? services.GetRequiredService() : Growing>(services, true); + public static Type ReadGrowing(bool stop) => stop ? typeof(T) : ReadGrowing>(true); + public static async Task ResolveAsync(IServiceProvider services) where T : notnull + { + await Task.Yield(); + return services.GetRequiredService(); + } + } + private static class IntegrationDeliveryProbeGenericResolver where T : notnull + { + public static T Resolve(IServiceProvider services) => IntegrationDeliveryProbeGenericHelpers.Forward(services); + } + private static class IntegrationDeliveryProbeCalls + { + public static Task Direct(IntegrationDeliveryProbeHandler handler, TenantCreatedIntegrationEventV1 fact) => handler.HandleAsync(fact); + public static Task ViaInterface(IIntegrationEventHandler handler, TenantCreatedIntegrationEventV1 fact) => handler.HandleAsync(fact); + public static Task ViaInherited(IntegrationDeliveryProbeInheritedHandler handler, TenantCreatedIntegrationEventV1 fact) => handler.HandleAsync(fact); + public static Task ViaPlainInherited(IntegrationDeliveryProbePlainInheritedHandler handler, TenantCreatedIntegrationEventV1 fact) => handler.HandleAsync(fact); + public static Task ViaGenericPlainInherited(IntegrationDeliveryProbeGenericPlainInheritedHandler handler, TenantCreatedIntegrationEventV1 fact) => handler.HandleAsync(fact); + public static Task ViaPlainVirtualSlot(IntegrationDeliveryProbePlainOverrideHandler handler, TenantCreatedIntegrationEventV1 fact) => ((IntegrationDeliveryProbePlainVirtualBase)handler).HandleAsync(fact); + public static Func CapturePlainInherited(IntegrationDeliveryProbePlainInheritedHandler handler) => handler.HandleAsync; + public static async Task AsyncPlainInherited(IntegrationDeliveryProbePlainInheritedHandler handler, TenantCreatedIntegrationEventV1 fact) + { + await Task.Yield(); + await handler.HandleAsync(fact); + } + public static Func Capture(IIntegrationEventHandler handler) => handler.HandleAsync; + public static async Task Async(IIntegrationEventHandler handler, TenantCreatedIntegrationEventV1 fact) + { + await Task.Yield(); + await handler.HandleAsync(fact); + } + public static Task ViaHelper(IIntegrationDeliveryProbeHelper helper, IIntegrationEventHandler handler, TenantCreatedIntegrationEventV1 fact) => helper.Run(handler, fact); + public static InitializeContentTypeDefaultHandler Construct(ISender sender) => new(sender); + public static IntegrationDeliveryProbeHandler? ResolveOptional(IServiceProvider services) => services.GetService(); + public static IntegrationDeliveryProbeHandler ResolveRequired(IServiceProvider services) => services.GetRequiredService(); + public static IEnumerable> ResolveAll(IServiceProvider services) => services.GetServices>(); + public static IEnumerable> ResolveWrapped(IServiceProvider services) => services.GetRequiredService>>(); + public static IntegrationDeliveryProbeHandler? ResolveKeyed(IServiceProvider services) => services.GetKeyedService("probe"); + public static IntegrationDeliveryProbeHandler ResolveRequiredKeyed(IServiceProvider services) => services.GetRequiredKeyedService("probe"); + public static IEnumerable> ResolveAllKeyed(IServiceProvider services) => services.GetKeyedServices>("probe"); + public static IntegrationDeliveryProbeHandler Activate(IServiceProvider services) => ActivatorUtilities.CreateInstance(services); + public static IntegrationDeliveryProbeHandler ResolveOrActivate(IServiceProvider services) => ActivatorUtilities.GetServiceOrCreateInstance(services); + public static T ResolveConstrained(IServiceProvider services) where T : IIntegrationEventHandler => services.GetRequiredService(); + public static async Task ResolveAsync(IServiceProvider services) + { + await Task.Yield(); + return services.GetRequiredService(); + } + public static Func CaptureResolution() => ServiceProviderServiceExtensions.GetRequiredService; + public static IntegrationDeliveryProbeHandler ResolveForwarded(IServiceProvider services) => IntegrationDeliveryProbeGenericHelpers.Forward(services); + public static IEnumerable> ResolveForwardedWrapped(IServiceProvider services) => IntegrationDeliveryProbeGenericHelpers.Forward>>(services); + public static IntegrationDeliveryProbeHandler ResolveFromGenericType(IServiceProvider services) => IntegrationDeliveryProbeGenericResolver.Resolve(services); + public static object ResolveRecursive(IServiceProvider services) => IntegrationDeliveryProbeGenericHelpers.Recursive(services, false); + public static object ResolveGrowing(IServiceProvider services) => IntegrationDeliveryProbeGenericHelpers.Growing(services, false); + public static Task ResolveForwardedAsync(IServiceProvider services) => IntegrationDeliveryProbeGenericHelpers.ResolveAsync(services); + public static Func CaptureForwardedResolution() => IntegrationDeliveryProbeGenericHelpers.Forward; + } + private abstract class IntegrationDeliveryProbeCleanPlainBase + { + private readonly Task _completion = Task.CompletedTask; + public Task HandleAsync(TenantCreatedIntegrationEventV1 fact, CancellationToken cancellationToken = default) => _completion; + } + private sealed class IntegrationDeliveryProbeCleanDerived : IntegrationDeliveryProbeCleanPlainBase; + private static class IntegrationDeliveryProbeClean + { + public static Task HandleAsync(TenantCreatedIntegrationEventV1 fact) => Task.CompletedTask; + public static async Task Async(TenantCreatedIntegrationEventV1 fact) => await HandleAsync(fact); + public static Task ViaHelper(IIntegrationDeliveryProbeCleanHelper helper, TenantCreatedIntegrationEventV1 fact) => helper.Run(fact); + public static Type Read(IntegrationEventSubscription subscription) => subscription.EventType; + public static object? ResolveOptional(IServiceProvider services) => services.GetService(); + public static object ResolveRequired(IServiceProvider services) => services.GetRequiredService(); + public static IEnumerable ResolveAll(IServiceProvider services) => services.GetServices(); + public static IEnumerable ResolveWrapped(IServiceProvider services) => services.GetRequiredService>(); + public static object? ResolveKeyed(IServiceProvider services) => services.GetKeyedService("probe"); + public static object ResolveRequiredKeyed(IServiceProvider services) => services.GetRequiredKeyedService("probe"); + public static IEnumerable ResolveAllKeyed(IServiceProvider services) => services.GetKeyedServices("probe"); + public static object Activate(IServiceProvider services) => ActivatorUtilities.CreateInstance(services); + public static object ResolveOrActivate(IServiceProvider services) => ActivatorUtilities.GetServiceOrCreateInstance(services); + public static T ResolveConstrained(IServiceProvider services) where T : class, IComparable => services.GetRequiredService(); + public static async Task ResolveAsync(IServiceProvider services) + { + await Task.Yield(); + return services.GetRequiredService(); + } + public static Func CaptureResolution() => ServiceProviderServiceExtensions.GetRequiredService; + public static object ResolveForwarded(IServiceProvider services) => IntegrationDeliveryProbeGenericHelpers.Forward(services); + public static IEnumerable ResolveForwardedWrapped(IServiceProvider services) => IntegrationDeliveryProbeGenericHelpers.Forward>(services); + public static object ResolveFromGenericType(IServiceProvider services) => IntegrationDeliveryProbeGenericResolver.Resolve(services); + public static object ResolveRecursive(IServiceProvider services) => IntegrationDeliveryProbeGenericHelpers.Recursive(services, false); + public static object ResolveGrowing(IServiceProvider services) => IntegrationDeliveryProbeGenericHelpers.Growing(services, false); + public static Task ResolveForwardedAsync(IServiceProvider services) => IntegrationDeliveryProbeGenericHelpers.ResolveAsync(services); + public static Type ReadHandlerMetadata() => IntegrationDeliveryProbeGenericHelpers.ReadGrowing(false); + public static Task ViaPlainInherited(IntegrationDeliveryProbeCleanDerived helper, TenantCreatedIntegrationEventV1 fact) => helper.HandleAsync(fact); + public static Func CapturePlainInherited(IntegrationDeliveryProbeCleanDerived helper) => helper.HandleAsync; + } +} diff --git a/backend/tests/LearnStack.Tests.Architecture/IntegrationDeliveryLifecycleBoundaryTests.cs b/backend/tests/LearnStack.Tests.Architecture/IntegrationDeliveryLifecycleBoundaryTests.cs new file mode 100644 index 00000000..c4f48da9 --- /dev/null +++ b/backend/tests/LearnStack.Tests.Architecture/IntegrationDeliveryLifecycleBoundaryTests.cs @@ -0,0 +1,363 @@ +using System.Data; +using System.Data.Common; +using FluentAssertions; +using LearnStack.Infrastructure.Persistence; +using LearnStack.SharedKernel.Persistence; +using LearnStack.SharedKernel.Tenancy; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Storage; +using Mono.Cecil; +using Mono.Cecil.Cil; +using Npgsql; +using Xunit; + +namespace LearnStack.Tests.Architecture; + +public sealed partial class OutboxDispatchConventionTests +{ + private const string LifecycleUnit = "LearnStack.SharedKernel.Persistence.IUnitOfWork"; + private const string LifecycleScope = "LearnStack.SharedKernel.Persistence.IUnitOfWorkScope"; + private const string LifecycleAdapter = "LearnStack.Infrastructure.Persistence.NpgsqlUnitOfWork"; + private const string LifecycleFrame = LifecycleAdapter + "/Frame"; + private const string LifecycleAcceptance = "LearnStack.Infrastructure.Messaging.PostgresDeliveryAcceptance"; + private const string LifecyclePlatform = "LearnStack.Infrastructure.MultiTenancy.PlatformAdminScope"; + private const string LifecycleTask = "System.Threading.Tasks.Task"; + private const string LifecycleScopeTask = "System.Threading.Tasks.Task`1<" + LifecycleScope + ">"; + private static readonly string[] LifecycleTerminalMethods = ["Commit", "Rollback", "CommitAsync", "RollbackAsync"]; + private static readonly string[] LifecycleFacadeTerminalMethods = ["CommitTransaction", "RollbackTransaction", "CommitTransactionAsync", "RollbackTransactionAsync"]; + + /// + /// ADR-0046's completion gate and ADR-0057's common owner: typed unit/frame and + /// database transaction completion and tenant announcement stay with the enumerated owners. This protects + /// the whole production helper surface without banning ambient-unit reads. It + /// does not infer arbitrary reflection or SQL transaction-control dataflow. + /// + [Fact] + public void Integration_delivery_lifecycle_stays_with_its_enumerated_owners() + { + using var census = ProductionCensus.Read(); + IntegrationLifecycleOffenders(census).Should().BeEmpty( + "Fix: let the common owner fence and complete business delivery; retain only the named acceptance, persistence and standalone transaction owners"); + } + + [Fact] + public void Integration_delivery_lifecycle_census_rejects_typed_helper_async_delegate_and_missing_paths() + { + using var census = ProductionCensus.Read(); + var types = census.Types; + var unit = types[(KernelAssembly, LifecycleUnit)]; + var scope = types[(KernelAssembly, LifecycleScope)]; + var adapter = types[(InfrastructureAssembly, LifecycleAdapter)]; + var frame = types[(InfrastructureAssembly, LifecycleFrame)]; + var application = census.Modules.Single(module => module.Assembly.Name.Name == DeliveryApplication); + var capabilities = new[] { unit, adapter }.SelectMany(type => type.Methods.Where(method => + method.Name is "BeginTransactionAsync" or "CommitAsync" or "RollbackAsync" or "SetTenantContextAsync")) + .Concat(new[] { scope, frame }.SelectMany(type => type.Methods.Where(method => method.Name is "CompleteAsync" or "FailAsync"))) + .Cast().Concat(new[] { typeof(DbTransaction), typeof(IDbTransaction), typeof(NpgsqlTransaction), + typeof(IDbContextTransaction), typeof(RelationalTransaction), typeof(DatabaseFacade) } + .SelectMany(type => type.GetMethods().Where(method => LifecycleTerminalMethods.Contains(method.Name, StringComparer.Ordinal) + || LifecycleFacadeTerminalMethods.Contains(method.Name, StringComparer.Ordinal)) + .Where(method => + method.GetParameters().Length == 0 || method.GetParameters() is [{ ParameterType: var parameter }] && parameter == typeof(CancellationToken))) + .Select(method => application.ImportReference(method))).ToArray(); + foreach (var capability in capabilities) + { + foreach (var opcode in new[] { OpCodes.Call, OpCodes.Callvirt, OpCodes.Ldftn, OpCodes.Ldvirtftn }) + AssertMutation(census, "extra typed lifecycle authority through " + opcode.Name, () => + AddCallProbe(application, "IntegrationLifecycleEscape", capability, opcode), IntegrationLifecycleOffenders); + AssertMutation(census, "an asynchronous lifecycle helper", () => + AddAsyncCallProbe(application, "AsyncIntegrationLifecycleEscape", capability), IntegrationLifecycleOffenders); + } + foreach (var module in census.Modules) + AssertMutation(census, "a lifecycle bypass in " + module.Name, () => + AddCallProbe(module, "IntegrationLifecycleLayerEscape", unit.Methods.Single(method => method.Name == "CommitAsync"), OpCodes.Callvirt), IntegrationLifecycleOffenders); + + foreach (var subject in new[] { adapter, frame }) + { + var module = subject.Module; + var declaring = subject.DeclaringType; + AssertMutation(census, "a missing real lifecycle implementation", () => + { + if (declaring is null) module.Types.Remove(subject); + else declaring.NestedTypes.Remove(subject); + return () => + { + if (declaring is null) module.Types.Add(subject); + else declaring.NestedTypes.Add(subject); + }; + }, IntegrationLifecycleOffenders); + } + AssertMutation(census, "an extra unit implementation", () => + { + var probe = AddProbe(application, "AdditionalIntegrationUnit"); + probe.Interfaces.Add(new InterfaceImplementation(application.ImportReference(unit))); + return () => application.Types.Remove(probe); + }, IntegrationLifecycleOffenders); + var complete = OwnedBodies(census).Where(item => item.Owner.DeclaringType.FullName == DeliveryCoordinator + && item.Owner.Name == "ExecuteFrameAsync").SelectMany(item => item.Body.Body.Instructions) + .Single(instruction => instruction.Operand is MethodReference target && target.Name == "CompleteAsync" + && SameType(target.DeclaringType, scope)); + AssertMutation(census, "a missing real owner completion site", () => + ReplaceInstruction(complete, OpCodes.Ldstr, "missing owner completion"), IntegrationLifecycleOffenders); + var physicalCommit = OwnedBodies(census).Where(item => item.Owner.DeclaringType.FullName == LifecycleAdapter + && item.Owner.Name == "CommitFrameAsync").SelectMany(item => item.Body.Body.Instructions) + .Single(instruction => instruction.Operand is MethodReference target + && IntegrationLifecycleCapability(target, types) == "database.CommitAsync"); + AssertMutation(census, "a missing real physical completion site", () => + ReplaceInstruction(physicalCommit, OpCodes.Ldstr, "missing physical completion"), IntegrationLifecycleOffenders); + var announcements = OwnedBodies(census).SelectMany(item => item.Body.Body.Instructions) + .Where(instruction => instruction.Operand is MethodReference target + && IntegrationLifecycleCapability(target, types) == "unit.announce").ToArray(); + announcements.Should().HaveCount(3, "Fix: retain the ordinary owner and both durable transport announcement sites"); + foreach (var announcement in announcements) + AssertMutation(census, "a missing real tenant announcement site", () => + ReplaceInstruction(announcement, OpCodes.Ldstr, "missing tenant announcement"), IntegrationLifecycleOffenders); + var clean = AddProbe(application, "CleanIntegrationLifecycleObserver"); + foreach (var method in unit.Methods.Where(method => method.IsGetter || method.Name == "MarkRollbackOnly") + .Concat(scope.Methods.Where(method => method.IsGetter))) + AddCall(clean, "Observe" + method.Name, method, OpCodes.Callvirt); + try { IntegrationLifecycleOffenders(census).Should().BeEmpty("ambient reads and poisoning cannot grant commit ownership"); } + finally { application.Types.Remove(clean); } + + using var probes = ModuleDefinition.ReadModule(typeof(OutboxDispatchConventionTests).Assembly.Location); + census.Modules.Add(probes); + try + { + var sites = CallSites(census).Where(site => !site.Owner.StartsWith(probes.Assembly.Name.Name + ":", StringComparison.Ordinal) + || site.Owner.Contains("/IntegrationLifecycleProbe", StringComparison.Ordinal)); + var offenders = IntegrationLifecycleCallsiteOffenders(census.Types, sites); + foreach (var name in new[] { "Commit", "Rollback", "Begin", "Complete", "Fail", "Concrete", "RawCommit", "RawRollback", + "ConcreteRaw", "InheritedUnit", "InheritedRaw", "Async", "Capture", "Run", + "Announce", "ConcreteAnnounce", "InheritedAnnounce", "AsyncAnnounce", "CaptureAnnouncement", + "EfCommit", "EfRollback", "EfCommitAsync", "EfRollbackAsync", "EfConcreteCommitAsync", "EfCurrentCommitAsync", + "EfFacadeCommit", "EfFacadeRollback", "EfFacadeCommitAsync", "EfFacadeRollbackAsync", "EfAsync", "EfCapture" }) + offenders.Should().Contain(offender => offender.Contains("::" + name + "(", StringComparison.Ordinal), + "Fix: the real lifecycle predicate must reject compiled " + name); + offenders.Should().NotContain(offender => offender.Contains("/IntegrationLifecycleProbeClean::", StringComparison.Ordinal), + "ordinary unit observations and rollback-only marking remain available"); + } + finally { census.Modules.Remove(probes); } + } + + private static SortedSet IntegrationLifecycleOffenders(ProductionCensus census) + { + var offenders = new SortedSet(StringComparer.Ordinal); + CheckSet(offenders, "lifecycle production assemblies", census.Modules.Select(module => module.Assembly.Name.Name), ProductionAssemblies.Names()); + var types = census.Types; + foreach (var subject in new[] { (KernelAssembly, LifecycleUnit), (KernelAssembly, LifecycleScope), + (InfrastructureAssembly, LifecycleAdapter), (InfrastructureAssembly, LifecycleFrame) }) + if (!types.ContainsKey(subject)) offenders.Add("Missing lifecycle subject: " + subject.Item2); + if (offenders.Count != 0) return offenders; + var unit = types[(KernelAssembly, LifecycleUnit)]; + var scope = types[(KernelAssembly, LifecycleScope)]; + var adapter = types[(InfrastructureAssembly, LifecycleAdapter)]; + var frame = types[(InfrastructureAssembly, LifecycleFrame)]; + CheckSet(offenders, "unit lifecycle implementations", types.Values.Where(type => !type.IsInterface && Derives(type, unit, types)).Select(TypeId), [TypeId(adapter)]); + CheckSet(offenders, "frame lifecycle implementations", types.Values.Where(type => !type.IsInterface && Derives(type, scope, types)).Select(TypeId), [TypeId(frame)]); + if (!adapter.IsSealed || !frame.IsSealed || !frame.IsNestedPrivate) offenders.Add("The unit adapter and its private frame must remain sealed"); + foreach (var type in new[] { unit, adapter }) + { + CheckLifecycleDeclarations(type, "BeginTransactionAsync", LifecycleScopeTask, + [[BoundaryCancellation], ["LearnStack.SharedKernel.Persistence.TransactionMode", BoundaryCancellation]]); + CheckLifecycleDeclarations(type, "CommitAsync", LifecycleTask, [[BoundaryCancellation]]); + CheckLifecycleDeclarations(type, "RollbackAsync", LifecycleTask, [[BoundaryCancellation]]); + CheckLifecycleDeclarations(type, "SetTenantContextAsync", LifecycleTask, + [["LearnStack.SharedKernel.Tenancy.ITenantContext", BoundaryCancellation]]); + } + foreach (var type in new[] { scope, frame }) + { + CheckLifecycleDeclarations(type, "CompleteAsync", LifecycleTask, [[BoundaryCancellation]]); + CheckLifecycleDeclarations(type, "FailAsync", LifecycleTask, [[BoundaryCancellation]]); + } + offenders.UnionWith(IntegrationLifecycleCallsiteOffenders(types, CallSites(census))); + return offenders; + + void CheckLifecycleDeclarations(TypeDefinition type, string name, string returnType, string[][] signatures) + { + var declarations = type.Methods.Where(method => method.Name == name).ToArray(); + if (declarations.Length != signatures.Length || declarations.Any(method => method.IsStatic || !method.IsPublic || method.ReturnType.FullName != returnType) + || signatures.Any(signature => declarations.Count(method => ParametersMatch(method, signature)) != 1)) + offenders.Add("Unexpected lifecycle declaration: " + TypeId(type) + "." + name); + } + } + + private static SortedSet IntegrationLifecycleCallsiteOffenders( + Dictionary<(string Assembly, string Type), TypeDefinition> types, + IEnumerable<(string Owner, MethodReference Target)> sites) + { + var offenders = new SortedSet(StringComparer.Ordinal); + var expected = new Dictionary(StringComparer.Ordinal) + { + ["unit.begin"] = [Owner(InfrastructureAssembly, LifecycleAcceptance, "AcceptAsync")], + ["unit.begin-mode"] = [Owner(DeliveryApplication, DeliveryCoordinator, "ExecuteAsync"), + Owner(DeliveryApplication, DeliveryPrepared, "CreateAsync"), Owner(InfrastructureAssembly, LifecycleAdapter, "BeginTransactionAsync", [BoundaryCancellation])], + ["unit.commit"] = [Owner(InfrastructureAssembly, LifecycleFrame, "CompleteAsync")], + ["unit.rollback"] = [Owner(InfrastructureAssembly, LifecycleFrame, "FailAsync")], + ["unit.announce"] = [Owner(DeliveryApplication, DeliveryCoordinator, "ExecuteFrameAsync"), + Owner(DeliveryApplication, DeliveryPrepared, "CreateAsync"), Owner(InfrastructureAssembly, LifecycleAcceptance, "AcceptAsync")], + ["scope.complete"] = [Owner(DeliveryApplication, DeliveryCoordinator, "ExecuteFrameAsync"), Owner(InfrastructureAssembly, LifecycleAcceptance, "AcceptAsync")], + ["scope.fail"] = [Owner(DeliveryApplication, DeliveryCoordinator, "ExecuteFrameAsync"), Owner(DeliveryApplication, DeliveryCoordinator, "ExecuteFrameAsync"), + Owner(DeliveryApplication, DeliveryPrepared, "CreateAsync"), Owner(InfrastructureAssembly, LifecycleAcceptance, "AcceptAsync"), Owner(InfrastructureAssembly, LifecycleFrame, "DisposeAsync")], + ["database.CommitAsync"] = + [ + Owner(InfrastructureAssembly, LifecycleAdapter, "CommitFrameAsync"), + Owner(InfrastructureAssembly, Store, "AcquireAsync"), Owner(InfrastructureAssembly, Store, "TryTerminalizeAsync"), + Owner(InfrastructureAssembly, DeliveryStore, "AcquireAsync"), Owner(InfrastructureAssembly, DeliveryStore, "UpdateAsync"), + Owner(InfrastructureAssembly, DeliveryStore, "TryTerminalizeAsync"), Owner(InfrastructureAssembly, DeliveryStore, "ReplayAsync"), + Owner(AuditAssembly, AuditStore, "WriteOwnTransactionAsync"), + Owner(AuditAssembly, "LearnStack.Infrastructure.Audit.AuditConfigService", "LoadAsync"), + Owner(InfrastructureAssembly, LifecyclePlatform, "RecordAsync"), Owner(InfrastructureAssembly, LifecyclePlatform + "/Handle", "CommitAsync"), + Owner(InfrastructureAssembly, "LearnStack.Infrastructure.MultiTenancy.CachedHostToTenantResolver", "ReadAsync"), + Owner(InfrastructureAssembly, "LearnStack.Infrastructure.MultiTenancy.OrganizationScopeValidator", "BelongsToTenantAsync"), + Owner("LearnStack.Modules.Tenancy.Infrastructure", "LearnStack.Modules.Tenancy.Infrastructure.FeatureFlags", "LoadTenantFlagsAsync"), + ], + ["database.RollbackAsync"] = [Owner(InfrastructureAssembly, LifecycleAdapter, "RollbackCoreAsync"), Owner(InfrastructureAssembly, LifecyclePlatform + "/Handle", "DisposeAsync")], + ["database.Commit"] = [], + ["database.Rollback"] = [], + }; + var actual = sites.Select(site => (site.Owner, site.Target, Capability: IntegrationLifecycleCapability(site.Target, types))) + .Where(site => site.Capability is not null).ToArray(); + foreach (var (capability, owners) in expected) + CheckSet(offenders, "typed lifecycle " + capability, actual.Where(site => site.Capability == capability).Select(site => site.Owner), owners); + foreach (var site in actual.Where(site => !expected.ContainsKey(site.Capability!))) + offenders.Add("Unexpected lifecycle signature: " + site.Owner + " -> " + site.Target.FullName); + return offenders; + + string Owner(string assembly, string typeName, string name, string[]? parameters = null) + { + if (!types.TryGetValue((assembly, typeName), out var type)) + { + offenders.Add("Missing actual lifecycle owner: " + typeName); + return "missing:" + typeName + "." + name; + } + var methods = type.Methods.Where(method => method.Name == name && (parameters is null || ParametersMatch(method, parameters))).ToArray(); + if (methods.Length == 1) return MethodId(methods[0]); + offenders.Add("Missing or ambiguous lifecycle owner: " + typeName + "." + name); + return "missing:" + typeName + "." + name; + } + } + + private static string? IntegrationLifecycleCapability(MethodReference target, + Dictionary<(string Assembly, string Type), TypeDefinition> types) + { + var name = target.Name[(target.Name.LastIndexOf('.') + 1)..]; + var cancellation = ParametersMatch(target, [BoundaryCancellation]); + if (types.TryGetValue((KernelAssembly, LifecycleUnit), out var unit) + && (SameType(target.DeclaringType, unit) || Derives(target.DeclaringType, unit, types))) + return name switch + { + "BeginTransactionAsync" when cancellation && target.ReturnType.FullName == LifecycleScopeTask => "unit.begin", + "BeginTransactionAsync" when ParametersMatch(target, ["LearnStack.SharedKernel.Persistence.TransactionMode", BoundaryCancellation]) + && target.ReturnType.FullName == LifecycleScopeTask => "unit.begin-mode", + "CommitAsync" when cancellation && target.ReturnType.FullName == LifecycleTask => "unit.commit", + "RollbackAsync" when cancellation && target.ReturnType.FullName == LifecycleTask => "unit.rollback", + "SetTenantContextAsync" when ParametersMatch(target, ["LearnStack.SharedKernel.Tenancy.ITenantContext", BoundaryCancellation]) + && target.ReturnType.FullName == LifecycleTask => "unit.announce", + "BeginTransactionAsync" or "CommitAsync" or "RollbackAsync" or "SetTenantContextAsync" => "invalid-unit", + _ => null, + }; + if (types.TryGetValue((KernelAssembly, LifecycleScope), out var scope) + && (SameType(target.DeclaringType, scope) || Derives(target.DeclaringType, scope, types))) + return name switch + { + "CompleteAsync" when cancellation && target.ReturnType.FullName == LifecycleTask => "scope.complete", + "FailAsync" when cancellation && target.ReturnType.FullName == LifecycleTask => "scope.fail", + "CompleteAsync" or "FailAsync" => "invalid-scope", + _ => null, + }; + if (LifecycleFacadeTerminalMethods.Contains(name, StringComparer.Ordinal) + && IsIntegrationDatabaseType(target.DeclaringType, types, + [("Microsoft.EntityFrameworkCore", "Microsoft.EntityFrameworkCore.Infrastructure.DatabaseFacade")])) + name = name.Replace("Transaction", string.Empty, StringComparison.Ordinal); + else if (!LifecycleTerminalMethods.Contains(name, StringComparer.Ordinal) || !IsIntegrationDatabaseTransaction(target.DeclaringType, types)) return null; + if ((name is "CommitAsync" or "RollbackAsync") && cancellation && target.ReturnType.FullName == LifecycleTask + || (name is "Commit" or "Rollback") && target.Parameters.Count == 0 && target.ReturnType.FullName == "System.Void") + return "database." + name; + return "invalid-database"; + } + + private static bool IsIntegrationDatabaseTransaction(TypeReference type, + Dictionary<(string Assembly, string Type), TypeDefinition> types) => + IsIntegrationDatabaseType(type, types, + [ + ("System.Data.Common", "System.Data.Common.DbTransaction"), + ("System.Data.Common", "System.Data.IDbTransaction"), + ("Npgsql", "Npgsql.NpgsqlTransaction"), + ("Microsoft.EntityFrameworkCore", "Microsoft.EntityFrameworkCore.Storage.IDbContextTransaction"), + ("Microsoft.EntityFrameworkCore.Relational", "Microsoft.EntityFrameworkCore.Storage.RelationalTransaction"), + ]); + + private static bool IsIntegrationDatabaseType(TypeReference type, + Dictionary<(string Assembly, string Type), TypeDefinition> types, (string Assembly, string Type)[] contracts) + { + var pending = new Queue(); + var visited = new HashSet<(string Assembly, string Type)>(); + pending.Enqueue(type); + while (pending.TryDequeue(out var current)) + { + if (!visited.Add(Key(current))) continue; + if (contracts.Contains(Key(current))) return true; + if (!types.TryGetValue(Key(current), out var definition)) continue; + if (definition.BaseType is { } parent) pending.Enqueue(parent); + foreach (var contract in definition.Interfaces) pending.Enqueue(contract.InterfaceType); + } + return false; + } + + private interface IIntegrationLifecycleProbeUnit : IUnitOfWork + { + new Task CommitAsync(CancellationToken cancellationToken = default); + } + private abstract class IntegrationLifecycleProbeTransaction : DbTransaction + { + public override Task CommitAsync(CancellationToken cancellationToken = default) => Task.CompletedTask; + } + private interface IIntegrationLifecycleProbeHelper { Task Run(IUnitOfWork unit); } + private sealed class IntegrationLifecycleProbeHelper : IIntegrationLifecycleProbeHelper + { + public Task Run(IUnitOfWork unit) => unit.CommitAsync(); + } + private static class IntegrationLifecycleProbeCalls + { + public static Task Commit(IUnitOfWork unit) => unit.CommitAsync(); + public static Task Rollback(IUnitOfWork unit) => unit.RollbackAsync(); + public static Task Begin(IUnitOfWork unit) => unit.BeginTransactionAsync(); + public static Task Complete(IUnitOfWorkScope frame) => frame.CompleteAsync(); + public static Task Fail(IUnitOfWorkScope frame) => frame.FailAsync(); + public static Task Concrete(NpgsqlUnitOfWork unit) => unit.CommitAsync(); + public static Task RawCommit(DbTransaction transaction) => transaction.CommitAsync(); + public static void RawRollback(IDbTransaction transaction) => transaction.Rollback(); + public static Task ConcreteRaw(NpgsqlTransaction transaction) => transaction.CommitAsync(); + public static Task InheritedUnit(IIntegrationLifecycleProbeUnit unit) => unit.CommitAsync(); + public static Task InheritedRaw(IntegrationLifecycleProbeTransaction transaction) => transaction.CommitAsync(); + public static Task ViaHelper(IIntegrationLifecycleProbeHelper helper, IUnitOfWork unit) => helper.Run(unit); + public static async Task Async(IUnitOfWork unit) { await Task.Yield(); await unit.CommitAsync(); } + public static Func Capture(IUnitOfWork unit) => unit.CommitAsync; + public static Task Announce(IUnitOfWork unit, ITenantContext context) => unit.SetTenantContextAsync(context); + public static Task ConcreteAnnounce(NpgsqlUnitOfWork unit, ITenantContext context) => unit.SetTenantContextAsync(context); + public static Task InheritedAnnounce(IIntegrationLifecycleProbeUnit unit, ITenantContext context) => unit.SetTenantContextAsync(context); + public static async Task AsyncAnnounce(IUnitOfWork unit, ITenantContext context) { await Task.Yield(); await unit.SetTenantContextAsync(context); } + public static Func CaptureAnnouncement(IUnitOfWork unit) => unit.SetTenantContextAsync; + public static void EfCommit(IDbContextTransaction transaction) => transaction.Commit(); + public static void EfRollback(IDbContextTransaction transaction) => transaction.Rollback(); + public static Task EfCommitAsync(IDbContextTransaction transaction) => transaction.CommitAsync(); + public static Task EfRollbackAsync(IDbContextTransaction transaction) => transaction.RollbackAsync(); + public static Task EfConcreteCommitAsync(RelationalTransaction transaction) => transaction.CommitAsync(); + public static Task EfCurrentCommitAsync(DbContext context) => context.Database.CurrentTransaction!.CommitAsync(); + public static void EfFacadeCommit(DatabaseFacade database) => database.CommitTransaction(); + public static void EfFacadeRollback(DatabaseFacade database) => database.RollbackTransaction(); + public static Task EfFacadeCommitAsync(DatabaseFacade database) => database.CommitTransactionAsync(); + public static Task EfFacadeRollbackAsync(DatabaseFacade database) => database.RollbackTransactionAsync(); + public static async Task EfAsync(IDbContextTransaction transaction) { await Task.Yield(); await transaction.CommitAsync(); } + public static Func EfCapture(IDbContextTransaction transaction) => transaction.CommitAsync; + } + private static class IntegrationLifecycleProbeClean + { + public static DbTransaction? Read(IUnitOfWork unit) => unit.Transaction; + public static bool ReadFrame(IUnitOfWorkScope frame) => frame.IsOwner; + public static void Refuse(IUnitOfWork unit) => unit.MarkRollbackOnly(); + public static IDbContextTransaction? ReadEfTransaction(DbContext context) => context.Database.CurrentTransaction; + public static Guid ReadEfIdentity(IDbContextTransaction transaction) => transaction.TransactionId; + } +} diff --git a/backend/tests/LearnStack.Tests.Architecture/OutboxDispatchBoundaryTests.cs b/backend/tests/LearnStack.Tests.Architecture/OutboxDispatchBoundaryTests.cs index 1475aa5f..5821bf0b 100644 --- a/backend/tests/LearnStack.Tests.Architecture/OutboxDispatchBoundaryTests.cs +++ b/backend/tests/LearnStack.Tests.Architecture/OutboxDispatchBoundaryTests.cs @@ -19,6 +19,9 @@ public sealed partial class OutboxDispatchConventionTests private const string BoundarySource = "LearnStack.Infrastructure.Messaging.OutboxDispatcherDataSource"; private const string BoundaryWorker = "LearnStack.Api.Messaging.OutboxProcessor"; private const string BoundaryAttempt = "LearnStack.Api.Messaging.OutboxPublishAttempt"; + private const string DeliveryWorker = "LearnStack.Api.Messaging.DeliveryProcessor"; + private const string DeliveryAttempt = "LearnStack.Api.Messaging.DeliveryAttempt"; + private const string DeliveryClaimType = "LearnStack.Infrastructure.Messaging.DeliveryClaim"; private const string BoundaryPlatform = "LearnStack.Infrastructure.MultiTenancy.PlatformAdminScope"; private const string BoundaryBus = "LearnStack.SharedKernel.Messaging.IEventBus"; private const string BoundaryTransport = "LearnStack.Infrastructure.Messaging.InProcessEventBus"; @@ -126,7 +129,7 @@ public void Credential_census_rejects_generic_direct_delegate_async_and_extra_he return () => worker.Methods.Remove(method); }, CredentialOffenders); var boundaryWorker = census.Types[(BoundaryApi, BoundaryWorker)]; - foreach (var name in new[] { "ExecuteAsync", "DispatchAsync", "DispatchNormalAsync", "RefreshAsync" }) + foreach (var name in new[] { "ExecuteAsync", "ClaimNextAsync", "DispatchAsync", "DispatchNormalAsync", "RefreshAsync" }) AssertMutation(census, "a public store-forwarding overload borrowing the approved worker name " + name, () => AddBoundaryFieldGetter(boundaryWorker, name, "_store"), CredentialOffenders); AssertMutation(census, "a public raw-source overload borrowing the approved worker name StartAsync", () => @@ -486,7 +489,7 @@ private static SortedSet CredentialOffenders(ProductionCensus census) var types = census.Types; CheckSet(offenders, "production assemblies", census.Modules.Select(module => module.Assembly.Name.Name), ProductionAssemblies.Names()); foreach (var subject in new[] { (BoundaryApi, BoundaryComposition), (BoundaryApi, BoundaryPersistence), - (BoundaryApi, BoundaryWorker), (InfrastructureAssembly, BoundaryPlatform), + (BoundaryApi, BoundaryWorker), (BoundaryApi, DeliveryWorker), (InfrastructureAssembly, DeliveryStore), (InfrastructureAssembly, BoundaryPlatform), (InfrastructureAssembly, BoundarySource), (InfrastructureAssembly, Store) }) if (!types.ContainsKey(subject)) offenders.Add("Missing credential subject: " + subject.Item2); if (offenders.Count != 0) return offenders; @@ -547,6 +550,18 @@ private static SortedSet CredentialOffenders(ProductionCensus census) CheckLiteralKeyResolver(offenders, storeFactory, BoundaryLazySource, OutboxDispatcherDataSource.Key); } + var deliveryDescriptor = services.Single(descriptor => !descriptor.IsKeyedService + && descriptor.ServiceType == typeof(DeliveryDispatchStore)); + var deliveryFactory = FindRuntimeMethod(census, deliveryDescriptor.ImplementationFactory!.Method); + if (deliveryFactory is null) offenders.Add("Missing actual delivery-store factory"); + else + { + expectedResolvers.Add(MethodId(deliveryFactory)); + if (!DelegateTargets(outboxRoot).Any(target => MethodId(target) == MethodId(deliveryFactory))) + offenders.Add("Delivery store factory must be registered by the outbox root"); + CheckLiteralKeyResolver(offenders, deliveryFactory, BoundaryLazySource, OutboxDispatcherDataSource.Key); + } + var sites = CallSites(census).ToArray(); CheckSet(offenders, "keyed resolver callers", sites.Where(site => IsKeyedResolution(site.Target)).Select(site => site.Owner), expectedResolvers); CheckSet(offenders, "keyed registration callers", sites.Where(site => IsKeyedRegistration(site.Target)).Select(site => site.Owner), expectedFactories); @@ -600,7 +615,8 @@ private static SortedSet CredentialOffenders(ProductionCensus census) var scopeConstructor = types[(InfrastructureAssembly, BoundaryPlatform)].Methods.Single(method => method.IsConstructor && !method.IsStatic); CheckSet(offenders, "keyed parameter bindings", injections, [workerConstructor + ":" + BoundaryLazySource + ":" + OutboxDispatcherDataSource.Key, - MethodId(scopeConstructor) + ":" + BoundaryLazySource + ":" + PlatformAdminScope.PlatformDataSourceKey]); + MethodId(scopeConstructor) + ":" + BoundaryLazySource + ":" + PlatformAdminScope.PlatformDataSourceKey, + DeliveryWorkerConstructor(offenders, types[(BoundaryApi, DeliveryWorker)]) + ":" + BoundaryLazySource + ":" + OutboxDispatcherDataSource.Key]); CheckSourceFieldAccess(offenders, census, worker, workerConstructor, [RequireBoundaryMethod(offenders, worker, "StartAsync", BoundaryTask, MethodAttributes.Public, BoundaryCancellation)]); var store = types[(InfrastructureAssembly, Store)]; @@ -621,6 +637,7 @@ private static SortedSet CredentialOffenders(ProductionCensus census) MethodAttributes.Public, BoundaryStrings, BoundaryCancellation), ]); if (storeFactory is not null) CheckStoreConsumers(offenders, census, outboxRoot, storeFactory); + CheckDeliveryCredentials(offenders, census, outboxRoot, deliveryFactory); return offenders; } @@ -639,7 +656,8 @@ private static void CheckStoreConsumers(SortedSet offenders, ProductionC var workerMethods = new[] { BoundaryWorkerConstructor(offenders, worker), - RequireBoundaryMethod(offenders, worker, "ExecuteAsync", BoundaryTask, MethodAttributes.Family, BoundaryCancellation), + RequireBoundaryMethod(offenders, worker, "ClaimNextAsync", "System.Threading.Tasks.Task`1<" + BoundaryClaim + ">", + MethodAttributes.Private, "System.String[]", BoundaryCancellation), RequireBoundaryMethod(offenders, worker, "DispatchAsync", BoundaryTask, MethodAttributes.Private, BoundaryClaim, BoundaryCancellation), RequireBoundaryMethod(offenders, worker, "DispatchNormalAsync", BoundaryTask, MethodAttributes.Private, BoundaryClaim, BoundaryCancellation), RequireBoundaryMethod(offenders, worker, "RefreshAsync", BoundaryTask, MethodAttributes.Private, "System.Boolean", BoundaryCancellation), diff --git a/backend/tests/LearnStack.Tests.Architecture/OutboxDispatchConventionTests.cs b/backend/tests/LearnStack.Tests.Architecture/OutboxDispatchConventionTests.cs index 956bf997..25dd4ad2 100644 --- a/backend/tests/LearnStack.Tests.Architecture/OutboxDispatchConventionTests.cs +++ b/backend/tests/LearnStack.Tests.Architecture/OutboxDispatchConventionTests.cs @@ -17,6 +17,8 @@ public sealed partial class OutboxDispatchConventionTests private const string AuditPort = "LearnStack.SharedKernel.Audit.IAuditStore"; private const string AuditStore = "LearnStack.Infrastructure.Audit.PostgresAuditStore"; private const string Mint = "FromProducerUpdate"; + private const string SubscriberMint = "FromSubscriberUpdate"; + private const string DeliveryStore = "LearnStack.Infrastructure.Messaging.DeliveryDispatchStore"; private const string Terminalize = "TryTerminalizeAsync"; private const string TerminalWrite = "WriteEventTerminalAsync"; private static readonly string[] MintParameters = @@ -34,7 +36,7 @@ public sealed partial class OutboxDispatchConventionTests [ "LearnStack.Infrastructure.Messaging.OutboxEventRegistry", "LearnStack.Infrastructure.Messaging.OutboxEnvelopeReader", - Store, + Store, DeliveryStore, ]; /// @@ -47,10 +49,19 @@ public void Outbox_Terminal_Audit_Identity_Is_Row_Bound() { using var census = ProductionCensus.Read(); TerminalOffenders(census).Should().BeEmpty( - "Fix: keep the immutable private witness, its sole dispatcher minting subtype, " + "Fix: keep the immutable private witness, its exact dispatcher minting subtypes, " + "and the terminal writer/call sites closed; keep row/transaction identity checks in the real audit seam"); } + /// ADR-0046 subscriber terminal witness uses the same closed actual-row authority. + [Fact] + public void Delivery_Terminal_Audit_Identity_Is_Row_Bound() + { + using var census = ProductionCensus.Read(); + TerminalOffenders(census).Should().BeEmpty( + "Fix: mint subscriber witnesses only from DeliveryDispatchStore's successful fenced UPDATE and write through the fifth seam"); + } + [Fact] public void Terminal_census_detects_extra_derivers_constructors_and_mint_callers() { @@ -183,11 +194,11 @@ public void Terminal_census_detects_mutability_accessibility_missing_subjects_an parameter.ParameterType = witness.Module.TypeSystem.String; return () => parameter.ParameterType = original; }); - foreach (var name in new[] { Mint, TerminalWrite }) + foreach (var name in new[] { Mint, SubscriberMint, TerminalWrite }) { AssertTerminalMutation(census, "an empty " + name + " call-site census", () => { - var owner = census.Types.Values.SelectMany(type => type.Methods).Single(method => method.HasBody + var owner = census.Types.Values.SelectMany(type => type.Methods).First(method => method.HasBody && method.Body.Instructions.Any(instruction => instruction.Operand is MethodReference target && target.Name == name)); var instruction = owner.Body.Instructions.Single(instruction => instruction.Operand is MethodReference target @@ -320,7 +331,7 @@ private static SortedSet TerminalOffenders(ProductionCensus census) var types = census.Types; CheckSet(offenders, "production assemblies", census.Modules.Select(module => module.Assembly.Name.Name), ProductionAssemblies.Names()); foreach (var key in new[] { (KernelAssembly, Witness), (KernelAssembly, MintingSource), - (InfrastructureAssembly, Store), (KernelAssembly, AuditPort), (AuditAssembly, AuditStore) }) + (InfrastructureAssembly, Store), (InfrastructureAssembly, DeliveryStore), (KernelAssembly, AuditPort), (AuditAssembly, AuditStore) }) { if (!types.ContainsKey(key)) offenders.Add("Missing production subject: " + key.Item2); } @@ -329,11 +340,12 @@ private static SortedSet TerminalOffenders(ProductionCensus census) var witness = types[(KernelAssembly, Witness)]; var source = types[(KernelAssembly, MintingSource)]; var store = types[(InfrastructureAssembly, Store)]; + var deliveryStore = types[(InfrastructureAssembly, DeliveryStore)]; var port = types[(KernelAssembly, AuditPort)]; var auditStore = types[(AuditAssembly, AuditStore)]; var constructors = witness.Methods.Where(method => method.IsConstructor && !method.IsStatic).ToArray(); if (!witness.IsSealed || witness.BaseType?.FullName != "System.Object") offenders.Add(Witness + ": must be sealed with no custom base"); - if (constructors.Length != 1 || constructors.Any(method => !method.IsPrivate || !ParametersMatch(method, MintParameters))) + if (constructors.Length != 1 || constructors.Any(method => !method.IsPrivate || !ParametersMatch(method, [.. MintParameters, "System.Boolean"]))) offenders.Add(Witness + ": requires exactly one private live-reader constructor"); if (witness.Fields.All(field => field.IsStatic) || witness.Fields.Any(field => !field.IsStatic && !field.IsInitOnly)) offenders.Add(Witness + ": instance fields must be nonempty and readonly"); @@ -347,13 +359,15 @@ private static SortedSet TerminalOffenders(ProductionCensus census) attribute.AttributeType.FullName == "System.Runtime.CompilerServices.InternalsVisibleToAttribute")) offenders.Add(KernelAssembly + ": friend assemblies reopen restricted authority"); - var factories = source.Methods.Where(method => method.Name == Mint).ToArray(); - if (factories.Length != 1 || factories.Any(method => !method.IsFamily || !method.IsStatic + var factories = source.Methods.Where(method => method.Name is Mint or SubscriberMint).ToArray(); + if (factories.Length != 2 || factories.Select(method => method.Name).Distinct(StringComparer.Ordinal).Count() != 2 || factories.Any(method => !method.IsFamily || !method.IsStatic || method.ReturnType.FullName != Witness || !ParametersMatch(method, MintParameters))) offenders.Add(MintingSource + "." + Mint + ": requires the protected static live-reader mint signature"); - var terminalizers = store.Methods.Where(method => method.Name == Terminalize).ToArray(); - if (terminalizers.Length != 1) offenders.Add(Store + "." + Terminalize + ": requires one concrete terminalization owner"); - CheckSet(offenders, "minting subtypes", types.Values.Where(type => Derives(type, source, types)).Select(TypeId), [TypeId(store)]); + var producerTerminalizers = store.Methods.Where(method => method.Name == Terminalize).ToArray(); + var subscriberTerminalizers = deliveryStore.Methods.Where(method => method.Name == Terminalize).ToArray(); + var terminalizers = producerTerminalizers.Concat(subscriberTerminalizers).ToArray(); + if (producerTerminalizers.Length != 1 || subscriberTerminalizers.Length != 1) offenders.Add(Store + "." + Terminalize + ": requires one concrete terminalization owner"); + CheckSet(offenders, "minting subtypes", types.Values.Where(type => Derives(type, source, types)).Select(TypeId), [TypeId(store), TypeId(deliveryStore)]); CheckSet(offenders, "witness-producing methods", types.Values.SelectMany(type => type.Methods) .Where(method => method.ReturnType.FullName == Witness).Select(MethodId), factories.Select(MethodId)); @@ -374,9 +388,10 @@ private static SortedSet TerminalOffenders(ProductionCensus census) var sites = CallSites(census).ToArray(); CheckSet(offenders, "witness constructor call sites", sites.Where(site => site.Target.Name == ".ctor" && SameType(site.Target.DeclaringType, witness)).Select(site => site.Owner), factories.Select(MethodId)); - CheckSet(offenders, "mint call sites", sites.Where(site => site.Target.Name == Mint - && (SameType(site.Target.DeclaringType, source) || Derives(site.Target.DeclaringType, source, types))) - .Select(site => site.Owner), terminalizers.Select(MethodId)); + foreach (var (mint, owners) in new[] { (Mint, producerTerminalizers), (SubscriberMint, subscriberTerminalizers) }) + CheckSet(offenders, mint + " call sites", sites.Where(site => site.Target.Name == mint + && (SameType(site.Target.DeclaringType, source) || Derives(site.Target.DeclaringType, source, types))) + .Select(site => site.Owner), owners.Select(MethodId)); CheckSet(offenders, "terminal audit call sites", sites.Where(site => IsTerminalWriteName(site.Target) && (SameType(site.Target.DeclaringType, port) || Derives(site.Target.DeclaringType, port, types))) .Select(site => site.Owner), terminalizers.Select(MethodId)); diff --git a/backend/tests/LearnStack.Tests.Architecture/OutboxDispatchObservationTests.cs b/backend/tests/LearnStack.Tests.Architecture/OutboxDispatchObservationTests.cs index 55080d44..f5847d10 100644 --- a/backend/tests/LearnStack.Tests.Architecture/OutboxDispatchObservationTests.cs +++ b/backend/tests/LearnStack.Tests.Architecture/OutboxDispatchObservationTests.cs @@ -3,6 +3,8 @@ using LearnStack.Api.Composition; using LearnStack.Api.Messaging; using LearnStack.Infrastructure.Messaging; +using LearnStack.SharedKernel.Hosting; +using Microsoft.AspNetCore.Builder; using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Diagnostics.HealthChecks; @@ -341,10 +343,20 @@ private static SortedSet ObservationOffenders(ProductionCensus census) return offenders; } - private static ServiceCollection ObservationServices() + private static IServiceCollection ObservationServices() { - var services = new ServiceCollection(); - services.AddSingleton(IntegrationEventHandlerRegistry.Discover()); + var builder = WebApplication.CreateBuilder([]); + builder.Configuration.Sources.Clear(); + builder.AddLearnStackCrossCuttingFoundation(DeploymentMode.Development, + typeof(LearnStack.Modules.Customization.Application.AssemblyMarker).Assembly); + IServiceCollection services = new ServiceCollection(); + // Import the construction-free manifest from its real registration site. + // This guard isolates persistence's worker/check set; unrelated framework + // hosted services from Foundation are not its subject. + foreach (var descriptor in builder.Services.Where(descriptor => + descriptor.ServiceType == typeof(IntegrationEventHandlerRegistry) + || descriptor.ServiceType == typeof(OutboxEventRegistry))) + services.Add(descriptor); services.AddLearnStackPersistence(new ConfigurationBuilder().Build()); return services; } @@ -356,7 +368,8 @@ private static SortedSet ObservationRegistrationOffenders(IServiceCollec // factory/instance duplicates cannot hide behind the dispatcher type filter. var workers = services.Where(descriptor => descriptor.ServiceType == typeof(IHostedService)).ToArray(); CheckSet(offenders, "hosted-service descriptor types", workers.Select(descriptor => descriptor.ImplementationType?.FullName ?? "non-type registration"), - [typeof(OutboxProcessor).FullName!, "Microsoft.Extensions.Diagnostics.HealthChecks.HealthCheckPublisherHostedService"]); + [typeof(OutboxProcessor).FullName!, typeof(DeliveryProcessor).FullName!, + "Microsoft.Extensions.Diagnostics.HealthChecks.HealthCheckPublisherHostedService"]); if (workers.Any(descriptor => descriptor.Lifetime != ServiceLifetime.Singleton)) offenders.Add("Mandatory singleton worker registration"); // The helper graph walks production contracts. Its one framework-owned // virtual clock capability is admitted only as the shipped system instance. diff --git a/backend/tests/LearnStack.Tests.Architecture/PlatformAdminScopeConventionTests.cs b/backend/tests/LearnStack.Tests.Architecture/PlatformAdminScopeConventionTests.cs index be114051..c7d1dd54 100644 --- a/backend/tests/LearnStack.Tests.Architecture/PlatformAdminScopeConventionTests.cs +++ b/backend/tests/LearnStack.Tests.Architecture/PlatformAdminScopeConventionTests.cs @@ -35,6 +35,7 @@ public sealed class PlatformAdminScopeConventionTests private const string OutboxCompositionFile = "LearnStack.Api/Composition/OutboxCompositionExtensions.cs"; private const string OutboxWorkerFile = "LearnStack.Api/Messaging/OutboxProcessor.cs"; + private const string DeliveryWorkerFile = "LearnStack.Api/Messaging/DeliveryProcessor.cs"; [Fact] public void Platform_DataSource_Resolved_Only_By_PlatformAdminScope() @@ -57,7 +58,7 @@ public void Platform_DataSource_Resolved_Only_By_PlatformAdminScope() .ToList(); resolvers.Should().BeEquivalentTo( - [ScopeFile, CompositionFile, OutboxCompositionFile, OutboxWorkerFile], + [ScopeFile, CompositionFile, OutboxCompositionFile, OutboxWorkerFile, DeliveryWorkerFile], "the platform scope and the separately confined dispatcher are the only " + "credential consumers, alongside their composition roots"); @@ -69,6 +70,14 @@ public void Platform_DataSource_Resolved_Only_By_PlatformAdminScope() File.ReadAllText(Path.Combine(SourceScan.SourceRoot, OutboxWorkerFile)) .Should().NotContain("PlatformAdmin"); + typeof(DeliveryProcessor).GetConstructors().SelectMany(constructor => constructor.GetParameters()) + .Select(parameter => parameter.GetCustomAttribute()) + .OfType().Should().ContainSingle().Which.Key + .Should().Be(OutboxDispatcherDataSource.Key, + "the subscriber scheduler reuses only the confined dispatcher credential"); + File.ReadAllText(Path.Combine(SourceScan.SourceRoot, DeliveryWorkerFile)) + .Should().NotContain("PlatformAdmin"); + // Leg 2 — IConfiguration.GetConnectionString is called in exactly one file under // backend/src. Not "every credential is read in one file": the two design-time // DbContext factories read ConnectionStrings__Migration straight from the diff --git a/backend/tests/LearnStack.Tests.Architecture/ProductionAssemblies.cs b/backend/tests/LearnStack.Tests.Architecture/ProductionAssemblies.cs index 182ef35d..f03ea4c8 100644 --- a/backend/tests/LearnStack.Tests.Architecture/ProductionAssemblies.cs +++ b/backend/tests/LearnStack.Tests.Architecture/ProductionAssemblies.cs @@ -19,8 +19,12 @@ namespace LearnStack.Tests.Architecture; /// internal static class ProductionAssemblies { - public static IEnumerable Names() => - Directory.EnumerateFiles(RepositoryPaths.BackendSrc(), "LearnStack.*.csproj", SearchOption.AllDirectories) + public static IEnumerable Names() => Names(RepositoryPaths.BackendSrc()); + + internal static IEnumerable Names(string sourceRoot) => + Directory.EnumerateFiles(sourceRoot, "*.csproj", SearchOption.AllDirectories) + .Where(path => !path.Contains( + $"{Path.DirectorySeparatorChar}obj{Path.DirectorySeparatorChar}", StringComparison.Ordinal)) .Select(Path.GetFileNameWithoutExtension) .Where(name => !string.IsNullOrEmpty(name)) .Select(name => name!) diff --git a/backend/tests/LearnStack.Tests.Architecture/ProductionAssembliesTests.cs b/backend/tests/LearnStack.Tests.Architecture/ProductionAssembliesTests.cs new file mode 100644 index 00000000..d4b26476 --- /dev/null +++ b/backend/tests/LearnStack.Tests.Architecture/ProductionAssembliesTests.cs @@ -0,0 +1,29 @@ +using FluentAssertions; +using Xunit; + +namespace LearnStack.Tests.Architecture; + +public sealed class ProductionAssembliesTests +{ + [Fact] + public void Project_census_includes_every_project_name_and_excludes_generated_projects() + { + var directory = Directory.CreateTempSubdirectory("learnstack-project-census-"); + try + { + File.WriteAllText(Path.Combine(directory.FullName, "LearnStack.Existing.csproj"), ""); + var added = Directory.CreateDirectory(Path.Combine(directory.FullName, "module")); + File.WriteAllText(Path.Combine(added.FullName, "OtherProduction.csproj"), ""); + var generated = Directory.CreateDirectory(Path.Combine(added.FullName, "obj", "nested")); + File.WriteAllText(Path.Combine(generated.FullName, "Generated.csproj"), ""); + + ProductionAssemblies.Names(directory.FullName).Should() + .Equal(["LearnStack.Existing", "OtherProduction"], + "a differently named production project must not silently escape every assembly census"); + } + finally + { + directory.Delete(recursive: true); + } + } +} diff --git a/backend/tests/LearnStack.Tests.Architecture/TenantContextConstructionTests.cs b/backend/tests/LearnStack.Tests.Architecture/TenantContextConstructionTests.cs index b88c16c8..2c55e3b3 100644 --- a/backend/tests/LearnStack.Tests.Architecture/TenantContextConstructionTests.cs +++ b/backend/tests/LearnStack.Tests.Architecture/TenantContextConstructionTests.cs @@ -165,7 +165,8 @@ public void SetTenant_Callers_Are_The_Enumerated_Four() // `ITenantContextAccessor.Current`, and writes to it have exactly four // callers — TenantResolverMiddleware (HTTP), HubCorrelationMiddleware // (/api/internal/*), the Hangfire JobActivator (jobs) and the outbox / inbox - // handler scope (integration events). Two exist today; the rule is written + // handler scope (integration events). HTTP and the acceptance/execution + // halves of the integration-event scope exist today; the rule is written // over the whole set so the third to arrive is a deliberate edit here rather // than a silent addition there. // @@ -197,9 +198,10 @@ public void SetTenant_Callers_Are_The_Enumerated_Four() [ "LearnStack.Api/Tenancy/TenantResolverMiddleware.cs", "LearnStack.Infrastructure/Messaging/InProcessEventBus.cs", + "LearnStack.Infrastructure/Messaging/DeliveryExecutor.cs", ], - "two of the four enumerated writers have landed — the HTTP one and the " - + "integration-event handler scope. HubCorrelationMiddleware and the " + "two of the four logical writers have landed — HTTP and the integration-event " + + "acceptance/execution scopes. HubCorrelationMiddleware and the " + "Hangfire JobActivator are later phases, and a fifth writer is how a " + "request runs under a tenant nothing resolved"); } diff --git a/backend/tests/LearnStack.Tests.Contract/DevelopmentWebApplicationFactory.cs b/backend/tests/LearnStack.Tests.Contract/DevelopmentWebApplicationFactory.cs index 31e7f255..8532bb59 100644 --- a/backend/tests/LearnStack.Tests.Contract/DevelopmentWebApplicationFactory.cs +++ b/backend/tests/LearnStack.Tests.Contract/DevelopmentWebApplicationFactory.cs @@ -19,9 +19,9 @@ namespace LearnStack.Tests.Contract; /// in CI without it. Packet 4 removed that gate — the document is the contract /// the SDK generates from and CI diffs, so it is served in every environment — /// and the environment override keeps the contract suite on one known environment. -/// This served-contract fixture also removes the dispatcher worker because it owns -/// no dispatcher database. Its retained private outbox health check intentionally -/// remains Unhealthy; the static /healthz liveness endpoint stays independent. +/// This served-contract fixture also removes the dispatcher workers because it owns +/// no dispatcher database. Its retained private outbox and delivery health checks +/// intentionally remain Unhealthy; the static /healthz liveness endpoint stays independent. /// public sealed class DevelopmentWebApplicationFactory : WebApplicationFactory { @@ -29,12 +29,13 @@ protected override void ConfigureWebHost(IWebHostBuilder builder) { builder.UseEnvironment(Environments.Development); // This served-contract proof owns no dispatcher database. Remove only - // that worker; production startup tests retain the mandatory registration. + // those two workers; production startup tests retain the mandatory registration. builder.ConfigureTestServices(services => { foreach (var descriptor in services.Where(descriptor => descriptor.ServiceType == typeof(IHostedService) - && descriptor.ImplementationType == typeof(OutboxProcessor)).ToArray()) + && (descriptor.ImplementationType == typeof(OutboxProcessor) + || descriptor.ImplementationType == typeof(DeliveryProcessor))).ToArray()) services.Remove(descriptor); }); } diff --git a/backend/tests/LearnStack.Tests.Integration/BackendDiscovery.cs b/backend/tests/LearnStack.Tests.Integration/BackendDiscovery.cs index 2d927140..c1c493e5 100644 --- a/backend/tests/LearnStack.Tests.Integration/BackendDiscovery.cs +++ b/backend/tests/LearnStack.Tests.Integration/BackendDiscovery.cs @@ -1,5 +1,6 @@ using System.Reflection; using LearnStack.SharedKernel.Audit; +using LearnStack.SharedKernel.Messaging; using MediatR; namespace LearnStack.Tests.Integration; @@ -24,6 +25,24 @@ [.. Types() public static List ShippedRequestTypes() => [.. HandlerContracts().Select(contract => contract.GenericTypeArguments[0]).Distinct()]; + /// + /// Every audit identity: request types for MediatR and concrete handler types for + /// integration delivery, including inherited contracts under ADR-0057. + /// + public static List ShippedAuditSubjects() => + [.. Types() + .Where(type => type is { IsAbstract: false, IsInterface: false, IsGenericTypeDefinition: false }) + .SelectMany(type => type.GetInterfaces() + .Where(contract => contract.IsGenericType + && (contract.GetGenericTypeDefinition() == typeof(IIntegrationEventHandler<>) + || contract.GetGenericTypeDefinition() == typeof(IRequestHandler<,>) + || contract.GetGenericTypeDefinition() == typeof(IRequestHandler<>))) + .Select(contract => contract.GetGenericTypeDefinition() == typeof(IIntegrationEventHandler<>) + ? type + : contract.GenericTypeArguments[0])) + .Distinct() + .OrderBy(type => type.FullName, StringComparer.Ordinal)]; + /// Every a backend assembly ships, constructed directly. public static List CatalogueSources() => [.. Types() @@ -36,18 +55,22 @@ private static IEnumerable Types() => .EnumerateFiles(BackendSrc(), "*.csproj", SearchOption.AllDirectories) .Where(path => !path.Contains( $"{Path.DirectorySeparatorChar}obj{Path.DirectorySeparatorChar}", StringComparison.Ordinal)) + .Order(StringComparer.Ordinal) .Select(path => Assembly.Load(Path.GetFileNameWithoutExtension(path))) - .SelectMany(LoadableTypes); + .SelectMany(TypesIn); - private static IEnumerable LoadableTypes(Assembly assembly) + private static Type[] TypesIn(Assembly assembly) { try { return assembly.GetTypes(); } - catch (ReflectionTypeLoadException partial) + catch (ReflectionTypeLoadException exception) { - return partial.Types.OfType(); + // A partial load is a failed census, never a smaller green population. + throw new InvalidOperationException( + $"Could not enumerate every type in {assembly.FullName}; the backend census is incomplete.", + exception); } } diff --git a/backend/tests/LearnStack.Tests.Integration/CompositionRootCatalogueTests.cs b/backend/tests/LearnStack.Tests.Integration/CompositionRootCatalogueTests.cs index 22dc27d8..e4a0a6a0 100644 --- a/backend/tests/LearnStack.Tests.Integration/CompositionRootCatalogueTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/CompositionRootCatalogueTests.cs @@ -1,5 +1,6 @@ using FluentAssertions; using LearnStack.Infrastructure.Audit; +using LearnStack.Modules.Customization.Application.IntegrationEvents; using LearnStack.SharedKernel.Audit; using LearnStack.Tools.Seeder; using Microsoft.AspNetCore.Mvc.Testing; @@ -25,8 +26,9 @@ namespace LearnStack.Tests.Integration; /// /// /// So each root's own is resolved from its own container and held -/// to two things: every request type with a handler is registered in it, and its entries are -/// exactly the entries the discovered sources declare — no source missing, none extra. +/// to two things: every request and concrete integration handler is registered in it, +/// including explicit Off coordinators, and its entries are exactly the entries the +/// discovered sources declare — no source missing, none extra. /// /// public sealed class CompositionRootCatalogueTests : IClassFixture @@ -52,18 +54,65 @@ public async Task The_seeder_root_builds_the_discovered_catalogue() AssertIsTheDiscoveredCatalogue(provider.GetRequiredService(), "the seeder"); } + /// + /// ADR-0057 + /// classifies each concrete handler even when Off contributes no operation entries. + /// + [Fact] + public void The_root_catalogue_sweep_rejects_one_missing_Off_handler_with_unchanged_operation_entries() + { + var built = _factory.Services.GetRequiredService(); + var filtered = new MissingHandlerCatalogue(built, typeof(InitializeContentTypeDefaultHandler)); + + filtered.All.Should().BeSameAs(built.All, + "removing one Off registration leaves the entire operation catalogue unchanged; Fix: check handler identities as well as operation entries"); + BackendDiscovery.ShippedAuditSubjects().Where(subject => !filtered.TryGet(subject, out _)) + .Should().Equal([typeof(InitializeContentTypeDefaultHandler)], + "the other consumer of TenantCreated remains registered; Fix: distinguish each concrete handler sharing an event"); + + Action check = () => AssertIsTheDiscoveredCatalogue(filtered, "the missing-Off control"); + check.Should().Throw() + .WithMessage("*InitializeContentTypeDefaultHandler*", + "the same assertion used for both actual roots must reject the missing Off handler; Fix: retain the handler registration census"); + } + private static void AssertIsTheDiscoveredCatalogue(IAuditCatalog built, string root) { var discovered = new AuditCatalog(BackendDiscovery.CatalogueSources()); - var shipped = BackendDiscovery.ShippedRequestTypes(); + var shipped = BackendDiscovery.ShippedAuditSubjects(); discovered.All.Should().NotBeEmpty("a comparison against an empty catalogue proves nothing"); - shipped.Where(request => !built.TryGet(request, out _)).Select(request => request.Name) + shipped.Should().Contain([typeof(InitializeContentTypeDefaultHandler), typeof(InitializeTaxonomyDefaultHandler)], + "both useful production coordinators must be in the population; Fix: discover concrete integration handlers across every production assembly"); + + shipped.Where(subject => !built.TryGet(subject, out _)).Select(subject => subject.Name) .Should().BeEmpty( - "{0} refuses every request its catalogue does not register, as unclassified", root); + "{0} refuses requests and integration handlers its catalogue does not register. Fix: register each concrete handler, including Off coordinators", root); + + foreach (var handler in new[] { typeof(InitializeContentTypeDefaultHandler), typeof(InitializeTaxonomyDefaultHandler) }) + { + built.TryGet(handler, out var registration).Should().BeTrue( + "{0} must classify {1} before construction; Fix: register the concrete coordinator in the actual root", root, handler.Name); + registration.WritesNoRow.Should().BeTrue( + "{0} classifies {1} as Off while its nested command supplies MUST intents; Fix: preserve the coordinator's explicit Off registration", root, handler.Name); + } built.All.Should().BeEquivalentTo(discovered.All, "{0} registers every audit source the backend ships, and nothing else", root); } + + private sealed class MissingHandlerCatalogue(IAuditCatalog inner, Type missingHandler) : IAuditCatalog + { + public IReadOnlyCollection All => inner.All; + + public bool TryGet(Type requestType, out AuditRegistration registration) + { + if (requestType != missingHandler) return inner.TryGet(requestType, out registration); + registration = null!; + return false; + } + + public bool TryGetOffPath(string operation, out AuditCatalogEntry entry) => inner.TryGetOffPath(operation, out entry); + } } diff --git a/backend/tests/LearnStack.Tests.Integration/ConstructionOnlyApiFactory.cs b/backend/tests/LearnStack.Tests.Integration/ConstructionOnlyApiFactory.cs index 39a8832b..6a2fa93d 100644 --- a/backend/tests/LearnStack.Tests.Integration/ConstructionOnlyApiFactory.cs +++ b/backend/tests/LearnStack.Tests.Integration/ConstructionOnlyApiFactory.cs @@ -35,7 +35,8 @@ internal static void RemoveDispatcher(IServiceCollection services) // test composition, never a production flag or a blanket hosted-service ban. foreach (var descriptor in services.Where(descriptor => descriptor.ServiceType == typeof(IHostedService) - && descriptor.ImplementationType == typeof(OutboxProcessor)).ToArray()) + && (descriptor.ImplementationType == typeof(OutboxProcessor) + || descriptor.ImplementationType == typeof(DeliveryProcessor))).ToArray()) { services.Remove(descriptor); } diff --git a/backend/tests/LearnStack.Tests.Integration/ConstructionOnlyApiFactoryTests.cs b/backend/tests/LearnStack.Tests.Integration/ConstructionOnlyApiFactoryTests.cs index 09e17e8a..5de85d97 100644 --- a/backend/tests/LearnStack.Tests.Integration/ConstructionOnlyApiFactoryTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/ConstructionOnlyApiFactoryTests.cs @@ -11,12 +11,13 @@ namespace LearnStack.Tests.Integration; public sealed class ConstructionOnlyApiFactoryTests { [Fact] - public void Request_only_test_composition_removes_exactly_the_dispatcher_and_preserves_other_workers() + public void Request_only_test_composition_removes_exactly_both_dispatchers_and_preserves_other_workers() { var services = new ServiceCollection(); services.AddHostedService(); + services.AddHostedService(); services.AddHostedService(); - services.Count(descriptor => descriptor.ServiceType == typeof(IHostedService)).Should().Be(2); + services.Count(descriptor => descriptor.ServiceType == typeof(IHostedService)).Should().Be(3); ConstructionOnlyApiFactory.RemoveDispatcher(services); @@ -30,7 +31,7 @@ public async Task Construction_only_host_leaves_the_dispatcher_source_unresolved await using var host = new ConstructionOnlyApiFactory(); using var client = host.CreateClient(); - host.Services.GetServices().Should().NotContain(service => service is OutboxProcessor); + host.Services.GetServices().Should().NotContain(service => service is OutboxProcessor || service is DeliveryProcessor); host.Services.GetRequiredKeyedService>(OutboxDispatcherDataSource.Key) .IsValueCreated.Should().BeFalse(); (await client.GetAsync("/healthz")).IsSuccessStatusCode.Should().BeTrue(); diff --git a/backend/tests/LearnStack.Tests.Integration/Database/AuditPipelineTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/AuditPipelineTests.cs index aed92947..7fd7fb16 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/AuditPipelineTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/AuditPipelineTests.cs @@ -88,10 +88,8 @@ public async Task MustClass_Audit_Writes_Share_The_Business_Transaction() "tenancy.organization.create", "tenancy.organization.create", "tenancy.hostmapping.write", - "customization.content_type.register", - "customization.content_type.publish", - "customization.level_taxonomy.register", - "customization.level_taxonomy.publish"); + "customization.content_type.initialize", + "customization.level_taxonomy.initialize"); rows.Count(row => row.Operation == "tenancy.organization.create").Should().Be(2, "the seed creates a second organization with CreateOrganizationCommand, and " @@ -354,7 +352,7 @@ public async Task Audit_Survives_Transaction_Rollback() // The repeated commands REFUSE, and every refusal is recorded. That is the point rather // than an inconvenience: a repeated provisioning attempt is exactly the shape a // probe takes, and Audit Coverage justifies the whole `denied` class with it. - rows.Count(row => row.Outcome == "success").Should().Be(8, + rows.Count(row => row.Outcome == "success").Should().Be(6, "the first run's rows are untouched"); rows.Where(row => row.Outcome != "success").Should().NotBeEmpty( @@ -485,10 +483,8 @@ private async Task CountAsync(string sql, object key) "tenancy.tenant.create", "tenancy.organization.create", "tenancy.hostmapping.write", - "customization.content_type.register", - "customization.content_type.publish", - "customization.level_taxonomy.register", - "customization.level_taxonomy.publish", + "customization.content_type.initialize", + "customization.level_taxonomy.initialize", ]; /// diff --git a/backend/tests/LearnStack.Tests.Integration/Database/AuditStoreTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/AuditStoreTests.cs index a474336a..0aad064f 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/AuditStoreTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/AuditStoreTests.cs @@ -2,6 +2,7 @@ using LearnStack.Infrastructure.Audit; using LearnStack.Infrastructure.Persistence; using LearnStack.SharedKernel.Audit; +using LearnStack.SharedKernel.Tenancy; using LearnStack.SharedKernel.Identifiers; using LearnStack.SharedKernel.Persistence; using Microsoft.Extensions.DependencyInjection; @@ -408,7 +409,7 @@ public async Task A_best_effort_failure_is_logged_and_dropped() logger, MeterFactory, health, - new AuditCatalog([])); + new AuditCatalog([]), StaticTenantContextAccessor.Unresolved); // TenantB is a real tenant, but the draft's own announcement makes the row legal — // so to force a failure the row has to break something the announcement cannot @@ -758,7 +759,7 @@ public async Task A_best_effort_write_swallows_a_data_source_that_cannot_be_buil NullLogger.Instance, MeterFactory, new AuditHealth(), - new AuditCatalog([])); + new AuditCatalog([]), StaticTenantContextAccessor.Unresolved); var bestEffort = async () => await store.WriteBestEffortAsync(Draft(organizationId: null)); @@ -943,7 +944,7 @@ public async Task A_standalone_failure_is_Critical_counted_and_leaves_the_check_ logger, MeterFactory, health, - new AuditCatalog([])); + new AuditCatalog([]), StaticTenantContextAccessor.Unresolved); var counted = 0L; using var listener = new MeterListener(); @@ -999,7 +1000,7 @@ public async Task A_failure_the_store_cannot_translate_is_still_Critical_counted logger, MeterFactory, health, - new AuditCatalog([])); + new AuditCatalog([]), StaticTenantContextAccessor.Unresolved); var counted = 0L; using var listener = new MeterListener(); @@ -1116,7 +1117,7 @@ private static PostgresAuditStore Store(AuditStateCapture capture, NpgsqlDataSou private static PostgresAuditStore Store( AuditStateCapture capture, NpgsqlDataSource dataSource, IAuditHealth health) => new(capture, new Lazy(() => dataSource), - NullLogger.Instance, MeterFactory, health, new AuditCatalog([])); + NullLogger.Instance, MeterFactory, health, new AuditCatalog([]), StaticTenantContextAccessor.Unresolved); /// A real meter factory, so the counter the store increments is a real one. private static readonly IMeterFactory MeterFactory = diff --git a/backend/tests/LearnStack.Tests.Integration/Database/AuditWorkflowTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/AuditWorkflowTests.cs index 5e59c8fc..817e5dfc 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/AuditWorkflowTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/AuditWorkflowTests.cs @@ -2,6 +2,7 @@ using FluentAssertions; using LearnStack.Modules.Customization.Application.Abstractions; using LearnStack.Modules.Customization.Application.Contracts.Customization; +using LearnStack.Modules.Customization.Application.Contracts.Seeding; using LearnStack.Modules.Customization.Domain; using LearnStack.Modules.Customization.Infrastructure.Persistence; using LearnStack.Modules.Tenancy.Application.Contracts.Tenant; @@ -72,7 +73,7 @@ public async Task Replacing_a_live_content_type_is_one_row_about_the_successor() await using var dataSource = NpgsqlDataSource.Create(_database.AppConnectionString); await SeedAsync(dataSource); - var incumbent = Tenant.BuiltInContentTypeId; + var incumbent = await BuiltInIdAsync(dataSource, taxonomy: false); var successor = Guid.CreateVersion7(); await using (var provider = Compose(dataSource)) @@ -91,9 +92,8 @@ public async Task Replacing_a_live_content_type_is_one_row_about_the_successor() var rows = (await RowsAsync()).Where(row => row.Operation == "customization.content_type.publish").ToList(); - // The FIRST publication — the seed's — and the replacement: one row each. - rows.Select(row => row.EntityId).Should().BeEquivalentTo( - [incumbent.ToString(), successor.ToString()]); + // Initialization creates the incumbent Active; only the successor is published. + rows.Should().ContainSingle().Which.EntityId.Should().Be(successor.ToString()); var replacement = rows.Single(row => row.EntityId == successor.ToString()); replacement.Outcome.Should().Be("success"); @@ -116,7 +116,7 @@ public async Task Replacing_a_live_taxonomy_is_one_row_about_the_successor() await using var dataSource = NpgsqlDataSource.Create(_database.AppConnectionString); await SeedAsync(dataSource); - var incumbent = Tenant.BuiltInTaxonomyId; + var incumbent = await BuiltInIdAsync(dataSource, taxonomy: true); var successor = Guid.CreateVersion7(); await using (var provider = Compose(dataSource)) @@ -135,8 +135,7 @@ [.. BuiltInCustomizations.Plain.Bands.Select(band => var rows = (await RowsAsync()).Where(row => row.Operation == "customization.level_taxonomy.publish").ToList(); - rows.Select(row => row.EntityId).Should().BeEquivalentTo( - [incumbent.ToString(), successor.ToString()]); + rows.Should().ContainSingle().Which.EntityId.Should().Be(successor.ToString()); var replacement = rows.Single(row => row.EntityId == successor.ToString()); replacement.Outcome.Should().Be("success"); @@ -192,7 +191,7 @@ public async Task The_bands_a_tenant_authors_are_in_the_row_that_records_the_tax // And only this aggregate: the seed's own taxonomy — a different instance of the // same type, registered in another request — is not in this row. row.AfterState.Should().NotContain(BuiltInCustomizations.Plain.Bands[0].DisplayName["en"]); - row.Changes.Should().NotContain(Tenant.BuiltInTaxonomyId.ToString()); + row.Changes.Should().NotContain((await BuiltInIdAsync(dataSource, taxonomy: true)).ToString()); } [Fact] @@ -281,6 +280,7 @@ public async Task A_partially_loaded_taxonomy_is_never_recorded_as_owning_only_w var intentId = AuditEntryId.From(Guid.CreateVersion7()); var firstBand = BuiltInCustomizations.Plain.Bands[0].Key; + var builtInId = TenantLevelTaxonomyId.From(await BuiltInIdAsync(dataSource, taxonomy: true)); await using var provider = Compose(dataSource); await using (var scope = provider.CreateAsyncScope()) @@ -310,7 +310,7 @@ public async Task A_partially_loaded_taxonomy_is_never_recorded_as_owning_only_w var db = services.GetRequiredService(); var taxonomy = await db.TenantLevelTaxonomies .Include(candidate => candidate.Items.Where(band => band.Key == firstBand)) - .SingleAsync(candidate => candidate.Id == TenantLevelTaxonomyId.From(Tenant.BuiltInTaxonomyId)); + .SingleAsync(candidate => candidate.Id == builtInId); taxonomy.Items.Should().ContainSingle("the premise: the load brought one band"); db.Entry(taxonomy).Collection(nameof(TenantLevelTaxonomy.Items)).IsLoaded.Should().BeTrue( @@ -453,10 +453,11 @@ public async Task A_refused_publication_is_recorded_against_its_subject_with_act // the reconcile's standalone row names the instance it refused. await using var dataSource = NpgsqlDataSource.Create(_database.AppConnectionString); await SeedAsync(dataSource); + var builtInId = await BuiltInIdAsync(dataSource, taxonomy: false); await using (var provider = Compose(dataSource)) { - var refused = await SendAsync(provider, new PublishTenantContentTypeCommand(Tenant.BuiltInContentTypeId)); + var refused = await SendAsync(provider, new PublishTenantContentTypeCommand(builtInId)); refused.IsFailure.Should().BeTrue("the precondition: that revision is already live"); } @@ -467,7 +468,7 @@ public async Task A_refused_publication_is_recorded_against_its_subject_with_act // field detail of that error and not a column. row.Outcome.Should().Be("failed"); row.ErrorKey.Should().Be("lockey_business_rule_violation"); - row.EntityId.Should().Be(Tenant.BuiltInContentTypeId.ToString()); + row.EntityId.Should().Be(builtInId.ToString()); row.ActorUserId.Should().Be(Actor); row.CorrelationId.Should().Be(Correlation); } @@ -536,6 +537,19 @@ private static async Task SeedAsync(NpgsqlDataSource dataSource) (await runner.RunAsync(CancellationToken.None, [Tenant])).Should().Be(0); } + private static async Task BuiltInIdAsync(NpgsqlDataSource source, bool taxonomy) + { + await using var provider = Compose(source); + await using var scope = provider.CreateAsyncScope(); + var sender = scope.ServiceProvider.GetRequiredService(); + var result = taxonomy + ? await sender.Send(new GetActiveTaxonomySeedRevisionQuery(BuiltInCustomizations.Plain.Key)) + : await sender.Send(new GetActiveContentTypeSeedRevisionQuery(BuiltInCustomizations.Card.Key)); + result.IsSuccess.Should().BeTrue(); + result.Value!.State.Should().NotBeNull(); + return result.Value!.State!.Id; + } + /// The seeder's graph, acting for the tenant as an authenticated principal would. private static ServiceProvider Compose(NpgsqlDataSource dataSource) => SeedComposition.Build( diff --git a/backend/tests/LearnStack.Tests.Integration/Database/CustomizationDefaultDeliveryTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/CustomizationDefaultDeliveryTests.cs new file mode 100644 index 00000000..5c97f69a --- /dev/null +++ b/backend/tests/LearnStack.Tests.Integration/Database/CustomizationDefaultDeliveryTests.cs @@ -0,0 +1,426 @@ +using FluentAssertions; +using LearnStack.Api.Messaging; +using LearnStack.Infrastructure.Messaging; +using LearnStack.Modules.Tenancy.Application.Contracts.IntegrationEvents; +using LearnStack.Modules.Tenancy.Application.Contracts.Tenant; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Messaging; +using System.Text.Json; +using LearnStack.Tools.Seeder; +using MediatR; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.Mvc.Testing; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using Xunit; + +namespace LearnStack.Tests.Integration.Database; + +/// Production TenantCreated delivery and Seeder converge on the same tenant defaults. +[Trait(RequiresDocker.Key, RequiresDocker.Value)] +[Collection(SharedSchema.Name)] +public sealed class CustomizationDefaultDeliveryTests(SchemaFixture schema) +{ + private const string ContentSubscription = "customization.initialize-content-type-default.v1"; + private const string TaxonomySubscription = "customization.initialize-taxonomy-default.v1"; + private static readonly TimeSpan Deadline = TimeSpan.FromSeconds(45); + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Stored_backlog_and_fresh_provisioning_reach_both_real_subscribers(bool fresh) + { + await using var fixture = await Fixture.CreateAsync(schema); + if (!fresh) await fixture.ProvisionAsync(); + fixture.StartApi(); + fixture.AssertProductionSubscriptions(); + if (fresh) await fixture.ProvisionAsync(fixture.ApiServices); + + await fixture.WaitForBothDeliveriesAsync(); + await fixture.AssertCreatedDefaultsAsync(); + (await fixture.CountAsync("outbox_messages", "processed_at IS NOT NULL")).Should().Be(1); + (await fixture.CountAsync("event_deliveries", "processed_at IS NOT NULL")).Should().Be(2); + (await fixture.CountAsync("inbox_messages", "recorded_at IS NOT NULL")).Should().Be(2); + } + + [Fact] + public async Task A_failed_MUST_audit_rolls_back_both_business_effects_and_inbox_completion() + { + await using var fixture = await Fixture.CreateAsync(schema); + await fixture.ProvisionAsync(); + await fixture.AlterSchemaAsync(""" + ALTER TABLE audit_log ADD CONSTRAINT refuse_default_delivery_audit + CHECK (operation NOT IN ('customization.content_type.initialize', + 'customization.level_taxonomy.initialize')) + """); + fixture.StartApi(); + await Fixture.WaitUntilAsync(async () => + await fixture.CountAsync("event_deliveries", "last_error IS NOT NULL") == 2); + + (await fixture.CountAsync("tenant_content_types", "key = 'card'")).Should().Be(0); + (await fixture.CountAsync("tenant_level_taxonomies", "key = 'plain'")).Should().Be(0); + (await fixture.CountAsync("inbox_messages", "recorded_at IS NOT NULL")).Should().Be(0); + (await fixture.CountAsync("event_deliveries", "processed_at IS NOT NULL")).Should().Be(0); + (await fixture.GenerationAsync()).Should().Be(0); + + await fixture.AlterSchemaAsync("ALTER TABLE audit_log DROP CONSTRAINT refuse_default_delivery_audit"); + await fixture.MakeFailedDeliveriesDueAsync(); + await fixture.WaitForBothDeliveriesAsync(); + await fixture.AssertCreatedDefaultsAsync(); + } + + [Fact] + public async Task Reaccepting_the_exact_provisioning_fact_does_not_repeat_inbox_effect_or_audit() + { + await using var fixture = await Fixture.CreateAsync(schema); + await fixture.ProvisionAsync(); + fixture.StartApi(); + await fixture.WaitForBothDeliveriesAsync(); + var before = await fixture.DefaultStateAsync(); + + await fixture.RepublishOriginalAsync(); + + (await fixture.DefaultStateAsync()).Should().Be(before); + (await fixture.CountAsync("event_deliveries", "processed_at IS NOT NULL")).Should().Be(2); + (await fixture.CountAsync("inbox_messages", "recorded_at IS NOT NULL")).Should().Be(2); + } + + [Theory] + [InlineData(false, false)] + [InlineData(false, true)] + [InlineData(true, false)] + [InlineData(true, true)] + public async Task Real_Seeder_and_real_subscriber_serialize_on_each_builtin_concept(bool taxonomy, bool consumerFirst) + { + await using var fixture = await Fixture.CreateAsync(schema); + await fixture.ProvisionAsync(); + await fixture.InstallDefinitionInsertGateAsync(taxonomy); + Task? seed = null; + try + { + if (consumerFirst) + { + fixture.StartApi(); + await fixture.WaitForGateWinnerAsync(); + seed = fixture.RunSeederAsync(); + } + else + { + seed = fixture.RunSeederAsync(); + await fixture.WaitForGateWinnerAsync(); + fixture.StartApi(); + } + + await fixture.WaitForConceptLoserAsync(); + (await fixture.CountAsync(taxonomy ? "tenant_level_taxonomies" : "tenant_content_types", + taxonomy ? "key = 'plain'" : "key = 'card'")).Should().Be(0, + "the winning insert is still inside its uncommitted transaction"); + } + finally + { + await fixture.ReleaseDefinitionInsertGateAsync(); + if (seed is not null) await seed.WaitAsync(Deadline); + } + + seed.Should().NotBeNull(); + (await seed!).Should().Be(0); + await fixture.WaitForBothDeliveriesAsync(); + await fixture.AssertCreatedDefaultsAsync(); + (await fixture.CountAsync("tenant_content_types", "key = 'card'")).Should().Be(1); + (await fixture.CountAsync("audit_log", "operation = 'customization.content_type.initialize' AND outcome = 'success' AND operation_type = 'Create' AND after_state IS NOT NULL")).Should().Be(1, + "the first writer creates exactly one root; a preserved attempt may have a separate classified audit"); + } + + private sealed class Fixture : IAsyncDisposable + { + private readonly DisposableSchemaDatabase _database; + private readonly NpgsqlDataSource _seedSource; + private readonly SeedTenant _tenant; + private ApiFactory? _api; + private NpgsqlConnection? _gateOwner; + private long _gateKey; + private int _gateOwnerPid; + private string _gateTable = ""; + private CancellationTokenSource? _seedCancellation; + private Task? _seedTask; + + private Fixture(DisposableSchemaDatabase database) + { + _database = database; + _seedSource = NpgsqlDataSource.Create(database.AppConnectionString); + _tenant = SeedData.English with { Curriculum = null }; + } + + public IServiceProvider ApiServices => _api?.Services ?? throw new InvalidOperationException("API has not started."); + + public static async Task CreateAsync(SchemaFixture schema) => + new(await DisposableSchemaDatabase.CreateAsync(schema.Postgres)); + + public void StartApi() + { + _api = new ApiFactory(_database); + _ = _api.Services; + } + + public void AssertProductionSubscriptions() + { + var registry = ApiServices.GetRequiredService(); + var resolved = registry.Resolve(typeof(LearnStack.Modules.Tenancy.Application.Contracts.IntegrationEvents.TenantCreatedIntegrationEventV1).AssemblyQualifiedName); + resolved.Subscriptions.Select(subscription => (subscription.Id, subscription.Readiness, subscription.HandlerType)) + .Should().BeEquivalentTo(new (string, OutboxSubscriptionReadiness, Type)[] { + (ContentSubscription, OutboxSubscriptionReadiness.Active, + typeof(LearnStack.Modules.Customization.Application.IntegrationEvents.InitializeContentTypeDefaultHandler)), + (TaxonomySubscription, OutboxSubscriptionReadiness.Active, + typeof(LearnStack.Modules.Customization.Application.IntegrationEvents.InitializeTaxonomyDefaultHandler)), + }); + ApiServices.GetServices().OfType().Should().ContainSingle(); + ApiServices.GetServices().OfType().Should().ContainSingle(); + } + + public async Task ProvisionAsync(IServiceProvider? provider = null) + { + if (provider is null) + { + await using var seed = SeedComposition.Build(_seedSource, null, NullLoggerFactory.Instance); + await ProvisionInAsync(seed); + } + else await ProvisionInAsync(provider); + } + + private async Task ProvisionInAsync(IServiceProvider provider) + { + await using var scope = provider.CreateAsyncScope(); + var result = await scope.ServiceProvider.GetRequiredService().Send(new ProvisionTenantCommand( + _tenant.TenantId, _tenant.Slug, _tenant.DisplayName, + _tenant.DefaultOrganization.OrganizationId, _tenant.DefaultOrganization.Slug, + _tenant.DefaultOrganization.DisplayName)); + result.IsSuccess.Should().BeTrue(); + } + + public Task RunSeederAsync() + { + _seedCancellation = new CancellationTokenSource(Deadline); + _seedTask = new SeedRunner( + context => SeedComposition.Build(_seedSource, context, NullLoggerFactory.Instance), + NullLogger.Instance).RunAsync(_seedCancellation.Token, [_tenant]); + return _seedTask; + } + + public async Task RepublishOriginalAsync() + { + IntegrationEventEnvelope envelope; + await using (var connection = (NpgsqlConnection)await PostgresFixture.OpenAsync(_database.AppConnectionString)) + { + await SchemaQueries.AssertApplicationRoleAsync(connection); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.SetTenantAsync(connection, transaction, _tenant.TenantId.Value); + await using var query = new NpgsqlCommand(""" + SELECT payload::text, correlation_id, organization_id, causation_id, actor_user_id + FROM outbox_messages WHERE tenant_id = @tenant + """, connection, transaction); + query.Parameters.AddWithValue("tenant", _tenant.TenantId.Value); + await using var row = await query.ExecuteReaderAsync(); + (await row.ReadAsync()).Should().BeTrue(); + var fact = JsonSerializer.Deserialize( + row.GetString(0), IntegrationEventBase.PayloadJsonOptions)!; + envelope = new IntegrationEventEnvelope(fact, row.GetString(1), + row.IsDBNull(2) ? null : row.GetGuid(2), + row.IsDBNull(3) ? null : row.GetGuid(3), + row.IsDBNull(4) ? null : UserId.From(row.GetGuid(4))); + (await row.ReadAsync()).Should().BeFalse(); + } + await using var scope = ApiServices.CreateAsyncScope(); + await scope.ServiceProvider.GetRequiredService().PublishAsync(envelope); + } + + public async Task<(Guid Content, Guid Taxonomy, long Generation, long Audits, long Inbox)> DefaultStateAsync() => + (await DefinitionIdAsync("tenant_content_types", "card"), + await DefinitionIdAsync("tenant_level_taxonomies", "plain"), + await GenerationAsync(), + await CountAsync("audit_log", "operation IN ('customization.content_type.initialize', 'customization.level_taxonomy.initialize')"), + await CountAsync("inbox_messages", $"subscription_id IN ('{ContentSubscription}', '{TaxonomySubscription}')")); + + private async Task DefinitionIdAsync(string table, string key) + { + await using var connection = (NpgsqlConnection)await PostgresFixture.OpenAsync(_database.AppConnectionString); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.SetTenantAsync(connection, transaction, _tenant.TenantId.Value); + await using var query = new NpgsqlCommand($"SELECT id FROM {table} WHERE key = @key", connection, transaction); + query.Parameters.AddWithValue("key", key); + return (Guid)(await query.ExecuteScalarAsync())!; + } + + public async Task WaitForBothDeliveriesAsync() => + await WaitUntilAsync(async () => await CountAsync("event_deliveries", "processed_at IS NOT NULL") == 2 + && await CountAsync("inbox_messages", "recorded_at IS NOT NULL") == 2); + + public async Task AssertCreatedDefaultsAsync() + { + (await CountAsync("tenant_content_types", "key = 'card' AND status = 'Active' AND deleted_at IS NULL")) + .Should().Be(1); + (await CountAsync("tenant_level_taxonomies", "key = 'plain' AND status = 'Active' AND deleted_at IS NULL")) + .Should().Be(1); + (await CountAsync("tenant_level_taxonomy_items", "taxonomy_key = 'plain' AND schema_version = 1")).Should().Be(3); + (await GenerationAsync()).Should().Be(2, "each create-only initializer advances its own definition generation once"); + (await CountAsync("audit_log", "operation = 'customization.content_type.initialize' AND operation_class = 'Must' AND outcome = 'success' AND operation_type = 'Create' AND after_state IS NOT NULL")) + .Should().Be(1); + (await CountAsync("audit_log", "operation = 'customization.level_taxonomy.initialize' AND operation_class = 'Must' AND outcome = 'success' AND operation_type = 'Create' AND after_state IS NOT NULL")) + .Should().Be(1); + (await CountAsync("inbox_messages inbox JOIN outbox_messages source ON source.id = inbox.event_id AND source.tenant_id = inbox.tenant_id", + $"inbox.subscription_id IN ('{ContentSubscription}', '{TaxonomySubscription}') AND inbox.recorded_at IS NOT NULL")) + .Should().Be(2); + } + + public async Task CountAsync(string table, string predicate) + { + // Closed, test-owned identifiers and predicates only; the first transaction + // statement announces the tenant and PostgreSQL RLS scopes this read. + await using var connection = (NpgsqlConnection)await PostgresFixture.OpenAsync(_database.AppConnectionString); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.SetTenantAsync(connection, transaction, _tenant.TenantId.Value); + await using var query = new NpgsqlCommand($"SELECT count(*) FROM {table} WHERE {predicate}", connection, transaction); + return (long)(await query.ExecuteScalarAsync())!; + } + + public async Task GenerationAsync() + { + await using var connection = (NpgsqlConnection)await PostgresFixture.OpenAsync(_database.AppConnectionString); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.SetTenantAsync(connection, transaction, _tenant.TenantId.Value); + await using var query = new NpgsqlCommand("SELECT COALESCE((SELECT generation FROM customization_generations), 0)", connection, transaction); + return (long)(await query.ExecuteScalarAsync())!; + } + + public async Task AlterSchemaAsync(string sql) + { + await using var owner = await PostgresFixture.OpenAsync(_database.MigrationConnectionString); + await using var command = new NpgsqlCommand(sql, (NpgsqlConnection)owner); + await command.ExecuteNonQueryAsync(); + } + + public async Task MakeFailedDeliveriesDueAsync() + { + await using var dispatcher = (NpgsqlConnection)await PostgresFixture.OpenAsync(_database.OutboxConnectionString); + await using var command = new NpgsqlCommand(""" + UPDATE event_deliveries SET available_after = clock_timestamp() + WHERE processed_at IS NULL AND last_error IS NOT NULL + """, dispatcher); + await command.ExecuteNonQueryAsync(); + } + + public async Task InstallDefinitionInsertGateAsync(bool taxonomy) + { + _gateKey = Random.Shared.NextInt64(1, long.MaxValue); + _gateTable = taxonomy ? "tenant_level_taxonomies" : "tenant_content_types"; + var conceptKey = taxonomy ? "plain" : "card"; + _gateOwner = (NpgsqlConnection)await PostgresFixture.OpenAsync(_database.MigrationConnectionString); + await using (var setup = new NpgsqlCommand($""" + CREATE FUNCTION default_definition_insert_gate() RETURNS trigger LANGUAGE plpgsql AS $$ + BEGIN + IF NEW.tenant_id = '{_tenant.TenantId.Value:D}'::uuid AND NEW.key = '{conceptKey}' THEN + PERFORM pg_advisory_xact_lock({_gateKey}); + END IF; + RETURN NEW; + END $$; + CREATE TRIGGER default_definition_insert_gate BEFORE INSERT ON {_gateTable} + FOR EACH ROW EXECUTE FUNCTION default_definition_insert_gate(); + SELECT pg_advisory_lock({_gateKey}); + """, _gateOwner)) + await setup.ExecuteNonQueryAsync(); + await using var pid = new NpgsqlCommand("SELECT pg_backend_pid()", _gateOwner); + _gateOwnerPid = (int)(await pid.ExecuteScalarAsync())!; + } + + public async Task WaitForGateWinnerAsync() => await WaitUntilAsync(async () => + await BlockedByAsync(_gateOwnerPid) == 1); + + public async Task WaitForConceptLoserAsync() => await WaitUntilAsync(async () => + { + await using var observer = (NpgsqlConnection)await PostgresFixture.OpenAsync(_database.AppConnectionString); + await SchemaQueries.AssertApplicationRoleAsync(observer); + await using var query = new NpgsqlCommand(""" + SELECT count(*) FROM pg_locks waiting + JOIN pg_locks held ON held.locktype = 'advisory' AND held.granted + AND held.database = waiting.database + AND held.classid = waiting.classid AND held.objid = waiting.objid + AND held.objsubid = waiting.objsubid AND held.pid <> waiting.pid + JOIN pg_stat_activity loser ON loser.pid = waiting.pid + JOIN pg_stat_activity winner ON winner.pid = held.pid + WHERE waiting.locktype = 'advisory' AND NOT waiting.granted + AND loser.datname = current_database() AND winner.datname = current_database() + AND loser.usename = 'learnstack_app' AND winner.usename = 'learnstack_app' + AND loser.wait_event_type = 'Lock' AND winner.wait_event_type = 'Lock' + AND @gate = ANY(pg_blocking_pids(winner.pid)) + """, observer); + query.Parameters.AddWithValue("gate", _gateOwnerPid); + return (long)(await query.ExecuteScalarAsync())! == 1; + }); + + private async Task BlockedByAsync(int blockerPid) + { + await using var observer = (NpgsqlConnection)await PostgresFixture.OpenAsync(_database.AppConnectionString); + await SchemaQueries.AssertApplicationRoleAsync(observer); + await using var query = new NpgsqlCommand(""" + SELECT count(*) FROM pg_stat_activity + WHERE datname = current_database() AND usename = 'learnstack_app' + AND wait_event_type = 'Lock' AND @blocker = ANY(pg_blocking_pids(pid)) + AND query LIKE @table + """, observer); + query.Parameters.AddWithValue("blocker", blockerPid); + query.Parameters.AddWithValue("table", $"%{_gateTable}%"); + return (long)(await query.ExecuteScalarAsync())!; + } + + public async Task ReleaseDefinitionInsertGateAsync() + { + if (_gateOwner is null) return; + await using var release = new NpgsqlCommand($"SELECT pg_advisory_unlock({_gateKey})", _gateOwner); + ((bool)(await release.ExecuteScalarAsync())!).Should().BeTrue(); + await _gateOwner.DisposeAsync(); + _gateOwner = null; + } + + public static async Task WaitUntilAsync(Func> condition) + { + using var deadline = new CancellationTokenSource(Deadline); + while (!await condition()) await Task.Delay(20, deadline.Token); + } + + public async ValueTask DisposeAsync() + { + await ReleaseDefinitionInsertGateAsync(); + _seedCancellation?.Cancel(); + if (_seedTask is { IsCompleted: false }) + { + try { await _seedTask.WaitAsync(Deadline); } + catch (OperationCanceledException) { } + } + if (_api is not null) + { + foreach (var worker in _api.Services.GetServices() + .Where(service => service is OutboxProcessor or DeliveryProcessor)) + await worker.StopAsync(CancellationToken.None).WaitAsync(Deadline); + await _api.DisposeAsync(); + } + _seedCancellation?.Dispose(); + await _seedSource.DisposeAsync(); + await _database.DisposeAsync(); + } + + private sealed class ApiFactory(DisposableSchemaDatabase database) : WebApplicationFactory + { + protected override void ConfigureWebHost(IWebHostBuilder builder) + { + builder.UseEnvironment(Environments.Development); + builder.UseSetting("ConnectionStrings:Default", database.AppConnectionString); + builder.UseSetting("ConnectionStrings:PlatformAdmin", database.PlatformConnectionString); + builder.UseSetting("ConnectionStrings:OutboxDispatcher", database.OutboxConnectionString); + } + } + } +} diff --git a/backend/tests/LearnStack.Tests.Integration/Database/CustomizationDefaultInitializationTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/CustomizationDefaultInitializationTests.cs new file mode 100644 index 00000000..f26f72b8 --- /dev/null +++ b/backend/tests/LearnStack.Tests.Integration/Database/CustomizationDefaultInitializationTests.cs @@ -0,0 +1,444 @@ +using System.Text.Json.Nodes; +using FluentAssertions; +using LearnStack.Modules.Customization.Application.Abstractions; +using LearnStack.Modules.Customization.Application.Contracts.Customization; +using LearnStack.Modules.Customization.Domain; +using LearnStack.Modules.Customization.Infrastructure.Persistence; +using LearnStack.Modules.Customization.Infrastructure.Projections; +using LearnStack.Modules.Tenancy.Application.Contracts.Tenant; +using LearnStack.SharedKernel.Audit; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Persistence; +using LearnStack.SharedKernel.Results; +using LearnStack.SharedKernel.Tenancy; +using LearnStack.SharedKernel.Time; +using LearnStack.Tools.Seeder; +using MediatR; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.Mvc.Testing; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Diagnostics; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using Xunit; + +namespace LearnStack.Tests.Integration.Database; + +[Trait(RequiresDocker.Key, RequiresDocker.Value)] +[Collection(SharedSchema.Name)] +public sealed class CustomizationDefaultInitializationTests(SchemaFixture schema, ConstructionOnlyApiFactory factory) + : IClassFixture +{ + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task An_untouched_concept_is_created_active_with_one_save_generation_and_must_audit(bool taxonomy) + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = NpgsqlDataSource.Create(database.AppConnectionString); + var tenant = await ProvisionAsync(source); + var saves = new SaveProbe(); + await using var host = Host(database, saves); + + (await InitializeAsync(host.Services, tenant, taxonomy)).IsSuccess.Should().BeTrue(); + + saves.Statuses.Should().Equal("Active"); + var state = await StateAsync(source, tenant, taxonomy); + state.Generation.Should().Be(1); + var root = JsonNode.Parse(state.Roots.Should().ContainSingle().Which)!; + var id = Guid.Parse(root["id"]!.GetValue()); + id.Version.Should().Be(7); + root["key"]!.GetValue().Should().Be(Key(taxonomy)); + root["tenant_id"]!.GetValue().Should().Be(tenant.TenantId.Value.ToString()); + root["status"]!.GetValue().Should().Be("Active"); + root["schema_version"]!.GetValue().Should().Be(1); + root["created_by"]!.GetValue().Should().Be(UserId.SystemActor.Value.ToString()); + root["updated_by"]!.GetValue().Should().Be(UserId.SystemActor.Value.ToString()); + root["deleted_at"].Should().BeNull(); + root["deleted_by"].Should().BeNull(); + AssertNames(root["display_name"]!, taxonomy ? BuiltInCustomizations.Plain.DisplayName : BuiltInCustomizations.Card.DisplayName); + if (taxonomy) + { + root["schema_revision"]!.GetValue().Should().Be(3); + root["row_version"]!.GetValue().Should().Be(4); + state.Items.Should().HaveCount(3); + foreach (var (itemJson, band) in state.Items.Zip(BuiltInCustomizations.Plain.Bands)) + { + var item = JsonNode.Parse(itemJson)!; + item["key"]!.GetValue().Should().Be(band.Key); + item["taxonomy_key"]!.GetValue().Should().Be("plain"); + item["schema_version"]!.GetValue().Should().Be(1); + item["sort"]!.GetValue().Should().Be(band.Sort); + item["metadata"].Should().BeNull(); + AssertNames(item["display_name"]!, band.DisplayName); + } + } + else + { + root["schema_revision"]!.GetValue().Should().Be(0); + root["row_version"]!.GetValue().Should().Be(1); + root["renderer_key"]!.GetValue().Should().Be(BuiltInCustomizations.Card.RendererKey); + JsonNode.DeepEquals(root["json_schema"], JsonNode.Parse(BuiltInCustomizations.Card.JsonSchema)).Should().BeTrue(); + state.Items.Should().BeEmpty(); + } + + var audit = (await AuditsAsync(source, tenant, taxonomy)).Should().ContainSingle().Which; + audit.OperationType.Should().Be("Create"); + audit.Class.Should().Be("Must"); + audit.Outcome.Should().Be("success"); + audit.EntityType.Should().Be(taxonomy ? nameof(TenantLevelTaxonomy) : nameof(TenantContentType)); + audit.EntityId.Should().Be(id.ToString()); + audit.Actor.Should().Be(UserId.SystemActor.Value); + audit.Before.Should().BeNull(); + JsonNode.Parse(audit.After!)!["Status"]!.GetValue().Should().Be("Active"); + JsonNode.Parse(audit.Changes!)!.AsArray().Should().NotBeEmpty(); + } + + [Theory] + [InlineData(false, "Draft")] + [InlineData(false, "Active")] + [InlineData(false, "Deprecated")] + [InlineData(false, "Deleted")] + [InlineData(false, "Successor")] + [InlineData(true, "Draft")] + [InlineData(true, "Active")] + [InlineData(true, "Deprecated")] + [InlineData(true, "Deleted")] + [InlineData(true, "Successor")] + public async Task Any_concept_history_preserves_every_field_and_generation_and_classifies_a_no_change_attempt(bool taxonomy, string history) + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = NpgsqlDataSource.Create(database.AppConnectionString); + var tenant = await ProvisionAsync(source); + await ArrangeHistoryAsync(source, tenant, taxonomy, history); + var before = await StateAsync(source, tenant, taxonomy); + var saves = new SaveProbe(); + await using var host = Host(database, saves); + + (await InitializeAsync(host.Services, tenant, taxonomy)).IsSuccess.Should().BeTrue(); + + var after = await StateAsync(source, tenant, taxonomy); + after.Roots.Should().Equal(before.Roots); + after.Items.Should().Equal(before.Items); + after.Generation.Should().Be(before.Generation); + saves.Statuses.Should().BeEmpty(); + var audit = (await AuditsAsync(source, tenant, taxonomy)).Should().ContainSingle().Which; + audit.Class.Should().Be("Must"); + audit.Outcome.Should().Be("success"); + audit.EntityId.Should().BeNull(); + audit.Before.Should().BeNull(); + audit.After.Should().BeNull(); + audit.Changes.Should().BeNull(); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Must_audit_refusal_rolls_back_the_active_root_and_generation(bool taxonomy) + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = NpgsqlDataSource.Create(database.AppConnectionString); + var tenant = await ProvisionAsync(source); + await using (var owner = await PostgresFixture.OpenAsync(database.MigrationConnectionString)) + { + await using var revoke = new NpgsqlCommand("REVOKE INSERT ON audit_log FROM learnstack_app", (NpgsqlConnection)owner); + await revoke.ExecuteNonQueryAsync(); + } + var saves = new SaveProbe(); + await using var host = Host(database, saves); + Func initialize = async () => await InitializeAsync(host.Services, tenant, taxonomy); + await initialize.Should().ThrowAsync(); + saves.Statuses.Should().Equal(["Active"], "the real save happened before audit refused the owning transaction"); + var state = await StateAsync(source, tenant, taxonomy); + state.Roots.Should().BeEmpty(); + state.Items.Should().BeEmpty(); + state.Generation.Should().Be(0); + (await AuditsAsync(source, tenant, taxonomy)).Should().BeEmpty(); + } + + [Theory] + [InlineData(false, false, 1)] + [InlineData(false, false, 2)] + [InlineData(false, true, 1)] + [InlineData(false, true, 2)] + [InlineData(true, false, 1)] + [InlineData(true, false, 2)] + [InlineData(true, true, 1)] + [InlineData(true, true, 2)] + public async Task Registration_and_initialization_share_the_concept_lock_in_both_winning_orders(bool taxonomy, bool initializeFirst, int registrationVersion) + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = NpgsqlDataSource.Create(database.AppConnectionString); + var tenant = await ProvisionAsync(source); + var gate = new ConceptGate(); + await using var host = Host(database, gate: gate); + var authoredId = Guid.CreateVersion7(); + async Task Initialize() => await InitializeAsync(host.Services, tenant, taxonomy); + async Task Register() => await RegisterAsync(host.Services, tenant, taxonomy, authoredId, registrationVersion); + var first = initializeFirst ? Initialize() : Register(); + await gate.Held.Task.WaitAsync(TimeSpan.FromSeconds(20)); + var second = initializeFirst ? Register() : Initialize(); + try + { + var waitingPid = await gate.Contender.Task.WaitAsync(TimeSpan.FromSeconds(20)); + await AssertAdvisoryWaitAsync(source, waitingPid); + second.IsCompleted.Should().BeFalse("the actual PostgreSQL transaction lock holds the competitor"); + } + finally { gate.Release.TrySetResult(); } + (await first).IsSuccess.Should().BeTrue(); + var secondResult = await second; + secondResult.IsSuccess.Should().Be(!initializeFirst || registrationVersion == 2); + var state = await StateAsync(source, tenant, taxonomy); + state.Generation.Should().Be(initializeFirst && registrationVersion == 2 ? 2 : 1); + state.Roots.Should().HaveCount(initializeFirst && registrationVersion == 2 ? 2 : 1); + var roots = state.Roots.Select(json => JsonNode.Parse(json)!).ToArray(); + if (initializeFirst) + { + roots.Single(root => root["schema_version"]!.GetValue() == 1)["status"]!.GetValue().Should().Be("Active"); + if (registrationVersion == 2) + roots.Single(root => root["schema_version"]!.GetValue() == 2)["id"]!.GetValue().Should().Be(authoredId.ToString()); + } + else + { + roots.Single()["id"]!.GetValue().Should().Be(authoredId.ToString()); + roots.Single()["status"]!.GetValue().Should().Be("Draft"); + roots.Single()["schema_version"]!.GetValue().Should().Be(registrationVersion); + } + var audit = (await AuditsAsync(source, tenant, taxonomy)).Should().ContainSingle().Which; + audit.Outcome.Should().Be("success"); + if (initializeFirst) audit.After.Should().NotBeNull(); + else + { + audit.EntityId.Should().BeNull(); + audit.After.Should().BeNull(); + } + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Another_tenants_same_key_history_does_not_suppress_initialization(bool taxonomy) + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = NpgsqlDataSource.Create(database.AppConnectionString); + var first = await ProvisionAsync(source); + var second = await ProvisionAsync(source); + await ArrangeHistoryAsync(source, first, taxonomy, "Deleted"); + var before = await StateAsync(source, first, taxonomy); + await using var host = Host(database); + (await InitializeAsync(host.Services, second, taxonomy)).IsSuccess.Should().BeTrue(); + (await StateAsync(source, second, taxonomy)).Roots.Should().ContainSingle(); + (await StateAsync(source, first, taxonomy)).Roots.Should().Equal(before.Roots); + } + + private WebApplicationFactory Host(DisposableSchemaDatabase database, SaveProbe? saves = null, ConceptGate? gate = null) => + factory.WithWebHostBuilder(builder => builder + .UseSetting("ConnectionStrings:Default", database.AppConnectionString) + .ConfigureServices(services => + { + if (saves is not null) services.AddSingleton(saves); + if (gate is null) return; + services.AddScoped(provider => new GatedContentStore( + new TenantContentTypeStore(provider.GetRequiredService(), provider.GetRequiredService()), + provider.GetRequiredService(), gate)); + services.AddScoped(provider => new GatedTaxonomyStore( + new TenantLevelTaxonomyStore(provider.GetRequiredService(), provider.GetRequiredService()), + provider.GetRequiredService(), gate)); + })); + + private static async Task ProvisionAsync(NpgsqlDataSource source) + { + var tenant = TenantId.From(Guid.CreateVersion7()); + await using var provider = SeedComposition.Build(source, null, NullLoggerFactory.Instance); + (await SendAsync(provider, null, new ProvisionTenantCommand(tenant, "default-" + tenant.Value.ToString("N"), "Default proof", + OrganizationId.From(Guid.CreateVersion7()), "main", "Main"))).IsSuccess.Should().BeTrue(); + return new(tenant); + } + + private static async Task> SendAsync(IServiceProvider services, ITenantContext? tenant, IRequest> command) + { + await using var scope = services.CreateAsyncScope(); + if (tenant is not null) scope.ServiceProvider.GetRequiredService().Current = tenant; + return await scope.ServiceProvider.GetRequiredService().Send(command); + } + + private static Task> InitializeAsync(IServiceProvider services, ITenantContext tenant, bool taxonomy) => taxonomy + ? SendAsync(services, tenant, new InitializeTenantLevelTaxonomyDefaultCommand()) + : SendAsync(services, tenant, new InitializeTenantContentTypeDefaultCommand()); + + private static async Task RegisterAsync(IServiceProvider services, ITenantContext? tenant, bool taxonomy, Guid id, int version) => taxonomy + ? await SendAsync(services, tenant, new RegisterTenantLevelTaxonomyCommand(id, "plain", version, + new Dictionary { ["en"] = "Authored taxonomy" }, + [new("custom", new Dictionary { ["en"] = "Authored band" }, 9, "{\"color\":\"custom\"}")])) + : await SendAsync(services, tenant, new RegisterTenantContentTypeCommand(id, "card", version, + new Dictionary { ["en"] = "Authored content" }, BuiltInCustomizations.Card.JsonSchema, BuiltInCustomizations.Card.RendererKey)); + + private static async Task ArrangeHistoryAsync(NpgsqlDataSource source, Context tenant, bool taxonomy, string history) + { + var id = Guid.CreateVersion7(); + await using var provider = SeedComposition.Build(source, tenant, NullLoggerFactory.Instance); + // This composition fixes its tenant; no accessor write or privileged setup is needed. + (await RegisterAsync(provider, null, taxonomy, id, history == "Successor" ? 2 : 1)).IsSuccess.Should().BeTrue(); + if (history is "Draft" or "Successor") return; + await using var scope = provider.CreateAsyncScope(); + var unit = scope.ServiceProvider.GetRequiredService(); + await using var frame = await unit.BeginTransactionAsync(); + await unit.SetTenantContextAsync(tenant); + var clock = scope.ServiceProvider.GetRequiredService(); + if (taxonomy) + { + var store = scope.ServiceProvider.GetRequiredService(); + var root = (await store.FindAsync(TenantLevelTaxonomyId.From(id)))!; + root.Publish(clock, UserId.SystemActor); + if (history == "Deprecated") root.Deprecate(clock, UserId.SystemActor); + if (history == "Deleted") root.SoftDelete(clock.UtcNow, UserId.SystemActor); + await store.UpdateAsync(root); + } + else + { + var store = scope.ServiceProvider.GetRequiredService(); + var root = (await store.FindAsync(TenantContentTypeId.From(id)))!; + root.Publish(clock, UserId.SystemActor); + if (history == "Deprecated") root.Deprecate(clock, UserId.SystemActor); + if (history == "Deleted") root.SoftDelete(clock.UtcNow, UserId.SystemActor); + await store.UpdateAsync(root); + } + await frame.CompleteAsync(); + } + + private static string Key(bool taxonomy) => taxonomy ? "plain" : "card"; + + private static async Task StateAsync(NpgsqlDataSource source, ITenantContext tenant, bool taxonomy) + { + await using var connection = await source.OpenConnectionAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.SetTenantAsync(connection, transaction, tenant.TenantId.Value); + var table = taxonomy ? "tenant_level_taxonomies" : "tenant_content_types"; + var roots = await RowsAsync($"SELECT to_jsonb(t)::text FROM {table} t WHERE tenant_id = @tenant AND key = @key ORDER BY schema_version"); + var items = taxonomy + ? await RowsAsync("SELECT to_jsonb(t)::text FROM tenant_level_taxonomy_items t WHERE tenant_id = @tenant AND taxonomy_key = @key ORDER BY schema_version, sort") + : []; + await using var generation = new NpgsqlCommand("SELECT COALESCE((SELECT generation FROM customization_generations WHERE tenant_id = @tenant), 0)", connection, transaction); + generation.Parameters.AddWithValue("tenant", tenant.TenantId.Value); + return new(roots, items, (long)(await generation.ExecuteScalarAsync())!); + + async Task> RowsAsync(string sql) + { + await using var command = new NpgsqlCommand(sql, connection, transaction); + command.Parameters.AddWithValue("tenant", tenant.TenantId.Value); + command.Parameters.AddWithValue("key", Key(taxonomy)); + var rows = new List(); + await using var reader = await command.ExecuteReaderAsync(); + while (await reader.ReadAsync()) rows.Add(reader.GetString(0)); + return rows; + } + } + + private static async Task> AuditsAsync(NpgsqlDataSource source, ITenantContext tenant, bool taxonomy) + { + await using var connection = await source.OpenConnectionAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.SetTenantAsync(connection, transaction, tenant.TenantId.Value); + await using var command = new NpgsqlCommand(""" + SELECT operation_type, operation_class, outcome, entity_type, entity_id, actor_user_id, + before_state::text, after_state::text, changes::text + FROM audit_log WHERE tenant_id = @tenant AND operation = @operation + """, connection, transaction); + command.Parameters.AddWithValue("tenant", tenant.TenantId.Value); + command.Parameters.AddWithValue("operation", taxonomy ? "customization.level_taxonomy.initialize" : "customization.content_type.initialize"); + var rows = new List(); + await using var reader = await command.ExecuteReaderAsync(); + while (await reader.ReadAsync()) + rows.Add(new(reader.GetString(0), reader.GetString(1), reader.GetString(2), Text(3), Text(4), reader.GetGuid(5), Text(6), Text(7), Text(8))); + return rows; + string? Text(int index) => reader.IsDBNull(index) ? null : reader.GetString(index); + } + + private static void AssertNames(JsonNode actual, IReadOnlyDictionary expected) + { + actual.AsObject().Count.Should().Be(expected.Count); + foreach (var (locale, text) in expected) actual[locale]!.GetValue().Should().Be(text); + } + + private static async Task AssertAdvisoryWaitAsync(NpgsqlDataSource source, int pid) + { + using var timeout = new CancellationTokenSource(TimeSpan.FromSeconds(10)); + await using var connection = await source.OpenConnectionAsync(timeout.Token); + while (!timeout.IsCancellationRequested) + { + await using var query = new NpgsqlCommand("SELECT wait_event FROM pg_stat_activity WHERE pid = @pid", connection); + query.Parameters.AddWithValue("pid", pid); + if (await query.ExecuteScalarAsync(timeout.Token) is "advisory") return; + await Task.Delay(20, timeout.Token); + } + throw new InvalidOperationException("The competing writer never waited on the concept lock."); + } + + private sealed record State(List Roots, List Items, long Generation); + private sealed record Context(TenantId TenantId) : ITenantContext + { + public bool IsResolved => true; + public TenantContextOrigin? Origin => TenantContextOrigin.Ambient; + public OrganizationId? OrganizationId => null; + public UserId? UserId => LearnStack.SharedKernel.Identifiers.UserId.SystemActor; + public string? CorrelationId => null; + public string? ModuleName => "customization"; + } + private sealed record Audit(string OperationType, string Class, string Outcome, string? EntityType, + string? EntityId, Guid Actor, string? Before, string? After, string? Changes); + + private sealed class SaveProbe : SaveChangesInterceptor + { + internal List Statuses { get; } = []; + public override ValueTask> SavingChangesAsync(DbContextEventData eventData, + InterceptionResult result, CancellationToken cancellationToken = default) + { + if (eventData.Context is CustomizationDbContext db) + { + var roots = db.ChangeTracker.Entries().Where(entry => entry.State == EntityState.Added) + .Select(entry => entry.Entity).Where(entity => entity is TenantContentType or TenantLevelTaxonomy).ToArray(); + Statuses.AddRange(roots.Select(root => root is TenantContentType content ? content.Status.ToString() : ((TenantLevelTaxonomy)root).Status.ToString())); + roots.Should().ContainSingle("each initializer saves one complete aggregate root"); + } + return ValueTask.FromResult(result); + } + } + + private sealed class ConceptGate + { + private int _calls; + internal TaskCompletionSource Held { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + internal TaskCompletionSource Contender { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + internal TaskCompletionSource Release { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + internal async Task LockAsync(Func acquire, IUnitOfWork unit, CancellationToken cancellationToken) + { + var first = Interlocked.Increment(ref _calls) == 1; + if (!first) Contender.TrySetResult(((NpgsqlConnection)unit.Connection).ProcessID); + await acquire(); + if (!first) return; + Held.TrySetResult(); + await Release.Task.WaitAsync(cancellationToken); + } + } + + private sealed class GatedContentStore(ITenantContentTypeStore inner, IUnitOfWork unit, ConceptGate gate) : ITenantContentTypeStore + { + public Task LockConceptAsync(string key, CancellationToken cancellationToken = default) => gate.LockAsync(() => inner.LockConceptAsync(key, cancellationToken), unit, cancellationToken); + public Task HasConceptHistoryAsync(string key, CancellationToken cancellationToken = default) => inner.HasConceptHistoryAsync(key, cancellationToken); + public Task FindAsync(TenantContentTypeId id, CancellationToken cancellationToken = default) => inner.FindAsync(id, cancellationToken); + public Task FindActiveAsync(string key, CancellationToken cancellationToken = default) => inner.FindActiveAsync(key, cancellationToken); + public Task AddAsync(TenantContentType root, CancellationToken cancellationToken = default) => inner.AddAsync(root, cancellationToken); + public Task UpdateAsync(TenantContentType root, CancellationToken cancellationToken = default) => inner.UpdateAsync(root, cancellationToken); + } + + private sealed class GatedTaxonomyStore(ITenantLevelTaxonomyStore inner, IUnitOfWork unit, ConceptGate gate) : ITenantLevelTaxonomyStore + { + public Task LockConceptAsync(string key, CancellationToken cancellationToken = default) => gate.LockAsync(() => inner.LockConceptAsync(key, cancellationToken), unit, cancellationToken); + public Task HasConceptHistoryAsync(string key, CancellationToken cancellationToken = default) => inner.HasConceptHistoryAsync(key, cancellationToken); + public Task FindAsync(TenantLevelTaxonomyId id, CancellationToken cancellationToken = default) => inner.FindAsync(id, cancellationToken); + public Task FindActiveAsync(string key, CancellationToken cancellationToken = default) => inner.FindActiveAsync(key, cancellationToken); + public Task AddAsync(TenantLevelTaxonomy root, CancellationToken cancellationToken = default) => inner.AddAsync(root, cancellationToken); + public Task UpdateAsync(TenantLevelTaxonomy root, CancellationToken cancellationToken = default) => inner.UpdateAsync(root, cancellationToken); + } +} diff --git a/backend/tests/LearnStack.Tests.Integration/Database/CustomizationProjectionCompositionTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/CustomizationProjectionCompositionTests.cs index 241aa877..0c8c3ab1 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/CustomizationProjectionCompositionTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/CustomizationProjectionCompositionTests.cs @@ -135,8 +135,10 @@ public async Task Seeded_local_measurement_records_statement_plans_payload_volum var context = new SeedTenantContext(tenant.TenantId, null); await using var read = await ReadSession.OpenAsync(source, context, cache: cache); await AssertAppRoleAsync(read.Unit); - var request = Request([.. SeedData.ContentTypes(tenant).Select(type => new DefinitionRevision(type.Key, type.SchemaVersion))], - [.. SeedData.Taxonomies(tenant).Select(taxonomy => new DefinitionRevision(taxonomy.Key, taxonomy.SchemaVersion))]); + var request = Request([new(BuiltInCustomizations.Card.Key, BuiltInCustomizations.SchemaVersion), + .. SeedData.ContentTypes(tenant).Select(type => new DefinitionRevision(type.Key, type.SchemaVersion))], + [new(BuiltInCustomizations.Plain.Key, BuiltInCustomizations.SchemaVersion), + .. SeedData.Taxonomies(tenant).Select(taxonomy => new DefinitionRevision(taxonomy.Key, taxonomy.SchemaVersion))]); var initial = (await read.Reader.ReadAsync(request)).Value!; initial.ContentTypes.Should().HaveCount(2); initial.Taxonomies.Should().HaveCount(2); diff --git a/backend/tests/LearnStack.Tests.Integration/Database/CustomizationPublicationConcurrencyTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/CustomizationPublicationConcurrencyTests.cs index e65838e4..f45ebe77 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/CustomizationPublicationConcurrencyTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/CustomizationPublicationConcurrencyTests.cs @@ -236,6 +236,8 @@ private static void Fail(bool concurrency, bool taxonomy) } private sealed class ControlledContentStore(ITenantContentTypeStore inner, ActiveReadGate? gate = null, bool? failure = null) : ITenantContentTypeStore { + public Task LockConceptAsync(string key, CancellationToken cancellationToken = default) => inner.LockConceptAsync(key, cancellationToken); + public Task HasConceptHistoryAsync(string key, CancellationToken cancellationToken = default) => inner.HasConceptHistoryAsync(key, cancellationToken); public Task FindAsync(TenantContentTypeId id, CancellationToken ct = default) => inner.FindAsync(id, ct); public async Task FindActiveAsync(string key, CancellationToken ct = default) { @@ -251,6 +253,8 @@ public async Task UpdateAsync(TenantContentType root, CancellationToken ct = def } private sealed class ControlledTaxonomyStore(ITenantLevelTaxonomyStore inner, ActiveReadGate? gate = null, bool? failure = null) : ITenantLevelTaxonomyStore { + public Task LockConceptAsync(string key, CancellationToken cancellationToken = default) => inner.LockConceptAsync(key, cancellationToken); + public Task HasConceptHistoryAsync(string key, CancellationToken cancellationToken = default) => inner.HasConceptHistoryAsync(key, cancellationToken); public Task FindAsync(TenantLevelTaxonomyId id, CancellationToken ct = default) => inner.FindAsync(id, ct); public async Task FindActiveAsync(string key, CancellationToken ct = default) { diff --git a/backend/tests/LearnStack.Tests.Integration/Database/DeliveryDispatchStoreTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/DeliveryDispatchStoreTests.cs new file mode 100644 index 00000000..1d879c60 --- /dev/null +++ b/backend/tests/LearnStack.Tests.Integration/Database/DeliveryDispatchStoreTests.cs @@ -0,0 +1,797 @@ +using System.Data.Common; +using System.Text.Json.Nodes; +using FluentAssertions; +using LearnStack.Infrastructure.Messaging; +using LearnStack.SharedKernel.Audit; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Messaging; +using LearnStack.SharedKernel.Persistence; +using LearnStack.SharedKernel.Time; +using LearnStack.Tools.Seeder; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using NpgsqlTypes; +using Xunit; + +namespace LearnStack.Tests.Integration.Database; + +[Trait(RequiresDocker.Key, RequiresDocker.Value)] +[Collection(SharedSchema.Name)] +public sealed class DeliveryDispatchStoreTests(SchemaFixture schema) +{ + private const string First = "test.first.v1"; + private const string Second = "test.second.v1"; + private const string Pending = "test.pending.v1"; + private const string Privileged = "test.privileged.v1"; + private const string Alias = "declared.previous.delivery"; + private const string Trace = "00-11111111111111111111111111111111-2222222222222222-01"; + private const string AttemptTrace = "00-33333333333333333333333333333333-4444444444444444-01"; + private static readonly string TypeIdentity = typeof(DeliveryEvent).AssemblyQualifiedName!; + private static readonly DateTimeOffset Occurred = new(2026, 10, 1, 12, 0, 0, TimeSpan.Zero); + private static readonly string[] NoBlocked = []; + + [Fact] + public async Task Two_stores_acquire_one_tuple_once_and_siblings_spend_independent_budgets() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + var claims = await Task.WhenAll(test.Store().TryClaimAsync(NoBlocked), test.Store().TryClaimAsync(NoBlocked)); + var first = claims.OfType().Should().ContainSingle().Which; + first.EventId.Should().Be(id); + first.SubscriptionId.Should().Be(First); + first.Attempts.Should().Be(1); + first.Token.Should().Be(Guid.ParseExact(first.Token, "D").ToString("D")); + (await test.Store().TryClaimAsync(NoBlocked)).Should().BeNull(); + await test.InsertAsync(id, Second); + var sibling = (await test.Store().TryClaimAsync(NoBlocked))!; + sibling.SubscriptionId.Should().Be(Second); + sibling.Attempts.Should().Be(1); + (await test.Store().FailAsync(first, DeliveryFailureReason.ExecutionFailed)).Should().BeTrue(); + (await test.ReadAsync(id, Second)).Token.Should().Be(sibling.Token); + (await test.ReadAsync(id, Second)).Attempts.Should().Be(1); + (await test.Store().FailAsync(first with { SubscriptionId = Second }, DeliveryFailureReason.ExecutionFailed)) + .Should().BeFalse(); + } + + [Fact] + public async Task Normal_unknown_commit_and_expired_recovery_have_distinct_stored_diagnostics() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + var store = test.Store(); + var claim = (await store.TryClaimAsync(NoBlocked))!; + + (await store.FailAsync(claim, DeliveryFailureReason.UnknownOutcome)).Should().BeTrue(); + (await test.ReadAsync(id)).Error.Should().Be("execute:unknown:commit"); + + await test.DueAsync(id); + (await store.TryClaimAsync(NoBlocked)).Should().NotBeNull(); + await test.ExpireAsync(id); + var recovery = (await store.TryAcquireRecoveryAsync(NoBlocked))!; + (await store.FailAsync(recovery, DeliveryFailureReason.UnknownOutcome)).Should().BeTrue(); + (await test.ReadAsync(id)).Error.Should().Be("recovery:unknown:expired"); + } + + [Fact] + public async Task Normal_claim_cannot_terminalize_without_its_separate_acquisition() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + var store = test.Store(); + var claim = (await store.TryClaimAsync(NoBlocked))!; + var before = await test.ReadAsync(id); + + var rejected = () => store.TryTerminalizeAsync(claim, AttemptTrace); + await rejected.Should().ThrowAsync(); + + (await test.ReadAsync(id)).Should().Be(before); + (await test.AuditCountAsync()).Should().Be(0); + (await store.RequestTerminalAsync(claim, DeliveryTerminalReason.InvalidEnvelope)).Should().BeTrue(); + var terminal = (await store.TryClaimTerminalAsync(NoBlocked))!; + (await store.TryTerminalizeAsync(terminal, AttemptTrace)).Should().BeTrue(); + } + + [Fact] + public async Task Terminal_claim_cannot_spend_an_execution_retry() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + var store = test.Store(); + var claim = (await store.TryClaimAsync(NoBlocked))!; + (await store.RequestTerminalAsync(claim, DeliveryTerminalReason.InvalidEnvelope)).Should().BeTrue(); + var terminal = (await store.TryClaimTerminalAsync(NoBlocked))!; + var before = await test.ReadAsync(id); + + var rejected = () => store.FailAsync(terminal, DeliveryFailureReason.ExecutionFailed); + await rejected.Should().ThrowAsync(); + + (await test.ReadAsync(id)).Should().Be(before); + (await test.AuditCountAsync()).Should().Be(0); + (await store.TryTerminalizeAsync(terminal, AttemptTrace)).Should().BeTrue(); + } + + [Fact] + public async Task Actual_claims_follow_all_five_index_order_keys() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var firstId = Guid.Parse("11111111-1111-7111-8111-111111111111"); + var laterId = Guid.Parse("22222222-2222-7222-8222-222222222222"); + await test.InsertAsync(firstId, Second); + await test.InsertAsync(firstId, First, tenant: SchemaFixture.TenantB); + await test.InsertAsync(firstId, First); + await test.InsertAsync(laterId, First); + var laterOccurrence = await test.InsertAsync(occurred: Occurred.AddHours(1)); + var laterAvailability = await test.InsertAsync(); + await test.SetOrderingAsync(laterAvailability); + var store = test.Store(); + var claims = new List(); + while (await store.TryClaimAsync(NoBlocked) is { } claim) claims.Add(claim); + claims.Select(claim => (claim.EventId, claim.TenantId, claim.SubscriptionId)).Should().Equal( + (firstId, SchemaFixture.TenantA, First), (firstId, SchemaFixture.TenantA, Second), + (firstId, SchemaFixture.TenantB, First), (laterId, SchemaFixture.TenantA, First), + (laterOccurrence, SchemaFixture.TenantA, First), (laterAvailability, SchemaFixture.TenantA, First)); + } + + [Theory] + [InlineData(1, 1)] + [InlineData(2, 5)] + [InlineData(3, 30)] + [InlineData(4, 300)] + [InlineData(5, 3600)] + public async Task Normal_failure_uses_each_database_delay_and_clears_ownership(int attempt, int seconds) + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + await test.UpdateAsync(id, First, $"attempts = {attempt - 1}"); + var store = test.Store(); + var claim = (await store.TryClaimAsync(NoBlocked))!; + var before = await test.NowAsync(); + (await store.FailAsync(claim, DeliveryFailureReason.ExecutionFailed)).Should().BeTrue(); + var after = await test.NowAsync(); + var row = await test.ReadAsync(id); + row.Available.Should().BeOnOrAfter(before.AddSeconds(seconds)).And.BeOnOrBefore(after.AddSeconds(seconds)); + row.Token.Should().BeNull(); + row.LockedUntil.Should().BeNull(); + row.Attempts.Should().Be(attempt); + row.Error.Should().Be("execute:failed:handler"); + } + + [Fact] + public async Task Pair_specific_holds_preserve_pending_and_privileged_but_allow_active_pending_sibling_and_unknown_pair() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var pending = await test.InsertAsync(subscription: Pending, type: Alias); + var privileged = await test.InsertAsync(subscription: Privileged); + var active = await test.InsertAsync(); + var unknown = await test.InsertAsync(subscription: Pending, type: "private.unknown.type"); + var blocked = await test.InsertAsync(partition: "local-block"); + var store = test.Store(); + var snapshot = await store.ReadSnapshotAsync(["local-block"]); + snapshot.PendingCount.Should().Be(5); + snapshot.DeferredCount.Should().Be(2); + var claimed = new[] { (await store.TryClaimAsync(["local-block"]))!, (await store.TryClaimAsync(["local-block"]))! }; + claimed.Select(c => c.EventId).Should().BeEquivalentTo([active, unknown]); + (await store.TryClaimAsync(["local-block"])).Should().BeNull(); + (await test.ReadAsync(pending, Pending)).Attempts.Should().Be(0); + (await test.ReadAsync(privileged, Privileged)).Attempts.Should().Be(0); + (await store.TryClaimAsync(NoBlocked))!.EventId.Should().Be(blocked); + } + + [Theory] + [InlineData(1, 1)] + [InlineData(2, 5)] + [InlineData(3, 30)] + [InlineData(4, 300)] + [InlineData(5, 3600)] + public async Task Recovery_preserves_attempt_budget_and_sets_database_time_backoff(int attempt, int delay) + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + await test.UpdateAsync(id, First, $"attempts = {attempt - 1}"); + var store = test.Store(); + var stale = (await store.TryClaimAsync(NoBlocked))!; + await test.ExpireAsync(id); + (await store.FailAsync(stale, DeliveryFailureReason.ExecutionFailed)).Should().BeFalse(); + (await store.RequestTerminalAsync(stale, DeliveryTerminalReason.InvalidEnvelope)).Should().BeFalse(); + (await store.TryClaimAsync(NoBlocked)).Should().BeNull(); + var recovery = (await store.TryAcquireRecoveryAsync(NoBlocked))!; + recovery.Kind.Should().Be(DeliveryClaimKind.Recovery); + recovery.Token.Should().NotBe(stale.Token); + recovery.Attempts.Should().Be(attempt); + (await store.FailAsync(stale, DeliveryFailureReason.ExecutionFailed)).Should().BeFalse(); + var before = await test.NowAsync(); + (await store.FailAsync(recovery, DeliveryFailureReason.UnknownOutcome)).Should().BeTrue(); + var after = await test.NowAsync(); + var row = await test.ReadAsync(id); + row.Available.Should().BeOnOrAfter(before.AddSeconds(delay)).And.BeOnOrBefore(after.AddSeconds(delay)); + row.Attempts.Should().Be(attempt); + row.Token.Should().BeNull(); + row.TerminalReason.Should().BeNull(); + row.Error.Should().Be("recovery:unknown:expired"); + (await store.TryClaimAsync(NoBlocked)).Should().BeNull(); + await test.DueAsync(id); + (await store.TryClaimAsync(NoBlocked))!.Attempts.Should().Be(attempt + 1); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Sixth_failure_or_recovery_is_terminal_only_and_never_spends_a_seventh_claim(bool recovery) + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + await test.UpdateAsync(id, First, "attempts = 5"); + var store = test.Store(); + var claim = (await store.TryClaimAsync(NoBlocked))!; + claim.Attempts.Should().Be(6); + if (recovery) + { + await test.ExpireAsync(id); + claim = (await store.TryAcquireRecoveryAsync(NoBlocked))!; + } + (await store.FailAsync(claim, recovery ? DeliveryFailureReason.UnknownOutcome : DeliveryFailureReason.ExecutionTimeout)) + .Should().BeTrue(); + (await store.TryClaimAsync(NoBlocked)).Should().BeNull(); + (await store.TryAcquireRecoveryAsync(NoBlocked)).Should().BeNull(); + var terminal = (await store.TryClaimTerminalAsync(NoBlocked))!; + terminal.Attempts.Should().Be(6); + (await store.TryTerminalizeAsync(terminal, AttemptTrace)).Should().BeTrue(); + (await test.ReadAsync(id)).TerminalReason.Should().Be("retry_exhausted"); + (await test.AuditCountAsync()).Should().Be(1); + } + + [Fact] + public async Task Completed_sixth_attempt_with_inert_lease_is_excluded_from_recovery_without_inbox_access() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + await test.UpdateAsync(id, First, "attempts = 5"); + var store = test.Store(); + var claim = (await store.TryClaimAsync(NoBlocked))!; + await test.CompleteAsync(id, First, inbox: true); + await test.ExpireAsync(id); + (await store.TryAcquireRecoveryAsync(NoBlocked)).Should().BeNull(); + (await store.TryClaimAsync(NoBlocked)).Should().BeNull(); + (await store.TryClaimTerminalAsync(NoBlocked)).Should().BeNull(); + (await store.FailAsync(claim, DeliveryFailureReason.UnknownOutcome)).Should().BeFalse(); + (await store.ReplayAsync(SchemaFixture.TenantA, First, id)).Should().BeFalse(); + (await test.InboxCountAsync()).Should().Be(1); + } + + [Theory] + [InlineData("failure")] + [InlineData("request")] + [InlineData("terminal")] + public async Task Unchanged_row_lock_wait_cannot_use_pre_wait_lease_time(string action) + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + var store = test.Store(); + var claim = (await store.TryClaimAsync(NoBlocked))!; + if (action == "terminal") + { + (await store.RequestTerminalAsync(claim, DeliveryTerminalReason.InvalidEnvelope)).Should().BeTrue(); + claim = (await store.TryClaimTerminalAsync(NoBlocked))!; + } + await test.UpdateAsync(id, First, "locked_until = clock_timestamp() + interval '1 second'"); + await using var connection = await test.Dispatch.OpenConnectionAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await using var rowLock = new NpgsqlCommand("SELECT event_id FROM event_deliveries WHERE event_id = @id FOR UPDATE", connection, transaction); + rowLock.Parameters.AddWithValue("id", id); + await rowLock.ExecuteScalarAsync(); + var transition = action switch + { + "failure" => store.FailAsync(claim, DeliveryFailureReason.ExecutionFailed), + "request" => store.RequestTerminalAsync(claim, DeliveryTerminalReason.InvalidEnvelope), + _ => store.TryTerminalizeAsync(claim, AttemptTrace), + }; + await test.WaitForLockWaitAsync(); + using var expiryDeadline = new CancellationTokenSource(TimeSpan.FromSeconds(5)); + while ((await test.ReadAsync(id, cancellationToken: expiryDeadline.Token)).LockedUntil + > await test.NowAsync(expiryDeadline.Token)) + { + await Task.Delay(20, expiryDeadline.Token); + } + await transaction.CommitAsync(); + (await transition).Should().BeFalse(); + (await test.ReadAsync(id)).Token.Should().Be(claim.Token); + (await test.AuditCountAsync()).Should().Be(0); + } + + [Fact] + public async Task Audit_refusal_rolls_back_transition_then_reacquires_terminal_only_with_actual_metadata() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(organization: SchemaFixture.OrgA1); + var store = test.Store(); + var terminal = await TestDatabase.RequestTerminalAsync(store); + var before = await test.ReadAsync(id); + await test.AuditGrantAsync(false); + var attempt = () => store.TryTerminalizeAsync(terminal, AttemptTrace); + await attempt.Should().ThrowAsync(); + (await test.ReadAsync(id)).Should().Be(before); + (await test.AuditCountAsync()).Should().Be(0); + await test.AuditGrantAsync(true); + await test.ExpireAsync(id); + (await store.TryClaimAsync(NoBlocked)).Should().BeNull(); + var fresh = (await store.TryClaimTerminalAsync(NoBlocked))!; + fresh.Token.Should().NotBe(terminal.Token); + (await store.TryTerminalizeAsync(terminal, AttemptTrace)).Should().BeFalse(); + (await store.TryTerminalizeAsync(fresh, AttemptTrace)).Should().BeTrue(); + var audit = await test.ReadAuditAsync(); + audit.Tenant.Should().Be(SchemaFixture.TenantA); + audit.Organization.Should().Be(SchemaFixture.OrgA1); + audit.Actor.Should().Be(UserId.SystemActor.Value); + audit.Correlation.Should().Be(AttemptTrace); + audit.Operation.Should().Be("platform.delivery.dead_letter"); + audit.Class.Should().Be("Must"); + audit.Type.Should().Be("SecurityEvent"); + var metadata = JsonNode.Parse(audit.Metadata)!.AsObject(); + metadata.Select(pair => pair.Key).Should().BeEquivalentTo(["event_id", "subscription_id", "reason", "attempts"]); + metadata["event_id"]!.GetValue().Should().Be(id); + metadata["subscription_id"]!.GetValue().Should().Be(First); + (await store.TryTerminalizeAsync(fresh, AttemptTrace)).Should().BeFalse(); + (await test.AuditCountAsync()).Should().Be(1); + } + + [Theory] + [InlineData("tenant")] + [InlineData("organization")] + [InlineData("event")] + [InlineData("subscription")] + [InlineData("operation")] + [InlineData("class")] + [InlineData("type")] + [InlineData("actor")] + [InlineData("trace")] + [InlineData("transaction")] + public async Task Substitution_against_actual_subscriber_witness_rolls_back_both_writes(string substitution) + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(organization: SchemaFixture.OrgA1); + var calls = 0; + var store = test.Store(decorate: inner => new AuditDecorator(inner, async (draft, witness, connection, transaction, token) => + { + calls++; + witness.Kind.Should().Be("subscriber"); + witness.SubscriptionId.Should().Be(First); + witness.EventId.Should().Be(id); + await using var truth = new NpgsqlCommand("SELECT dead_lettered_at IS NOT NULL FROM event_deliveries WHERE event_id = @id", + (NpgsqlConnection)connection, (NpgsqlTransaction)transaction); + truth.Parameters.AddWithValue("id", id); + (await truth.ExecuteScalarAsync(token)).Should().Be(true); + if (substitution == "transaction") + { + await using var other = await test.Dispatch.OpenConnectionAsync(token); + await using var otherTransaction = await other.BeginTransactionAsync(token); + await inner.WriteEventTerminalAsync(draft, witness, other, otherTransaction, token); + return; + } + var metadata = JsonNode.Parse(draft.Metadata!)!.AsObject(); + if (substitution == "event") metadata["event_id"] = Guid.NewGuid(); + if (substitution == "subscription") metadata["subscription_id"] = Second; + var changed = substitution switch + { + "tenant" => draft with { TenantId = TenantId.From(SchemaFixture.TenantB) }, + "organization" => draft with { OrganizationId = OrganizationId.From(SchemaFixture.OrgA2) }, + "operation" => draft with { Operation = "platform.outbox.dead_letter" }, + "class" => draft with { OperationClass = OperationClass.Should }, + "type" => draft with { OperationType = OperationType.Update }, + "actor" => draft with { ActorUserId = UserId.From(Guid.CreateVersion7()) }, + "trace" => draft with { CorrelationId = Trace }, + _ => draft with { Metadata = metadata.ToJsonString() }, + }; + await inner.WriteEventTerminalAsync(changed, witness, connection, transaction, token); + })); + var terminal = await TestDatabase.RequestTerminalAsync(store); + var before = await test.ReadAsync(id); + var attempt = () => store.TryTerminalizeAsync(terminal, AttemptTrace); + await attempt.Should().ThrowAsync(); + calls.Should().Be(1); + (await test.ReadAsync(id)).Should().Be(before); + (await test.AuditCountAsync()).Should().Be(0); + (await test.Store().TryTerminalizeAsync(terminal, AttemptTrace)).Should().BeTrue(); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Lost_terminal_commit_ack_reconciles_once_or_preserves_both_failures(bool rejectProbe) + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + var terminal = await TestDatabase.RequestTerminalAsync(test.Store()); + await using var proxy = new CommitResponseDroppingProxy(test.OutboxConnectionString, rejectReconciliationConnections: rejectProbe); + await using var source = OutboxDispatcherDataSource.Build(proxy.ConnectionString); + var store = test.Store(source: source); + if (rejectProbe) + { + var attempt = () => store.TryTerminalizeAsync(terminal, AttemptTrace); + var error = (await attempt.Should().ThrowAsync()).Which; + error.InnerExceptions.Should().HaveCount(2); + error.InnerExceptions[0].StackTrace.Should().Contain("NpgsqlTransaction.Commit"); + error.InnerExceptions[1].StackTrace.Should().Contain("IsDurablyTerminalAsync"); + } + else (await store.TryTerminalizeAsync(terminal, AttemptTrace)).Should().BeTrue(); + proxy.DroppedCommitResponses.Should().Be(1); + proxy.AcceptedConnections.Should().BeGreaterThanOrEqualTo(2); + (await test.ReadAsync(id)).DeadLettered.Should().NotBeNull(); + (await test.AuditCountAsync()).Should().Be(1); + (await test.Store().TryTerminalizeAsync(terminal, AttemptTrace)).Should().BeFalse(); + } + + [Fact] + public async Task Replay_resets_only_exact_active_terminal_tuple_and_keeps_completed_sibling_inbox_and_stale_fences() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + await test.InsertAsync(id, Second); + var store = test.Store(); + var terminal = await TestDatabase.RequestTerminalAsync(store); + // Mark only sibling completed through the actual app completion grant. + await test.CompleteAsync(id, Second, inbox: true); + var sibling = await test.ReadAsync(id, Second); + (await store.TryTerminalizeAsync(terminal, AttemptTrace)).Should().BeTrue(); + (await store.ReplayAsync(SchemaFixture.TenantB, First, id)).Should().BeFalse(); + (await store.ReplayAsync(SchemaFixture.TenantA, Second, id)).Should().BeFalse(); + (await store.ReplayAsync(SchemaFixture.TenantA, First, id)).Should().BeTrue(); + var reset = await test.ReadAsync(id); + reset.Attempts.Should().Be(0); + reset.Error.Should().BeNull(); + reset.TerminalReason.Should().BeNull(); + reset.DeadLettered.Should().BeNull(); + reset.ReplayCount.Should().Be(1); + reset.Replayed.Should().NotBeNull(); + reset.Token.Should().BeNull(); + (await test.ReadAsync(id, Second)).Should().Be(sibling); + (await test.InboxCountAsync()).Should().Be(1); + (await store.TryTerminalizeAsync(terminal, AttemptTrace)).Should().BeFalse(); + var fresh = (await store.TryClaimAsync(NoBlocked))!; + fresh.SubscriptionId.Should().Be(First); + fresh.Token.Should().NotBe(terminal.Token); + fresh.Attempts.Should().Be(1); + (await store.ReplayAsync(SchemaFixture.TenantA, First, id)).Should().BeFalse(); + } + + [Fact] + public async Task Replay_preserves_other_terminal_rows_differing_in_each_tuple_component() + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + await test.InsertAsync(id, Second); + await test.InsertAsync(id, tenant: SchemaFixture.TenantB); + var unrelatedId = await test.InsertAsync(); + var store = test.Store(); + for (var index = 0; index < 4; index++) + { + var terminal = await TestDatabase.RequestTerminalAsync(store); + (await store.TryTerminalizeAsync(terminal, AttemptTrace)).Should().BeTrue(); + } + (await store.TryClaimAsync(NoBlocked)).Should().BeNull(); + var target = await test.ReadAsync(id); + var sibling = await test.ReadAsync(id, Second); + var foreign = await test.ReadAsync(id, tenant: SchemaFixture.TenantB); + var unrelated = await test.ReadAsync(unrelatedId); + foreach (var row in new[] { target, sibling, foreign, unrelated }) + row.DeadLettered.Should().NotBeNull("every control must itself satisfy the terminal replay state predicate"); + + (await store.ReplayAsync(SchemaFixture.TenantA, First, id)).Should().BeTrue(); + + var reset = await test.ReadAsync(id); + reset.Attempts.Should().Be(0); + reset.TerminalReason.Should().BeNull(); + reset.DeadLettered.Should().BeNull(); + reset.Error.Should().BeNull(); + reset.Token.Should().BeNull(); + reset.LockedUntil.Should().BeNull(); + reset.ReplayCount.Should().Be(target.ReplayCount + 1); + reset.Replayed.Should().NotBeNull(); + (await test.ReadAsync(id, Second)).Should().Be(sibling); + (await test.ReadAsync(id, tenant: SchemaFixture.TenantB)).Should().Be(foreign); + (await test.ReadAsync(unrelatedId)).Should().Be(unrelated); + } + + [Theory] + [InlineData("pending")] + [InlineData("privileged")] + [InlineData("unknown")] + [InlineData("retired")] + [InlineData("requested")] + [InlineData("live")] + public async Task Replay_refuses_nonadmitted_or_unfinished_state(string state) + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + var subscription = state == "pending" ? Pending : state == "privileged" ? Privileged : First; + var type = state is "unknown" or "retired" ? state : TypeIdentity; + var id = await test.InsertAsync(subscription: subscription, type: type); + await test.UpdateAsync(id, subscription, state switch + { + "live" => "locked_by = '11111111-1111-4111-8111-111111111111', locked_until = clock_timestamp() + interval '1 hour'", + "requested" => "terminal_reason = 'invalid_envelope'", + _ => "terminal_reason = 'invalid_envelope', dead_lettered_at = clock_timestamp()", + }); + var before = await test.ReadAsync(id, subscription); + (await test.Store().ReplayAsync(SchemaFixture.TenantA, subscription, id)).Should().BeFalse(); + (await test.ReadAsync(id, subscription)).Should().Be(before); + } + + [Theory] + [InlineData("payload")] + [InlineData("correlation_id")] + [InlineData("causation_id")] + [InlineData("actor_user_id")] + [InlineData("inbox")] + public async Task Real_scheduler_role_cannot_read_payload_causality_or_inbox(string column) + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + await test.InsertAsync(); + await using var connection = await test.Dispatch.OpenConnectionAsync(); + await using var command = new NpgsqlCommand(column == "inbox" ? "SELECT event_id FROM inbox_messages" + : $"SELECT {column} FROM event_deliveries", connection); + var read = () => command.ExecuteScalarAsync(); + (await read.Should().ThrowAsync()).Which.SqlState.Should().Be(PostgresErrorCodes.InsufficientPrivilege); + (await test.Store().TryClaimAsync(NoBlocked)).Should().NotBeNull(); + } + + private static OutboxEventRegistry Registry() + { + var handlers = IntegrationEventHandlerRegistry.FromServiceDescriptors([ + ServiceDescriptor.Scoped, FirstHandler>(), + ServiceDescriptor.Scoped, SecondHandler>(), + ServiceDescriptor.Scoped, PrivilegedHandler>(), + ]); + return OutboxEventRegistry.Create([OutboxEventRegistration.Required(typeof(DeliveryEvent), [ + OutboxSubscriptionManifest.Active(First, typeof(FirstHandler)), + OutboxSubscriptionManifest.Active(Second, typeof(SecondHandler)), + OutboxSubscriptionManifest.Pending(Pending), + OutboxSubscriptionManifest.Active(Privileged, typeof(PrivilegedHandler), OutboxSubscriptionExecutionScope.PrivilegedPlatform), + ], Alias)], ["retired"], handlers, [typeof(DeliveryEvent).Assembly]); + } + + [IntegrationEventPartition(nameof(TenantId), tenantWide: true)] + public sealed record DeliveryEvent : IntegrationEventBase + { + public override string Topic => "learnstack.test.delivery"; + public override string PartitionKey => TenantId.ToString("D"); + } + public sealed class FirstHandler : IIntegrationEventHandler + { + public Task HandleAsync(DeliveryEvent fact, CancellationToken cancellationToken) => Task.CompletedTask; + } + public sealed class SecondHandler : IIntegrationEventHandler + { + public Task HandleAsync(DeliveryEvent fact, CancellationToken cancellationToken) => Task.CompletedTask; + } + public sealed class PrivilegedHandler : IIntegrationEventHandler + { + public Task HandleAsync(DeliveryEvent fact, CancellationToken cancellationToken) => Task.CompletedTask; + } + + private sealed record Row(int Attempts, string? Token, DateTimeOffset? LockedUntil, DateTimeOffset Available, + DateTimeOffset? Processed, string? TerminalReason, DateTimeOffset? DeadLettered, string? Error, + int ReplayCount, DateTimeOffset? Replayed); + private sealed record AuditRow(Guid Tenant, Guid? Organization, Guid Actor, string Operation, + string Type, string Class, string Correlation, string Metadata); + + private sealed class TestDatabase : IAsyncDisposable + { + private readonly DisposableSchemaDatabase _database; + private readonly NpgsqlDataSource _app; + private readonly ServiceProvider _provider; + public NpgsqlDataSource Dispatch { get; } + public string OutboxConnectionString => _database.OutboxConnectionString; + + private TestDatabase(DisposableSchemaDatabase database) + { + _database = database; + _app = NpgsqlDataSource.Create(database.AppConnectionString); + Dispatch = OutboxDispatcherDataSource.Build(database.OutboxConnectionString); + _provider = SeedComposition.Build(_app, null, NullLoggerFactory.Instance); + } + + public static async Task CreateAsync(PostgresFixture postgres) + { + var test = new TestDatabase(await DisposableSchemaDatabase.CreateAsync(postgres)); + try + { + await using var connection = await test._app.OpenConnectionAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await SchemaQueries.SetTenantAsync(connection, transaction, SchemaFixture.TenantA); + await SchemaQueries.ExecuteAsync(connection, transaction, """ + INSERT INTO tenants (id, slug, display_name, status, created_at, created_by, row_version) + VALUES (@tenant, 'delivery-store', 'Delivery store', 'Trial', now(), @actor, 0); + INSERT INTO organizations (id, tenant_id, slug, display_name, status, created_at, created_by, row_version) + VALUES (@organization, @tenant, 'hq', 'HQ', 'Active', now(), @actor, 0); + """, ("tenant", SchemaFixture.TenantA), ("organization", SchemaFixture.OrgA1), ("actor", UserId.SystemActor.Value)); + await transaction.CommitAsync(); + return test; + } + catch { await test.DisposeAsync(); throw; } + } + + public DeliveryDispatchStore Store(NpgsqlDataSource? source = null, Func? decorate = null) => + new(new Lazy(() => source ?? Dispatch), Registry(), new DeliveryDispatchOptions(), + new SystemGuidFactory(), new FixedClock(Occurred), decorate is null + ? _provider.GetRequiredService() + : new DecoratingScopes(_provider.GetRequiredService(), decorate)); + + public async Task InsertAsync(Guid? id = null, string subscription = First, string? type = null, + Guid? organization = null, string? partition = null, Guid? tenant = null, DateTimeOffset? occurred = null) + { + var fact = new DeliveryEvent { EventId = id ?? Guid.CreateVersion7(), TenantId = tenant ?? SchemaFixture.TenantA, OccurredAt = occurred ?? Occurred }; + await using var connection = await _app.OpenConnectionAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await SchemaQueries.SetTenantAsync(connection, transaction, fact.TenantId); + if (organization is { } org) await SchemaQueries.SetSettingAsync(connection, transaction, "app.organization_id", org.ToString("D")); + await using var command = new NpgsqlCommand(""" + INSERT INTO event_deliveries (tenant_id, subscription_id, event_id, organization_id, occurred_at, + correlation_id, type, topic, partition_key, payload) + VALUES (@tenant, @subscription, @event, @organization, @occurred, @trace, @type, @topic, @partition, @payload) + """, connection, transaction); + Bind(command, fact.EventId, subscription, fact.TenantId); + command.Parameters.AddWithValue("organization", NpgsqlDbType.Uuid, (object?)organization ?? DBNull.Value); + command.Parameters.AddWithValue("occurred", fact.OccurredAt); + command.Parameters.AddWithValue("trace", Trace); + command.Parameters.AddWithValue("type", type ?? TypeIdentity); + command.Parameters.AddWithValue("topic", fact.Topic); + command.Parameters.AddWithValue("partition", partition ?? fact.PartitionKey); + command.Parameters.AddWithValue("payload", NpgsqlDbType.Jsonb, fact.ToPayloadJson()); + await command.ExecuteNonQueryAsync(); + await transaction.CommitAsync(); + return fact.EventId; + } + + public async Task ReadAsync(Guid id, string subscription = First, Guid? tenant = null, + CancellationToken cancellationToken = default) + { + await using var connection = await Dispatch.OpenConnectionAsync(cancellationToken); + await using var command = new NpgsqlCommand(""" + SELECT attempts, locked_by, locked_until, available_after, processed_at, terminal_reason, + dead_lettered_at, last_error, replay_count, replayed_at + FROM event_deliveries WHERE tenant_id = @tenant AND subscription_id = @subscription AND event_id = @event + """, connection); + Bind(command, id, subscription, tenant); + await using var row = await command.ExecuteReaderAsync(cancellationToken); + (await row.ReadAsync(cancellationToken)).Should().BeTrue(); + return new(row.GetInt32(0), Text(row, 1), Time(row, 2), row.GetFieldValue(3), Time(row, 4), + Text(row, 5), Time(row, 6), Text(row, 7), row.GetInt32(8), Time(row, 9)); + } + + public async Task UpdateAsync(Guid id, string subscription, string assignments) + { + await using var connection = await Dispatch.OpenConnectionAsync(); + await using var command = new NpgsqlCommand($"UPDATE event_deliveries SET {assignments} WHERE tenant_id = @tenant AND subscription_id = @subscription AND event_id = @event", connection); + Bind(command, id, subscription); + (await command.ExecuteNonQueryAsync()).Should().Be(1); + } + public async Task SetOrderingAsync(Guid last) + { + await using var connection = await Dispatch.OpenConnectionAsync(); + await using var command = new NpgsqlCommand( + "UPDATE event_deliveries SET available_after = @time + CASE WHEN event_id = @last THEN interval '1 hour' ELSE interval '0 seconds' END", connection); + command.Parameters.AddWithValue("time", Occurred); + command.Parameters.AddWithValue("last", last); + await command.ExecuteNonQueryAsync(); + } + public Task ExpireAsync(Guid id) => UpdateAsync(id, First, "locked_until = clock_timestamp() - interval '1 second'"); + public Task DueAsync(Guid id) => UpdateAsync(id, First, "available_after = clock_timestamp() - interval '1 second'"); + public async Task NowAsync(CancellationToken cancellationToken = default) + { + await using var connection = await Dispatch.OpenConnectionAsync(cancellationToken); + await using var command = new NpgsqlCommand("SELECT clock_timestamp()", connection); + await using var row = await command.ExecuteReaderAsync(cancellationToken); + await row.ReadAsync(cancellationToken); + return row.GetFieldValue(0); + } + public static async Task RequestTerminalAsync(DeliveryDispatchStore store) + { + var claim = (await store.TryClaimAsync(NoBlocked))!; + (await store.RequestTerminalAsync(claim, DeliveryTerminalReason.InvalidEnvelope)).Should().BeTrue(); + return (await store.TryClaimTerminalAsync(NoBlocked))!; + } + public async Task CompleteAsync(Guid id, string subscription, bool inbox) + { + await using var connection = await _app.OpenConnectionAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await SchemaQueries.SetTenantAsync(connection, transaction, SchemaFixture.TenantA); + await SchemaQueries.ExecuteAsync(connection, transaction, """ + UPDATE event_deliveries SET processed_at = clock_timestamp() + WHERE tenant_id = @tenant AND subscription_id = @subscription AND event_id = @event + """, ("tenant", SchemaFixture.TenantA), ("subscription", subscription), ("event", id)); + if (inbox) await SchemaQueries.ExecuteAsync(connection, transaction, + "INSERT INTO inbox_messages (tenant_id, subscription_id, event_id) VALUES (@tenant, @subscription, @event)", + ("tenant", SchemaFixture.TenantA), ("subscription", subscription), ("event", id)); + await transaction.CommitAsync(); + } + public async Task AuditGrantAsync(bool grant) + { + await using var connection = (NpgsqlConnection)await PostgresFixture.OpenAsync(_database.MigrationConnectionString); + await using var command = new NpgsqlCommand(grant ? "GRANT INSERT ON audit_log TO learnstack_outbox_admin" + : "REVOKE INSERT ON audit_log FROM learnstack_outbox_admin", connection); + await command.ExecuteNonQueryAsync(); + } + public Task AuditCountAsync() => CountAsync("SELECT count(*) FROM audit_log WHERE operation = 'platform.delivery.dead_letter'"); + public Task InboxCountAsync() => CountAsync("SELECT count(*) FROM inbox_messages"); + private async Task CountAsync(string sql) + { + await using var connection = await _app.OpenConnectionAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await SchemaQueries.SetTenantAsync(connection, transaction, SchemaFixture.TenantA); + await SchemaQueries.SetSettingAsync(connection, transaction, "app.scope", "tenant"); + await using var command = new NpgsqlCommand(sql, connection, transaction); + return (long)(await command.ExecuteScalarAsync())!; + } + public async Task ReadAuditAsync() + { + await using var connection = await _app.OpenConnectionAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.SetTenantAsync(connection, transaction, SchemaFixture.TenantA); + await SchemaQueries.SetSettingAsync(connection, transaction, "app.scope", "tenant"); + await using var command = new NpgsqlCommand(""" + SELECT tenant_id, organization_id, actor_user_id, operation, operation_type, operation_class, correlation_id, metadata::text + FROM audit_log WHERE operation = 'platform.delivery.dead_letter' + """, connection, transaction); + await using var row = await command.ExecuteReaderAsync(); + (await row.ReadAsync()).Should().BeTrue(); + return new(row.GetGuid(0), row.IsDBNull(1) ? null : row.GetGuid(1), row.GetGuid(2), row.GetString(3), + row.GetString(4), row.GetString(5), row.GetString(6), row.GetString(7)); + } + public async Task WaitForLockWaitAsync() + { + using var deadline = new CancellationTokenSource(TimeSpan.FromSeconds(3)); + await using var connection = await Dispatch.OpenConnectionAsync(); + while (true) + { + await using var command = new NpgsqlCommand("SELECT EXISTS (SELECT 1 FROM pg_stat_activity WHERE datname = current_database() AND wait_event_type = 'Lock')", connection); + if (await command.ExecuteScalarAsync(deadline.Token) is true) return; + await Task.Delay(10, deadline.Token); + } + } + public async ValueTask DisposeAsync() + { + await _provider.DisposeAsync(); + await Dispatch.DisposeAsync(); + await _app.DisposeAsync(); + await _database.DisposeAsync(); + } + private static void Bind(NpgsqlCommand command, Guid id, string subscription, Guid? tenant = null) + { + command.Parameters.AddWithValue("tenant", tenant ?? SchemaFixture.TenantA); + command.Parameters.AddWithValue("subscription", subscription); + command.Parameters.AddWithValue("event", id); + } + private static string? Text(NpgsqlDataReader row, int index) => row.IsDBNull(index) ? null : row.GetString(index); + private static DateTimeOffset? Time(NpgsqlDataReader row, int index) => row.IsDBNull(index) ? null : row.GetFieldValue(index); + } + + private sealed class DecoratingScopes(IServiceScopeFactory inner, Func decorate) : IServiceScopeFactory + { + public IServiceScope CreateScope() => new DecoratingScope(inner.CreateScope(), decorate); + } + private sealed class DecoratingScope(IServiceScope inner, Func decorate) : IServiceScope, IServiceProvider, IAsyncDisposable + { + public IServiceProvider ServiceProvider => this; + public object? GetService(Type type) => type == typeof(IAuditStore) ? decorate(inner.ServiceProvider.GetRequiredService()) : inner.ServiceProvider.GetService(type); + public void Dispose() => inner.Dispose(); + public ValueTask DisposeAsync() + { + if (inner is IAsyncDisposable asyncScope) return asyncScope.DisposeAsync(); + inner.Dispose(); + return ValueTask.CompletedTask; + } + } + private sealed class AuditDecorator(IAuditStore inner, + Func terminal) : IAuditStore + { + public Task WritePendingAsync(IUnitOfWork unit, CancellationToken token = default) => inner.WritePendingAsync(unit, token); + public Task WriteStandaloneAsync(AuditEntryDraft entry, CancellationToken token = default) => inner.WriteStandaloneAsync(entry, token); + public Task WriteBestEffortAsync(AuditEntryDraft entry, CancellationToken token = default) => inner.WriteBestEffortAsync(entry, token); + public Task WritePlatformScopeAsync(AuditEntryDraft entry, DbConnection connection, DbTransaction transaction, CancellationToken token = default) => inner.WritePlatformScopeAsync(entry, connection, transaction, token); + public Task WriteEventTerminalAsync(AuditEntryDraft entry, EventTerminalTransition witness, DbConnection connection, DbTransaction transaction, CancellationToken token = default) => terminal(entry, witness, connection, transaction, token); + } +} diff --git a/backend/tests/LearnStack.Tests.Integration/Database/DeliveryExecutionTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/DeliveryExecutionTests.cs new file mode 100644 index 00000000..9770f1b3 --- /dev/null +++ b/backend/tests/LearnStack.Tests.Integration/Database/DeliveryExecutionTests.cs @@ -0,0 +1,1143 @@ +using System.Collections.Concurrent; +using System.Diagnostics; +using System.Reflection; +using LearnStack.Application.Audit; +using LearnStack.Infrastructure.Audit; +using FluentAssertions; +using LearnStack.Application.Messaging; +using LearnStack.Application.Persistence; +using LearnStack.Infrastructure.Messaging; +using LearnStack.Infrastructure.Persistence; +using LearnStack.Modules.Customization.Domain; +using LearnStack.Modules.Customization.Infrastructure.Persistence; +using LearnStack.Modules.Tenancy.Domain; +using LearnStack.Modules.Tenancy.Infrastructure.Persistence; +using LearnStack.SharedKernel.Audit; +using LearnStack.SharedKernel.Domain; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Localization; +using LearnStack.SharedKernel.Messaging; +using LearnStack.SharedKernel.Persistence; +using LearnStack.SharedKernel.Results; +using LearnStack.SharedKernel.Tenancy; +using LearnStack.SharedKernel.Time; +using MediatR; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.Mvc.Testing; +using Microsoft.AspNetCore.TestHost; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; +using Microsoft.Extensions.Logging; +using Npgsql; +using NpgsqlTypes; +using Xunit; + +namespace LearnStack.Tests.Integration.Database; + +[Trait(RequiresDocker.Key, RequiresDocker.Value)] +[Collection(SharedSchema.Name)] +public sealed class DeliveryExecutionTests(SchemaFixture schema) +{ + private const string Subscription = "proof.execute.v1"; + private const string PendingSubscription = "proof.future.v1"; + private const string SensitiveCanary = "private_delivery_canary_84e315"; + private static readonly Guid PayloadCanary = new("b90387b1-c760-4c58-a276-8a7fa9a2f98a"); + private static readonly Guid CausalActor = new("019bf000-7d6c-7000-8000-000000000057"); + private const string Operation = "customization.content_type.delivery_probe"; + private const string Trace = "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"; + + /// + /// ADR-0046 + /// restores resolved tenant/organization context and causal identity before handler + /// construction, then commits business, inbox, outbox, audit and ledger together. + /// + [Fact] + public async Task Integration_Event_Handler_Restores_Tenant_Context() + { + await using var fixture = await Fixture.CreateAsync(schema); + CausalActor.Should().NotBe(UserId.SystemActor.Value); + var fact = Fact(); + var claim = await fixture.AcceptAndClaimAsync(fact); + using var cancellation = new CancellationTokenSource(); + fixture.Observation.ExpectedToken = cancellation.Token; + fixture.Observation.ObserveBareUnit = true; + var previous = new ProbeContext(SchemaFixture.TenantB, SchemaFixture.OrgA2); + fixture.Accessor.Current = previous; + var ambientActivity = Activity.Current; + Activity.Current = null; + try + { + var gate = new DeliveryCommitGate(); + var result = await fixture.Executor.ExecuteAsync(claim, gate, cancellation.Token); + + result.Should().Be(new DeliveryExecutionResult(DeliveryExecutionOutcome.Completed, true, false)); + gate.CommitAcknowledged.Should().BeTrue(); + fixture.Observation.Constructions.Should().Be(1); + fixture.Observation.Executions.Should().Be(1); + fixture.Observation.ObservedSavedWork.Should().BeTrue(); + fixture.Observation.InboxObservedAtConstruction.Should().BeTrue(); + fixture.Observation.BareUnitObserved.Should().BeTrue(); + fixture.Accessor.Current.Should().BeSameAs(previous); + Activity.Current.Should().BeNull(); + var state = await fixture.StateAsync(fact); + state.Business.Should().Be(1); + state.Inbox.Should().Be(1); + state.Outbox.Should().Be(1); + state.SuccessAudit.Should().Be(1); + state.Processed.Should().BeTrue(); + state.OutboxCausation.Should().Be(fact.EventId); + state.OutboxActor.Should().Be(CausalActor); + fixture.Observation.OutgoingAnnouncement.Should().NotBeNull(); + var announcement = fixture.Observation.OutgoingAnnouncement!; + announcement.TenantId.Value.Should().Be(fact.TenantId); + announcement.OrganizationId!.Value.Value.Should().Be(SchemaFixture.OrgA1); + state.OutboxTenant.Should().Be(announcement.TenantId.Value); + state.OutboxOrganization.Should().Be(announcement.OrganizationId?.Value); + state.OutboxPartition.Should().NotBeNullOrEmpty().And.Be(fact.PartitionKey); + state.OutboxCorrelation.Should().Be(fixture.Observation.OutgoingActivity); + ActivityContext.TryParse(state.OutboxCorrelation, null, out var persisted).Should().BeTrue(); + ActivityContext.TryParse(fixture.Observation.OutgoingActivity, null, out var enqueuing).Should().BeTrue(); + persisted.TraceId.Should().Be(enqueuing.TraceId); + state.AuditActor.Should().Be(UserId.SystemActor.Value); + state.AuditCorrelation.Should().Be(Trace); + state.AuditOrganization.Should().Be(SchemaFixture.OrgA1); + state.AuditCausation.Should().Be(fact.EventId); + state.AuditCausalActor.Should().Be(CausalActor); + state.LockedBy.Should().Be(claim.Token, "application completion cannot rewrite scheduler provenance"); + } + finally + { + fixture.Accessor.Current = null; + Activity.Current = ambientActivity; + } + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Durable_acceptance_does_not_wait_for_same_partition_handler_or_acknowledged_cleanup(bool holdCleanup) + { + await using var fixture = await Fixture.CreateAsync(schema); + var first = Fact(); + var claim = await fixture.AcceptAndClaimAsync(first); + var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + if (holdCleanup) fixture.Observation.CleanupMode = "hang"; + else fixture.Observation.AfterSave = async () => + { + entered.TrySetResult(); + await release.Task; + }; + var gate = new DeliveryCommitGate(); + var execution = fixture.Executor.ExecuteAsync(claim, gate); + Task? accepting = null; + try + { + await (holdCleanup ? fixture.Observation.CleanupEntered.Task : entered.Task) + .WaitAsync(TimeSpan.FromSeconds(10)); + gate.CommitAcknowledged.Should().Be(holdCleanup); + var next = first with { EventId = Guid.CreateVersion7() }; + next.PartitionKey.Should().Be(first.PartitionKey); + accepting = fixture.AcceptAndClaimAsync(next); + var accepted = await accepting.WaitAsync(TimeSpan.FromSeconds(5)); + + accepted.EventId.Should().Be(next.EventId, "the real bus durably accepted another event while execution was held"); + execution.IsCompleted.Should().BeFalse(); + fixture.Observation.Constructions.Should().Be(1, "acceptance constructs no second handler"); + gate.CommitAcknowledged.Should().Be(holdCleanup); + } + finally + { + release.TrySetResult(); + fixture.Observation.CleanupRelease.TrySetResult(); + await execution.WaitAsync(TimeSpan.FromSeconds(10)); + if (accepting is not null) await accepting.WaitAsync(TimeSpan.FromSeconds(10)); + } + (await fixture.StateAsync(first)).Processed.Should().BeTrue(); + } + + [Theory] + [InlineData("handler")] + [InlineData("constructor")] + [InlineData("null-task")] + [InlineData("foreign-cancel")] + public async Task Faults_roll_back_actual_attempted_work_and_a_fresh_delivery_scope_remains_usable(string fault) + { + await using var fixture = await Fixture.CreateAsync(schema); + var fact = Fact(); + var claim = await fixture.AcceptAndClaimAsync(fact); + fixture.Observation.Fault = fault; + var previous = new ProbeContext(SchemaFixture.TenantB, null); + fixture.Accessor.Current = previous; + var result = await fixture.Executor.ExecuteAsync(claim, new DeliveryCommitGate()); + + result.CommitAcknowledged.Should().BeFalse(); + result.Outcome.Should().Be(DeliveryExecutionOutcome.Failed); + fixture.Accessor.Current.Should().BeSameAs(previous); + fixture.Logs.Text.Should().NotContain(SensitiveCanary).And.NotContain(PayloadCanary.ToString()).And.NotContain(typeof(Handler).FullName!); + (await fixture.StateAsync(fact)).Should().Match(state => + state.Business == 0 && state.Inbox == 0 && state.Outbox == 0 + && state.SuccessAudit == 0 && !state.Processed); + if (fault is "handler" or "foreign-cancel") fixture.Observation.ObservedSavedWork.Should().BeTrue(); + var failedOrigin = await fixture.FailedAuditOriginAsync(); + failedOrigin.Actor.Should().Be(UserId.SystemActor.Value); + failedOrigin.Causation.Should().Be(fact.EventId); + failedOrigin.CausalActor.Should().Be(CausalActor); + + fixture.Observation.Fault = null; + var retry = await fixture.Executor.ExecuteAsync(claim, new DeliveryCommitGate()); + retry.Outcome.Should().Be(DeliveryExecutionOutcome.Completed); + retry.CommitAcknowledged.Should().BeTrue(); + (await fixture.StateAsync(fact)).Business.Should().Be(1); + fixture.Accessor.Current.Should().BeSameAs(previous); + fixture.Accessor.Current = null; + } + + [Fact] + public async Task A_durable_inbox_duplicate_completes_the_claim_without_constructing_or_classifying_the_handler() + { + await using var fixture = await Fixture.CreateAsync(schema); + var fact = Fact(); + var claim = await fixture.AcceptAndClaimAsync(fact); + await fixture.InsertInboxAsync(fact); + fixture.Observation.Fault = "constructor"; + var result = await fixture.Executor.ExecuteAsync(claim, new DeliveryCommitGate()); + + result.Should().Be(new DeliveryExecutionResult(DeliveryExecutionOutcome.Duplicate, true, false)); + fixture.Observation.Constructions.Should().Be(0); + var state = await fixture.StateAsync(fact); + state.Inbox.Should().Be(1); + state.Processed.Should().BeTrue(); + state.Business.Should().Be(0); + state.Outbox.Should().Be(0); + state.SuccessAudit.Should().Be(0); + } + + [Theory] + [InlineData("token")] + [InlineData("expired")] + [InlineData("gate")] + public async Task Final_ownership_loss_rolls_back_saved_business_inbox_outbox_and_success_audit(string loss) + { + await using var fixture = await Fixture.CreateAsync(schema); + var fact = Fact(); + var claim = await fixture.AcceptAndClaimAsync(fact); + var gate = new DeliveryCommitGate(); + fixture.Observation.AfterSave = async () => + { + if (loss == "gate") gate.Close(); + else await fixture.ChangeLeaseAsync(fact.EventId, loss); + }; + var result = await fixture.Executor.ExecuteAsync(claim, gate); + + fixture.Observation.ObservedSavedWork.Should().BeTrue("ownership must be lost after an actual saved write"); + result.CommitAcknowledged.Should().BeFalse(); + gate.CommitStarted.Should().BeFalse(); + var state = await fixture.StateAsync(fact); + state.Business.Should().Be(0); + state.Inbox.Should().Be(0); + state.Outbox.Should().Be(0); + state.SuccessAudit.Should().Be(0); + state.Processed.Should().BeFalse(); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Closing_the_gate_while_a_callback_ignores_cancellation_prevents_a_late_commit(bool cancelToken) + { + await using var fixture = await Fixture.CreateAsync(schema); + var fact = Fact(); + var claim = await fixture.AcceptAndClaimAsync(fact); + var gate = new DeliveryCommitGate(); + var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + fixture.Observation.AfterSave = async () => + { + entered.SetResult(); + await release.Task; + }; + using var callbackCancellation = new CancellationTokenSource(); + fixture.Observation.ExpectedToken = callbackCancellation.Token; + var execution = fixture.Executor.ExecuteAsync(claim, gate, callbackCancellation.Token); + await entered.Task.WaitAsync(TimeSpan.FromSeconds(10)); + gate.Close(); + if (cancelToken) callbackCancellation.Cancel(); + callbackCancellation.IsCancellationRequested.Should().Be(cancelToken, + "the live-token control proves gate closure independently carries the timeout decision"); + execution.IsCompleted.Should().BeFalse("the executor still owns the live callback and transaction"); + (await fixture.StateAsync(fact)).Business.Should().Be(0, "the observed saved write is still uncommitted"); + release.SetResult(); + var result = await execution.WaitAsync(TimeSpan.FromSeconds(10)); + + result.Outcome.Should().Be(DeliveryExecutionOutcome.Canceled); + result.CommitAcknowledged.Should().BeFalse(); + (await fixture.StateAsync(fact)).Should().Match(state => + state.Business == 0 && state.Inbox == 0 && state.Outbox == 0 + && state.SuccessAudit == 0 && !state.Processed); + } + + [Fact] + public async Task Retained_active_subscription_executes_despite_a_pending_sibling_and_pending_target_is_refused() + { + await using var fixture = await Fixture.CreateAsync(schema, pendingSibling: true); + var fact = Fact(); + var claim = await fixture.InsertAndClaimAsync(fact); + fixture.Registry.Resolve(claim.Type).Disposition.Should().Be(OutboxEventDisposition.Pending); + var result = await fixture.Executor.ExecuteAsync(claim, new DeliveryCommitGate()); + result.Outcome.Should().Be(DeliveryExecutionOutcome.Completed); + (await fixture.StateAsync(fact)).Business.Should().Be(1); + + var pending = Fact(); + var pendingClaim = await fixture.InsertAndClaimAsync(pending, PendingSubscription); + var refused = await fixture.Executor.ExecuteAsync(pendingClaim, new DeliveryCommitGate()); + refused.Outcome.Should().Be(DeliveryExecutionOutcome.InvalidSubscription); + fixture.Observation.Constructions.Should().Be(1); + (await fixture.StateAsync(pending, PendingSubscription)).Inbox.Should().Be(0); + } + + [Theory] + [InlineData("payload")] + [InlineData("type")] + [InlineData("topic")] + [InlineData("partition")] + [InlineData("claim-time")] + [InlineData("unclassified")] + [InlineData("privileged")] + public async Task Invalid_metadata_or_unclassified_handler_refuses_before_business_construction(string invalid) + { + await using var fixture = await Fixture.CreateAsync(schema, + classified: invalid != "unclassified", privileged: invalid == "privileged"); + var fact = Fact(); + var claim = await fixture.InsertAndClaimAsync(fact, corruption: invalid); + if (invalid == "claim-time") claim = claim with { OccurredAt = claim.OccurredAt.AddSeconds(1) }; + var result = await fixture.Executor.ExecuteAsync(claim, new DeliveryCommitGate()); + + result.Outcome.Should().Be(invalid switch + { + "privileged" => DeliveryExecutionOutcome.InvalidSubscription, + "unclassified" => DeliveryExecutionOutcome.Refused, + _ => DeliveryExecutionOutcome.InvalidEnvelope, + }); + result.CommitAcknowledged.Should().BeFalse(); + fixture.Observation.Constructions.Should().Be(0); + fixture.Logs.Text.Should().NotContain(SensitiveCanary).And.NotContain(PayloadCanary.ToString()).And.NotContain(typeof(Handler).FullName!); + if (invalid == "unclassified") fixture.Logs.Entries.Should().Contain(entry => entry.Level == LogLevel.Error); + var state = await fixture.StateAsync(fact); + state.Inbox.Should().Be(0); + state.Processed.Should().BeFalse(); + state.Business.Should().Be(0); + } + + [Fact] + public async Task A_lease_expiring_behind_an_unchanged_row_lock_is_rechecked_after_lock_acquisition() + { + await using var fixture = await Fixture.CreateAsync(schema); + var fact = Fact(); + var claim = await fixture.AcceptAndClaimAsync(fact); + var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + fixture.Observation.AfterSave = async () => + { + await using var command = fixture.Observation.Unit!.Connection.CreateCommand(); + command.Transaction = fixture.Observation.Unit.Transaction; + command.CommandText = "SELECT pg_backend_pid()"; + entered.SetResult((int)(await command.ExecuteScalarAsync())!); + await release.Task; + }; + var execution = fixture.Executor.ExecuteAsync(claim, new DeliveryCommitGate()); + var pid = await entered.Task.WaitAsync(TimeSpan.FromSeconds(10)); + await fixture.ChangeLeaseAsync(fact.EventId, "short"); + await using var blocker = (NpgsqlConnection)await PostgresFixture.OpenAsync(fixture.Database.OutboxConnectionString); + await using var blockerTransaction = await blocker.BeginTransactionAsync(); + try + { + await using (var rowLock = new NpgsqlCommand(""" + SELECT event_id FROM event_deliveries WHERE event_id = @event FOR UPDATE + """, blocker, blockerTransaction)) + { + rowLock.Parameters.AddWithValue("event", fact.EventId); + (await rowLock.ExecuteScalarAsync()).Should().Be(fact.EventId); + } + release.SetResult(); + await WaitUntilBlockedAsync(fixture.Database.AppConnectionString, pid); + execution.IsCompleted.Should().BeFalse(); + // This transaction changes no row. The final fence has already reached + // the lock, and database time is the only change before its release. + await using (var wait = new NpgsqlCommand(""" + SELECT pg_sleep(GREATEST(0, EXTRACT(EPOCH FROM (locked_until - clock_timestamp()))) + 0.05) + FROM event_deliveries WHERE event_id = @event + """, blocker, blockerTransaction)) + { + wait.Parameters.AddWithValue("event", fact.EventId); + await wait.ExecuteNonQueryAsync(); + } + await blockerTransaction.RollbackAsync(); + } + finally { release.TrySetResult(); } + var result = await execution.WaitAsync(TimeSpan.FromSeconds(10)); + result.CommitAcknowledged.Should().BeFalse(); + fixture.Observation.ObservedSavedWork.Should().BeTrue(); + var state = await fixture.StateAsync(fact); + state.Business.Should().Be(0); + state.Inbox.Should().Be(0); + state.Outbox.Should().Be(0); + state.Processed.Should().BeFalse(); + } + + [Theory] + [InlineData(1)] + [InlineData(6)] + public async Task Lost_COMMIT_acknowledgement_retains_atomic_completion_and_never_repeats_business(int attempts) + { + await using var fixture = await Fixture.CreateAsync(schema); + var fact = Fact(); + var claim = await fixture.AcceptAndClaimAsync(fact, attempts); + await using var proxy = new CommitResponseDroppingProxy(fixture.Database.AppConnectionString); + await using var uncertain = fixture.ForConnection(proxy.ConnectionString); + var gate = new DeliveryCommitGate(); + var result = await uncertain.Executor.ExecuteAsync(claim, gate); + + result.Outcome.Should().Be(DeliveryExecutionOutcome.Indeterminate); + result.CommitAcknowledged.Should().BeFalse(); + gate.CommitStarted.Should().BeTrue(); + proxy.DroppedCommitResponses.Should().Be(1); + uncertain.Observation.ObservedSavedWork.Should().BeTrue(); + var state = await fixture.StateAsync(fact); + state.Business.Should().Be(1); + state.Inbox.Should().Be(1); + state.Outbox.Should().Be(1); + state.SuccessAudit.Should().Be(1); + state.Processed.Should().BeTrue(); + + var retry = await fixture.Executor.ExecuteAsync(claim, new DeliveryCommitGate()); + retry.Outcome.Should().Be(DeliveryExecutionOutcome.LostClaim); + fixture.Observation.Constructions.Should().Be(0, "durable completion excludes every subsequent execution"); + (await fixture.StateAsync(fact)).Should().BeEquivalentTo(state); + uncertain.Logs.Text.Should().NotContain(SensitiveCanary).And.NotContain(PayloadCanary.ToString()); + } + + [Fact] + public async Task Post_acknowledgement_scope_disposal_fault_retains_completion_and_blocks_gate_reuse() + { + await using var fixture = await Fixture.CreateAsync(schema); + var fact = Fact(); + var claim = await fixture.AcceptAndClaimAsync(fact); + fixture.Observation.CleanupMode = "fault"; + var gate = new DeliveryCommitGate(); + var result = await fixture.Executor.ExecuteAsync(claim, gate); + + result.Should().Be(new DeliveryExecutionResult(DeliveryExecutionOutcome.Completed, true, true)); + gate.CommitAcknowledged.Should().BeTrue(); + var state = await fixture.StateAsync(fact); + state.Business.Should().Be(1); + state.Inbox.Should().Be(1); + state.Processed.Should().BeTrue(); + await FluentActions.Awaiting(() => fixture.Executor.ExecuteAsync(claim, gate)) + .Should().ThrowAsync().WithMessage("*gate*one execution*"); + fixture.Observation.CleanupMode = null; + (await fixture.Executor.ExecuteAsync(claim, new DeliveryCommitGate())).Outcome + .Should().Be(DeliveryExecutionOutcome.LostClaim); + fixture.Observation.Constructions.Should().Be(1); + (await fixture.StateAsync(fact)).Should().BeEquivalentTo(state); + fixture.Logs.Text.Should().NotContain(SensitiveCanary).And.NotContain(PayloadCanary.ToString()); + } + + [Fact] + public async Task Post_acknowledgement_hanging_scope_cleanup_retains_receipt_and_live_resource_until_observed_completion() + { + await using var fixture = await Fixture.CreateAsync(schema); + var fact = Fact(); + var claim = await fixture.AcceptAndClaimAsync(fact); + fixture.Observation.CleanupMode = "hang"; + var gate = new DeliveryCommitGate(); + var execution = fixture.Executor.ExecuteAsync(claim, gate); + await fixture.Observation.CleanupEntered.Task.WaitAsync(TimeSpan.FromSeconds(10)); + try + { + execution.IsCompleted.Should().BeFalse(); + fixture.Observation.CleanupAlive.Should().BeTrue(); + gate.CommitAcknowledged.Should().BeTrue("the receipt is published before asynchronous resource cleanup"); + gate.Close(); + var state = await fixture.StateAsync(fact); + state.Business.Should().Be(1); + state.Inbox.Should().Be(1); + state.Outbox.Should().Be(1); + state.SuccessAudit.Should().Be(1); + state.Processed.Should().BeTrue(); + } + finally { fixture.Observation.CleanupRelease.TrySetResult(); } + (await execution.WaitAsync(TimeSpan.FromSeconds(10))).Should() + .Be(new DeliveryExecutionResult(DeliveryExecutionOutcome.Completed, true, false)); + fixture.Observation.CleanupAlive.Should().BeFalse(); + (await fixture.Executor.ExecuteAsync(claim, new DeliveryCommitGate())).Outcome + .Should().Be(DeliveryExecutionOutcome.LostClaim); + fixture.Observation.Constructions.Should().Be(1); + } + + [Fact] + public async Task Retained_envelope_read_failure_rolls_back_before_construction_and_fresh_scope_can_retry() + { + await using var fixture = await Fixture.CreateAsync(schema); + var fact = Fact(); + var claim = await fixture.AcceptAndClaimAsync(fact); + await fixture.AlterSchemaAsync("REVOKE SELECT ON event_deliveries FROM learnstack_app"); + DeliveryExecutionResult result; + try { result = await fixture.Executor.ExecuteAsync(claim, new DeliveryCommitGate()); } + finally { await fixture.AlterSchemaAsync("GRANT SELECT ON event_deliveries TO learnstack_app"); } + + result.Outcome.Should().Be(DeliveryExecutionOutcome.Failed); + result.CommitAcknowledged.Should().BeFalse(); + fixture.Observation.Constructions.Should().Be(0); + (await fixture.StateAsync(fact)).Should().Match(state => + state.Business == 0 && state.Inbox == 0 && state.Outbox == 0 && !state.Processed); + (await fixture.Executor.ExecuteAsync(claim, new DeliveryCommitGate())).Outcome + .Should().Be(DeliveryExecutionOutcome.Completed); + (await fixture.StateAsync(fact)).Business.Should().Be(1); + } + + [Fact] + public async Task A_real_MUST_audit_write_fault_rolls_back_every_effect_and_logs_only_bounded_diagnostics() + { + await using var fixture = await Fixture.CreateAsync(schema); + var fact = Fact(); + var claim = await fixture.AcceptAndClaimAsync(fact); + await fixture.AlterSchemaAsync("ALTER TABLE audit_log ADD CONSTRAINT private_delivery_canary_84e315 CHECK (false)"); + var result = await fixture.Executor.ExecuteAsync(claim, new DeliveryCommitGate()); + + fixture.Observation.ObservedSavedWork.Should().BeTrue(); + result.Outcome.Should().Be(DeliveryExecutionOutcome.Failed); + result.CommitAcknowledged.Should().BeFalse(); + var state = await fixture.StateAsync(fact); + state.Business.Should().Be(0); + state.Inbox.Should().Be(0); + state.Outbox.Should().Be(0); + state.SuccessAudit.Should().Be(0); + state.Processed.Should().BeFalse(); + fixture.Logs.Entries.Should().Contain(entry => entry.Level == LogLevel.Critical, + "the actual failed standalone MUST reconciliation logs through this capture"); + fixture.Logs.Text.Should().NotContain(SensitiveCanary).And.NotContain(PayloadCanary.ToString()).And.NotContain(typeof(Handler).FullName!); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Actual_delivery_owner_drains_saved_domain_facts_and_rolls_back_translation_failure(bool failTranslation) + { + await using var fixture = await Fixture.CreateAsync(schema); + var fact = Fact(); + var claim = await fixture.AcceptAndClaimAsync(fact); + fixture.Observation.EmitDomainEvent = true; + fixture.Observation.FailTranslation = failTranslation; + var gate = new DeliveryCommitGate(); + var result = await fixture.Executor.ExecuteAsync(claim, gate); + + fixture.Observation.ObservedSavedWork.Should().BeTrue(); + fixture.Observation.TranslationObserved.Should().BeTrue(); + fixture.Observation.Source!.DomainEvents.Should().BeEmpty("owner completion clears saved facts on every exit"); + var state = await fixture.StateAsync(fact); + if (failTranslation) + { + result.Outcome.Should().Be(DeliveryExecutionOutcome.Failed); + gate.CommitStarted.Should().BeFalse(); + state.Business.Should().Be(0); + state.Inbox.Should().Be(0); + state.Outbox.Should().Be(0); + state.SuccessAudit.Should().Be(0); + state.Processed.Should().BeFalse(); + } + else + { + result.Outcome.Should().Be(DeliveryExecutionOutcome.Completed); + gate.CommitAcknowledged.Should().BeTrue(); + state.Business.Should().Be(1); + state.Inbox.Should().Be(1); + state.Outbox.Should().Be(2); + state.SuccessAudit.Should().Be(1); + state.Processed.Should().BeTrue(); + (await fixture.OutboxOriginsAsync(fact)).Should().BeEquivalentTo(new[] + { + (fact.EventId, CausalActor), + (fixture.Observation.DomainEventId, CausalActor), + }); + } + fixture.Logs.Text.Should().NotContain(SensitiveCanary).And.NotContain(PayloadCanary.ToString()); + } + + [Theory] + [InlineData("malformed")] + [InlineData("compact")] + [InlineData("uppercase")] + [InlineData("zero")] + [InlineData("sentinel")] + public async Task Invalid_claim_token_or_platform_tenant_refuses_before_creating_any_application_scope(string invalid) + { + await using var fixture = await Fixture.CreateAsync(schema); + var valid = await fixture.AcceptAndClaimAsync(Fact()); + var claim = invalid switch + { + "malformed" => valid with { Token = SensitiveCanary }, + "compact" => valid with { Token = "aaaaaaaabbbb4ccc8dddeeeeeeeeeeee" }, + "uppercase" => valid with { Token = "AAAAAAAA-BBBB-4CCC-8DDD-EEEEEEEEEEEE" }, + "zero" => valid with { Token = Guid.Empty.ToString("D") }, + _ => valid with { TenantId = TenantId.PlatformSentinel.Value }, + }; + var factory = new RefusingScopeFactory(); + var handlers = IntegrationEventHandlerRegistry.FromServiceDescriptors( + [ServiceDescriptor.Scoped, Handler>()]); + var executor = new DeliveryExecutor(factory, fixture.Accessor, new PartitionSerializer(), fixture.Registry, handlers); + var previous = new ProbeContext(SchemaFixture.TenantB, null); + fixture.Accessor.Current = previous; + try + { + (await executor.ExecuteAsync(claim, new DeliveryCommitGate())).Should() + .Be(new DeliveryExecutionResult(DeliveryExecutionOutcome.InvalidEnvelope, false, false)); + factory.CreatedScopes.Should().Be(0); + fixture.Accessor.Current.Should().BeSameAs(previous); + + // A valid claim reaches the deliberately failing factory, proving the + // negative result above was early validation, not an inert probe. + (await executor.ExecuteAsync(valid, new DeliveryCommitGate())).Outcome + .Should().Be(DeliveryExecutionOutcome.Failed); + factory.CreatedScopes.Should().Be(1); + fixture.Accessor.Current.Should().BeSameAs(previous); + } + finally { fixture.Accessor.Current = null; } + } + + private sealed class RefusingScopeFactory : IServiceScopeFactory + { + public int CreatedScopes { get; private set; } + public IServiceScope CreateScope() + { + CreatedScopes++; + throw new InvalidOperationException(SensitiveCanary); + } + } + + private static async Task WaitUntilBlockedAsync(string connectionString, int pid) + { + using var deadline = new CancellationTokenSource(TimeSpan.FromSeconds(10)); + await using var connection = (NpgsqlConnection)await PostgresFixture.OpenAsync(connectionString); + await SchemaQueries.AssertApplicationRoleAsync(connection); + while (true) + { + deadline.Token.ThrowIfCancellationRequested(); + await using var read = new NpgsqlCommand("SELECT cardinality(pg_blocking_pids(@pid)) > 0", connection); + read.Parameters.AddWithValue("pid", pid); + if (await read.ExecuteScalarAsync(deadline.Token) is true) return; + await Task.Yield(); + } + } + + private static ProofEvent Fact() => new() + { + EventId = Guid.CreateVersion7(), + TenantId = SchemaFixture.TenantA, + OccurredAt = new DateTimeOffset(2026, 10, 13, 9, 0, 0, TimeSpan.Zero), + SubjectId = Guid.CreateVersion7(), + }; + + [IntegrationEventPartition(nameof(SubjectId))] + public sealed record ProofEvent : IntegrationEventBase + { + public required Guid SubjectId { get; init; } + public Guid PrivateReferenceId { get; init; } = PayloadCanary; + public override string Topic => "learnstack.proof.delivery"; + public override string PartitionKey => SubjectId.ToString("D"); + } + + public sealed class Handler + : IIntegrationEventHandler + { + private readonly IServiceProvider _services; + private readonly Observation _observation; + public Handler(IServiceProvider services, Observation observation) + { + _services = services; + _observation = observation; + observation.Constructions++; + _ = services.GetRequiredService(); + AssertContext(services); + var unit = services.GetRequiredService(); + unit.HasActiveTransaction.Should().BeTrue(); + using (var inbox = unit.Connection.CreateCommand()) + { + inbox.Transaction = unit.Transaction; + inbox.CommandText = "SELECT count(*) FROM inbox_messages WHERE subscription_id = @subscription"; + var subscription = inbox.CreateParameter(); + subscription.ParameterName = "subscription"; + subscription.Value = Subscription; + inbox.Parameters.Add(subscription); + inbox.ExecuteScalar().Should().Be(1L, "the transport claimed inbox before entering the business constructor"); + observation.InboxObservedAtConstruction = true; + } + services.GetRequiredService().HasOpenFrame.Should().BeTrue(); + services.GetRequiredService().RequireAnnouncement().TenantId.Value.Should().Be(SchemaFixture.TenantA); + if (observation.Fault == "constructor") throw new InvalidOperationException(SensitiveCanary); + } + public Task HandleAsync(ProofEvent @event, CancellationToken cancellationToken = default) + { + _observation.Executions++; + cancellationToken.Should().Be(_observation.ExpectedToken); + return _observation.Fault == "null-task" ? null! : WorkAsync(@event, cancellationToken); + } + private async Task WorkAsync(ProofEvent fact, CancellationToken cancellationToken) + { + await Task.Yield(); + AssertContext(_services); + var unit = _services.GetRequiredService(); + _observation.Unit = unit; + await SchemaQueries.AssertApplicationRoleAsync(unit.Connection); + var db = _services.GetRequiredService(); + var id = TenantContentTypeId.From(fact.SubjectId); + var source = TenantContentType.Create(id, TenantId.From(fact.TenantId), $"delivery-{fact.SubjectId:N}"[..24], 1, + LocalizedText.From(("en", "Delivery execution proof")), """{"type":"object"}""", "default-card", + _services.GetRequiredService(), UserId.SystemActor); + _observation.Source = source; + if (_observation.EmitDomainEvent) + { + typeof(Entity).GetMethod("RaiseDomainEvent", BindingFlags.Instance | BindingFlags.NonPublic)! + .Invoke(source, [new DeliveryDomainEvent + { + EventId = _observation.DomainEventId, OccurredAt = fact.OccurredAt, SubjectId = fact.SubjectId, + }]); + } + db.Add(source); + await db.SaveChangesAsync(cancellationToken); + _observation.OutgoingActivity = Activity.Current?.Id; + _observation.OutgoingAnnouncement = unit.GetTenantContextAnnouncement(unit.Transaction); + await _services.GetRequiredService().EnqueueAsync(fact with { EventId = Guid.CreateVersion7() }, cancellationToken); + await using var read = unit.Connection.CreateCommand(); + read.Transaction = unit.Transaction; + read.CommandText = "SELECT count(*) FROM tenant_content_types WHERE id = @id"; + var parameter = read.CreateParameter(); + parameter.ParameterName = "id"; + parameter.Value = fact.SubjectId; + read.Parameters.Add(parameter); + (await read.ExecuteScalarAsync(cancellationToken)).Should().Be(1L); + _observation.ObservedSavedWork = true; + if (_observation.AfterSave is { } callback) await callback(); + if (_observation.Fault == "handler") throw new InvalidOperationException(SensitiveCanary); + if (_observation.Fault == "foreign-cancel") + { + using var foreign = new CancellationTokenSource(); + foreign.Cancel(); + throw new OperationCanceledException(SensitiveCanary, foreign.Token); + } + } + private static void AssertContext(IServiceProvider services) + { + var context = services.GetRequiredService(); + context.IsResolved.Should().BeTrue("delivery restores resolved context before handler construction and execution"); + context.TenantId.Value.Should().Be(SchemaFixture.TenantA); + context.OrganizationId!.Value.Value.Should().Be(SchemaFixture.OrgA1); + context.UserId.Should().Be(UserId.SystemActor); + context.CausalActorUserId.Should().Be(UserId.From(CausalActor)); + context.CorrelationId.Should().Be(Trace); + context.ModuleName.Should().Be("unknown"); + Activity.Current.Should().NotBeNull(); + Activity.Current!.IdFormat.Should().Be(ActivityIdFormat.W3C); + Activity.Current.TraceId.ToHexString().Should().Be("4bf92f3577b34da6a3ce929d0e0e4736"); + } + } + + public sealed class Observation + { + public int Constructions { get; set; } + public bool InboxObservedAtConstruction { get; set; } + public bool ObserveBareUnit { get; set; } + public bool BareUnitObserved { get; set; } + public bool EmitDomainEvent { get; set; } + public bool FailTranslation { get; set; } + public bool TranslationObserved { get; set; } + public Guid DomainEventId { get; } = Guid.CreateVersion7(); + public TenantContentType? Source { get; set; } + public int Executions { get; set; } + public bool ObservedSavedWork { get; set; } + public string? OutgoingActivity { get; set; } + public TenantContextAnnouncement? OutgoingAnnouncement { get; set; } + public string? Fault { get; set; } + public Func? AfterSave { get; set; } + public CancellationToken ExpectedToken { get; set; } + public IUnitOfWork? Unit { get; set; } + public string? CleanupMode { get; set; } + public bool CleanupAlive { get; set; } + public TaskCompletionSource CleanupEntered { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + public TaskCompletionSource CleanupRelease { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + } + + public sealed record DeliveryDomainEvent : DomainEvent + { + public required Guid SubjectId { get; init; } + } + + public sealed class DomainTranslator(IOutbox outbox, IUnitOfWork unit, Observation observation) + : INotificationHandler + { + public async Task Handle(DeliveryDomainEvent notification, CancellationToken cancellationToken) + { + observation.Source!.DomainEvents.Should().ContainSingle(); + await outbox.EnqueueAsync(new ProofEvent + { + EventId = Guid.CreateVersion7(), + TenantId = SchemaFixture.TenantA, + OccurredAt = notification.OccurredAt, + SubjectId = notification.SubjectId, + }, cancellationToken); + await using var read = unit.Connection.CreateCommand(); + read.Transaction = unit.Transaction; + read.CommandText = "SELECT count(*) FROM outbox_messages WHERE payload ->> 'SubjectId' = @subject"; + var parameter = read.CreateParameter(); + parameter.ParameterName = "subject"; + parameter.Value = notification.SubjectId.ToString(); + read.Parameters.Add(parameter); + (await read.ExecuteScalarAsync(cancellationToken)).Should().Be(2L, + "both direct and translated production existed on the actual owner transaction"); + observation.TranslationObserved = true; + if (observation.FailTranslation) throw new InvalidOperationException(SensitiveCanary); + } + } + + public sealed class CleanupProbe : IAsyncDisposable + { + private readonly Observation _observation; + private readonly ITenantContextAccessor _accessor; + public CleanupProbe(Observation observation, ITenantContextAccessor accessor) + { + _observation = observation; + _accessor = accessor; + observation.CleanupAlive = true; + } + public async ValueTask DisposeAsync() + { + try + { + _accessor.Current.Should().BeOfType(); + _accessor.Current!.TenantId.Value.Should().Be(SchemaFixture.TenantA); + _accessor.Current.OrganizationId!.Value.Value.Should().Be(SchemaFixture.OrgA1); + _accessor.Current.UserId.Should().Be(UserId.SystemActor); + Activity.Current.Should().NotBeNull("delivery metadata outlives owned scope disposal"); + if (_observation.CleanupMode == "hang") + { + _observation.CleanupEntered.TrySetResult(); + await _observation.CleanupRelease.Task; + } + if (_observation.CleanupMode == "fault") throw new IOException(SensitiveCanary); + } + finally { _observation.CleanupAlive = false; } + } + } + + private sealed record LogEntry(LogLevel Level, string Text); + + private sealed class DeliveryLogs : ILoggerProvider + { + public ConcurrentQueue Entries { get; } = new(); + public string Text => string.Join("\n", Entries.Select(entry => entry.Text)); + public ILogger CreateLogger(string categoryName) => new CapturedLogger(this); + public void Dispose() { } + } + + private sealed class CapturedLogger(DeliveryLogs logs) : ILogger + { + public IDisposable? BeginScope(TState state) where TState : notnull => null; + public bool IsEnabled(LogLevel logLevel) => true; + public void Log(LogLevel logLevel, EventId eventId, TState state, + Exception? exception, Func formatter) => + logs.Entries.Enqueue(new LogEntry(logLevel, formatter(state, exception) + " " + exception)); + } + + private sealed class CatalogSource : IAuditCatalogSource + { + public string ModuleName => "customization"; + public void Describe(IAuditCatalogBuilder builder) => + builder.MustAudit(Operation, OperationType.Create, typeof(TenantContentType)); + } + + private sealed class ProbeContext(Guid tenant, Guid? organization) : ITenantContext + { + public bool IsResolved => true; + public TenantId TenantId => TenantId.From(tenant); + public OrganizationId? OrganizationId => organization is { } id ? LearnStack.SharedKernel.Identifiers.OrganizationId.From(id) : null; + public UserId? UserId => LearnStack.SharedKernel.Identifiers.UserId.From(SchemaFixture.Actor); + public string? CorrelationId => Trace; + public string? ModuleName => "customization"; + } + + private sealed record DurableState(long Business, long Inbox, long Outbox, long SuccessAudit, + bool Processed, Guid? OutboxCausation, Guid? OutboxActor, Guid? AuditActor, + string? AuditCorrelation, Guid? AuditOrganization, string? LockedBy, + Guid? AuditCausation, Guid? AuditCausalActor, string? OutboxCorrelation, + Guid? OutboxTenant, Guid? OutboxOrganization, string? OutboxPartition); + + private sealed class Fixture : IAsyncDisposable + { + private readonly WebApplicationFactory _host; + private readonly bool _ownsDatabase; + private Fixture(DisposableSchemaDatabase database, bool pendingSibling, bool classified, bool privileged, + string? connectionString = null, bool ownsDatabase = true) + { + Database = database; + _ownsDatabase = ownsDatabase; + Observation = new Observation(); + Logs = new DeliveryLogs(); + var handlers = IntegrationEventHandlerRegistry.FromServiceDescriptors( + [ServiceDescriptor.Scoped, Handler>()]); + var selected = OutboxSubscriptionManifest.Active(Subscription, typeof(Handler), privileged + ? OutboxSubscriptionExecutionScope.PrivilegedPlatform : OutboxSubscriptionExecutionScope.OrdinaryTenant); + Registry = OutboxEventRegistry.Create([ + OutboxEventRegistration.Required(typeof(ProofEvent), pendingSibling + ? [selected, OutboxSubscriptionManifest.Pending(PendingSubscription)] : [selected]), + ], [], handlers, [typeof(ProofEvent).Assembly]); + _host = new ConstructionOnlyApiFactory().WithWebHostBuilder(builder => + { + builder.UseSetting("ConnectionStrings:Default", connectionString ?? database.AppConnectionString); + builder.ConfigureTestServices(services => + { + services.RemoveAll(); + services.AddSingleton(Registry); + services.RemoveAll(); + services.AddSingleton(handlers); + services.AddSingleton(Observation); + services.RemoveAll(); + services.AddScoped(provider => + { + if (Observation.ObserveBareUnit) + { + provider.GetRequiredService().Current.Should().BeNull( + "routing SQL must resolve its bare unit without another caller's ambient identity"); + Observation.BareUnitObserved = true; + } + return ActivatorUtilities.CreateInstance(provider); + }); + services.AddScoped(); + services.AddScoped(); + services.AddScoped, DomainTranslator>(); + services.AddSingleton(Logs); + services.AddSingleton>(new CapturedLogger(Logs)); + services.AddSingleton>(new CapturedLogger(Logs)); + services.AddSingleton>(new CapturedLogger(Logs)); + services.AddSingleton>(new CapturedLogger(Logs)); + if (classified) services.AddSingleton(); + services.AddSingleton(); + }); + }); + } + public DisposableSchemaDatabase Database { get; } + public Observation Observation { get; } + public DeliveryLogs Logs { get; } + public Fixture ForConnection(string connectionString) => + new(Database, false, true, false, connectionString, ownsDatabase: false); + public OutboxEventRegistry Registry { get; } + public ITenantContextAccessor Accessor => _host.Services.GetRequiredService(); + public DeliveryExecutor Executor => _host.Services.GetRequiredService(); + public static async Task CreateAsync(SchemaFixture schema, bool pendingSibling = false, + bool classified = true, bool privileged = false) + { + var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + var fixture = new Fixture(database, pendingSibling, classified, privileged); + try { await fixture.SeedAsync(); return fixture; } + catch { await fixture.DisposeAsync(); throw; } + } + private async Task SeedAsync() + { + Accessor.Current = new ProbeContext(SchemaFixture.TenantA, null); + try + { + await using var scope = _host.Services.CreateAsyncScope(); + var services = scope.ServiceProvider; + var unit = services.GetRequiredService(); + await using var frame = await unit.BeginTransactionAsync(); + await unit.SetTenantContextAsync(services.GetRequiredService()); + await SchemaQueries.AssertApplicationRoleAsync(unit.Connection); + var db = services.GetRequiredService(); + db.Add(Tenant.Create(TenantId.From(SchemaFixture.TenantA), "delivery-proof", "Delivery proof", + services.GetRequiredService(), UserId.From(SchemaFixture.Actor))); + await db.SaveChangesAsync(); + db.Add(Organization.Create(OrganizationId.From(SchemaFixture.OrgA1), TenantId.From(SchemaFixture.TenantA), + "proof-branch", "Proof branch", services.GetRequiredService(), UserId.From(SchemaFixture.Actor))); + await db.SaveChangesAsync(); + await frame.CompleteAsync(); + } + finally { Accessor.Current = null; } + } + public async Task AcceptAndClaimAsync(ProofEvent fact, int attempts = 1) + { + await _host.Services.GetRequiredService().PublishAsync(Envelope(fact)); + return await ClaimAsync(fact, Subscription, attempts); + } + public async Task InsertAndClaimAsync(ProofEvent fact, string subscription = Subscription, + string? corruption = null) + { + var envelope = Envelope(fact); + await using var connection = (NpgsqlConnection)await PostgresFixture.OpenAsync(Database.AppConnectionString); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await SchemaQueries.SetTenantAsync(connection, transaction, SchemaFixture.TenantA); + await SchemaQueries.SetSettingAsync(connection, transaction, "app.organization_id", SchemaFixture.OrgA1.ToString()); + await using var insert = new NpgsqlCommand(""" + INSERT INTO event_deliveries + (tenant_id, subscription_id, event_id, organization_id, occurred_at, + correlation_id, causation_id, actor_user_id, type, topic, partition_key, payload) + VALUES (@tenant, @subscription, @event, @organization, @occurred, + @correlation, @causation, @actor, @type, @topic, @partition, @payload) + """, connection, transaction); + insert.Parameters.AddWithValue("tenant", fact.TenantId); + insert.Parameters.AddWithValue("subscription", subscription); + insert.Parameters.AddWithValue("event", fact.EventId); + insert.Parameters.AddWithValue("organization", SchemaFixture.OrgA1); + insert.Parameters.AddWithValue("occurred", fact.OccurredAt); + insert.Parameters.AddWithValue("correlation", Trace); + insert.Parameters.AddWithValue("causation", Guid.CreateVersion7()); + insert.Parameters.AddWithValue("actor", CausalActor); + insert.Parameters.AddWithValue("type", corruption == "type" ? SensitiveCanary : typeof(ProofEvent).AssemblyQualifiedName!); + insert.Parameters.AddWithValue("topic", corruption == "topic" ? "learnstack.proof.wrong" : fact.Topic); + insert.Parameters.AddWithValue("partition", corruption == "partition" ? "wrong-partition" : fact.PartitionKey); + insert.Parameters.AddWithValue("payload", NpgsqlDbType.Jsonb, corruption == "payload" ? "{}" : fact.ToPayloadJson()); + await insert.ExecuteNonQueryAsync(); + await transaction.CommitAsync(); + return await ClaimAsync(fact, subscription); + } + private static IntegrationEventEnvelope Envelope(ProofEvent fact) => + new(fact, Trace, SchemaFixture.OrgA1, Guid.CreateVersion7(), UserId.From(CausalActor)); + private async Task ClaimAsync(ProofEvent fact, string subscription, int attempts = 1) + { + await using var connection = (NpgsqlConnection)await PostgresFixture.OpenAsync(Database.OutboxConnectionString); + await using var update = new NpgsqlCommand(""" + UPDATE event_deliveries SET locked_by = @token, + locked_until = clock_timestamp() + interval '30 seconds', attempts = @attempts + WHERE event_id = @event AND subscription_id = @subscription + RETURNING locked_until, type, topic, partition_key + """, connection); + var token = Guid.CreateVersion7().ToString("D"); + update.Parameters.AddWithValue("token", token); + update.Parameters.AddWithValue("attempts", attempts); + update.Parameters.AddWithValue("event", fact.EventId); + update.Parameters.AddWithValue("subscription", subscription); + await using var read = await update.ExecuteReaderAsync(); + (await read.ReadAsync()).Should().BeTrue(); + return new DeliveryClaim(fact.TenantId, subscription, fact.EventId, SchemaFixture.OrgA1, + token, attempts, read.GetFieldValue(0), DeliveryClaimKind.Normal, + read.GetString(1), read.GetString(2), read.GetString(3), fact.OccurredAt); + } + public async Task ChangeLeaseAsync(Guid eventId, string loss) + { + await using var connection = (NpgsqlConnection)await PostgresFixture.OpenAsync(Database.OutboxConnectionString); + var sql = loss switch + { + "token" => "UPDATE event_deliveries SET locked_by = @replacement WHERE event_id = @event", + "short" => "UPDATE event_deliveries SET locked_until = clock_timestamp() + interval '1 second' WHERE event_id = @event", + _ => "UPDATE event_deliveries SET locked_until = clock_timestamp() - interval '1 second' WHERE event_id = @event", + }; + await using var update = new NpgsqlCommand(sql, connection); + update.Parameters.AddWithValue("event", eventId); + if (loss == "token") update.Parameters.AddWithValue("replacement", Guid.CreateVersion7().ToString("D")); + (await update.ExecuteNonQueryAsync()).Should().Be(1); + } + public async Task AlterSchemaAsync(string sql) + { + await using var connection = (NpgsqlConnection)await PostgresFixture.OpenAsync(Database.MigrationConnectionString); + await using var command = new NpgsqlCommand(sql, connection); + await command.ExecuteNonQueryAsync(); + } + public async Task InsertInboxAsync(ProofEvent fact) + { + Accessor.Current = new ProbeContext(SchemaFixture.TenantA, SchemaFixture.OrgA1); + try + { + await using var scope = _host.Services.CreateAsyncScope(); + var unit = scope.ServiceProvider.GetRequiredService(); + await using var frame = await unit.BeginTransactionAsync(); + await unit.SetTenantContextAsync(scope.ServiceProvider.GetRequiredService()); + (await scope.ServiceProvider.GetRequiredService() + .TryClaimAsync(TenantId.From(fact.TenantId), Subscription, fact.EventId)).Should().BeTrue(); + await frame.CompleteAsync(); + } + finally { Accessor.Current = null; } + } + public async Task> OutboxOriginsAsync(ProofEvent fact) + { + await using var connection = (NpgsqlConnection)await PostgresFixture.OpenAsync(Database.AppConnectionString); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await SchemaQueries.SetTenantAsync(connection, transaction, SchemaFixture.TenantA); + await SchemaQueries.SetSettingAsync(connection, transaction, "app.organization_id", SchemaFixture.OrgA1.ToString()); + await using var command = new NpgsqlCommand(""" + SELECT causation_id, actor_user_id FROM outbox_messages WHERE payload ->> 'SubjectId' = @subject + """, connection, transaction); + command.Parameters.AddWithValue("subject", fact.SubjectId.ToString()); + await using var read = await command.ExecuteReaderAsync(); + var rows = new List<(Guid, Guid)>(); + while (await read.ReadAsync()) rows.Add((read.GetGuid(0), read.GetGuid(1))); + return rows; + } + public async Task<(Guid Actor, Guid Causation, Guid CausalActor)> FailedAuditOriginAsync() + { + await using var connection = (NpgsqlConnection)await PostgresFixture.OpenAsync(Database.AppConnectionString); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await SchemaQueries.SetTenantAsync(connection, transaction, SchemaFixture.TenantA); + await SchemaQueries.SetSettingAsync(connection, transaction, "app.organization_id", SchemaFixture.OrgA1.ToString()); + await using var command = new NpgsqlCommand(""" + SELECT actor_user_id, (metadata ->> 'causation_id')::uuid, + (metadata ->> 'causal_actor_user_id')::uuid + FROM audit_log WHERE operation = @operation AND outcome = 'failed' + """, connection, transaction); + command.Parameters.AddWithValue("operation", Operation); + await using var read = await command.ExecuteReaderAsync(); + (await read.ReadAsync()).Should().BeTrue(); + var origin = (read.GetGuid(0), read.GetGuid(1), read.GetGuid(2)); + (await read.ReadAsync()).Should().BeFalse(); + return origin; + } + public async Task StateAsync(ProofEvent fact, string subscription = Subscription) + { + await using var connection = (NpgsqlConnection)await PostgresFixture.OpenAsync(Database.AppConnectionString); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await SchemaQueries.SetTenantAsync(connection, transaction, fact.TenantId); + await SchemaQueries.SetSettingAsync(connection, transaction, "app.organization_id", SchemaFixture.OrgA1.ToString()); + await using var command = new NpgsqlCommand(""" + SELECT + (SELECT count(*) FROM tenant_content_types WHERE id = @subject), + (SELECT count(*) FROM inbox_messages WHERE event_id = @event AND subscription_id = @subscription), + (SELECT count(*) FROM outbox_messages WHERE payload ->> 'SubjectId' = @subject_text), + (SELECT count(*) FROM audit_log WHERE operation = @operation AND outcome = 'success' AND entity_id = @subject_text), + (SELECT processed_at IS NOT NULL FROM event_deliveries WHERE event_id = @event AND subscription_id = @subscription), + (SELECT causation_id FROM outbox_messages WHERE payload ->> 'SubjectId' = @subject_text LIMIT 1), + (SELECT actor_user_id FROM outbox_messages WHERE payload ->> 'SubjectId' = @subject_text LIMIT 1), + (SELECT actor_user_id FROM audit_log WHERE operation = @operation AND outcome = 'success' AND entity_id = @subject_text LIMIT 1), + (SELECT correlation_id FROM audit_log WHERE operation = @operation AND outcome = 'success' AND entity_id = @subject_text LIMIT 1), + (SELECT organization_id FROM audit_log WHERE operation = @operation AND outcome = 'success' AND entity_id = @subject_text LIMIT 1), + (SELECT locked_by FROM event_deliveries WHERE event_id = @event AND subscription_id = @subscription), + (SELECT (metadata ->> 'causation_id')::uuid FROM audit_log WHERE operation = @operation AND outcome = 'success' AND entity_id = @subject_text LIMIT 1), + (SELECT (metadata ->> 'causal_actor_user_id')::uuid FROM audit_log WHERE operation = @operation AND outcome = 'success' AND entity_id = @subject_text LIMIT 1), + (SELECT correlation_id FROM outbox_messages WHERE payload ->> 'SubjectId' = @subject_text LIMIT 1), + (SELECT tenant_id FROM outbox_messages WHERE payload ->> 'SubjectId' = @subject_text LIMIT 1), + (SELECT organization_id FROM outbox_messages WHERE payload ->> 'SubjectId' = @subject_text LIMIT 1), + (SELECT partition_key FROM outbox_messages WHERE payload ->> 'SubjectId' = @subject_text LIMIT 1) + """, connection, transaction); + command.Parameters.AddWithValue("subject", fact.SubjectId); + command.Parameters.AddWithValue("subject_text", fact.SubjectId.ToString()); + command.Parameters.AddWithValue("event", fact.EventId); + command.Parameters.AddWithValue("subscription", subscription); + command.Parameters.AddWithValue("operation", Operation); + await using var read = await command.ExecuteReaderAsync(); + (await read.ReadAsync()).Should().BeTrue(); + return new DurableState(read.GetInt64(0), read.GetInt64(1), read.GetInt64(2), read.GetInt64(3), + !read.IsDBNull(4) && read.GetBoolean(4), read.IsDBNull(5) ? null : read.GetGuid(5), + read.IsDBNull(6) ? null : read.GetGuid(6), read.IsDBNull(7) ? null : read.GetGuid(7), + read.IsDBNull(8) ? null : read.GetString(8), read.IsDBNull(9) ? null : read.GetGuid(9), + read.IsDBNull(10) ? null : read.GetString(10), read.IsDBNull(11) ? null : read.GetGuid(11), + read.IsDBNull(12) ? null : read.GetGuid(12), read.IsDBNull(13) ? null : read.GetString(13), + read.IsDBNull(14) ? null : read.GetGuid(14), read.IsDBNull(15) ? null : read.GetGuid(15), + read.IsDBNull(16) ? null : read.GetString(16)); + } + public async ValueTask DisposeAsync() + { + await _host.DisposeAsync(); + if (_ownsDatabase) await Database.DisposeAsync(); + } + } +} diff --git a/backend/tests/LearnStack.Tests.Integration/Database/DeliveryProcessorTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/DeliveryProcessorTests.cs new file mode 100644 index 00000000..8c76388d --- /dev/null +++ b/backend/tests/LearnStack.Tests.Integration/Database/DeliveryProcessorTests.cs @@ -0,0 +1,907 @@ +using System.Collections.Concurrent; +using System.Diagnostics; +using FluentAssertions; +using LearnStack.Api.Messaging; +using LearnStack.Application.Messaging; +using LearnStack.Infrastructure.Messaging; +using LearnStack.Modules.Customization.Domain; +using LearnStack.Modules.Customization.Infrastructure.Persistence; +using LearnStack.SharedKernel.Audit; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Localization; +using LearnStack.SharedKernel.Messaging; +using LearnStack.SharedKernel.Persistence; +using LearnStack.SharedKernel.Time; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.Mvc.Testing; +using Microsoft.AspNetCore.TestHost; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using Npgsql; +using NpgsqlTypes; +using Xunit; + +namespace LearnStack.Tests.Integration.Database; + +[Trait(RequiresDocker.Key, RequiresDocker.Value)] +[Collection(SharedSchema.Name)] +public sealed class DeliveryProcessorTests(SchemaFixture schema) +{ + private const string First = "proof.first.v1"; + private const string Second = "proof.second.v1"; + private const string Pending = "proof.pending.v1"; + private const string Privileged = "proof.privileged.v1"; + private const string Canary = "private_delivery_worker_canary_84e315"; + private const string Trace = "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"; + private static readonly TimeSpan Deadline = TimeSpan.FromSeconds(30); + private static readonly DeliveryDispatchOptions Bounds = new() + { + SqlTimeout = TimeSpan.FromSeconds(1), + ExecutionTimeout = TimeSpan.FromSeconds(2), + LeaseDuration = TimeSpan.FromSeconds(8), + ProgressTimeout = TimeSpan.FromSeconds(12), + IdlePoll = TimeSpan.FromMilliseconds(20), + SnapshotInterval = TimeSpan.FromMilliseconds(40), + ShutdownTimeout = TimeSpan.FromMilliseconds(200), + }; + + [Fact] + public async Task Actual_API_registers_one_delivery_worker_with_shared_guarded_source_and_private_health() + { + await using var fixture = await Fixture.CreateAsync(schema); + fixture.Start(); + fixture.Services.GetServices().OfType().Should().ContainSingle(); + await UntilAsync(() => fixture.Observation.IsHealthy); + var source = fixture.Services.GetRequiredKeyedService>(OutboxDispatcherDataSource.Key); + source.IsValueCreated.Should().BeTrue(); + await using var connection = await source.Value.OpenConnectionAsync(); + await using var command = new NpgsqlCommand("SELECT current_user", connection); + (await command.ExecuteScalarAsync()).Should().Be(OutboxDispatcherDataSource.Role); + using var client = fixture.Host.CreateClient(); + (await client.GetAsync("/healthz")).IsSuccessStatusCode.Should().BeTrue(); + (await client.GetAsync("/health/delivery")).StatusCode.Should().Be(System.Net.HttpStatusCode.NotFound); + } + + [Fact] + public async Task Siblings_have_independent_retries_and_Active_delivery_runs_beside_exact_Pending_and_privileged_holds() + { + await using var fixture = await Fixture.CreateAsync(schema); + var fact = Fact(); + fixture.Controls.For(fact, First).Fault = "handler"; + await fixture.InsertAsync(fact, First); + await fixture.InsertAsync(fact, Second); + await fixture.InsertAsync(fact, Pending); + await fixture.InsertAsync(fact, Privileged); + fixture.Start(); + await UntilAsync(async () => (await fixture.RowAsync(fact, First)).Error is not null + && (await fixture.RowAsync(fact, Second)).Processed); + await fixture.PostponeAsync(fact, First); + var failed = await fixture.RowAsync(fact, First); + var completed = await fixture.RowAsync(fact, Second); + failed.Attempts.Should().Be(1); + completed.Attempts.Should().Be(1); + (await fixture.RowAsync(fact, Pending)).Attempts.Should().Be(0); + (await fixture.RowAsync(fact, Privileged)).Attempts.Should().Be(0); + (await fixture.CountAsync("inbox_messages", fact.EventId)).Should().Be(1); + fixture.Controls.For(fact, First).Fault = null; + await fixture.MakeDueAsync(fact, First); + await UntilAsync(async () => (await fixture.RowAsync(fact, First)).Processed); + (await fixture.RowAsync(fact, First)).Attempts.Should().Be(2); + (await fixture.RowAsync(fact, Second)).Should().Be(completed); + fixture.Controls.For(fact, Second).Executions.Should().Be(1); + fixture.Controls.FirstConstructions.Should().Be(2); + } + + [Theory] + [InlineData(1)] + [InlineData(5)] + [InlineData(6)] + public async Task Expired_claim_recovery_spends_no_execution_and_six_has_no_seventh_invocation(int attempts) + { + await using var fixture = await Fixture.CreateAsync(schema); + var fact = Fact(); + await fixture.InsertAsync(fact); + await fixture.ScheduleAsync(fact, First, attempts, expired: true); + fixture.Start(); + await UntilAsync(async () => (await fixture.RowAsync(fact)).Error is not null); + if (attempts < 6) await fixture.PostponeAsync(fact, First); + var row = await fixture.RowAsync(fact); + row.Attempts.Should().Be(attempts); + fixture.Controls.For(fact).Executions.Should().Be(0); + if (attempts == 6) + { + await UntilAsync(async () => (await fixture.RowAsync(fact)).Deadlettered); + (await fixture.RowAsync(fact)).Terminal.Should().Be("retry_exhausted"); + } + else row.Terminal.Should().BeNull(); + } + + [Theory] + [InlineData("handler")] + [InlineData("foreign-cancel")] + [InlineData("owned-cancel")] + [InlineData("null-task")] + [InlineData("constructor")] + public async Task Worker_bounds_handler_faults_and_a_different_partition_keeps_progressing(string fault) + { + await using var fixture = await Fixture.CreateAsync(schema); + var bad = Fact(); + var good = Fact(); + fixture.Controls.For(bad).Fault = fault; + if (fault == "constructor") fixture.Controls.ConstructorFaults = 1; + await fixture.InsertAsync(bad); + await fixture.InsertAsync(good, Second); + fixture.Start(); + await UntilAsync(async () => (await fixture.RowAsync(bad)).Error is not null + && (await fixture.RowAsync(good, Second)).Processed); + await fixture.PostponeAsync(bad, First); + (await fixture.CountAsync("inbox_messages", bad.EventId)).Should().Be(0); + (await fixture.BusinessAsync(fixture.Controls.For(bad).EffectId)).Should().Be(0); + fixture.Logs.Text.Should().NotContain(Canary).And.NotContain(bad.EventId.ToString()); + (await fixture.RowAsync(bad)).Error.Should().NotContain(Canary); + } + + [Fact] + public async Task Terminal_audit_backlog_cannot_starve_normal_execution_or_expired_claim_recovery() + { + await using var fixture = await Fixture.CreateAsync(schema); + var bounds = Bounds with + { + LeaseDuration = TimeSpan.FromSeconds(5), + ProgressTimeout = TimeSpan.FromSeconds(6), + IdlePoll = TimeSpan.FromMilliseconds(50), + ShutdownTimeout = TimeSpan.FromSeconds(2), + }; + var terminals = Enumerable.Range(0, 100).Select(_ => Fact()).ToArray(); + foreach (var fact in terminals) + { + await fixture.InsertAsync(fact); + await fixture.RequestTerminalAsync(fact); + } + var recovery = Fact(); + await fixture.InsertAsync(recovery); + await fixture.ScheduleAsync(recovery, First, 5, expired: true); + await fixture.SchemaAsync("REVOKE INSERT ON audit_log FROM learnstack_outbox_admin"); + fixture.Start(bounds: bounds); + + // More failures than rows proves an acquisition was retried after its + // database lease expired. The failure remains live throughout this test. + await UntilAsync(() => fixture.Logs.IterationFailures > terminals.Length); + fixture.Observation.Snapshot.TerminalAuditFailed.Should().BeTrue(); + var normal = Fact(); + await fixture.InsertAsync(normal, Second); + await UntilAsync(async () => (await fixture.RowAsync(normal, Second)).Processed + && (await fixture.RowAsync(recovery)).Error == "recovery:unknown:expired"); + await fixture.Worker.StopAsync(CancellationToken.None).WaitAsync(Deadline); + + var completed = await fixture.RowAsync(normal, Second); + completed.Attempts.Should().Be(1); + completed.Error.Should().BeNull(); + (await fixture.CountAsync("inbox_messages", normal.EventId)).Should().Be(1); + (await fixture.BusinessAsync(fixture.Controls.For(normal, Second).EffectId)).Should().Be(1); + (await fixture.AuditCountAsync("customization.content_type.worker_second")).Should().Be(1); + var recovered = await fixture.RowAsync(recovery); + recovered.Attempts.Should().Be(5); + recovered.Processed.Should().BeFalse(); + recovered.Terminal.Should().BeNull(); + fixture.Controls.For(recovery).Executions.Should().Be(0); + (await fixture.CountAsync("inbox_messages", recovery.EventId)).Should().Be(0); + + (await fixture.PendingTerminalCountAsync()).Should().Be(terminals.Length, + "failed MUST audit must leave every terminal request intact without business execution"); + terminals.Should().OnlyContain(fact => fixture.Controls.For(fact).Executions == 0); + (await fixture.TotalCountAsync("inbox_messages")).Should().Be(1); + (await fixture.TotalCountAsync("tenant_content_types")).Should().Be(1); + (await fixture.AuditCountAsync("platform.delivery.dead_letter")).Should().Be(0); + (await fixture.SchedulerCanInsertAuditAsync()).Should().BeFalse(); + fixture.Observation.Snapshot.TerminalAuditFailed.Should().BeTrue(); + } + + [Fact] + public async Task Unclassified_handler_is_refused_before_construction_with_a_distinct_closed_descriptor() + { + await using var fixture = await Fixture.CreateAsync(schema); + var fact = Fact(); + await fixture.InsertAsync(fact); + fixture.Start(unclassifiedFirst: true); + await UntilAsync(async () => (await fixture.RowAsync(fact)).Error is not null + && fixture.Logs.Text.Contains("outcome catalogue_refused", StringComparison.Ordinal)); + await fixture.PostponeAsync(fact, First); + var refused = await fixture.RowAsync(fact); + refused.Error.Should().Be("execute:refused:catalogue"); + refused.Attempts.Should().Be(1); + refused.Processed.Should().BeFalse(); + refused.Terminal.Should().BeNull(); + fixture.Controls.FirstConstructions.Should().Be(0); + fixture.Controls.For(fact).Executions.Should().Be(0); + (await fixture.CountAsync("inbox_messages", fact.EventId)).Should().Be(0); + (await fixture.BusinessAsync(fixture.Controls.For(fact).EffectId)).Should().Be(0); + fixture.Logs.Text.Should().Contain("outcome catalogue_refused"); + } + + [Fact] + public async Task Failed_timeout_transition_still_closes_gate_and_ignored_token_cannot_commit_late() + { + await using var fixture = await Fixture.CreateAsync(schema); + var fact = Fact(); + var plan = fixture.Controls.For(fact); + plan.HoldHandler = true; + await fixture.InsertAsync(fact); + fixture.Start(); + await plan.Entered.Task.WaitAsync(Deadline); + await fixture.SchemaAsync("REVOKE UPDATE (attempts, available_after, locked_by, locked_until, terminal_reason, dead_lettered_at, last_error) ON event_deliveries FROM learnstack_outbox_admin"); + await UntilAsync(() => fixture.Observation.Snapshot.LoopFailed); + fixture.Observation.Snapshot.OutstandingExecutions.Should().Be(1); + plan.ReleaseHandler.TrySetResult(); + await plan.Disposed.Task.WaitAsync(Deadline); + (await fixture.BusinessAsync(plan.EffectId)).Should().Be(0); + (await fixture.CountAsync("inbox_messages", fact.EventId)).Should().Be(0); + (await fixture.RowAsync(fact)).Processed.Should().BeFalse(); + } + + [Fact] + public async Task Synchronous_handler_construction_cannot_monopolize_worker_wait_or_other_partitions() + { + await using var fixture = await Fixture.CreateAsync(schema); + fixture.Controls.HoldConstructor = true; + var held = Fact(); + var following = Fact(); + await fixture.InsertAsync(held); + await fixture.InsertAsync(following, Second); + fixture.Start(); + await fixture.Controls.ConstructorEntered.Task.WaitAsync(Deadline); + await UntilAsync(async () => (await fixture.RowAsync(following, Second)).Processed); + fixture.Observation.Snapshot.OutstandingExecutions.Should().Be(1); + (await fixture.RowAsync(held)).Processed.Should().BeFalse(); + fixture.Controls.ReleaseConstructor.TrySetResult(); + await UntilAsync(() => fixture.Observation.Snapshot.OutstandingExecutions == 0); + (await fixture.BusinessAsync(fixture.Controls.For(held).EffectId)).Should().Be(0); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Acknowledged_commit_survives_hung_or_faulted_scope_cleanup_without_retry(bool fault) + { + await using var fixture = await Fixture.CreateAsync(schema); + var fact = Fact(); + var plan = fixture.Controls.For(fact); + plan.HoldCleanup = !fault; + plan.FaultCleanup = fault; + await fixture.InsertAsync(fact); + fixture.Start(); + await plan.CleanupEntered.Task.WaitAsync(Deadline); + await UntilAsync(async () => (await fixture.RowAsync(fact)).Processed); + if (!fault) + { + await UntilAsync(() => fixture.Observation.Snapshot.OutstandingExecutions == 1); + fixture.Observation.IsHealthy.Should().BeFalse(); + plan.Disposed.Task.IsCompleted.Should().BeFalse(); + } + var completed = await fixture.RowAsync(fact); + completed.Error.Should().BeNull(); + completed.Attempts.Should().Be(1); + plan.ReleaseCleanup.TrySetResult(); + await plan.Disposed.Task.WaitAsync(Deadline); + await UntilAsync(() => fixture.Observation.IsHealthy); + (await fixture.RowAsync(fact)).Should().Be(completed); + (await fixture.BusinessAsync(plan.EffectId)).Should().Be(1); + plan.Executions.Should().Be(1); + } + + [Fact] + public async Task Four_retained_scopes_saturate_capacity_and_release_unblocks_only_available_partitions() + { + await using var fixture = await Fixture.CreateAsync(schema); + var held = Enumerable.Range(0, 4).Select(_ => Fact()).ToArray(); + foreach (var fact in held) + { + fixture.Controls.For(fact).HoldHandler = true; + await fixture.InsertAsync(fact); + } + var samePartition = Fact() with { SubjectId = held[0].SubjectId }; + var otherPartition = Fact(); + await fixture.InsertAsync(samePartition); + await fixture.InsertAsync(otherPartition); + fixture.Start(); + await UntilAsync(() => fixture.Observation.Snapshot.OutstandingExecutions == 4); + fixture.Observation.IsHealthy.Should().BeFalse(); + (await fixture.RowAsync(otherPartition)).Attempts.Should().Be(0); + fixture.Controls.For(held[1]).ReleaseHandler.TrySetResult(); + await UntilAsync(async () => (await fixture.RowAsync(otherPartition)).Processed); + (await fixture.RowAsync(samePartition)).Attempts.Should().Be(0); + fixture.Controls.For(held[0]).ReleaseHandler.TrySetResult(); + await UntilAsync(async () => (await fixture.RowAsync(samePartition)).Processed); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Blocked_cancellation_callback_retains_actual_scope_and_shutdown_observes_one_shared_budget(bool hostCanceled) + { + await using var fixture = await Fixture.CreateAsync(schema); + var fact = Fact(); + var plan = fixture.Controls.For(fact); + plan.HoldCancellation = true; + plan.Fault = "owned-cancel"; + await fixture.InsertAsync(fact); + var time = new ShutdownTimeProvider(Bounds.ShutdownTimeout); + fixture.Start(time: time); + await plan.CancellationEntered.Task.WaitAsync(Deadline); + plan.CleanupEntered.Task.IsCompleted.Should().BeFalse("the actual application scope must outlive callbacks"); + using var shutdown = new CancellationTokenSource(); + if (hostCanceled) shutdown.Cancel(); + time.ControlShutdownTimers(); + var stopping = fixture.Worker.StopAsync(shutdown.Token); + await time.BudgetRegistered.Task.WaitAsync(Deadline); + time.Advance(Bounds.ShutdownTimeout); + await stopping.WaitAsync(Deadline); + time.RegisteredCount.Should().Be(1, "base service and callback drain share one total deadline"); + plan.Disposed.Task.IsCompleted.Should().BeFalse(); + fixture.Observation.Snapshot.OutstandingExecutions.Should().Be(1); + plan.ReleaseCancellation.TrySetResult(); + await plan.Disposed.Task.WaitAsync(Deadline); + plan.CallbackObservedLiveScope.Should().BeTrue(); + (await fixture.BusinessAsync(plan.EffectId)).Should().Be(0); + } + + [Fact] + public async Task Faulting_cancellation_callback_is_observed_and_actual_scope_still_disposes() + { + await using var fixture = await Fixture.CreateAsync(schema); + var fact = Fact(); + var plan = fixture.Controls.For(fact); + plan.FaultCancellation = true; + plan.Fault = "owned-cancel"; + await fixture.InsertAsync(fact); + fixture.Start(); + await plan.Disposed.Task.WaitAsync(Deadline); + await UntilAsync(async () => (await fixture.RowAsync(fact)).Error is not null); + await fixture.PostponeAsync(fact, First); + (await fixture.BusinessAsync(plan.EffectId)).Should().Be(0); + fixture.Logs.Text.Should().NotContain(Canary); + } + + [Fact] + public async Task Controlled_executor_reuse_refuses_without_disposing_the_original_live_control() + { + await using var fixture = await Fixture.CreateAsync(schema); + var fact = Fact(); + var plan = fixture.Controls.For(fact); + plan.HoldHandler = true; + await fixture.InsertAsync(fact); + fixture.Start(removeWorker: true); + var claim = await fixture.Services.GetRequiredService().TryClaimAsync([], CancellationToken.None); + var executor = fixture.Services.GetRequiredService(); + var control = new DeliveryExecutionControl(); + var running = Task.Run(() => executor.ExecuteControlledAsync(claim!, control)); + await plan.Entered.Task.WaitAsync(Deadline); + await FluentActions.Awaiting(() => executor.ExecuteControlledAsync(claim!, control)) + .Should().ThrowAsync(); + running.IsCompleted.Should().BeFalse(); + control.RequestCancellation(); + control.Gate.IsClosed.Should().BeTrue(); + plan.ReleaseHandler.TrySetResult(); + var result = await running.WaitAsync(Deadline); + result.CommitAcknowledged.Should().BeFalse(); + (await fixture.BusinessAsync(plan.EffectId)).Should().Be(0); + } + + [Fact] + public async Task Lost_sixth_COMMIT_acknowledgement_is_reconciled_from_durable_completion_without_seventh_execution() + { + await using var fixture = await Fixture.CreateAsync(schema); + var fact = Fact(); + await fixture.InsertAsync(fact); + await fixture.PrimeLastAttemptAsync(fact); + await using var proxy = new CommitResponseDroppingProxy(fixture.AppConnectionString); + fixture.Start(appConnection: proxy.ConnectionString); + await UntilAsync(async () => (await fixture.RowAsync(fact)).Processed); + await UntilAsync(() => fixture.Observation.IsHealthy); + proxy.DroppedCommitResponses.Should().Be(1); + var completed = await fixture.RowAsync(fact); + completed.Attempts.Should().Be(6); + completed.Terminal.Should().BeNull(); + (await fixture.CountAsync("inbox_messages", fact.EventId)).Should().Be(1); + (await fixture.BusinessAsync(fixture.Controls.For(fact).EffectId)).Should().Be(1); + fixture.Controls.For(fact).Executions.Should().Be(1); + await fixture.ScheduleAsync(fact, First, 6, expired: true); + var before = fixture.Observation.Snapshot.LastSnapshotTimestamp; + await UntilAsync(() => fixture.Observation.Snapshot.LastSnapshotTimestamp > before); + (await fixture.RowAsync(fact)).Should().Be(completed); + fixture.Controls.For(fact).Executions.Should().Be(1); + } + + [Fact] + public async Task Successful_executions_do_not_force_scans_and_refresh_failure_keeps_the_previous_snapshot() + { + await using var fixture = await Fixture.CreateAsync(schema); + var time = new SnapshotTimeProvider(); + fixture.Start(time: time, bounds: Bounds with { SnapshotInterval = TimeSpan.FromSeconds(5) }); + await UntilAsync(() => fixture.Observation.IsHealthy); + var initial = fixture.Observation.Snapshot; + var fact = Fact(); + await fixture.InsertAsync(fact); + await UntilAsync(async () => (await fixture.RowAsync(fact)).Processed); + fixture.Observation.Snapshot.LastSnapshotTimestamp.Should().Be(initial.LastSnapshotTimestamp); + time.Advance(TimeSpan.FromSeconds(4)); + var second = Fact(); + await fixture.InsertAsync(second); + await UntilAsync(async () => (await fixture.RowAsync(second)).Processed); + fixture.Observation.Snapshot.LastSnapshotTimestamp.Should().Be(initial.LastSnapshotTimestamp); + time.Advance(TimeSpan.FromSeconds(1)); + await UntilAsync(() => fixture.Observation.Snapshot.LastSnapshotTimestamp > initial.LastSnapshotTimestamp); + var good = fixture.Observation.Snapshot; + foreach (var held in Enumerable.Range(0, 4).Select(_ => Fact())) + { + fixture.Controls.For(held).HoldHandler = true; + await fixture.InsertAsync(held); + } + await UntilAsync(() => fixture.Observation.Snapshot.OutstandingExecutions == 4); + await fixture.SchemaAsync("REVOKE SELECT (occurred_at) ON event_deliveries FROM learnstack_outbox_admin"); + time.Advance(TimeSpan.FromSeconds(5)); + await UntilAsync(() => fixture.Observation.Snapshot.RefreshFailed); + fixture.Observation.Snapshot.Queue.Should().BeSameAs(good.Queue); + fixture.Observation.Snapshot.LastSnapshotTimestamp.Should().Be(good.LastSnapshotTimestamp); + fixture.Observation.IsHealthy.Should().BeFalse(); + + await fixture.SchemaAsync("GRANT SELECT (occurred_at) ON event_deliveries TO learnstack_outbox_admin"); + await UntilAsync(() => !fixture.Observation.Snapshot.RefreshFailed); + fixture.Observation.Snapshot.LastSnapshotTimestamp.Should().Be(time.GetTimestamp(), + "a failed refresh must retry without advancing the successful-snapshot interval"); + } + + private sealed class SnapshotTimeProvider : TimeProvider + { + private long _timestamp; + public override long TimestampFrequency => TimeSpan.TicksPerSecond; + public override long GetTimestamp() => Interlocked.Read(ref _timestamp); + public void Advance(TimeSpan duration) => Interlocked.Add(ref _timestamp, duration.Ticks); + } + + private sealed class ShutdownTimeProvider(TimeSpan shutdownTimeout) : TimeProvider + { + private readonly object _gate = new(); + private readonly List _timers = []; + private bool _controlled; + private long? _origin; + private TimeSpan _elapsed; + private TimeSpan _lastRegistration; + public TaskCompletionSource BudgetRegistered { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + + public int RegisteredCount { get { lock (_gate) return _timers.Count; } } + + public void ControlShutdownTimers() { lock (_gate) _controlled = true; } + + public override ITimer CreateTimer(TimerCallback callback, object? state, TimeSpan dueTime, TimeSpan period) + { + lock (_gate) + { + if (!_controlled || dueTime != shutdownTimeout) + return System.CreateTimer(callback, state, dueTime, period); + period.Should().Be(Timeout.InfiniteTimeSpan); + _origin ??= Stopwatch.GetTimestamp(); + var registration = _timers.Count == 0 ? TimeSpan.Zero : Stopwatch.GetElapsedTime(_origin.Value); + // Preserve the real ordering even below TimeSpan's resolution: + // a later budget cannot expire at the first budget's deadline. + if (_timers.Count > 0 && registration <= _lastRegistration) + registration = _lastRegistration + TimeSpan.FromTicks(1); + _lastRegistration = registration; + var timer = new ShutdownTimer(this, callback, state, + (_elapsed > registration ? _elapsed : registration) + dueTime); + _timers.Add(timer); + BudgetRegistered.TrySetResult(); + return timer; + } + } + + public void Advance(TimeSpan duration) + { + ShutdownTimer[] due; + lock (_gate) + { + _elapsed += duration; + due = _timers.Where(timer => timer.Due is { } at && at <= _elapsed).ToArray(); + foreach (var timer in due) timer.Due = null; + } + foreach (var timer in due) timer.Fire(); + } + + private sealed class ShutdownTimer(ShutdownTimeProvider owner, TimerCallback callback, + object? state, TimeSpan due) : ITimer + { + private bool _disposed; + public TimeSpan? Due { get; set; } = due; + public void Fire() => callback(state); + public bool Change(TimeSpan dueTime, TimeSpan period) + { + lock (owner._gate) + { + if (_disposed) return false; + period.Should().Be(Timeout.InfiniteTimeSpan); + Due = dueTime == Timeout.InfiniteTimeSpan ? null : owner._elapsed + dueTime; + return true; + } + } + public void Dispose() + { + lock (owner._gate) { _disposed = true; Due = null; } + } + public ValueTask DisposeAsync() { Dispose(); return ValueTask.CompletedTask; } + } + } + + private static async Task UntilAsync(Func condition) => + await UntilAsync(() => Task.FromResult(condition())); + private static async Task UntilAsync(Func> condition) + { + using var deadline = new CancellationTokenSource(Deadline); + while (!await condition()) await Task.Delay(10, deadline.Token); + } + + private static ProofEvent Fact() => new() + { + EventId = Guid.CreateVersion7(), + TenantId = SchemaFixture.TenantA, + OccurredAt = new DateTimeOffset(2026, 10, 13, 9, 0, 0, TimeSpan.Zero), + SubjectId = Guid.CreateVersion7(), + }; + [IntegrationEventPartition(nameof(SubjectId))] + public sealed record ProofEvent : IntegrationEventBase + { + public required Guid SubjectId { get; init; } + public override string Topic => "learnstack.proof.worker"; + public override string PartitionKey => SubjectId.ToString("D"); + } + public sealed class Controls + { + private readonly ConcurrentDictionary<(Guid, string), Plan> _plans = new(); + internal int ConstructorFaults; + private int _firstConstructions; + private volatile bool _holdConstructor; + public int FirstConstructions => Volatile.Read(ref _firstConstructions); + public bool HoldConstructor { get => _holdConstructor; set => _holdConstructor = value; } + internal void ConstructedFirst() => Interlocked.Increment(ref _firstConstructions); + public TaskCompletionSource ConstructorEntered { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + public TaskCompletionSource ReleaseConstructor { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + public Plan For(ProofEvent fact, string subscription = First) => + _plans.GetOrAdd((fact.EventId, subscription), _ => new Plan()); + public void ReleaseAll() + { + ReleaseConstructor.TrySetResult(); + foreach (var plan in _plans.Values) + { + plan.ReleaseHandler.TrySetResult(); + plan.ReleaseCancellation.TrySetResult(); + plan.ReleaseCleanup.TrySetResult(); + } + } + public Task DrainAsync() => Task.WhenAll(_plans.Values.Where(plan => plan.Entered.Task.IsCompleted) + .Select(plan => plan.Disposed.Task)); + } + public sealed class Plan + { + private volatile string? _fault; + private volatile bool _holdHandler; + private volatile bool _holdCleanup; + private volatile bool _faultCleanup; + private volatile bool _holdCancellation; + private volatile bool _faultCancellation; + private volatile bool _callbackObservedLiveScope; + public Guid EffectId { get; } = Guid.CreateVersion7(); + public string? Fault { get => _fault; set => _fault = value; } + public bool HoldHandler { get => _holdHandler; set => _holdHandler = value; } + public bool HoldCleanup { get => _holdCleanup; set => _holdCleanup = value; } + public bool FaultCleanup { get => _faultCleanup; set => _faultCleanup = value; } + public bool HoldCancellation { get => _holdCancellation; set => _holdCancellation = value; } + public bool FaultCancellation { get => _faultCancellation; set => _faultCancellation = value; } + public bool CallbackObservedLiveScope { get => _callbackObservedLiveScope; set => _callbackObservedLiveScope = value; } + internal int Executions; + public TaskCompletionSource Entered { get; } = Signal(); + public TaskCompletionSource ReleaseHandler { get; } = Signal(); + public TaskCompletionSource CleanupEntered { get; } = Signal(); + public TaskCompletionSource ReleaseCleanup { get; } = Signal(); + public TaskCompletionSource CancellationEntered { get; } = Signal(); + public TaskCompletionSource ReleaseCancellation { get; } = Signal(); + public TaskCompletionSource Disposed { get; } = Signal(); + private static TaskCompletionSource Signal() => new(TaskCreationOptions.RunContinuationsAsynchronously); + } + public sealed class FirstHandler(IServiceProvider services, Controls controls) + : Handler(services, controls, First), IIntegrationEventHandler; + public sealed class SecondHandler(IServiceProvider services, Controls controls) + : Handler(services, controls, Second), IIntegrationEventHandler; + public sealed class PrivilegedHandler(IServiceProvider services, Controls controls) + : Handler(services, controls, Privileged), IIntegrationEventHandler; + public abstract class Handler + { + private readonly IServiceProvider _services; + private readonly Controls _controls; + private readonly string _subscription; + protected Handler(IServiceProvider services, Controls controls, string subscription) + { + _services = services; + _controls = controls; + _subscription = subscription; + if (subscription == First) controls.ConstructedFirst(); + if (subscription == First && controls.HoldConstructor) + { + controls.ConstructorEntered.TrySetResult(); + controls.ReleaseConstructor.Task.GetAwaiter().GetResult(); + } + if (subscription == First && Interlocked.CompareExchange(ref controls.ConstructorFaults, 0, 1) == 1) + throw new InvalidOperationException(Canary); + } + public Task HandleAsync(ProofEvent fact, CancellationToken cancellationToken = default) + { + var plan = _controls.For(fact, _subscription); + Interlocked.Increment(ref plan.Executions); + var cleanup = _services.GetRequiredService(); + cleanup.Plan = plan; + return plan.Fault == "null-task" ? null! : WorkAsync(fact, plan, cleanup, cancellationToken); + } + private async Task WorkAsync(ProofEvent fact, Plan plan, Cleanup cleanup, CancellationToken cancellationToken) + { + await Task.Yield(); + var unit = _services.GetRequiredService(); + await SchemaQueries.AssertApplicationRoleAsync(unit.Connection); + var db = _services.GetRequiredService(); + db.Add(TenantContentType.Create(TenantContentTypeId.From(plan.EffectId), TenantId.From(fact.TenantId), + $"worker-{plan.EffectId:N}"[..24], 1, LocalizedText.From(("en", "Worker proof")), + """{"type":"object"}""", "default-card", _services.GetRequiredService(), UserId.SystemActor)); + await db.SaveChangesAsync(cancellationToken); + if (plan.HoldCancellation || plan.FaultCancellation) + cleanup.Registration = cancellationToken.Register(() => + { + plan.CancellationEntered.TrySetResult(); + if (plan.HoldCancellation) plan.ReleaseCancellation.Task.GetAwaiter().GetResult(); + plan.CallbackObservedLiveScope = !plan.Disposed.Task.IsCompleted && !plan.CleanupEntered.Task.IsCompleted; + if (plan.FaultCancellation) throw new InvalidOperationException(Canary); + }); + plan.Entered.TrySetResult(); + if (plan.HoldHandler) await plan.ReleaseHandler.Task; + if (plan.Fault == "owned-cancel") await Task.Delay(Timeout.InfiniteTimeSpan, cancellationToken); + if (plan.Fault == "handler") throw new InvalidOperationException(Canary); + if (plan.Fault == "foreign-cancel") throw new OperationCanceledException(Canary, new CancellationToken(true)); + } + } + public sealed class Cleanup : IAsyncDisposable + { + public Plan? Plan { get; set; } + public CancellationTokenRegistration Registration { get; set; } + public async ValueTask DisposeAsync() + { + if (Plan is not { } plan) return; + plan.CleanupEntered.TrySetResult(); + if (plan.HoldCleanup) await plan.ReleaseCleanup.Task; + Registration.Dispose(); + plan.Disposed.TrySetResult(); + if (plan.FaultCleanup) throw new InvalidOperationException(Canary); + } + } + private sealed class Catalog(bool unclassifiedFirst) : IAuditCatalogSource + { + public string ModuleName => "customization"; + public void Describe(IAuditCatalogBuilder builder) + { + if (!unclassifiedFirst) + builder.MustAudit("customization.content_type.worker_first", OperationType.Create, typeof(TenantContentType)); + builder.MustAudit("customization.content_type.worker_second", OperationType.Create, typeof(TenantContentType)); + builder.MustAudit("customization.content_type.worker_privileged", OperationType.Create, typeof(TenantContentType)); + } + } + private sealed record Row(int Attempts, bool Processed, string? Error, string? Terminal, bool Deadlettered); + private sealed class Fixture : IAsyncDisposable + { + private readonly DisposableSchemaDatabase _database; + private WorkerFactory? _host; + private Fixture(DisposableSchemaDatabase database) => _database = database; + public string AppConnectionString => _database.AppConnectionString; + public Controls Controls { get; } = new(); + public Logs Logs { get; } = new(); + public WorkerFactory Host => _host!; + public IServiceProvider Services => Host.Services; + public DeliveryObservation Observation => Services.GetRequiredService(); + public DeliveryProcessor Worker => Services.GetServices().OfType().Single(); + public static async Task CreateAsync(SchemaFixture schema) + { + var fixture = new Fixture(await DisposableSchemaDatabase.CreateAsync(schema.Postgres)); + await using var connection = await fixture.AppAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.SetTenantAsync(connection, transaction, SchemaFixture.TenantA); + await SchemaQueries.ExecuteAsync(connection, transaction, """ + INSERT INTO tenants (id, slug, display_name, status, created_at, created_by, row_version) + VALUES (@tenant, 'delivery-worker', 'Delivery worker', 'Trial', now(), @actor, 0) + """, ("tenant", SchemaFixture.TenantA), ("actor", UserId.SystemActor.Value)); + await transaction.CommitAsync(); + return fixture; + } + public void Start(bool removeWorker = false, TimeProvider? time = null, DeliveryDispatchOptions? bounds = null, + string? appConnection = null, bool unclassifiedFirst = false) + { + _host = new WorkerFactory(this, removeWorker, time, bounds, appConnection, unclassifiedFirst); + _ = Services; + } + public async Task InsertAsync(ProofEvent fact, string subscription = First) + { + await using var connection = await AppAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.SetTenantAsync(connection, transaction, fact.TenantId); + await using var command = new NpgsqlCommand(""" + INSERT INTO event_deliveries + (tenant_id, subscription_id, event_id, occurred_at, correlation_id, type, topic, partition_key, payload) + VALUES (@tenant, @subscription, @event, @occurred, @trace, @type, @topic, @partition, @payload) + """, connection, transaction); + command.Parameters.AddWithValue("tenant", fact.TenantId); + command.Parameters.AddWithValue("subscription", subscription); + command.Parameters.AddWithValue("event", fact.EventId); + command.Parameters.AddWithValue("occurred", fact.OccurredAt); + command.Parameters.AddWithValue("trace", Trace); + command.Parameters.AddWithValue("type", typeof(ProofEvent).AssemblyQualifiedName!); + command.Parameters.AddWithValue("topic", fact.Topic); + command.Parameters.AddWithValue("partition", fact.PartitionKey); + command.Parameters.AddWithValue("payload", NpgsqlDbType.Jsonb, fact.ToPayloadJson()); + await command.ExecuteNonQueryAsync(); + await transaction.CommitAsync(); + } + public async Task RowAsync(ProofEvent fact, string subscription = First) + { + await using var connection = await DispatchAsync(); + await using var command = new NpgsqlCommand(""" + SELECT attempts, processed_at IS NOT NULL, last_error, terminal_reason, dead_lettered_at IS NOT NULL + FROM event_deliveries WHERE event_id = @event AND subscription_id = @subscription + """, connection); + command.Parameters.AddWithValue("event", fact.EventId); + command.Parameters.AddWithValue("subscription", subscription); + await using var reader = await command.ExecuteReaderAsync(); + (await reader.ReadAsync()).Should().BeTrue(); + return new(reader.GetInt32(0), reader.GetBoolean(1), reader.IsDBNull(2) ? null : reader.GetString(2), + reader.IsDBNull(3) ? null : reader.GetString(3), reader.GetBoolean(4)); + } + public Task PrimeLastAttemptAsync(ProofEvent fact) => UpdateAsync(fact, First, + "UPDATE event_deliveries SET attempts = 5 WHERE event_id = @event AND subscription_id = @subscription"); + public Task RequestTerminalAsync(ProofEvent fact) => UpdateAsync(fact, First, + "UPDATE event_deliveries SET attempts = 6, terminal_reason = 'retry_exhausted' WHERE event_id = @event AND subscription_id = @subscription"); + public Task MakeDueAsync(ProofEvent fact, string subscription) => UpdateAsync(fact, subscription, + "UPDATE event_deliveries SET available_after = clock_timestamp() WHERE event_id = @event AND subscription_id = @subscription"); + public Task PostponeAsync(ProofEvent fact, string subscription) => UpdateAsync(fact, subscription, + "UPDATE event_deliveries SET available_after = clock_timestamp() + interval '1 hour' WHERE event_id = @event AND subscription_id = @subscription"); + public Task ScheduleAsync(ProofEvent fact, string subscription, int attempts, bool expired) => UpdateAsync(fact, subscription, + $"UPDATE event_deliveries SET attempts = {attempts}, locked_by = '{Guid.CreateVersion7():D}', locked_until = clock_timestamp() {(expired ? "-" : "+")} interval '1 minute' WHERE event_id = @event AND subscription_id = @subscription"); + private async Task UpdateAsync(ProofEvent fact, string subscription, string sql) + { + await using var connection = await DispatchAsync(); + await using var command = new NpgsqlCommand(sql, connection); + command.Parameters.AddWithValue("event", fact.EventId); + command.Parameters.AddWithValue("subscription", subscription); + await command.ExecuteNonQueryAsync(); + } + public async Task CountAsync(string table, Guid eventId) => await ScalarAsync( + $"SELECT count(*) FROM {table} WHERE event_id = @id", eventId); + public Task BusinessAsync(Guid id) => ScalarAsync("SELECT count(*) FROM tenant_content_types WHERE id = @id", id); + public async Task TotalCountAsync(string table) + { + await using var connection = await AppAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.SetTenantAsync(connection, transaction, SchemaFixture.TenantA); + await using var command = new NpgsqlCommand($"SELECT count(*) FROM {table}", connection, transaction); + return (long)(await command.ExecuteScalarAsync())!; + } + public async Task AuditCountAsync(string operation) + { + await using var connection = await AppAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.SetTenantAsync(connection, transaction, SchemaFixture.TenantA); + await using var command = new NpgsqlCommand("SELECT count(*) FROM audit_log WHERE operation = @operation", connection, transaction); + command.Parameters.AddWithValue("operation", operation); + return (long)(await command.ExecuteScalarAsync())!; + } + public async Task PendingTerminalCountAsync() + { + await using var connection = await DispatchAsync(); + await using var command = new NpgsqlCommand(""" + SELECT count(*) FROM event_deliveries + WHERE terminal_reason = 'retry_exhausted' AND attempts = 6 + AND processed_at IS NULL AND dead_lettered_at IS NULL + """, connection); + return (long)(await command.ExecuteScalarAsync())!; + } + public async Task SchedulerCanInsertAuditAsync() + { + await using var connection = await DispatchAsync(); + await using var command = new NpgsqlCommand("SELECT has_table_privilege(current_user, 'audit_log', 'INSERT')", connection); + return (bool)(await command.ExecuteScalarAsync())!; + } + private async Task ScalarAsync(string sql, Guid id) + { + await using var connection = await AppAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.SetTenantAsync(connection, transaction, SchemaFixture.TenantA); + await using var command = new NpgsqlCommand(sql, connection, transaction); + command.Parameters.AddWithValue("id", id); + return (long)(await command.ExecuteScalarAsync())!; + } + public async Task SchemaAsync(string sql) + { + await using var connection = (NpgsqlConnection)await PostgresFixture.OpenAsync(_database.MigrationConnectionString); + await using var command = new NpgsqlCommand(sql, connection); + await command.ExecuteNonQueryAsync(); + } + private async Task AppAsync() + { + var connection = (NpgsqlConnection)await PostgresFixture.OpenAsync(_database.AppConnectionString); + await SchemaQueries.AssertApplicationRoleAsync(connection); + return connection; + } + private async Task DispatchAsync() => + (NpgsqlConnection)await PostgresFixture.OpenAsync(_database.OutboxConnectionString); + public async ValueTask DisposeAsync() + { + Controls.ReleaseAll(); + if (_host is not null) + { + var worker = Services.GetServices().OfType().SingleOrDefault(); + if (worker is not null) await worker.StopAsync(CancellationToken.None).WaitAsync(Deadline); + await Controls.DrainAsync().WaitAsync(Deadline); + await _host.DisposeAsync(); + } + await _database.DisposeAsync(); + } + public sealed class WorkerFactory(Fixture fixture, bool removeWorker, TimeProvider? time, + DeliveryDispatchOptions? bounds, string? appConnection, bool unclassifiedFirst) : WebApplicationFactory + { + protected override void ConfigureWebHost(IWebHostBuilder builder) + { + builder.UseEnvironment(Environments.Development); + builder.UseSetting("ConnectionStrings:Default", appConnection ?? fixture._database.AppConnectionString); + builder.UseSetting("ConnectionStrings:PlatformAdmin", fixture._database.PlatformConnectionString); + builder.UseSetting("ConnectionStrings:OutboxDispatcher", fixture._database.OutboxConnectionString); + builder.ConfigureTestServices(services => + { + foreach (var descriptor in services.Where(descriptor => descriptor.ServiceType == typeof(IHostedService) + && (descriptor.ImplementationType == typeof(OutboxProcessor) + || removeWorker && descriptor.ImplementationType == typeof(DeliveryProcessor))).ToArray()) + services.Remove(descriptor); + services.RemoveAll(); + services.AddSingleton(bounds ?? Bounds); + if (time is not null) { services.RemoveAll(); services.AddSingleton(time); } + var handlers = IntegrationEventHandlerRegistry.FromServiceDescriptors([ + ServiceDescriptor.Scoped, FirstHandler>(), + ServiceDescriptor.Scoped, SecondHandler>(), + ServiceDescriptor.Scoped, PrivilegedHandler>(), + ]); + services.RemoveAll(); + services.AddSingleton(handlers); + services.RemoveAll(); + services.AddSingleton(OutboxEventRegistry.Create([ + OutboxEventRegistration.Required(typeof(ProofEvent), [ + OutboxSubscriptionManifest.Active(First, typeof(FirstHandler)), + OutboxSubscriptionManifest.Active(Second, typeof(SecondHandler)), + OutboxSubscriptionManifest.Pending(Pending), + OutboxSubscriptionManifest.Active(Privileged, typeof(PrivilegedHandler), OutboxSubscriptionExecutionScope.PrivilegedPlatform), + ]), + ], [], handlers, [typeof(ProofEvent).Assembly])); + services.AddSingleton(fixture.Controls); + services.AddScoped(); + services.AddScoped(); + services.AddScoped(); + services.AddScoped(); + services.AddSingleton(new Catalog(unclassifiedFirst)); + services.AddSingleton>(fixture.Logs); + }); + } + } + } + public sealed class Logs : ILogger + { + private readonly ConcurrentQueue _messages = new(); + public string Text => string.Join('\n', _messages); + public int IterationFailures => _messages.Count(message => message.StartsWith( + "Delivery dispatch iteration failed with outcome iteration_failed.", StringComparison.Ordinal)); + public IDisposable? BeginScope(TState state) where TState : notnull => null; + public bool IsEnabled(LogLevel logLevel) => true; + public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, + Func formatter) => _messages.Enqueue(formatter(state, exception) + exception); + } +} diff --git a/backend/tests/LearnStack.Tests.Integration/Database/DomainEventProductionTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/DomainEventProductionTests.cs index 59fbea34..71ff982a 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/DomainEventProductionTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/DomainEventProductionTests.cs @@ -777,7 +777,7 @@ public async Task Disposing_an_unfinished_scope_discards_collected_and_uncollect await unit.SetTenantContextAsync(new ActingContext()); using var invocation = (ProducerExecution.Invocation)typeof(ProducerExecution) .GetMethod("BeginInvocation", BindingFlags.Instance | BindingFlags.NonPublic)! - .Invoke(services.GetRequiredService(), [true, null])!; + .Invoke(services.GetRequiredService(), [true, null, null])!; source = Source(services); var db = services.GetRequiredService(); db.Add(source); diff --git a/backend/tests/LearnStack.Tests.Integration/Database/DurableEventAcceptanceTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/DurableEventAcceptanceTests.cs new file mode 100644 index 00000000..9b74a858 --- /dev/null +++ b/backend/tests/LearnStack.Tests.Integration/Database/DurableEventAcceptanceTests.cs @@ -0,0 +1,782 @@ +using FluentAssertions; +using System.Text.Json; +using LearnStack.Infrastructure.Messaging; +using LearnStack.Infrastructure.Observability; +using LearnStack.Infrastructure.Persistence; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Messaging; +using LearnStack.SharedKernel.Persistence; +using LearnStack.SharedKernel.Tenancy; +using Microsoft.Extensions.DependencyInjection; +using Npgsql; +using NpgsqlTypes; +using Xunit; + +namespace LearnStack.Tests.Integration.Database; + +/// Durable publication through the real app-role unit of work and PostgreSQL. +[Trait(RequiresDocker.Key, RequiresDocker.Value)] +[Collection(SharedSchema.Name)] +public sealed class DurableEventAcceptanceTests(SchemaFixture schema) +{ + private const string Trace = "00-11111111111111111111111111111111-2222222222222222-01"; + private const string OtherTrace = "00-33333333333333333333333333333333-4444444444444444-01"; + private const string First = "proof.first.v1"; + private const string Second = "proof.second.v1"; + private const string EarlierIdentity = "LearnStack.Previous.ProofEvent, LearnStack.Previous.Contracts, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"; + private const string OtherEarlierIdentity = "LearnStack.Previous.OtherEvent, LearnStack.Previous.Contracts, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"; + private const string RetiredIdentity = "LearnStack.Retired.ProofEvent, LearnStack.Retired.Contracts"; + private static readonly DateTimeOffset Occurred = new(2026, 10, 13, 10, 20, 30, TimeSpan.Zero); + + [Fact] + public async Task Integration_Event_Publication_Is_Durable_Acceptance() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var services = Compose(source); + var fact = Event(); + + await services.GetRequiredService().PublishAsync(Envelope(fact)); + + var rows = await RowsAsync(database.AppConnectionString, fact.EventId); + rows.Select(row => row.Subscription).Should().BeEquivalentTo([First, Second]); + rows.Should().OnlyContain(row => row.Tenant == SchemaFixture.TenantA + && row.Organization == SchemaFixture.OrgA1 && row.EventId == fact.EventId + && row.Attempts == 0 && row.ProcessedAt == null && row.LastError == null); + ExplodingHandler.Constructions.Should().Be(0); + ExplodingHandler.Invocations.Should().Be(0); + } + + [Fact] + public async Task Identical_republication_preserves_status_lease_and_row_version() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var services = Compose(source); + var envelope = Envelope(Event()); + var bus = services.GetRequiredService(); + await bus.PublishAsync(envelope); + await using (var admin = await OpenAsync(database.OutboxConnectionString)) + await using (var update = new NpgsqlCommand(""" + UPDATE event_deliveries SET attempts = 2, last_error = 'publish:failed:closed', + available_after = clock_timestamp() + interval '1 hour' + WHERE event_id = @event + """, admin)) + { + update.Parameters.AddWithValue("event", envelope.Event.EventId); + (await update.ExecuteNonQueryAsync()).Should().Be(2); + } + var before = await RowsAsync(database.AppConnectionString, envelope.Event.EventId); + + await bus.PublishAsync(envelope); + + (await RowsAsync(database.AppConnectionString, envelope.Event.EventId)).Should().BeEquivalentTo(before); + } + + [Fact] + public async Task Canonical_republication_preserves_compatible_alias_copies_and_their_scheduling_state() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var services = Compose(source, withAliases: true); + var envelope = Envelope(Event()); + var registry = services.GetRequiredService(); + registry.Resolve(EarlierIdentity).Disposition.Should().Be(OutboxEventDisposition.Active); + registry.Resolve(EarlierIdentity).EventType.Should().Be(); + await InsertCopyAsync(database.AppConnectionString, envelope, First, storedType: EarlierIdentity); + await InsertCopyAsync(database.AppConnectionString, envelope, Second, storedType: EarlierIdentity); + await using (var admin = await OpenAsync(database.OutboxConnectionString)) + await using (var update = new NpgsqlCommand(""" + UPDATE event_deliveries SET attempts = 2, last_error = 'execute:failed:handler', + available_after = clock_timestamp() + interval '1 hour', + locked_by = @token, locked_until = clock_timestamp() + interval '1 hour' + WHERE event_id = @event + """, admin)) + { + update.Parameters.AddWithValue("token", Guid.CreateVersion7().ToString("D")); + update.Parameters.AddWithValue("event", envelope.Event.EventId); + (await update.ExecuteNonQueryAsync()).Should().Be(2); + } + var before = await RowsAsync(database.AppConnectionString, envelope.Event.EventId); + + await services.GetRequiredService().PublishAsync(envelope); + + (await RowsAsync(database.AppConnectionString, envelope.Event.EventId)).Should().BeEquivalentTo(before); + ExplodingHandler.Constructions.Should().Be(0); + } + + [Fact] + public async Task A_compatible_alias_subset_is_completed_with_the_current_canonical_identity() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var services = Compose(source, withAliases: true); + var envelope = Envelope(Event()); + await InsertCopyAsync(database.AppConnectionString, envelope, First, storedType: EarlierIdentity); + var before = (await RowsAsync(database.AppConnectionString, envelope.Event.EventId)).Single(); + + await services.GetRequiredService().PublishAsync(envelope); + + var rows = await RowsAsync(database.AppConnectionString, envelope.Event.EventId); + rows.Select(row => row.Subscription).Should().BeEquivalentTo([First, Second]); + rows.Single(row => row.Subscription == First).Should().BeEquivalentTo(before); + rows.Single(row => row.Subscription == Second).Type.Should().Be(typeof(ProofEvent).AssemblyQualifiedName); + ExplodingHandler.Constructions.Should().Be(0); + } + + [Theory] + [InlineData("other-canonical")] + [InlineData("other-alias")] + [InlineData("unknown")] + [InlineData("retired")] + [InlineData("alias-payload")] + public async Task Alias_compatibility_never_accepts_a_different_contract_or_changed_payload(string collision) + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var services = Compose(source, withAliases: true); + var envelope = Envelope(Event()); + var storedType = collision switch + { + "other-canonical" => typeof(OtherEvent).AssemblyQualifiedName!, + "other-alias" => OtherEarlierIdentity, + "unknown" => "LearnStack.Undeclared.ProofEvent, LearnStack.Undeclared.Contracts", + "retired" => RetiredIdentity, + "alias-payload" => EarlierIdentity, + _ => throw new ArgumentOutOfRangeException(nameof(collision)), + }; + var expectedType = collision switch + { + "other-canonical" or "other-alias" => typeof(OtherEvent), + "alias-payload" => typeof(ProofEvent), + _ => null, + }; + services.GetRequiredService().Resolve(storedType).EventType.Should().Be(expectedType); + var storedEnvelope = collision == "alias-payload" + ? Envelope(((ProofEvent)envelope.Event) with { RelatedId = Guid.CreateVersion7() }) + : envelope; + // The earlier missing subscription would be inserted before this later + // collision; refusal must roll that insertion back with the whole set. + await InsertCopyAsync(database.AppConnectionString, storedEnvelope, Second, storedType: storedType, + storedPartition: envelope.PartitionKey); + var before = await RowsAsync(database.AppConnectionString, envelope.Event.EventId); + + await FluentActions.Awaiting(() => services.GetRequiredService().PublishAsync(envelope)) + .Should().ThrowAsync(); + + (await RowsAsync(database.AppConnectionString, envelope.Event.EventId)).Should().BeEquivalentTo(before); + ExplodingHandler.Constructions.Should().Be(0); + } + + [Theory] + [InlineData("payload")] + [InlineData("occurred-microsecond")] + [InlineData("organization")] + [InlineData("causation")] + [InlineData("actor")] + [InlineData("correlation")] + public async Task Any_immutable_envelope_collision_refuses_without_overwriting(string change) + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var services = Compose(source); + var fact = Event(); + var original = Envelope(fact); + var bus = services.GetRequiredService(); + await bus.PublishAsync(original); + var before = await RowsAsync(database.AppConnectionString, fact.EventId); + var collision = change switch + { + "payload" => Envelope(fact with { SubjectId = Guid.CreateVersion7() }), + "occurred-microsecond" => Envelope(fact with { OccurredAt = fact.OccurredAt.AddTicks(10) }), + "organization" => Envelope(fact, organization: SchemaFixture.OrgA2), + "causation" => Envelope(fact, causation: Guid.CreateVersion7()), + "actor" => Envelope(fact, actor: UserId.From(Guid.CreateVersion7())), + "correlation" => Envelope(fact, trace: OtherTrace), + _ => throw new ArgumentOutOfRangeException(nameof(change)), + }; + + await FluentActions.Awaiting(() => bus.PublishAsync(collision)).Should().ThrowAsync(); + + (await RowsAsync(database.AppConnectionString, fact.EventId)).Should().BeEquivalentTo(before); + } + + [Fact] + public async Task A_matching_existing_subset_is_completed_without_changing_its_copy() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var services = Compose(source); + var envelope = Envelope(Event()); + await InsertCopyAsync(database.AppConnectionString, envelope, First); + var before = (await RowsAsync(database.AppConnectionString, envelope.Event.EventId)).Single(); + + await services.GetRequiredService().PublishAsync(envelope); + + var rows = await RowsAsync(database.AppConnectionString, envelope.Event.EventId); + rows.Should().HaveCount(2); + rows.Single(row => row.Subscription == First).Should().BeEquivalentTo(before); + rows.Single(row => row.Subscription == Second).Should().NotBeNull(); + } + + [Fact] + public async Task Equivalent_json_with_different_object_key_order_is_an_identical_copy() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var services = Compose(source); + var envelope = Envelope(Event()); + using var document = JsonDocument.Parse(((IntegrationEventBase)envelope.Event).ToPayloadJson()); + var reordered = JsonSerializer.Serialize(document.RootElement.EnumerateObject() + .Reverse().ToDictionary(property => property.Name, property => property.Value)); + await InsertCopyAsync(database.AppConnectionString, envelope, First, reordered); + var existing = (await RowsAsync(database.AppConnectionString, envelope.Event.EventId)).Single(); + + await services.GetRequiredService().PublishAsync(envelope); + + var rows = await RowsAsync(database.AppConnectionString, envelope.Event.EventId); + rows.Should().HaveCount(2); + rows.Single(row => row.Subscription == First).Should().BeEquivalentTo(existing); + } + + [Fact] + public async Task A_conflicting_later_copy_rolls_back_the_new_earlier_copy() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var services = Compose(source); + var envelope = Envelope(Event()); + await InsertCopyAsync(database.AppConnectionString, Envelope( + ((ProofEvent)envelope.Event) with { SubjectId = Guid.CreateVersion7() }), Second); + var before = await RowsAsync(database.AppConnectionString, envelope.Event.EventId); + + await FluentActions.Awaiting(() => services.GetRequiredService().PublishAsync(envelope)) + .Should().ThrowAsync(); + + (await RowsAsync(database.AppConnectionString, envelope.Event.EventId)).Should().BeEquivalentTo(before); + } + + [Theory] + [InlineData("type")] + [InlineData("topic")] + [InlineData("partition")] + public async Task Stored_contract_field_collision_refuses_without_overwriting(string field) + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var services = Compose(source); + var envelope = Envelope(Event()); + await InsertCopyAsync(database.AppConnectionString, envelope, First, + storedType: field == "type" ? "proof.previous.contract" : null, + storedTopic: field == "topic" ? "learnstack.proof.other" : null, + storedPartition: field == "partition" ? Guid.CreateVersion7().ToString() : null); + var before = await RowsAsync(database.AppConnectionString, envelope.Event.EventId); + + await FluentActions.Awaiting(() => services.GetRequiredService().PublishAsync(envelope)) + .Should().ThrowAsync(); + + (await RowsAsync(database.AppConnectionString, envelope.Event.EventId)).Should().BeEquivalentTo(before); + } + + [Fact] + public async Task Simultaneous_duplicate_publishers_commit_only_the_declared_tuple_copies() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var first = Compose(source); + await using var second = Compose(source); + var envelope = Envelope(Event()); + var buses = Enumerable.Range(0, 8) + .Select(index => (index % 2 == 0 ? first : second).GetRequiredService()) + .ToArray(); + + await Task.WhenAll(buses.Select(bus => Task.Run(() => bus.PublishAsync(envelope)))); + + (await RowsAsync(database.AppConnectionString, envelope.Event.EventId)) + .Select(row => row.Subscription).Should().BeEquivalentTo([First, Second]); + ExplodingHandler.Constructions.Should().Be(0); + } + + [Theory] + [InlineData(true)] + [InlineData(false)] + public async Task A_publisher_waits_for_an_uncommitted_duplicate_then_reconciles_its_outcome( + bool firstCommits) + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var services = Compose(source); + var envelope = Envelope(Event()); + await using var blocker = await OpenAsync(database.AppConnectionString); + await using var blockerTransaction = await blocker.BeginTransactionAsync(); + await SchemaQueries.SetTenantAsync(blocker, blockerTransaction, SchemaFixture.TenantA); + await SchemaQueries.SetSettingAsync(blocker, blockerTransaction, "app.organization_id", + SchemaFixture.OrgA1.ToString()); + await InsertMatchingFirstAsync(blocker, blockerTransaction, envelope); + await using var pid = new NpgsqlCommand("SELECT pg_backend_pid()", blocker, blockerTransaction); + var blockerPid = (int)(await pid.ExecuteScalarAsync())!; + + var publish = services.GetRequiredService().PublishAsync(envelope); + await WaitForBlockedAcceptanceAsync(database.AppConnectionString, blockerPid); + publish.IsCompleted.Should().BeFalse("a separate app transaction owns the tuple until commit or rollback"); + if (firstCommits) await blockerTransaction.CommitAsync(); + else await blockerTransaction.RollbackAsync(); + var committedFirst = firstCommits + ? (await RowsAsync(database.AppConnectionString, envelope.Event.EventId)) + .Single(row => row.Subscription == First) + : null; + + await publish.WaitAsync(TimeSpan.FromSeconds(10)); + + var rows = await RowsAsync(database.AppConnectionString, envelope.Event.EventId); + rows.Select(row => row.Subscription).Should().BeEquivalentTo([First, Second]); + if (firstCommits) + rows.Single(row => row.Subscription == First).Should().BeEquivalentTo(committedFirst); + ExplodingHandler.Constructions.Should().Be(0); + } + + [Fact] + public async Task Publication_uses_a_fresh_transaction_and_restores_the_caller_ambient_context() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + var caller = EventTenantContext.FromEnvelope(Envelope(Event(SchemaFixture.TenantB), + organization: SchemaFixture.OrgB1)); + await using var services = Compose(source); + var accessor = services.GetRequiredService(); + accessor.Current = caller; + await using var publisher = new NpgsqlUnitOfWork(source, + Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance); + await using var publisherFrame = await publisher.BeginTransactionAsync(); + await publisher.SetTenantContextAsync(caller); + var marker = Guid.CreateVersion7(); + await using (var insert = new NpgsqlCommand(""" + INSERT INTO inbox_messages (tenant_id, subscription_id, event_id) + VALUES (@tenant, @subscription, @event) + """, (NpgsqlConnection)publisher.Connection, (NpgsqlTransaction)publisher.Transaction!)) + { + insert.Parameters.AddWithValue("tenant", SchemaFixture.TenantB); + insert.Parameters.AddWithValue("subscription", First); + insert.Parameters.AddWithValue("event", marker); + (await insert.ExecuteNonQueryAsync()).Should().Be(1); + } + var envelope = Envelope(Event()); + + await services.GetRequiredService().PublishAsync(envelope); + + accessor.Current.Should().BeSameAs(caller); + publisher.HasActiveTransaction.Should().BeTrue(); + (await RowsAsync(database.AppConnectionString, envelope.Event.EventId)).Should().HaveCount(2, + "acceptance committed while the publisher's separate transaction stayed open"); + (await InboxCountAsync(database.AppConnectionString, SchemaFixture.TenantB, marker)).Should().Be(0); + await publisherFrame.FailAsync(); + (await InboxCountAsync(database.AppConnectionString, SchemaFixture.TenantB, marker)).Should().Be(0); + } + + [Fact] + public async Task Accepted_copies_obey_the_actual_application_role_tenant_and_organization_scope() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var services = Compose(source); + var bus = services.GetRequiredService(); + var a1 = Envelope(Event()); + var a2 = Envelope(Event(), organization: SchemaFixture.OrgA2); + var b1 = Envelope(Event(SchemaFixture.TenantB), organization: SchemaFixture.OrgB1); + await bus.PublishAsync(a1); + await bus.PublishAsync(a2); + await bus.PublishAsync(b1); + + (await VisibleCountAsync(database.AppConnectionString, SchemaFixture.TenantA, + SchemaFixture.OrgA1, a1.Event.EventId)).Should().Be(2); + (await VisibleCountAsync(database.AppConnectionString, SchemaFixture.TenantA, + SchemaFixture.OrgA1, a2.Event.EventId)).Should().Be(0); + (await VisibleCountAsync(database.AppConnectionString, SchemaFixture.TenantA, + SchemaFixture.OrgA1, b1.Event.EventId)).Should().Be(0); + (await VisibleCountAsync(database.AppConnectionString, SchemaFixture.TenantA, + SchemaFixture.OrgA2, a2.Event.EventId)).Should().Be(2); + (await VisibleCountAsync(database.AppConnectionString, SchemaFixture.TenantB, + SchemaFixture.OrgB1, b1.Event.EventId)).Should().Be(2); + } + + [Fact] + public async Task Blocked_publication_does_not_leak_ambient_scope_or_block_another_tenant_partition() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var services = Compose(source); + var bus = services.GetRequiredService(); + var accessor = services.GetRequiredService(); + var a1 = Envelope(Event()); + var b1 = Envelope(Event(SchemaFixture.TenantB), organization: SchemaFixture.OrgB1); + a1.PartitionKey.Should().NotBe(b1.PartitionKey); + await using var blocker = await OpenAsync(database.AppConnectionString); + await using var blockerTransaction = await blocker.BeginTransactionAsync(); + await SchemaQueries.SetTenantAsync(blocker, blockerTransaction, SchemaFixture.TenantA); + await SchemaQueries.SetSettingAsync(blocker, blockerTransaction, "app.organization_id", + SchemaFixture.OrgA1.ToString()); + await InsertMatchingFirstAsync(blocker, blockerTransaction, a1); + await using var pid = new NpgsqlCommand("SELECT pg_backend_pid()", blocker, blockerTransaction); + var blockerPid = (int)(await pid.ExecuteScalarAsync())!; + var blockedA = bus.PublishAsync(a1); + await WaitForBlockedAcceptanceAsync(database.AppConnectionString, blockerPid); + + try + { + await bus.PublishAsync(b1).WaitAsync(TimeSpan.FromSeconds(5)); + blockedA.IsCompleted.Should().BeFalse(); + (await VisibleCountAsync(database.AppConnectionString, SchemaFixture.TenantB, + SchemaFixture.OrgB1, b1.Event.EventId)).Should().Be(2); + (await VisibleCountAsync(database.AppConnectionString, SchemaFixture.TenantA, + SchemaFixture.OrgA1, b1.Event.EventId)).Should().Be(0); + (await RowsAsync(database.AppConnectionString, a1.Event.EventId)).Should().BeEmpty(); + accessor.Current.Should().BeNull("the publisher's ambient context was empty before both calls"); + } + finally + { + await blockerTransaction.CommitAsync(); + await blockedA.WaitAsync(TimeSpan.FromSeconds(10)); + } + + (await RowsAsync(database.AppConnectionString, a1.Event.EventId)).Should().HaveCount(2); + accessor.Current.Should().BeNull(); + } + + [Fact] + public async Task Cancellation_before_publication_writes_no_copies() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var services = Compose(source); + var envelope = Envelope(Event()); + using var cancellation = new CancellationTokenSource(); + cancellation.Cancel(); + + await FluentActions.Awaiting(() => services.GetRequiredService() + .PublishAsync(envelope, cancellation.Token)).Should().ThrowAsync(); + + (await RowsAsync(database.AppConnectionString, envelope.Event.EventId)).Should().BeEmpty(); + } + + [Fact] + public async Task Cancellation_while_the_second_copy_waits_rolls_back_the_first_copy() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var services = Compose(source); + var envelope = Envelope(Event()); + await using var blocker = await OpenAsync(database.AppConnectionString); + await using var blockerTransaction = await blocker.BeginTransactionAsync(); + await SchemaQueries.SetTenantAsync(blocker, blockerTransaction, SchemaFixture.TenantA); + await SchemaQueries.SetSettingAsync(blocker, blockerTransaction, "app.organization_id", + SchemaFixture.OrgA1.ToString()); + await using (var insert = new NpgsqlCommand(""" + INSERT INTO event_deliveries + (tenant_id, subscription_id, event_id, organization_id, occurred_at, + correlation_id, type, topic, partition_key, payload) + VALUES (@tenant, @subscription, @event, @organization, @occurred, + @trace, @type, @topic, @partition, @payload) + """, blocker, blockerTransaction)) + { + insert.Parameters.AddWithValue("tenant", envelope.Event.TenantId); + insert.Parameters.AddWithValue("subscription", Second); + insert.Parameters.AddWithValue("event", envelope.Event.EventId); + insert.Parameters.AddWithValue("organization", SchemaFixture.OrgA1); + insert.Parameters.AddWithValue("occurred", envelope.Event.OccurredAt); + insert.Parameters.AddWithValue("trace", Trace); + insert.Parameters.AddWithValue("type", envelope.Event.GetType().AssemblyQualifiedName!); + insert.Parameters.AddWithValue("topic", envelope.Topic); + insert.Parameters.AddWithValue("partition", envelope.PartitionKey); + insert.Parameters.Add(new NpgsqlParameter("payload", NpgsqlDbType.Jsonb) + { Value = ((IntegrationEventBase)envelope.Event).ToPayloadJson() }); + (await insert.ExecuteNonQueryAsync()).Should().Be(1); + } + await using var pid = new NpgsqlCommand("SELECT pg_backend_pid()", blocker, blockerTransaction); + var blockerPid = (int)(await pid.ExecuteScalarAsync())!; + using var cancellation = new CancellationTokenSource(); + var publish = services.GetRequiredService().PublishAsync(envelope, cancellation.Token); + + await WaitForBlockedAcceptanceAsync(database.AppConnectionString, blockerPid); + publish.IsCompleted.Should().BeFalse(); + cancellation.Cancel(); + await FluentActions.Awaiting(() => publish).Should().ThrowAsync(); + await blockerTransaction.RollbackAsync(); + + (await RowsAsync(database.AppConnectionString, envelope.Event.EventId)).Should().BeEmpty(); + } + + [Fact] + public async Task Dropped_commit_acknowledgement_leaves_durable_copies_and_republication_is_idempotent() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var proxy = new CommitResponseDroppingProxy(database.AppConnectionString); + await using var uncertainSource = ApplicationDataSource.Build(proxy.ConnectionString); + await using var uncertain = Compose(uncertainSource); + var envelope = Envelope(Event()); + + await FluentActions.Awaiting(() => uncertain.GetRequiredService().PublishAsync(envelope)) + .Should().ThrowAsync(); + proxy.DroppedCommitResponses.Should().Be(1); + var accepted = await RowsAsync(database.AppConnectionString, envelope.Event.EventId); + accepted.Should().HaveCount(2); + + await using var retrySource = ApplicationDataSource.Build(database.AppConnectionString); + await using var retry = Compose(retrySource); + await retry.GetRequiredService().PublishAsync(envelope); + (await RowsAsync(database.AppConnectionString, envelope.Event.EventId)).Should().BeEquivalentTo(accepted); + } + + [Fact] + public async Task Unknown_pending_and_retired_contracts_refuse_and_explicit_absence_is_validated() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + var fact = Event(); + var envelope = Envelope(fact); + + foreach (var disposition in new[] { "unknown", "pending", "retired" }) + { + await using var services = Compose(source, disposition); + await FluentActions.Awaiting(() => services.GetRequiredService().PublishAsync(envelope)) + .Should().ThrowAsync(); + (await RowsAsync(database.AppConnectionString, fact.EventId)).Should().BeEmpty(); + } + + await using var absent = Compose(source, "absent"); + await absent.GetRequiredService().PublishAsync(envelope); + (await RowsAsync(database.AppConnectionString, fact.EventId)).Should().BeEmpty(); + } + + private static ServiceProvider Compose(NpgsqlDataSource source, string disposition = "active", bool withAliases = false) + { + var services = new ServiceCollection(); + services.AddLogging(); + services.AddSingleton(source); + services.AddLearnStackObservabilityServices(); + services.AddScoped(provider => provider.GetRequiredService().Current + ?? UnresolvedTenantContext.Instance); + services.AddScoped(); + services.AddScoped(); + services.AddSingleton(); + if (disposition == "active") + { + services.AddScoped, ExplodingHandler>(); + services.AddScoped, OtherExplodingHandler>(); + } + var handlers = IntegrationEventHandlerRegistry.FromServiceDescriptors(services); + OutboxEventRegistration[] registrations = disposition switch + { + "active" => [OutboxEventRegistration.Required(typeof(ProofEvent), + [OutboxSubscriptionManifest.Active(First, typeof(ExplodingHandler)), + OutboxSubscriptionManifest.Active(Second, typeof(OtherExplodingHandler))], + withAliases ? [EarlierIdentity] : [])], + "pending" => [OutboxEventRegistration.Required(typeof(ProofEvent), + [OutboxSubscriptionManifest.Pending(First)])], + "absent" => [OutboxEventRegistration.IntentionallyUnsubscribed(typeof(ProofEvent))], + _ => [OutboxEventRegistration.IntentionallyUnsubscribed(typeof(OtherEvent))], + }; + if (withAliases) + registrations = [.. registrations, + OutboxEventRegistration.IntentionallyUnsubscribed(typeof(OtherEvent), OtherEarlierIdentity)]; + string[] retired = disposition == "retired" ? [typeof(ProofEvent).AssemblyQualifiedName!] + : withAliases ? [RetiredIdentity] : []; + services.AddSingleton(OutboxEventRegistry.Create(registrations, retired, handlers, + [typeof(ProofEvent).Assembly])); + services.AddSingleton(); + return services.BuildServiceProvider(validateScopes: true); + } + + private static ProofEvent Event(Guid? tenant = null) => new() + { + EventId = Guid.CreateVersion7(), + TenantId = tenant ?? SchemaFixture.TenantA, + OccurredAt = Occurred, + SubjectId = Guid.CreateVersion7(), + }; + + private static IntegrationEventEnvelope Envelope(ProofEvent fact, Guid? organization = null, + Guid? causation = null, UserId? actor = null, string trace = Trace) => + new(fact, trace, organization ?? SchemaFixture.OrgA1, causation, actor); + + private static async Task InsertCopyAsync(string connectionString, IntegrationEventEnvelope envelope, + string subscription, string? payload = null, string? storedType = null, + string? storedTopic = null, string? storedPartition = null) + { + await using var connection = await OpenAsync(connectionString); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await SchemaQueries.SetTenantAsync(connection, transaction, envelope.Event.TenantId); + await SchemaQueries.SetSettingAsync(connection, transaction, "app.organization_id", + envelope.OrganizationId?.ToString() ?? string.Empty); + await using var insert = new NpgsqlCommand(""" + INSERT INTO event_deliveries + (tenant_id, subscription_id, event_id, organization_id, occurred_at, + correlation_id, causation_id, actor_user_id, type, topic, partition_key, payload) + VALUES (@tenant, @subscription, @event, @organization, @occurred, + @correlation, @causation, @actor, @type, @topic, @partition, @payload) + """, connection, transaction); + insert.Parameters.AddWithValue("tenant", envelope.Event.TenantId); + insert.Parameters.AddWithValue("subscription", subscription); + insert.Parameters.AddWithValue("event", envelope.Event.EventId); + insert.Parameters.AddWithValue("organization", (object?)envelope.OrganizationId ?? DBNull.Value); + insert.Parameters.AddWithValue("occurred", envelope.Event.OccurredAt); + insert.Parameters.AddWithValue("correlation", envelope.CorrelationId); + insert.Parameters.AddWithValue("causation", (object?)envelope.CausationId ?? DBNull.Value); + insert.Parameters.AddWithValue("actor", (object?)envelope.ActorUserId?.Value ?? DBNull.Value); + insert.Parameters.AddWithValue("type", storedType ?? envelope.Event.GetType().AssemblyQualifiedName!); + insert.Parameters.AddWithValue("topic", storedTopic ?? envelope.Topic); + insert.Parameters.AddWithValue("partition", storedPartition ?? envelope.PartitionKey); + insert.Parameters.Add(new NpgsqlParameter("payload", NpgsqlDbType.Jsonb) + { Value = payload ?? ((IntegrationEventBase)envelope.Event).ToPayloadJson() }); + (await insert.ExecuteNonQueryAsync()).Should().Be(1); + await transaction.CommitAsync(); + } + + private static async Task InsertMatchingFirstAsync(NpgsqlConnection connection, + NpgsqlTransaction transaction, IntegrationEventEnvelope envelope) + { + await using var insert = new NpgsqlCommand(""" + INSERT INTO event_deliveries + (tenant_id, subscription_id, event_id, organization_id, occurred_at, + correlation_id, type, topic, partition_key, payload) + VALUES (@tenant, @subscription, @event, @organization, @occurred, + @trace, @type, @topic, @partition, @payload) + """, connection, transaction); + insert.Parameters.AddWithValue("tenant", envelope.Event.TenantId); + insert.Parameters.AddWithValue("subscription", First); + insert.Parameters.AddWithValue("event", envelope.Event.EventId); + insert.Parameters.AddWithValue("organization", envelope.OrganizationId!.Value); + insert.Parameters.AddWithValue("occurred", envelope.Event.OccurredAt); + insert.Parameters.AddWithValue("trace", envelope.CorrelationId); + insert.Parameters.AddWithValue("type", envelope.Event.GetType().AssemblyQualifiedName!); + insert.Parameters.AddWithValue("topic", envelope.Topic); + insert.Parameters.AddWithValue("partition", envelope.PartitionKey); + insert.Parameters.Add(new NpgsqlParameter("payload", NpgsqlDbType.Jsonb) + { Value = ((IntegrationEventBase)envelope.Event).ToPayloadJson() }); + (await insert.ExecuteNonQueryAsync()).Should().Be(1); + } + + private static async Task> RowsAsync(string connectionString, Guid eventId) + { + await using var connection = await OpenAsync(connectionString); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await SchemaQueries.SetTenantAsync(connection, transaction, SchemaFixture.TenantA); + await SchemaQueries.SetSettingAsync(connection, transaction, "app.organization_id", + SchemaFixture.OrgA1.ToString()); + await SchemaQueries.SetSettingAsync(connection, transaction, "app.scope", "tenant"); + await using var command = new NpgsqlCommand(""" + SELECT tenant_id, subscription_id, event_id, organization_id, attempts, + processed_at, last_error, available_after, locked_by, locked_until, + correlation_id, causation_id, actor_user_id, occurred_at, type, topic, + partition_key, payload::text, xmin::text + FROM event_deliveries WHERE event_id = @event ORDER BY subscription_id + """, connection, transaction); + command.Parameters.AddWithValue("event", eventId); + var rows = new List(); + await using var reader = await command.ExecuteReaderAsync(); + while (await reader.ReadAsync()) + rows.Add(new DeliveryRow(reader.GetGuid(0), reader.GetString(1), reader.GetGuid(2), + reader.IsDBNull(3) ? null : reader.GetGuid(3), reader.GetInt32(4), + reader.IsDBNull(5) ? null : reader.GetFieldValue(5), + reader.IsDBNull(6) ? null : reader.GetString(6), + reader.GetFieldValue(7), reader.IsDBNull(8) ? null : reader.GetString(8), + reader.IsDBNull(9) ? null : reader.GetFieldValue(9), + reader.GetString(10), reader.IsDBNull(11) ? null : reader.GetGuid(11), + reader.IsDBNull(12) ? null : reader.GetGuid(12), reader.GetFieldValue(13), + reader.GetString(14), reader.GetString(15), reader.GetString(16), reader.GetString(17), + reader.GetString(18))); + return rows; + } + + private static async Task InboxCountAsync(string connectionString, Guid tenantId, Guid eventId) + { + await using var connection = await OpenAsync(connectionString); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.SetTenantAsync(connection, transaction, tenantId); + await using var command = new NpgsqlCommand(""" + SELECT count(*) FROM inbox_messages WHERE tenant_id = @tenant AND event_id = @event + """, connection, transaction); + command.Parameters.AddWithValue("tenant", tenantId); + command.Parameters.AddWithValue("event", eventId); + return (long)(await command.ExecuteScalarAsync())!; + } + + private static async Task VisibleCountAsync(string connectionString, Guid tenantId, + Guid organizationId, Guid eventId) + { + await using var connection = await OpenAsync(connectionString); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await SchemaQueries.SetTenantAsync(connection, transaction, tenantId); + await SchemaQueries.SetSettingAsync(connection, transaction, "app.organization_id", + organizationId.ToString()); + await using var command = new NpgsqlCommand( + "SELECT count(*) FROM event_deliveries WHERE event_id = @event", connection, transaction); + command.Parameters.AddWithValue("event", eventId); + return (long)(await command.ExecuteScalarAsync())!; + } + + private static async Task WaitForBlockedAcceptanceAsync(string connectionString, int blockerPid) + { + using var deadline = new CancellationTokenSource(TimeSpan.FromSeconds(10)); + await using var connection = await OpenAsync(connectionString, deadline.Token); + while (true) + { + deadline.Token.ThrowIfCancellationRequested(); + await using var command = new NpgsqlCommand(""" + SELECT count(*) FROM pg_stat_activity + WHERE wait_event_type = 'Lock' AND @blocker = ANY(pg_blocking_pids(pid)) + AND query LIKE 'INSERT INTO event_deliveries%' + """, connection); + command.Parameters.AddWithValue("blocker", blockerPid); + if ((long)(await command.ExecuteScalarAsync(deadline.Token))! > 0) return; + await Task.Yield(); + } + } + + [IntegrationEventPartition(nameof(SubjectId))] + private sealed record ProofEvent : IntegrationEventBase + { + public required Guid SubjectId { get; init; } + public Guid? RelatedId { get; init; } + public override string Topic => "learnstack.proof.delivery"; + public override string PartitionKey => SubjectId.ToString(); + } + + [IntegrationEventPartition(nameof(SubjectId))] + private sealed record OtherEvent : IntegrationEventBase + { + public required Guid SubjectId { get; init; } + public override string Topic => "learnstack.proof.other"; + public override string PartitionKey => SubjectId.ToString(); + } + + private class ExplodingHandler : IIntegrationEventHandler + { + public static int Constructions; + public static int Invocations; + public ExplodingHandler() { Interlocked.Increment(ref Constructions); throw new InvalidOperationException("handler construction is forbidden"); } + public Task HandleAsync(ProofEvent @event, CancellationToken cancellationToken = default) + { + Interlocked.Increment(ref Invocations); + throw new InvalidOperationException("handler invocation is forbidden"); + } + } + + private sealed class OtherExplodingHandler : ExplodingHandler { } + + private static async Task OpenAsync(string connectionString, + CancellationToken cancellationToken = default) + { + var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(cancellationToken); + return connection; + } + + private sealed record DeliveryRow(Guid Tenant, string Subscription, Guid EventId, Guid? Organization, + int Attempts, DateTimeOffset? ProcessedAt, string? LastError, DateTimeOffset AvailableAfter, + string? LockedBy, DateTimeOffset? LockedUntil, string CorrelationId, Guid? CausationId, + Guid? ActorUserId, DateTimeOffset OccurredAt, string Type, string Topic, string Partition, + string Payload, string Version); +} diff --git a/backend/tests/LearnStack.Tests.Integration/Database/EventDeliveryMigrationTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/EventDeliveryMigrationTests.cs new file mode 100644 index 00000000..09a3dd47 --- /dev/null +++ b/backend/tests/LearnStack.Tests.Integration/Database/EventDeliveryMigrationTests.cs @@ -0,0 +1,948 @@ +using System.Data.Common; +using System.Reflection; +using FluentAssertions; +using LearnStack.Infrastructure.Messaging; +using LearnStack.Infrastructure.Persistence; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Migrations; +using Npgsql; +using Xunit; + +namespace LearnStack.Tests.Integration.Database; + +[Trait(RequiresDocker.Key, RequiresDocker.Value)] +[Collection(SharedSchema.Name)] +public sealed class EventDeliveryMigrationTests(SchemaFixture schema) +{ + private const string Predecessor = "20261012222654_align_outbox_recovery_index"; + private const string Subscription = "customization.initialize-content-type-default.v1"; + private const string Token = "12345678-1234-4234-8234-123456789abc"; + private const string InsertDelivery = """ + INSERT INTO event_deliveries + (tenant_id, subscription_id, event_id, organization_id, occurred_at, + correlation_id, causation_id, actor_user_id, type, topic, partition_key, payload) + VALUES (@tenant, @subscription, @event, @organization, now(), + '00-11111111111111111111111111111111-1111111111111111-01', + NULL, @actor, 'TenantCreatedIntegrationEventV1', + 'learnstack.tenancy.tenant', @partition, '{"version":1}') + """; + + private static readonly string[] AcceptanceColumns = + [ + "tenant_id", "subscription_id", "event_id", "organization_id", "occurred_at", + "correlation_id", "causation_id", "actor_user_id", "type", "topic", "partition_key", "payload", + ]; + + private static readonly string[] SchedulingColumns = + [ + "attempts", "last_error", "available_after", "locked_by", "locked_until", + "terminal_reason", "dead_lettered_at", "replay_count", "replayed_at", + ]; + + private static readonly string[] SchedulerReadColumns = + [ + "tenant_id", "subscription_id", "event_id", "organization_id", "occurred_at", + "type", "topic", "partition_key", "processed_at", "attempts", "last_error", + "available_after", "locked_by", "locked_until", "terminal_reason", "dead_lettered_at", + "replay_count", "replayed_at", + ]; + + [Fact] + public async Task Fresh_down_reapply_preserves_the_producer_and_restores_the_complete_schema() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var context = Platform(database); + context.Database.HasPendingModelChanges().Should().BeFalse(); + var before = await InventoryAsync(database.MigrationConnectionString); + before.Should().HaveCountGreaterThan(50, "the fingerprint must cover actual columns, constraints, indexes, policies and grants"); + + await using (var app = await PostgresFixture.OpenAsync(database.AppConnectionString)) + await using (var transaction = await app.BeginTransactionAsync()) + { + await SchemaQueries.AssertApplicationRoleAsync(app); + await AnnounceAsync(app, transaction, SchemaFixture.TenantA); + await SchemaQueries.ExecuteAsync(app, transaction, """ + INSERT INTO outbox_messages (tenant_id, correlation_id, type, topic, partition_key, payload) + VALUES (@tenant, 'producer-fact', 'TenantCreated', 'learnstack.tenancy.tenant', 'retained', '{}') + """, ("tenant", SchemaFixture.TenantA)); + await InsertAsync(app, transaction, SchemaFixture.TenantA, null, Guid.NewGuid()); + await transaction.CommitAsync(); + } + + await context.GetService().MigrateAsync(Predecessor); + await using (var owner = await PostgresFixture.OpenAsync(database.MigrationConnectionString)) + { + (await SchemaQueries.CountAsync(owner, "SELECT count(*) FROM pg_proc WHERE proname = 'fn_inbox_messages_immutable'")) + .Should().Be(0, "Down must remove the table-owned function as well as its triggers"); + (await SchemaQueries.CountAsync(owner, """ + SELECT count(*) FROM pg_class WHERE relnamespace = 'public'::regnamespace + AND relname IN ('inbox_messages','event_deliveries') + """)).Should().Be(0); + } + await using (var scheduler = await PostgresFixture.OpenAsync(database.OutboxConnectionString)) + { + (await SchemaQueries.CountAsync(scheduler, "SELECT count(*) FROM outbox_messages WHERE correlation_id = 'producer-fact'")) + .Should().Be(1); + } + + await context.Database.MigrateAsync(); + (await InventoryAsync(database.MigrationConnectionString)).Should().Equal(before); + await using var restored = await PostgresFixture.OpenAsync(database.AppConnectionString); + await using var restoredTransaction = await restored.BeginTransactionAsync(); + await AnnounceAsync(restored, restoredTransaction, SchemaFixture.TenantA); + await InsertAsync(restored, restoredTransaction, SchemaFixture.TenantA, null, Guid.NewGuid()); + } + + [Fact] + public async Task Canonical_document_ddl_produces_the_same_applied_inventory_as_the_migration() + { + await using var canonical = await DisposableSchemaDatabase.CreateAsync(schema.Postgres, applyMigrations: false); + await MigrationChains.ApplyTenancyAsync(canonical.MigrationConnectionString); + var document = await File.ReadAllTextAsync(Path.GetFullPath(Path.Combine(Path.GetDirectoryName(RepositoryPath.RolesScript())!, "../../../docs/standards/05-database.md"))); + var section = document[document.IndexOf("## Inbox\n", StringComparison.Ordinal)..]; + var start = section.IndexOf("```sql\n", StringComparison.Ordinal) + "```sql\n".Length; + var end = section.IndexOf("\n```", start, StringComparison.Ordinal); + start.Should().BeGreaterThan("```sql\n".Length); + end.Should().BeGreaterThan(start); + await using (var owner = await PostgresFixture.OpenAsync(canonical.MigrationConnectionString)) + { + await SchemaQueries.ExecuteAsync(owner, null, section[start..end]); + } + (await InventoryAsync(canonical.MigrationConnectionString)) + .Should().Equal(await InventoryAsync(schema.Postgres.MigrationConnectionString), + "the canonical DDL must build the actual shipped columns, checks, RLS, indexes and exact grants"); + } + + [Fact] + public async Task Exact_column_grants_exclude_envelope_reads_completion_forgery_and_purge() + { + await using var owner = await PostgresFixture.OpenAsync(schema.Postgres.MigrationConnectionString); + var ledgerColumns = await SchemaQueries.ReadStringsAsync(owner, """ + SELECT column_name FROM information_schema.columns + WHERE table_schema = 'public' AND table_name = 'event_deliveries' ORDER BY column_name + """); + ledgerColumns.Should().HaveCount(22).And.NotContain("metadata"); + foreach (var role in new[] { "learnstack_app", "learnstack_platform", "learnstack_outbox_admin", "PUBLIC" }) + { + foreach (var privilege in new[] { "SELECT", "INSERT", "UPDATE", "DELETE" }) + { + var expected = (role, privilege) switch + { + ("learnstack_app", "SELECT") => ledgerColumns, + ("learnstack_app", "INSERT") => AcceptanceColumns.ToList(), + ("learnstack_app", "UPDATE") => ["processed_at"], + ("learnstack_outbox_admin", "SELECT") => SchedulerReadColumns.ToList(), + ("learnstack_outbox_admin", "UPDATE") => SchedulingColumns.ToList(), + _ => [], + }; + (await GrantedColumnsAsync(owner, role, "event_deliveries", privilege)) + .Should().BeEquivalentTo(expected, $"{role} has an exact {privilege} column budget"); + var inboxExpected = (role, privilege) switch + { + ("learnstack_app", "SELECT") => new[] { "tenant_id", "subscription_id", "event_id", "recorded_at" }, + ("learnstack_app", "INSERT") => ["tenant_id", "subscription_id", "event_id"], + _ => [], + }; + (await GrantedColumnsAsync(owner, role, "inbox_messages", privilege)) + .Should().BeEquivalentTo(inboxExpected); + } + } + (await SchemaQueries.CountAsync(owner, """ + SELECT count(*) FROM information_schema.table_privileges + WHERE table_schema = 'public' AND table_name IN ('event_deliveries','inbox_messages') + AND grantee <> 'learnstack_migration' + AND NOT (grantee = 'learnstack_app' AND privilege_type = 'SELECT') + """)).Should().Be(0, "column grants must not be widened by table privileges"); + } + + [Fact] + public async Task Real_roles_accept_claim_complete_and_retain_inert_lease_provenance() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + var eventId = Guid.NewGuid(); + await using (var app = await PostgresFixture.OpenAsync(database.AppConnectionString)) + await using (var transaction = await app.BeginTransactionAsync()) + { + await SchemaQueries.AssertApplicationRoleAsync(app); + await AnnounceAsync(app, transaction, SchemaFixture.TenantA, SchemaFixture.OrgA1); + await InsertAsync(app, transaction, SchemaFixture.TenantA, SchemaFixture.OrgA1, eventId); + await transaction.CommitAsync(); + } + await using (var scheduler = await PostgresFixture.OpenAsync(database.OutboxConnectionString)) + { + await using var claim = new NpgsqlCommand(""" + WITH candidate AS ( + SELECT tenant_id, subscription_id, event_id FROM event_deliveries + WHERE processed_at IS NULL AND dead_lettered_at IS NULL + AND terminal_reason IS NULL AND attempts < 6 AND locked_by IS NULL + AND available_after <= clock_timestamp() + ORDER BY available_after, occurred_at, event_id, tenant_id, subscription_id + FOR UPDATE SKIP LOCKED LIMIT 1) + UPDATE event_deliveries d + SET locked_by = @token, locked_until = clock_timestamp() + interval '30 seconds', attempts = attempts + 1 + FROM candidate c WHERE d.tenant_id = c.tenant_id + AND d.subscription_id = c.subscription_id AND d.event_id = c.event_id + RETURNING d.event_id + """, (NpgsqlConnection)scheduler); + claim.Parameters.AddWithValue("token", Token); + (await claim.ExecuteScalarAsync()).Should().Be(eventId, "column UPDATE grants must permit actual row locking and claiming"); + } + await using (var app = await PostgresFixture.OpenAsync(database.AppConnectionString)) + await using (var transaction = await app.BeginTransactionAsync()) + { + await AnnounceAsync(app, transaction, SchemaFixture.TenantA, SchemaFixture.OrgA1); + await SchemaQueries.ExecuteAsync(app, transaction, """ + INSERT INTO inbox_messages (tenant_id, subscription_id, event_id) + VALUES (@tenant, @subscription, @event) + """, ("tenant", SchemaFixture.TenantA), ("subscription", Subscription), ("event", eventId)); + await using var complete = new NpgsqlCommand(""" + UPDATE event_deliveries SET processed_at = clock_timestamp() + WHERE tenant_id = @tenant AND subscription_id = @subscription AND event_id = @event + AND processed_at IS NULL AND dead_lettered_at IS NULL AND terminal_reason IS NULL + AND locked_by = @token AND locked_until > clock_timestamp() + """, (NpgsqlConnection)app, (NpgsqlTransaction)transaction); + complete.Parameters.AddWithValue("tenant", SchemaFixture.TenantA); + complete.Parameters.AddWithValue("subscription", Subscription); + complete.Parameters.AddWithValue("event", eventId); + complete.Parameters.AddWithValue("token", Token); + (await complete.ExecuteNonQueryAsync()).Should().Be(1); + (await SchemaQueries.CountAsync(app, """ + SELECT count(*) FROM inbox_messages WHERE recorded_at = transaction_timestamp() + """, transaction)).Should().Be(1); + await transaction.CommitAsync(); + } + await using var verify = await PostgresFixture.OpenAsync(database.OutboxConnectionString); + (await SchemaQueries.CountAsync(verify, $"SELECT count(*) FROM event_deliveries WHERE processed_at IS NOT NULL AND locked_by = '{Token}' AND locked_until IS NOT NULL")) + .Should().Be(1); + foreach (var predicate in new[] + { + "terminal_reason IS NULL AND attempts < 6 AND locked_by IS NULL", + "terminal_reason IS NULL AND (locked_by IS NOT NULL OR attempts >= 6)", + "terminal_reason IS NOT NULL", + }) + { + (await SchemaQueries.CountAsync(verify, $"SELECT count(*) FROM event_deliveries WHERE processed_at IS NULL AND dead_lettered_at IS NULL AND ({predicate})")) + .Should().Be(0, "completed rows with retained ownership must never reenter a scheduler path"); + } + } + + [Theory] + [InlineData(false, "SELECT payload FROM event_deliveries")] + [InlineData(false, "SELECT correlation_id FROM event_deliveries")] + [InlineData(false, "SELECT causation_id FROM event_deliveries")] + [InlineData(false, "SELECT actor_user_id FROM event_deliveries")] + [InlineData(false, "SELECT * FROM inbox_messages")] + [InlineData(false, "TRUNCATE inbox_messages")] + [InlineData(false, "TRUNCATE event_deliveries")] + [InlineData(false, "UPDATE event_deliveries SET processed_at = now()")] + [InlineData(false, "UPDATE event_deliveries SET topic = 'other'")] + [InlineData(false, "INSERT INTO event_deliveries (tenant_id) VALUES ('11111111-1111-7111-8111-111111111111')")] + [InlineData(false, "DELETE FROM event_deliveries")] + [InlineData(false, "INSERT INTO inbox_messages (tenant_id) VALUES ('11111111-1111-7111-8111-111111111111')")] + [InlineData(true, "UPDATE event_deliveries SET attempts = 0")] + [InlineData(true, "UPDATE event_deliveries SET locked_by = NULL, locked_until = NULL")] + [InlineData(true, "UPDATE event_deliveries SET replay_count = 0, replayed_at = NULL")] + [InlineData(true, "UPDATE event_deliveries SET payload = '{}'::jsonb")] + [InlineData(true, "UPDATE event_deliveries SET organization_id = NULL")] + [InlineData(true, "DELETE FROM event_deliveries")] + [InlineData(true, "INSERT INTO event_deliveries (attempts) VALUES (0)")] + [InlineData(true, "INSERT INTO inbox_messages (recorded_at) VALUES (now())")] + [InlineData(true, "UPDATE inbox_messages SET recorded_at = now()")] + [InlineData(true, "DELETE FROM inbox_messages")] + [InlineData(true, "TRUNCATE inbox_messages")] + [InlineData(true, "TRUNCATE event_deliveries")] + public async Task Actual_sql_refuses_privileges_outside_each_role_budget(bool application, string sql) + { + await using var connection = await PostgresFixture.OpenAsync(application + ? schema.Postgres.AppConnectionString : schema.Postgres.OutboxConnectionString); + await using var transaction = await connection.BeginTransactionAsync(); + if (application) + { + await SchemaQueries.AssertApplicationRoleAsync(connection); + await AnnounceAsync(connection, transaction, SchemaFixture.TenantA); + } + var execute = async () => await SchemaQueries.ExecuteAsync(connection, transaction, sql); + (await execute.Should().ThrowAsync()).Which.SqlState.Should().Be(PostgresErrorCodes.InsufficientPrivilege); + } + + [Fact] + public async Task Composite_identity_deduplicates_each_subscription_and_tenant_independently() + { + await using var connection = await PostgresFixture.OpenAsync(schema.Postgres.AppConnectionString); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + var eventId = Guid.NewGuid(); + foreach (var tenant in new[] { SchemaFixture.TenantA, SchemaFixture.TenantB }) + { + await AnnounceAsync(connection, transaction, tenant); + foreach (var subscription in new[] { Subscription, "customization.initialize-taxonomy-default.v1" }) + { + await InsertAsync(connection, transaction, tenant, null, eventId, subscription); + await SchemaQueries.ExecuteAsync(connection, transaction, """ + INSERT INTO inbox_messages (tenant_id, subscription_id, event_id) + VALUES (@tenant, @subscription, @event) + """, ("tenant", tenant), ("subscription", subscription), ("event", eventId)); + } + (await CountEventAsync(connection, transaction, "event_deliveries", eventId)).Should().Be(2); + (await CountEventAsync(connection, transaction, "inbox_messages", eventId)).Should().Be(2); + } + await transaction.SaveAsync("duplicate"); + var duplicate = async () => await InsertAsync(connection, transaction, SchemaFixture.TenantB, null, eventId); + (await duplicate.Should().ThrowAsync()).Which.ConstraintName.Should().Be("pk_event_deliveries"); + await transaction.RollbackAsync("duplicate"); + var duplicateMarker = async () => await SchemaQueries.ExecuteAsync(connection, transaction, """ + INSERT INTO inbox_messages (tenant_id, subscription_id, event_id) + VALUES (@tenant, @subscription, @event) + """, ("tenant", SchemaFixture.TenantB), ("subscription", Subscription), ("event", eventId)); + (await duplicateMarker.Should().ThrowAsync()).Which.ConstraintName.Should().Be("pk_inbox_messages"); + } + + [Theory] + [InlineData("customization.initialize-content-type-default.v1", true)] + [InlineData("module1.effect-2.v12", true)] + [InlineData("Module.effect.v1", false)] + [InlineData("module.éffect.v1", false)] + [InlineData("module.effect.v0", false)] + [InlineData("module.effect.v01", false)] + [InlineData("module.effect.other.v1", false)] + [InlineData("module.effect.v1\n", false)] + [InlineData("module.effect.v1 ", false)] + public async Task Subscription_grammar_has_ascii_accepting_and_refusing_controls(string subscription, bool valid) + { + await using var app = await PostgresFixture.OpenAsync(schema.Postgres.AppConnectionString); + await using var transaction = await app.BeginTransactionAsync(); + await AnnounceAsync(app, transaction, SchemaFixture.TenantA); + foreach (var table in new[] { "event_deliveries", "inbox_messages" }) + { + await transaction.SaveAsync("grammar"); + var insert = async () => + { + if (table == "event_deliveries") + { + await InsertAsync(app, transaction, SchemaFixture.TenantA, null, Guid.NewGuid(), subscription); + } + else + { + await SchemaQueries.ExecuteAsync(app, transaction, """ + INSERT INTO inbox_messages (tenant_id, subscription_id, event_id) + VALUES (@tenant, @subscription, @event) + """, ("tenant", SchemaFixture.TenantA), ("subscription", subscription), ("event", Guid.NewGuid())); + } + }; + if (valid) await insert.Should().NotThrowAsync(); + else (await insert.Should().ThrowAsync()).Which.ConstraintName.Should().Be($"ck_{table}_subscription"); + await transaction.RollbackAsync("grammar"); + } + } + + [Theory] + [InlineData("attempts = -1", "ck_event_deliveries_attempts")] + [InlineData("attempts = 7", null)] + [InlineData("last_error = repeat('é', 128)", null)] + [InlineData("last_error = repeat('é', 129)", "ck_event_deliveries_last_error_size")] + [InlineData("locked_by = 'worker'", "ck_event_deliveries_lease")] + [InlineData("locked_by = 'worker', locked_until = now()", "ck_event_deliveries_lease")] + [InlineData("locked_by = '12345678-1234-4234-8234-123456789abc', locked_until = now()", null)] + [InlineData("locked_by = '12345678-1234-4234-8234-123456789ABC', locked_until = now()", "ck_event_deliveries_lease")] + [InlineData("locked_by = '00000000-0000-0000-0000-000000000000', locked_until = now()", "ck_event_deliveries_lease")] + [InlineData("locked_until = now()", "ck_event_deliveries_lease")] + [InlineData("terminal_reason = 'unknown_type'", null)] + [InlineData("terminal_reason = 'retired_type'", null)] + [InlineData("terminal_reason = 'invalid_envelope'", null)] + [InlineData("terminal_reason = 'invalid_subscription'", null)] + [InlineData("terminal_reason = 'retry_exhausted'", null)] + [InlineData("terminal_reason = 'undeclared'", "ck_event_deliveries_terminal_reason")] + [InlineData("dead_lettered_at = now()", "ck_event_deliveries_state")] + [InlineData("dead_lettered_at = now(), terminal_reason = 'retry_exhausted'", null)] + [InlineData("processed_at = now(), terminal_reason = 'retry_exhausted'", "ck_event_deliveries_state")] + [InlineData("replay_count = -1", "ck_event_deliveries_replay")] + [InlineData("replay_count = 1", "ck_event_deliveries_replay")] + [InlineData("replayed_at = now()", "ck_event_deliveries_replay")] + [InlineData("replay_count = 1, replayed_at = now()", null)] + public async Task Ledger_state_checks_preserve_valid_facts_and_refuse_invalid_combinations(string assignment, string? constraint) + { + // CHECK semantics are independent of the caller's privileges. Owner setup + // keeps the row and mutation in one rolled-back transaction; role budgets + // and actual permitted transitions are separately proved above. + await using var owner = await PostgresFixture.OpenAsync(schema.Postgres.MigrationConnectionString); + await using var transaction = await owner.BeginTransactionAsync(); + await AnnounceAsync(owner, transaction, SchemaFixture.TenantA); + var eventId = Guid.NewGuid(); + await InsertAsync(owner, transaction, SchemaFixture.TenantA, null, eventId); + var update = async () => + { + await using var command = new NpgsqlCommand($"UPDATE event_deliveries SET {assignment} WHERE event_id = @event", + (NpgsqlConnection)owner, (NpgsqlTransaction)transaction); + command.Parameters.AddWithValue("event", eventId); + (await command.ExecuteNonQueryAsync()).Should().Be(1); + }; + if (constraint is null) await update.Should().NotThrowAsync(); + else (await update.Should().ThrowAsync()).Which.ConstraintName.Should().Be(constraint); + } + + [Fact] + public async Task Organization_immutability_is_enforced_even_when_the_owner_has_column_update_privilege() + { + await using var owner = await PostgresFixture.OpenAsync(schema.Postgres.MigrationConnectionString); + await using var transaction = await owner.BeginTransactionAsync(); + await AnnounceAsync(owner, transaction, SchemaFixture.TenantA, SchemaFixture.OrgA1); + var eventId = Guid.NewGuid(); + await InsertAsync(owner, transaction, SchemaFixture.TenantA, SchemaFixture.OrgA1, eventId); + var move = async () => await SchemaQueries.ExecuteAsync(owner, transaction, + "UPDATE event_deliveries SET organization_id = @organization WHERE event_id = @event", + ("organization", SchemaFixture.OrgA2), ("event", eventId)); + var error = (await move.Should().ThrowAsync()).Which; + error.SqlState.Should().Be(PostgresErrorCodes.CheckViolation); + error.MessageText.Should().Contain("organization_id is immutable after insert"); + } + + [Theory] + [InlineData("UPDATE inbox_messages SET recorded_at = recorded_at + interval '1 second'", "tg_inbox_messages_immutable")] + [InlineData("DELETE FROM inbox_messages", "tg_inbox_messages_immutable")] + [InlineData("TRUNCATE inbox_messages", "tg_inbox_messages_no_truncate")] + public async Task Immutable_inbox_guard_refuses_the_owner_and_a_removed_guard_is_a_failing_control(string sql, string trigger) + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var owner = await PostgresFixture.OpenAsync(database.MigrationConnectionString); + await using var transaction = await owner.BeginTransactionAsync(); + await AnnounceAsync(owner, transaction, SchemaFixture.TenantA); + await SchemaQueries.ExecuteAsync(owner, transaction, """ + INSERT INTO inbox_messages (tenant_id, subscription_id, event_id) + VALUES (@tenant, @subscription, @event) + """, ("tenant", SchemaFixture.TenantA), ("subscription", Subscription), ("event", Guid.NewGuid())); + await transaction.SaveAsync("guard"); + var mutation = async () => await SchemaQueries.ExecuteAsync(owner, transaction, sql); + var error = (await mutation.Should().ThrowAsync()).Which; + error.SqlState.Should().Be(PostgresErrorCodes.InsufficientPrivilege); + error.MessageText.Should().Contain("inbox_messages is immutable"); + await transaction.RollbackAsync("guard"); + // Remove only the owned trigger in a disposable database. The same SQL + // must now reach the seeded row, proving that the guard above mattered. + await SchemaQueries.ExecuteAsync(owner, transaction, $"DROP TRIGGER {trigger} ON inbox_messages"); + await using var dirty = new NpgsqlCommand(sql, (NpgsqlConnection)owner, (NpgsqlTransaction)transaction); + (await dirty.ExecuteNonQueryAsync()).Should().Be(sql.StartsWith("TRUNCATE", StringComparison.Ordinal) ? -1 : 1); + } + + [Theory] + [InlineData("tenant", "00000000-0000-0000-0000-000000000000")] + [InlineData("tenant", "00000000-0000-7000-8000-000000000002")] + [InlineData("event", "00000000-0000-0000-0000-000000000000")] + [InlineData("organization", "00000000-0000-0000-0000-000000000000")] + [InlineData("actor", "00000000-0000-0000-0000-000000000000")] + [InlineData("causation", "00000000-0000-0000-0000-000000000000")] + public async Task Application_acceptance_refuses_invalid_identity_without_repair(string member, string invalid) + { + await using var app = await PostgresFixture.OpenAsync(schema.Postgres.AppConnectionString); + await using var transaction = await app.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(app); + var tenant = member == "tenant" ? Guid.Parse(invalid) : SchemaFixture.TenantA; + var eventId = member == "event" ? Guid.Parse(invalid) : Guid.NewGuid(); + Guid? organization = member == "organization" ? Guid.Parse(invalid) : null; + await AnnounceAsync(app, transaction, tenant, organization); + await transaction.SaveAsync("identity"); + var insert = async () => await SchemaQueries.ExecuteAsync(app, transaction, + InsertDelivery.Replace("NULL, @actor", "@causation, @actor", StringComparison.Ordinal), + ("tenant", tenant), ("subscription", Subscription), ("event", eventId), + ("organization", (object?)organization ?? DBNull.Value), ("partition", tenant.ToString()), + ("actor", member == "actor" ? Guid.Parse(invalid) : SchemaFixture.Actor), + ("causation", member == "causation" ? Guid.Parse(invalid) : DBNull.Value)); + (await insert.Should().ThrowAsync()).Which.ConstraintName.Should().Be("ck_event_deliveries_identity"); + await transaction.RollbackAsync("identity"); + if (member is "tenant" or "event") + { + var marker = async () => await SchemaQueries.ExecuteAsync(app, transaction, """ + INSERT INTO inbox_messages (tenant_id, subscription_id, event_id) + VALUES (@tenant, @subscription, @event) + """, ("tenant", tenant), ("subscription", Subscription), ("event", eventId)); + (await marker.Should().ThrowAsync()).Which.ConstraintName.Should().Be("ck_inbox_messages_identity"); + } + } + + [Fact] + public async Task Scheduling_indexes_cover_eligibility_and_total_order_including_tied_event_ids() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + var tiedEvent = Guid.Parse("11111111-0000-4000-8000-000000000001"); + var taxonomy = "customization.initialize-taxonomy-default.v1"; + await using (var owner = await PostgresFixture.OpenAsync(database.MigrationConnectionString)) + await using (var transaction = await owner.BeginTransactionAsync()) + { + // Equal availability, occurrence and event identities force selection + // to use the final tenant/subscription tie-breakers. + foreach (var (tenant, subscription) in new[] + { + (SchemaFixture.TenantB, Subscription), (SchemaFixture.TenantA, taxonomy), (SchemaFixture.TenantA, Subscription), + }) + { + await AnnounceAsync(owner, transaction, tenant); + await InsertAsync(owner, transaction, tenant, null, tiedEvent, subscription); + await SchemaQueries.ExecuteAsync(owner, transaction, """ + UPDATE event_deliveries SET available_after = '2026-01-01', occurred_at = '2026-01-01' + WHERE tenant_id = @tenant AND subscription_id = @subscription AND event_id = @event + """, ("tenant", tenant), ("subscription", subscription), ("event", tiedEvent)); + } + await AnnounceAsync(owner, transaction, SchemaFixture.TenantA); + foreach (var assignment in new[] + { + "available_after = now() + interval '1 day'", + $"locked_by = '{Token}', locked_until = now() + interval '1 day'", + $"locked_by = '{Token}', locked_until = now() - interval '1 day'", + "attempts = 6", + "terminal_reason = 'invalid_subscription'", + "terminal_reason = 'retry_exhausted', dead_lettered_at = now()", + $"processed_at = now(), locked_by = '{Token}', locked_until = now() - interval '1 day', attempts = 6", + }) + { + var eventId = Guid.NewGuid(); + await InsertAsync(owner, transaction, SchemaFixture.TenantA, null, eventId); + await SchemaQueries.ExecuteAsync(owner, transaction, $"UPDATE event_deliveries SET {assignment} WHERE event_id = @event", ("event", eventId)); + } + await transaction.CommitAsync(); + } + await using var scheduler = await PostgresFixture.OpenAsync(database.OutboxConnectionString); + await using var queryTransaction = await scheduler.BeginTransactionAsync(); + // This proves each partial index can satisfy the production candidate query/order; + // disabling sequential scans makes no claim about production cost estimates. + await SchemaQueries.ExecuteAsync(scheduler, queryTransaction, "SET LOCAL enable_seqscan = off"); + var indexQueries = new[] + { + ("pending", "Normal", 3), + ("recovery", "Recovery", 2), + ("terminal", "Terminal", 1), + }; + foreach (var (suffix, predicate, expected) in indexQueries) + { + var rows = await SchemaQueries.ReadStringsAsync(scheduler, $""" + SELECT event_id::text || ':' || tenant_id::text || ':' || subscription_id FROM event_deliveries + WHERE {DeliveryPredicate(predicate)} + ORDER BY available_after, occurred_at, event_id, tenant_id, subscription_id + """, queryTransaction); + rows.Should().HaveCount(expected); + if (suffix == "pending") + { + rows.Should().Equal( + $"{tiedEvent}:{SchemaFixture.TenantA}:{Subscription}", + $"{tiedEvent}:{SchemaFixture.TenantA}:{taxonomy}", + $"{tiedEvent}:{SchemaFixture.TenantB}:{Subscription}"); + } + await using var selection = DeliverySelection(scheduler, queryTransaction, predicate); + selection.CommandText = "EXPLAIN (FORMAT JSON) " + selection.CommandText; + var plan = (string)(await selection.ExecuteScalarAsync())!; + plan.Should().Contain($"\"Index Name\": \"ix_event_deliveries_{suffix}\"") + .And.NotContain("\"Node Type\": \"Sort\"") + .And.NotContain("\"Node Type\": \"Incremental Sort\""); + } + } + + private static string DeliveryPredicate(string name) => + typeof(DeliveryDispatchStore).GetField(name, BindingFlags.Static | BindingFlags.NonPublic)?.GetRawConstantValue() as string + ?? throw new InvalidOperationException("The production delivery eligibility predicate is missing: " + name); + + private static NpgsqlCommand DeliverySelection(DbConnection scheduler, DbTransaction transaction, string predicate) + { + // Read the actual predicates, including time/lease and manifest/partition + // admission, so a production change cannot leave an independently copied + // WHERE clause passing. Locking and LIMIT match AcquireAsync's candidate. + var command = new NpgsqlCommand($""" + SELECT tenant_id, subscription_id, event_id FROM event_deliveries + WHERE {DeliveryPredicate(predicate)}{DeliveryPredicate("Admitted")} + ORDER BY available_after, occurred_at, event_id, tenant_id, subscription_id + FOR UPDATE SKIP LOCKED LIMIT 1 + """, (NpgsqlConnection)scheduler, (NpgsqlTransaction)transaction); + command.Parameters.AddWithValue("held_types", Array.Empty()); + command.Parameters.AddWithValue("held_subscriptions", Array.Empty()); + command.Parameters.AddWithValue("blocked", Array.Empty()); + return command; + } + + [Fact] + public async Task Scheduler_status_grants_allow_terminalization_and_targeted_replay_without_retention_dependencies() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + var eventId = Guid.NewGuid(); + await using (var app = await PostgresFixture.OpenAsync(database.AppConnectionString)) + await using (var transaction = await app.BeginTransactionAsync()) + { + await SchemaQueries.AssertApplicationRoleAsync(app); + await AnnounceAsync(app, transaction, SchemaFixture.TenantA); + await InsertAsync(app, transaction, SchemaFixture.TenantA, null, eventId); + await SchemaQueries.ExecuteAsync(app, transaction, """ + INSERT INTO outbox_messages (id, tenant_id, correlation_id, type, topic, partition_key, payload) + VALUES (@event, @tenant, 'retention-control', 'TenantCreated', 'learnstack.tenancy.tenant', 'retained', '{}'); + INSERT INTO inbox_messages (tenant_id, subscription_id, event_id) + VALUES (@tenant, @subscription, @marker) + """, ("event", eventId), ("tenant", SchemaFixture.TenantA), ("subscription", Subscription), ("marker", Guid.NewGuid())); + await transaction.CommitAsync(); + } + await using (var scheduler = await PostgresFixture.OpenAsync(database.OutboxConnectionString)) + { + await using var status = new NpgsqlCommand(""" + UPDATE event_deliveries SET attempts = 6, last_error = 'execution:handler:failed', + terminal_reason = 'retry_exhausted', dead_lettered_at = clock_timestamp(), + locked_by = NULL, locked_until = NULL + WHERE tenant_id = @tenant AND subscription_id = @subscription AND event_id = @event + AND processed_at IS NULL + RETURNING event_id + """, (NpgsqlConnection)scheduler); + status.Parameters.AddWithValue("tenant", SchemaFixture.TenantA); + status.Parameters.AddWithValue("subscription", Subscription); + status.Parameters.AddWithValue("event", eventId); + (await status.ExecuteScalarAsync()).Should().Be(eventId); + status.CommandText = """ + UPDATE event_deliveries SET attempts = 0, last_error = NULL, + terminal_reason = NULL, dead_lettered_at = NULL, + available_after = clock_timestamp(), locked_by = NULL, locked_until = NULL, + replay_count = replay_count + 1, replayed_at = clock_timestamp() + WHERE tenant_id = @tenant AND subscription_id = @subscription AND event_id = @event + AND processed_at IS NULL AND dead_lettered_at IS NOT NULL AND locked_by IS NULL + RETURNING replay_count + """; + (await status.ExecuteScalarAsync()).Should().Be(1, "all nine granted status columns support terminal and replay transitions"); + } + await using (var platform = await PostgresFixture.OpenAsync(database.PlatformConnectionString)) + { + await SchemaQueries.ExecuteAsync(platform, null, "DELETE FROM outbox_messages WHERE id = @event", ("event", eventId)); + } + await using var verify = await PostgresFixture.OpenAsync(database.AppConnectionString); + await using var read = await verify.BeginTransactionAsync(); + await AnnounceAsync(verify, read, SchemaFixture.TenantA); + (await SchemaQueries.CountAsync(verify, "SELECT count(*) FROM event_deliveries WHERE replay_count = 1 AND payload = '{\"version\":1}'::jsonb", read)) + .Should().Be(1, "producer retention must not cascade into retained subscription envelopes"); + (await SchemaQueries.CountAsync(verify, "SELECT count(*) FROM inbox_messages", read)) + .Should().Be(1, "replay and producer retention never reset completed deduplication history"); + (await SchemaQueries.CountAsync(verify, "SELECT count(*) FROM outbox_messages", read)).Should().Be(0); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Application_organization_reads_widen_only_with_the_tenant_hatch(bool tenantHatch) + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + var canaries = await SeedOrganizationCanariesAsync(database); + var expected = new List { canaries["tenant"].EventId.ToString(), canaries["own"].EventId.ToString() }; + if (tenantHatch) expected.Add(canaries["sibling"].EventId.ToString()); + (await ReadOrganizationCanariesAsync(database, tenantHatch)).Should().BeEquivalentTo(expected, + "an A1 session reads its own and tenant-wide envelopes; only the tenant hatch adds A2, never tenant B"); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Application_acceptance_requires_exact_organization_even_with_the_tenant_read_hatch(bool tenantHatch) + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await SeedOrganizationCanariesAsync(database); + (string Name, Guid? SessionOrg, Guid TargetTenant, Guid? TargetOrg, bool Allowed)[] cases = + [ + ("tenant session / tenant row", null, SchemaFixture.TenantA, null, true), + ("tenant session / A1 row", null, SchemaFixture.TenantA, SchemaFixture.OrgA1, false), + ("tenant session / A2 row", null, SchemaFixture.TenantA, SchemaFixture.OrgA2, false), + ("tenant session / foreign org", null, SchemaFixture.TenantA, SchemaFixture.OrgB1, false), + ("A1 session / A1 row", SchemaFixture.OrgA1, SchemaFixture.TenantA, SchemaFixture.OrgA1, true), + ("A1 session / tenant row", SchemaFixture.OrgA1, SchemaFixture.TenantA, null, false), + ("A1 session / A2 row", SchemaFixture.OrgA1, SchemaFixture.TenantA, SchemaFixture.OrgA2, false), + ("A1 session / foreign org", SchemaFixture.OrgA1, SchemaFixture.TenantA, SchemaFixture.OrgB1, false), + ("A1 session / foreign tenant", SchemaFixture.OrgA1, SchemaFixture.TenantB, SchemaFixture.OrgA1, false), + ("tenant session / foreign tenant", null, SchemaFixture.TenantB, null, false), + ]; + await using var app = await PostgresFixture.OpenAsync(database.AppConnectionString); + await SchemaQueries.AssertApplicationRoleAsync(app); + await using var transaction = await app.BeginTransactionAsync(); + foreach (var testCase in cases) + { + await AnnounceAsync(app, transaction, SchemaFixture.TenantA, testCase.SessionOrg); + await SchemaQueries.SetSettingAsync(app, transaction, "app.scope", tenantHatch ? "tenant" : ""); + await transaction.SaveAsync("acceptance"); + var eventId = Guid.NewGuid(); + var insert = async () => await InsertAsync(app, transaction, testCase.TargetTenant, testCase.TargetOrg, eventId); + if (testCase.Allowed) + { + await insert.Should().NotThrowAsync(testCase.Name); + (await CountEventAsync(app, transaction, "event_deliveries", eventId)).Should().Be(1, testCase.Name); + } + else + { + var error = (await insert.Should().ThrowAsync(testCase.Name)).Which; + error.SqlState.Should().Be(PostgresErrorCodes.InsufficientPrivilege); + error.MessageText.Should().Contain("row-level security", "column permissions alone cannot prove scope enforcement"); + } + await transaction.RollbackAsync("acceptance"); + } + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Fenced_application_completion_writes_only_its_exact_organization_even_with_the_read_hatch(bool tenantHatch) + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + var canaries = await SeedOrganizationCanariesAsync(database); + foreach (var name in new[] { "tenant", "sibling", "foreign-tenant", "foreign-org" }) + { + (await CompleteOrganizationCanaryAsync(database, canaries[name], tenantHatch)).Should().Be(0, + $"A1 must not complete {name}, even with its correct token and live lease"); + } + (await CompleteOrganizationCanaryAsync(database, canaries["own"], tenantHatch)).Should().Be(1, + "the same fenced statement and granted completion column must succeed for A1's own live delivery"); + await using var scheduler = await PostgresFixture.OpenAsync(database.OutboxConnectionString); + (await SchemaQueries.CountAsync(scheduler, "SELECT count(*) FROM event_deliveries WHERE processed_at IS NOT NULL")) + .Should().Be(1); + } + + [Fact] + public async Task Organization_read_proof_detects_an_isolated_tenant_only_using_policy_mutant() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + var canaries = await SeedOrganizationCanariesAsync(database); + (await ReadOrganizationCanariesAsync(database, tenantHatch: false)).Should().BeEquivalentTo( + [canaries["tenant"].EventId.ToString(), canaries["own"].EventId.ToString()]); + await using (var owner = await PostgresFixture.OpenAsync(database.MigrationConnectionString)) + { + // Change only the disposable database's READ predicate; retain tenant + // isolation, WITH CHECK and both restrictive write guards. + await SchemaQueries.ExecuteAsync(owner, null, """ + ALTER POLICY event_deliveries_isolation ON event_deliveries + USING (tenant_id = NULLIF(current_setting('app.tenant_id', true), '')::uuid) + """); + } + (await ReadOrganizationCanariesAsync(database, tenantHatch: false)).Should().BeEquivalentTo( + [canaries["tenant"].EventId.ToString(), canaries["own"].EventId.ToString(), canaries["sibling"].EventId.ToString()], + "the planted missing organization READ term must expose A2 while tenant B stays hidden"); + } + + [Fact] + public async Task Acceptance_scope_proof_detects_an_isolated_read_hatch_in_with_check_mutant() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await SeedOrganizationCanariesAsync(database); + await using (var app = await PostgresFixture.OpenAsync(database.AppConnectionString)) + await using (var transaction = await app.BeginTransactionAsync()) + { + await SchemaQueries.AssertApplicationRoleAsync(app); + await AnnounceAsync(app, transaction, SchemaFixture.TenantA, SchemaFixture.OrgA1); + await SchemaQueries.SetSettingAsync(app, transaction, "app.scope", "tenant"); + var insert = async () => await InsertAsync(app, transaction, SchemaFixture.TenantA, null, Guid.NewGuid()); + var error = (await insert.Should().ThrowAsync()).Which; + error.SqlState.Should().Be(PostgresErrorCodes.InsufficientPrivilege); + error.MessageText.Should().Contain("row-level security"); + } + await using (var owner = await PostgresFixture.OpenAsync(database.MigrationConnectionString)) + { + // The historical failure shape copies the permissive READ hatch into + // INSERT's WITH CHECK. Restrictive UPDATE/DELETE guards cannot fix INSERT. + await SchemaQueries.ExecuteAsync(owner, null, """ + ALTER POLICY event_deliveries_isolation ON event_deliveries + WITH CHECK ( + tenant_id = NULLIF(current_setting('app.tenant_id', true), '')::uuid + AND (organization_id IS NULL + OR organization_id = NULLIF(current_setting('app.organization_id', true), '')::uuid + OR current_setting('app.scope', true) = 'tenant')) + """); + } + await using var dirtyApp = await PostgresFixture.OpenAsync(database.AppConnectionString); + await SchemaQueries.AssertApplicationRoleAsync(dirtyApp); + await using var dirtyTransaction = await dirtyApp.BeginTransactionAsync(); + await AnnounceAsync(dirtyApp, dirtyTransaction, SchemaFixture.TenantA, SchemaFixture.OrgA1); + await SchemaQueries.SetSettingAsync(dirtyApp, dirtyTransaction, "app.scope", "tenant"); + var dirtyEvent = Guid.NewGuid(); + await InsertAsync(dirtyApp, dirtyTransaction, SchemaFixture.TenantA, null, dirtyEvent); + (await CountEventAsync(dirtyApp, dirtyTransaction, "event_deliveries", dirtyEvent)).Should().Be(1, + "a weakened WITH CHECK must make the forbidden A1-to-tenant-wide acceptance observable"); + } + + [Theory] + [InlineData("tenant")] + [InlineData("sibling")] + public async Task Restrictive_update_guard_stops_targeting_before_the_remaining_with_check_backstop(string target) + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + var canaries = await SeedOrganizationCanariesAsync(database); + (await CompleteOrganizationCanaryAsync(database, canaries[target], tenantHatch: true)).Should().Be(0, + "the tenant READ hatch must not admit a readable foreign-organization row as an UPDATE target"); + await using (var owner = await PostgresFixture.OpenAsync(database.MigrationConnectionString)) + { + // Drop only the restrictive UPDATE guard in this disposable database. + // WITH CHECK remains exact, so the mutant targets the row but still + // cannot commit it: its observable result changes from zero to 42501. + await SchemaQueries.ExecuteAsync(owner, null, + "DROP POLICY event_deliveries_org_write_guard ON event_deliveries"); + } + var dirtyCompletion = async () => await CompleteOrganizationCanaryAsync(database, canaries[target], tenantHatch: true); + var error = (await dirtyCompletion.Should().ThrowAsync()).Which; + error.SqlState.Should().Be(PostgresErrorCodes.InsufficientPrivilege); + error.MessageText.Should().Contain("row-level security"); + (await CompleteOrganizationCanaryAsync(database, canaries["own"], tenantHatch: true)).Should().Be(1, + "the unchanged WITH CHECK still accepts A1's own leased delivery"); + await using var scheduler = await PostgresFixture.OpenAsync(database.OutboxConnectionString); + (await SchemaQueries.CountAsync(scheduler, "SELECT count(*) FROM event_deliveries WHERE processed_at IS NOT NULL")) + .Should().Be(1, "the removed guard does not bypass the remaining exact WITH CHECK"); + } + + private sealed record OrganizationCanary(Guid TenantId, Guid? OrganizationId, Guid EventId, string LeaseToken); + + private static async Task> SeedOrganizationCanariesAsync(DisposableSchemaDatabase database) + { + var canaries = new Dictionary(StringComparer.Ordinal) + { + ["tenant"] = new(SchemaFixture.TenantA, null, Guid.NewGuid(), Guid.NewGuid().ToString()), + ["own"] = new(SchemaFixture.TenantA, SchemaFixture.OrgA1, Guid.NewGuid(), Guid.NewGuid().ToString()), + ["sibling"] = new(SchemaFixture.TenantA, SchemaFixture.OrgA2, Guid.NewGuid(), Guid.NewGuid().ToString()), + ["foreign-tenant"] = new(SchemaFixture.TenantB, null, Guid.NewGuid(), Guid.NewGuid().ToString()), + ["foreign-org"] = new(SchemaFixture.TenantB, SchemaFixture.OrgB1, Guid.NewGuid(), Guid.NewGuid().ToString()), + }; + await using (var app = await PostgresFixture.OpenAsync(database.AppConnectionString)) + { + await SchemaQueries.AssertApplicationRoleAsync(app); + await using var transaction = await app.BeginTransactionAsync(); + foreach (var canary in canaries.Values) + { + await AnnounceAsync(app, transaction, canary.TenantId, canary.OrganizationId); + await InsertAsync(app, transaction, canary.TenantId, canary.OrganizationId, canary.EventId); + } + await transaction.CommitAsync(); + } + await using var scheduler = await PostgresFixture.OpenAsync(database.OutboxConnectionString); + foreach (var canary in canaries.Values) + { + await using var lease = new NpgsqlCommand(""" + UPDATE event_deliveries SET locked_by = @token, + locked_until = clock_timestamp() + interval '30 seconds', attempts = 1 + WHERE tenant_id = @tenant AND subscription_id = @subscription AND event_id = @event + AND processed_at IS NULL AND terminal_reason IS NULL AND dead_lettered_at IS NULL + AND locked_by IS NULL + """, (NpgsqlConnection)scheduler); + lease.Parameters.AddWithValue("tenant", canary.TenantId); + lease.Parameters.AddWithValue("subscription", Subscription); + lease.Parameters.AddWithValue("event", canary.EventId); + lease.Parameters.AddWithValue("token", canary.LeaseToken); + (await lease.ExecuteNonQueryAsync()).Should().Be(1); + } + (await SchemaQueries.CountAsync(scheduler, """ + SELECT count(*) FROM event_deliveries + WHERE processed_at IS NULL AND locked_by IS NOT NULL AND locked_until > clock_timestamp() + """)).Should().Be(5, "every refusal must have a real live leased row, rather than a missing or expired target"); + return canaries; + } + + private static async Task> ReadOrganizationCanariesAsync(DisposableSchemaDatabase database, bool tenantHatch) + { + await using var app = await PostgresFixture.OpenAsync(database.AppConnectionString); + await SchemaQueries.AssertApplicationRoleAsync(app); + await using var transaction = await app.BeginTransactionAsync(); + await AnnounceAsync(app, transaction, SchemaFixture.TenantA, SchemaFixture.OrgA1); + await SchemaQueries.SetSettingAsync(app, transaction, "app.scope", tenantHatch ? "tenant" : ""); + // No application WHERE clause can hide a missing tenant/org policy here. + return await SchemaQueries.ReadStringsAsync(app, "SELECT event_id::text FROM event_deliveries", transaction); + } + + private static async Task CompleteOrganizationCanaryAsync( + DisposableSchemaDatabase database, OrganizationCanary canary, bool tenantHatch) + { + await using var app = await PostgresFixture.OpenAsync(database.AppConnectionString); + await SchemaQueries.AssertApplicationRoleAsync(app); + await using var transaction = await app.BeginTransactionAsync(); + await AnnounceAsync(app, transaction, SchemaFixture.TenantA, SchemaFixture.OrgA1); + await SchemaQueries.SetSettingAsync(app, transaction, "app.scope", tenantHatch ? "tenant" : ""); + await using var complete = new NpgsqlCommand(""" + UPDATE event_deliveries SET processed_at = clock_timestamp() + WHERE tenant_id = @tenant AND subscription_id = @subscription AND event_id = @event + AND processed_at IS NULL AND dead_lettered_at IS NULL AND terminal_reason IS NULL + AND locked_by = @token AND locked_until > clock_timestamp() + """, (NpgsqlConnection)app, (NpgsqlTransaction)transaction); + complete.Parameters.AddWithValue("tenant", canary.TenantId); + complete.Parameters.AddWithValue("subscription", Subscription); + complete.Parameters.AddWithValue("event", canary.EventId); + complete.Parameters.AddWithValue("token", canary.LeaseToken); + var affected = await complete.ExecuteNonQueryAsync(); + await transaction.CommitAsync(); + return affected; + } + + private static PlatformDbContext Platform(DisposableSchemaDatabase database) => new( + new DbContextOptionsBuilder().UseNpgsql(database.MigrationConnectionString, + provider => provider.MigrationsHistoryTable(PlatformDbContextFactory.HistoryTable)).Options); + + private static async Task AnnounceAsync(DbConnection connection, DbTransaction transaction, Guid tenant, Guid? organization = null) + { + await SchemaQueries.SetTenantAsync(connection, transaction, tenant); + await SchemaQueries.SetSettingAsync(connection, transaction, "app.organization_id", organization?.ToString() ?? ""); + await SchemaQueries.SetSettingAsync(connection, transaction, "app.scope", ""); + } + + private static Task InsertAsync(DbConnection connection, DbTransaction transaction, Guid tenant, + Guid? organization, Guid eventId, string subscription = Subscription) => + SchemaQueries.ExecuteAsync(connection, transaction, InsertDelivery, + ("tenant", tenant), ("subscription", subscription), ("event", eventId), + ("organization", (object?)organization ?? DBNull.Value), ("actor", SchemaFixture.Actor), ("partition", tenant.ToString())); + + private static async Task CountEventAsync(DbConnection connection, DbTransaction transaction, string table, Guid eventId) + { + // Table comes only from the two literal callers in this test. + await using var command = new NpgsqlCommand($"SELECT count(*) FROM {table} WHERE event_id = @event", + (NpgsqlConnection)connection, (NpgsqlTransaction)transaction); + command.Parameters.AddWithValue("event", eventId); + return (long)(await command.ExecuteScalarAsync())!; + } + + private static async Task> GrantedColumnsAsync(DbConnection connection, string role, string table, string privilege) + { + // DELETE has no column privilege in PostgreSQL; table privileges still + // appear in column_privileges for SELECT/INSERT/UPDATE automatically. + await using var command = new NpgsqlCommand(""" + SELECT column_name FROM information_schema.column_privileges + WHERE table_schema = 'public' AND grantee = @role AND table_name = @table + AND privilege_type = @privilege ORDER BY column_name + """, (NpgsqlConnection)connection); + command.Parameters.AddWithValue("role", role); + command.Parameters.AddWithValue("table", table); + command.Parameters.AddWithValue("privilege", privilege); + var result = new List(); + await using var reader = await command.ExecuteReaderAsync(); + while (await reader.ReadAsync()) result.Add(reader.GetString(0)); + return result; + } + + private static async Task> InventoryAsync(string connectionString) + { + await using var owner = await PostgresFixture.OpenAsync(connectionString); + return await SchemaQueries.ReadStringsAsync(owner, """ + SELECT item FROM ( + SELECT 'column:' || table_name || ':' || ordinal_position || ':' || column_name || ':' || + data_type || ':' || is_nullable || ':' || coalesce(column_default, '') AS item + FROM information_schema.columns + WHERE table_schema = 'public' AND table_name IN ('event_deliveries','inbox_messages') + UNION ALL + SELECT 'constraint:' || c.relname || ':' || conname || ':' || pg_get_constraintdef(k.oid) + FROM pg_constraint k JOIN pg_class c ON c.oid = k.conrelid + WHERE c.relnamespace = 'public'::regnamespace AND c.relname IN ('event_deliveries','inbox_messages') + UNION ALL + SELECT 'index:' || tablename || ':' || indexname || ':' || indexdef FROM pg_indexes + WHERE schemaname = 'public' AND tablename IN ('event_deliveries','inbox_messages') + UNION ALL + SELECT 'policy:' || tablename || ':' || policyname || ':' || permissive || ':' || roles::text || ':' || cmd || ':' || + coalesce(qual, '') || ':' || coalesce(with_check, '') FROM pg_policies + WHERE schemaname = 'public' AND tablename IN ('event_deliveries','inbox_messages') + UNION ALL + SELECT 'rls:' || relname || ':' || relrowsecurity || ':' || relforcerowsecurity FROM pg_class + WHERE relnamespace = 'public'::regnamespace AND relname IN ('event_deliveries','inbox_messages') + UNION ALL + SELECT 'grant:' || table_name || ':' || grantee || ':' || column_name || ':' || privilege_type + FROM information_schema.column_privileges + WHERE table_schema = 'public' AND table_name IN ('event_deliveries','inbox_messages') + UNION ALL + SELECT 'table-grant:' || table_name || ':' || grantee || ':' || privilege_type + FROM information_schema.table_privileges + WHERE table_schema = 'public' AND table_name IN ('event_deliveries','inbox_messages') + UNION ALL + SELECT 'trigger:' || c.relname || ':' || pg_get_triggerdef(t.oid) + FROM pg_trigger t JOIN pg_class c ON c.oid = t.tgrelid + WHERE c.relnamespace = 'public'::regnamespace AND c.relname IN ('event_deliveries','inbox_messages') AND NOT t.tgisinternal + UNION ALL + SELECT 'function:' || pg_get_functiondef(oid) FROM pg_proc + WHERE pronamespace = 'public'::regnamespace AND proname = 'fn_inbox_messages_immutable' + ) inventory ORDER BY item + """); + } +} diff --git a/backend/tests/LearnStack.Tests.Integration/Database/InboxGuardTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/InboxGuardTests.cs new file mode 100644 index 00000000..3f5a7fe2 --- /dev/null +++ b/backend/tests/LearnStack.Tests.Integration/Database/InboxGuardTests.cs @@ -0,0 +1,328 @@ +using FluentAssertions; +using LearnStack.Infrastructure.Messaging; +using LearnStack.Infrastructure.Persistence; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Persistence; +using LearnStack.SharedKernel.Tenancy; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using Xunit; + +namespace LearnStack.Tests.Integration.Database; + +/// The transport-owned inbox claim on an announced application transaction. +[Trait(RequiresDocker.Key, RequiresDocker.Value)] +[Collection(SharedSchema.Name)] +public sealed class InboxGuardTests(SchemaFixture schema) +{ + private const string First = "proof.first.v1"; + private const string Second = "proof.second.v1"; + + [Fact] + public async Task A_committed_claim_is_false_on_duplicate_but_distinct_tuples_are_independent() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + var eventId = Guid.CreateVersion7(); + await using (var unit = Unit(source)) + { + var context = Context(SchemaFixture.TenantA); + await using var frame = await AnnounceAsync(unit, context); + var guard = new PostgresInboxGuard(unit, context); + (await guard.TryClaimAsync(TenantId.From(SchemaFixture.TenantA), First, eventId)).Should().BeTrue(); + (await guard.TryClaimAsync(TenantId.From(SchemaFixture.TenantA), First, eventId)).Should().BeFalse(); + (await guard.TryClaimAsync(TenantId.From(SchemaFixture.TenantA), Second, eventId)).Should().BeTrue(); + (await guard.TryClaimAsync(TenantId.From(SchemaFixture.TenantA), First, Guid.CreateVersion7())) + .Should().BeTrue(); + await frame.CompleteAsync(); + } + + await using (var unit = Unit(source)) + { + var context = Context(SchemaFixture.TenantA); + await using var frame = await AnnounceAsync(unit, context); + (await new PostgresInboxGuard(unit, context) + .TryClaimAsync(TenantId.From(SchemaFixture.TenantA), First, eventId)).Should().BeFalse(); + await frame.CompleteAsync(); + } + await using (var unit = Unit(source)) + { + var context = Context(SchemaFixture.TenantB); + await using var frame = await AnnounceAsync(unit, context); + (await new PostgresInboxGuard(unit, context) + .TryClaimAsync(TenantId.From(SchemaFixture.TenantB), First, eventId)).Should().BeTrue(); + await frame.CompleteAsync(); + } + (await CountAsync(database.AppConnectionString, SchemaFixture.TenantA, eventId)).Should().Be(2); + (await CountAsync(database.AppConnectionString, SchemaFixture.TenantB, eventId)).Should().Be(1); + } + + [Fact] + public async Task Guard_refuses_missing_wrong_or_nonwritable_announcement_before_inserting() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + var eventId = Guid.CreateVersion7(); + var tenantA = TenantId.From(SchemaFixture.TenantA); + + await using (var unit = Unit(source)) + { + var context = Context(SchemaFixture.TenantA); + var guard = new PostgresInboxGuard(unit, context); + await FluentActions.Awaiting(() => guard.TryClaimAsync(tenantA, First, eventId)) + .Should().ThrowAsync(); + await using var frame = await unit.BeginTransactionAsync(); + await FluentActions.Awaiting(() => guard.TryClaimAsync(tenantA, First, eventId)) + .Should().ThrowAsync(); + await frame.FailAsync(); + } + + await using (var unit = Unit(source)) + { + var context = Context(SchemaFixture.TenantA); + await using var frame = await AnnounceAsync(unit, context); + await FluentActions.Awaiting(() => new PostgresInboxGuard(unit, context) + .TryClaimAsync(TenantId.From(SchemaFixture.TenantB), First, eventId)) + .Should().ThrowAsync(); + await frame.FailAsync(); + } + + await using (var unit = Unit(source)) + { + var announced = Context(SchemaFixture.TenantA, SchemaFixture.OrgA1); + await using var frame = await AnnounceAsync(unit, announced); + var wrongOrganization = Context(SchemaFixture.TenantA, SchemaFixture.OrgA2); + await FluentActions.Awaiting(() => new PostgresInboxGuard(unit, wrongOrganization) + .TryClaimAsync(tenantA, First, eventId)) + .Should().ThrowAsync(); + await frame.FailAsync(); + } + + await using (var unit = Unit(source)) + { + var context = Context(SchemaFixture.TenantA); + await using var frame = await AnnounceAsync(unit, context, TransactionMode.ReadOnly); + await FluentActions.Awaiting(() => new PostgresInboxGuard(unit, context) + .TryClaimAsync(tenantA, First, eventId)) + .Should().ThrowAsync(); + await frame.FailAsync(); + } + + await using (var unit = Unit(source)) + { + var context = Context(SchemaFixture.TenantA); + await using var frame = await AnnounceAsync(unit, context); + unit.MarkRollbackOnly(); + await FluentActions.Awaiting(() => new PostgresInboxGuard(unit, context) + .TryClaimAsync(tenantA, First, eventId)) + .Should().ThrowAsync(); + await frame.FailAsync(); + } + + (await CountAsync(database.AppConnectionString, SchemaFixture.TenantA, eventId)).Should().Be(0); + } + + [Theory] + [InlineData("")] + [InlineData("proof.first.v0")] + [InlineData("proof.first.v01")] + [InlineData("Proof.first.v1")] + [InlineData("proof.bad/name.v1")] + [InlineData("proof.first.v1 ")] + public async Task Invalid_subscription_identity_refuses_before_SQL(string subscription) + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var unit = Unit(source); + var context = Context(SchemaFixture.TenantA); + await using var frame = await AnnounceAsync(unit, context); + var eventId = Guid.CreateVersion7(); + + await FluentActions.Awaiting(() => new PostgresInboxGuard(unit, context) + .TryClaimAsync(TenantId.From(SchemaFixture.TenantA), subscription, eventId)) + .Should().ThrowAsync(); + + unit.IsRollbackOnly.Should().BeFalse("input rejection occurs before any SQL statement"); + (await CountInUnitAsync(unit, eventId)).Should().Be(0); + await frame.CompleteAsync(); + } + + [Fact] + public async Task Empty_event_identity_refuses_before_SQL_and_recorded_at_is_database_owned() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var unit = Unit(source); + var context = Context(SchemaFixture.TenantA); + await using var frame = await AnnounceAsync(unit, context); + var guard = new PostgresInboxGuard(unit, context); + await FluentActions.Awaiting(() => guard.TryClaimAsync(TenantId.From(SchemaFixture.TenantA), First, + Guid.Empty)).Should().ThrowAsync(); + var eventId = Guid.CreateVersion7(); + (await guard.TryClaimAsync(TenantId.From(SchemaFixture.TenantA), First, eventId)).Should().BeTrue(); + await using var read = new NpgsqlCommand(""" + SELECT recorded_at <= clock_timestamp() AND recorded_at > clock_timestamp() - interval '1 minute' + FROM inbox_messages WHERE tenant_id = @tenant AND subscription_id = @subscription AND event_id = @event + """, (NpgsqlConnection)unit.Connection, (NpgsqlTransaction)unit.Transaction!); + read.Parameters.AddWithValue("tenant", SchemaFixture.TenantA); + read.Parameters.AddWithValue("subscription", First); + read.Parameters.AddWithValue("event", eventId); + (await read.ExecuteScalarAsync()).Should().Be(true); + await frame.CompleteAsync(); + + await using var connection = await OpenAsync(database.AppConnectionString); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.SetTenantAsync(connection, transaction, SchemaFixture.TenantA); + await using var overrideTimestamp = new NpgsqlCommand(""" + INSERT INTO inbox_messages (tenant_id, subscription_id, event_id, recorded_at) + VALUES (@tenant, @subscription, @event, @timestamp) + """, connection, transaction); + overrideTimestamp.Parameters.AddWithValue("tenant", SchemaFixture.TenantA); + overrideTimestamp.Parameters.AddWithValue("subscription", First); + overrideTimestamp.Parameters.AddWithValue("event", Guid.CreateVersion7()); + overrideTimestamp.Parameters.AddWithValue("timestamp", DateTimeOffset.UtcNow); + var denied = await FluentActions.Awaiting(() => overrideTimestamp.ExecuteNonQueryAsync()) + .Should().ThrowAsync(); + denied.Which.SqlState.Should().Be(PostgresErrorCodes.InsufficientPrivilege); + } + + [Fact] + public async Task Default_empty_and_platform_tenants_refuse_before_SQL_and_completed_frame_cannot_claim() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var unit = Unit(source); + var context = Context(SchemaFixture.TenantA); + await using var frame = await AnnounceAsync(unit, context); + var guard = new PostgresInboxGuard(unit, context); + var eventId = Guid.CreateVersion7(); + // An array element supplies the uninitialized Vogen value without using + // its prohibited default constructor in application code. + var uninitialized = (new TenantId[1])[0]; + foreach (var invalid in new[] { uninitialized, TenantId.From(Guid.Empty), + TenantId.PlatformSentinel }) + { + await FluentActions.Awaiting(() => guard.TryClaimAsync(invalid, First, eventId)) + .Should().ThrowAsync(); + unit.IsRollbackOnly.Should().BeFalse(); + } + (await CountInUnitAsync(unit, eventId)).Should().Be(0); + await frame.CompleteAsync(); + + await FluentActions.Awaiting(() => guard.TryClaimAsync( + TenantId.From(SchemaFixture.TenantA), First, eventId)) + .Should().ThrowAsync(); + (await CountAsync(database.AppConnectionString, SchemaFixture.TenantA, eventId)).Should().Be(0); + } + + [Theory] + [InlineData(true)] + [InlineData(false)] + public async Task Concurrent_claim_waits_on_uniqueness_and_observes_first_transaction_outcome(bool firstCommits) + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var first = Unit(source); + await using var second = Unit(source); + var context = Context(SchemaFixture.TenantA); + var eventId = Guid.CreateVersion7(); + await using var firstFrame = await AnnounceAsync(first, context); + await using var secondFrame = await AnnounceAsync(second, context); + var tenant = TenantId.From(SchemaFixture.TenantA); + (await new PostgresInboxGuard(first, context).TryClaimAsync(tenant, First, eventId)).Should().BeTrue(); + var secondPid = await BackendPidAsync(second); + + var pending = new PostgresInboxGuard(second, context).TryClaimAsync(tenant, First, eventId); + await BlockedByAsync(database.AppConnectionString, secondPid, first); + pending.IsCompleted.Should().BeFalse("PostgreSQL must wait for the first transaction's unique key"); + if (firstCommits) await firstFrame.CompleteAsync(); + else await firstFrame.FailAsync(); + + (await pending.WaitAsync(TimeSpan.FromSeconds(10))).Should().Be(!firstCommits); + await secondFrame.CompleteAsync(); + (await CountAsync(database.AppConnectionString, SchemaFixture.TenantA, eventId)).Should().Be(1); + } + + private static NpgsqlUnitOfWork Unit(NpgsqlDataSource source) => + new(source, NullLogger.Instance); + + private static ClaimContext Context(Guid tenant, Guid? organization = null) => + new ClaimContext(TenantId.From(tenant), organization is { } id ? OrganizationId.From(id) : null); + + private static async Task AnnounceAsync(NpgsqlUnitOfWork unit, ITenantContext context, + TransactionMode mode = TransactionMode.ReadWrite) + { + var frame = await unit.BeginTransactionAsync(mode); + await unit.SetTenantContextAsync(context); + return frame; + } + + private static async Task BackendPidAsync(NpgsqlUnitOfWork unit) + { + await using var command = new NpgsqlCommand("SELECT pg_backend_pid()", + (NpgsqlConnection)unit.Connection, (NpgsqlTransaction)unit.Transaction!); + return (int)(await command.ExecuteScalarAsync())!; + } + + private static async Task BlockedByAsync(string connectionString, int secondPid, NpgsqlUnitOfWork first) + { + var firstPid = await BackendPidAsync(first); + using var deadline = new CancellationTokenSource(TimeSpan.FromSeconds(10)); + await using var connection = await OpenAsync(connectionString, deadline.Token); + await SchemaQueries.AssertApplicationRoleAsync(connection); + while (true) + { + deadline.Token.ThrowIfCancellationRequested(); + await using var command = new NpgsqlCommand(""" + SELECT @first = ANY(pg_blocking_pids(@second)) + FROM pg_stat_activity WHERE pid = @second AND wait_event_type = 'Lock' + """, connection); + command.Parameters.AddWithValue("first", firstPid); + command.Parameters.AddWithValue("second", secondPid); + if (await command.ExecuteScalarAsync(deadline.Token) is true) return; + // The observed blocker, rather than elapsed time, releases this proof. + await Task.Yield(); + } + } + + private static async Task CountAsync(string connectionString, Guid tenantId, Guid eventId) + { + await using var connection = await OpenAsync(connectionString); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await SchemaQueries.SetTenantAsync(connection, transaction, tenantId); + await using var command = new NpgsqlCommand(""" + SELECT count(*) FROM inbox_messages WHERE tenant_id = @tenant AND event_id = @event + """, connection, transaction); + command.Parameters.AddWithValue("tenant", tenantId); + command.Parameters.AddWithValue("event", eventId); + return (long)(await command.ExecuteScalarAsync())!; + } + + private static async Task CountInUnitAsync(NpgsqlUnitOfWork unit, Guid eventId) + { + await using var command = new NpgsqlCommand( + "SELECT count(*) FROM inbox_messages WHERE event_id = @event", + (NpgsqlConnection)unit.Connection, (NpgsqlTransaction)unit.Transaction!); + command.Parameters.AddWithValue("event", eventId); + return (long)(await command.ExecuteScalarAsync())!; + } + + private sealed class ClaimContext(TenantId tenant, OrganizationId? organization) : ITenantContext + { + public bool IsResolved => true; + public TenantId TenantId => tenant; + public OrganizationId? OrganizationId => organization; + public UserId? UserId => LearnStack.SharedKernel.Identifiers.UserId.SystemActor; + public string? CorrelationId => null; + public string? ModuleName => null; + } + + private static async Task OpenAsync(string connectionString, + CancellationToken cancellationToken = default) + { + var connection = new NpgsqlConnection(connectionString); + await connection.OpenAsync(cancellationToken); + return connection; + } +} diff --git a/backend/tests/LearnStack.Tests.Integration/Database/MigrationChains.cs b/backend/tests/LearnStack.Tests.Integration/Database/MigrationChains.cs index 8aa9324f..3cd4462d 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/MigrationChains.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/MigrationChains.cs @@ -33,12 +33,7 @@ internal static class MigrationChains /// Applies every chain, as learnstack_migration. public static async Task ApplyAllAsync(string migrationConnectionString) { - await using (var tenancy = new TenancyDbContext( - Options(migrationConnectionString, TenancyDbContextFactory.HistoryTable), - StaticTenantContextAccessor.Unresolved)) - { - await tenancy.Database.MigrateAsync(); - } + await ApplyTenancyAsync(migrationConnectionString); await using (var platform = new PlatformDbContext( Options(migrationConnectionString, PlatformDbContextFactory.HistoryTable))) @@ -74,6 +69,15 @@ public static async Task ApplyAllAsync(string migrationConnectionString) await education.Database.MigrateAsync(); } + /// Installs the shared scope function before a dependent chain is tested alone. + public static async Task ApplyTenancyAsync(string migrationConnectionString) + { + await using var tenancy = new TenancyDbContext( + Options(migrationConnectionString, TenancyDbContextFactory.HistoryTable), + StaticTenantContextAccessor.Unresolved); + await tenancy.Database.MigrateAsync(); + } + /// The history table every chain declares, for a fixture that counts them. public static readonly string[] HistoryTables = [ diff --git a/backend/tests/LearnStack.Tests.Integration/Database/MigrationRollbackTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/MigrationRollbackTests.cs index bb554db8..b9c2376a 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/MigrationRollbackTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/MigrationRollbackTests.cs @@ -49,8 +49,8 @@ public async Task EveryChainReversesToAnEmptySchema() // Applied state first, so a rollback that reversed nothing because nothing // was there cannot pass. - (await CountAsync(connection, TablesQuery)).Should().Be(26L, - "eight tenancy tables, three platform tables, four customization tables, " + (await CountAsync(connection, TablesQuery)).Should().Be(28L, + "eight tenancy tables, five platform tables, four customization tables, " + "two audit tables, four Education tables, and five history tables"); (await CountAsync(connection, FunctionQuery)).Should().Be(1L, "fn_organization_id_immutable backs the tenant_settings trigger"); @@ -59,6 +59,8 @@ public async Task EveryChainReversesToAnEmptySchema() + "append-only triggers"); (await CountAsync(connection, EducationFunctionQuery)).Should().Be(3L, "each Education parent relation has its own invoker scope check"); + (await CountAsync(connection, InboxFunctionQuery)).Should().Be(1L, + "fn_inbox_messages_immutable backs the shared inbox append-only trigger"); await _fixture.RollBackAsync(); @@ -73,14 +75,16 @@ public async Task EveryChainReversesToAnEmptySchema() // with a matching signature is not an error the migration surfaces. (await CountAsync(connection, AuditFunctionQuery)).Should().Be(0L); (await CountAsync(connection, EducationFunctionQuery)).Should().Be(0L); + (await CountAsync(connection, InboxFunctionQuery)).Should().Be(0L); (await CountAsync(connection, PolicyQuery)).Should().Be(0L); // A successful Down must also leave a state that can be installed again. await MigrationChains.ApplyAllAsync(_fixture.Postgres.MigrationConnectionString); - (await CountAsync(connection, TablesQuery)).Should().Be(26L); + (await CountAsync(connection, TablesQuery)).Should().Be(28L); (await CountAsync(connection, FunctionQuery)).Should().Be(1L); (await CountAsync(connection, AuditFunctionQuery)).Should().Be(2L); (await CountAsync(connection, EducationFunctionQuery)).Should().Be(3L); + (await CountAsync(connection, InboxFunctionQuery)).Should().Be(1L); await EducationSchemaSeed.SeedAsync(_fixture.Postgres.AppConnectionString); await using var app = await PostgresFixture.OpenAsync(_fixture.Postgres.AppConnectionString); await using var transaction = await app.BeginTransactionAsync(); @@ -105,6 +109,10 @@ public async Task EveryChainReversesToAnEmptySchema() private const string PolicyQuery = "SELECT count(*) FROM pg_policies WHERE schemaname = 'public'"; + private const string InboxFunctionQuery = + "SELECT count(*) FROM pg_proc WHERE pronamespace = 'public'::regnamespace " + + "AND proname = 'fn_inbox_messages_immutable'"; + private const string EducationFunctionQuery = "SELECT count(*) FROM pg_proc WHERE pronamespace = 'public'::regnamespace " + "AND proname IN ('fn_lessons_parent_scope', 'fn_course_translations_parent_scope', " diff --git a/backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatchMigrationTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatchMigrationTests.cs index e6e68601..11dfd206 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatchMigrationTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatchMigrationTests.cs @@ -29,6 +29,7 @@ public sealed class OutboxDispatchMigrationTests(SchemaFixture schema) public async Task Legacy_facts_survive_up_down_reapply_and_diagnostics_are_redacted_across_tenants() { await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres, applyMigrations: false); + await MigrationChains.ApplyTenancyAsync(database.MigrationConnectionString); await using var context = Platform(database); await context.GetService().MigrateAsync(PlatformPredecessor); await InsertLegacyAsync(database, Guid.NewGuid(), SchemaFixture.TenantA, null, processed: false); @@ -70,6 +71,7 @@ SELECT count(*) FROM pg_constraint public async Task Recovery_index_covers_expired_and_exhausted_selection_without_scanning_pending_backlog(bool expiredLease) { await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres, applyMigrations: false); + await MigrationChains.ApplyTenancyAsync(database.MigrationConnectionString); await using var context = Platform(database); await context.GetService().MigrateAsync(DispatchPredecessor); var target = Guid.Parse("10000000-0000-0000-0000-000000000002"); @@ -101,6 +103,7 @@ public async Task Recovery_index_covers_expired_and_exhausted_selection_without_ public async Task Invalid_legacy_identity_refuses_migration_without_repair(string member, string invalid) { await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres, applyMigrations: false); + await MigrationChains.ApplyTenancyAsync(database.MigrationConnectionString); await using var context = Platform(database); await context.GetService().MigrateAsync(PlatformPredecessor); await InsertLegacyAsync(database, diff --git a/backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatchStoreTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatchStoreTests.cs index c8629cbe..5c0affdf 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatchStoreTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatchStoreTests.cs @@ -512,6 +512,30 @@ public async Task Terminal_audit_uses_the_stored_tenant_organization_and_validat (await test.ReadAsync(id)).DeadLetteredAt.Should().NotBeNull(); } + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Terminal_audit_keeps_valid_trace_for_absent_causal_values_but_replaces_it_for_empty_values(bool empty) + { + await using var test = await TestDatabase.CreateAsync(schema.Postgres); + Guid? causalValue = empty ? Guid.Empty : null; + var id = await test.InsertAsync(correlation: Traceparent, causation: causalValue, actor: causalValue, payload: "{}"); + var store = test.Store(); + var claim = (await store.TryClaimAsync(NoBlockedPartitions))!; + (await store.RequestTerminalAsync(claim, OutboxTerminalReason.InvalidEnvelope)).Should().BeTrue(); + var terminal = (await store.TryClaimTerminalAsync(NoBlockedPartitions))!; + + (await store.TryTerminalizeAsync(terminal, AttemptTraceparent)).Should().BeTrue(); + + var audit = await test.ReadAuditAsync(); + audit.Correlation.Should().Be(empty ? AttemptTraceparent : Traceparent, + "SQL NULL is valid optional absence, while an empty stored identifier is invalid metadata"); + using var metadata = JsonDocument.Parse(audit.Metadata); + metadata.RootElement.GetProperty("event_id").GetGuid().Should().Be(id); + metadata.RootElement.TryGetProperty("causation_id", out _).Should().BeFalse(); + metadata.RootElement.TryGetProperty("causal_actor_user_id", out _).Should().BeFalse(); + } + [Theory] [InlineData("tenant")] [InlineData("organization")] @@ -853,14 +877,17 @@ public async Task Empty_and_future_occurrence_snapshots_never_report_negative_el } [Fact] - public async Task Reserved_subscriber_terminal_operation_remains_unregistered() + public async Task Subscriber_terminal_operation_is_declared_as_MUST_security_event() { await using var test = await TestDatabase.CreateAsync(schema.Postgres); var catalog = test.Provider.GetRequiredService(); catalog.TryGetOffPath("platform.outbox.dead_letter", out var producer).Should().BeTrue(); producer.OperationClass.Should().Be(OperationClass.Must); producer.OperationType.Should().Be(OperationType.SecurityEvent); - catalog.TryGetOffPath("platform.delivery.dead_letter", out _).Should().BeFalse(); + catalog.TryGetOffPath("platform.delivery.dead_letter", out var subscriber).Should().BeTrue(); + subscriber.Operation.Should().Be("platform.delivery.dead_letter"); + subscriber.OperationClass.Should().Be(OperationClass.Must); + subscriber.OperationType.Should().Be(OperationType.SecurityEvent); } private static Task TransitionAsync(OutboxDispatchStore store, OutboxDispatchClaim claim, string transition) => transition switch diff --git a/backend/tests/LearnStack.Tests.Integration/Database/OutboxProcessorTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/OutboxProcessorTests.cs index 63e0af8f..917cafc7 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/OutboxProcessorTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/OutboxProcessorTests.cs @@ -24,7 +24,11 @@ namespace LearnStack.Tests.Integration.Database; -/// The registered production worker, real PostgreSQL roles and actual transport scopes. +/// The registered producer worker and real PostgreSQL publication acceptance. +/// +/// A test-owned scoped wrapper controls publication/cancellation/cleanup timing around +/// the actual durable transport. Subscriber business execution belongs to its own worker. +/// [Trait(RequiresDocker.Key, RequiresDocker.Value)] [Collection(SharedSchema.Name)] public sealed class OutboxProcessorTests(SchemaFixture schema) @@ -36,7 +40,7 @@ public sealed class OutboxProcessorTests(SchemaFixture schema) private static readonly OutboxDispatchOptions Bounds = new() { SqlTimeout = TimeSpan.FromSeconds(1), - PublishTimeout = TimeSpan.FromMilliseconds(150), + PublishTimeout = TimeSpan.FromMilliseconds(500), LeaseDuration = TimeSpan.FromSeconds(3), IdlePoll = TimeSpan.FromMilliseconds(20), ProgressTimeout = TimeSpan.FromSeconds(5), @@ -63,6 +67,44 @@ public async Task The_actual_API_registers_exactly_one_worker_and_boots_its_guar reader.GetString(1).Should().Be(OutboxDispatcherDataSource.Role); } + [Fact] + public async Task A_real_unwrapped_publication_acknowledges_retained_delivery_without_business_construction() + { + await using var test = await WorkerDatabase.CreateAsync(schema.Postgres); + var id = await test.InsertAsync(); + test.Start(Bounds with + { + PublishTimeout = TimeSpan.FromSeconds(10), + LeaseDuration = TimeSpan.FromSeconds(30), + ProgressTimeout = TimeSpan.FromSeconds(45), + }, controlPublication: false); + await UntilAsync(async () => (await test.ReadAsync(id)).ProcessedAt is not null); + + test.Callbacks.Publications.Should().Be(0, "this proof uses the shipped transport without the timing wrapper"); + var producer = await test.ReadAsync(id); + producer.Attempts.Should().Be(1); + await using var app = await PostgresFixture.OpenAsync(test.AppConnectionString); + await SchemaQueries.AssertApplicationRoleAsync(app); + await using var transaction = await app.BeginTransactionAsync(); + await SchemaQueries.SetTenantAsync(app, transaction, SchemaFixture.TenantA); + await using var delivery = new NpgsqlCommand(""" + SELECT subscription_id, payload::text, processed_at FROM event_deliveries + WHERE tenant_id = @tenant AND event_id = @event + """, (NpgsqlConnection)app, (NpgsqlTransaction)transaction); + delivery.Parameters.AddWithValue("tenant", SchemaFixture.TenantA); + delivery.Parameters.AddWithValue("event", id); + await using var reader = await delivery.ExecuteReaderAsync(); + (await reader.ReadAsync()).Should().BeTrue(); + reader.GetString(0).Should().Be("test.worker.v1"); + using var actual = System.Text.Json.JsonDocument.Parse(reader.GetString(1)); + using var expected = System.Text.Json.JsonDocument.Parse(producer.Payload); + System.Text.Json.JsonElement.DeepEquals(actual.RootElement, expected.RootElement).Should().BeTrue(); + reader.IsDBNull(2).Should().BeTrue("publication accepts durable work; the subscriber has not executed it"); + (await reader.ReadAsync()).Should().BeFalse(); + // WorkerHandler's constructor throws, so reaching durable acknowledgement + // also proves that publication did not construct its business graph. + } + [Fact] public async Task Two_independent_actual_processors_cannot_publish_or_claim_one_live_lease_twice() { @@ -111,7 +153,7 @@ await UntilAsync(() => test.Observation.Snapshot.LastSnapshotTimestamp > firstPr (await test.LeaseIsLiveAsync(id).WaitAsync(proof.Token)).Should().BeTrue("the contention assertion must run before expiry or a timeout"); (await test.ReadAsync(id).WaitAsync(proof.Token)).Should().Be(acquired); - test.Callbacks.Constructions.Should().Be(1); + test.Callbacks.Publications.Should().Be(1); observer.Snapshot.Queue.PendingCount.Should().Be(1); observer.Snapshot.Queue.OldestEligibleAgeSeconds.Should().Be(0); held.ReleaseCallback.TrySetResult(); @@ -119,7 +161,7 @@ await UntilAsync(() => test.Observation.Snapshot.LastSnapshotTimestamp > firstPr var completed = await test.ReadAsync(id).WaitAsync(proof.Token); completed.Attempts.Should().Be(1); completed.Token.Should().BeNull(); - test.Callbacks.Constructions.Should().Be(1); + test.Callbacks.Publications.Should().Be(1); } finally { @@ -189,7 +231,7 @@ public async Task Real_provisioning_rows_remain_Pending_across_hosted_polls_with (await test.ReadAsync(id)).Should().Be(before); before.Type.Should().Be(typeof(TenantCreatedIntegrationEventV1).AssemblyQualifiedName); before.Attempts.Should().Be(0); - test.Callbacks.Constructions.Should().Be(0); + test.Callbacks.Publications.Should().Be(0); test.Observation.Snapshot.Queue.PendingCount.Should().Be(1); test.Observation.Snapshot.Queue.DeferredCount.Should().Be(1); (await test.ReadTerminalAuditsAsync()).Should().BeEmpty(); @@ -240,12 +282,12 @@ public async Task Successful_claims_keep_the_snapshot_until_its_interval_and_ref test.Observation.Snapshot.Queue.Should().NotBeSameAs(initial.Queue); test.Observation.Snapshot.Queue.PendingCount.Should().Be(0); test.Observation.Snapshot.LastSnapshotTimestamp.Should().Be(bounds.SnapshotInterval.Ticks); - test.Callbacks.Constructions.Should().Be(3); + test.Callbacks.Publications.Should().Be(3); test.Observation.IsHealthy.Should().BeTrue(); } [Fact] - public async Task A_poison_first_row_terminalizes_with_MUST_audit_and_the_following_real_handler_progresses() + public async Task A_poison_first_row_terminalizes_with_MUST_audit_and_following_acceptance_progresses() { await using var test = await WorkerDatabase.CreateAsync(schema.Postgres); using var traces = new DispatcherTraces(); @@ -260,7 +302,7 @@ await UntilAsync(async () => (await test.ReadAsync(poison)).DeadLetteredAt is no row.TerminalReason.Should().Be("unknown_type"); row.Attempts.Should().Be(1); row.ProcessedAt.Should().BeNull(); - test.Callbacks.Constructions.Should().Be(1); + test.Callbacks.Publications.Should().Be(1); var audit = (await test.ReadTerminalAuditsAsync()).Should().ContainSingle().Which; audit.OperationClass.Should().Be("Must"); audit.OperationType.Should().Be("SecurityEvent"); @@ -276,7 +318,61 @@ await UntilAsync(async () => (await test.ReadAsync(poison)).DeadLetteredAt is no } [Fact] - public async Task Ordinary_handler_failure_persists_a_closed_descriptor_and_allows_following_work() + public async Task Terminal_audit_backlog_cannot_starve_normal_publication_or_expired_claim_recovery() + { + await using var test = await WorkerDatabase.CreateAsync(schema.Postgres); + var bounds = Bounds with + { + PublishTimeout = TimeSpan.FromSeconds(2), + LeaseDuration = TimeSpan.FromSeconds(5), + ProgressTimeout = TimeSpan.FromSeconds(6), + IdlePoll = TimeSpan.FromMilliseconds(50), + ShutdownTimeout = TimeSpan.FromSeconds(2), + }; + var terminals = new List(); + for (var index = 0; index < 100; index++) + { + var id = await test.InsertAsync(); + terminals.Add(id); + await test.RequestTerminalAsync(id); + } + var recovery = await test.InsertAsync(); + await test.ExpireAsync(recovery, attempts: 5); + await test.AuditInsertGrantAsync(false); + test.Start(bounds, controlPublication: false); + + // A repeated terminal acquisition requires a database lease to expire. + // The audit failure remains live when fresh publication is enqueued. + await UntilAsync(() => test.Logs.TerminalAuditFailures > terminals.Count, + proofTimeout: TimeSpan.FromSeconds(30)); + test.Observation.Snapshot.TerminalAuditFailed.Should().BeTrue(); + var normal = await test.InsertAsync(); + await UntilAsync(async () => (await test.ReadAsync(normal)).ProcessedAt is not null + && (await test.ReadAsync(recovery)).LastError == "recovery:unknown:expired"); + await test.Worker.StopAsync(CancellationToken.None).WaitAsync(ProofDeadline); + + var published = await test.ReadAsync(normal); + published.Attempts.Should().Be(1); + published.LastError.Should().BeNull(); + (await test.RetainedDeliveryCountAsync()).Should().Be(1, + "only the valid normal row reaches real app-role durable acceptance"); + (await test.RetainedDeliveryCountAsync(normal)).Should().Be(1); + // WorkerHandler throws in its constructor; durable acceptance also proves + // that publication never constructed or invoked the business handler. + var recovered = await test.ReadAsync(recovery); + recovered.Attempts.Should().Be(5); + recovered.ProcessedAt.Should().BeNull(); + recovered.TerminalReason.Should().BeNull(); + test.Callbacks.Publications.Should().Be(0, "the shipped transport is unwrapped"); + (await test.PendingTerminalCountAsync()).Should().Be(terminals.Count, + "failed MUST audit preserves every terminal request without publication"); + (await test.ReadTerminalAuditsAsync()).Should().BeEmpty(); + (await test.SchedulerCanInsertAuditAsync()).Should().BeFalse(); + test.Observation.Snapshot.TerminalAuditFailed.Should().BeTrue(); + } + + [Fact] + public async Task Publication_failure_persists_a_closed_descriptor_and_allows_following_work() { await using var test = await WorkerDatabase.CreateAsync(schema.Postgres); using var traces = new DispatcherTraces(); @@ -293,7 +389,7 @@ public async Task Ordinary_handler_failure_persists_a_closed_descriptor_and_allo row.Attempts.Should().Be(1); row.Token.Should().BeNull(); row.ProcessedAt.Should().BeNull(); - test.Callbacks.Constructions.Should().Be(2); + test.Callbacks.Publications.Should().Be(2); test.Logs.Text.Should().NotContain(SensitiveCanary); traces.Records.Should().NotBeEmpty(); string.Join("\n", traces.Records).Should().NotContain(SensitiveCanary); @@ -316,7 +412,7 @@ public async Task Explicit_intentional_absence_acknowledges_without_constructing await UntilAsync(async () => (await test.ReadAsync(id)).ProcessedAt is not null); (await test.ReadAsync(id)).Attempts.Should().Be(1); - test.Callbacks.Constructions.Should().Be(0); + test.Callbacks.Publications.Should().Be(0); (await test.ReadTerminalAuditsAsync()).Should().BeEmpty(); } @@ -353,7 +449,7 @@ public async Task Timeout_persists_failure_and_holds_the_partition_until_actual_ failed.ProcessedAt.Should().BeNull("late callback success cannot acknowledge a timed-out claim"); failed.Attempts.Should().Be(1); failed.Token.Should().BeNull(); - held.Disposals.Should().Be(2, "the released partition subsequently constructs and disposes its sibling's handler"); + held.Disposals.Should().Be(2, "the released partition subsequently publishes and disposes its sibling's acceptance scope"); } [Fact] @@ -376,7 +472,7 @@ public async Task Four_retained_callbacks_stop_all_acquisition_and_releasing_one await test.NextSnapshotAsync(); (await test.ReadAsync(fifth)).Attempts.Should().Be(0, "the four retained callbacks exhaust the global allowance"); - test.Callbacks.Constructions.Should().Be(4); + test.Callbacks.Publications.Should().Be(4); test.Observation.IsHealthy.Should().BeFalse(); held[0].ReleaseCallback.TrySetResult(); await UntilAsync(async () => (await test.ReadAsync(fifth)).ProcessedAt is not null); @@ -483,10 +579,17 @@ private static bool TokenSourceDisposed(CancellationToken token) private static Task UntilAsync(Func condition, CancellationToken cancellationToken = default) => UntilAsync(() => Task.FromResult(condition()), cancellationToken); - private static async Task UntilAsync(Func> condition, CancellationToken cancellationToken = default) + private static Task UntilAsync(Func condition, TimeSpan proofTimeout) + => UntilAsync(() => Task.FromResult(condition()), proofTimeout, CancellationToken.None); + + private static Task UntilAsync(Func> condition, CancellationToken cancellationToken = default) + => UntilAsync(condition, ProofDeadline, cancellationToken); + + private static async Task UntilAsync(Func> condition, TimeSpan proofTimeout, + CancellationToken cancellationToken) { using var deadline = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); - deadline.CancelAfter(ProofDeadline); + deadline.CancelAfter(proofTimeout); while (!await condition().WaitAsync(deadline.Token)) await Task.Delay(TimeSpan.FromMilliseconds(10), deadline.Token); } @@ -506,19 +609,35 @@ public sealed record UnsubscribedEventV1 : IntegrationEventBase public override string PartitionKey => SubjectId.ToString("D"); } - /// One discovered, configurable real-bus handler; no production subscription is activated. - public sealed class WorkerHandler(CallbackControls controls, ITenantContextAccessor accessor) - : IIntegrationEventHandler, IAsyncDisposable + /// Manifest metadata only: publication must never construct business handlers. + public sealed class WorkerHandler : IIntegrationEventHandler + { + public WorkerHandler() => throw new InvalidOperationException("Publication constructed a business handler."); + public Task HandleAsync(WorkerEventV1 integrationEvent, CancellationToken cancellationToken) => + throw new InvalidOperationException("Publication invoked a business handler."); + } + + /// Controllable publication around the real acceptance adapter, not a consumer. + private sealed class ControlledPublicationBus(InProcessEventBus inner, PublicationScope scope) : IEventBus + { + public async Task PublishAsync(IntegrationEventEnvelope envelope, CancellationToken cancellationToken = default) + { + await scope.EnterAsync(envelope, cancellationToken); + // Deliberately model an adapter that ignores cancellation after entry. + // Its late successful acceptance must never acknowledge an expired attempt. + await inner.PublishAsync(envelope, CancellationToken.None); + } + } + + /// The production publish attempt owns this real DI scope through async cleanup. + private sealed class PublicationScope(CallbackControls controls) : IAsyncDisposable { - private readonly CallbackControls _controls = controls.RecordConstruction(); + private readonly CallbackControls _controls = controls.RecordPublication(); private CallbackBarrier? _barrier; - public Task HandleAsync(WorkerEventV1 integrationEvent, CancellationToken cancellationToken) + public Task EnterAsync(IntegrationEventEnvelope envelope, CancellationToken cancellationToken) { - accessor.Current.Should().NotBeNull(); - accessor.Current!.TenantId.Value.Should().Be(integrationEvent.TenantId); - accessor.Current.UserId.Should().Be(UserId.SystemActor); - _barrier = _controls.Enter(integrationEvent.SubjectId, cancellationToken); + _barrier = _controls.Enter(Guid.Parse(envelope.PartitionKey), cancellationToken); if (_barrier.Failure is { } failure) throw failure; if (_barrier.SynchronousBlock) _barrier.ReleaseCallback.Task.GetAwaiter().GetResult(); return _barrier.ReleaseCallback.Task; @@ -536,8 +655,8 @@ public async ValueTask DisposeAsync() public sealed class CallbackControls { private readonly ConcurrentDictionary _partitions = new(); - private int _constructions; - public int Constructions => Volatile.Read(ref _constructions); + private int _publications; + public int Publications => Volatile.Read(ref _publications); public CallbackBarrier Hold(Guid partition, bool holdCleanup = false, bool synchronousBlock = false) { @@ -549,9 +668,9 @@ public CallbackBarrier Hold(Guid partition, bool holdCleanup = false, bool synch public void Fail(Guid partition, Exception failure) => _partitions.TryAdd(partition, new CallbackBarrier(false, false, false) { Failure = failure }).Should().BeTrue(); - public CallbackControls RecordConstruction() + public CallbackControls RecordPublication() { - Interlocked.Increment(ref _constructions); + Interlocked.Increment(ref _publications); return this; } @@ -687,7 +806,8 @@ public void Dispose() } private sealed class WorkerApiFactory(WorkerDatabase database, string? credential, - OutboxDispatchOptions? bounds = null, TimeProvider? time = null) : WebApplicationFactory + OutboxDispatchOptions? bounds = null, TimeProvider? time = null, + bool controlPublication = true) : WebApplicationFactory { protected override void ConfigureWebHost(IWebHostBuilder builder) { @@ -697,6 +817,10 @@ protected override void ConfigureWebHost(IWebHostBuilder builder) builder.UseSetting("ConnectionStrings:OutboxDispatcher", credential); builder.ConfigureTestServices(services => { + // This fixture proves producer acceptance independently of business delivery. + foreach (var descriptor in services.Where(descriptor => descriptor.ServiceType == typeof(IHostedService) + && descriptor.ImplementationType == typeof(DeliveryProcessor)).ToArray()) + services.Remove(descriptor); services.RemoveAll(); services.AddSingleton(bounds ?? Bounds); if (time is not null) @@ -705,7 +829,8 @@ protected override void ConfigureWebHost(IWebHostBuilder builder) services.AddSingleton(time); } services.RemoveAll(); - var handlers = IntegrationEventHandlerRegistry.Discover(typeof(WorkerHandler).Assembly); + var handlers = IntegrationEventHandlerRegistry.FromServiceDescriptors( + [ServiceDescriptor.Scoped, WorkerHandler>()]); handlers.All.Should().ContainSingle().Which.HandlerType.Should().Be(); services.AddSingleton(handlers); services.RemoveAll(); @@ -720,6 +845,17 @@ protected override void ConfigureWebHost(IWebHostBuilder builder) ], [], handlers, [typeof(WorkerEventV1).Assembly, typeof(TenantCreatedIntegrationEventV1).Assembly])); services.AddSingleton(database.Callbacks); services.AddScoped(); + // Preserve the shipped selection factory and wrap only this fixture's + // publication. The worker still owns a real async DI scope and the + // inner adapter persists actual subscription envelopes as learnstack_app. + if (controlPublication) + { + var transport = services.Single(descriptor => descriptor.ServiceType == typeof(IEventBus)); + services.RemoveAll(); + services.AddSingleton(provider => (InProcessEventBus)transport.ImplementationFactory!(provider)); + services.AddScoped(); + services.AddScoped(); + } services.AddSingleton>(database.Logs); }); } @@ -761,11 +897,12 @@ public static async Task CreateAsync(PostgresFixture postgres, b catch { await test.DisposeAsync(); throw; } } - public void Start(OutboxDispatchOptions? bounds = null, TimeProvider? time = null) + public void Start(OutboxDispatchOptions? bounds = null, TimeProvider? time = null, bool controlPublication = true) { - _factory = new WorkerApiFactory(this, OutboxConnectionString, bounds, time); + _factory = new WorkerApiFactory(this, OutboxConnectionString, bounds, time, controlPublication); _client = _factory.CreateClient(new WebApplicationFactoryClientOptions { BaseAddress = new Uri("http://localhost") }); - Services.GetRequiredService().Should().BeOfType(); + if (controlPublication) Services.GetRequiredService().Should().BeOfType(); + else Services.GetRequiredService().Should().BeOfType(); } public async Task NextSnapshotAsync() @@ -850,6 +987,64 @@ public async Task PostponeAsync(Guid id) (await update.ExecuteNonQueryAsync()).Should().Be(1); } + public async Task RequestTerminalAsync(Guid id) + { + await using var connection = await _dispatchSource.OpenConnectionAsync(); + await using var update = new NpgsqlCommand(""" + UPDATE outbox_messages SET attempts = 6, terminal_reason = 'retry_exhausted' + WHERE id = @id + """, connection); + update.Parameters.AddWithValue("id", id); + (await update.ExecuteNonQueryAsync()).Should().Be(1); + } + + public async Task ExpireAsync(Guid id, int attempts) + { + await using var connection = await _dispatchSource.OpenConnectionAsync(); + await using var update = new NpgsqlCommand(""" + UPDATE outbox_messages SET attempts = @attempts, locked_by = @token, + locked_until = clock_timestamp() - interval '1 minute' + WHERE id = @id + """, connection); + update.Parameters.AddWithValue("attempts", attempts); + update.Parameters.AddWithValue("token", Guid.CreateVersion7().ToString("D")); + update.Parameters.AddWithValue("id", id); + (await update.ExecuteNonQueryAsync()).Should().Be(1); + } + + public async Task PendingTerminalCountAsync() + { + await using var connection = await _dispatchSource.OpenConnectionAsync(); + await using var count = new NpgsqlCommand(""" + SELECT count(*) FROM outbox_messages + WHERE terminal_reason = 'retry_exhausted' AND attempts = 6 + AND processed_at IS NULL AND dead_lettered_at IS NULL + """, connection); + return (long)(await count.ExecuteScalarAsync())!; + } + + public async Task RetainedDeliveryCountAsync(Guid? eventId = null) + { + await using var connection = await _appSource.OpenConnectionAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await SchemaQueries.SetTenantAsync(connection, transaction, SchemaFixture.TenantA); + await using var count = new NpgsqlCommand(""" + SELECT count(*) FROM event_deliveries WHERE tenant_id = @tenant + AND (@event IS NULL OR event_id = @event) + """, connection, transaction); + count.Parameters.AddWithValue("tenant", SchemaFixture.TenantA); + count.Parameters.Add(new NpgsqlParameter("event", NpgsqlDbType.Uuid) { Value = (object?)eventId ?? DBNull.Value }); + return (long)(await count.ExecuteScalarAsync())!; + } + + public async Task SchedulerCanInsertAuditAsync() + { + await using var connection = await _dispatchSource.OpenConnectionAsync(); + await using var command = new NpgsqlCommand("SELECT has_table_privilege(current_user, 'audit_log', 'INSERT')", connection); + return (bool)(await command.ExecuteScalarAsync())!; + } + public async Task AuditInsertGrantAsync(bool grant) { await using var connection = await PostgresFixture.OpenAsync(_database.MigrationConnectionString); @@ -939,6 +1134,8 @@ private sealed class RecordingLogger : ILogger { private readonly ConcurrentQueue _messages = new(); public string Text => string.Join('\n', _messages); + public int TerminalAuditFailures => _messages.Count(message => message.EndsWith( + "outcome terminal_audit_failed.", StringComparison.Ordinal)); public IDisposable? BeginScope(TState state) where TState : notnull => null; public bool IsEnabled(LogLevel logLevel) => true; public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, diff --git a/backend/tests/LearnStack.Tests.Integration/Database/OutboxProducerHttpTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/OutboxProducerHttpTests.cs new file mode 100644 index 00000000..16530ba3 --- /dev/null +++ b/backend/tests/LearnStack.Tests.Integration/Database/OutboxProducerHttpTests.cs @@ -0,0 +1,130 @@ +using System.Diagnostics; +using FluentAssertions; +using LearnStack.Api.Common; +using LearnStack.Infrastructure.Messaging; +using LearnStack.Modules.Tenancy.Application.Contracts.Tenant; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Messaging; +using LearnStack.SharedKernel.Persistence; +using LearnStack.SharedKernel.Tenancy; +using MediatR; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.Mvc; +using Microsoft.AspNetCore.Mvc.Testing; +using Microsoft.AspNetCore.TestHost; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; +using Microsoft.Extensions.Hosting; +using Xunit; + +namespace LearnStack.Tests.Integration.Database; + +public sealed partial class OutboxProducerTests +{ + /// + /// ADR-0057 + /// preserves the HTTP producer's trace and successful transaction announcement in + /// the committed outbox. The test-only transport makes no authorization or + /// production-route admission claim. + /// + [Fact] + [Trait(RequiresDocker.Key, RequiresDocker.Value)] + public async Task An_HTTP_request_commits_the_enqueuing_trace_announced_scope_and_exact_partition() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + var observation = new HttpProducerObservation(); + await using var host = new ConstructionOnlyApiFactory().WithWebHostBuilder(builder => + { + builder.UseEnvironment(Environments.Development); + builder.UseSetting("ConnectionStrings:Default", database.AppConnectionString); + builder.ConfigureTestServices(services => + { + services.AddControllers(options => options.Conventions.Insert(0, + new TestControllerFilter(typeof(OutboxHttpProbeController)))) + .AddApplicationPart(typeof(OutboxHttpProbeController).Assembly); + services.AddSingleton(observation); + services.RemoveAll(); + services.AddScoped(); + services.AddScoped(); + }); + }); + using var client = host.CreateClient(new WebApplicationFactoryClientOptions + { + BaseAddress = new Uri("http://localhost"), + }); + using var request = new HttpRequestMessage(HttpMethod.Post, "/api/v1/outbox-http-probe"); + request.Headers.Add("traceparent", AmbientCorrelation); + using var response = await client.SendAsync(request); + response.EnsureSuccessStatusCode(); + + observation.Calls.Should().Be(1, "actual provisioning translates exactly one saved fact"); + observation.UnresolvedAtEntry.Should().BeTrue( + "the test-only caller uses existing unresolved provisioning and tracing supplies no authority"); + observation.Announcement.Should().NotBeNull(); + var announcement = observation.Announcement!; + announcement.TenantId.Value.Should().Be(SchemaFixture.TenantA); + announcement.OrganizationId.Should().BeNull("provisioning deliberately announces a tenant-wide transaction"); + var row = (await ReadRowAsync(database.AppConnectionString, observation.EventId))!; + row.Should().NotBeNull("a separate application connection must observe the committed row"); + row.TenantId.Should().Be(announcement.TenantId.Value); + row.OrganizationId.Should().Be(announcement.OrganizationId?.Value); + row.PartitionKey.Should().NotBeNullOrEmpty().And.Be(observation.PartitionKey); + row.CorrelationId.Should().Be(observation.EnqueueActivity); + ActivityContext.TryParse(row.CorrelationId, null, out var persisted).Should().BeTrue(); + ActivityContext.TryParse(observation.EnqueueActivity, null, out var enqueuing).Should().BeTrue(); + ActivityContext.TryParse(observation.HttpActivity, null, out var http).Should().BeTrue(); + ActivityContext.TryParse(AmbientCorrelation, null, out var incoming).Should().BeTrue(); + persisted.TraceId.Should().Be(enqueuing.TraceId).And.Be(http.TraceId).And.Be(incoming.TraceId); + } + + private sealed class HttpObservedOutbox( + PostgresOutbox actual, IUnitOfWork unit, HttpProducerObservation observation) : IOutbox + { + public async Task EnqueueAsync(IIntegrationEvent integrationEvent, CancellationToken cancellationToken = default) + { + observation.Calls++; + observation.EventId = integrationEvent.EventId; + observation.PartitionKey = integrationEvent.PartitionKey; + observation.EnqueueActivity = Activity.Current?.Id; + observation.Announcement = unit.GetTenantContextAnnouncement(unit.Transaction); + (await ScalarInUnitAsync(unit, "SELECT current_user")).Should().Be("learnstack_app"); + (await ScalarInUnitAsync(unit, + "SELECT (rolbypassrls OR rolsuper)::text FROM pg_roles WHERE rolname = current_user")) + .Should().Be("false"); + await actual.EnqueueAsync(integrationEvent, cancellationToken); + } + } +} + +/// Observation only; no request metadata supplies tenancy or transaction authority. +internal sealed class HttpProducerObservation +{ + public int Calls { get; set; } + public Guid EventId { get; set; } + public string? PartitionKey { get; set; } + public string? HttpActivity { get; set; } + public string? EnqueueActivity { get; set; } + public TenantContextAnnouncement? Announcement { get; set; } + public bool UnresolvedAtEntry { get; set; } +} + +/// +/// Test-only HTTP transport for the existing tests-only provisioning caller. It does +/// not introduce a production route or exercise authorization/route admission policy. +/// +[Route("outbox-http-probe")] +public sealed class OutboxHttpProbeController(ISender sender, IServiceProvider services) + : ApiControllerBase, ITestOnlyController +{ + [HttpPost] + public async Task Produce(CancellationToken cancellationToken) + { + var observation = services.GetRequiredService(); + observation.HttpActivity = Activity.Current?.Id; + observation.UnresolvedAtEntry = !services.GetRequiredService().IsResolved; + var result = await sender.Send(new ProvisionTenantCommand( + TenantId.From(SchemaFixture.TenantA), "http-producer-proof", "HTTP producer proof", + OrganizationId.From(Guid.CreateVersion7()), "hq", "HQ"), cancellationToken); + return result.ToActionResult(); + } +} diff --git a/backend/tests/LearnStack.Tests.Integration/Database/OutboxProducerTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/OutboxProducerTests.cs index 4dcb6719..c7c356ed 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/OutboxProducerTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/OutboxProducerTests.cs @@ -37,7 +37,7 @@ namespace LearnStack.Tests.Integration.Database; /// [Trait(RequiresDocker.Key, RequiresDocker.Value)] [Collection(SharedSchema.Name)] -public sealed class OutboxProducerTests(SchemaFixture schema) +public sealed partial class OutboxProducerTests(SchemaFixture schema) { private const string DirectOperation = "customization.content_type.outbox_probe"; private const string AmbientCorrelation = "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"; diff --git a/backend/tests/LearnStack.Tests.Integration/Database/PlatformAdminScopeTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/PlatformAdminScopeTests.cs index 8050a9f9..44f0edff 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/PlatformAdminScopeTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/PlatformAdminScopeTests.cs @@ -712,6 +712,7 @@ private static IServiceScopeFactory AuditStoreScopeFactory() => .AddScoped() .AddSingleton() .AddSingleton(new AuditCatalog([new TenancyAuditCatalogSource()])) + .AddSingleton(StaticTenantContextAccessor.Unresolved) .AddSingleton(new Lazy( () => throw new InvalidOperationException( "The platform-scope row is written on the scope's own connection, so " diff --git a/backend/tests/LearnStack.Tests.Integration/Database/PublicReadTransactionTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/PublicReadTransactionTests.cs index b0383421..639439f1 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/PublicReadTransactionTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/PublicReadTransactionTests.cs @@ -315,7 +315,8 @@ private ServiceProvider BuildProvider(ILoggerFactory? injectedLogs = null) private static PostgresAuditStore Store(IServiceProvider services, AuditStateCapture capture) => new( capture, new Lazy(services.GetRequiredService), NullLogger.Instance, - services.GetRequiredService(), new AuditHealth(), new AuditCatalog([])); + services.GetRequiredService(), new AuditHealth(), new AuditCatalog([]), + services.GetRequiredService()); private static async Task AssertRoleAsync(IUnitOfWork unit) { diff --git a/backend/tests/LearnStack.Tests.Integration/Database/SchemaFixture.cs b/backend/tests/LearnStack.Tests.Integration/Database/SchemaFixture.cs index 046b0341..fcf4d68e 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/SchemaFixture.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/SchemaFixture.cs @@ -58,7 +58,7 @@ public sealed class SchemaFixture : IAsyncLifetime public const string HostB = "beta.example.com"; /// - /// The twenty-one tables the five chains create, used only to prove that a + /// The twenty-three tables the five chains create, used only to prove that a /// catalogue sweep read something. /// /// @@ -71,7 +71,7 @@ public sealed class SchemaFixture : IAsyncLifetime "tenants", "organizations", "tenant_domains", "tenant_locales", "tenant_settings", "tenant_feature_flags", "platform_entitlement_cache", "platform_host_to_tenant", - "outbox_messages", "idempotency_keys", + "outbox_messages", "idempotency_keys", "event_deliveries", "inbox_messages", "tenant_content_types", "tenant_level_taxonomies", "tenant_level_taxonomy_items", "customization_generations", "audit_log", "audit_config", @@ -94,6 +94,8 @@ public sealed class SchemaFixture : IAsyncLifetime ["platform_host_to_tenant"] = 1, ["outbox_messages"] = 1, ["idempotency_keys"] = 1, + ["event_deliveries"] = 1, + ["inbox_messages"] = 1, ["tenant_content_types"] = 1, ["tenant_level_taxonomies"] = 1, ["tenant_level_taxonomy_items"] = 1, @@ -127,6 +129,8 @@ public sealed class SchemaFixture : IAsyncLifetime ["platform_host_to_tenant"] = 1, ["outbox_messages"] = 1, ["idempotency_keys"] = 1, + ["event_deliveries"] = 1, + ["inbox_messages"] = 1, ["tenant_content_types"] = 1, ["tenant_level_taxonomies"] = 1, ["tenant_level_taxonomy_items"] = 1, @@ -154,11 +158,54 @@ public async Task InitializeAsync() await MigrationChains.ApplyAllAsync(Postgres.MigrationConnectionString); await SeedAsync(); + await SeedDeliveryAsync(); await EducationSchemaSeed.SeedAsync(Postgres.AppConnectionString); } public async Task DisposeAsync() => await Postgres.DisposeAsync(); + private async Task SeedDeliveryAsync() + { + await using var app = await PostgresFixture.OpenAsync(Postgres.AppConnectionString); + await SchemaQueries.AssertApplicationRoleAsync(app); + (Guid Tenant, Guid? Organization)[] scopes = + [ + (TenantA, null), (TenantA, OrgA1), (TenantA, OrgA2), + (TenantB, null), (TenantB, OrgB1), + ]; + foreach (var (tenant, organization) in scopes) + { + await using var transaction = await app.BeginTransactionAsync(); + await SchemaQueries.SetTenantAsync(app, transaction, tenant); + await SchemaQueries.SetSettingAsync(app, transaction, "app.organization_id", organization?.ToString() ?? string.Empty); + await SchemaQueries.ExecuteAsync(app, transaction, """ + INSERT INTO event_deliveries + (tenant_id, subscription_id, event_id, organization_id, occurred_at, + correlation_id, type, topic, partition_key, payload) + VALUES (@tenant, 'schema.fixture.v1', @event, @organization, clock_timestamp(), + '00-11111111111111111111111111111111-2222222222222222-01', + 'schema.fixture', 'learnstack.schema.fixture', @partition, '{"fixture":true}'); + UPDATE event_deliveries SET processed_at = clock_timestamp() + WHERE tenant_id = @tenant AND event_id = @event + AND subscription_id = 'schema.fixture.v1'; + """, ("tenant", tenant), ("event", Guid.CreateVersion7()), + ("organization", organization is { } org ? org : DBNull.Value), + ("partition", tenant.ToString())); + await transaction.CommitAsync(); + } + + foreach (var tenant in new[] { TenantA, TenantB }) + { + await using var transaction = await app.BeginTransactionAsync(); + await SchemaQueries.SetTenantAsync(app, transaction, tenant); + await SchemaQueries.ExecuteAsync(app, transaction, """ + INSERT INTO inbox_messages (tenant_id, subscription_id, event_id) + VALUES (@tenant, 'schema.fixture.v1', @event); + """, ("tenant", tenant), ("event", Guid.CreateVersion7())); + await transaction.CommitAsync(); + } + } + private async Task SeedAsync() { await using (var owner = await PostgresFixture.OpenAsync(Postgres.MigrationConnectionString)) diff --git a/backend/tests/LearnStack.Tests.Integration/Database/SeederTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/SeederTests.cs index f3bc57ce..4de969d8 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/SeederTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/SeederTests.cs @@ -172,10 +172,8 @@ SELECT count(*) FROM tenant_level_taxonomy_items """, "tenant", tenant.TenantId.Value)) .Should().Be(3L, "a level vocabulary with no bands resolves everything to nothing"); - // The counter every cache key embeds. Both built-in and declared definitions - // each register and publish; a reader holding a pre-seed key cannot reach a - // stale entry — and a generation still at its default would mean the writes - // happened without invalidating anything. + // Each built-in initialization bumps once; each authored definition registers + // and publishes. A reader holding a pre-seed key cannot reach stale data. (await ScalarAsPlatformAsync(""" SELECT generation FROM customization_generations WHERE tenant_id = @tenant """, "tenant", tenant.TenantId.Value)) @@ -228,9 +226,8 @@ public async Task Running_the_seed_twice_changes_nothing_and_still_succeeds() (await ScalarAsPlatformAsync("SELECT count(*) FROM organizations WHERE tenant_id = ANY(@ids)", "ids", SeedData.All.Select(tenant => tenant.TenantId.Value).ToArray())) .Should().Be(4L); - // The built-ins have two ways to double that the tenancy rows do not: the - // register conflicts on a versioned key, and the publish refuses because the - // definition is already live. Both are "already seeded" and neither may write. + // Semantic built-in verification and exact authored verification both skip + // before a writer, so no initialization attempt or publication is repeated. // Scoped to the seed's own tenants: the shared fixture seeds customization rows // for two OTHER tenants, and an unfiltered count would be measuring both. (await ScalarAsPlatformAsync( @@ -284,7 +281,7 @@ public async Task A_failure_that_is_not_a_conflict_stops_the_run() [Theory] [InlineData("content-type")] [InlineData("taxonomy")] - public async Task A_built_in_whose_id_another_tenant_holds_stops_the_run(string subject) + public async Task An_authored_definition_whose_id_another_tenant_holds_stops_the_run(string subject) { // The ownership verification, for the two customization acts. A definition's // id is a uuid primary key and therefore GLOBAL, while its key is per tenant — @@ -306,10 +303,18 @@ public async Task A_built_in_whose_id_another_tenant_holds_stops_the_run(string (await Runner(dataSource) .RunAsync(CancellationToken.None, [SeedData.English])).Should().Be(0); - // The second tenant, re-declared with the first tenant's built-in id. + // Authored IDs remain exact even though built-in IDs are discovered semantically. var collidingYoga = subject == "content-type" - ? SeedData.Yoga with { BuiltInContentTypeId = SeedData.English.BuiltInContentTypeId } - : SeedData.Yoga with { BuiltInTaxonomyId = SeedData.English.BuiltInTaxonomyId }; + ? SeedData.Yoga with + { + Curriculum = SeedData.Yoga.Curriculum! with + { ContentType = SeedData.Yoga.Curriculum!.ContentType with { Id = SeedData.English.Curriculum!.ContentType.Id } } + } + : SeedData.Yoga with + { + Curriculum = SeedData.Yoga.Curriculum! with + { Taxonomy = SeedData.Yoga.Curriculum!.Taxonomy with { Id = SeedData.English.Curriculum!.Taxonomy.Id } } + }; var seed = async () => await Runner(dataSource) .RunAsync(CancellationToken.None, [collidingYoga]); @@ -600,8 +605,21 @@ ServiceProvider Compose(ITenantContext? context) outcomes.Should().OnlyContain(code => code == 0); (await Runner(source).RunAsync(CancellationToken.None)).Should().Be(0); await AssertInventoryAsync(source); + var preservedInitializations = await ScalarAsPlatformAsync(""" + SELECT count(*) FROM audit_log + WHERE operation IN ('customization.content_type.initialize', 'customization.level_taxonomy.initialize') + AND outcome = 'success' AND entity_id IS NULL AND before_state IS NULL AND after_state IS NULL + AND changes IS NULL + """); + preservedInitializations.Should().BeInRange(0, 2, "each competing runner can preserve each default once"); (await ScalarAsPlatformAsync("SELECT count(*) FROM audit_log WHERE outcome = 'success'")) - .Should().Be(SeedData.ExpectedAuditWrites, "only one successful writer wins each act"); + .Should().Be(SeedData.ExpectedAuditWrites + preservedInitializations, + "only one successful writer wins each act; additional successes are precisely classified no-change initializers"); + (await ScalarAsPlatformAsync(""" + SELECT count(*) FROM audit_log + WHERE operation IN ('customization.content_type.initialize', 'customization.level_taxonomy.initialize') + AND outcome = 'success' AND after_state IS NOT NULL + """)).Should().Be(SeedData.All.Count * 2L, "every built-in has exactly one creating initialization"); var completed = await SnapshotAsync(); (await Runner(source).RunAsync(CancellationToken.None)).Should().Be(0); (await SnapshotAsync()).Should().Be(completed); @@ -723,6 +741,8 @@ SELECT count(*) FROM pg_stat_activity [InlineData("host scope")] [InlineData("locale")] [InlineData("undeclared locale")] + [InlineData("type id")] + [InlineData("taxonomy id")] [InlineData("type key")] [InlineData("type revision")] [InlineData("schema")] @@ -760,6 +780,8 @@ public async Task Completed_seed_mismatch_fails_without_mutating_any_existing_ro "host scope" => original with { MapHostToDefaultOrganization = true }, "locale" => original with { Curriculum = content with { Locales = [content.Locales[0] with { Sort = 4 }] } }, "undeclared locale" => original, + "type id" => original with { Curriculum = content with { ContentType = content.ContentType with { Id = Guid.CreateVersion7() } } }, + "taxonomy id" => original with { Curriculum = content with { Taxonomy = content.Taxonomy with { Id = Guid.CreateVersion7() } } }, "type key" => original with { Curriculum = content with { ContentType = content.ContentType with { Key = "different" } } }, "type revision" => original with { Curriculum = content with { ContentType = content.ContentType with { SchemaVersion = 2 } } }, "schema" => original with { Curriculum = content with { ContentType = content.ContentType with { JsonSchema = "{}" } } }, @@ -818,28 +840,34 @@ static string Reordered(string value) [InlineData(false, true)] [InlineData(true, false)] [InlineData(true, true)] - public async Task Seed_publication_refuses_a_different_active_revision_without_retiring_or_rebinding_it(bool taxonomy, bool expectedDraftExists) + public async Task Built_in_verification_refuses_a_different_active_revision_without_retiring_or_rebinding_it(bool taxonomy, bool expectedDraftExists) { await using var source = DataSource(); var tenant = SeedData.English with { Curriculum = null }; var interrupted = new SeedRunner(context => SeedComposition.Build(source, context, NullLoggerFactory.Instance), - new InterruptAfter(taxonomy ? (expectedDraftExists ? "level taxonomy" : "content type publication") - : (expectedDraftExists ? "content type" : "host mapping"))); + new InterruptAfter("host mapping")); var initial = () => interrupted.RunAsync(CancellationToken.None, [tenant]); await initial.Should().ThrowAsync(); var incumbent = Guid.CreateVersion7(); var context = new SeedTenantContext(tenant.TenantId, null); if (taxonomy) { - var definition = SeedData.Taxonomies(tenant).First(); - await Write(new RegisterTenantLevelTaxonomyCommand(incumbent, definition.Key, definition.SchemaVersion + 1, - definition.DisplayName, [.. definition.Bands.Select(band => new TaxonomyItemInput(band.Key, band.DisplayName, band.Sort, band.Metadata))])); + await Write(new InitializeTenantContentTypeDefaultCommand()); + var definition = BuiltInCustomizations.Plain; + if (expectedDraftExists) + await Write(new RegisterTenantLevelTaxonomyCommand(Guid.CreateVersion7(), definition.Key, 1, + definition.DisplayName, [.. definition.Bands.Select(band => new TaxonomyItemInput(band.Key, band.DisplayName, band.Sort))])); + await Write(new RegisterTenantLevelTaxonomyCommand(incumbent, definition.Key, BuiltInCustomizations.SchemaVersion + 1, + definition.DisplayName, [.. definition.Bands.Select(band => new TaxonomyItemInput(band.Key, band.DisplayName, band.Sort))])); await Write(new PublishTenantLevelTaxonomyCommand(incumbent)); } else { - var definition = SeedData.ContentTypes(tenant).First(); - await Write(new RegisterTenantContentTypeCommand(incumbent, definition.Key, definition.SchemaVersion + 1, + var definition = BuiltInCustomizations.Card; + if (expectedDraftExists) + await Write(new RegisterTenantContentTypeCommand(Guid.CreateVersion7(), definition.Key, 1, + definition.DisplayName, definition.JsonSchema, definition.RendererKey)); + await Write(new RegisterTenantContentTypeCommand(incumbent, definition.Key, BuiltInCustomizations.SchemaVersion + 1, definition.DisplayName, definition.JsonSchema, definition.RendererKey)); await Write(new PublishTenantContentTypeCommand(incumbent)); } @@ -861,6 +889,120 @@ async Task Write(IRequest> request) } } + [Fact] + public async Task Matching_built_ins_with_independently_generated_ids_are_preserved_without_any_writer() + { + await using var source = DataSource(); + var tenant = SeedData.English with { Curriculum = null }; + var interrupted = new SeedRunner(context => SeedComposition.Build(source, context, NullLoggerFactory.Instance), + new InterruptAfter("host mapping")); + await ((Func)(() => interrupted.RunAsync(CancellationToken.None, [tenant]))) + .Should().ThrowAsync(); + var typeId = Guid.CreateVersion7(); + var taxonomyId = Guid.CreateVersion7(); + var type = BuiltInCustomizations.Card; + var taxonomy = BuiltInCustomizations.Plain; + var context = new SeedTenantContext(tenant.TenantId, null); + await WriteAsync(source, context, new RegisterTenantContentTypeCommand(typeId, type.Key, 1, + type.DisplayName, type.JsonSchema, type.RendererKey)); + await WriteAsync(source, context, new PublishTenantContentTypeCommand(typeId)); + await WriteAsync(source, context, new RegisterTenantLevelTaxonomyCommand(taxonomyId, taxonomy.Key, 1, + taxonomy.DisplayName, [.. taxonomy.Bands.Select(band => new TaxonomyItemInput(band.Key, band.DisplayName, band.Sort))])); + await WriteAsync(source, context, new PublishTenantLevelTaxonomyCommand(taxonomyId)); + var before = await SnapshotAsync(); + (await Runner(source).RunAsync(CancellationToken.None, [tenant])).Should().Be(0); + (await SnapshotAsync()).Should().Be(before, "semantically complete warm defaults skip all initialization attempts"); + (await ReadAsync(source, context, new GetActiveContentTypeSeedRevisionQuery(type.Key))).State!.Id.Should().Be(typeId); + (await ReadAsync(source, context, new GetActiveTaxonomySeedRevisionQuery(taxonomy.Key))).State!.Id.Should().Be(taxonomyId); + } + + [Theory] + [InlineData("type labels")] + [InlineData("type schema")] + [InlineData("type renderer")] + [InlineData("taxonomy labels")] + [InlineData("band key")] + [InlineData("band labels")] + [InlineData("band sort")] + [InlineData("band metadata")] + [InlineData("missing band")] + public async Task A_live_built_in_with_a_different_semantic_shape_is_refused_before_any_writer(string mismatch) + { + await using var source = DataSource(); + var tenant = SeedData.English with { Curriculum = null }; + (await Runner(source).RunAsync(CancellationToken.None, [tenant])).Should().Be(0); + var sql = mismatch switch + { + "type labels" => """UPDATE tenant_content_types SET display_name = '{"en":"Different"}' WHERE tenant_id = @tenant AND key = 'card'""", + "type schema" => "UPDATE tenant_content_types SET json_schema = '{}' WHERE tenant_id = @tenant AND key = 'card'", + "type renderer" => "UPDATE tenant_content_types SET renderer_key = 'different' WHERE tenant_id = @tenant AND key = 'card'", + "taxonomy labels" => """UPDATE tenant_level_taxonomies SET display_name = '{"en":"Different"}' WHERE tenant_id = @tenant AND key = 'plain'""", + "band key" => "UPDATE tenant_level_taxonomy_items SET key = 'different' WHERE tenant_id = @tenant AND taxonomy_key = 'plain' AND key = 'beginner'", + "band labels" => """UPDATE tenant_level_taxonomy_items SET display_name = '{"en":"Different"}' WHERE tenant_id = @tenant AND taxonomy_key = 'plain' AND key = 'beginner'""", + "band sort" => "UPDATE tenant_level_taxonomy_items SET sort = 99 WHERE tenant_id = @tenant AND taxonomy_key = 'plain' AND key = 'beginner'", + "band metadata" => "UPDATE tenant_level_taxonomy_items SET metadata = '{}' WHERE tenant_id = @tenant AND taxonomy_key = 'plain' AND key = 'beginner'", + "missing band" => "DELETE FROM tenant_level_taxonomy_items WHERE tenant_id = @tenant AND taxonomy_key = 'plain' AND key = 'beginner'", + _ => throw new ArgumentOutOfRangeException(nameof(mismatch)), + }; + await ChangeFixtureAsync(sql, tenant); + var before = await SnapshotAsync(); + var run = () => Runner(source).RunAsync(CancellationToken.None, [tenant]); + (await run.Should().ThrowAsync()).WithMessage("*Seed mismatch*"); + (await SnapshotAsync()).Should().Be(before, "the Active row is inspected in full before choosing an initializer"); + } + + [Theory] + [InlineData(false, "Draft")] + [InlineData(false, "Deprecated")] + [InlineData(false, "Deleted")] + [InlineData(true, "Draft")] + [InlineData(true, "Deprecated")] + [InlineData(true, "Deleted")] + public async Task Preserved_built_in_history_without_an_active_default_refuses_without_business_mutation(bool taxonomy, string state) + { + await using var source = DataSource(); + var tenant = SeedData.English with { Curriculum = null }; + (await Runner(source).RunAsync(CancellationToken.None, [tenant])).Should().Be(0); + var table = taxonomy ? "tenant_level_taxonomies" : "tenant_content_types"; + var key = taxonomy ? BuiltInCustomizations.Plain.Key : BuiltInCustomizations.Card.Key; + var assignment = state == "Deleted" ? "deleted_at = now()" : $"status = '{state}'"; + await ChangeFixtureAsync($"UPDATE {table} SET {assignment} WHERE tenant_id = @tenant AND key = '{key}'", tenant); + var before = await SnapshotAsync(includeAudit: false); + var audits = await AuditRowsAsync(); + var operation = taxonomy ? "customization.level_taxonomy.initialize" : "customization.content_type.initialize"; + var previous = await ScalarAsPlatformAsync("SELECT count(*) FROM audit_log WHERE operation = @operation", "operation", operation); + var run = () => Runner(source).RunAsync(CancellationToken.None, [tenant]); + (await run.Should().ThrowAsync()).WithMessage("*Seed mismatch*"); + (await SnapshotAsync(includeAudit: false)).Should().Be(before, + "the shared create-only initializer preserves every lifecycle and soft-deleted revision"); + (await AuditRowsAsync()).Should().HaveCount(audits.Length + 1).And.Contain(audits); + (await ScalarAsPlatformAsync(""" + SELECT count(*) FROM audit_log WHERE operation = @operation AND outcome = 'success' + AND entity_id IS NULL AND before_state IS NULL AND after_state IS NULL + AND changes IS NULL + """, "operation", operation)).Should().Be(1, "the only additional row classifies the preserved initialization attempt"); + (await ScalarAsPlatformAsync("SELECT count(*) FROM audit_log WHERE operation = @operation", "operation", operation)) + .Should().Be(previous + 1); + } + + private async Task ChangeFixtureAsync(string sql, SeedTenant tenant) + { + await using var connection = await PostgresFixture.OpenAsync(_database.AppConnectionString); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.SetTenantAsync(connection, transaction, tenant.TenantId.Value); + await using var command = new NpgsqlCommand(sql, (NpgsqlConnection)connection, (NpgsqlTransaction)transaction); + command.Parameters.AddWithValue("tenant", tenant.TenantId.Value); + (await command.ExecuteNonQueryAsync()).Should().Be(1, "the deliberate test-owned mismatch must exist"); + await transaction.CommitAsync(); + } + + private static async Task WriteAsync(NpgsqlDataSource source, ITenantContext context, IRequest> request) + { + await using var provider = SeedComposition.Build(source, context, NullLoggerFactory.Instance); + await using var scope = provider.CreateAsyncScope(); + (await scope.ServiceProvider.GetRequiredService().Send(request)).IsSuccess.Should().BeTrue(); + } + // ── Harness ────────────────────────────────────────────────────────────── /// @@ -961,7 +1103,7 @@ private static async Task ProvisionForeignAsync(NpgsqlDataSource source) result.IsSuccess.Should().BeTrue(); } - private async Task SnapshotAsync() + private async Task SnapshotAsync(bool includeAudit = true) { var output = new SortedDictionary(StringComparer.Ordinal); var ids = SeedData.All.Select(tenant => tenant.TenantId.Value).ToArray(); @@ -970,6 +1112,7 @@ private async Task SnapshotAsync() "tenant_content_types", "tenant_level_taxonomies", "tenant_level_taxonomy_items", "customization_generations", "courses", "lessons", "course_translations", "lesson_translations", "audit_log" }) { + if (table == "audit_log" && !includeAudit) continue; var column = table == "tenants" ? "id" : "tenant_id"; // Table/column are a closed test-owned list, never external SQL input. await using var query = new NpgsqlCommand($"SELECT COALESCE(jsonb_agg(to_jsonb(row) ORDER BY to_jsonb(row)::text), '[]'::jsonb)::text FROM {table} row WHERE {column} = ANY(@ids)", (NpgsqlConnection)connection); diff --git a/backend/tests/LearnStack.Tests.Integration/Database/TenancySchemaTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/TenancySchemaTests.cs index caf1b085..109e8315 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/TenancySchemaTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/TenancySchemaTests.cs @@ -429,6 +429,20 @@ INSERT INTO platform_host_to_tenant (host, tenant_id, is_active, is_publicly_liv INSERT INTO outbox_messages (tenant_id, correlation_id, type, topic, partition_key, payload) VALUES (@foreign, '00-sneak-span-01', 'Sneak', 'learnstack.tenancy.tenant', 'k', '{}') """, + ["event_deliveries"] = + """ + INSERT INTO event_deliveries + (tenant_id, subscription_id, event_id, organization_id, occurred_at, + correlation_id, type, topic, partition_key, payload) + VALUES (@foreign, 'schema.fixture.v1', uuidv7(), NULL, now(), + '00-11111111111111111111111111111111-2222222222222222-01', + 'schema.fixture', 'learnstack.schema.fixture', @foreign::text, '{}') + """, + ["inbox_messages"] = + """ + INSERT INTO inbox_messages (tenant_id, subscription_id, event_id) + VALUES (@foreign, 'schema.fixture.v1', uuidv7()) + """, ["idempotency_keys"] = """ INSERT INTO idempotency_keys (tenant_id, key, fingerprint, claim_token, state, expires_at) @@ -1204,6 +1218,10 @@ AND grantee IN ('learnstack_app', 'learnstack_platform', 'learnstack_outbox_admi "learnstack_app platform_host_to_tenant DELETE,INSERT,SELECT,UPDATE", "learnstack_app outbox_messages INSERT,SELECT", "learnstack_app idempotency_keys INSERT,SELECT,UPDATE", + // INSERT acceptance/claim columns and UPDATE completion are column + // grants, checked with their actual denial/success probes separately. + "learnstack_app event_deliveries SELECT", + "learnstack_app inbox_messages SELECT", "learnstack_platform tenants DELETE,INSERT,SELECT,UPDATE", "learnstack_platform organizations DELETE,INSERT,SELECT,UPDATE", "learnstack_platform tenant_domains SELECT", @@ -1249,6 +1267,7 @@ AND grantee IN ('learnstack_app', 'learnstack_platform', 'learnstack_outbox_admi "learnstack_platform lesson_translations SELECT", ], "every line is one line of the migration chains' grant matrices, and " - + "learnstack_outbox_admin holds only outbox SELECT/status UPDATE and terminal audit INSERT"); + + "learnstack_outbox_admin holds only declared outbox/delivery scheduling columns " + + "and terminal audit INSERT"); } } diff --git a/backend/tests/LearnStack.Tests.Integration/Database/TenantAssertionAuditTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/TenantAssertionAuditTests.cs index bdbdf611..022cad36 100644 --- a/backend/tests/LearnStack.Tests.Integration/Database/TenantAssertionAuditTests.cs +++ b/backend/tests/LearnStack.Tests.Integration/Database/TenantAssertionAuditTests.cs @@ -4,6 +4,7 @@ using LearnStack.Infrastructure.Audit; using LearnStack.Modules.Tenancy.Application.Audit; using LearnStack.SharedKernel.Audit; +using LearnStack.SharedKernel.Tenancy; using LearnStack.SharedKernel.Time; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Logging.Abstractions; @@ -49,7 +50,7 @@ public async Task An_authenticated_mismatch_lands_a_denied_row_the_tenant_can_re NullLogger.Instance, MeterFactory, new AuditHealth(), - new AuditCatalog([])); + new AuditCatalog([]), StaticTenantContextAccessor.Unresolved); var recorder = Recorder(store, threshold: 10); @@ -101,7 +102,7 @@ public async Task An_anonymous_burst_lands_one_row_however_long_the_run_is() NullLogger.Instance, MeterFactory, new AuditHealth(), - new AuditCatalog([])); + new AuditCatalog([]), StaticTenantContextAccessor.Unresolved); var recorder = Recorder(store, threshold: 3); diff --git a/backend/tests/LearnStack.Tests.Integration/Database/UnitOfWorkCommitCleanupTests.cs b/backend/tests/LearnStack.Tests.Integration/Database/UnitOfWorkCommitCleanupTests.cs new file mode 100644 index 00000000..703e7ffc --- /dev/null +++ b/backend/tests/LearnStack.Tests.Integration/Database/UnitOfWorkCommitCleanupTests.cs @@ -0,0 +1,224 @@ +using System.Data; +using System.Data.Common; +using System.Reflection; +using FluentAssertions; +using LearnStack.Infrastructure.Persistence; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Tenancy; +using Microsoft.Extensions.Logging.Abstractions; +using Npgsql; +using Xunit; + +namespace LearnStack.Tests.Integration.Database; + +/// Physical COMMIT acknowledgement is distinct from owned transaction cleanup. +[Trait(RequiresDocker.Key, RequiresDocker.Value)] +[Collection(SharedSchema.Name)] +public sealed class UnitOfWorkCommitCleanupTests(SchemaFixture schema) +{ + private const string Subscription = "proof.commit-cleanup.v1"; + private static readonly CleanupContext Context = new(); + + [Fact] + public async Task Acknowledged_commit_precedes_a_disposal_fault_and_remains_durable() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var unit = new NpgsqlUnitOfWork(source, NullLogger.Instance); + var frame = await unit.BeginTransactionAsync(); + await unit.SetTenantContextAsync(Context); + var marker = await InsertMarkerAsync(unit); + var cleanupFailure = new IOException("cleanup proof fault"); + var transaction = Interpose(unit, cleanupFailure: cleanupFailure); + + await frame.CompleteAsync(); + + transaction.Commits.Should().Be(1); + transaction.Disposals.Should().Be(0, "physical acknowledgement must reach the owner before cleanup"); + unit.HasActiveTransaction.Should().BeFalse(); + (await CountMarkerAsync(source, marker)).Should().Be(1); + await FluentActions.Awaiting(() => frame.DisposeAsync().AsTask()) + .Should().ThrowAsync().Where(failure => ReferenceEquals(failure, cleanupFailure)); + unit.IsRollbackOnly.Should().BeTrue("failed cleanup prevents reuse, not the preceding durable commit"); + (await CountMarkerAsync(source, marker)).Should().Be(1); + await FluentActions.Awaiting(() => unit.BeginTransactionAsync()) + .Should().ThrowAsync(); + } + + [Fact] + public async Task Hanging_cleanup_remains_owned_after_acknowledged_commit() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var unit = new NpgsqlUnitOfWork(source, NullLogger.Instance); + var frame = await unit.BeginTransactionAsync(); + await unit.SetTenantContextAsync(Context); + var marker = await InsertMarkerAsync(unit); + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var transaction = Interpose(unit, releaseCleanup: release.Task); + + await frame.CompleteAsync().WaitAsync(TimeSpan.FromSeconds(5)); + transaction.Disposals.Should().Be(0); + var cleanup = frame.DisposeAsync().AsTask(); + try + { + await transaction.CleanupEntered.Task.WaitAsync(TimeSpan.FromSeconds(5)); + cleanup.IsCompleted.Should().BeFalse("the actual cleanup callback still owns its transaction"); + unit.HasActiveTransaction.Should().BeFalse(); + (await CountMarkerAsync(source, marker)).Should().Be(1); + } + finally + { + release.TrySetResult(); + await cleanup.WaitAsync(TimeSpan.FromSeconds(5)); + } + transaction.Disposals.Should().Be(1); + } + + [Fact] + public async Task Reuse_cleans_the_previous_commit_and_a_stale_frame_cannot_clean_the_next_one() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + await using var unit = new NpgsqlUnitOfWork(source, NullLogger.Instance); + var first = await unit.BeginTransactionAsync(); + await unit.SetTenantContextAsync(Context); + var marker = await InsertMarkerAsync(unit); + var previous = Interpose(unit); + await first.CompleteAsync(); + previous.Disposals.Should().Be(0); + + var second = await unit.BeginTransactionAsync(); + previous.Disposals.Should().Be(1, "reuse must clean the completed transaction before BEGIN"); + await unit.SetTenantContextAsync(Context); + var current = Interpose(unit); + await first.DisposeAsync(); + unit.HasActiveTransaction.Should().BeTrue(); + current.Disposals.Should().Be(0, "the first frame belongs to an older generation"); + await second.CompleteAsync(); + current.Disposals.Should().Be(0); + await first.DisposeAsync(); + current.Disposals.Should().Be(0, "stale disposal cannot clean a later committed generation either"); + await second.DisposeAsync(); + current.Disposals.Should().Be(1); + (await CountMarkerAsync(source, marker)).Should().Be(1); + } + + [Fact] + public async Task Unknown_commit_preserves_its_primary_exception_when_disposal_also_fails() + { + await using var database = await DisposableSchemaDatabase.CreateAsync(schema.Postgres); + await using var source = ApplicationDataSource.Build(database.AppConnectionString); + var unit = new NpgsqlUnitOfWork(source, NullLogger.Instance); + var frame = await unit.BeginTransactionAsync(); + await unit.SetTenantContextAsync(Context); + var marker = await InsertMarkerAsync(unit); + var primary = new IOException("commit proof fault"); + var transaction = Interpose(unit, primary, new IOException("cleanup proof fault")); + try + { + await FluentActions.Awaiting(() => frame.CompleteAsync()).Should() + .ThrowAsync().Where(failure => ReferenceEquals(failure, primary)); + transaction.Commits.Should().Be(1); + transaction.Disposals.Should().Be(1); + unit.HasActiveTransaction.Should().BeFalse(); + await frame.DisposeAsync(); + } + finally + { + // The injected COMMIT failed before sending; connection cleanup owns + // the real uncommitted transaction even when its wrapper disposal faults. + await unit.DisposeAsync(); + } + (await CountMarkerAsync(source, marker)).Should().Be(0); + } + + private static CleanupTransaction Interpose(NpgsqlUnitOfWork unit, + Exception? commitFailure = null, Exception? cleanupFailure = null, Task? releaseCleanup = null) + { + var field = typeof(NpgsqlUnitOfWork).GetField("_transaction", BindingFlags.Instance | BindingFlags.NonPublic)!; + var current = (DbTransaction)field.GetValue(unit)!; + var wrapper = new CleanupTransaction(current, commitFailure, cleanupFailure, releaseCleanup); + // Test-only failure injection: all SQL/COMMIT still uses the actual app-role + // connection; no production transaction factory or alternate owner is added. + field.SetValue(unit, wrapper); + return wrapper; + } + + private static async Task InsertMarkerAsync(NpgsqlUnitOfWork unit) + { + var eventId = Guid.CreateVersion7(); + await SchemaQueries.AssertApplicationRoleAsync(unit.Connection); + await using var insert = new NpgsqlCommand(""" + INSERT INTO inbox_messages (tenant_id, subscription_id, event_id) + VALUES (@tenant, @subscription, @event) + """, (NpgsqlConnection)unit.Connection, (NpgsqlTransaction)unit.Transaction!); + insert.Parameters.AddWithValue("tenant", SchemaFixture.TenantA); + insert.Parameters.AddWithValue("subscription", Subscription); + insert.Parameters.AddWithValue("event", eventId); + (await insert.ExecuteNonQueryAsync()).Should().Be(1); + return eventId; + } + + private static async Task CountMarkerAsync(NpgsqlDataSource source, Guid eventId) + { + await using var connection = await source.OpenConnectionAsync(); + await using var transaction = await connection.BeginTransactionAsync(); + await SchemaQueries.AssertApplicationRoleAsync(connection); + await SchemaQueries.SetTenantAsync(connection, transaction, SchemaFixture.TenantA); + await using var query = new NpgsqlCommand(""" + SELECT count(*) FROM inbox_messages + WHERE tenant_id = @tenant AND subscription_id = @subscription AND event_id = @event + """, connection, transaction); + query.Parameters.AddWithValue("tenant", SchemaFixture.TenantA); + query.Parameters.AddWithValue("subscription", Subscription); + query.Parameters.AddWithValue("event", eventId); + return (long)(await query.ExecuteScalarAsync())!; + } + + private sealed class CleanupContext : ITenantContext + { + public bool IsResolved => true; + public TenantId TenantId => TenantId.From(SchemaFixture.TenantA); + public OrganizationId? OrganizationId => null; + public UserId? UserId => null; + public string? CorrelationId => null; + public string? ModuleName => "cleanup-proof"; + } + + private sealed class CleanupTransaction(DbTransaction inner, Exception? commitFailure, + Exception? cleanupFailure, Task? releaseCleanup) : DbTransaction + { + public int Commits { get; private set; } + public int Disposals { get; private set; } + public TaskCompletionSource CleanupEntered { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + public override IsolationLevel IsolationLevel => inner.IsolationLevel; + protected override DbConnection? DbConnection => inner.Connection; + public override void Commit() => inner.Commit(); + public override void Rollback() => inner.Rollback(); + public override Task RollbackAsync(CancellationToken cancellationToken = default) => inner.RollbackAsync(cancellationToken); + + public override async Task CommitAsync(CancellationToken cancellationToken = default) + { + Commits++; + if (commitFailure is not null) throw commitFailure; + await inner.CommitAsync(cancellationToken); + } + + public override async ValueTask DisposeAsync() + { + try + { + Disposals++; + CleanupEntered.TrySetResult(); + if (releaseCleanup is not null) await releaseCleanup; + if (cleanupFailure is not null) throw cleanupFailure; + await inner.DisposeAsync(); + } + finally + { + await base.DisposeAsync(); + } + } + } +} diff --git a/backend/tests/LearnStack.Tests.Unit/Api/Composition/OutboxProductionCompositionTests.cs b/backend/tests/LearnStack.Tests.Unit/Api/Composition/OutboxProductionCompositionTests.cs index 3ddb427c..137b5366 100644 --- a/backend/tests/LearnStack.Tests.Unit/Api/Composition/OutboxProductionCompositionTests.cs +++ b/backend/tests/LearnStack.Tests.Unit/Api/Composition/OutboxProductionCompositionTests.cs @@ -2,7 +2,11 @@ using LearnStack.Api.Composition; using LearnStack.Api.Messaging; using LearnStack.Infrastructure.Messaging; +using LearnStack.Modules.Customization.Application.IntegrationEvents; using LearnStack.Modules.Tenancy.Application.Contracts.IntegrationEvents; +using LearnStack.SharedKernel.Hosting; +using LearnStack.SharedKernel.Messaging; +using Microsoft.AspNetCore.Builder; using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Diagnostics.HealthChecks; @@ -31,28 +35,35 @@ public void Production_registers_exactly_one_mandatory_worker_and_the_unmapped_o } [Fact] - public void Real_provisioning_contract_remains_pending_for_both_distinct_default_consumers() + public void Real_provisioning_contract_activates_both_distinct_default_consumers_with_exact_handler_bindings() { - var handlers = IntegrationEventHandlerRegistry.Discover( + var builder = WebApplication.CreateBuilder([]); + builder.Configuration.Sources.Clear(); + builder.AddLearnStackCrossCuttingFoundation(DeploymentMode.Development, typeof(LearnStack.Modules.Tenancy.Application.AssemblyMarker).Assembly, typeof(LearnStack.Modules.Customization.Application.AssemblyMarker).Assembly, typeof(LearnStack.Modules.Education.Application.AssemblyMarker).Assembly); - var services = new ServiceCollection(); - services.AddSingleton(handlers); - services.AddLearnStackPersistence(new ConfigurationBuilder().Build()); - using var provider = services.BuildServiceProvider(); + using var provider = builder.Services.BuildServiceProvider(); + var handlers = provider.GetRequiredService(); var registry = provider.GetRequiredService(); + provider.GetRequiredService().Should().BeOfType( + "the construction-free foundation resolves the transport without database registration"); var identity = typeof(TenantCreatedIntegrationEventV1).AssemblyQualifiedName!; - registry.PendingIdentities.Should().ContainSingle().Which.Should().Be(identity); + registry.PendingIdentities.Should().BeEmpty(); var resolution = registry.Resolve(identity); resolution.EventType.Should().Be(); - resolution.Disposition.Should().Be(OutboxEventDisposition.Pending); + resolution.Disposition.Should().Be(OutboxEventDisposition.Active); resolution.Subscriptions.Select(subscription => subscription.Id).Should().BeEquivalentTo( ["customization.initialize-content-type-default.v1", "customization.initialize-taxonomy-default.v1"]); resolution.Subscriptions.Should().OnlyContain(subscription => - subscription.Readiness == OutboxSubscriptionReadiness.Pending && subscription.HandlerType == null); - handlers.For(typeof(TenantCreatedIntegrationEventV1)).Should().BeEmpty( - "P02b-3 activates real consumers; P02b-2 must preserve their facts"); + subscription.Readiness == OutboxSubscriptionReadiness.Active); + resolution.Subscriptions.Single(subscription => subscription.Id == "customization.initialize-content-type-default.v1") + .HandlerType.Should().Be(); + resolution.Subscriptions.Single(subscription => subscription.Id == "customization.initialize-taxonomy-default.v1") + .HandlerType.Should().Be(); + handlers.All.Should().HaveCount(2, "both production consumers must be discovered"); + handlers.For(typeof(TenantCreatedIntegrationEventV1)).Select(subscription => subscription.HandlerType) + .Should().BeEquivalentTo([typeof(InitializeContentTypeDefaultHandler), typeof(InitializeTaxonomyDefaultHandler)]); } } diff --git a/backend/tests/LearnStack.Tests.Unit/Api/Messaging/DeliveryAttemptTests.cs b/backend/tests/LearnStack.Tests.Unit/Api/Messaging/DeliveryAttemptTests.cs new file mode 100644 index 00000000..0ac4d8f0 --- /dev/null +++ b/backend/tests/LearnStack.Tests.Unit/Api/Messaging/DeliveryAttemptTests.cs @@ -0,0 +1,51 @@ +using FluentAssertions; +using LearnStack.Infrastructure.Messaging; +using Xunit; + +namespace LearnStack.Tests.Unit.Api.Messaging; + +public sealed class DeliveryAttemptTests +{ + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Cancellation_closes_gate_before_blocking_callbacks_and_retains_token_until_actual_cleanup(bool callbackFault) + { + var control = new DeliveryExecutionControl(); + var token = control.Token; + var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var observedClosedGate = false; + var observedLiveToken = false; + using var registration = token.Register(() => + { + observedClosedGate = control.Gate.IsClosed; + entered.TrySetResult(); + release.Task.GetAwaiter().GetResult(); + observedLiveToken = token.WaitHandle.WaitOne(0); + if (callbackFault) throw new InvalidOperationException("private-cancellation-canary"); + }); + Task? finishing = null; + try + { + control.RequestCancellation(); + await entered.Task.WaitAsync(TimeSpan.FromSeconds(5)); + observedClosedGate.Should().BeTrue("the gate decision precedes every cancellation callback"); + control.Gate.IsClosed.Should().BeTrue(); + token.IsCancellationRequested.Should().BeTrue(); + finishing = control.FinishCallbacksAsync(); + finishing.IsCompleted.Should().BeFalse("the live callback still owns cancellation resources"); + control.RequestCancellation(); + } + finally + { + release.TrySetResult(); + finishing ??= control.FinishCallbacksAsync(); + (await finishing.WaitAsync(TimeSpan.FromSeconds(5))).Should().Be(!callbackFault); + control.DisposeCancellation(); + } + observedLiveToken.Should().BeTrue(); + FluentActions.Invoking(() => _ = token.WaitHandle).Should().Throw(); + control.RequestCancellation(); + } +} diff --git a/backend/tests/LearnStack.Tests.Unit/Api/Messaging/DeliveryObservationTests.cs b/backend/tests/LearnStack.Tests.Unit/Api/Messaging/DeliveryObservationTests.cs new file mode 100644 index 00000000..8956ac3d --- /dev/null +++ b/backend/tests/LearnStack.Tests.Unit/Api/Messaging/DeliveryObservationTests.cs @@ -0,0 +1,216 @@ +using System.Diagnostics; +using System.Diagnostics.Metrics; +using System.Reflection; +using FluentAssertions; +using LearnStack.Api.Messaging; +using LearnStack.Infrastructure.Messaging; +using LearnStack.SharedKernel.Messaging; +using LearnStack.Tests.Unit.Application.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Diagnostics.HealthChecks; +using Xunit; + +namespace LearnStack.Tests.Unit.Api.Messaging; + +[Collection(DeliveryCommitGateIsolation.Name)] +public sealed class DeliveryObservationTests +{ + [Fact] + public async Task Health_requires_progress_and_fresh_snapshot_on_the_monotonic_clock() + { + var time = new ManualTimeProvider(); + using var observation = Create(time); + var check = new DeliveryHealthCheck(observation); + observation.IsHealthy.Should().BeFalse(); + observation.Started(); + observation.Progress(); + observation.IsHealthy.Should().BeFalse("there is no successful queue observation yet"); + observation.Refreshed(new(9, 4, 20)); + (await check.CheckHealthAsync(new HealthCheckContext())).Status.Should().Be(HealthStatus.Healthy); + + time.Advance(TimeSpan.FromSeconds(44)); + observation.IsHealthy.Should().BeTrue(); + observation.HeartbeatAge.Should().Be(TimeSpan.FromSeconds(44)); + time.UtcNow = time.UtcNow.AddDays(-30); + observation.IsHealthy.Should().BeTrue("wall-clock adjustments do not reset elapsed progress"); + time.Advance(TimeSpan.FromSeconds(1)); + observation.IsHealthy.Should().BeFalse("45 seconds is the stale boundary"); + observation.Progress(); + observation.IsHealthy.Should().BeFalse("fresh progress cannot conceal a stale SQL snapshot"); + observation.Refreshed(new(8, 3, 25)); + observation.IsHealthy.Should().BeTrue(); + observation.Stopped(); + (await check.CheckHealthAsync(new HealthCheckContext())).Status.Should().Be(HealthStatus.Unhealthy); + } + + [Fact] + public void Failed_refresh_retains_immutable_snapshot_and_terminal_failure_survives_empty_polls() + { + using var observation = Create(new ManualTimeProvider()); + observation.Started(); + observation.Progress(); + observation.Refreshed(new(7, 5, 13)); + var before = observation.Snapshot; + observation.RefreshFailed(); + observation.Snapshot.Queue.Should().BeSameAs(before.Queue); + before.RefreshFailed.Should().BeFalse("published snapshots are immutable"); + observation.IsHealthy.Should().BeFalse(); + observation.Refreshed(new(6, 5, 13)); + observation.IsHealthy.Should().BeTrue(); + + observation.TerminalAuditFailed(); + observation.Progress(); + observation.Refreshed(new(0, 0, 0)); + observation.IsHealthy.Should().BeFalse("a successful empty poll is not terminal audit recovery"); + observation.TerminalAuditSucceeded(); + observation.IsHealthy.Should().BeTrue(); + observation.Outstanding(1); + observation.IsHealthy.Should().BeFalse(); + observation.Outstanding(0); + observation.IsHealthy.Should().BeTrue(); + observation.LoopFailed(); + observation.IsHealthy.Should().BeFalse(); + observation.Progress(); + observation.IsHealthy.Should().BeTrue(); + } + + [Fact] + public void Exact_metric_inventory_has_memory_only_gauges_and_finite_event_labels() + { + var instruments = new List(); + var measurements = new List<(Instrument Instrument, double Value, KeyValuePair[] Tags)>(); + using var listener = new MeterListener(); + listener.InstrumentPublished = (instrument, owner) => + { + if (instrument.Meter.Name != DeliveryObservation.InstrumentationName) return; + lock (instruments) instruments.Add(instrument); + owner.EnableMeasurementEvents(instrument); + }; + listener.SetMeasurementEventCallback((instrument, value, tags, _) => + { + lock (measurements) measurements.Add((instrument, value, tags.ToArray())); + }); + listener.SetMeasurementEventCallback((instrument, value, tags, _) => + { + lock (measurements) measurements.Add((instrument, value, tags.ToArray())); + }); + listener.Start(); + var time = new ManualTimeProvider(); + using var observation = Create(time); + observation.Started(); + observation.Progress(); + observation.Refreshed(new(17, 11, 23)); + observation.Duration(nameof(TestEvent), "proof.subscription.v1", TimeSpan.FromSeconds(3.5)); + Meter ownMeter; + lock (measurements) ownMeter = measurements.Single(value => value.Value == 3.5).Instrument.Meter; + observation.Failed("private-stored-type-and-tenant", "private-subscription"); + observation.Deadletter("retired", "proof.subscription.v1"); + observation.Deduplicated(nameof(TestEvent), "proof.subscription.v1"); + observation.LeaseLost(); + time.Advance(TimeSpan.FromSeconds(8)); + listener.RecordObservableInstruments(); + + Instrument[] ownInstruments; + lock (instruments) ownInstruments = instruments.Where(value => ReferenceEquals(value.Meter, ownMeter)).ToArray(); + ownInstruments.Select(value => value.Name).Should().BeEquivalentTo( + [ + "learnstack_delivery_pending_count", "learnstack_delivery_execute_duration_seconds", + "learnstack_delivery_failed_total", "learnstack_inbox_deadletter_total", + "learnstack_delivery_lease_lost_total", "learnstack_delivery_oldest_eligible_age_seconds", + "learnstack_delivery_deferred_count", "learnstack_delivery_heartbeat_age_seconds", "learnstack_inbox_dedup_total", + ]); + lock (measurements) + { + var own = measurements.Where(value => ReferenceEquals(value.Instrument.Meter, ownMeter)).ToArray(); + own.Single(value => value.Instrument.Name == "learnstack_delivery_pending_count").Value.Should().Be(17); + own.Single(value => value.Instrument.Name == "learnstack_delivery_deferred_count").Value.Should().Be(11); + own.Single(value => value.Instrument.Name == "learnstack_delivery_oldest_eligible_age_seconds").Value.Should().Be(23); + own.Single(value => value.Instrument.Name == "learnstack_delivery_heartbeat_age_seconds").Value.Should().Be(8); + own.Single(value => value.Instrument.Name == "learnstack_delivery_failed_total").Tags + .Should().Equal([new KeyValuePair("event_type", "invalid"), new("subscription", "invalid")]); + own.Where(value => value.Instrument.IsObservable || value.Instrument.Name == "learnstack_delivery_lease_lost_total") + .Should().OnlyContain(value => value.Tags.Length == 0); + own.SelectMany(value => value.Tags).Should().OnlyContain(value => value.Key == "event_type" || value.Key == "subscription"); + } + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void Attempts_get_fresh_canonical_trace_without_inheriting_ambient_identity(bool listen) + { + using var listener = new ActivityListener + { + ShouldListenTo = source => listen && source.Name == DeliveryObservation.InstrumentationName, + Sample = (ref ActivityCreationOptions _) => ActivitySamplingResult.AllDataAndRecorded, + }; + ActivitySource.AddActivityListener(listener); + using var ambient = new Activity("private-caller").SetIdFormat(ActivityIdFormat.W3C).Start(); + using var observation = Create(new ManualTimeProvider()); + var actualSource = (ActivitySource)typeof(DeliveryObservation) + .GetField("_activities", BindingFlags.Instance | BindingFlags.NonPublic)!.GetValue(observation)!; + actualSource.HasListeners().Should().Be(listen, + "the isolated collection verifies the actual scheduler source's listener state"); + if (!listen) + { + using var controlSource = new ActivitySource("learnstack.delivery.no-listener-proof"); + using var control = controlSource.StartActivity("control"); + control.Should().BeNull(); + } + using var first = observation.BeginAttempt("private-undeclared-type", "private-subscription"); + first.Should().NotBeNull("the scheduler must create an audit trace even when its actual source has no listener"); + using var second = observation.BeginAttempt(nameof(TestEvent), "proof.subscription.v1"); + ActivityContext.TryParse(first.Id, null, out var firstContext).Should().BeTrue(); + ActivityContext.TryParse(second.Id, null, out var secondContext).Should().BeTrue(); + first.Id.Should().HaveLength(55); + firstContext.TraceId.Should().NotBe(ambient.TraceId).And.NotBe(secondContext.TraceId); + first.GetTagItem("event_type").Should().Be("invalid"); + first.GetTagItem("subscription").Should().Be("invalid"); + second.GetTagItem("subscription").Should().Be("proof.subscription.v1"); + first.GetTagItem("correlation.id").Should().BeNull(); + } + + [Fact] + public void Production_bounds_validate_and_unsafe_margins_or_unbounded_capacity_are_refused() + { + var defaults = new DeliveryDispatchOptions(); + defaults.Validate(); + defaults.ExecutionTimeout.Should().Be(TimeSpan.FromSeconds(10)); + defaults.SqlTimeout.Should().Be(TimeSpan.FromSeconds(5)); + defaults.LeaseDuration.Should().Be(TimeSpan.FromSeconds(30)); + defaults.MaxOutstandingExecutions.Should().Be(4); + defaults.ShutdownTimeout.Should().Be(TimeSpan.FromSeconds(30)); + DeliveryDispatchOptions[] invalid = [ + defaults with { ExecutionTimeout = TimeSpan.Zero }, + defaults with { SqlTimeout = TimeSpan.FromSeconds(6) }, + defaults with { LeaseDuration = TimeSpan.FromSeconds(20) }, + defaults with { ProgressTimeout = TimeSpan.FromSeconds(30) }, + defaults with { IdlePoll = TimeSpan.Zero }, + defaults with { SnapshotInterval = TimeSpan.FromSeconds(6) }, + defaults with { ShutdownTimeout = TimeSpan.FromSeconds(31) }, + defaults with { MaxOutstandingExecutions = 5 }, + ]; + foreach (var option in invalid) + FluentActions.Invoking(option.Validate).Should().Throw(); + } + + private static DeliveryObservation Create(TimeProvider time) => new(time, new DeliveryDispatchOptions(), + OutboxEventRegistry.Create([OutboxEventRegistration.Required(typeof(TestEvent), [OutboxSubscriptionManifest.Pending("proof.subscription.v1")])], [], + IntegrationEventHandlerRegistry.FromServiceDescriptors(new ServiceCollection()), [typeof(TestEvent).Assembly])); + + private sealed record TestEvent : IntegrationEventBase + { + public override string Topic => "learnstack.test.observation"; + public override string PartitionKey => TenantId.ToString("D"); + } + + private sealed class ManualTimeProvider : TimeProvider + { + private long _timestamp; + public DateTimeOffset UtcNow { get; set; } = new(2026, 10, 12, 0, 0, 0, TimeSpan.Zero); + public override long TimestampFrequency => TimeSpan.TicksPerSecond; + public override long GetTimestamp() => _timestamp; + public override DateTimeOffset GetUtcNow() => UtcNow; + public void Advance(TimeSpan duration) => _timestamp += duration.Ticks; + } +} diff --git a/backend/tests/LearnStack.Tests.Unit/Api/Tenancy/AuditingTenantAssertionRecorderTests.cs b/backend/tests/LearnStack.Tests.Unit/Api/Tenancy/AuditingTenantAssertionRecorderTests.cs index fb4fd589..668e051d 100644 --- a/backend/tests/LearnStack.Tests.Unit/Api/Tenancy/AuditingTenantAssertionRecorderTests.cs +++ b/backend/tests/LearnStack.Tests.Unit/Api/Tenancy/AuditingTenantAssertionRecorderTests.cs @@ -250,8 +250,9 @@ public async Task The_real_store_reports_a_failure_once_and_the_recorder_does_no var health = new AuditHealth(); var storeLogger = new CapturingLogger(); var recorderLogger = new CapturingLogger(); - var meterFactory = new ServiceCollection().AddMetrics() - .BuildServiceProvider().GetRequiredService(); + using var provider = new ServiceCollection().AddMetrics().BuildServiceProvider(); + var meterFactory = provider.GetRequiredService(); + var meter = meterFactory.Create(PostgresAuditStore.MeterName); var store = new PostgresAuditStore( new AuditStateCapture(), @@ -260,13 +261,13 @@ public async Task The_real_store_reports_a_failure_once_and_the_recorder_does_no storeLogger, meterFactory, health, - new AuditCatalog([])); + new AuditCatalog([]), StaticTenantContextAccessor.Unresolved); var counted = 0L; using var listener = new MeterListener(); listener.InstrumentPublished = (instrument, active) => { - if (instrument.Meter.Name == PostgresAuditStore.MeterName + if (ReferenceEquals(instrument.Meter, meter) && instrument.Name == PostgresAuditStore.StandaloneWriteFailureCounterName) { active.EnableMeasurementEvents(instrument); @@ -275,6 +276,21 @@ public async Task The_real_store_reports_a_failure_once_and_the_recorder_does_no listener.SetMeasurementEventCallback((_, measurement, _, _) => counted += measurement); listener.Start(); + // A concurrent store can publish the same instrument name from another meter. + using var unrelatedMeter = new Meter(PostgresAuditStore.MeterName); + var unrelatedCounter = unrelatedMeter.CreateCounter(PostgresAuditStore.StandaloneWriteFailureCounterName); + var unrelatedCount = 0L; + using var unrelatedListener = new MeterListener(); + unrelatedListener.InstrumentPublished = (instrument, active) => + { + if (ReferenceEquals(instrument, unrelatedCounter)) active.EnableMeasurementEvents(instrument); + }; + unrelatedListener.SetMeasurementEventCallback((_, measurement, _, _) => unrelatedCount += measurement); + unrelatedListener.Start(); + unrelatedCounter.Add(7); + unrelatedCount.Should().Be(7, "the unrelated instrument must actually emit its planted measurement"); + counted.Should().Be(0, "another store's meter must not contribute to this store's failure count"); + var recorder = Recorder(store, out _, logger: recorderLogger); var act = async () => await recorder.RecordRejectionAsync(Rejection(authenticated: true)); diff --git a/backend/tests/LearnStack.Tests.Unit/Application/Messaging/DeliveryCommitGateTests.cs b/backend/tests/LearnStack.Tests.Unit/Application/Messaging/DeliveryCommitGateTests.cs new file mode 100644 index 00000000..87d5af4f --- /dev/null +++ b/backend/tests/LearnStack.Tests.Unit/Application/Messaging/DeliveryCommitGateTests.cs @@ -0,0 +1,350 @@ +using System.Diagnostics; +using System.Reflection; +using System.Data; +using System.Data.Common; +using FluentAssertions; +using LearnStack.Application.Messaging; +using LearnStack.Application.Persistence; +using LearnStack.Infrastructure.Audit; +using LearnStack.Infrastructure.Messaging; +using LearnStack.SharedKernel.Audit; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Messaging; +using LearnStack.SharedKernel.Persistence; +using LearnStack.SharedKernel.Results; +using LearnStack.SharedKernel.Tenancy; +using MediatR; +using Microsoft.Extensions.Logging.Abstractions; +using NSubstitute; +using Xunit; + +namespace LearnStack.Tests.Unit.Application.Messaging; + +[CollectionDefinition(Name, DisableParallelization = true)] +public sealed class DeliveryCommitGateIsolation +{ + public const string Name = "Delivery commit gate without telemetry"; +} + +[Collection(DeliveryCommitGateIsolation.Name)] +public sealed class DeliveryCommitGateTests +{ + [Fact] + public void The_actual_delivery_activity_falls_back_to_W3C_with_no_listener_and_restores_its_caller() + { + const string trace = "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"; + var parent = Activity.Current; + Activity.Current = null; + Activity? activity = null; + try + { + using var controlSource = new ActivitySource("learnstack.delivery.no-listener-proof"); + using var control = controlSource.StartActivity("control"); + control.Should().BeNull(); + var actualSource = (ActivitySource)typeof(DeliveryExecutor) + .GetField("Activities", BindingFlags.Static | BindingFlags.NonPublic)!.GetValue(null)!; + actualSource.HasListeners().Should().BeFalse("this collection runs without any host or telemetry provider"); + activity = (Activity)typeof(DeliveryExecutor) + .GetMethod("BeginConsumerActivity", BindingFlags.Static | BindingFlags.NonPublic)! + .Invoke(null, [new IntegrationEventEnvelope(Fact(), trace)])!; + + activity.Should().NotBeNull(); + Activity.Current.Should().BeSameAs(activity); + activity.IdFormat.Should().Be(ActivityIdFormat.W3C); + activity.ParentId.Should().Be(trace); + activity.TraceId.ToHexString().Should().Be("4bf92f3577b34da6a3ce929d0e0e4736"); + activity.Id.Should().MatchRegex("^00-[0-9a-f]{32}-[0-9a-f]{16}-[0-9a-f]{2}$"); + activity.Dispose(); + Activity.Current.Should().BeNull(); + } + finally + { + activity?.Dispose(); + Activity.Current = parent; + } + } + + [Fact] + public async Task Concurrent_execution_attachment_has_exactly_one_winner_and_closed_gate_never_resets() + { + var gate = new DeliveryCommitGate(); + gate.Close(); + var starts = await Task.WhenAll(Enumerable.Range(0, 32).Select(_ => Task.Run(gate.TryStartExecution))); + starts.Count(started => started).Should().Be(1); + gate.TryStartExecution().Should().BeFalse(); + gate.CommitStarted.Should().BeFalse(); + gate.CommitAcknowledged.Should().BeFalse(); + } + + [Fact] + public async Task Closing_after_saved_work_refuses_the_actual_owner_commit_and_rolls_back() + { + var unit = new RecordingUnit(); + var gate = new DeliveryCommitGate(); + var capture = new AuditStateCapture(); + var producer = new ProducerExecution(unit); + var coordinator = Coordinator(unit, capture, producer); + await using var owner = await TransactionExecutionCoordinator.PrepareDeliveryOwnerAsync(unit, Context); + var participant = new Participant(unit, gate, () => unit.Calls.Add("fence")); + var run = () => coordinator.ExecutePreparedDeliveryAsync(owner, participant, Guid.CreateVersion7(), () => + { + unit.Calls.Add("saved-business"); + producer.RecordProducedWork(); + gate.Close(); + return Task.FromResult(Result.Ok(None.Value)); + }); + + await run.Should().ThrowAsync(); + unit.Calls.Should().ContainInOrder("saved-business", "audit", "fence", "rollback"); + unit.Calls.Should().NotContain("commit"); + unit.IsRollbackOnly.Should().BeTrue(); + capture.State.Should().Be(AuditIntentState.RolledBack); + gate.CommitStarted.Should().BeFalse(); + gate.CommitAcknowledged.Should().BeFalse(); + producer.HasInvocation.Should().BeFalse(); + } + + [Fact] + public async Task Closing_after_admission_cannot_revoke_a_commit_or_its_acknowledgement() + { + var unit = new RecordingUnit(); + var gate = new DeliveryCommitGate(); + var capture = new AuditStateCapture(); + unit.OnCommit = () => + { + gate.CommitStarted.Should().BeTrue(); + gate.CommitAcknowledged.Should().BeFalse(); + gate.Close(); + }; + await using var owner = await TransactionExecutionCoordinator.PrepareDeliveryOwnerAsync(unit, Context); + var result = await Coordinator(unit, capture).ExecutePreparedDeliveryAsync(owner, + new Participant(unit, gate, () => unit.Calls.Add("fence")), Guid.CreateVersion7(), + () => Task.FromResult(Result.Ok(None.Value))); + + result.IsSuccess.Should().BeTrue(); + unit.Calls.Should().ContainInOrder("audit", "fence", "commit", "dispose"); + gate.Close(); + gate.CommitAcknowledged.Should().BeTrue(); + capture.State.Should().Be(AuditIntentState.Committed); + } + + [Fact] + public async Task A_post_acknowledgement_disposal_fault_retains_definitive_commit_evidence() + { + var fault = new IOException("owned frame cleanup failed"); + var unit = new RecordingUnit { DisposeFailure = fault }; + var gate = new DeliveryCommitGate(); + var capture = new AuditStateCapture(); + await using var owner = await TransactionExecutionCoordinator.PrepareDeliveryOwnerAsync(unit, Context); + var run = () => Coordinator(unit, capture).ExecutePreparedDeliveryAsync(owner, + new Participant(unit, gate), Guid.CreateVersion7(), () => Task.FromResult(Result.Ok(None.Value))); + + (await run.Should().ThrowAsync()).Which.Should().BeSameAs(fault); + gate.CommitAcknowledged.Should().BeTrue(); + capture.State.Should().Be(AuditIntentState.Committed); + unit.Calls.Should().NotContain("rollback"); + } + + [Fact] + public async Task An_unacknowledged_commit_fault_is_indeterminate_and_cleanup_cannot_replace_it() + { + var fault = new IOException("COMMIT response lost"); + var unit = new RecordingUnit + { + CommitFailure = fault, + DisposeFailure = new IOException("secondary cleanup failure"), + }; + var gate = new DeliveryCommitGate(); + var capture = new AuditStateCapture(); + await using var owner = await TransactionExecutionCoordinator.PrepareDeliveryOwnerAsync(unit, Context); + var run = () => Coordinator(unit, capture).ExecutePreparedDeliveryAsync(owner, + new Participant(unit, gate), Guid.CreateVersion7(), () => Task.FromResult(Result.Ok(None.Value))); + + (await run.Should().ThrowAsync()).Which.Should().BeSameAs(fault); + gate.CommitStarted.Should().BeTrue(); + gate.CommitAcknowledged.Should().BeFalse(); + capture.State.Should().Be(AuditIntentState.Indeterminate); + unit.Calls.Should().NotContain("rollback"); + } + + [Fact] + public async Task A_completion_participant_cannot_cross_units_even_when_tenant_and_generation_match() + { + var unit = new RecordingUnit(); + var other = new RecordingUnit(); + var gate = new DeliveryCommitGate(); + var constructed = false; + await using (var owner = await TransactionExecutionCoordinator.PrepareDeliveryOwnerAsync(unit, Context)) + { + var run = () => Coordinator(unit, new AuditStateCapture()).ExecutePreparedDeliveryAsync(owner, + new Participant(other, gate), Guid.CreateVersion7(), () => + { + constructed = true; + return Task.FromResult(Result.Ok(None.Value)); + }); + await run.Should().ThrowAsync().WithMessage("*share*unit*"); + unit.HasActiveTransaction.Should().BeTrue("the rejected participant never consumes the owner"); + } + constructed.Should().BeFalse(); + unit.Calls.Should().Contain("rollback"); + gate.CommitStarted.Should().BeFalse(); + } + + [Fact] + public async Task Delivery_causation_is_inherited_by_nested_production_and_domain_identity_takes_precedence() + { + var unit = new RecordingUnit(); + var producer = new ProducerExecution(unit); + var coordinator = Coordinator(unit, new AuditStateCapture(), producer); + var incoming = Guid.CreateVersion7(); + var domain = Guid.CreateVersion7(); + await using var owner = await TransactionExecutionCoordinator.PrepareDeliveryOwnerAsync(unit, Context); + (await coordinator.ExecutePreparedDeliveryAsync(owner, new Participant(unit, new DeliveryCommitGate()), + incoming, async () => + { + producer.CreateEnvelope(Fact()).CausationId.Should().Be(incoming); + await coordinator.ExecuteAsync(null, TransactionMode.ReadWrite, "nested", () => + { + producer.CreateEnvelope(Fact()).CausationId.Should().Be(incoming); + return Task.FromResult(Result.Ok(None.Value)); + }); + using (producer.EnterDomainDispatch(producer.CaptureOrigin(), domain)) + producer.CreateEnvelope(Fact()).CausationId.Should().Be(domain); + return Result.Ok(None.Value); + })).IsSuccess.Should().BeTrue(); + producer.HasInvocation.Should().BeFalse(); + } + + private static readonly DeliveryContext Context = new(); + + private static TransactionExecutionCoordinator Coordinator(RecordingUnit unit, AuditStateCapture capture, + ProducerExecution? producer = null) + { + producer ??= new ProducerExecution(unit); + var store = Substitute.For(); + store.WritePendingAsync(unit, Arg.Any()).Returns(_ => + { + unit.Calls.Add("audit"); + return Task.CompletedTask; + }); + return new TransactionExecutionCoordinator(unit, Context, store, capture, producer, + new DomainEventCollector(producer, Substitute.For()), + NullLogger.Instance); + } + + private static FactEvent Fact() => new() + { + TenantId = Context.TenantId.Value, + EventId = Guid.CreateVersion7(), + OccurredAt = DateTimeOffset.UtcNow, + }; + + private sealed record FactEvent : IntegrationEventBase + { + public override string Topic => "learnstack.proof.delivery"; + public override string PartitionKey => TenantId.ToString(); + } + + private sealed class DeliveryContext : ITenantContext + { + public bool IsResolved => true; + public TenantId TenantId { get; } = TenantId.From(Guid.CreateVersion7()); + public OrganizationId? OrganizationId => null; + public UserId? UserId => LearnStack.SharedKernel.Identifiers.UserId.SystemActor; + public string? CorrelationId => null; + public string? ModuleName => "proof"; + } + + private sealed class Participant(IUnitOfWork unit, DeliveryCommitGate gate, Action? fence = null) + : DeliveryCompletionParticipant(unit, gate) + { + protected override Task FenceAsync(CancellationToken cancellationToken) + { + fence?.Invoke(); + return Task.CompletedTask; + } + } + + private sealed class RecordingUnit : IUnitOfWork + { + private int _depth; + private TenantContextAnnouncement? _announcement; + public List Calls { get; } = []; + public Action? OnCommit { get; set; } + public Exception? CommitFailure { get; init; } + public Exception? DisposeFailure { get; init; } + public DbConnection Connection => throw new NotSupportedException(); + public DbTransaction? Transaction { get; private set; } + public bool HasActiveTransaction => _depth > 0; + public TransactionMode? Mode { get; private set; } + public bool IsRollbackOnly { get; private set; } + public Task BeginTransactionAsync(CancellationToken cancellationToken = default) => + BeginTransactionAsync(TransactionMode.ReadWrite, cancellationToken); + public Task BeginTransactionAsync(TransactionMode mode, + CancellationToken cancellationToken = default) + { + if (_depth == 0) { Transaction = new TestTransaction(); Mode = mode; } + return Task.FromResult(new Frame(this, ++_depth)); + } + public Task SetTenantContextAsync(ITenantContext context, CancellationToken cancellationToken = default) + { + if (_depth == 1) _announcement = new TenantContextAnnouncement(1, context.TenantId, context.OrganizationId); + return Task.CompletedTask; + } + public TenantContextAnnouncement? GetTenantContextAnnouncement(DbTransaction? transaction) => + transaction is not null && ReferenceEquals(Transaction, transaction) ? _announcement : null; + public bool IsTenantContextIssuedOn(DbTransaction? transaction) => GetTenantContextAnnouncement(transaction) is not null; + public Task SetProvisioningTenantContextAsync(TenantId tenantId, CancellationToken cancellationToken = default) => + throw new NotSupportedException(); + public Task CommitAsync(CancellationToken cancellationToken = default) + { + if (--_depth > 0) return Task.CompletedTask; + Calls.Add("commit"); + OnCommit?.Invoke(); + Transaction = null; + Mode = null; + _announcement = null; + return CommitFailure is { } fault ? Task.FromException(fault) : Task.CompletedTask; + } + public Task RollbackAsync(CancellationToken cancellationToken = default) + { + Calls.Add("rollback"); + _depth = 0; + Transaction = null; + Mode = null; + _announcement = null; + return Task.CompletedTask; + } + public void MarkRollbackOnly() => IsRollbackOnly = true; + public ValueTask DisposeAsync() => ValueTask.CompletedTask; + private sealed class Frame(RecordingUnit unit, int depth) : IUnitOfWorkScope + { + private bool _resolved; + public bool IsOwner => depth == 1; + public Task CompleteAsync(CancellationToken cancellationToken = default) + { + _resolved = true; + return unit.CommitAsync(cancellationToken); + } + public Task FailAsync(CancellationToken cancellationToken = default) + { + _resolved = true; + return unit.RollbackAsync(cancellationToken); + } + public async ValueTask DisposeAsync() + { + unit.Calls.Add("dispose"); + if (!_resolved) await unit.RollbackAsync(); + if (unit.DisposeFailure is { } fault) throw fault; + } + } + } + + private sealed class TestTransaction : DbTransaction + { + public override IsolationLevel IsolationLevel => IsolationLevel.ReadCommitted; + protected override DbConnection? DbConnection => null; + public override void Commit() => throw new NotSupportedException(); + public override void Rollback() => throw new NotSupportedException(); + } +} diff --git a/backend/tests/LearnStack.Tests.Unit/Infrastructure/Audit/AuditChangeTrackerInterceptorTests.cs b/backend/tests/LearnStack.Tests.Unit/Infrastructure/Audit/AuditChangeTrackerInterceptorTests.cs index 3b2cbc55..cac9eacd 100644 --- a/backend/tests/LearnStack.Tests.Unit/Infrastructure/Audit/AuditChangeTrackerInterceptorTests.cs +++ b/backend/tests/LearnStack.Tests.Unit/Infrastructure/Audit/AuditChangeTrackerInterceptorTests.cs @@ -309,6 +309,8 @@ public void A_converted_property_is_captured_as_the_column_holds_it() // excluded regardless, because "cannot arise" is a property of today's store while // the exclusion is a property of the design. [InlineData(typeof(OutboxMessage))] + [InlineData(typeof(InboxMessage))] + [InlineData(typeof(EventDelivery))] [InlineData(typeof(IdempotencyKey))] [InlineData(typeof(AuditEntry))] [InlineData(typeof(AuditConfig))] @@ -902,6 +904,8 @@ protected override void OnModelCreating(ModelBuilder builder) entity.Property(documented => documented.Text).HasColumnType("text"); }); builder.Entity(); + builder.Entity(); + builder.Entity(); builder.Entity(); builder.Entity(); builder.Entity(); @@ -1099,10 +1103,14 @@ private readonly record struct ProbeId(Guid Value) : IStronglyTypedId public bool IsInitialized() => Value != Guid.Empty; } - // The four excluded names, as the interceptor matches them: by CLR type name, so a + // The six excluded names, as the interceptor matches them: by CLR type name, so a // stand-in with the same name is exactly what it will see. private sealed class OutboxMessage { public int Id { get; set; } } + private sealed class InboxMessage { public int Id { get; set; } } + + private sealed class EventDelivery { public int Id { get; set; } } + private sealed class IdempotencyKey { public int Id { get; set; } } private sealed class AuditEntry { public int Id { get; set; } } diff --git a/backend/tests/LearnStack.Tests.Unit/Infrastructure/Audit/DeliveryAuditDiagnosticsTests.cs b/backend/tests/LearnStack.Tests.Unit/Infrastructure/Audit/DeliveryAuditDiagnosticsTests.cs new file mode 100644 index 00000000..9f09bdfd --- /dev/null +++ b/backend/tests/LearnStack.Tests.Unit/Infrastructure/Audit/DeliveryAuditDiagnosticsTests.cs @@ -0,0 +1,215 @@ +using System.Diagnostics.Metrics; +using FluentAssertions; +using LearnStack.Infrastructure.Audit; +using LearnStack.Infrastructure.Audit.Capture; +using LearnStack.Infrastructure.Caching; +using LearnStack.Modules.Tenancy.Application.Contracts.IntegrationEvents; +using LearnStack.SharedKernel.Audit; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Messaging; +using LearnStack.SharedKernel.Tenancy; +using LearnStack.SharedKernel.Time; +using Microsoft.Extensions.Logging; +using Npgsql; +using Xunit; + +namespace LearnStack.Tests.Unit.Infrastructure.Audit; + +/// Actual audit failures retain their posture while delivery diagnostics exclude provider details. +public sealed class DeliveryAuditDiagnosticsTests +{ + private const string Canary = "private-provider-detail-must-not-leave-delivery"; + private const string Operation = "tenancy.tenant.provision"; + private const string Traceparent = "00-12345678901234567890123456789012-1234567890123456-01"; + + [Fact] + public void Store_requires_an_accessor_even_when_no_context_is_currently_resolved() + { + using var meters = new IsolatedMeterFactory(); + var create = () => new PostgresAuditStore( + new AuditStateCapture(), + new Lazy(() => throw new InvalidOperationException("No database is needed.")), + new CaptureLogger(), meters, new AuditHealth(), new AuditCatalog([]), null!); + + create.Should().Throw().WithParameterName("tenantAccessor"); + } + + [Theory] + [InlineData(true)] + [InlineData(false)] + public async Task Override_read_failure_preserves_the_declared_tier_and_reports_with_context_appropriate_detail( + bool delivery) + { + var tenant = TenantId.From(Guid.CreateVersion7()); + var failure = new IOException(Canary); + var logger = new CaptureLogger(); + using var meters = new IsolatedMeterFactory(); + var cache = new InMemoryCacheService(new FixedClock(), meters); + var attempted = 0; + var source = new Lazy(() => + { + attempted++; + throw failure; + }); + var classifier = new AuditConfigService(cache, source, logger, Accessor(tenant, delivery)); + var declaration = new AuditCatalogEntry( + "tenancy", Operation, OperationType.Create, OperationClass.Should, null); + + var classified = await classifier.ClassifyAsync(tenant, declaration); + + attempted.Should().Be(1, "a real cache miss must reach the lazy database loader"); + classified.Should().Be(AuditClassification.Should, + "an unavailable override cannot lower or refuse the declared coverage"); + var log = logger.Entries.Should().ContainSingle().Subject; + log.Level.Should().Be(LogLevel.Error, "override fallback retains its established severity"); + AssertDiagnostic(log, delivery, failure); + } + + [Theory] + [InlineData(true, true)] + [InlineData(false, true)] + [InlineData(true, false)] + [InlineData(false, false)] + public async Task Store_failure_preserves_durability_health_and_counter_posture_while_minimizing_delivery_logs( + bool delivery, bool standaloneMust) + { + var tenant = TenantId.From(Guid.CreateVersion7()); + var failure = new IOException(Canary); + var logger = new CaptureLogger(); + var health = new AuditHealth(); + using var meters = new IsolatedMeterFactory(); + var sourceAttempts = 0; + var store = new PostgresAuditStore( + new AuditStateCapture(), + new Lazy(() => + { + sourceAttempts++; + throw failure; + }), + logger, meters, health, new AuditCatalog([]), Accessor(tenant, delivery)); + var measurements = new List<(long Count, KeyValuePair[] Tags)>(); + using var listener = new MeterListener(); + listener.InstrumentPublished = (instrument, active) => + { + // Meter identity excludes measurements from concurrently running tests. + if (ReferenceEquals(instrument.Meter, meters.Meter) + && instrument.Name == PostgresAuditStore.StandaloneWriteFailureCounterName) + active.EnableMeasurementEvents(instrument); + }; + listener.SetMeasurementEventCallback((_, count, tags, _) => + measurements.Add((count, tags.ToArray()))); + listener.Start(); + var draft = Draft(tenant, standaloneMust ? OperationClass.Must : OperationClass.Should); + + if (standaloneMust) + { + var thrown = await Assert.ThrowsAsync(() => store.WriteStandaloneAsync(draft)); + thrown.Should().BeSameAs(failure, "diagnostic minimization must not change the caller's failure contract"); + health.IsHealthy.Should().BeFalse(); + health.FailingOperation.Should().Be(Operation); + var measurement = measurements.Should().ContainSingle().Subject; + measurement.Count.Should().Be(1); + measurement.Tags.Should().ContainSingle().Which + .Should().Be(new KeyValuePair("operation", Operation)); + } + else + { + await store.WriteBestEffortAsync(draft); + health.IsHealthy.Should().BeTrue("a best-effort loss does not mark MUST durability unavailable"); + health.FailingOperation.Should().BeNull(); + measurements.Should().BeEmpty("only failed MUST standalone writes increment the failure counter"); + } + + sourceAttempts.Should().Be(1, "the actual store must attempt its standalone database path"); + var log = logger.Entries.Should().ContainSingle().Subject; + log.Level.Should().Be(standaloneMust ? LogLevel.Critical : LogLevel.Error); + log.Message.Should().Contain(Operation); + AssertDiagnostic(log, delivery, failure); + } + + private static void AssertDiagnostic(LogEntry log, bool delivery, Exception failure) + { + log.Message.Should().NotBeNullOrWhiteSpace("bounded diagnostics must still report the failure"); + log.Message.Should().NotContain(Canary).And.NotContain(typeof(IOException).FullName!); + if (delivery) + { + log.Exception.Should().BeNull("the logger must not receive provider text, stack or raw exception type"); + } + else + { + log.Exception.Should().BeSameAs(failure, "ordinary request diagnostics retain the original exception"); + } + } + + private static StaticTenantContextAccessor Accessor(TenantId tenant, bool delivery) + { + if (!delivery) return new(new RequestContext(tenant)); + var fact = new TenantCreatedIntegrationEventV1 + { + EventId = Guid.CreateVersion7(), + TenantId = tenant.Value, + OccurredAt = DateTimeOffset.UnixEpoch, + }; + return new(EventTenantContext.FromEnvelope(new IntegrationEventEnvelope(fact, Traceparent))); + } + + private static AuditEntryDraft Draft(TenantId tenant, OperationClass classification) => new() + { + Id = AuditEntryId.From(Guid.CreateVersion7()), + TenantId = tenant, + OrganizationId = null, + ActorUserId = UserId.SystemActor, + ActorEmail = null, + ModuleName = "tenancy", + Operation = Operation, + OperationType = OperationType.Create, + OperationClass = classification, + EntityType = null, + EntityId = null, + Outcome = AuditOutcome.Failed, + ErrorKey = null, + Reason = null, + BeforeState = null, + AfterState = null, + Changes = "[]", + CorrelationId = Traceparent, + IpAddress = null, + UserAgent = null, + Timestamp = DateTimeOffset.UnixEpoch, + Metadata = null, + }; + + private sealed class RequestContext(TenantId tenant) : ITenantContext + { + public bool IsResolved => true; + public TenantId TenantId => tenant; + public OrganizationId? OrganizationId => null; + public UserId? UserId { get; } = LearnStack.SharedKernel.Identifiers.UserId.From(Guid.CreateVersion7()); + public string? CorrelationId => Traceparent; + public string? ModuleName => "tenancy"; + } + + private sealed class FixedClock : IClock + { + public DateTimeOffset UtcNow => DateTimeOffset.UnixEpoch; + } + + private sealed class IsolatedMeterFactory : IMeterFactory + { + public Meter Meter { get; } = new(PostgresAuditStore.MeterName); + public Meter Create(MeterOptions options) => Meter; + public void Dispose() => Meter.Dispose(); + } + + private sealed record LogEntry(LogLevel Level, string Message, Exception? Exception); + + private sealed class CaptureLogger : ILogger + { + public List Entries { get; } = []; + public IDisposable? BeginScope(TState state) where TState : notnull => null; + public bool IsEnabled(LogLevel logLevel) => true; + public void Log(LogLevel logLevel, EventId eventId, TState state, + Exception? exception, Func formatter) => + Entries.Add(new(logLevel, formatter(state, exception), exception)); + } +} diff --git a/backend/tests/LearnStack.Tests.Unit/Infrastructure/Messaging/InProcessEventBusTests.cs b/backend/tests/LearnStack.Tests.Unit/Infrastructure/Messaging/InProcessEventBusTests.cs index af23b514..3d561d4c 100644 --- a/backend/tests/LearnStack.Tests.Unit/Infrastructure/Messaging/InProcessEventBusTests.cs +++ b/backend/tests/LearnStack.Tests.Unit/Infrastructure/Messaging/InProcessEventBusTests.cs @@ -1,1057 +1,334 @@ -using System.Collections.Concurrent; -using System.Diagnostics; using FluentAssertions; using LearnStack.Infrastructure.Messaging; using LearnStack.SharedKernel.Identifiers; using LearnStack.SharedKernel.Messaging; using LearnStack.SharedKernel.Tenancy; using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Logging.Abstractions; using Xunit; namespace LearnStack.Tests.Unit.Infrastructure.Messaging; -/// -/// The default , per -/// ADR-0035. -/// -/// -/// Every case here asserts one of the four obligations ADR-0035 makes a condition -/// of the gating — same handler contract, same deduplication seam, same -/// tenant-context restoration, same per-partition ordering. A transport that -/// dropped any of them would be a development path where the production -/// behaviour is never exercised, which is the opposite of what a default -/// implementation is for. -/// +/// Publication guards before the PostgreSQL acceptance boundary. +/// Database acceptance and handler execution have separate proofs. public sealed class InProcessEventBusTests { private static readonly Guid Tenant = Guid.Parse("018f4d40-0000-7000-8000-00000000000a"); - private static readonly Guid OtherTenant = Guid.Parse("018f4d40-0000-7000-8000-00000000000b"); - private static readonly TimeSpan Timeout = TimeSpan.FromSeconds(10); + private static readonly Guid Subject = Guid.Parse("018f4d40-0000-7000-8000-00000000000b"); + private static readonly Guid Organization = Guid.Parse("018f4d40-0000-7000-8000-00000000000c"); private const string Trace = "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"; [Fact] - public async Task A_Handler_Receives_The_Event() + public async Task Active_publication_restores_envelope_scope_before_acceptance_scope_creation() { - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - services.AddScoped, ThingHandler>()); - - await bus.PublishAsync(Envelope(NewThing("a"))); - - recorder.Handled.Should().ContainSingle().Which.Should().Be("a"); - } - - [Fact] - public async Task An_Internal_Handler_Is_Invoked_Too() - { - // The reason handlers are called through the interface's MethodInfo - // rather than by `dynamic`: the dynamic binder honours accessibility, so - // an internal handler — the normal shape for a module's own consumer — - // would fail to bind at runtime, and the failure would be a - // RuntimeBinderException out of the transport rather than anything a - // consumer could diagnose. - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - services.AddScoped, InternalThingHandler>()); - - await bus.PublishAsync(Envelope(NewThing("a"))); - - recorder.Handled.Should().ContainSingle().Which.Should().Be("internal:a"); - } - - [Fact] - public async Task Publishing_Through_The_Base_Interface_Still_Reaches_The_Handler() - { - // The reason PublishAsync is not generic. The outbox processor - // deserialises to object and publishes through the base interface, so a - // generic parameter would bind to IIntegrationEvent — and resolving - // IIntegrationEventHandler finds nothing, because no - // concrete consumer implements it. The publish would reach zero handlers - // and report success, which is the worst shape a bug can take. - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - services.AddScoped, ThingHandler>()); - - IIntegrationEvent asBase = NewThing("a"); - await bus.PublishAsync(new IntegrationEventEnvelope(asBase, Trace)); - - recorder.Handled.Should().ContainSingle(); - } - - [Fact] - public async Task Every_Handler_For_The_Event_Runs() - { - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - { - services.AddScoped, ThingHandler>(); - services.AddScoped, SecondThingHandler>(); - }); - - await bus.PublishAsync(Envelope(NewThing("a"))); - - recorder.Handled.Should().BeEquivalentTo(["a", "second:a"]); - } - - [Fact] - public async Task An_Event_With_No_Handler_Is_Not_An_Error() - { - var (bus, _) = Build(new Recorder(), _ => { }); - - var act = () => bus.PublishAsync(Envelope(NewThing("a"))); - - await act.Should().NotThrowAsync(); - } - - // ---- obligation: tenant context is restored, and put back ---------------- - - [Fact] - public async Task The_Handler_Runs_Under_The_Events_Tenant() - { - // A consumer runs outside the request that produced the fact, so there is - // no ambient context to inherit. Without this the handler executes with - // no tenant and every query filter and RLS policy is evaluated against - // nothing. - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - services.AddScoped, TenantReadingHandler>()); - - await bus.PublishAsync(Envelope(NewThing("a"))); - - recorder.Tenants.Should().ContainSingle().Which.Should().Be(Tenant); - } - - [Fact] - public async Task The_Publishers_Own_Context_Is_Put_Back() - { - // Dispatch is synchronous here, so a transport that set the context and - // walked away would leak the event's tenant into the caller that - // published it — and that caller goes on to run its own queries. - var recorder = new Recorder(); - var (bus, accessor) = Build(recorder, services => - services.AddScoped, TenantReadingHandler>()); - - var publisher = EventTenantContext.FromEnvelope( - Envelope(NewThing("x") with { TenantId = Guid.NewGuid() })); + var accessor = new FieldAccessor(); + var publisher = EventTenantContext.FromEnvelope(Envelope(New())); accessor.Current = publisher; - - await bus.PublishAsync(Envelope(NewThing("a"))); - + var scopes = new ScopeProbe(accessor); + var partitions = new RecordingPartitions(); + var bus = Build(scopes, accessor, partitions); + var actor = UserId.From(Guid.NewGuid()); + var envelope = new IntegrationEventEnvelope(New(), Trace, + OrganizationId: Organization, CausationId: Guid.NewGuid(), ActorUserId: actor); + + await ((Func)(() => bus.PublishAsync(envelope))) + .Should().ThrowAsync(); + + scopes.Calls.Should().Be(1); + var restored = scopes.Context.Should().BeOfType().Subject; + restored.TenantId.Value.Should().Be(Tenant); + restored.OrganizationId!.Value.Value.Should().Be(Organization); + restored.CorrelationId.Should().Be(Trace); + restored.UserId.Should().Be(UserId.SystemActor); + restored.CausalActorUserId.Should().Be(actor); accessor.Current.Should().BeSameAs(publisher); + partitions.Keys.Should().ContainSingle().Which.Should().Be(Subject.ToString("D")); + DeclaredHandler.Constructions.Should().Be(0); } [Fact] - public async Task A_Handler_That_Throws_Leaves_No_Context_Behind() + public async Task Failed_acceptance_scope_creation_restores_a_null_previous_context() { - var recorder = new Recorder(); - var (bus, accessor) = Build(recorder, services => - services.AddScoped, ThrowingHandler>()); - accessor.Current = null; + var accessor = new FieldAccessor(); + var scopes = new ScopeProbe(accessor); + var bus = Build(scopes, accessor); - var act = () => bus.PublishAsync(Envelope(NewThing("a"))); + await ((Func)(() => bus.PublishAsync(Envelope(New())))) + .Should().ThrowAsync(); - await act.Should().ThrowAsync(); + scopes.Context.Should().BeOfType(); accessor.Current.Should().BeNull(); } [Fact] - public async Task A_Handler_Reads_The_Scoped_Tenant_Context_Too() - { - // Setting only the ambient accessor left the SCOPED ITenantContext — the - // one ITenantContext's own doc says is handed to MediatR handlers, EF - // interceptors and the audit pipeline — unresolved inside the dispatch - // scope. A handler injecting it threw, and one sending a MediatR command - // was short-circuited by TenantContextBehavior before its business logic - // ran. The obligation the transport advertises was half-delivered. - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - services.AddScoped, ScopedContextReadingHandler>()); - - await bus.PublishAsync(Envelope(NewThing("a"))); - - recorder.Tenants.Should().ContainSingle().Which.Should().Be(Tenant); - } - - [Fact] - public async Task The_Consumer_Acts_As_The_System_When_The_Envelope_Names_No_Actor() + public async Task Base_interface_publication_uses_runtime_contract_and_canonical_partition() { - // AuditableEntity.MarkCreated refuses default(UserId) and Guid.Empty, so - // a null actor left every state-writing consumer with no value it could - // legally pass — it could not create an aggregate at all. - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - services.AddScoped, ActorReadingHandler>()); + var accessor = new FieldAccessor(); + var scopes = new ScopeProbe(accessor); + var partitions = new RecordingPartitions(); + var bus = Build(scopes, accessor, partitions); + IIntegrationEvent fact = New(); - await bus.PublishAsync(Envelope(NewThing("a"))); + await ((Func)(() => bus.PublishAsync(Envelope(fact)))) + .Should().ThrowAsync(); - recorder.Actors.Should().ContainSingle().Which.Should().Be(UserId.SystemActor); + scopes.Calls.Should().Be(1); + partitions.Keys.Should().ContainSingle().Which.Should().Be(Subject.ToString("D")); } [Fact] - public async Task The_Consumer_Uses_The_System_Actor_And_Preserves_The_Causal_Actor() + public async Task Intentional_absence_validates_but_does_not_construct_an_acceptance_scope() { - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - services.AddScoped, ActorReadingHandler>()); - - var actor = UserId.From(Guid.Parse("018f4d40-0000-7000-8000-0000000000aa")); - var organization = Guid.Parse("018f4d40-0000-7000-8000-0000000000c1"); + var accessor = new FieldAccessor(); + var publisher = EventTenantContext.FromEnvelope(Envelope(New())); + accessor.Current = publisher; + var scopes = new ScopeProbe(accessor); + var partitions = new RecordingPartitions(); + var bus = Build(scopes, accessor, partitions); - await bus.PublishAsync(new IntegrationEventEnvelope( - NewThing("a"), Trace, OrganizationId: organization, ActorUserId: actor)); + await bus.PublishAsync(Envelope(New())); - recorder.Actors.Should().ContainSingle().Which.Should().Be(UserId.SystemActor); - recorder.CausalActors.Should().ContainSingle().Which.Should().Be(actor); - recorder.Organizations.Should().ContainSingle().Which.Should().Be(organization); + scopes.Calls.Should().Be(0); + partitions.Keys.Should().ContainSingle().Which.Should().Be(Subject.ToString("D")); + accessor.Current.Should().BeSameAs(publisher); + DeclaredHandler.Constructions.Should().Be(0); } [Fact] - public async Task An_Event_With_No_Tenant_Is_Refused_Rather_Than_Dispatched() + public async Task Pending_unknown_and_retired_contracts_refuse_before_any_scope() { - // A confidently-resolved context for a tenant that does not exist is - // worse than an unresolved one: once SET LOCAL app.tenant_id runs, every - // query silently returns nothing instead of failing. - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - services.AddScoped, ThingHandler>()); + var accessor = new FieldAccessor(); + var scopes = new ScopeProbe(accessor); + var partitions = new RecordingPartitions(); + var bus = Build(scopes, accessor, partitions); - var act = () => bus.PublishAsync( - Envelope(NewThing("a") with { TenantId = Guid.Empty })); - - await act.Should().ThrowAsync(); - recorder.Handled.Should().BeEmpty(); - } - - // ---- failure isolation --------------------------------------------------- - - [Fact] - public async Task One_Failing_Handler_Does_Not_Deny_The_Others_The_Event() - { - // Poison-message containment is per subscription: one module's broken - // handler must not stop another module consuming the same event, and - // in-process there is no retry and no dead-letter to make up for it. - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => + foreach (var fact in new IntegrationEventBase[] + { New(), New(), New() }) { - services.AddScoped, ThrowingHandler>(); - services.AddScoped, SecondThingHandler>(); - }); - - var act = () => bus.PublishAsync(Envelope(NewThing("a"))); + await ((Func)(() => bus.PublishAsync(Envelope(fact)))) + .Should().ThrowAsync(); + } - await act.Should().ThrowAsync(); - recorder.Handled.Should().ContainSingle().Which.Should().Be("second:a", - "the surviving subscription still got its delivery"); + scopes.Calls.Should().Be(0); + partitions.Keys.Should().BeEmpty(); + accessor.Current.Should().BeNull(); } [Fact] - public async Task Several_Failures_Are_Reported_Together() + public async Task Invalid_tenant_payload_or_partition_refuses_even_intentional_absence() { - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - { - services.AddScoped, ThrowingHandler>(); - services.AddScoped, ThrowingHandler>(); - }); - - var act = () => bus.PublishAsync(Envelope(NewThing("a"))); - - (await act.Should().ThrowAsync()) - .Which.InnerExceptions.Should().HaveCount(2); - } + var accessor = new FieldAccessor(); + var scopes = new ScopeProbe(accessor); + var partitions = new RecordingPartitions(); + var bus = Build(scopes, accessor, partitions); + var sentinel = New() with { TenantId = TenantId.PlatformSentinel.Value }; - [Fact] - public async Task Each_Handler_Gets_Its_Own_Scope() - { - // Under a broker each subscription gets its own. Sharing one here would - // hand two modules' consumers the same DbContext and the same unit of - // work, so a failed handler's dirty state would cross a module boundary - // the architecture otherwise enforces hard. - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => + foreach (var fact in new IntegrationEventBase[] + { sentinel, New(), New() }) { - services.AddScoped(); - services.AddScoped, ScopeReadingHandler>(); - services.AddScoped, SecondScopeReadingHandler>(); - }); - - await bus.PublishAsync(Envelope(NewThing("a"))); - - recorder.Scopes.Should().HaveCount(2); - recorder.Scopes.Distinct().Should().HaveCount(2, "two scopes, two markers"); - } - - // ---- cancellation -------------------------------------------------------- - - [Fact] - public async Task A_Publish_On_An_Already_Cancelled_Token_Does_Not_Dispatch() - { - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - services.AddScoped, ThingHandler>()); - using var cancelled = new CancellationTokenSource(); - await cancelled.CancelAsync(); - - var act = () => bus.PublishAsync(Envelope(NewThing("a")), cancelled.Token); - - await act.Should().ThrowAsync(); - recorder.Handled.Should().BeEmpty("a broker-backed transport would fail before any I/O"); - } - - [Fact] - public async Task A_Handler_Cancelled_By_A_Foreign_Token_Fails_Rather_Than_Cancels() - { - // An outbox processor cannot distinguish "we are shutting down, retry - // later" from "the handler ran and gave up" if both arrive as a cancelled - // task — and its shutdown path swallows the former. - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - services.AddScoped, ForeignCancelHandler>()); - - var publish = bus.PublishAsync(Envelope(NewThing("a"))); - - var act = () => publish; - await act.Should().ThrowAsync(); - publish.IsCanceled.Should().BeFalse(); - } - - [Fact] - public async Task A_Handler_That_Faults_After_An_Await_Still_Faults_The_Publish() - { - // Every throwing handler in this suite threw SYNCHRONOUSLY, which comes - // out of MethodInfo.Invoke and is rethrown before `await delivery` is - // ever reached. So the path every real async handler takes — the one - // that hits a database — had no coverage for faults at all: wrapping the - // await in a swallowing catch survived the whole suite. - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - services.AddScoped, AsyncThrowingHandler>()); - - var act = () => bus.PublishAsync(Envelope(NewThing("a"))); - - (await act.Should().ThrowAsync()) - .Which.Message.Should().Be("async handler failed"); - } - - [Fact] - public async Task The_Tenant_Is_Still_Set_After_A_Handler_Awaits() - { - // Every tenant-reading handler read it synchronously, so the suite - // proved the context was set when a handler STARTED — not that it was - // still set when its continuation resumed, which is when the query RLS - // evaluates actually runs. Restoring the publisher's context just before - // the await survived every test. - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - services.AddScoped, LateTenantReadingHandler>()); - - await bus.PublishAsync(Envelope(NewThing("a"))); - - recorder.Tenants.Should().ContainSingle().Which.Should().Be(Tenant); - } - - [Fact] - public async Task The_Publish_Token_Reaches_The_Handler() - { - // The token was threaded through but never asserted to arrive: passing - // CancellationToken.None instead survived, so a shutdown or a timeout - // would never reach a consumer. - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - services.AddScoped, TokenReadingHandler>()); - using var cancelled = new CancellationTokenSource(); - - await bus.PublishAsync(Envelope(NewThing("a")), cancelled.Token); + await ((Func)(() => bus.PublishAsync(Envelope(fact)))) + .Should().ThrowAsync(); + } - // The token ITSELF, not merely "a cancellable token" — asserting - // CanBeCanceled was true of any token at all, so threading a freshly - // minted CancellationTokenSource through instead would have passed while - // a shutdown never reached a consumer, which is the failure this test - // names. - recorder.HandlerToken.Should().Be(cancelled.Token); + scopes.Calls.Should().Be(0); + partitions.Keys.Should().BeEmpty(); + accessor.Current.Should().BeNull(); } [Fact] - public async Task Publish_Cancellation_Stops_Later_Subscriptions() + public async Task Null_envelope_and_pre_canceled_token_never_construct_a_scope() { - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - { - services.AddScoped, CancellationWaitingHandler>(); - services.AddScoped, SecondThingHandler>(); - }); + var accessor = new FieldAccessor(); + var scopes = new ScopeProbe(accessor); + var bus = Build(scopes, accessor); using var cancellation = new CancellationTokenSource(); + cancellation.Cancel(); - var publish = bus.PublishAsync(Envelope(NewThing("a")), cancellation.Token); - await recorder.HandlerEntered.Task.WaitAsync(Timeout); - await cancellation.CancelAsync(); - - await ((Func)(() => publish)).Should().ThrowAsync(); - publish.IsCanceled.Should().BeTrue(); - recorder.Handled.Should().BeEmpty( - "the later subscription must not start after publish cancellation"); - } - - [Fact] - public async Task A_Handler_Constructor_Sees_The_Events_Tenant() - { - // Counting the handlers resolves them — the container materialises the - // array to count it — and that happened before the tenant context was - // set, so every handler CONSTRUCTOR ran under the publisher's tenant - // instead of the event's. A constructor that injects ITenantContext - // captured the wrong one and used it for the rest of the handler's life. - var recorder = new Recorder(); - var (bus, accessor) = Build(recorder, services => - services.AddScoped, TenantCapturingHandler>()); - - accessor.Current = EventTenantContext.FromEnvelope( - Envelope(NewThing("publisher") with { TenantId = OtherTenant })); - - await bus.PublishAsync(Envelope(NewThing("a"))); - - recorder.Tenants.Should().NotBeEmpty(); - recorder.Tenants.Should().OnlyContain(t => t == Tenant, - "every construction and every handle runs under the event's tenant"); - } - - [Fact] - public async Task A_Handler_That_Cannot_Be_Built_Is_Reported_As_Such() - { - // Subscription metadata is construction-free. A broken graph therefore - // belongs only to that subscription and cannot deny a healthy sibling. - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - { - services.AddScoped, ThingHandler>(); - services.AddScoped, UnconstructableHandler>(); - }); - - var act = () => bus.PublishAsync(Envelope(NewThing("a"))); - - var thrown = await act.Should().ThrowAsync(); - thrown.Which.Message.Should().Contain("failed to construct"); - thrown.Which.InnerException!.Message.Should().Be("this handler cannot be built"); - recorder.Handled.Should().ContainSingle().Which.Should().Be("a", - "the healthy subscription is constructed and invoked independently"); - } + await ((Func)(() => bus.PublishAsync(null!))) + .Should().ThrowAsync(); + await ((Func)(() => bus.PublishAsync(Envelope(New()), cancellation.Token))) + .Should().ThrowAsync(); - [Fact] - public async Task A_Handler_Returning_No_Task_Is_Named() - { - // Otherwise the caller gets a bare NullReferenceException from inside a - // transport it did not know it was in. - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - services.AddScoped, NullTaskHandler>()); - - var act = () => bus.PublishAsync(Envelope(NewThing("a"))); - - (await act.Should().ThrowAsync()) - .Which.Message.Should().Contain(nameof(NullTaskHandler)); - } - - [Fact] - public async Task The_Dispatch_Scope_Is_Disposed() - { - // An undisposed scope leaks a DbContext per publish, and nothing noticed. - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - { - services.AddScoped(); - services.AddScoped, DisposalProbingHandler>(); - }); - - await bus.PublishAsync(Envelope(NewThing("a"))); - - recorder.Probes.Should().ContainSingle().Which.Disposed.Should().BeTrue(); + scopes.Calls.Should().Be(0); + accessor.Current.Should().BeNull(); } [Fact] - public async Task An_Async_Only_Disposable_Dependency_Is_Disposed_Exactly_Once() + public async Task Cancellation_while_queued_prevents_acceptance_scope_creation() { - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => + var accessor = new FieldAccessor(); + var scopes = new ScopeProbe(accessor); + var partitions = new PartitionSerializer(); + var bus = Build(scopes, accessor, partitions); + var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var release = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var holding = partitions.RunSequentiallyFor(Subject.ToString("D"), async () => { - services.AddScoped(); - services.AddScoped, AsyncDisposalProbingHandler>(); + entered.SetResult(); + await release.Task; }); - - await bus.PublishAsync(Envelope(NewThing("a"))); - - recorder.AsyncProbes.Should().ContainSingle(); - recorder.AsyncProbes.Single().DisposeCalls.Should().Be(1); - } - - [Fact] - public async Task Each_Subscription_Continues_The_Producer_Trace() - { - var stopped = new ConcurrentBag(); - using var listener = Listen(stopped); - - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - services.AddScoped, ActivityReadingHandler>()); - - await bus.PublishAsync(Envelope(NewThing("a"))); - - stopped.Should().ContainSingle(); - var activity = stopped.Single(); - activity.Kind.Should().Be(ActivityKind.Consumer); - activity.TraceId.ToString().Should().Be("0af7651916cd43dd8448eb211c80319c"); - activity.ParentSpanId.ToString().Should().Be("b7ad6b7169203331"); - recorder.Modules.Should().ContainSingle().Which.Should().Be("unknown"); - } - - [Fact] - public async Task A_Failed_Delivery_Marks_Its_Consumer_Span() - { - // The span and the log line describe the same delivery, so a span left - // Unset when the handler threw is worse than no span: an operator - // filtering a trace backend for errors finds a green consumer next to - // the error log, and concludes the failure happened somewhere else. - var stopped = new ConcurrentBag(); - using var listener = Listen(stopped); - - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - services.AddScoped, ThrowingHandler>()); - - var act = () => bus.PublishAsync(Envelope(NewThing("a"))); - - await act.Should().ThrowAsync(); - var activity = stopped.Should().ContainSingle().Which; - activity.Status.Should().Be(ActivityStatusCode.Error); - activity.StatusDescription.Should().Be(nameof(InvalidOperationException)); - activity.Events.Should().ContainSingle() - .Which.Tags.Should().Contain( - tag => tag.Key == "exception.type" - && (string?)tag.Value == typeof(InvalidOperationException).FullName); - } - - [Fact] - public async Task Publish_Cancellation_Leaves_Its_Consumer_Span_Unset() - { - // The other half: shutdown is not a failure. Marking it Error would put - // one Error span per in-flight subscription into the 100%-sampled error - // traces every time the host stops, which is the same reason Standards - // 10 leaves a client disconnect Unset. - var stopped = new ConcurrentBag(); - using var listener = Listen(stopped); - - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - services.AddScoped, CancellationWaitingHandler>()); + await entered.Task.WaitAsync(TimeSpan.FromSeconds(10)); using var cancellation = new CancellationTokenSource(); - var publish = bus.PublishAsync(Envelope(NewThing("a")), cancellation.Token); - await recorder.HandlerEntered.Task.WaitAsync(Timeout); - await cancellation.CancelAsync(); - - await ((Func)(() => publish)).Should().ThrowAsync(); - var activity = stopped.Should().ContainSingle().Which; - activity.Status.Should().Be(ActivityStatusCode.Unset); - activity.Events.Should().BeEmpty(); - } - - private static ActivityListener Listen(ConcurrentBag stopped) - { - var listener = new ActivityListener + try { - ShouldListenTo = source => source.Name == InProcessEventBus.ActivitySourceName, - Sample = static (ref ActivityCreationOptions _) => - ActivitySamplingResult.AllDataAndRecorded, - ActivityStopped = stopped.Add, - }; - ActivitySource.AddActivityListener(listener); - return listener; - } - - // ---- obligation: ordering per partition key ------------------------------ + var queued = bus.PublishAsync(Envelope(New()), cancellation.Token); + await cancellation.CancelAsync(); + release.SetResult(); + await holding.WaitAsync(TimeSpan.FromSeconds(10)); - [Fact] - public async Task Two_Events_On_One_Partition_Key_Do_Not_Overlap() - { - // Ordering is guaranteed per partition key and nowhere else. An - // in-process transport that dispatched concurrently would let an - // ordering assumption pass every test and fail in production. - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - services.AddScoped, OverlapDetectingHandler>()); - - var first = bus.PublishAsync(Envelope(NewThing("a", "same-key"))); - var second = bus.PublishAsync(Envelope(NewThing("b", "same-key"))); - await Task.WhenAll(first, second).WaitAsync(Timeout); - - recorder.Overlapped.Should().BeFalse("dispatch is sequential within one key"); - recorder.Handled.Should().BeEquivalentTo(["a", "b"], o => o.WithStrictOrdering()); - } - - [Fact] - public async Task Different_Partition_Keys_Run_Concurrently() - { - // The other half of the guarantee: serialising everything would be - // correct and useless, so the test that proves ordering must be paired - // with one that proves it is not global. - // TWO gates, each side waiting on the OTHER's. Sharing one semaphore - // let each side consume its own release, so it never waited for - // anything and passed with both keys on a single chain. - var recorder = new Recorder(); - using var firstArrived = new SemaphoreSlim(0); - using var secondArrived = new SemaphoreSlim(0); - var (bus, _) = Build(recorder, services => - { - services.AddSingleton(new RendezvousGates(firstArrived, secondArrived)); - services.AddScoped, RendezvousHandler>(); - }); - - var first = bus.PublishAsync(Envelope(NewThing("a", "key-1"))); - var second = bus.PublishAsync(Envelope(NewThing("b", "key-2"))); - - await Task.WhenAll(first, second).WaitAsync(Timeout); - recorder.Rendezvoused.Should().Be(2, "neither key waited for the other"); - } - - [Fact] - public async Task A_Failed_Delivery_Does_Not_Block_The_Rest_Of_Its_Partition() - { - // A handler that throws must not stop every later event for that - // aggregate. The failure belongs to the one publish that caused it. - var recorder = new Recorder(); - var (bus, _) = Build(recorder, services => - { - services.AddScoped, ThrowOnFirstHandler>(); - }); - - var failing = bus.PublishAsync(Envelope(NewThing("boom", "same-key"))); - await ((Func)(() => failing)).Should().ThrowAsync(); - - await bus.PublishAsync(Envelope(NewThing("after", "same-key"))).WaitAsync(Timeout); - - recorder.Handled.Should().Contain("after"); - } - - // ---- helpers ------------------------------------------------------------- - - /// - /// Wraps an event for dispatch. - /// - /// - /// The partition key is the event's own — the envelope reads it and cannot - /// disagree with it. An earlier two-parameter shape could: every test here - /// published an event declaring one key with a different one passed - /// alongside, the transport used the parameter and never the event, and - /// nothing noticed. Ordering is guaranteed per partition key, so a key that - /// can differ from itself is a guarantee that cannot be stated. - /// - private static IntegrationEventEnvelope Envelope(Thing @event) => - new(@event, Trace); - - private static Thing NewThing(string payload, string? partitionKey = null) => new() - { - EventId = Guid.NewGuid(), - TenantId = Tenant, - OccurredAt = DateTimeOffset.UnixEpoch, - Payload = payload, - Key = partitionKey, - }; - - private static (IEventBus Bus, ITenantContextAccessor Accessor) Build( - Recorder recorder, Action register) - { - var accessor = new TestTenantContextAccessor(); - - var services = new ServiceCollection(); - services.AddSingleton(recorder); - - // The SAME accessor the bus writes through, so a handler resolved from - // the dispatch scope reads what the transport restored rather than a - // second, empty one. - services.AddSingleton(accessor); - - // The production binding, verbatim (CrossCuttingFoundationExtensions): - // the transient context forwards to the accessor on every resolution. Registering anything - // else here would test a container this application never builds. - services.AddTransient(sp => - sp.GetRequiredService().Current - ?? UnresolvedTenantContext.Instance); - - register(services); - - var handlers = IntegrationEventHandlerRegistry.FromServiceDescriptors(services); - foreach (var subscription in handlers.All) - { - if (services.All(descriptor => descriptor.ServiceType != subscription.HandlerType)) - { - services.AddScoped(subscription.HandlerType, subscription.HandlerType); - } + await ((Func)(() => queued)).Should().ThrowAsync(); + scopes.Calls.Should().Be(0); + accessor.Current.Should().BeNull(); } - - var provider = services.BuildServiceProvider(); - - return ( - new InProcessEventBus( - provider.GetRequiredService(), - accessor, - new PartitionSerializer(), - handlers, - NullLogger.Instance), - accessor); - } - - /// - /// A plain field-backed accessor, deliberately NOT . - /// - /// - /// promises nothing about flow - /// isolation, and the production implementation being AsyncLocal-backed is a - /// detail of another assembly. With an AsyncLocal accessor a leaked context - /// is invisible here — dispatch runs in its own flow, so the write never - /// reaches the publisher and the restore looks unnecessary even when it is - /// removed. This accessor makes the guarantee observable, which is the only - /// way the test constrains the transport rather than the accessor. - /// - private sealed class TestTenantContextAccessor : ITenantContextAccessor - { - public ITenantContext? Current { get; set; } - } - - public sealed class Recorder - { - private int _inFlight; - - public ConcurrentQueue Handled { get; } = new(); - - public ConcurrentQueue Tenants { get; } = new(); - - public ConcurrentQueue Actors { get; } = new(); - - public ConcurrentQueue CausalActors { get; } = new(); - - public ConcurrentQueue Organizations { get; } = new(); - - public ConcurrentQueue Scopes { get; } = new(); - - public ConcurrentQueue Probes { get; } = new(); - - public ConcurrentQueue AsyncProbes { get; } = new(); - - public ConcurrentQueue Modules { get; } = new(); - - public TaskCompletionSource HandlerEntered { get; } = - new(TaskCreationOptions.RunContinuationsAsynchronously); - - public CancellationToken HandlerToken { get; set; } - - public int Rendezvoused => _rendezvoused; - - private int _rendezvoused; - - public void Rendezvous() => Interlocked.Increment(ref _rendezvoused); - - /// Whether two handlers were ever inside the dispatch at once. - public bool Overlapped { get; private set; } - - public void Enter() - { - if (Interlocked.Increment(ref _inFlight) > 1) - { - Overlapped = true; - } - } - - public void Exit() => Interlocked.Decrement(ref _inFlight); - } - - public sealed record Thing : IntegrationEventBase - { - public required string Payload { get; init; } - - /// An ordering domain independent of the payload, for the ordering cases. - public string? Key { get; init; } - - public override string Topic => "learnstack.test.thing"; - - public override string PartitionKey => Key ?? Payload; - } - - public sealed class ThingHandler(Recorder recorder) : IIntegrationEventHandler - { - public Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) - { - recorder.Handled.Enqueue(@event.Payload); - return Task.CompletedTask; - } - } - - public sealed class SecondThingHandler(Recorder recorder) : IIntegrationEventHandler - { - public Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) + finally { - recorder.Handled.Enqueue($"second:{@event.Payload}"); - return Task.CompletedTask; + release.TrySetResult(); + await holding.WaitAsync(TimeSpan.FromSeconds(10)); } } - internal sealed class InternalThingHandler(Recorder recorder) : IIntegrationEventHandler - { - public Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) - { - recorder.Handled.Enqueue($"internal:{@event.Payload}"); - return Task.CompletedTask; - } - } - - public sealed class TenantReadingHandler(Recorder recorder, ITenantContextAccessor accessor) - : IIntegrationEventHandler - { - public Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) - { - recorder.Tenants.Enqueue(accessor.Current!.TenantId.Value); - return Task.CompletedTask; - } - } - - public sealed class ScopedContextReadingHandler(Recorder recorder, ITenantContext context) - : IIntegrationEventHandler - { - public Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) - { - recorder.Tenants.Enqueue(context.TenantId.Value); - return Task.CompletedTask; - } - } - - public sealed class ActorReadingHandler(Recorder recorder, ITenantContext context) - : IIntegrationEventHandler - { - public Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) - { - recorder.Actors.Enqueue(context.UserId!.Value); - - if (context.CausalActorUserId is { } causalActor) - { - recorder.CausalActors.Enqueue(causalActor); - } - - if (context.OrganizationId is { } organization) - { - recorder.Organizations.Enqueue(organization.Value); - } - - return Task.CompletedTask; - } - } - - public sealed class ScopeMarker - { - public Guid Id { get; } = Guid.NewGuid(); - } - - public sealed class ScopeReadingHandler(Recorder recorder, ScopeMarker marker) - : IIntegrationEventHandler - { - public Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) - { - recorder.Scopes.Enqueue(marker.Id); - return Task.CompletedTask; - } - } - - public sealed class SecondScopeReadingHandler(Recorder recorder, ScopeMarker marker) - : IIntegrationEventHandler - { - public Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) - { - recorder.Scopes.Enqueue(marker.Id); - return Task.CompletedTask; - } - } - - public sealed class ForeignCancelHandler : IIntegrationEventHandler - { - public Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) - { - using var unrelated = new CancellationTokenSource(); - unrelated.Cancel(); - unrelated.Token.ThrowIfCancellationRequested(); - return Task.CompletedTask; - } - } - - public sealed class AsyncThrowingHandler : IIntegrationEventHandler - { - public async Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) - { - await Task.Yield(); - throw new InvalidOperationException("async handler failed"); - } - } + private static InProcessEventBus Build( + IServiceScopeFactory scopes, ITenantContextAccessor accessor, + IPartitionSerializer? partitions = null) + { + DeclaredHandler.Constructions = 0; + var descriptors = new ServiceCollection(); + descriptors.AddScoped, DeclaredHandler>(); + var handlers = IntegrationEventHandlerRegistry.FromServiceDescriptors(descriptors); + var registry = OutboxEventRegistry.Create( + [ + OutboxEventRegistration.Required(typeof(ActiveEvent), + [OutboxSubscriptionManifest.Active("test.active.v1", typeof(DeclaredHandler))]), + OutboxEventRegistration.IntentionallyUnsubscribed(typeof(AbsentEvent)), + OutboxEventRegistration.Required(typeof(PendingEvent), + [OutboxSubscriptionManifest.Pending("test.pending.v1")]), + OutboxEventRegistration.IntentionallyUnsubscribed(typeof(InvalidShapeEvent)), + OutboxEventRegistration.IntentionallyUnsubscribed(typeof(InvalidPartitionEvent)), + ], + [typeof(RetiredEvent).AssemblyQualifiedName!], + handlers, + [typeof(ActiveEvent).Assembly]); + return new InProcessEventBus(scopes, accessor, partitions ?? new PartitionSerializer(), registry); + } + + private static IntegrationEventEnvelope Envelope(IIntegrationEvent fact) => new(fact, Trace); + + private static T New() where T : IntegrationEventBase => typeof(T) switch + { + var type when type == typeof(ActiveEvent) => (T)(IntegrationEventBase)new ActiveEvent + { EventId = Guid.NewGuid(), TenantId = Tenant, OccurredAt = DateTimeOffset.UnixEpoch, SubjectId = Subject }, + var type when type == typeof(AbsentEvent) => (T)(IntegrationEventBase)new AbsentEvent + { EventId = Guid.NewGuid(), TenantId = Tenant, OccurredAt = DateTimeOffset.UnixEpoch, SubjectId = Subject }, + var type when type == typeof(PendingEvent) => (T)(IntegrationEventBase)new PendingEvent + { EventId = Guid.NewGuid(), TenantId = Tenant, OccurredAt = DateTimeOffset.UnixEpoch, SubjectId = Subject }, + var type when type == typeof(UnknownEvent) => (T)(IntegrationEventBase)new UnknownEvent + { EventId = Guid.NewGuid(), TenantId = Tenant, OccurredAt = DateTimeOffset.UnixEpoch, SubjectId = Subject }, + var type when type == typeof(RetiredEvent) => (T)(IntegrationEventBase)new RetiredEvent + { EventId = Guid.NewGuid(), TenantId = Tenant, OccurredAt = DateTimeOffset.UnixEpoch, SubjectId = Subject }, + var type when type == typeof(InvalidShapeEvent) => (T)(IntegrationEventBase)new InvalidShapeEvent + { EventId = Guid.NewGuid(), TenantId = Tenant, OccurredAt = DateTimeOffset.UnixEpoch, SubjectId = Subject, Label = "invalid" }, + var type when type == typeof(InvalidPartitionEvent) => (T)(IntegrationEventBase)new InvalidPartitionEvent + { EventId = Guid.NewGuid(), TenantId = Tenant, OccurredAt = DateTimeOffset.UnixEpoch, SubjectId = Subject }, + _ => throw new ArgumentOutOfRangeException(nameof(T)), + }; - public sealed class LateTenantReadingHandler(Recorder recorder, ITenantContextAccessor accessor) - : IIntegrationEventHandler + private sealed class FieldAccessor : ITenantContextAccessor { - public async Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) - { - await Task.Delay(TimeSpan.FromMilliseconds(20), CancellationToken.None); - recorder.Tenants.Enqueue(accessor.Current!.TenantId.Value); - } + public ITenantContext? Current { get; set; } } - public sealed class TokenReadingHandler(Recorder recorder) : IIntegrationEventHandler + private sealed class ScopeProbe(ITenantContextAccessor accessor) : IServiceScopeFactory { - public Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) - { - recorder.HandlerToken = cancellationToken; - return Task.CompletedTask; - } - } + public int Calls { get; private set; } + public ITenantContext? Context { get; private set; } - public sealed class CancellationWaitingHandler(Recorder recorder) - : IIntegrationEventHandler - { - public async Task HandleAsync( - Thing @event, - CancellationToken cancellationToken = default) + public IServiceScope CreateScope() { - recorder.HandlerEntered.TrySetResult(); - await Task.Delay(System.Threading.Timeout.InfiniteTimeSpan, cancellationToken); + Calls++; + Context = accessor.Current; + throw new ScopeProbeException(); } } - public sealed class ActivityReadingHandler(Recorder recorder, ITenantContext context) - : IIntegrationEventHandler - { - public Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) - { - recorder.Modules.Enqueue(context.ModuleName!); - Activity.Current.Should().NotBeNull(); - return Task.CompletedTask; - } - } + private sealed class ScopeProbeException : Exception { } - public sealed class TenantCapturingHandler : IIntegrationEventHandler + private sealed class RecordingPartitions : IPartitionSerializer { - private readonly Recorder _recorder; - - public TenantCapturingHandler(Recorder recorder, ITenantContext context) - { - _recorder = recorder; - - // Captured at CONSTRUCTION, which is the point. - recorder.Tenants.Enqueue(context.TenantId.Value); - } - - public Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) + public List Keys { get; } = []; + public Task RunSequentiallyFor(string partitionKey, Func work) { - _recorder.Tenants.Enqueue(@event.TenantId); - return Task.CompletedTask; + Keys.Add(partitionKey); + return work(); } } - public sealed class NullTaskHandler : IIntegrationEventHandler - { - public Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) => - null!; - } - - public sealed class UnconstructableHandler : IIntegrationEventHandler + [IntegrationEventPartition(nameof(SubjectId))] + private sealed record ActiveEvent : IntegrationEventBase { - public UnconstructableHandler() => - throw new InvalidOperationException("this handler cannot be built"); - - public Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) => - Task.CompletedTask; + public required Guid SubjectId { get; init; } + public override string Topic => "learnstack.test.active"; + public override string PartitionKey => SubjectId.ToString("D"); } - public sealed class DisposalProbe : IDisposable + [IntegrationEventPartition(nameof(SubjectId))] + private sealed record AbsentEvent : IntegrationEventBase { - public bool Disposed { get; private set; } - - public void Dispose() => Disposed = true; + public required Guid SubjectId { get; init; } + public override string Topic => "learnstack.test.absent"; + public override string PartitionKey => SubjectId.ToString("D"); } - public sealed class DisposalProbingHandler(Recorder recorder, DisposalProbe probe) - : IIntegrationEventHandler + [IntegrationEventPartition(nameof(SubjectId))] + private sealed record PendingEvent : IntegrationEventBase { - public Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) - { - recorder.Probes.Enqueue(probe); - return Task.CompletedTask; - } + public required Guid SubjectId { get; init; } + public override string Topic => "learnstack.test.pending"; + public override string PartitionKey => SubjectId.ToString("D"); } - public sealed class AsyncDisposalProbe : IAsyncDisposable + [IntegrationEventPartition(nameof(SubjectId))] + private sealed record UnknownEvent : IntegrationEventBase { - public int DisposeCalls { get; private set; } - - public ValueTask DisposeAsync() - { - DisposeCalls++; - return ValueTask.CompletedTask; - } + public required Guid SubjectId { get; init; } + public override string Topic => "learnstack.test.unknown"; + public override string PartitionKey => SubjectId.ToString("D"); } - public sealed class AsyncDisposalProbingHandler( - Recorder recorder, - AsyncDisposalProbe probe) : IIntegrationEventHandler + [IntegrationEventPartition(nameof(SubjectId))] + private sealed record RetiredEvent : IntegrationEventBase { - public Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) - { - recorder.AsyncProbes.Enqueue(probe); - return Task.CompletedTask; - } + public required Guid SubjectId { get; init; } + public override string Topic => "learnstack.test.retired"; + public override string PartitionKey => SubjectId.ToString("D"); } - public sealed class ThrowingHandler : IIntegrationEventHandler + [IntegrationEventPartition(nameof(SubjectId))] + private sealed record InvalidShapeEvent : IntegrationEventBase { - public Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) => - throw new InvalidOperationException("handler failed"); + public required Guid SubjectId { get; init; } + public required string Label { get; init; } + public override string Topic => "learnstack.test.invalidshape"; + public override string PartitionKey => SubjectId.ToString("D"); } - public sealed class ThrowOnFirstHandler(Recorder recorder) : IIntegrationEventHandler + [IntegrationEventPartition(nameof(SubjectId))] + private sealed record InvalidPartitionEvent : IntegrationEventBase { - public Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) - { - if (@event.Payload == "boom") - { - throw new InvalidOperationException("handler failed"); - } - - recorder.Handled.Enqueue(@event.Payload); - return Task.CompletedTask; - } + public required Guid SubjectId { get; init; } + public override string Topic => "learnstack.test.invalidpartition"; + public override string PartitionKey => TenantId.ToString("D"); } - public sealed class OverlapDetectingHandler(Recorder recorder) : IIntegrationEventHandler + private sealed class DeclaredHandler : IIntegrationEventHandler { - public async Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) + public static int Constructions; + public DeclaredHandler() { - recorder.Enter(); - await Task.Delay(TimeSpan.FromMilliseconds(30), CancellationToken.None); - recorder.Handled.Enqueue(@event.Payload); - recorder.Exit(); + Constructions++; + throw new InvalidOperationException("Publication constructed a business handler."); } - } - - public sealed record RendezvousGates(SemaphoreSlim First, SemaphoreSlim Second); - public sealed class RendezvousHandler(Recorder recorder, RendezvousGates gates) - : IIntegrationEventHandler - { - public async Task HandleAsync(Thing @event, CancellationToken cancellationToken = default) - { - // The key decides which gate is this side's, so the two invocations - // never pick the same one — a shared counter would be shared across - // tests running in parallel. - var mine = @event.PartitionKey == "key-1" ? gates.First : gates.Second; - var theirs = ReferenceEquals(mine, gates.First) ? gates.Second : gates.First; - - mine.Release(); - (await theirs.WaitAsync(Timeout, CancellationToken.None)).Should().BeTrue( - "the other key's handler must be running at the same time"); - - recorder.Rendezvous(); - } + public Task HandleAsync(ActiveEvent integrationEvent, + CancellationToken cancellationToken = default) => Task.CompletedTask; } } diff --git a/backend/tests/LearnStack.Tests.Unit/Infrastructure/Messaging/OutboxEventRegistryTests.cs b/backend/tests/LearnStack.Tests.Unit/Infrastructure/Messaging/OutboxEventRegistryTests.cs index 10edf9ac..e2b26288 100644 --- a/backend/tests/LearnStack.Tests.Unit/Infrastructure/Messaging/OutboxEventRegistryTests.cs +++ b/backend/tests/LearnStack.Tests.Unit/Infrastructure/Messaging/OutboxEventRegistryTests.cs @@ -99,6 +99,20 @@ public void Subscription_ids_follow_the_closed_stable_grammar(string id) build.Should().Throw(); } + [Fact] + public void An_undefined_subscription_execution_scope_has_its_own_bounded_diagnostic() + { + var build = () => Create( + [OutboxEventRegistration.Required(typeof(SampleEvent), + [OutboxSubscriptionManifest.Active("test.sample.v1", typeof(SampleHandler), + (OutboxSubscriptionExecutionScope)int.MaxValue)])], + handlers: Handlers(services => + services.AddScoped, SampleHandler>())); + + build.Should().Throw() + .WithMessage("Outbox subscription execution scopes must be declared values."); + } + [Fact] public void Alias_and_retired_collisions_refuse_composition() { diff --git a/backend/tests/LearnStack.Tests.Unit/Modules/Customization/CustomizationCommandTests.cs b/backend/tests/LearnStack.Tests.Unit/Modules/Customization/CustomizationCommandTests.cs index ef992dad..85ce8e57 100644 --- a/backend/tests/LearnStack.Tests.Unit/Modules/Customization/CustomizationCommandTests.cs +++ b/backend/tests/LearnStack.Tests.Unit/Modules/Customization/CustomizationCommandTests.cs @@ -2,7 +2,9 @@ using FluentValidation; using LearnStack.Modules.Customization.Application.Abstractions; using LearnStack.Modules.Customization.Application.Contracts.Customization; +using LearnStack.Modules.Customization.Application.IntegrationEvents; using LearnStack.Modules.Customization.Domain; +using LearnStack.Modules.Tenancy.Application.Contracts.IntegrationEvents; using LearnStack.SharedKernel.Audit; using LearnStack.SharedKernel.Domain; using LearnStack.SharedKernel.Identifiers; @@ -77,6 +79,103 @@ private static RegisterTenantLevelTaxonomyCommand RegisterTaxonomy( // ── The write path ──────────────────────────────────────────────────── + [Fact] + public async Task Initialization_creates_the_complete_active_content_type_with_a_generated_id_in_one_write() + { + var gate = new RecordingGate(); + var (sender, stores) = Build(gate: gate); + + (await sender.Send(new InitializeTenantContentTypeDefaultCommand())).IsSuccess.Should().BeTrue(); + + stores.Coordination.Should().Equal("content-type:lock:card", "content-type:history:card", "content-type:add:Active", "generation:bump"); + stores.Writes.Should().Equal("content-type:add", "generation:bump"); + var root = stores.ContentTypes.Values.Should().ContainSingle().Which; + root.Id.Value.Should().Be(ContentTypeId); + root.TenantId.Should().Be(Tenant); + root.Key.Should().Be("card"); + root.SchemaVersion.Should().Be(1); + root.SchemaRevision.Should().Be(0); + root.DisplayName.Should().Be(LocalizedText.From(BuiltInCustomizations.Card.DisplayName)); + root.JsonSchema.Should().Be(BuiltInCustomizations.Card.JsonSchema); + root.RendererKey.Should().Be(BuiltInCustomizations.Card.RendererKey); + root.CreatedAt.Should().Be(Clock.UtcNow); + root.CreatedBy.Should().Be(UserId.SystemActor); + gate.Admitted.Should().Equal(BuiltInCustomizations.Card.JsonSchema); + } + + [Fact] + public async Task Initialization_creates_the_complete_active_taxonomy_and_its_bands_in_one_write() + { + var (sender, stores) = Build(); + + (await sender.Send(new InitializeTenantLevelTaxonomyDefaultCommand())).IsSuccess.Should().BeTrue(); + + stores.Coordination.Should().Equal("taxonomy:lock:plain", "taxonomy:history:plain", "taxonomy:add:Active", "generation:bump"); + stores.Writes.Should().Equal("taxonomy:add", "generation:bump"); + var root = stores.AddedTaxonomies.Should().ContainSingle().Which; + root.Id.Value.Should().Be(ContentTypeId, "the factory, not a demo constant, owns identity"); + root.TenantId.Should().Be(Tenant); + root.Key.Should().Be("plain"); + root.SchemaVersion.Should().Be(1); + root.DisplayName.Should().Be(LocalizedText.From(BuiltInCustomizations.Plain.DisplayName)); + root.Items.Select(item => (item.Key, item.DisplayName, item.Sort, item.Metadata)).Should().Equal( + BuiltInCustomizations.Plain.Bands.Select(band => (band.Key, LocalizedText.From(band.DisplayName), band.Sort, (string?)null))); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Initialization_preserves_successor_only_history_without_validation_or_writes(bool taxonomy) + { + var gate = new RecordingGate(); + var (sender, stores) = Build(gate: gate); + if (taxonomy) await sender.Send(RegisterTaxonomy(key: "plain", version: 2)); + else await sender.Send(RegisterContentType(key: "card", version: 2)); + stores.Writes.Clear(); + stores.Coordination.Clear(); + gate.Admitted.Clear(); + + var result = taxonomy + ? await sender.Send(new InitializeTenantLevelTaxonomyDefaultCommand()) + : await sender.Send(new InitializeTenantContentTypeDefaultCommand()); + + result.IsSuccess.Should().BeTrue(); + if (taxonomy) stores.Coordination.Should().Equal("taxonomy:lock:plain", "taxonomy:history:plain"); + else stores.Coordination.Should().Equal("content-type:lock:card", "content-type:history:card"); + stores.Writes.Should().BeEmpty(); + gate.Admitted.Should().BeEmpty(); + } + + [Fact] + public async Task Initialization_refuses_a_failed_builtin_schema_gate_before_any_write() + { + var (sender, stores) = Build(admitSchemas: false); + (await sender.Send(new InitializeTenantContentTypeDefaultCommand())).IsFailure.Should().BeTrue(); + stores.Writes.Should().BeEmpty(); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public async Task Each_consumer_throws_a_bounded_exception_when_its_initializer_refuses(bool taxonomy) + { + var (sender, stores) = Build(resolved: false); + var message = new TenantCreatedIntegrationEventV1 + { + EventId = ContentTypeId, + TenantId = Tenant.Value, + OccurredAt = Clock.UtcNow, + }; + Func execute = taxonomy + ? () => new InitializeTaxonomyDefaultHandler(sender).HandleAsync(message) + : () => new InitializeContentTypeDefaultHandler(sender).HandleAsync(message); + (await execute.Should().ThrowAsync()).WithMessage(taxonomy + ? "Taxonomy default initialization was refused." + : "Content type default initialization was refused."); + stores.Writes.Should().BeEmpty(); + stores.Coordination.Should().BeEmpty(); + } + [Fact] public async Task Registering_a_content_type_writes_a_draft_and_bumps_the_generation() { @@ -1274,6 +1373,7 @@ private static ServiceProvider Provider( services.AddSingleton(stores); services.AddSingleton(gate ?? new StubGate(admitSchemas)); services.AddSingleton(Clock); + services.AddSingleton(new FixedGuidFactory(ContentTypeId)); services.AddSingleton(); services.AddSingleton( resolved ? new ResolvedContext(Tenant) : new UnresolvedTenantContext()); @@ -1299,6 +1399,32 @@ private sealed class RecordingStores public List Writes { get; } = []; + public List Coordination { get; } = []; + + Task ITenantContentTypeStore.LockConceptAsync(string key, CancellationToken cancellationToken) + { + Coordination.Add("content-type:lock:" + key); + return Task.CompletedTask; + } + + Task ITenantLevelTaxonomyStore.LockConceptAsync(string key, CancellationToken cancellationToken) + { + Coordination.Add("taxonomy:lock:" + key); + return Task.CompletedTask; + } + + Task ITenantContentTypeStore.HasConceptHistoryAsync(string key, CancellationToken cancellationToken) + { + Coordination.Add("content-type:history:" + key); + return Task.FromResult(ContentTypes.Values.Any(root => root.Key == key)); + } + + Task ITenantLevelTaxonomyStore.HasConceptHistoryAsync(string key, CancellationToken cancellationToken) + { + Coordination.Add("taxonomy:history:" + key); + return Task.FromResult(Taxonomies.Values.Any(root => root.Key == key)); + } + /// Every instance a handler designated as its audit row's subject, in order. /// /// Kept apart from : a designation is not a write, and the @@ -1357,6 +1483,7 @@ Task IAggregateWriteStore.AddAsync( { Conflict(); Writes.Add("content-type:add"); + Coordination.Add("content-type:add:" + aggregate.Status); ContentTypes[aggregate.Id.Value] = aggregate; return Task.CompletedTask; } @@ -1387,6 +1514,7 @@ Task IAggregateWriteStore.AddAsync( { Conflict(); Writes.Add("taxonomy:add"); + Coordination.Add("taxonomy:add:" + aggregate.Status); Taxonomies[aggregate.Id.Value] = aggregate; AddedTaxonomies.Add(aggregate); return Task.CompletedTask; @@ -1436,6 +1564,7 @@ Task> ITenantLevelTaxonomyCatalog.ExistingAsync( public Task BumpAsync(TenantId tenantId, CancellationToken cancellationToken = default) { Writes.Add("generation:bump"); + Coordination.Add("generation:bump"); return Task.FromResult(++_generation); } diff --git a/backend/tests/LearnStack.Tests.Unit/SharedKernel/Audit/DeliveryAuditMetadataTests.cs b/backend/tests/LearnStack.Tests.Unit/SharedKernel/Audit/DeliveryAuditMetadataTests.cs new file mode 100644 index 00000000..6255076b --- /dev/null +++ b/backend/tests/LearnStack.Tests.Unit/SharedKernel/Audit/DeliveryAuditMetadataTests.cs @@ -0,0 +1,136 @@ +using System.Text.Json; +using FluentAssertions; +using LearnStack.SharedKernel.Audit; +using LearnStack.SharedKernel.Identifiers; +using LearnStack.SharedKernel.Messaging; +using LearnStack.SharedKernel.Tenancy; +using Xunit; + +namespace LearnStack.Tests.Unit.SharedKernel.Audit; + +public sealed class DeliveryAuditMetadataTests +{ + private const string Traceparent = "00-11111111111111111111111111111111-2222222222222222-01"; + private static readonly DateTimeOffset Declared = DateTimeOffset.UnixEpoch.AddMinutes(1); + private static readonly Guid IncomingEventId = Guid.Parse("bbbbbbbb-0000-7000-8000-000000000001"); + private static readonly UserId CausalActor = UserId.From(Guid.Parse("cccccccc-0000-7000-8000-000000000001")); + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void Both_writers_preserve_delivery_identifiers_without_impersonating_the_causal_actor(bool hasActor) + { + var intent = Intent() with + { + CausationId = IncomingEventId, + CausalActorUserId = hasActor ? CausalActor : null, + }; + + foreach (var row in BothPaths(intent)) + { + row.ActorUserId.Should().Be(UserId.SystemActor); + row.CorrelationId.Should().Be(Traceparent); + using var metadata = JsonDocument.Parse(row.Metadata!); + var properties = metadata.RootElement.EnumerateObject().ToArray(); + string[] expectedNames = hasActor + ? ["causation_id", "causal_actor_user_id"] + : ["causation_id"]; + properties.Select(property => property.Name).Should().Equal(expectedNames); + properties.Should().OnlyContain(property => property.Value.ValueKind == JsonValueKind.String); + metadata.RootElement.GetProperty("causation_id").GetGuid().Should().Be(IncomingEventId); + if (hasActor) + { + metadata.RootElement.GetProperty("causal_actor_user_id").GetGuid().Should().Be(CausalActor.Value); + } + } + } + + [Fact] + public void Ordinary_request_intents_keep_null_metadata_and_their_effective_actor() + { + var intent = Intent() with { ActorUserId = CausalActor }; + foreach (var row in BothPaths(intent)) + { + row.Metadata.Should().BeNull(); + row.ActorUserId.Should().Be(CausalActor); + } + } + + [Fact] + public void A_causal_actor_alone_is_still_identifier_only_metadata() + { + foreach (var row in BothPaths(Intent() with { CausalActorUserId = CausalActor })) + { + using var metadata = JsonDocument.Parse(row.Metadata!); + metadata.RootElement.EnumerateObject().Should().ContainSingle() + .Which.Name.Should().Be("causal_actor_user_id"); + metadata.RootElement.GetProperty("causal_actor_user_id").GetGuid().Should().Be(CausalActor.Value); + } + } + + [Theory] + [InlineData("empty_event")] + [InlineData("empty_actor")] + [InlineData("uninitialized_actor")] + public void Both_writers_refuse_invalid_causal_identifiers(string invalid) + { + var intent = invalid switch + { + "empty_event" => Intent() with { CausationId = Guid.Empty }, + "empty_actor" => Intent() with { CausalActorUserId = UserId.From(Guid.Empty) }, + _ => Intent() with { CausalActorUserId = (new UserId[1])[0] }, + }; + + FluentActions.Invoking(() => AuditDraftComposer.InTransaction(intent, [])) + .Should().Throw(); + FluentActions.Invoking(() => AuditDraftComposer.Reconciled( + intent, [], AuditIntentState.Indeterminate, Declared.AddSeconds(1))) + .Should().Throw(); + } + + [Fact] + public void Event_context_carries_the_incoming_identity_separately_from_its_ancestors_causation() + { + var incoming = new ProbeEvent + { + EventId = IncomingEventId, + TenantId = Guid.CreateVersion7(), + OccurredAt = Declared, + }; + var ancestor = Guid.CreateVersion7(); + var context = EventTenantContext.FromEnvelope(new IntegrationEventEnvelope( + incoming, Traceparent, CausationId: ancestor, ActorUserId: CausalActor)); + + context.EventId.Should().Be(IncomingEventId).And.NotBe(ancestor); + context.CausalActorUserId.Should().Be(CausalActor); + context.UserId.Should().Be(UserId.SystemActor); + } + + private static IEnumerable BothPaths(AuditIntent intent) + { + yield return AuditDraftComposer.InTransaction(intent, []); + foreach (var state in new[] { AuditIntentState.RolledBack, AuditIntentState.Indeterminate, AuditIntentState.Committed }) + { + yield return AuditDraftComposer.Reconciled(intent, [], state, Declared.AddSeconds(1)); + } + } + + private static AuditIntent Intent() => new( + AuditEntryId.From(Guid.CreateVersion7()), + TenantId.From(Guid.CreateVersion7()), + OrganizationId: null, + ActorUserId: UserId.SystemActor, + CorrelationId: Traceparent, + ModuleName: "customization", + Operation: "customization.content_type.initialize", + OperationType: OperationType.Create, + OperationClass: OperationClass.Must, + EntityType: null, + DeclaredAt: Declared); + + private sealed record ProbeEvent : IntegrationEventBase + { + public override string Topic => "learnstack.customization.probe.v1"; + public override string PartitionKey => TenantId.ToString("D"); + } +} diff --git a/docs/architecture/03-module-boundaries.md b/docs/architecture/03-module-boundaries.md index 7942073f..1cfe39c3 100644 --- a/docs/architecture/03-module-boundaries.md +++ b/docs/architecture/03-module-boundaries.md @@ -239,7 +239,7 @@ Owns external provider credentials, webhooks, API keys, LTI/xAPI readiness, inte | Subscribe to another module's **integration event** — delivered from the outbox through `IEventBus` (`InProcessEventBus` today, Dapr pub/sub on its trigger). | Cross-module EF navigation properties. | | Read another module's **public read model** (projection table). | Joining across module-owned tables in SQL. | | Use the **shared kernel** (ids, audit fields, errors, pagination, base types, `ICacheService`, `ISecretProvider`, `IFeatureFlags`). | Importing another module's `Domain` namespace. | -| Publish an integration event by enqueueing it in the **outbox** ([Phase 02b](../roadmap/phase-02b-events-auth.md) ships `IOutbox`). | Holding `IEventBus`: a module that publishes directly gets a synchronous cross-module call with no durability (`Modules_Do_Not_Inject_IEventBus_Directly`). | +| Publish an integration event by enqueueing it in the **outbox** through `IOutbox`, implemented by [P02b-1](../roadmap/phase-02b-events-auth.md#p02b-1-step-2--immediate-outbox-production-2026-10-12). | Holding `IEventBus`: direct publication bypasses the transactional producer outbox (`Modules_Do_Not_Inject_IEventBus_Directly`). | | Provide an **adapter implementation** at the composition root. | Domain-specific names (`CEFR`, `Asana`, `Kyu`, …) anywhere in a core module — those belong to tenant customization data, not code. | | Ask **`IFeatureFlags`** for an entitlement and **`IHostToTenantResolver`** for a host — the two ports over the Hub-mirrored projections. | Reading `platform_entitlement_cache` or `platform_host_to_tenant` directly: the entitlement provider and the host resolver are their only readers — not `IFeatureFlags`, and not the Tenancy module that maps both tables (`Modules_Do_Not_Read_Entitlement_Cache_Directly`). Direct HTTPS calls to Hub from anywhere except the `IEntitlementProvider` / `IUsageReporter` / `IHubTenantSync` adapter implementations; resolving a host by calling the Hub at all — `IHostToTenantResolver` reads `platform_host_to_tenant` and nothing else ([ADR-0034](../decisions/0034-hub-contract-surface-invariant.md)). | | Reference the `LearnStack.Modules.Audit` module via integration events (Audit subscribes to events from other modules). | Writing to `audit_log` directly from outside the Audit infrastructure pipeline. | diff --git a/docs/architecture/04-technical-architecture.md b/docs/architecture/04-technical-architecture.md index 14df334d..f5f37e3c 100644 --- a/docs/architecture/04-technical-architecture.md +++ b/docs/architecture/04-technical-architecture.md @@ -175,23 +175,31 @@ flowchart LR outbox[(outbox table)] dispatcher[API OutboxProcessor
BackgroundService] bus[In-process event bus] + ledger[(Shared delivery ledger)] + consumer[API DeliveryProcessor] + executor[App-role delivery executor] handlers[Integration event handlers] - externals[External calls
email / search index / etc.] trans -- domain change + integration event row --> outbox dispatcher -- claim one admitted row --> outbox dispatcher -- dispatch --> bus - bus --> handlers - bus --> externals + bus -- durable acceptance --> ledger + consumer -- claim one subscription --> ledger + consumer --> executor + executor -- inbox before construction --> handlers outbox -. mark processed .- dispatcher ``` - All side-effects beyond the database transaction are deferred to **integration events** written to a transactional `outbox` table in the same transaction as the domain change. - The API-host `OutboxProcessor` polls the outbox and publishes admitted events through `IEventBus`. P02b-2 Step 3 implements hosted execution, with - both independent review rounds complete; real required subscriptions remain Pending - until P02b-3 supplies durable consumer execution. -- Idempotency keys plus consumer-side deduplication guarantee at-least-once with effective once-only semantics. + both independent review rounds complete. P02b-3 Steps 1–5 implement durable + acceptance and independent subscriber execution with both real defaults Active. + The [packet record](../roadmap/phase-02b-events-auth.md#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) + owns current enforcement, validation and review status. +- Publication is at-least-once. The transport-owned shared inbox and business + completion commit together for ordinary database effects; external effects need + their own idempotency contract under ADR-0046. Full details: [Events & Outbox](15-event-and-outbox.md). diff --git a/docs/architecture/09-tenant-isolation.md b/docs/architecture/09-tenant-isolation.md index bc2136e4..925c04b3 100644 --- a/docs/architecture/09-tenant-isolation.md +++ b/docs/architecture/09-tenant-isolation.md @@ -162,7 +162,12 @@ set, host factory ceiling and canonical RLS placement remain unchanged under [ADR-0057](../decisions/0057-transactional-event-production-and-completion.md). P02b-1 Steps 2–3 deliver immediate outbox enqueue, successful-save collection and owner-only completion. P02b-2 Step 3 implements confined hosted dispatch; -both independent review rounds are complete. Consumer delivery remains P02b-3. +both independent review rounds are complete. P02b-3 Steps 1–5 supply durable +acceptance, shared tenant/org-protected delivery storage, app-role business scopes, +independent scheduling and both real defaults. Its +[Step 6 record](../roadmap/phase-02b-events-auth.md#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) +owns current validation and review status. The scheduler cannot read payload or +inbox columns; business execution retains the ordinary tenant/org RLS boundary. ## Default org semantics diff --git a/docs/architecture/10-cross-module-contracts.md b/docs/architecture/10-cross-module-contracts.md index 4623064a..d30f3219 100644 --- a/docs/architecture/10-cross-module-contracts.md +++ b/docs/architecture/10-cross-module-contracts.md @@ -21,6 +21,17 @@ construction; handlers do not call the guard. The four communication mechanisms and demand-gated broker trigger remain unchanged. This records the accepted target, not delivered subscriber machinery. +**2026-10-13 implementation note.** P02b-3 Steps 1–4 deliver shared storage, +durable acceptance, transport-owned inbox/business completion and independent +subscriber scheduling, terminal audit and internal replay. The +[Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-3-step-4--subscriber-scheduling-terminalization-and-replay-2026-10-13) +owns passing proofs and both completed review rounds. Production default +subscriptions activated together in Step 5 with the shared initializer/Seeder +handoff; its [record](../roadmap/phase-02b-events-auth.md#p02b-3-step-5--useful-defaults-and-semantic-seed-handoff-2026-10-13) +records that step's validation and reviews. The +[Step 6 record](../roadmap/phase-02b-events-auth.md#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) +owns current enforcement, validation and review/PR status. + ## Allowed Contract Types ```mermaid diff --git a/docs/architecture/15-event-and-outbox.md b/docs/architecture/15-event-and-outbox.md index d2d09a61..07e64bb8 100644 --- a/docs/architecture/15-event-and-outbox.md +++ b/docs/architecture/15-event-and-outbox.md @@ -26,21 +26,24 @@ not land together. |---|---|---| | `outbox_messages` table, its schema, and its LearnStack ownership | [Phase 02a Packet 6](../roadmap/phase-02a-kernel-tenancy.md) | Schema and ownership shipped in Packet 6; producer dispatch is implemented in P02b-2 Step 3 | | `IEventBus` port + `InProcessEventBus` | [Phase 02a Packet 5](../roadmap/phase-02a-kernel-tenancy.md) | The only registered implementation | -| `IOutbox` and `TenantCreatedIntegrationEventV1` | [P02b-1](../roadmap/phase-02b-events-auth.md#p02b-1-decision-package-2026-10-12) | Implemented immediate transactional production; no delivery yet | -| `OutboxProcessor` and the producer claim protocol | [P02b-2](../roadmap/phase-02b-events-auth.md#p02b-2-step-3--hosted-lifecycle-and-observability-2026-10-12) | Producer runtime and enforcement locally complete; all four steps completed both independent review rounds; CI/merge status: [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) | -| `IInboxGuard` and durable consumer transactions | [P02b-3](../roadmap/phase-02b-events-auth.md) | Pending | +| `IOutbox` and `TenantCreatedIntegrationEventV1` | [P02b-1](../roadmap/phase-02b-events-auth.md#p02b-1-decision-package-2026-10-12) | Implemented immediate transactional production; P02b-3 delivers the two independent default effects | +| `OutboxProcessor` and the producer claim protocol | [P02b-2](../roadmap/phase-02b-events-auth.md#p02b-2-step-3--hosted-lifecycle-and-observability-2026-10-12) | Producer runtime and enforcement complete and merged; all four steps completed both independent review rounds; CI/merge status: [merge closeout](../roadmap/phase-02b-events-auth.md#p02b-2-merge-and-closeout-2026-10-13) | +| Shared ledger/inbox and durable acceptance | [P02b-3](../roadmap/phase-02b-events-auth.md) | Steps 1–4 implement schema, acceptance/inbox, transactional business execution and independent scheduling/terminalization/replay; Step 5 activates both real defaults and semantic Seeder handoff | | Domain events collected and dispatched in-process, inside the ambient transaction | [P02b-1](../roadmap/phase-02b-events-auth.md#p02b-1-step-3--domain-completion-and-provisioning-publisher-2026-10-12) | Successful-save collection and owner-only completion implemented | | `DaprEventBus` → Dapr pub/sub → Kafka | [Phase 11](../roadmap/phase-11-production-hardening.md) | Demand-gated; trigger: a second process needs to consume an integration event, or event volume / replay / cross-process ordering is required ([ADR-0035](../decisions/0035-demand-gated-infrastructure.md)) | -**`InProcessEventBus` is a first-class transport.** It already supplies the handler -interface, scoped ambient tenant restoration and per-key serialisation. It does -not yet open a database transaction, announce RLS context, deduplicate deliveries -or persist retries. `IInboxGuard` is a documented seam, not a declared type. +**`InProcessEventBus` is a first-class transport.** P02b-3 Step 2 changes publish +success to durable acceptance of all required subscription envelopes on a fresh, +announced application transaction. It constructs no business handler. Step 1 +supplies the shared ledger/inbox schema; Step 2 declares `IInboxGuard` and its +transaction-bound claim primitive. Step 3 implements the business owner and +completion gate; Step 4 adds independent scheduling, terminalization, replay and +bounded observation. Step 5 activates both real defaults and semantic Seeder handoff. [Phase 02b's entry verification](../roadmap/phase-02b-events-auth.md#p02b-0-entry-verification-2026-10-12) -records that baseline. The producer/consumer diagrams and examples below describe -the durable target: P02b-1 supplies enqueue, P02b-2 dispatch and P02b-3 consumer -transactions, inbox and recovery. Accepted ADR-0046 now assigns inbox ownership -and stable subscriber identity; implementation remains P02b-3. +records the earlier immediate-invocation baseline. The producer/consumer diagrams +and examples below describe the delivered ADR-0046 runtime. The +[Step 6 record](../roadmap/phase-02b-events-auth.md#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) +owns production enforcement, validation and review/PR status. The [P02b-1 decision package](../roadmap/phase-02b-events-auth.md#p02b-1-decision-package-2026-10-12) and [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md) @@ -50,8 +53,8 @@ successful-save collection and the completed-provisioning publisher. Step 4 implements production-census guards and catalogue enforcement; its [delivery record](../roadmap/phase-02b-events-auth.md#p02b-1-step-4--producer-enforcement-and-closeout-2026-10-12) owns final verification and review status. ADR-0046 accepts dispatcher/inbox -contracts; producer dispatch is implemented through P02b-2 Step 3, while -inbox/consumer implementation remains P02b-3. +contracts; producer dispatch is implemented through P02b-2 Step 3 and shared +inbox/consumer execution through P02b-3 Steps 1–5. The [P02b-2 decision package](../roadmap/phase-02b-events-auth.md#p02b-2-decision-package-2026-10-12) and [Accepted ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md) @@ -62,9 +65,15 @@ Step 3 implements hosted execution and observation, with both independent review rounds complete in its [delivery record](../roadmap/phase-02b-events-auth.md#p02b-2-step-3--hosted-lifecycle-and-observability-2026-10-12). Step 4 enforcement and local closeout are complete; all four steps completed both -independent review rounds. The [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) +independent review rounds. The [merge closeout](../roadmap/phase-02b-events-auth.md#p02b-2-merge-and-closeout-2026-10-13) owns CI/merge status. -P02b-3 inbox/consumer delivery remains pending. Numerical +P02b-3 implements consumer execution through Step 3 and independent scheduling, +terminalization, replay and bounded observation in Step 4. Its production default +subscriptions are Active with Step 5 initialization and semantic Seeder handoff. The +[Step 5 record](../roadmap/phase-02b-events-auth.md#p02b-3-step-5--useful-defaults-and-semantic-seed-handoff-2026-10-13) +records that step's validation and reviews; the +[Step 6 record](../roadmap/phase-02b-events-auth.md#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) +owns current enforcement, validation and review/PR status. Numerical runtime bounds live in the package, canonical DDL/grants in Standards 05 and metric names in Standards 10. @@ -108,9 +117,9 @@ claims one admitted row immediately before publishing and fences its acknowledge Required Pending subscriptions hold rows before acquisition. P02b-3 durably accepts required subscriptions, then executes each on its own app-role transaction. Shared inbox completion and business/audit/ledger completion commit together. Producer -production and the API worker are implemented; both Step 3 review rounds are -complete. Durable consumer execution remains P02b-3, and real required -subscriptions remain Pending. +production and the API producer worker are implemented; both producer Step 3 +review rounds are complete. P02b-3 Steps 1–4 implement durable consumer acceptance, +execution and scheduling; Step 5 activates both real required subscriptions. ## Outbox table @@ -144,6 +153,7 @@ attributes and no privilege-escalating membership. budget: named outbox status UPDATEs and audit INSERT-only in P02b-2, with no audit read/update/delete or business access. P02b-3 may add only enumerated delivery identity/routing/lease/status SELECT and status UPDATE, never payload SELECT. +Step 4 implements that subscriber scheduler/store on the same confined source. It executes business work separately as `learnstack_app` under tenant RLS. Routine confined poll/claim/mark writes bounded structured outcomes and metrics, @@ -156,24 +166,25 @@ implemented for transition-witness construction and terminal audit in Step 2; both independent review rounds passed. Step 3 implements hosted execution and observation, with both independent review rounds complete. Step 4 implements credential/publication censuses and planted controls; all four steps completed -both independent review rounds and local closeout. The [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) +both independent review rounds and local closeout. The [merge closeout](../roadmap/phase-02b-events-auth.md#p02b-2-merge-and-closeout-2026-10-13) owns CI/merge status. ## Inbox table (shared, transport-owned) -**Accepted ADR-0046 target; implementation P02b-3.** One tenant-owned +**Implemented by P02b-3 Steps 1–3.** One tenant-owned `inbox_messages` deduplicates `(tenant_id, subscription_id, event_id)`. The transport claims before handler construction on the same announced app-role -transaction as business work. New claims roll back on failure; only successful -completion persists a completed row. Completed rows are never reset by replay. +transaction as business work. The immutable marker is inserted at claim time; +only successful owner completion makes it durable. New claims roll back on +failure. Completed rows are never reset by replay. The separate delivery ledger retains validated envelopes and owns independent -attempts, availability, leases and terminal/replay state. P02b-3 chooses envelope -copies/references and retention dependencies before its canonical DDL, then -P02b-8 inventories all retained stores for erasure. DDL, class/RLS and exact grants -belong in [Database Standards § Inbox](../standards/05-database.md#inbox); this +attempts, availability, leases and terminal/replay state. `event_deliveries` holds +one validated envelope copy per tenant/subscription/event, independent of the +producer row. P02b-8 inventories these retained copies for erasure. DDL, class/RLS +and exact grants belong in [Database Standards § Inbox](../standards/05-database.md#inbox); this document carries no second schema. ## Producer pattern @@ -261,8 +272,10 @@ Step 7's `OutboxFlushBehavior` is an enrollment/invariant guard. The reusable audit coordinator gives a controlled non-MediatR owner the existing classification, MUST-write, boundary-reporting and outer reconciliation duties. -P02b-3 owns delivery/inbox wiring under ADR-0046; the P02b-1 fixture proves -that entry without claiming delivered subscribers. +P02b-3 Step 3 adds a private, single-use prepared-owner token: routing announcement +and retained-envelope validation precede module resolution, then the same owner +body completes work without a second BEGIN or a joiner masquerading as owner. +The P02b-1 fixture remains the earlier controlled direct-entry proof. ### First useful flow @@ -290,14 +303,17 @@ everything. ## Consumer pattern -**Accepted target, unimplemented until P02b-3.** Each subscription has an explicit +**Owner implemented by P02b-3 Step 3; scheduler implemented by Step 4.** +Step 5 activates both real defaults and semantic Seeder handoff. +Each subscription has an explicit stable `{module}.{subscription}.v{N}` ID, independent of handler class/assembly. ADR-0046 owns its closed grammar and version/migration rules. The transport opens an app-role transaction, restores validated tenant and organization, announces RLS context, and claims the shared inbox before constructing the handler. Absence of the organization marker never erases a present envelope organization; a marked event requires one. Effective actor is `UserId.SystemActor`; human actor remains -causal metadata. +causal metadata. Business audit metadata retains `causation_id` and optional +`causal_actor_user_id` through the same composer for success and reconciliation. The common transaction/audit owner classifies the handler and refuses undeclared mutating work. A handler changes only its own root and saves; it calls neither @@ -307,11 +323,39 @@ completion checks the current token and database-time expiry, plus a sticky gate that refuses timed-out/canceled execution. Zero rows or a closed gate poison and roll back the whole transaction. Cancellation after physical COMMIT starts is Indeterminate; recover from atomic durable state, not a rollback assumption. +The final fence locks the exact row first and checks database time in a subsequent +completion UPDATE, so an unchanged lock wait cannot lend an expired lease authority. +The attempt's one-use gate publishes acknowledged COMMIT independently of later +frame/scope cleanup; a cleanup fault cannot turn committed business into retryable +work. Live cleanup retains its task and resources until actual completion. External effects require provider idempotency and reconciliation. Privileged Audit erasure belongs to its separate Phase 03 execution contract; P3's ordinary executor refuses it. An app-role inbox cannot make two connections atomic. +### Independent subscriber scheduling + +P02b-3 Step 4's API-owned `DeliveryProcessor` shares the confined dispatcher data +source. `DeliveryDispatchStore` selects scheduling/routing/status columns only; +the app-role executor reads the retained envelope under RLS. Exact Pending or +privileged type/subscription pairs are held without blocking an accepted Active +sibling. Terminal requests and expired claims take priority over new execution. +One normal claim increments only that tuple's budget; recovery replaces expired +ownership without execution or another attempt. Six claims use the accepted five +backoffs; the sixth cannot authorize a seventh invocation. Every status transition +locks the exact tuple, then checks current token and fresh database-time expiry. + +The controlled execution task owns construction, cancellation callbacks and actual +scope cleanup. Timeout closes its sticky commit gate before asynchronous callbacks. +An acknowledged COMMIT remains definitive through faulty or overlong cleanup; +unacknowledged commit admission requires durable reconciliation. Retained tasks +block their local execution partition and consume the accepted four-task capacity +until actual completion. Acceptance and execution use separate serializer lanes; +a held subscriber never queues new durable acceptance behind its callback/cleanup. +Shutdown uses one finite wait shared with the host token. +Numerical bounds and validation/review evidence belong to the +[accepted package and Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-3-step-4--subscriber-scheduling-terminalization-and-replay-2026-10-13). + ### Audit rides this outbox for fan-out, never for the audit row An earlier version of the example above built an `AuditEntry` from the event and saved it @@ -358,8 +402,8 @@ concurrent live owners; crash recovery and lease expiry can still redeliver. ### The claim protocol -**ADR-0046 producer runtime and enforcement locally complete in P02b-2; all four -steps completed both independent review rounds.** The [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) +**ADR-0046 producer runtime and enforcement complete and merged in P02b-2; all four +steps completed both independent review rounds.** The [merge closeout](../roadmap/phase-02b-events-auth.md#p02b-2-merge-and-closeout-2026-10-13) owns CI/merge status. The processor admits a construction-free manifest policy before acquisition. Any required Pending subscription holds the contract and all aliases unchanged, even if stored payload is malformed. Active @@ -413,10 +457,12 @@ across publication. A deployment-specific alternative requires a new decision. - Deterministic selection for one processor without failure, with no enqueue, commit, retry or cross-process order promise. - Ordinary failure and bounded overruns permit unrelated eligible work to progress. -- Pending real subscriptions retain useful facts until P02b-3 activates durable - acceptance. No placeholder consumer or global worker-disable switch is used. +- Required Pending subscriptions retain useful facts before activation. Both + real default subscriptions are now Active with P02b-3 durable acceptance; + no placeholder consumer or global worker-disable switch is used. - Terminal state commits with MUST audit; it does not prove that a timed-out - transport never accepted the event. P02b-3's inbox is still required. + transport never accepted the event. P02b-3's shared inbox provides independent + per-subscription deduplication during at-least-once recovery. ## `IEventBus` and `IntegrationEventEnvelope` @@ -515,19 +561,21 @@ Topic naming convention: `learnstack.{module}.{aggregate}`. Examples: ## `InProcessEventBus` `InProcessEventBus` remains the only registered transport in every deployment -mode until ADR-0035's Phase 11 trigger fires. The shipped baseline constructs one -async scope per discovered subscription, restores ambient tenant/organization, -serializes invocation per key and collects constructor/handler/disposal failures. -It opens no business transaction and has no durable inbox or subscriber ledger. -Its current exception logs/spans need P02b-3's bounded diagnostic reconciliation -before Active production consumers; P02b-2 minimizes its own dispatcher path. - -P02b-3 changes publish success to durable acceptance of every required delivery. -Duplicate acceptance is idempotent; partial or unknown acceptance can be retried. -The independent scheduler then invokes one internal delivery executor for each -subscription, with the shared inbox and app-role owner described above. Subscriber -business failure spends neither sibling nor producer attempts after acceptance. -The executor never rebroadcasts and receives no publication-rule exception. +mode until ADR-0035's Phase 11 trigger fires. P02b-3 Step 2 restores the validated +publication context before creating one fresh acceptance scope, announces its +app-role transaction and persists all required Active subscriptions atomically. +It never resolves a business handler. Identical canonical envelopes are idempotent; +conflicting or inaccessible copies refuse without changing earlier rows. Matching +partial acceptance can be completed, and uncertain COMMIT remains producer-retryable. +Pending, unknown and retired contracts cannot silently acknowledge zero deliveries; +only explicitly declared intentional absence succeeds without ledger writes. + +Per-key serialization covers publication acceptance inside this process. Handler +construction, execution, cancellation and cleanup no longer belong to publish. +Step 3 supplies the independent internal executor and app-role owner described +above; Step 4 implements its scheduler. Subscriber business failure spends +neither sibling nor producer attempts after acceptance. The executor never +rebroadcasts and receives no publication-rule exception. `PartitionSerializer` refuses reentrant work for its current key. Queuing behind itself would deadlock; running inline would let inherited AsyncLocal state permit @@ -571,7 +619,7 @@ P02b-1 accepts partition-source derivation under ADR-0057. It accepts no durable publish-order guarantee. ADR-0046 selects eligible rows in total order `(available_after, occurred_at, id)` for one processor without failure. This is no enqueue/commit/retry/cross-process promise; strict head-of-line blocking is rejected. -`InProcessEventBus` serializes invocation per key within its process. +`InProcessEventBus` serializes acceptance per key within its process. The rule, therefore, is that **every outbox row carries a non-null `partition_key`**: @@ -629,16 +677,22 @@ Phase 03 must decide producer reset/fencing before any operator replay endpoint. ### Subscriber side — dispatch succeeded, handling did not -**P02b-3 target.** Each stable subscription has its own durable ledger budget, -retained envelope and terminal/replay state. Terminal failure leaves no completed -inbox row and writes Tenancy-owned `platform.delivery.dead_letter` atomically with -terminal state. No `learnstack.dlq.*` topic or duplicate dead-letter table is needed. - -Targeted internal replay resets only that subscription with fresh ownership fencing -and invokes its executor, never `PublishAsync`. Completed siblings remain inbox -deduplicated; a completed inbox is never reset. Phase 03 owns operator permission, -HTTP routes and the separately decided producer replay contract. P02b-8 owns erasure -inventory, Phase 03 privileged redaction and Phase 11 retention purge. +**Implemented by P02b-3 Step 4.** Each stable subscription has its own durable +ledger budget, retained envelope and terminal/replay state. Terminal failure leaves +no completed inbox row and writes Tenancy-owned `platform.delivery.dead_letter` +atomically with terminal state. No `learnstack.dlq.*` topic or duplicate dead-letter +table is needed. + +Targeted internal replay admits exactly one declared tenant/subscription/event +tuple whose Active ordinary subscription has completed terminal state and no lease. +It refuses pending terminalization, completed business deliveries and Pending, +privileged, unknown or retired subscriptions. The reset clears attempts, bounded +error and terminal state, invalidates old ownership and makes that tuple due. The +scheduler later invokes only its executor, never `PublishAsync`. Completed siblings +remain inbox deduplicated; a completed inbox is never reset. Phase 03 owns operator +permission, HTTP routes and the separately decided producer replay contract. +P02b-8 owns erasure inventory, Phase 03 privileged redaction and Phase 11 +retention purge. ## Service extraction readiness @@ -663,8 +717,9 @@ separate service later: - `Integration_Events_Inherit_From_IntegrationEventBase` — every type implementing `IIntegrationEvent` extends `IntegrationEventBase` (which carries `EventId`, `OccurredAt`, `TenantId` and declares `Topic` / `PartitionKey`). -- `Integration_Event_Handlers_Use_InboxGuard` — Registered for P02b-3's shared - transport-owned claim before construction and atomic fenced completion. +- [`Integration_Event_Handlers_Use_InboxGuard`](../standards/21-architecture-tests-catalogue.md#integration_event_handlers_use_inboxguard) + — Implemented by P02b-3: nonempty all-layer handler/manifest and typed call-path + controls plus real transport-owned claim-before-construction and atomicity proofs. - `Dapr_PubSub_TopicNames_FollowConvention` — Phase 11 checks that Dapr component bindings agree with the already-validated topics declared by events. It is not a current `[Topic]`-attribute scan; no such attribute exists. @@ -690,12 +745,15 @@ attempt duration/failure/terminal/lease-loss observations. Gauges perform no SQL refresh failure preserves stale data and makes health unhealthy. Per-attempt spans and closed diagnostics contain no raw exception/type/payload/connection data. -P02b-3 adds dedup/subscriber-terminal counters with bounded module/event-type labels. -Producer and subscriber terminal outcomes remain separate observations: acceptance -failure and business execution failure have different retry owners. Metrics are -process observations, not an exactly-once durable event ledger. `outbox` is -registered and not publicly mapped; `/healthz` remains liveness-only, with Phase 11 -owning readiness exposure and dashboards. +P02b-3 Step 4 emits nine subscriber instruments: four snapshot/heartbeat gauges, +execution duration, failure, terminal, dedup and lease-loss observations. Event-type +and subscription labels use finite declared values and fixed invalid categories; +queue/heartbeat gauges and lease loss are unlabelled. Failed refresh retains the +prior snapshot; stale progress/snapshot, retained overruns and terminal-audit +failure make delivery health unhealthy. Producer and subscriber outcomes remain +separate process observations, not an exactly-once durable event ledger. `outbox` +and `delivery` are registered and not publicly mapped; `/healthz` remains +liveness-only, with Phase 11 owning readiness exposure and dashboards. ## References diff --git a/docs/architecture/29-dapr-integration.md b/docs/architecture/29-dapr-integration.md index 0857b48c..ae14fa06 100644 --- a/docs/architecture/29-dapr-integration.md +++ b/docs/architecture/29-dapr-integration.md @@ -206,18 +206,19 @@ only application code permitted to know Dapr types. ### Current default: `InProcessEventBus` -Every deployment mode currently registers `InProcessEventBus : IEventBus`, even when a -developer starts the gated sidecar for inspection. It is a **transport, not a stub**: it reads -construction-free subscription metadata, gives each subscription one consumer activity -and one async DI scope, restores tenant and module context before resolving exactly one -concrete `IIntegrationEventHandler`, and restores the publisher's context after the -full dispatch. It serializes invocation per partition key within its process; -it opens no business transaction and implements no durable inbox or subscriber ledger. -[ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md) assigns shared -`IInboxGuard` claim/completion to the transport before handler construction, with -implementation in P02b-3. Handlers do not call the guard. Phase 11's Dapr adapter -preserves that accepted contract. See [15-event-and-outbox.md](15-event-and-outbox.md) -for the shipped baseline and the P02b-3 durable acceptance/execution target. +Every deployment mode registers `InProcessEventBus : IEventBus`, even when a +sidecar is started for inspection. P02b-3 Step 2 makes publication durable +acceptance of all required subscription envelopes on a fresh app-role transaction; +it constructs no business handler. Steps 3–4 add the independent application +executor, transport-owned inbox and bounded subscriber scheduler. The executor +restores validated tenant/org and causal context before resolving a module or +handler, then restores its caller's context after actual cleanup. Handlers do not +call `IInboxGuard` or rebroadcast. Step 5 activates both real default subscriptions +and supplies their initialization and semantic Seeder handoff. + +Phase 11's Dapr adapter must preserve these accepted contracts. See +[Events and Outbox](15-event-and-outbox.md#consumer-pattern) for the lifecycle, +independent retry ownership and current delivery evidence. ## 4. Cache implementation (DaprCacheService) diff --git a/docs/architecture/31-audit-subsystem.md b/docs/architecture/31-audit-subsystem.md index 5deb611f..e1d63294 100644 --- a/docs/architecture/31-audit-subsystem.md +++ b/docs/architecture/31-audit-subsystem.md @@ -102,10 +102,13 @@ and remains the sole audit SQL writer. Mismatch rolls back both state and audit. GUCs cannot bind a bypass INSERT; real tenant/org/event/transaction mismatch proofs and a matching organization-scoped positive control are required. -P02b-2 supplies `platform.outbox.dead_letter`; P02b-3 supplies reserved -`platform.delivery.dead_letter`, both off-path under Tenancy. Register each only -when its writer ships. Outcome success describes terminal state, not publication; -invalid causal fields are omitted in favor of a fresh validated attempt trace. +P02b-2 supplies `platform.outbox.dead_letter`; P02b-3 Step 4 implements and +registers `platform.delivery.dead_letter`, both off-path under Tenancy. The +subscriber witness binds the actual tenant/subscription/event tuple and uses a +validated attempt trace without causal-envelope or payload SELECT. Outcome success +describes terminal state, not publication. Producer terminal audit omits invalid +causal fields in favor of a fresh validated attempt trace. +Absent optional causal IDs are valid and do not alone invalidate the stored trace. Closed terminal values use bounded metadata; `Reason` stays for platform entry. Audit failure retains terminal work for retry without delivery; uncertain COMMIT is reconciled before retry and cannot duplicate a completed terminal audit. @@ -114,8 +117,8 @@ The existing outbox role gains audit INSERT-only in the Audit chain, with exact grants owned by [Database Standards](../standards/05-database.md#grant-matrix). Routine confined poll/claim/mark uses the accepted bounded audit exemption; platform-admin pre-entry audit and other bypass operations are unchanged. -P02b-3 adds inbox/delivery machinery to capture exclusions with reasons, because -explicit intents audit business effects; these exclusions remain pending P3. +P02b-3 Step 3 adds inbox/delivery machinery to capture exclusions, because explicit +intents audit business effects; retry/completion bookkeeping is not another root. ## 2. Pipeline overview @@ -171,7 +174,9 @@ Five components, separated concerns: 4. **`AuditExecutionCoordinator`** — classifies, attributes intents and outcomes, catches/rethrows unexpected failures and reconciles in `finally` at the outermost capture frame. `AuditLogBehavior` remains its step-3 request adapter. - The same entry serves direct owners; production delivery remains P02b-3. + The same entry serves direct owners and P02b-3 Step 3's delivery executor; + Step 5 activates both real defaults with Off-classified coordinators and nested + MUST-classified initializer commands. 5. **`PlatformAdminScope`** — the one writer outside the pipeline. `EnterAsync` records `platform.admin_scope.enter` through `IAuditStore.WritePlatformScopeAsync`, on the scope's own platform-role connection and transaction, **before** the operation runs @@ -218,11 +223,13 @@ public sealed class AuditChangeTrackerInterceptor : ISaveChangesInterceptor // matrix singles out as mattering most. // // Excluded BY NAME, not by type: LearnStack.Infrastructure.Audit may not reference a - // module assembly (CoreInfrastructure_DoesNotDependOn_AnyModule), and two of the four - // names are module types. + // module assembly (CoreInfrastructure_DoesNotDependOn_AnyModule); AuditEntry and + // AuditConfig are module types. private static readonly HashSet Excluded = [ "OutboxMessage", // its payload IS the audited event; the row is machinery + "InboxMessage", // transport completion identity, not a business root + "EventDelivery", // retained envelope and retry/completion machinery "IdempotencyKey", // request plumbing "AuditEntry", // the audit tables themselves — unreachable anyway, since the "AuditConfig", // store writes parameterised SQL and never a DbContext @@ -672,8 +679,10 @@ declares its own intents; nested commands retain their own, without an extra business row. Step 1 supplies the common entry and producer ancestry groundwork; Steps 2–3 deliver immediate outbox enqueue, successful-save collection/promotion and owner-only domain drain. P02b-2 Step 3 implements hosted producer dispatch, -with both independent review rounds complete; production consumer delivery/inbox wiring -remains P02b-3 under Accepted ADR-0046. +with both independent review rounds complete. P02b-3 Step 3 implements the +app-role delivery owner and per-subscription audit frames; Step 4 implements +independent scheduling and terminal audit. Step 5 activates both real defaults +and semantic Seeder handoff under Accepted ADR-0046. ### The two catalogue faces, and the three columns nothing fills yet @@ -885,7 +894,9 @@ execution boundary creates a W3C Activity even without a telemetry listener and restores ancestry/Activity on disposal. P02b-1 Steps 2–3 deliver immediate outbox enqueue, successful-save collection/promotion and owner-only domain drain before MUST audit/commit. P02b-2 Step 3 implements hosted producer dispatch, with -both independent review rounds complete; consumer delivery/inbox remains P02b-3. +both independent review rounds complete. P02b-3 Steps 1–4 implement shared +storage, acceptance, app-role business ownership and independent scheduling; +Step 5 activates both real defaults and semantic Seeder handoff. `IUnitOfWork` remains the seam through which every module context and `IAuditStore` reach the one ambient connection/transaction. Packet 6 shipped that seam and the @@ -1631,7 +1642,7 @@ update it constrains. [ADR-0036](../decisions/0036-tenant-resolution-trusted-inputs.md)'s `tenancy.tenant_assertion.reject` and `tenancy.tenant_assertion.anonymous_burst`, and Audit's `audit.redaction.apply` and `audit.purge.apply`. They register by slug: the - four whose writer ships through `DeclareOffPath`, the other five — also + five whose writer ships through `DeclareOffPath`, the other four — also `(planned)` — when their writer lands. - *An `Off` registration carries no slug*, so it raises no matrix row and is outside both directions too (§ 5). @@ -1718,9 +1729,9 @@ for MUST-class would lock in exactly the defect ADR-0033 removes. | Phase | Deliverable | |-------|-------------| | [02a Packet 9](../roadmap/phase-02a-kernel-tenancy.md) | `AuditChangeTrackerInterceptor` with the broad capture predicate, the `[PiiSensitive]` redaction gate and the size cap; `IAuditStateCapture` + impl, holding an **ordered list** of intents; `AuditLogBehavior` lit up per ADR-0033 as amended; `IAuditStore` + `PostgresAuditStore` with **four** write methods, the fourth serving `EnterPlatformAdminScope`; the classification catalogue as a **pair** — the module-owned `IAuditCatalogSource` / `IAuditCatalogBuilder` and the `IAuditCatalog` the composition root merges them into once at startup — with the six value types beside the five ports in `LearnStack.SharedKernel.Audit`; `TenantId.PlatformSentinel` and the **four** guards that enforce its invariant — `SetProvisioningTenantContextAsync`, `TenantOwnership.EnsureRealTenant`, `EventTenantContext.FromEnvelope` and, deliberately redundant at the one site every announcement passes, `SetTenantContextAsync`; `AuditEntry` aggregate and `AuditConfig` with the fail-closed MUST floor; `audit_log` as a **single plain table** with the composite primary key, org-scoped, in a fourth migration chain, with its append-only grants and its two triggers; and the named `audit` health check §§ 1, 5 and 7 read, registered through `AddHealthChecks()` at the `LearnStack.Api` composition root. | -| P02b-1 Steps 1–3 | Common audit/transaction coordinators, capture-owned audit frames, producer ancestry/receipts and no-listener trace groundwork; Steps 2–3 deliver immediate outbox enqueue, successful-save collection/promotion and owner-only domain drain. Hosted dispatch is implemented in P02b-2 Step 3; consumer delivery/inbox remains P02b-3. | -| P02b-2 Steps 1–4 | Schema/grants and confined producer transitions with MUST audit through the row-bound fifth seam; real-role atomicity/identity proofs. Steps 1–2 completed both review rounds. Step 3 implements hosted dispatch and observation, with both independent review rounds complete; Step 4 enforcement and local closeout are complete; all four steps completed both independent review rounds. The [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) owns CI/merge status. | -| P02b-3 (Accepted contract; pending) | Shared transport-owned inbox and independent delivery ledger; per-subscription audit frames and Tenancy-owned terminal audit. Ordinary executor refuses privileged subscriptions. | +| P02b-1 Steps 1–3 | Common audit/transaction coordinators, capture-owned audit frames, producer ancestry/receipts and no-listener trace groundwork; Steps 2–3 deliver immediate outbox enqueue, successful-save collection/promotion and owner-only domain drain. Hosted dispatch is implemented in P02b-2 Step 3; P02b-3 implements consumer delivery/inbox. | +| P02b-2 Steps 1–4 | Schema/grants and confined producer transitions with MUST audit through the row-bound fifth seam; real-role atomicity/identity proofs. Steps 1–2 completed both review rounds. Step 3 implements hosted dispatch and observation, with both independent review rounds complete; Step 4 enforcement and local closeout are complete; all four steps completed both independent review rounds. The [merge closeout](../roadmap/phase-02b-events-auth.md#p02b-2-merge-and-closeout-2026-10-13) owns CI/merge status. | +| P02b-3 Steps 1–6 | Shared transport-owned inbox and retained delivery envelopes, durable acceptance, per-subscription audit frames and fenced business completion; independent scheduling/replay and Tenancy-owned subscriber terminal audit through the row-bound fifth seam. Ordinary executor refuses privileged subscriptions. Both real default subscriptions are Active with nested initializer MUST audits and semantic Seeder handoff; Step 6 extends classification to each concrete integration handler, with actual API/Seeder catalogue and missing-Off controls. The [Step 6 record](../roadmap/phase-02b-events-auth.md#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) owns validation/review/PR status. | | [03](../roadmap/phase-03-identity-admin.md) | Admin API endpoints over the audit stream; `UserGdprDeletedIntegrationEventHandler` + per-module `IUserReferenceLocator`. | | [06](../roadmap/phase-06-renderer-admin-studio.md) | Admin Studio audit UI: timeline view, filters, diff viewer, CSV / JSON export. | | [09](../roadmap/phase-09-billing-integrations-analytics.md) | Hub-side `hub_audit_log` + cross-stream correlation query. | diff --git a/docs/architecture/32-tenant-customization-model.md b/docs/architecture/32-tenant-customization-model.md index 23faa835..7b33ff89 100644 --- a/docs/architecture/32-tenant-customization-model.md +++ b/docs/architecture/32-tenant-customization-model.md @@ -13,6 +13,14 @@ tenant's database**. This document is the conceptual deep dive for ADR-0018: what gets stored, how renderers compose primitives, how customization is versioned, and what cannot be customized. +P02b-3 Step 5 activates the two create-only provisioning defaults and verifies +built-in seed rows by semantic identity. The +[Customization module spec](../modules/customization/README.md#integration-event-catalogue) +owns the shipped initializer/registration serialization and preservation contract; +the [packet record](../roadmap/phase-02b-events-auth.md#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) +owns current verification and review status. Earlier seed receipts remain as-of +history; authored definitions and content pins retain exact identity checks. + ## 1. Customization surfaces Eight tenant-database tables drive customization: diff --git a/docs/decisions/0023-strongly-typed-id-source-generator.md b/docs/decisions/0023-strongly-typed-id-source-generator.md index 06782c90..51cd06da 100644 --- a/docs/decisions/0023-strongly-typed-id-source-generator.md +++ b/docs/decisions/0023-strongly-typed-id-source-generator.md @@ -260,6 +260,13 @@ Three things outweighed the appeal: > surrogate column. See Amendment 4. Current authority: [Database Standards § > Idempotency](../standards/05-database.md). +> **Erratum — 2026-10-13.** The DB-side list below includes `inbox_messages`. +> It reuses the producer's EventId; it does not mint a second UUID. At acceptance +> commit `da1d37e3`, Architecture 15's Inbox fence already declares `event_id uuid +> PRIMARY KEY` without a default, and its consumer supplies `@event.EventId`. +> The UUIDv7 Decision is unchanged. Current authority: [Database Standards § +> Inbox](../standards/05-database.md#inbox). Recorded in Amendment 10. + - **UUIDv7 minting:** - DB-side (`gen_uuid_v7()` per ADR-0031) for `audit_log`, `outbox_messages`, `inbox_messages`, `idempotency_keys` — high-volume append-only tables. @@ -549,6 +556,25 @@ commit-in-doubt event. `DEFAULT uuidv7()` stays on the column as a backstop for inserted by something other than `PostgresAuditStore`; every row the store writes carries an id the store supplied. +### Amendment 10 — Inbox identity reuses the producer EventId (2026-10-13) + +The Implementation Notes DB-side minting list named `inbox_messages`, although +its deduplication identity is the incoming producer EventId. This was false when +the list entered the record: +`git show da1d37e3:docs/architecture/15-event-and-outbox.md` shows an +`event_id uuid PRIMARY KEY` inbox with no default and a consumer passing +`@event.EventId`. Under ADR-0041, an inline erratum precedes the original list; +the body, Decision, acceptance metadata and earlier amendments remain intact. + +[ADR-0046](0046-durable-event-delivery-and-recovery.md#subscriber-acceptance-and-inbox-ownership) +adds tenant and stable subscription to that natural identity; it still mints no +surrogate. P02b-3 implements the shared table under its +[accepted package](../roadmap/phase-02b-events-auth.md#p02b-3-decision-package-2026-10-13). +Carriers changed for this correction are this erratum/amendment, the ADR index, +Database Standards' Inbox/Identifiers text and the phase package. Database +Standards already excludes inbox from DB-side minting; its canonical DDL lands +with Step 1. This note changes no UUIDv7 source decision. + ## References - [Standards 02 § Strongly-Typed Identifiers](../standards/02-backend-coding.md) diff --git a/docs/decisions/0038-cross-cutting-port-and-event-contracts.md b/docs/decisions/0038-cross-cutting-port-and-event-contracts.md index 3122c8d7..8a84b606 100644 --- a/docs/decisions/0038-cross-cutting-port-and-event-contracts.md +++ b/docs/decisions/0038-cross-cutting-port-and-event-contracts.md @@ -247,6 +247,22 @@ This is a bounded new decision, not an ADR-0041 correction. Original acceptance metadata, body and earlier amendments remain intact. Acceptance claims no durable acceptance or consumer delivery. +### Amendment 4 — Durable acceptance and subscriber delivery implemented (2026-10-13) + +**Delivery note; no decision change.** P02b-3 implements Amendment 3's accepted +publish-outcome replacement. `InProcessEventBus` commits all required envelope +copies without constructing business handlers; unknown acceptance remains a +producer retry. `DeliveryExecutor` restores tenant/org, effective SystemActor +and separate human causation inside a fresh app-role scope, claims inbox before +construction and uses the common audit/domain-completion owner. Independent +subscriber scheduling owns retry, terminalization and targeted internal replay. +Transport diagnostics use closed reasons without logging raw handler exceptions +or copied envelope data. The two real default subscriptions are Active. + +The [Step 6 record](../roadmap/phase-02b-events-auth.md#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) +owns runtime proofs, bounded production censuses and current review/PR status. +Original acceptance metadata, body and previous amendments are unchanged. + ## References - [ADR-0006 — Events and Outbox](0006-events-and-outbox.md) diff --git a/docs/decisions/0044-audit-write-path.md b/docs/decisions/0044-audit-write-path.md index bda44db1..90827f0f 100644 --- a/docs/decisions/0044-audit-write-path.md +++ b/docs/decisions/0044-audit-write-path.md @@ -1153,6 +1153,46 @@ This additive seam is separate from Amendment 9's grant replacement and does not change ADR-0033's durability contract. Original acceptance metadata, body and prior amendments remain intact. Acceptance authorizes implementation, not delivery. +## Amendment 11 — Shared delivery capture exclusions (2026-10-13) + +**Status: Accepted.** [ADR-0046](0046-durable-event-delivery-and-recovery.md#bounded-supersession-and-additive-seams) +and the [P02b-3 package](../roadmap/phase-02b-events-auth.md#p02b-3-decision-package-2026-10-13) +add `InboxMessage` and `EventDelivery` to § 7's named capture exclusions. The inbox +is transport-owned completion identity; the delivery ledger retains envelopes +and retry/terminal machinery. Neither is a second business aggregate. Explicit +handler/nested-command intents continue to capture and audit all business effects. +The runtime exclusion list and its planted controls land with P02b-3; acceptance +does not claim those tests or subscriber delivery already exist. + +Amendment 10's fifth seam already authorizes the subscriber terminal writer and +`platform.delivery.dead_letter`. P02b-3 binds that operation to an actual-row +subscriber transition witness on the same transaction; register the off-path +MUST/SecurityEvent operation only with its writer. No new audit method, standalone +terminal writer, grant beyond ADR-0046 or changed durability rule is introduced. +The original body, Decision, acceptance metadata and earlier amendments survive. +This is the additive clarification ADR-0046 requires, not a factual erratum or +a normative supersession. + +## Amendment 12 — Subscriber audit delivery note (2026-10-13) + +**Delivery note.** P02b-3 Steps 3–4 implement Amendments 10–11's remaining +subscriber duties. `InboxMessage` and `EventDelivery` are excluded from entity +capture with planted controls; explicit business intents remain audited. +The confined subscriber terminal writer registers `platform.delivery.dead_letter` +as off-path MUST/SecurityEvent. Its actual-row witness and the fifth seam bind +stored tenant/org/subscription/event identity and the live transaction; terminal +state and audit commit together. The scheduler receives no payload or causal +SELECT grant. Real-role identity substitutions, audit refusal and lost-COMMIT +controls are recorded with the +[Step 4 implementation](../roadmap/phase-02b-events-auth.md#p02b-3-step-4--subscriber-scheduling-terminalization-and-replay-2026-10-13). +The catalogue names the +[row-bound terminal proof](../standards/21-architecture-tests-catalogue.md#delivery_terminal_audit_identity_is_row_bound). + +This records delivery of the accepted additive seam and exclusions. It changes +no earlier Decision, acceptance metadata, grant or durability rule. Production +default subscriptions and Seeder handoff remain Step 5; packet review/PR closeout +remain Step 6. + ## References - [ADR-0033 Audit Durability Model](0033-audit-durability-model.md) — the durability diff --git a/docs/decisions/0046-durable-event-delivery-and-recovery.md b/docs/decisions/0046-durable-event-delivery-and-recovery.md index 5a9ed9d2..92f8421f 100644 --- a/docs/decisions/0046-durable-event-delivery-and-recovery.md +++ b/docs/decisions/0046-durable-event-delivery-and-recovery.md @@ -487,6 +487,28 @@ independent subscriber retries/terminalization, both real default subscriptions and Seeder handoff remain its obligations. Required production subscriptions remain Pending; test-owned Active contracts prove producer runtime only. +### Amendment 3 — Consumer runtime and enforcement delivery (2026-10-13) + +**Delivery note; no decision change.** P02b-3 implements shared envelope retention +and immutable inbox storage under the accepted grants, durable all-subscription +acceptance, independent six-claim subscriber retries/recovery, the app-role +business owner and timeout completion gate, row-bound terminal MUST audit and +internal targeted replay. Controlled execution retains actual task/scope cleanup; +publication acceptance and business execution use distinct local lanes. Both +real Customization subscriptions are Active and preserve existing concept +history. Seeder shares the initializer commands and verifies semantic identity. + +Production enforcement includes all module layers, exact handler/manifest/DI +bindings, typed invocation and transaction-completion sites, concrete-handler +audit classification and the one-root constructor-port rule. Planted missing, +dirty and clean controls exercise these bounded checks; real PostgreSQL tests +own ordering, isolation and atomicity. They do not claim arbitrary reflection or +SQL transaction dataflow analysis. The +[Step 6 record](../roadmap/phase-02b-events-auth.md#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) +owns validation, independent reviews and PR/CI status. Earlier delivery notes +retain their dated state. No exactly-once publication, cross-process ordering, +operator replay route, purge, broker adapter or mapped readiness is delivered. + ## References - [ADR-0006](0006-events-and-outbox.md), [ADR-0010](0010-cross-module-communication.md) diff --git a/docs/decisions/0057-transactional-event-production-and-completion.md b/docs/decisions/0057-transactional-event-production-and-completion.md index 4baae490..46168e68 100644 --- a/docs/decisions/0057-transactional-event-production-and-completion.md +++ b/docs/decisions/0057-transactional-event-production-and-completion.md @@ -416,6 +416,25 @@ These are all carriers changed for this locator correction. The Decision and the exact bounded supersession are unchanged: the two historical separate-column requirements yield to `outbox_messages.id = EventId`; all other clauses survive. +### Amendment 3 — Useful default consumers and semantic seed handoff (2026-10-13) + +**Delivery note; no decision change.** P02b-3 Step 5 activates the two stable +subscriptions together: the content-type initializer creates `card@1`, and the +level-taxonomy initializer creates `plain@1`. Each creates one untouched Active +root with an owner-minted UUIDv7, one save, generation bump and MUST audit. Any +existing concept history is preserved under the same transaction-owned semantic +lock used by ordinary registration. Subscriber coordinator classifications are +Off; the nested initializer commands own the resource audits. + +Seeder uses those same commands and verifies built-ins by semantic identity and +exact Active shape. Warm/generated IDs survive; authored identities and exact +pins remain strict. App-role tests cover backlog and fresh provisioning, +independent inbox effects, audit rollback, duplicate acceptance and both winning +orders of Seeder/subscriber races. The +[Step 5 record](../roadmap/phase-02b-events-auth.md#p02b-3-step-5--useful-defaults-and-semantic-seed-handoff-2026-10-13) +owns validation and review status; Step 6 retains production-census enforcement +and full packet closeout. Earlier dated delivery notes retain their as-of state. + ## References - [ADR-0006: Events and Outbox](0006-events-and-outbox.md) diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 1f2a03f5..d8573093 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -54,7 +54,7 @@ name P02b-2/P02b-3 activation; acceptance claims no runtime delivery. | 0020 | [Triple Deployment + Hybrid License](0020-triple-deployment-hybrid-license.md) | SaaS / Dedicated / Self-Hosted from one codebase; phone-home + RSA-signed key + 30-day grace (Amendment 2026-09-07: the mode switch describes the **end state** — `NullEntitlementProvider` is the registered implementation in every mode until that mode's adapter exists, per ADR-0035's default-implementation gate) | | 0021 | [Feature-Based Entitlement](0021-feature-based-entitlement.md) | Feature flags + numeric limits per plan; typed `FeatureKeys` / `LimitKeys` registries (Amendment 2026-09-07: **`-1` unlimited, `0` denied** is normative — the inverse reading made the degraded read-only mode grant unlimited; `NullEntitlementProvider` returns `-1` not `null`; `IFeatureFlagService` / `IUsageLimitService` are withdrawn, leaving `IFeatureFlags` as the one read interface. Amendment 2026-05-18: `FeatureKey` / `LimitKey` value objects. Amendment 2026-09-08: `LimitKeys` takes the Hub's `limits.` vocabulary and `tenancy.max_learners` and its siblings are withdrawn. Amendment 2026-09-11: a registry declares every key the contract names, and enforcement, not membership, is what waits for a consumer) | | 0022 | [Custom Domain & TLS](0022-custom-domain-tls.md) | Hub-owned custom domain admin; DNS-01 + HTTP-01 + Let's Encrypt; APISIX hot-reload | -| 0023 | [Strongly-Typed ID Source Generator — Vogen](0023-strongly-typed-id-source-generator.md) | Vogen as the source generator for both IDs and value objects; `[ValueObject]` annotation; EF + JSON + ASP.NET + OpenAPI emitters out of the box (Amendment 1: Vogen 7.0.0 pin + architecture-test placement, 2026-05-21; Amendment 2: `UserId` / `TenantId` / `OrganizationId` are cross-cutting and live in `LearnStack.SharedKernel`, 2026-08-10; **Amendment 3: `IStronglyTypedId` gains `IsInitialized()` and is no longer a pure marker, 2026-08-10**; Amendments 4–7, 2026-08-27 to 2026-09-01: `idempotency_keys` mints no id, `gen_uuid_v7()` is not a PostgreSQL function, the retroactive disclosure of the 2026-05-21 edit, and an id's own formatting is not a wire format; **Amendment 8, 2026-09-07: a module-local identifier crosses a cross-module contract as `Guid`**) | +| 0023 | [Strongly-Typed ID Source Generator — Vogen](0023-strongly-typed-id-source-generator.md) | Vogen as the source generator for both IDs and value objects; `[ValueObject]` annotation; EF + JSON + ASP.NET + OpenAPI emitters out of the box (Amendment 1: Vogen 7.0.0 pin + architecture-test placement, 2026-05-21; Amendment 2: `UserId` / `TenantId` / `OrganizationId` are cross-cutting and live in `LearnStack.SharedKernel`, 2026-08-10; **Amendment 3: `IStronglyTypedId` gains `IsInitialized()` and is no longer a pure marker, 2026-08-10**; Amendments 4–7, 2026-08-27 to 2026-09-01: `idempotency_keys` mints no id, `gen_uuid_v7()` is not a PostgreSQL function, the retroactive disclosure of the 2026-05-21 edit, and an id's own formatting is not a wire format; **Amendment 8, 2026-09-07: a module-local identifier crosses a cross-module contract as `Guid`**) [Amendment 10 (2026-10-13)](0023-strongly-typed-id-source-generator.md#amendment-10--inbox-identity-reuses-the-producer-eventid-2026-10-13) corrects the inbox DB-side minting list through an inline erratum; inbox identity reuses producer EventId. | | 0024 | [API Versioning Policy](0024-api-versioning-policy.md) | URL-based `/v{N}/`; 6-month deprecation window; RFC 8594 `Sunset` + `Deprecation` headers; OpenAPI `deprecated` + `x-sunset` extensions; 410 Gone with RFC 7807 on sunset | | 0027 | [Frontend UI Localization with next-intl](0027-frontend-i18n.md) | Accepted 2026-10-09: next-intl 4.14.9, server-first app-local ICU catalogues, dotted UI keys and verified request-locale source; P02d-6 completes the foundation, catalog/course/lesson pages, automated product proofs and manual accessibility ([Amendment 3](0027-frontend-i18n.md#amendment-3--p02d-6-manual-accessibility-closeout-2026-10-10)); Phase 04 consumes the foundation | | 0028 | [`audit_log` Partition Management — Hangfire Recurring Job](0028-audit-log-partition-management.md) | Daily `learnstack:audit:partition-management` Hangfire job; create-ahead 2 months; drop only on platform-max retention horizon; row-level purge separate; no `pg_partman` dependency (Amendment 2026-09-07: Packet 9 creates a **plain, unpartitioned** table and executes none of this ADR's DDL; `Partition_Manager_Job_Is_Registered_AtStartup` moves to Phase 11 with the job it guards) | @@ -67,15 +67,15 @@ name P02b-2/P02b-3 activation; acceptance claims no runtime delivery. | 0035 | [Demand-Gated Infrastructure](0035-demand-gated-infrastructure.md) | The one-way-door test; ports + default implementations ship now, vendor adapters ship on a named trigger in a named phase; uncontracted deployment modes may not decide technical choices | | 0036 | [Trusted Inputs for Tenant and Organization Resolution](0036-tenant-resolution-trusted-inputs.md) | Resolution by **agreement, not priority** — every authoritative signal present is resolved independently and the request proceeds only on their intersection; no request header names a tenant or an organization, one header names a **host** over an authenticated hop and LearnStack still resolves it itself; `TenantContextOrigin` caps a host-only context to the `[PublicSurface]` read set; the platform-admin override leaves the resolution model | | 0037 | [What an Idempotency Key Identifies, Owns, and Replays](0037-idempotency-key-contract.md) | A client-chosen key is a **nonce inside a tenant's key space**, not an identity: `(tenant, key)` addresses the record and a fingerprint over organization, principal, method, path, query and body decides whether replaying it answers the question asked; a fencing token owns the claim; capacity is **admission, not eviction**, so nothing unexpired is ever displaced; the guarantee is at-most-once while a claim is live and at-least-once across process death | -| 0038 | [Cross-Cutting Port and Event Contracts](0038-cross-cutting-port-and-event-contracts.md) | **Supersedes ADR-0014.** Retains Dapr behind demand gates; fixes the event envelope, handler isolation, trace/audit scope and cache contracts. [Amendment 2 (2026-10-12)](0038-cross-cutting-port-and-event-contracts.md#amendment-2--additional-producer-payload-policy-2026-10-12) links ADR-0057's additional producer payload policy; original envelope, serializer and transport contracts survive. | +| 0038 | [Cross-Cutting Port and Event Contracts](0038-cross-cutting-port-and-event-contracts.md) | **Supersedes ADR-0014.** Retains Dapr behind demand gates; fixes the event envelope, handler isolation, trace/audit scope and cache contracts. [Amendment 2 (2026-10-12)](0038-cross-cutting-port-and-event-contracts.md#amendment-2--additional-producer-payload-policy-2026-10-12) links ADR-0057's additional producer payload policy; original envelope, serializer and transport contracts survive. [Amendment 3](0038-cross-cutting-port-and-event-contracts.md#amendment-3--bounded-publish-outcome-supersession-by-adr-0046-2026-10-12) links ADR-0046's publish-outcome replacement. [Amendment 4](0038-cross-cutting-port-and-event-contracts.md#amendment-4--durable-acceptance-and-subscriber-delivery-implemented-2026-10-13) records P02b-3 durable acceptance and independent subscriber execution delivery. | | 0039 | [The Optimistic Concurrency Token](0039-optimistic-concurrency-token.md) | An explicit `row_version bigint` (CLR `long`), project-wide, incremented inside `AuditableEntity` by the single primitive `MarkUpdated` and `SoftDelete` both route through, and mapped with `HasDefaultValue(0L).IsConcurrencyToken().ValueGeneratedNever()` — exactly those three calls (Amendment 2): `ValueGeneratedOnAddOrUpdate()` / `IsRowVersion()` make EF omit the column from the `UPDATE` entirely (Amendment 1, measured), and `HasDefaultValue` alone leaves `ValueGenerated` at `OnAdd`, which the registered rule rejects; `xmin` is rejected because the token is client-visible through ETag / `If-Match` and a dump-restore or logical-replication cutover changes it — measured, unlike the widely-cited `VACUUM FREEZE` objection, which does not | | 0040 | [The Ambient Unit of Work](0040-ambient-unit-of-work.md) | One `DbConnection` per scope, owned by `IUnitOfWork`; every module `DbContext`, `IAuditStore` and `IOutbox` enlists on it, because `SET LOCAL` is connection-local and a context on its own connection reads **zero rows** under the corrected RLS policy. Spans one aggregate's write plus audit plus outbox plus any reads — cross-aggregate writes stay forbidden, with the one enumerated exception [ADR-0042](0042-tenant-provisioning-cross-aggregate-transaction.md) carves for tenant provisioning (Amendment 4, 2026-08-30). Defines nesting, disposal, and the event-consumer entry point that never reaches MediatR. The closed set of `app.tenant_id` setters is seven by Amendment 3 (2026-08-30) and eight by **Amendment 7 (2026-09-11)**, which adds the tenant-flag projection loader `IFeatureFlags` fills on a cache miss. [Amendment 10 (2026-10-12)](0040-ambient-unit-of-work.md#amendment-10--bounded-produced-work-refusal-supersession-2026-10-12) discloses ADR-0057's bounded produced-work refusal supersession; [Amendment 11 (2026-10-12)](0040-ambient-unit-of-work.md#amendment-11--named-transaction-announcement-snapshot-reader-2026-10-12) separately adds the named read-only announcement snapshot seam. | | 0041 | [Correcting False Statements in Accepted ADRs](0041-correcting-false-statements-in-accepted-adrs.md) | Inline erratum beside the false text is the default; in-place replacement only where the text is a **canonical artifact for reuse** — a template others are told to copy, a DDL or command meant to be applied — never merely because something could be copied, and never because a non-ADR carrier holds the same token. Bounded to statements false **when they entered the record**, judged per statement: original body at the acceptance commit, amendment text at that amendment's date. A statement that has since gone stale is history, and gets an amendment or a superseding ADR. Every changed Accepted ADR carries its own dated Amendment | | 0042 | [Tenant Provisioning as a Bounded Cross-Aggregate Transaction](0042-tenant-provisioning-cross-aggregate-transaction.md) | A standing, **enumerated** exception to § Aggregate Ownership: tenant provisioning writes `Tenant` and its default `Organization` in one transaction, because `tenants.default_organization_id` carries an invariant an integration event cannot deliver — the substitute moves the second write to a later transaction and the window between them is exactly what the invariant forbids. Bounded by enumeration rather than by principle: two roots, one operation, a literal allow-list of one in the architecture test. Covers no child entity, no projection, and no cross-**module** write, which stays forbidden with no exception | | 0043 | [What LearnStack Does With a Tenant-Authored Payload](0043-customization-payload-validation.md) | JSON Schema draft 2020-12, evaluated by `JsonSchema.Net` **pinned at 8.0.5** (the last MIT-expression release) behind an `IJsonSchemaValidator` port. A tenant document passes **four ordered gates** — JSON, the LearnStack profile, the 2020-12 meta-schema, then the build — because the library's own defaults keep none of the rules the corpus states: its default dialect is not 2020-12, a tenant's `$schema` line overrides the pinned one, a tenant's `$id` becomes process-global state (first-writer-wins, a cross-tenant denial of service), and `true` / `false` / `{}` are all valid schemas that switch validation off. `pattern` is refused — the built `Regex` carries an infinite `MatchTimeout` and `maxLength` does not short-circuit it (measured, 30.2 s on one property). **No compiled-validator cache**: compiling is measured *cheaper* than evaluating (0.5× at the 100-property ceiling), so § 8.2's cache is deleted rather than re-keyed. Rule bodies are opaque `text` with a `dialect` discriminator naming the language of their `condition` expressions. Amendments 1–2 (2026-09-04/05) refuse **every** `$ref` cycle and bound the reference graph; Amendment 3 (2026-09-06) seals the port's exception surface and makes `ValidateInstance` enforce § 8.4's entry cap; **Amendment 4 (2026-09-07) refuses what the `jsonb` column refuses** — `U+0000`, an unpaired surrogate, a number outside `numeric`; **Amendment 5 (2026-10-02) adds ADR-0051’s optional ordered text-card profile** | -| 0044 | [The Audit Write Path](0044-audit-write-path.md) | **Amendments 1–5 (2026-09-08), 6–7 (2026-09-11).** The eleven questions ADR-0033 left open, decided before the first audit file: the reserved **non-nil** platform sentinel tenant id (the nil uuid is refused by three shipped guards that read all-zero as *no tenant*), and a `tenants` CHECK that keeps it unprovisionable; the row's `tenant_id` is the tenant the **transaction announced**, so a provisioning command audits under the tenant it creates and `make seed` still commits; **N intents per request**, one per audited resource, because two shipped commands already write two — and only the **owning** unit-of-work frame writes them or reports the commit boundary, which is what stops a nested `Send` from claiming durability nothing committed; `outcome` closes at four values (`indeterminate` included) and the writer always supplies `timestamp`, so ADR-0033's deliberate duplicate-id pair is legal under `(id, timestamp)` rather than a `23505`; the catalogue is **code** keyed by request type and the matrix is prose, with a test asserting they agree, and every request must be classified — there is no `RequestKind.Other`; the interceptor captures **every** tracked entity minus a named exclusion list, because the `AuditableEntity<>` predicate was blind to `platform_host_to_tenant`; `[PiiSensitive]` and an elision record that preserves the column's JSON type; `audit_log` is org-scoped, carries **no** FK to `tenants`, and both standalone writers announce **two** GUCs — measured: an org-scoped `denied` row is refused without the second. Amendment 1 makes `audit_config` tenant-**wide**; Amendment 2 corrects the redaction sentinel to the shipped constant and the reason `FORCE` needs a trigger (the policy binds the owner by tenant, not by immutability — measured); Amendment 3 gives the matrix↔catalogue join two directions with two domains, because 23 of 30 shipped matrix slugs have no request type and the rule as written was red on arrival, and moves the audit value types into `SharedKernel.Audit` to break the same project cycle `AuditEntryId` had; Amendment 4 makes `[PiiSensitive]` property-granular, `jsonb` included, ships `audit_config` unwritten and binds the matrix rule to modules with code; Amendment 5 guards the sentinel where it is announced and fills `entity_type` / `entity_id` from the declared aggregate; Amendment 6 — from an external review that reproduced four defects against real PostgreSQL — has a handler designate the subject when it writes two instances of its aggregate, carries the actor and correlation id on the intent, keeps each intent's own request's result, captures a contained entity with its aggregate, and instance-qualifies `changes` pointers; Amendment 7 corrects § 7's count of plain entities — seven today, five of them MUST — and names the Tenancy matrix alone as the one that singles out the host mapping. [Amendment 8 (2026-10-12)](0044-audit-write-path.md#amendment-8--common-audit-ownership-outside-mediatr-2026-10-12) clarifies common audit ownership outside MediatR under ADR-0057 without adding an audit writer. | +| 0044 | [The Audit Write Path](0044-audit-write-path.md) | **Amendments 1–5 (2026-09-08), 6–7 (2026-09-11).** The eleven questions ADR-0033 left open, decided before the first audit file: the reserved **non-nil** platform sentinel tenant id (the nil uuid is refused by three shipped guards that read all-zero as *no tenant*), and a `tenants` CHECK that keeps it unprovisionable; the row's `tenant_id` is the tenant the **transaction announced**, so a provisioning command audits under the tenant it creates and `make seed` still commits; **N intents per request**, one per audited resource, because two shipped commands already write two — and only the **owning** unit-of-work frame writes them or reports the commit boundary, which is what stops a nested `Send` from claiming durability nothing committed; `outcome` closes at four values (`indeterminate` included) and the writer always supplies `timestamp`, so ADR-0033's deliberate duplicate-id pair is legal under `(id, timestamp)` rather than a `23505`; the catalogue is **code** keyed by request type and the matrix is prose, with a test asserting they agree, and every request must be classified — there is no `RequestKind.Other`; the interceptor captures **every** tracked entity minus a named exclusion list, because the `AuditableEntity<>` predicate was blind to `platform_host_to_tenant`; `[PiiSensitive]` and an elision record that preserves the column's JSON type; `audit_log` is org-scoped, carries **no** FK to `tenants`, and both standalone writers announce **two** GUCs — measured: an org-scoped `denied` row is refused without the second. Amendment 1 makes `audit_config` tenant-**wide**; Amendment 2 corrects the redaction sentinel to the shipped constant and the reason `FORCE` needs a trigger (the policy binds the owner by tenant, not by immutability — measured); Amendment 3 gives the matrix↔catalogue join two directions with two domains, because 23 of 30 shipped matrix slugs have no request type and the rule as written was red on arrival, and moves the audit value types into `SharedKernel.Audit` to break the same project cycle `AuditEntryId` had; Amendment 4 makes `[PiiSensitive]` property-granular, `jsonb` included, ships `audit_config` unwritten and binds the matrix rule to modules with code; Amendment 5 guards the sentinel where it is announced and fills `entity_type` / `entity_id` from the declared aggregate; Amendment 6 — from an external review that reproduced four defects against real PostgreSQL — has a handler designate the subject when it writes two instances of its aggregate, carries the actor and correlation id on the intent, keeps each intent's own request's result, captures a contained entity with its aggregate, and instance-qualifies `changes` pointers; Amendment 7 corrects § 7's count of plain entities — seven today, five of them MUST — and names the Tenancy matrix alone as the one that singles out the host mapping. [Amendment 8 (2026-10-12)](0044-audit-write-path.md#amendment-8--common-audit-ownership-outside-mediatr-2026-10-12) clarifies common audit ownership outside MediatR under ADR-0057 without adding an audit writer. [Amendment 11 (2026-10-13)](0044-audit-write-path.md#amendment-11--shared-delivery-capture-exclusions-2026-10-13) accepts the inbox/delivery machinery capture exclusions under ADR-0046; P02b-3 implements them through the shared business owner. | | 0045 | [The Entitlement and Feature-Flag Socket](0045-entitlement-and-feature-flag-socket.md) | **Amendment 1 (2026-09-08)** reads it against the Hub's merged code: the limit vocabulary is the Hub's, `expires_at`/`valid_until` are nullable, the generation guard admits the equal case, and `platform_killswitches` ships **unwritten** because `DenyAllPlatformAdminGate` makes a writer unreachable until Phase 03. **Amendment 2 (2026-09-11):** a registry's membership is the vocabulary the contract names — fourteen Hub features, nine `limits.*`, two tenant flags, three killswitches — and enforcement, not membership, waits for a consumer. **Amendment 3 (2026-09-11):** a key's fail-open/fail-closed class decides only when no projection exists; one past its grace window is read-only, as ADR-0021 decides. **Amendment 4 (2026-09-11):** of the projection's names, `PlanCode` differs from the wire (`tier`) and `ExpiresAt` from the column (`valid_until`) — § 1 had called both wire differences. Declares the port twenty documents name and none define. `IEntitlementProvider.GetAsync(TenantId)` + `RefreshAsync(projection)`, the projection carrying every field `entitlement-v1.schema.json` requires — **`compliance` and `generation` included**, both already `NOT NULL` columns — and the refresh **generation-guarded inside the write statement**, so a stale push cannot resurrect a revoked plan. `IFeatureFlags` is the only module-facing read and **composes over the provider** rather than querying `platform_entitlement_cache`, which is the only reading under which the Phase 02a criterion — swapping the provider changes the answer — can be true. Limits: **`-1` unlimited, `0` denied**, `long` not `long?`; the inverse reading would have made the degraded read-only mode grant unlimited. `NullEntitlementProvider` is registered in **every** deployment mode, per ADR-0035's default-implementation gate rather than ADR-0020's Development-only switch. Killswitches leave `tenant_feature_flags` for a platform-scoped `platform_killswitches`: a foreign key to `tenants` is a constraint no role or `BYPASSRLS` moves, so the sentinel could never have satisfied it. Push-primary, **not** push-only — ADR-0034's `license/verify` fallback stands; the no-Hub absolute belongs to host resolution | -| 0046 | [Durable Event Delivery and Recovery](0046-durable-event-delivery-and-recovery.md) | Accepted 2026-10-12: fenced producer leases/retries, Pending readiness holds and atomic row-bound terminal audit; P02b-2 Steps 1–2 implemented with both review rounds complete; Step 3 hosted dispatch/observation implemented with both independent review rounds complete; Step 4 enforcement and local closeout complete; all four steps completed both independent review rounds; the [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) owns CI/merge status. [Amendment 2](0046-durable-event-delivery-and-recovery.md#amendment-2--producer-enforcement-and-local-closeout-2026-10-12) records verified local closeout. [Amendment 1](0046-durable-event-delivery-and-recovery.md#amendment-1--producer-runtime-delivery-2026-10-12) records producer runtime delivery and its review evidence. Shared inbox/durable subscription acceptance and independent retries remain P02b-3. Named bounded supersessions preserve earlier Accepted records. | +| 0046 | [Durable Event Delivery and Recovery](0046-durable-event-delivery-and-recovery.md) | Accepted 2026-10-12: fenced producer leases/retries, Pending readiness holds and atomic row-bound terminal audit; P02b-2 is complete and merged through PR #30; its [merge closeout](../roadmap/phase-02b-events-auth.md#p02b-2-merge-and-closeout-2026-10-13) owns final verification. [Amendment 2](0046-durable-event-delivery-and-recovery.md#amendment-2--producer-enforcement-and-local-closeout-2026-10-12) records verified local closeout. [Amendment 1](0046-durable-event-delivery-and-recovery.md#amendment-1--producer-runtime-delivery-2026-10-12) records producer runtime delivery and its review evidence. [Amendment 3](0046-durable-event-delivery-and-recovery.md#amendment-3--consumer-runtime-and-enforcement-delivery-2026-10-13) records P02b-3 consumer runtime, both Active defaults and production enforcement; its Step 6 record owns validation/review/PR status. Named bounded supersessions preserve earlier Accepted records. | | 0050 | [Publication and Course Content Access](0050-publication-and-course-content-access.md) | Accepted 2026-10-02: publication/access separation, restricted backfill and anonymous denial; supersedes ADR-0048; P02d-2 policy, migration, writers and seed delivered ([Amendment 1](0050-publication-and-course-content-access.md#amendment-1--p02d-2-implementation-delivery-2026-10-02)); P02d-4 public-read enforcement delivered, grants Phase 07 | | 0051 | [Ordered Text Card Presentation](0051-ordered-text-card-presentation.md) | Accepted 2026-10-02: optional root `x-fields`, localized ordered text cards; compatible extension linked by ADR-0043 Amendment 5; P02d-2 profile parsing, resolution and seed verification delivered ([Amendment 1](0051-ordered-text-card-presentation.md#amendment-1--p02d-2-implementation-delivery-2026-10-02)); P02d-4 public projection delivered; P02d-6 completes ordered plain-text rendering, automated product proofs and manual accessibility smoke | | 0052 | [Anonymous Public Read Boundary](0052-anonymous-public-read-boundary.md) | Accepted 2026-10-03: host-resolved bootstrap, immutable host ceiling, mandatory marked dispatch and read-only ambient frames; ADR-0036/0040 Amendment 8; P02d-4 Steps 1–3 deliver host provenance, physical read-only frames and all institution GET/HEAD reads; both Step 3 review rounds pass; Step 4 delivers OpenAPI/SDK/CI controls; both review rounds and focused fix verification pass; six real required CI jobs and live sixth-check activation verified; P02d-4 complete and merged through PR #25 on 2026-10-08 ([Amendment 13](0052-anonymous-public-read-boundary.md#amendment-13--merge-closeout-2026-10-08)) | @@ -83,7 +83,7 @@ name P02b-2/P02b-3 activation; acceptance claims no runtime delivery. | 0054 | [Bounded Public Renderer Admission](0054-bounded-public-renderer-admission.md) | Accepted 2026-10-09: coordinated anonymous accounting, native GET/HEAD/production-upgrade admission and equivalent redirect-query encoding are delivered. All five remediation steps are complete and merged through PR #26; the [merge closeout](../roadmap/phase-02d-walking-skeleton.md#p02d-5-merge-and-closeout-2026-10-09) owns final verification ([Amendment 6](0054-bounded-public-renderer-admission.md#amendment-6--merge-closeout-2026-10-09)) | | 0055 | [Handle Public Renderer Bootstrap Failures in Middleware](0055-public-renderer-bootstrap-failures.md) | Accepted 2026-10-10: one live middleware bootstrap and an immutable native same-request snapshot; exact bootstrap 404/429/503, bounded renderer Retry-After and two/one/three-call accounting implemented. Production replacement proofs pass; all three steps completed both review rounds and focused fix verification. [Step 3](../roadmap/phase-02d-walking-skeleton.md#adr-0055-step-3--production-admission-proof-and-closeout) owns delivery evidence, completed reviews and verified fixes; five new catalogue rules are Implemented | | 0056 | [Public Demo Browser and Lighthouse Enforcement](0056-public-demo-browser-and-lighthouse-enforcement.md) | Accepted 2026-10-11: pinned Lighthouse/Playwright Chromium, seed-derived nine-page browser proof, 27 serial audits, actual-page admission, normalized gzip/CLS/accessibility hard checks and advisory timings delivered; four proof rules Implemented, real PR CI/artifacts pass and seventh required context activated ([Amendment 1](0056-public-demo-browser-and-lighthouse-enforcement.md#amendment-1--p02d-7-enforcement-delivery-2026-10-11)); complete and merged through PR #28 on 2026-10-11 ([merge closeout](../roadmap/phase-02d-walking-skeleton.md#p02d-7-merge-and-closeout-2026-10-11)) | -| 0057 | [Transactional Event Production and Completion](0057-transactional-event-production-and-completion.md) | Accepted 2026-10-12: immediate ambient enqueue, owner-only domain completion, sticky produced-work refusal protection, identifier-only bounded payloads and completed-provisioning publisher; eight producer rules Implemented; Steps 1–3 implement shared execution, immediate enqueue, successful-save collection and the publisher; Step 4 adds production censuses and catalogue enforcement. P02b-1 is complete and merged through PR #29 on 2026-10-12; the [merge closeout](../roadmap/phase-02b-events-auth.md#p02b-1-merge-and-closeout-2026-10-12) owns final verification. Amendment 1 links delivery verification and review status. P02b-3 owns default-initialization subscriptions and Seeder handoff. [Amendment 2 (2026-10-12)](0057-transactional-event-production-and-completion.md#amendment-2--correct-adr-0032-clause-locators-2026-10-12) corrects the ADR-0032 clause locators through inline errata; the Decision and bounded supersession are unchanged. | +| 0057 | [Transactional Event Production and Completion](0057-transactional-event-production-and-completion.md) | Accepted 2026-10-12: immediate ambient enqueue, owner-only domain completion, sticky produced-work refusal protection, identifier-only bounded payloads and completed-provisioning publisher; eight producer rules Implemented; Steps 1–3 implement shared execution, immediate enqueue, successful-save collection and the publisher; Step 4 adds production censuses and catalogue enforcement. P02b-1 is complete and merged through PR #29 on 2026-10-12; the [merge closeout](../roadmap/phase-02b-events-auth.md#p02b-1-merge-and-closeout-2026-10-12) owns final verification. Amendment 1 links delivery verification and review status. [Amendment 2 (2026-10-12)](0057-transactional-event-production-and-completion.md#amendment-2--correct-adr-0032-clause-locators-2026-10-12) corrects the ADR-0032 clause locators through inline errata; the Decision and bounded supersession are unchanged. [Amendment 3](0057-transactional-event-production-and-completion.md#amendment-3--useful-default-consumers-and-semantic-seed-handoff-2026-10-13) records the two Active default subscriptions and semantic Seeder handoff delivered in P02b-3. | ## Proposed ADRs diff --git a/docs/glossary.md b/docs/glossary.md index 8b719f59..6e2a267b 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -115,8 +115,9 @@ This glossary defines LearnStack-specific terms. When a term is ambiguous across | **Integration Event** | An event published outward via the outbox so that other modules or external consumers can react. | | **Outbox** | The transactional buffer that turns domain changes into integration events without dual-write inconsistency. | | **Producer frame** | Per-invocation bookkeeping implemented by scoped `ProducerExecution` in P02b-1 Steps 1–3, including nested ancestry, sticky successful INSERT/save-promotion receipts and no-listener W3C Activity. Saved facts retain the saving invocation's span and causal actor through owner-only translation. Receipts are distinct from transaction and audit ownership. See [ADR-0057](decisions/0057-transactional-event-production-and-completion.md). | -| **Audit frame** | The boundary grouping classified operation intents and captured changes. Nested operations contribute intents; only the outermost frame reconciles and clears them. P02b-1 Step 1 supplies `AuditExecutionCoordinator` for request and direct entry; the scoped capture's `HasOpenFrame` identifies outermost reconciliation, independently of transaction/producer ownership. `AuditLogBehavior` is the step-3 adapter. Production delivery remains P02b-3 under [ADR-0057](decisions/0057-transactional-event-production-and-completion.md). | +| **Audit frame** | The boundary grouping classified operation intents and captured changes. Nested operations contribute intents; only the outermost frame reconciles and clears them. P02b-1 Step 1 supplies `AuditExecutionCoordinator` for request and direct entry; the scoped capture's `HasOpenFrame` identifies outermost reconciliation, independently of transaction/producer ownership. `AuditLogBehavior` is the step-3 adapter. P02b-3 Step 3 wires delivery through the same owner; Step 5 activates both real defaults under [ADR-0057](decisions/0057-transactional-event-production-and-completion.md). | | **Transaction announcement snapshot** | Read-only `TenantContextAnnouncement` returned by `IUnitOfWork.GetTenantContextAnnouncement(DbTransaction?)`: a record of the tenant/organization context successfully announced on one physical transaction generation. It supplies producer checks without granting a new tenant-context writer or setter. P02b-1 Steps 1–3 implement the reader/generation lifecycle, outbox admission and save-origin checks under [ADR-0057](decisions/0057-transactional-event-production-and-completion.md). | +| **Delivery completion gate** | Attempt-local, single-use arbitration implemented by P02b-3 Step 3. Closing an open gate prevents commit; admission and cancellation cannot both win. Acknowledged COMMIT remains definitive through later cleanup, while an unacknowledged admitted COMMIT requires durable reconciliation. The gate changes no SQL authority. See [ADR-0046](decisions/0046-durable-event-delivery-and-recovery.md#subscriber-acceptance-and-inbox-ownership). | | **Domain-event collector** | Scoped `DomainEventCollector`, implemented in P02b-1 Step 3. A SaveChanges interceptor stages only changed sources of the saving context and promotes them after nonzero successful persistence. A FIFO queue retains those facts and their producer origins until owner-only completion; a separate retained-source set clears even uncollected/detached facts at owner termination. Retention is never save eligibility. See [ADR-0057](decisions/0057-transactional-event-production-and-completion.md). | | **Learning Event** | An analytics event describing learner behavior (lesson viewed, assessment completed). | | **Commerce Event** | An analytics event describing the commerce funnel. | @@ -248,7 +249,7 @@ These entries distinguish P02d-2 contracts from P02d-3 internal read contracts: | **Audit Operation Key** | The catalogue key `(module, operation)`, where `operation` is the dotted slug `{module}.{resource}.{verb}` — `tenancy.tenant.create`, `customization.content_type.publish`. It borrows the **shape** of a *Permission Key* and shares its first two segments with the permission gating the same resource, but **not** its closed action set: an audit verb records what happened and comes from the module's own coverage matrix (`create`, `publish`, `revise`, `rename`, `soft_delete`, …), where a permission action bounds what a principal may do. There is no `RequestKind.Other` — every `IRequest>` reaching pipeline step 3 must be classified, an unregistered one is rejected with `audit_unclassified_operation` (500), and test-only request types register through the same builder in their fixture. Per [ADR-0044 § 6](decisions/0044-audit-write-path.md). | | **Audit Coverage Matrix** | The per-module table in `docs/modules//audit.md` mapping resources × operations to MUST / SHOULD / MAY / – classifications, with an `Operation` column carrying the dotted slug. Required for every module spec. It is the human-readable artifact; the **audit catalogue** — declared in code through `IAuditCatalogSource.Describe(IAuditCatalogBuilder)`, discovered from DI and keyed by request type — is the executable one. Neither is parsed from the other: `Every_TenantOwned_Command_HasAuditCoverage` (catalogue → matrix) and `Every_Matrix_Row_Whose_Command_Exists_Is_Registered` (matrix → catalogue) assert they agree, in **two directions with two domains**. *Catalogue → matrix is total* — every entry a module's source registers has a row in that module's matrix carrying the same slug, no exemption. *Matrix → catalogue binds only to slugs whose request type exists* — a row classified ahead of the command that will raise it carries **`(planned)`** in the `Operation` cell and is outside that direction, because classifying before the command ships is what the standard asks for. The marker is a claim the rule re-checks, not an escape: a `(planned)` row whose command **has** shipped fails. Operations that are not MediatR requests at all carry **`(off-path)`**, are outside the join in both directions, and register by slug rather than by type; the closed list of them is enumerated once, in the amendment that decides it. Per [ADR-0044 § 6 and Amendment 3](decisions/0044-audit-write-path.md). | | **AuditConfig** | The per-tenant override of the catalogue's coverage, held as a cached projection whose loader runs on a cache miss on its own short transaction — once per tenant and generation, never once per request. It **narrows and never elevates**: `is_enabled = false` silences a SHOULD or a MAY, `true` is the baseline, and neither touches a MUST. The classifier applies the override and then re-applies the catalogue's MUST floor. Elevation is refused rather than deferred: a lever that moved an operation onto the durable path would let a tenant admin provoke a `503` on operations the platform classified MAY ([ADR-0033 Amendment 4](decisions/0033-audit-durability-model.md)). A failure to **read** it does **not** reject the operation: classification falls back to the in-process catalogue, which carries the same MUST floor, so nothing proceeds unaudited and one cache outage does not deny every request platform-wide. The failure logs at `Error`; the audit health check does not move, because it answers only whether a MUST-class row can be written. Per [ADR-0033 and its Amendment 6](decisions/0033-audit-durability-model.md). | -| **Audit Capture Pipeline** | `AuditChangeTrackerInterceptor` (EF) → `IAuditStateCapture` (before/after/`changes` JSON) → `AuditExecutionCoordinator` (MediatR adapter or direct owner) → `IAuditStore`. Modules never write `audit_log` directly. The interceptor captures **every** `ChangeTracker` entry in state `Added`, `Modified` or `Deleted`, minus a named exclusion list — `OutboxMessage`, `IdempotencyKey`, `AuditEntry`, `AuditConfig`. The earlier `AuditableEntity<>`-descendants predicate is withdrawn: it was blind to every plain class the modules ship, `PlatformHostMapping` and the other MUST-classified ones among them — [Audit Coverage § Required Behaviours](standards/18-audit-coverage.md#required-behaviours) lists them with their classes. It sees only what the `ChangeTracker` holds, so EF Core's set-based writes (`ExecuteUpdate`, `ExecuteDelete`, `ExecuteSql*`) would escape it and backend source uses none. It **captures only** — no row construction, no SQL — and reaches every module `DbContext` through `AddModuleDbContext`, which resolves `IEnumerable` from the provider, because DI registration alone does not attach it. An entity that is not an aggregate root and is reached through its root's navigation — a taxonomy's bands — is captured **inside its root's** diff rather than on its own, and in its root's snapshot only when the root was created in the same request and every member it was seen with is still tracked, the one case its membership is known to be complete; a filtered `Include` marks a partial collection loaded, so a loaded one is left out as unknown. `changes` serialises as a JSON **array** of `{ path, before, after }` with an instance-qualified pointer — `/{EntityType}/{EntityId}` and then an RFC 6901 pointer into that instance's snapshot — never ADR-0016's polymorphic object-or-array shape. Per [ADR-0044 § 7 and Amendment 6 §§ 4–5](decisions/0044-audit-write-path.md). | +| **Audit Capture Pipeline** | `AuditChangeTrackerInterceptor` (EF) → `IAuditStateCapture` (before/after/`changes` JSON) → `AuditExecutionCoordinator` (MediatR adapter or direct owner) → `IAuditStore`. Modules never write `audit_log` directly. The interceptor captures **every** `ChangeTracker` entry in state `Added`, `Modified` or `Deleted`, minus the named machinery/audit exclusion list (including P3's shared inbox and delivery ledger), owned by Audit Coverage below. The earlier `AuditableEntity<>`-descendants predicate is withdrawn: it was blind to every plain class the modules ship, `PlatformHostMapping` and the other MUST-classified ones among them — [Audit Coverage § Required Behaviours](standards/18-audit-coverage.md#required-behaviours) lists them with their classes. It sees only what the `ChangeTracker` holds, so EF Core's set-based writes (`ExecuteUpdate`, `ExecuteDelete`, `ExecuteSql*`) would escape it and backend source uses none. It **captures only** — no row construction, no SQL — and reaches every module `DbContext` through `AddModuleDbContext`, which resolves `IEnumerable` from the provider, because DI registration alone does not attach it. An entity that is not an aggregate root and is reached through its root's navigation — a taxonomy's bands — is captured **inside its root's** diff rather than on its own, and in its root's snapshot only when the root was created in the same request and every member it was seen with is still tracked, the one case its membership is known to be complete; a filtered `Include` marks a partial collection loaded, so a loaded one is left out as unknown. `changes` serialises as a JSON **array** of `{ path, before, after }` with an instance-qualified pointer — `/{EntityType}/{EntityId}` and then an RFC 6901 pointer into that instance's snapshot — never ADR-0016's polymorphic object-or-array shape. Per [ADR-0044 § 7 and Amendment 6 §§ 4–5](decisions/0044-audit-write-path.md). | | **Elision Record** | What replaces a snapshot value too large to store: `{"_elided": true, "bytes": , "sha256": ""}` — never an empty object and never a silent truncation. Each of `before_state`, `after_state` and `changes` is capped at `JsonValue.MaxRowBytes` (256 KiB). The elision **preserves the column's JSON type**: `changes` is an array on both sides of the cap, so an elided `changes` is `[{"_elided": …}]`, while `before_state` and `after_state` take the object form — a reader must not have to branch on shape to know what it is reading. `audit_blob_id` is struck from the corpus: it exists in no DDL and names no blob store. Per [ADR-0044 § 8](decisions/0044-audit-write-path.md). | | **Audit Intent** | One parked declaration that an audited `(resource, operation)` happened, minted with its `AuditEntryId` by `AuditExecutionCoordinator` (through step 3 for requests) and held in `IAuditStateCapture` as an **ordered list** — one intent per audited `(resource, operation)`, **not** one per request. `ProvisionTenantCommand` declares two, `tenancy.tenant.create` and `tenancy.organization.create`, which is why "exactly one intent per request" is retired. Only the **owning** unit-of-work frame (`IUnitOfWorkScope.IsOwner`) flushes the intents, signals `Committed` / `RolledBack` / `Indeterminate`, and it drains every intent in the scope rather than only its own; a joiner writes nothing and signals nothing. Only the outermost audit frame reconciles and calls `Clear()`; a nested frame that cleared would erase the outer execution's intents before commit. It also carries the `TenantId` and optional `OrganizationId` its row will be written under, resolved by `AuditExecutionCoordinator` before deferred work — through step 3 for requests, where ADR-0044 § 2's four cases are decidable, and the reason `PostgresAuditStore` composes a row from the intent rather than resolving a tenant itself. It carries the actor and the correlation id too, read at the same moment, so both writers write them. Every classified `AuditExecutionCoordinator` invocation opens an audit **frame**: an intent belongs to the frame that declared it, a captured change to the frame whose handler flushed it, and closing a frame records on its intents what that request returned — so a nested request refused under an outer one that commits keeps its refusal. Per [ADR-0044 §§ 2, 3, 4, Amendment 3 and Amendment 6 §§ 2–3](decisions/0044-audit-write-path.md) and [ADR-0033 Amendment 2](decisions/0033-audit-durability-model.md). | | **Audit Subject** | The one aggregate instance an audit row is about: its `entity_id`, `before_state` and `after_state`. The composer finds it among the captures of the intent's declared type; when a request writes **two** instances of that type — a publication retires the incumbent revision and activates the successor — the handler names it through `IAuditSubject.Designate(aggregate)`, and every other instance stays in `changes` under its own instance-qualified pointer. Undesignated, two instances are refused rather than guessed between. A designation binds only the current request's intents of that type. Per [ADR-0044 Amendment 6 § 1](decisions/0044-audit-write-path.md). | @@ -324,12 +325,12 @@ These entries distinguish P02d-2 contracts from P02d-3 internal read contracts: | **Demand-Gated Building Block** | An infrastructure choice that ships as a **port plus a working default implementation** now, and as a **vendor adapter later**, in a named phase, when a written trigger fires. It qualifies only when all four exist: the port, the default implementation, the owning phase, and the trigger condition. A block missing any of the four is not demand-gated — it is simply missing. Distinguished from a one-way door by the test below. The gated set and each trigger are tabulated in [ADR-0035](decisions/0035-demand-gated-infrastructure.md), which also names its own two exceptions — `audit_log` partitioning, which is schema-internal and so has no port, and LiveKit, which has no default because its absence is a missing product feature rather than a missing implementation. | | **One-Way Door** | A decision that gets more expensive with every week of delay, tested by the question: *if I add this six months from now, will I have to touch code that is already written?* **Yes → one-way door; ship it now** (tenant and organization isolation, the `outbox_messages` table and its ownership, strongly-typed identifiers, the localization schema — each touches every query, migration, or job payload ever written). **No → additive; ship the port now and the adapter on demand.** The test is mechanical, not a matter of taste: tenant isolation's cost grows with the codebase, a Dapr adapter's cost does not. A corollary rule: **a deployment mode or customer segment without a signed contract cannot be the deciding factor in a technical choice** — it may break a tie between otherwise-equal options, nothing more. Per [ADR-0035](decisions/0035-demand-gated-infrastructure.md) and [Engineering Principles](standards/00-principles.md). | | **Dapr Building Blocks** | The three Dapr abstractions LearnStack uses **when it uses Dapr**: pub/sub (Kafka), state (Valkey), secrets (Vault) per [ADR-0038](decisions/0038-cross-cutting-port-and-event-contracts.md). Service invocation, workflow, bindings, and actors are out of scope. Demand-gated to [Phase 11](roadmap/phase-11-production-hardening.md); ADR-0038 decides *what*, [ADR-0035](decisions/0035-demand-gated-infrastructure.md) decides *when*. | -| **`IEventBus`** | Non-generic envelope publication port. `InProcessEventBus` is the only registered implementation until the Phase 11 trigger. P02b-3 changes success to durable acceptance by every required subscription, then independent execution owns business outcomes; failed/unknown acceptance remains producer retry. The outbox processor remains the only production publisher; subscriber retries invoke an internal executor. Current ambient restoration/per-key invocation exists; durable acceptance/inbox do not. See [ADR-0046](decisions/0046-durable-event-delivery-and-recovery.md). | +| **`IEventBus`** | Non-generic envelope publication port. `InProcessEventBus` is the only registered implementation until the Phase 11 trigger. P02b-3 Step 2 implements durable acceptance by every required subscription without business construction; Step 3 implements independent transactional execution and owns business outcomes; failed/unknown acceptance remains producer retry. The outbox processor remains the only production publisher; subscriber retries invoke an internal executor. Context restoration, per-key acceptance and test-owned business execution are implemented; Step 4 implements independent scheduling/terminalization/replay and bounded observation; Step 5 activates both real defaults and semantic Seeder handoff. See [ADR-0046](decisions/0046-durable-event-delivery-and-recovery.md). | | **`IOutbox`** | Immediate integration-event INSERT through the enrolled, announced, writable ambient transaction. `PostgresOutbox` is registered in API and Seeder by P02b-1 Step 2; it opens no connection, commits nothing and calls no transport. Identity is `EventId`; duplicates fail. Concrete payloads use the frozen serializer, identifiers only, and the inclusive 262,144-byte UTF-8 `OutboxPayloadLimits.MaxBytes`. `metadata` is SQL NULL. See [ADR-0057](decisions/0057-transactional-event-production-and-completion.md). | | **`IntegrationEventPartitionAttribute`** | The concrete event's explicit partition-source declaration: a non-nullable aggregate identifier, or `TenantId` with `tenantWide: true`. Runtime admission compares the actual canonical lowercase `D` GUID value. Inherited declarations and implicit tenant partitioning are refused. [ADR-0046](decisions/0046-durable-event-delivery-and-recovery.md#producer-dispatch-and-recovery) defines eligible-selection order and its limits; the declaration supplies no durable publish-order guarantee. | | **`IntegrationEventEnvelope`** | One integration event plus the dispatch metadata the outbox row carries and the event does not: W3C `CorrelationId`, optional `OrganizationId`, `CausationId`, and causal `ActorUserId`. Its `Topic` and `PartitionKey` are the event's own — both are properties of the event *type* rather than of one delivery, so neither can hold a second answer ([ADR-0038](decisions/0038-cross-cutting-port-and-event-contracts.md)). An event implementing `IOrganizationScopedIntegrationEvent` requires a non-empty organization in its envelope. Metadata describes the *delivery*; the event describes the *fact*. | | **`IPartitionSerializer`** | Runs work sequentially within one partition key and concurrently across different ones in the current process. This invocation serialization establishes no durable publish-order guarantee; [ADR-0046](decisions/0046-durable-event-delivery-and-recovery.md#producer-dispatch-and-recovery) defines eligible-selection order and its limits. Queuing work for the key you are already inside is refused rather than deadlocked: the caller that does it is publishing from inside a handler, which [Standards 20](standards/20-infrastructure-stack.md) forbids. | -| **`EventTenantContext`** | The `ITenantContext` a consumer runs under, rebuilt from the envelope by the transport before handler discovery or resolution. A consumer executes outside the request that produced the fact, so there is no ambient context to inherit. The effective principal is `UserId.SystemActor`; an envelope human remains available separately as `CausalActorUserId`. Restoring the tenant and optional organization is what makes query filters and RLS policies evaluate against the right scope. | +| **`EventTenantContext`** | The `ITenantContext` a consumer runs under, rebuilt from the validated envelope by the delivery executor before business-handler construction or DI resolution. A consumer executes outside the request that produced the fact, so there is no ambient context to inherit. The effective principal is `UserId.SystemActor`; an envelope human remains available separately as `CausalActorUserId`. Restoring the tenant and optional organization is what makes query filters and RLS policies evaluate against the right scope. | | **`TenantId`** | The strongly-typed tenant identifier, in `LearnStack.SharedKernel.Identifiers` per [ADR-0023 Amendment 2](decisions/0023-strongly-typed-id-source-generator.md)'s cross-cutting placement rule — it rides on `ITenantContext`, on every `[TenantOwned]` entity, in cache keys, job payloads and event envelopes, so a Tenancy-owned type would make each of those a reference to that module. It has **no `New()`**: a tenant id is assigned by the registry that owns the `Tenant` aggregate, because a handler minting its own could not satisfy the self-keyed policy's `WITH CHECK` ([Database Standards § Table classes](standards/05-database.md)). | | **`TenantId.PlatformSentinel`** | The reserved platform sentinel tenant id `00000000-0000-7000-8000-000000000002` — a UUIDv7-shaped constant on the precedent `UserId.SystemActor` sets, and **not** the nil UUID, which three shipped mechanisms read as *the absence of a tenant*. `tenants` carries a `CHECK` refusing it, so no tenant is ever provisioned under it and its rows are invisible to every tenant policy. Exactly one class of row carries it: a platform-scope operation with no resolvable tenant, written standalone — entry into `EnterPlatformAdminScope(reason)` and the operations inside it, of which `tenancy.killswitch.toggle` is the first — a row the Tenancy matrix carries as `(planned)`, because Packet 9 ships the killswitch table with no writer and [Phase 03](roadmap/phase-03-identity-admin.md) owns the toggle command. Never announced by `SetTenantContextAsync`, never written by a tenant request path, and never carrying a rejected tenant assertion — and that invariant has an **enforcer** rather than a hope: `SetProvisioningTenantContextAsync`, the one announcement path that takes a caller-supplied id, refuses the sentinel exactly as it already refuses `Guid.Empty`, and `Tenant.Create` refuses it in the factory. The `CHECK` is the **backstop, not the control** — a constraint on `tenants` cannot stop a GUC from being announced. Per [ADR-0044 § 1 and Amendment 3](decisions/0044-audit-write-path.md). | | **`OrganizationId`** | The strongly-typed identifier for a sub-unit within a tenant ([ADR-0017](decisions/0017-tenant-organization-hierarchy.md)), cross-cutting for the same reason as [`TenantId`]. Nullable at almost every use site, and the null is a **scope** rather than "unknown": a tenant-owned row with no organization is tenant-wide, which is why the canonical policy reads `organization_id IS NULL OR organization_id = …`. Modules hold it by value and read organization data through an application contract; the `Organization` aggregate lives only in `LearnStack.Modules.Tenancy.Domain`. | @@ -341,15 +342,18 @@ These entries distinguish P02d-2 contracts from P02d-3 internal read contracts: | **`IFeatureFlags`** | The **only** module-facing entitlement read: `IsEnabledAsync(FeatureKey)` and `GetLimitAsync(LimitKey)` returning `long`. It resolves the tenant from `ITenantContext` and throws when there is none, so module code can never name another tenant. It **composes** rather than queries: a plan-projected key resolves through `IEntitlementProvider.GetAsync` — not by reading `platform_entitlement_cache` — a tenant-flag key reads `tenant_feature_flags`, and the killswitch overlay is applied last and wins over both. Its implementation lives in `LearnStack.Modules.Tenancy.Infrastructure`, because `tenant_feature_flags` is that module's table, and is registered at the composition root. `IFeatureFlagService` and `IUsageLimitService` are **withdrawn**, not renamed ([ADR-0021](decisions/0021-feature-based-entitlement.md)). Per [ADR-0045 § 2](decisions/0045-entitlement-and-feature-flag-socket.md). | | **`IHostToTenantResolver` / `HostResolution`** | Interface for host → `(tenant_id, organization_id?)` resolution, returning the `HostResolution(TenantId, OrganizationId?)` record or null. Reads `platform_host_to_tenant` and **nothing else** — never the Hub, because an anonymous page load must not depend on a control plane being reachable ([ADR-0034](decisions/0034-hub-contract-surface-invariant.md)). | | **APISIX** | The gateway in standalone YAML-reload mode per [ADR-0015](decisions/0015-api-gateway-apisix.md). The intended tenant-facing ingress, demand-gated to [Phase 11](roadmap/phase-11-production-hardening.md); until then ASP.NET middleware carries the same responsibilities in-process. `/api/internal/*` is never proxied by it — that listener is mTLS-only inside the pod. | -| **`OutboxProcessor`** | The API-host BackgroundService that claims one admitted outbox row immediately before invocation, with a fresh acquisition token in `locked_by`, a database-time lease and one counted attempt. Token/expiry-fenced transitions own retry and explicit audited terminal state; stable worker identity is diagnostic only. Accepted [ADR-0046](decisions/0046-durable-event-delivery-and-recovery.md) governs P02b-2 implementation: Steps 1–2 completed both review rounds, and Step 3 implements hosted execution/observation with both independent review rounds complete. Step 4 enforcement and local closeout are complete; all four steps completed both independent review rounds. The [Step 4 record](roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) owns CI/merge status; at-least-once recovery still needs P02b-3 inbox deduplication. | +| **`OutboxProcessor`** | The API-host BackgroundService that claims one admitted outbox row immediately before invocation, with a fresh acquisition token in `locked_by`, a database-time lease and one counted attempt. Token/expiry-fenced transitions own retry and explicit audited terminal state; stable worker identity is diagnostic only. Accepted [ADR-0046](decisions/0046-durable-event-delivery-and-recovery.md) governs P02b-2 implementation: Steps 1–2 completed both review rounds, and Step 3 implements hosted execution/observation with both independent review rounds complete. Step 4 enforcement and local closeout are complete; all four steps completed both independent review rounds. P02b-2 is complete and merged; its [merge closeout](roadmap/phase-02b-events-auth.md#p02b-2-merge-and-closeout-2026-10-13) owns final verification. P02b-3 delivers the independent shared inbox and subscriber runtime for at-least-once recovery; its [Step 6 record](roadmap/phase-02b-events-auth.md#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) and [PR handoff](roadmap/phase-02b-events-auth.md#p02b-3-pr-handoff-2026-10-13) own implementation review and CI status. | | **`IUnitOfWork`** | The ambient unit of work: **one database connection per scope**, and the transaction on it ([ADR-0040](decisions/0040-ambient-unit-of-work.md)). Every module `DbContext` resolved in the scope is built on that connection and enlisted in that transaction, and `IAuditStore` and `IOutbox` reach it through the same seam. The connection count is a correctness property, not a performance one: `SET LOCAL app.tenant_id` is connection- **and** transaction-local, so a context on its own connection reads zero rows from every tenant-owned table — silently. It exposes no `SaveChangesAsync`; contexts save themselves and it owns only the transaction boundary. [ADR-0040 Amendment 8](decisions/0040-ambient-unit-of-work.md#amendment-8--read-only-public-frames-2026-10-03) accepts explicit read-only/read-write intent, defaulting to writable: same-mode joins preserve ownership, mixed-mode joins poison the unit, and successful read-only completion permits a later writer. P02d-4 Step 1 delivers these modes and lifecycle controls; poison still requires a fresh scope. | -| **`IInboxGuard`** | The Accepted shared, transport-owned inbox claim/completion seam, pending P02b-3. Deduplication is `(tenant_id, subscription_id, event_id)`, claimed before handler construction on the announced app-role transaction. Business state, completed inbox, produced outbox, MUST audit and fenced ledger completion commit together; rollback removes a new claim. Handlers do not call the guard, and targeted replay never resets a completed inbox. See [ADR-0046](decisions/0046-durable-event-delivery-and-recovery.md#subscriber-acceptance-and-inbox-ownership). | -| **Inbox** | Shared tenant-owned completion record for one stable subscription/event pair. A new claim is uncommitted business-transaction work; only successful completion persists it. The delivery ledger owns failed/terminal/replay state. Canonical DDL remains P02b-3 in [Database Standards](standards/05-database.md#inbox). | +| **`IInboxGuard`** | The shared, transport-owned inbox claim seam, implemented as a transaction-bound primitive in P02b-3 Step 2 and integrated with the business owner in Step 3. Deduplication is `(tenant_id, subscription_id, event_id)`, claimed before handler construction on the announced app-role transaction. Business state, completed inbox, produced outbox, MUST audit and fenced ledger completion commit together; rollback removes a new claim. Handlers do not call the guard, and targeted replay never resets a completed inbox. See [ADR-0046](decisions/0046-durable-event-delivery-and-recovery.md#subscriber-acceptance-and-inbox-ownership). | +| **Inbox** | Shared tenant-owned completion record for one stable subscription/event pair. A new claim is uncommitted business-transaction work; only successful completion persists it. The delivery ledger owns failed/terminal/replay state. P02b-3 Step 1 implements its canonical DDL in [Database Standards](standards/05-database.md#inbox). | +| **Delivery ledger** | Shared `event_deliveries` storage retaining one validated envelope copy per `(tenant_id, subscription_id, event_id)`, independent of the producer row. It owns attempts, leases, completion and terminal/replay state; shared inbox identity owns deduplication. P02b-3 Steps 1–4 implement storage, acceptance, execution and scheduling. See [Database Standards](standards/05-database.md#inbox). | +| **`DeliveryProcessor`** | API-host subscriber worker implemented in P02b-3 Step 4. The confined scheduler claims one exact delivery tuple; a separate app-role executor owns business execution. Retries spend only that subscription's budget, terminal state commits with row-bound MUST audit, and internal replay resets one eligible terminal tuple without broadcast or inbox reset. Controlled tasks retain callbacks/scope cleanup within bounded capacity; the `delivery` health check is registered and unmapped. Step 5 activates both real defaults; the [Step 6 record](roadmap/phase-02b-events-auth.md#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) and [external follow-ups](roadmap/phase-02b-events-auth.md#p02b-3-second-external-review-follow-up-2026-10-15) own validation/review status; the [PR handoff](roadmap/phase-02b-events-auth.md#p02b-3-pr-handoff-2026-10-13) owns CI status. | +| **Built-in customization initialization** | Create-only installation of the platform's `card@1` content type and `plain@1` taxonomy, each independently owned by Customization. Any existing concept history is preserved; new IDs are owner-minted, while Seeder verifies matching built-ins semantically and retains warm IDs. P02b-3 Step 5 shares the initializer commands between the two real subscriptions and Seeder. See [Customization's flow](modules/customization/README.md#primary-integration-event-flow-provisioning-defaults) and its [delivery record](roadmap/phase-02b-events-auth.md#p02b-3-step-5--useful-defaults-and-semantic-seed-handoff-2026-10-13). | | **Integration-event subscription** | One explicitly identified consumer reaction, independent of handler class or assembly. Its stable `{module}.{subscription}.v{N}` ID versions deduplication semantics, not event schema or built-in revision. Renaming needs a migration/replay decision; a handler refactor preserves identity. [ADR-0046](decisions/0046-durable-event-delivery-and-recovery.md#subscriber-acceptance-and-inbox-ownership) owns the exact grammar. | | **Subscription readiness** | Construction-free Pending, Active or IntentionallyUnsubscribed declaration in `OutboxEventRegistry`, the closed resolver of declared concrete contracts, compatible aliases and retired identities. Any required Pending subscription holds its contract/aliases before acquisition and validation. Active requires a real registered handler; intentional absence is explicit, never inferred from zero discovered handlers. Accepted P02b-2 policy under [ADR-0046](decisions/0046-durable-event-delivery-and-recovery.md). | | **Outbox lease** | Paired `locked_by` / `locked_until` fields; the former holds a fresh canonical UUID acquisition token, the latter a database-time expiry. Every transition checks both current token and unexpired lease. Recovery replaces expired ownership before another transition. Stable worker identity confers no ownership. | -| **Dead letter** | Explicit completed terminal state after invalid data or exhausted attempts, distinct from a terminal request or delayed availability. On the producer row, `terminal_reason` requests terminalization and excludes normal publication; only `dead_lettered_at` records completed terminal state. Producer terminal state retains its envelope and commits with MUST audit; subscriber terminal state belongs to its independent delivery ledger and never completes its inbox. Operator reset/replay is separately owned by Phase 03. See [ADR-0046](decisions/0046-durable-event-delivery-and-recovery.md#producer-dispatch-and-recovery). | -| **`EventTerminalTransition`** | Restricted immutable witness minted only from the successful fenced terminal UPDATE result in its live transaction. It binds actual row identity, acquisition and terminal values to `WriteEventTerminalAsync`; substituting draft tenant/org/event or transaction requires rollback of state and audit. Fifth seam implemented in P02b-2 Step 2; both independent review rounds complete. | +| **Dead letter** | Explicit completed terminal state after invalid data or exhausted attempts, distinct from a terminal request or delayed availability. On the producer row, `terminal_reason` requests terminalization and excludes normal publication; only `dead_lettered_at` records completed terminal state. Producer terminal state retains its envelope and commits with MUST audit; subscriber terminal state belongs to its independent delivery ledger and never completes its inbox. P02b-3 Step 4 implements targeted internal subscriber replay; operator routes/permissions and producer reset/replay are separately owned by Phase 03. See [ADR-0046](decisions/0046-durable-event-delivery-and-recovery.md#producer-dispatch-and-recovery). | +| **`EventTerminalTransition`** | Restricted immutable witness minted only from the successful fenced terminal UPDATE result in its live transaction. It binds actual row identity, acquisition and terminal values to `WriteEventTerminalAsync`; substituting draft tenant/org/event or transaction requires rollback of state and audit. Producer fifth seam implemented in P02b-2 Step 2, with both independent review rounds complete; P02b-3 Step 4 adds the subscriber tuple witness and terminal writer using a validated attempt trace without causal-envelope/payload access. | | **`DeploymentMode`** | Enum (`Development \| SaaS \| Dedicated \| SelfHostedOnline \| SelfHostedAirGapped`) read at the composition root to select provider implementations. Modules never read this enum. | diff --git a/docs/modules/audit/README.md b/docs/modules/audit/README.md index 141e184c..f58e116a 100644 --- a/docs/modules/audit/README.md +++ b/docs/modules/audit/README.md @@ -35,7 +35,7 @@ for the confined outbox role and validates the audit draft against a restricted witness from the actual fenced terminal UPDATE on the caller's transaction. Terminal state and MUST audit are atomic; this is not a standalone or platform- scope write. Tenancy owns the producer/subscriber operation declarations in -[its matrix](../tenancy/audit.md), with subscriber implementation in P02b-3. +[its matrix](../tenancy/audit.md); P02b-3 Step 4 implements the subscriber writer. The third module spec in the repository, per [Documentation Standards § Per-Module Specifications](../../standards/13-documentation.md). @@ -353,8 +353,9 @@ uses completed provisioning to initialize Customization defaults; Audit is not its mirror consumer. Existing source MUST rows remain on the source transaction. The reusable execution coordinator supplies non-MediatR classification/entry, owner write/boundary reporting and outermost reconciliation under ADR-0044/0033. -P02b-1 proves a controlled direct owner; P02b-3 owns production delivery and its -per-effect audit under ADR-0046's accepted G1 contract. Business audit retains +P02b-1 proves a controlled direct owner; P02b-3 Steps 3–5 implement production +delivery and per-effect audit under ADR-0046's accepted G1 contract. Business audit +retains the existing coordinator and durability model; explicit terminal audit uses the separate fifth seam described above. diff --git a/docs/modules/audit/audit.md b/docs/modules/audit/audit.md index 78607d8a..c39cf94b 100644 --- a/docs/modules/audit/audit.md +++ b/docs/modules/audit/audit.md @@ -7,8 +7,9 @@ Accepted [ADR-0046](../../decisions/0046-durable-event-delivery-and-recovery.md) adds the confined terminal-audit seam in P02b-2, not an Audit-module operation. `platform.outbox.dead_letter` and P02b-3's `platform.delivery.dead_letter` belong to [Tenancy's off-path matrix](../tenancy/audit.md). P02b-2 Step 2 implements -the producer writer, with both review rounds complete; the subscriber writer -remains P02b-3. Phase 03 owns privileged erasure execution and its bounded +the producer writer, with both review rounds complete; P02b-3 Step 4 implements +the subscriber writer with both review rounds complete. Phase 03 owns privileged +erasure execution and its bounded caller contract; Phase 11 owns retention purge. **The module that owns the log is audited like any other.** Reading someone's audit diff --git a/docs/modules/customization/README.md b/docs/modules/customization/README.md index bd1a6724..b897bf3f 100644 --- a/docs/modules/customization/README.md +++ b/docs/modules/customization/README.md @@ -177,11 +177,12 @@ Contextual Off queries check the logical key's Active identity before registrati Convergent seed also passes true: a different Active revision is refused before mutation as `business_rule_violation` / `lockey_customization_key_already_live`. Default false retains ordinary revision succession. The same-row race remains typed concurrency; -an Active winner from another revision is never retired or adopted by seed. +an Active winner from another revision is never retired or adopted by authored seed. +Built-ins use the separate semantic initialization path described below. ### Primary integration-event flow: provisioning defaults -**Accepted ADR-0046/0057 flow; runtime subscriptions pending P02b-3.** +**Implemented by P02b-3 Step 5; validation/review status is recorded below.** [ADR-0057](../../decisions/0057-transactional-event-production-and-completion.md#first-useful-production-flow) authorizes independent `card@1` content-type and `plain@1` taxonomy initialization on completed tenant @@ -293,26 +294,32 @@ keeps the library's types inside it. ## Integration-event catalogue -**No publisher or subscriber implemented yet.** Accepted ADR-0057 fixes the -useful effects; ADR-0046 fixes the stable identities and shared transport-owned -inbox. P02b-2 manifests both required subscriptions Pending, holding real rows and -aliases before acquisition/validation; P02b-3 activates both with durable -acceptance. Each effect owns one root, generation change, inbox outcome -and MUST audit. Customization mints UUIDv7 IDs and resolves semantic key/version +Customization publishes no event and implements two subscribers. ADR-0057 fixes +the useful effects; ADR-0046 fixes their stable identities and transport-owned +inbox. P02b-3 Step 5 activates both real subscriptions together over durable +acceptance and the semantic Seeder handoff. Each effect owns one root, generation +change, inbox outcome and MUST audit. Customization mints UUIDv7 IDs and resolves +semantic key/version identity; Tenancy carries neither schemas nor Customization IDs. | Consumed topic | Event | Stable subscription | Independent effect | Owner | |---|---|---|---|---| -| `learnstack.tenancy.tenant` | `TenantCreatedIntegrationEventV1` | `customization.initialize-content-type-default.v1` | Create untouched `card@1` content type (`BuiltInCustomizations.Card`) | P02b-3; Pending until then | -| `learnstack.tenancy.tenant` | `TenantCreatedIntegrationEventV1` | `customization.initialize-taxonomy-default.v1` | Create untouched `plain@1` taxonomy (`BuiltInCustomizations.Plain`) | P02b-3; Pending until then | +| `learnstack.tenancy.tenant` | `TenantCreatedIntegrationEventV1` | `customization.initialize-content-type-default.v1` | Create untouched `card@1` content type (`BuiltInCustomizations.Card`) | P02b-3 Step 5; Active | +| `learnstack.tenancy.tenant` | `TenantCreatedIntegrationEventV1` | `customization.initialize-taxonomy-default.v1` | Create untouched `plain@1` taxonomy (`BuiltInCustomizations.Plain`) | P02b-3 Step 5; Active | Initialization preserves Draft, Active, Deprecated, successor and deliberately -deleted definitions; it never overwrites, republishes or resurrects them. P02b-3 -switches Seeder built-in verification to owner-resolved key/version, preserves -warm IDs and exact tenant-authored curriculum IDs, and proves concurrent fresh -seed/delivery convergence and sibling recovery. It updates the implementation's -creator/Seeder/Hub installation remarks when delivery exists. P02b-1 keeps -current seed writes and pending outbox rows. +deleted definitions; it never overwrites, republishes or resurrects them. Each +parameterless initializer takes its tenant from trusted context, acquires the +transaction-owned semantic lock shared by ordinary registration, then checks all +revision/status/deletion history under RLS. A new root is validated and Active +before one save, generation bump and MUST audit; a preserved attempt changes no +root or generation. The coordinating handlers are explicitly Off and send one +classified nested command, refusing a failed result rather than completing inbox. + +Seeder discovers built-ins by key/version and verifies their exact Active shape, +retaining warm/generated IDs. Authored curriculum identities/pins stay exact. +The [Step 5 record](../../roadmap/phase-02b-events-auth.md#p02b-3-step-5--useful-defaults-and-semantic-seed-handoff-2026-10-13) +owns initialization, race and Seeder validation/review status. No other module reacts to a customization change through events today. Public readers use the delivered generation-keyed cache with fresh generation probes; a cache diff --git a/docs/modules/customization/audit.md b/docs/modules/customization/audit.md index fd3989ae..beb8eb7c 100644 --- a/docs/modules/customization/audit.md +++ b/docs/modules/customization/audit.md @@ -14,13 +14,13 @@ MediatR request or audited write. It creates no intent or business-state mutatio Any production request introduced for it must be audit Off; test-only requests stay test-only. [The module contract](README.md#primary-read-flow-resolving-a-tenants-shapes) owns the read. -Four of the operations below now exist, all written by Phase 02a Packet 8's -handlers: `ContentType` register and publish, and `LevelTaxonomy` register and -publish. Eight more rows are classification ahead of code and carry `(planned)` in the +Six operations below exist: Phase 02a Packet 8's `ContentType` and +`LevelTaxonomy` register/publish, plus P02b-3's separate default initializers. +Eight more rows are classification ahead of code and carry `(planned)` in the `Operation` column ([ADR-0044 Amendment 3](../../decisions/0044-audit-write-path.md)); the last row carries no slug at all, and its own cell says why. -**All four write MUST rows**, and publication is the one that matters most: it +**All six write MUST rows**, and publication is the one that matters most: it retires the incumbent and makes every subsequent write of that content type validate against a new shape, and the row is what records who changed what a tenant's data is allowed to look like. `CustomizationAuditCatalogSource` declares the same operations in @@ -35,9 +35,11 @@ is total: every entry this module's `IAuditCatalogSource` registers has a row he carrying the same slug, and one that does not fails. Matrix → catalogue binds only to a slug whose request type **exists**, so the eight `(planned)` rows are classification and not drift — and a `(planned)` row whose command has since shipped fails, which is what -keeps the marker from becoming a hole rather than a claim. Nothing in this module is -written off the request path, so no row here carries the `(off-path)` marker the -[Tenancy matrix](../tenancy/audit.md) needs for five of its own. +keeps the marker from becoming a hole rather than a claim. Both integration +handlers coordinate classified nested commands and are explicitly +Off; the commands own their business audit. Nothing in this module writes audit SQL +outside that path, so no row here carries the `(off-path)` marker the +[Tenancy matrix](../tenancy/audit.md) owns for its confined terminal writers. **Why lighting up the pipeline changed no handler's transaction:** every one runs inside the ambient transaction ([ADR-0040](../../decisions/0040-ambient-unit-of-work.md)), @@ -107,11 +109,13 @@ audited operation at all. | Resource | Operation | Class | Why | |---|---|---|---| +| `ContentType` | `customization.content_type.initialize` | **MUST** | Creates one untouched Active built-in; any concept history is preserved as an explicitly classified no-change attempt | | `ContentType` | `customization.content_type.register` | **MUST** | A tenant declaring a new shape for its own data; the row is what a later "who added this?" reads | | `ContentType` | `customization.content_type.publish` | **MUST** | Retires the incumbent and changes what every subsequent content write is validated against — the highest-blast-radius act in the module | | `ContentType` | `customization.content_type.revise` `(planned)` | **MUST** | Only a draft's body is mutable, but "additive" is claimed by the editor and not proved by the aggregate; the row is what makes a wrong claim traceable | | `ContentType` | `customization.content_type.rename` `(planned)` | **MUST** | The baseline lists these aggregates' **created / updated / deleted**, and a rename is an update. It is presentational, which changes the payload and not the class: the display name is what every Studio list, every editor and every renderer shows for the shape | | `ContentType` | `customization.content_type.soft_delete` `(planned)` | **MUST** | Frees the key for a successor, because the one-live-revision index is partial on `deleted_at IS NULL`; a retired definition releasing its name is the same class of act as a domain releasing its host | +| `LevelTaxonomy` | `customization.level_taxonomy.initialize` | **MUST** | Creates one untouched Active built-in and its bands; preserves every existing revision or tombstone without a generation change | | `LevelTaxonomy` | `customization.level_taxonomy.register` | **MUST** | Same reason as the content type's registration: a tenant declaring a vocabulary of its own, and the row is what a later "who added this?" reads | | `LevelTaxonomy` | `customization.level_taxonomy.publish` | **MUST** | Same reason as the content type's publication, and with a wider reach: every level reference in the tenant resolves through the vocabulary this retires and replaces | | `LevelTaxonomy` | `customization.level_taxonomy.add_band` `(planned)` | **MUST** | Extends the vocabulary every level reference resolves against, and the aggregate permits it while the revision is a draft | @@ -119,7 +123,7 @@ audited operation at all. | `LevelTaxonomy` | `customization.level_taxonomy.rename` `(planned)` | **MUST** | Same reason as the content type's rename: the baseline lists these aggregates' **created / updated / deleted**, and a rename is an update | | `LevelTaxonomy` | `customization.level_taxonomy.rename_band` `(planned)` | **MUST** | A band's label is what a learner sees where a level is named, so renaming one is an update of the taxonomy and not of its presentation | | `LevelTaxonomy` | `customization.level_taxonomy.soft_delete` `(planned)` | **MUST** | Same reason as the content type's | -| `customization_generations` | – | – | **The bump is deliberately unaudited.** It is not an aggregate ([ADR-0043 § 7](../../decisions/0043-customization-payload-validation.md)), it carries no decision, and it is written exactly once per audited operation above — a row for it would be a second entry for the same act, in the same transaction, saying less. Nor does the broad capture predicate pick it up: `CustomizationGenerationStore` bumps the row with one `ON CONFLICT … DO UPDATE` statement on the ambient connection, so it is never a `ChangeTracker` entry for `AuditChangeTrackerInterceptor` to see | +| `customization_generations` | – | – | **The bump is deliberately unaudited.** It is not an aggregate ([ADR-0043 § 7](../../decisions/0043-customization-payload-validation.md)), it carries no decision, and it advances once per state-changing write above; preserved initialization changes no generation — a row for it would be a second entry for the same act, in the same transaction, saying less. Nor does the broad capture predicate pick it up: `CustomizationGenerationStore` bumps the row with one `ON CONFLICT … DO UPDATE` statement on the ambient connection, so it is never a `ChangeTracker` entry for `AuditChangeTrackerInterceptor` to see | **A refused schema is not an operation of its own.** A document the four gates reject never becomes state: no definition row is written, no generation is bumped, @@ -133,7 +137,8 @@ decision with a different owner: on `security-event` and says nothing about validation failures, and a tenant fixing its own draft in a loop would be the noisiest writer in the system. -The four unmarked rows are written by the pipeline today; the eight `(planned)` rows +The six unmarked rows are written through the classified pipeline; the eight +`(planned)` rows register with the commands that raise them. ## P02d-2 contextual verification reads @@ -143,3 +148,15 @@ register with the commands that raise them. are explicitly Off in `CustomizationAuditCatalogSource`. These unrouted queries verify announced tenant state through the ambient transaction and declare no write operation. + +## P02b-3 default initialization + +The two coordinating integration handlers are explicitly Off; their nested +`InitializeTenantContentTypeDefaultCommand` and +`InitializeTenantLevelTaxonomyDefaultCommand` own one MUST/Create intent each. +A new root is Active at its single save and advances generation once. A preserved +concept has no entity change or generation bump, but its initialization attempt +remains classified. The transport skips construction for a durable inbox duplicate, +so that duplicate adds no business audit. Completed Seeder acts skip the command +before invocation. The [Step 5 record](../../roadmap/phase-02b-events-auth.md#p02b-3-step-5--useful-defaults-and-semantic-seed-handoff-2026-10-13) +owns validation and review status. diff --git a/docs/modules/customization/permissions.md b/docs/modules/customization/permissions.md index 79677bf0..40c9849a 100644 --- a/docs/modules/customization/permissions.md +++ b/docs/modules/customization/permissions.md @@ -58,16 +58,19 @@ same operation as MUST-audit; a matrix with no `delete` disagreed with both. Publication has none, because unpublishing is not deletion: the successor's publish is what retires a revision. -**Four handlers exist and all are deliberately unauthorized.** Reachability -stands in for authorization exactly as it does in -[Tenancy](../tenancy/permissions.md), and for the same two reasons: none has an -HTTP endpoint, so the only callers are `LearnStack.Tools.Seeder` and — from -[Phase 02c](../../roadmap/phase-02c-hub-foundation.md) — the Hub over -`/api/internal/*`; and every one takes its tenant from the context and never from -the request, so a caller cannot name another tenant even without a permission -check. The database refuses the write, and -`Write_With_Foreign_TenantId_Is_Rejected_By_WithCheck` is what proves it rather -than this paragraph. +**Six write commands are unrouted.** The current route set exposes none of +them, and permission enforcement remains Phase 03. The four authoring commands +are called by the Seeder. The two default-initialization commands are shared by +the Seeder and the transport-owned Customization subscriptions; they take tenant +authority only from the restored trusted context. Hub provisioning emits the +completed-tenant event and does not directly invoke a Customization HTTP writer. +The planned Studio editors below own human-facing authorization. + +Every command takes its tenant from context rather than request input. RLS and +`Write_With_Foreign_TenantId_Is_Rejected_By_WithCheck` protect the database write; +reachability is the current authorization boundary, not a new permission grant. +The [Step 5 record](../../roadmap/phase-02b-events-auth.md#p02b-3-step-5--useful-defaults-and-semantic-seed-handoff-2026-10-13) +owns implementation validation/review status. **Every key in the `{module}.{resource}.{action}` form.** The resources above are written PascalCase as the template writes them; the keys they generate are diff --git a/docs/modules/tenancy/README.md b/docs/modules/tenancy/README.md index 2c520ae9..bf540fd5 100644 --- a/docs/modules/tenancy/README.md +++ b/docs/modules/tenancy/README.md @@ -496,20 +496,21 @@ announcement while provisioning ambient context remains unresolved. State, event and the existing two MUST audit rows share one transaction. P02b-1 persists pending rows; P02b-2 implements hosted dispatch and production-census enforcement. All four steps completed both independent review rounds and local closeout. -The [Step 4 record](../../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) -owns CI/merge status. P02b-3 owns consumer delivery. - -The subscriptions below are accepted flow contracts, not delivered handlers. -[ADR-0046](../../decisions/0046-durable-event-delivery-and-recovery.md) accepts G1's -shared transport-owned inbox and separate delivery ledger. P02b-2 holds this -contract Pending, including aliases, before claim or payload validation. P02b-3 -activates `customization.initialize-content-type-default.v1` for the `card@1` -content type and `customization.initialize-taxonomy-default.v1` for the `plain@1` -taxonomy together with durable acceptance; there are no placeholder handlers. +The [merge closeout](../../roadmap/phase-02b-events-auth.md#p02b-2-merge-and-closeout-2026-10-13) +owns CI/merge status. P02b-3 Steps 1–4 implement consumer ownership and scheduling; +Step 5 supplies the two actual default subscriptions and semantic Seeder handoff. + +P02b-3 Step 5 activates both real Customization subscriptions together over +[ADR-0046](../../decisions/0046-durable-event-delivery-and-recovery.md)'s shared +transport-owned inbox and independent delivery ledger. The stable identities are +`customization.initialize-content-type-default.v1` for `card@1` content type and +`customization.initialize-taxonomy-default.v1` for `plain@1` taxonomy. The +[Step 5 record](../../roadmap/phase-02b-events-auth.md#p02b-3-step-5--useful-defaults-and-semantic-seed-handoff-2026-10-13) +owns validation/review status; neither subscription is a placeholder. | Topic | Payload | Publisher | Consumers | Phase | |---|---|---|---|---| -| `learnstack.tenancy.tenant` | `TenantCreatedIntegrationEventV1`, completed provisioning | Tenancy | Independent Customization `card@1` and `plain@1` initializers | P02b-1 publisher; P02b-3 subscribers after G1 | +| `learnstack.tenancy.tenant` | `TenantCreatedIntegrationEventV1`, completed provisioning | Tenancy | Independent Customization `card@1` and `plain@1` initializers | P02b-1 publisher; P02b-3 Step 5 subscribers | | `learnstack.hub.entitlement` | entitlement projection refreshed | Hub adapter | entitlement cache readers | 02c | | `learnstack.hub.custom-domain.activated` / `.deactivated` | host mapping changed | Hub adapter | `CachedHostToTenantResolver` | 02c | @@ -517,9 +518,9 @@ Initialization is create-only for untouched built-in concepts; Draft, Active, Deprecated, successor and deliberately deleted definitions are preserved. Each P02b-3 effect owns one Customization root, generation change, inbox outcome and MUST audit on its delivery transaction. Customization owns UUIDv7 IDs and semantic -key/version identity. P02b-3 moves Seeder verification away from fixed built-in -IDs, preserves warm IDs and exact authored curriculum IDs, and proves fresh -seed/delivery convergence. P02b-1 keeps existing seed writes. Typed settings are +key/version identity. Seeder verifies exact Active built-in shape by semantic +identity, preserves warm IDs and keeps authored curriculum IDs/pins exact. +The shared initializers preserve any existing revision or tombstone. Typed settings are uncached; no settings-invalidation event or Audit-mirror consumer is promised. Future tenant-status/organization publishers need their owning feature decision. diff --git a/docs/modules/tenancy/audit.md b/docs/modules/tenancy/audit.md index 1d1100a8..55f9632e 100644 --- a/docs/modules/tenancy/audit.md +++ b/docs/modules/tenancy/audit.md @@ -10,14 +10,18 @@ aggregate. A failed MUST write rolls back tenant, organization and outbox togeth The [integration-event catalogue](README.md#integration-event-catalogue) owns the flow; P02b-2 implements hosted producer dispatch and production-census enforcement. All four steps completed both independent review rounds and local closeout. -The [Step 4 record](../../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) -owns CI/merge status. Subscribers remain P02b-3. +The [merge closeout](../../roadmap/phase-02b-events-auth.md#p02b-2-merge-and-closeout-2026-10-13) +owns CI/merge status. P02b-3 Steps 1–5 implement durable subscriber delivery and +both real defaults; its +[Step 6 record](../../roadmap/phase-02b-events-auth.md#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) +owns current enforcement and review status. **ADR-0046 producer terminal writer implemented — 2026-10-12; both review rounds complete.** The infrastructure operations below belong to this catalogue/matrix, without a synthetic aggregate. P02b-2 Step 2 registers the delivered producer terminal -operation; P02b-3 registers subscriber terminalization when its writer ships. +operation; P02b-3 Step 4 adds the subscriber writer and registration, with +implementation validation and independent reviews recorded in its delivery record. Both use the restricted fifth audit seam on the fenced state-transition transaction. Routine confined dispatch has the explicit ADR-0046 exemption; platform-scope entry auditing is unchanged. @@ -48,8 +52,9 @@ separate allowlist. Verification request types are Off. Packet 7 ships four audited root writes through three commands: Tenant and default Organization provisioning, a separate Organization create, and host mapping creation. P02d-2 adds two locale commands (SHOULD over Tenant) and one branding command (MUST over -TenantSetting). P02b-2 Step 2 adds the producer terminal MUST security-event writer. -Fourteen rows remain `(planned)`; seven operations have no MediatR request and +TenantSetting). P02b-2 Step 2 adds the producer terminal MUST security-event writer; +P02b-3 Step 4 adds its independent subscriber terminal counterpart. +Thirteen rows remain `(planned)`; seven operations have no MediatR request and carry `(off-path)`. All four original writes are **MUST**. Their records commit with the business state, @@ -127,7 +132,7 @@ unmarked row is one a shipped command already raises. | Resource | Operation | Class | Why | |---|---|---|---| | `outbox_messages` | `platform.outbox.dead_letter` `(off-path)` | **MUST** (`security-event`) | P02b-2 Step 2's consequential BYPASSRLS terminal transition; state and row commit together through the row-bound fifth audit seam, with stored tenant/org/event identity and SystemActor ([ADR-0046](../../decisions/0046-durable-event-delivery-and-recovery.md)) | -| delivery ledger | `platform.delivery.dead_letter` `(off-path)` `(planned)` | **MUST** (`security-event`) | P02b-3 owns subscriber terminalization, stable subscription/event identity and the same atomic transition/audit guarantee; no writer or operation registration ships at acceptance | +| delivery ledger | `platform.delivery.dead_letter` `(off-path)` | **MUST** (`security-event`) | P02b-3 Step 4's subscriber terminal writer binds stored tenant/org/event/subscription, acquisition token, reason and attempts to the actual fenced transition; ledger and MUST audit commit together through the same restricted fifth seam. It receives no causal-envelope or payload SELECT grant | | `Tenant` | `tenancy.tenant.create` | **MUST** | The root of a customer's data; its existence is a contractual fact | | `Tenant` | `tenancy.tenant.change_status` `(planned)` | **MUST** | Suspension withdraws access; an operator must be able to say who and when | | `Tenant` | `tenancy.tenant.rename` `(planned)` | SHOULD | Presentational | @@ -184,13 +189,13 @@ whose refreshing implementation lands in Phase 02c; the two ([ADR-0033 Amendment 2](../../decisions/0033-audit-durability-model.md)); `platform.outbox.dead_letter`, written with its fenced producer terminal UPDATE through `IAuditStore.WriteEventTerminalAsync`; and `platform.delivery.dead_letter`, -whose subscriber writer belongs to P02b-3. Four are +whose subscriber writer is delivered by P02b-3 Step 4. Five are registered by slug through `DeclareOffPath` because their writer ships — -`platform.admin_scope.enter`, `platform.outbox.dead_letter` and the two +`platform.admin_scope.enter`, `platform.outbox.dead_letter`, +`platform.delivery.dead_letter` and the two `tenancy.tenant_assertion.*` keys. -`tenancy.killswitch.toggle`, `tenancy.entitlement.refresh`, -and `platform.delivery.dead_letter` also carry -`(planned)` and register by slug when Phase 03, Phase 02c and P02b-3, +`tenancy.killswitch.toggle` and `tenancy.entitlement.refresh` also carry +`(planned)` and register by slug when Phase 03 and Phase 02c, respectively, land their writers. Every other slug above is joined in both directions the day its command lands. diff --git a/docs/modules/tenancy/permissions.md b/docs/modules/tenancy/permissions.md index dff3c612..8130c4d2 100644 --- a/docs/modules/tenancy/permissions.md +++ b/docs/modules/tenancy/permissions.md @@ -9,8 +9,8 @@ permission, and inserts a pending outbox row under the provisioning announcement Existing command reachability is unchanged. The [integration-event catalogue](README.md#integration-event-catalogue) owns the flow. -**ADR-0046 P02b-2 implementation locally complete — 2026-10-12.** All four steps -completed both independent review rounds. The [Step 4 record](../../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) +**ADR-0046 P02b-2 complete and merged — 2026-10-13.** All four steps +completed both independent review rounds. The [merge closeout](../../roadmap/phase-02b-events-auth.md#p02b-2-merge-and-closeout-2026-10-13) owns CI/merge status. The API-owned dispatcher and terminal audit add no permission key or HTTP endpoint. P02b-3's diff --git a/docs/roadmap/README.md b/docs/roadmap/README.md index 1a7c5c1d..be239c3d 100644 --- a/docs/roadmap/README.md +++ b/docs/roadmap/README.md @@ -93,21 +93,31 @@ not deferred to the showcase phase. [merge closeout](phase-02d-walking-skeleton.md#p02d-7-merge-and-closeout-2026-10-11) owns final verification and the completed phase exit. - [Phase 02b: Events, Background Jobs, Identity, and Session](phase-02b-events-auth.md) - — **P02b-0 and P02b-1 complete; P02b-1 merged through PR #29 on 2026-10-12**. Its + — **P02b-0 through P02b-2 complete; P02b-2 merged through PR #30 on 2026-10-13**. Its [entry audit](phase-02b-events-auth.md#p02b-0-entry-verification-2026-10-12) verifies the inherited baseline; Accepted ADR-0057 and the [producer package](phase-02b-events-auth.md#p02b-1-decision-package-2026-10-12) authorize the implemented lifecycle, enqueue and provisioning publisher. The [P02b-1 merge closeout](phase-02b-events-auth.md#p02b-1-merge-and-closeout-2026-10-12) records final CI verification and the completed delivery/review history. - P02b-2's revised package and ADR-0046 are Accepted. Steps 1–2 completed both - review rounds; Step 3 implements hosted dispatch and observation, with - both independent review rounds complete in its - [delivery record](phase-02b-events-auth.md#p02b-2-step-3--hosted-lifecycle-and-observability-2026-10-12). - Step 4 enforcement and local closeout are complete; all four steps completed - both independent review rounds. CI/merge status lives in its - [delivery record](phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12); - consumer delivery remains P02b-3. + P02b-2 delivers hosted dispatch, fenced recovery, atomic terminal audit, + confined credentials and observation/enforcement under Accepted ADR-0046. + All four steps and correction bundles completed both independent review rounds; + its [merge closeout](phase-02b-events-auth.md#p02b-2-merge-and-closeout-2026-10-13) + owns final verification. P02b-3 Steps 1–5 implement durable acceptance, shared + inbox/ledger, independent subscriber retries, both real default subscriptions + and semantic Seeder handoff. Its + [Step 6 record](phase-02b-events-auth.md#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) + records completed enforcement/regression and both review rounds for all six + steps. The + [external follow-up](phase-02b-events-auth.md#p02b-3-external-review-follow-up-2026-10-15) + adds verified corrections and two further independent review rounds on the + same unmerged branch; the + [second follow-up](phase-02b-events-auth.md#p02b-3-second-external-review-follow-up-2026-10-15) + owns subsequent remediation status. PR review and merge remain pending. + Background jobs, API authentication, + realm reconciliation, BFF sessions + and phase-exit enforcement remain P02b-4–8. - [Phase 03: Identity Domain, Authorization, and Admin Foundation](phase-03-identity-admin.md) - [Phase 04: Headless CMS, Page Builder, and Media Library](phase-04-cms-media-pages.md) - [Phase 05: Education Catalog and Learning Content](phase-05-education-learning-content.md) diff --git a/docs/roadmap/phase-02b-events-auth.md b/docs/roadmap/phase-02b-events-auth.md index 34a58ade..7d781198 100644 --- a/docs/roadmap/phase-02b-events-auth.md +++ b/docs/roadmap/phase-02b-events-auth.md @@ -1,5 +1,19 @@ # Phase 02b: Events, Background Jobs, Identity, and Session +**Current status — 2026-10-15.** P02b-0 through P02b-2 are complete. +P02b-2 is merged through [PR #30](https://github.com/HodeTech/LearnStack/pull/30); +its [merge closeout](#p02b-2-merge-and-closeout-2026-10-13) owns final verification. +P02b-3's [decision package](#p02b-3-decision-package-2026-10-13) is Accepted; +All six steps completed implementation and both independent review rounds. +[Step 6](#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) owns +validation and PR/CI handoff; PR review and merge remain pending. +The [external review follow-up](#p02b-3-external-review-follow-up-2026-10-15) +records verified corrections and dispositions against the reviewed PR head. +The [second follow-up](#p02b-3-second-external-review-follow-up-2026-10-15) +owns the subsequent runtime and proof corrections. +P02b-4–8 remain unimplemented. +Phase 02b remains in progress. Earlier dated receipts retain their as-of state. + **P02b-0 complete — 2026-10-12.** Phase 02d is complete and merged through PR #28; its [merge closeout](phase-02d-walking-skeleton.md#p02d-7-merge-and-closeout-2026-10-11) records the verified exit. The @@ -21,7 +35,7 @@ production-census guards and catalogue enforcement; its backend verification and both completed review rounds. **P02b-1 is complete and merged** through [PR #29](https://github.com/HodeTech/LearnStack/pull/29) on 2026-10-12. The [merge closeout](#p02b-1-merge-and-closeout-2026-10-12) owns final -verification. The Accepted P02b-2 package below owns the current dispatcher work. +verification. The P02b-2 merge closeout below owns final dispatcher verification. **P02b-2 decision pass — 2026-10-12.** The [decision package](#p02b-2-decision-package-2026-10-12) and @@ -34,8 +48,9 @@ implements hosted lifecycle and observation; its [record](#p02b-2-step-3--hosted-lifecycle-and-observability-2026-10-12) owns verification and both completed review rounds. Step 4 enforcement and local closeout are complete; all four steps completed both independent review rounds. -The [PR handoff](#p02b-2-pr-handoff-2026-10-12) owns live CI/review/merge state. -P02b-3 implementation remains pending. +The [merge closeout](#p02b-2-merge-and-closeout-2026-10-13) owns final verification. +P02b-3 was pending at that decision pass; its current implementation and review +status is recorded in [Step 6](#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13). **Phase 02d handoff — 2026-10-11.** P02b-0 re-verifies the delivered `/api/v1` public contracts, native HTTPS on retained `*.learnstack.local` seed hosts, @@ -258,8 +273,8 @@ Accepted, and its catalogue rows registered, before its first line of code. |---|---|---| | **P02b-0** | Phase entry: every premise the register below cites re-verified against `HEAD` — the code, the corpus, the Hub repository — and the register corrected where Phase 02d moved it. It drafts no record: each gate closes in the decision pass of the first packet it shapes, and each glossary headword this phase needs — `inbox`, outbox lease, dead letter, audit frame, job activator, integration-event consumer — is written in the pass whose gate fixes its meaning | [Phase 02d](phase-02d-walking-skeleton.md) exits | | **P02b-1** | **Complete and merged — [PR #29 closeout](#p02b-1-merge-and-closeout-2026-10-12).** Producer: `IOutbox`, the `OutboxFlushBehavior` invariant guard, successful-save domain-event collection and owner-only completion, and the first event with its enqueue path | G2, G3 (the key's derivation), G5 (the stored identities), G6, G18 (the payload rule), G19 Accepted | -| **P02b-2** | **Decision Accepted; all four implementation steps and both independent review rounds complete; local closeout complete; [PR handoff](#p02b-2-pr-handoff-2026-10-12) owns CI/review/merge state.** Dispatcher: the lease migration and grant extension, claim / publish / mark, backoff, the producer terminal state, the credential and its boot guard, the metrics and the health check | G1, G3, G4, G5, G7, G8, G17, G18 (`last_error`) Accepted; P02b-1 | -| **P02b-3** | Consumer: durable acceptance, shared inbox/ledger DDL and `IInboxGuard`, the common per-delivery transaction, both real defaults subscriptions and Seeder handoff, independent attempts and audited terminal state, targeted internal ledger replay | G1, G6, G7 Accepted; P02b-2 | +| **P02b-2** | **Complete and merged — [PR #30 closeout](#p02b-2-merge-and-closeout-2026-10-13).** Dispatcher: the lease migration and grant extension, claim / publish / mark, backoff, the producer terminal state, the credential and its boot guard, the metrics and the health check | G1, G3, G4, G5, G7, G8, G17, G18 (`last_error`) Accepted; P02b-1 | +| **P02b-3** | **Implementation and both review rounds complete** ([Step 6](#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13)); PR review and merge pending. Consumer: durable acceptance, shared inbox/ledger DDL and `IInboxGuard`, the common per-delivery transaction, both real default subscriptions and Seeder handoff, independent attempts and audited terminal state, targeted internal ledger replay | G1, G6, G7 Accepted; P02b-2 | | **P02b-4** | Background jobs: the runner, its storage, the tenant-context hook, the enqueue guard, no job definition | G9 Accepted; P02b-3 | | **P02b-5** | API authentication: JwtBearer wiring and its insertion order, claim population, the host/JWT cross-check through the existing recorder, token-keyed rate limits, the 401 rows, one authenticated endpoint | G10, G11, G12 (the BFF's client), G14, G20 Accepted | | **P02b-6** | Realm and seed: the shared client scope, the mappers, the user-profile declaration, UUID attributes agreeing with `SeedData`, the warm-realm reconciler, the Keycloak test fixture | G12, G13 (the access-token lifetime's authority) Accepted; P02b-5 | @@ -287,25 +302,25 @@ premise a row cites is re-verified at that pass rather than trusted. | # | Question | Leaning | Vehicle | Blocks | Status | |---|---|---|---|---|---| -| G1 | What is the subscriber's fate when its handler exhausts retries, and what is the inbox table — one shared or per module, keyed `event_id` or `(tenant_id, consumer, event_id)`, claimed by the handler or by the transport? Three arms travel with it: the class of consumer whose effect cannot share the inbox transaction (an external call, the platform-scope erasure handler the Audit spec already carries) and what "effectively once" means for it; whether an event without the organization marker is delivered tenant-wide regardless of the producing request's organization; and whether the inbox entity joins [ADR-0044 § 7](../decisions/0044-audit-write-path.md)'s named audit-capture exclusion list once a delivery declares intents | One shared tenant-owned `inbox_messages` keyed by tenant/stable subscription/event, transport-claimed before construction; a separate delivery ledger owns retries, terminal state and targeted replay; completion shares the business/audit/outbox transaction | **ADR-0046** + a dated **ADR-0006 Amendment** (or an explicit supersession note in ADR-0046 naming Amendment 1's consumer-side paragraph, which fixes the per-module inbox) + a dated **ADR-0010 Amendment**, because its § Idempotency makes two things mandatory that the shared-table leaning changes — the inbox is "the module's **own** inbox table", and the guard is called by the consumer — + the `Integration_Event_Handlers_Use_InboxGuard` assertion re-written in the same record to name whoever ends up owning the call, since a transport-claimed delivery fails a structural rule that looks for the call inside the handler + Database Standards § Inbox + an ADR-0041 erratum beside ADR-0023's minting list | P02b-2 (the record and its amendments: the dispatcher's retry state and a failed delivery's next attempt are one state machine), P02b-3 (Database Standards § Inbox and the ADR-0023 erratum) | Accepted — 2026-10-12; decision/amendments delivered; subscriber implementation pending P02b-3 | +| G1 | What is the subscriber's fate when its handler exhausts retries, and what is the inbox table — one shared or per module, keyed `event_id` or `(tenant_id, consumer, event_id)`, claimed by the handler or by the transport? Three arms travel with it: the class of consumer whose effect cannot share the inbox transaction (an external call, the platform-scope erasure handler the Audit spec already carries) and what "effectively once" means for it; whether an event without the organization marker is delivered tenant-wide regardless of the producing request's organization; and whether the inbox entity joins [ADR-0044 § 7](../decisions/0044-audit-write-path.md)'s named audit-capture exclusion list once a delivery declares intents | One shared tenant-owned `inbox_messages` keyed by tenant/stable subscription/event, transport-claimed before construction; a separate delivery ledger owns retries, terminal state and targeted replay; completion shares the business/audit/outbox transaction | **ADR-0046** + a dated **ADR-0006 Amendment** (or an explicit supersession note in ADR-0046 naming Amendment 1's consumer-side paragraph, which fixes the per-module inbox) + a dated **ADR-0010 Amendment**, because its § Idempotency makes two things mandatory that the shared-table leaning changes — the inbox is "the module's **own** inbox table", and the guard is called by the consumer — + the `Integration_Event_Handlers_Use_InboxGuard` assertion re-written in the same record to name whoever ends up owning the call, since a transport-claimed delivery fails a structural rule that looks for the call inside the handler + Database Standards § Inbox + an ADR-0041 erratum beside ADR-0023's minting list | P02b-2 (the record and its amendments: the dispatcher's retry state and a failed delivery's next attempt are one state machine), P02b-3 (Database Standards § Inbox and the ADR-0023 erratum) | Accepted — 2026-10-12; decision/amendments delivered; P02b-3 Steps 1–6 implement subscriber runtime, inbox, defaults and production enforcement; Step 6 record owns PR/CI status | | G2 | Does `IOutbox.EnqueueAsync` write the row immediately on the ambient transaction, or buffer for `OutboxFlushBehavior` at step 7 — and what is the flush point for the two paths that have no step 7? | Immediate write, with the behavior asserting no buffered message survives the handler | [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md) bounded supersession of ADR-0032 Sub-decision 2's step-7 completion role, with a dated supersession disclosure in ADR-0032 under this acceptance; phase-doc + [Events and Outbox § Producer pattern](../architecture/15-event-and-outbox.md#producer-pattern) | P02b-1 | Accepted — immediate enqueue/owner completion | -| G3 | What replaces the unconditional same-key publish-order promise — a scoped bound, or strict per-key head-of-line blocking — and is the key's derivation checkable structurally, given that [§ Ordering](../architecture/15-event-and-outbox.md#ordering) admits an aggregate id or a deliberately declared tenant-wide key and nothing else? | Scope it: no column in `outbox_messages` carries an unconditional per-key guarantee, so the honest bound is one processor, no failure, deterministic eligible selection; no enqueue/commit-order guarantee | Derivation: [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md). Ordering: Accepted ADR-0046 and the P02b-2 package, followed by [§ Ordering](../architecture/15-event-and-outbox.md#ordering) alignment under this acceptance | P02b-1 (whether the key's derivation is checked structurally: the first event declares a key), P02b-2 (the ordering bound) | Derivation and ordering Accepted; scoped ordering runtime [implemented in P02b-2](#p02b-2-step-3--hosted-lifecycle-and-observability-2026-10-12); local closeout complete; [PR handoff](#p02b-2-pr-handoff-2026-10-12) owns CI/review/merge state | +| G3 | What replaces the unconditional same-key publish-order promise — a scoped bound, or strict per-key head-of-line blocking — and is the key's derivation checkable structurally, given that [§ Ordering](../architecture/15-event-and-outbox.md#ordering) admits an aggregate id or a deliberately declared tenant-wide key and nothing else? | Scope it: no column in `outbox_messages` carries an unconditional per-key guarantee, so the honest bound is one processor, no failure, deterministic eligible selection; no enqueue/commit-order guarantee | Derivation: [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md). Ordering: Accepted ADR-0046 and the P02b-2 package, followed by [§ Ordering](../architecture/15-event-and-outbox.md#ordering) alignment under this acceptance | P02b-1 (whether the key's derivation is checked structurally: the first event declares a key), P02b-2 (the ordering bound) | Derivation and ordering Accepted; scoped ordering runtime [implemented in P02b-2](#p02b-2-step-3--hosted-lifecycle-and-observability-2026-10-12); complete and merged; [merge closeout](#p02b-2-merge-and-closeout-2026-10-13) owns CI/review/merge state | | G4 | What counts as an attempt, what is `MaxAttempts` bound to, and what marker keeps a terminal row terminal as the clock advances? | Six attempts (one immediate plus the five documented delays) and an explicit terminal column, because a pushed-out `available_after` stops being terminal and never leaves the pending gauge | Accepted ADR-0046 + dated ADR-0006 disclosure under this acceptance; Database Standards § Outbox DDL, index and grant edits | P02b-2 | Accepted — 2026-10-12; producer attempts/terminal state [implemented](#p02b-2-step-2--durable-transitions-and-terminal-audit-2026-10-12) | -| G5 | Does the stored event identity stay the assembly-qualified type name with declared compatibility mappings, and what happens to an unresolvable type, a retired type and an event with no subscriber? And what is the **row's** identity — a surrogate `id` plus an `event_id` column, or `EventId` as the key — given that two documents list an `event_id` the canonical DDL does not have? | Keep the type identity, add a bounded resolver reading through `PayloadJsonOptions` and declared mappings; dead-letter a retired type at once rather than spending its attempt budget; reconcile the row's column list in one direction | Stored identities: [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md) bounded supersession of ADR-0032 Sub-decision 12 and the Phase 02b deliverables table in Implementation Notes, with a dated supersession disclosure under this acceptance; Database Standards alignment. ADR-0046 accepts declared resolution, Pending holds and intentional absence; dispatch runtime remains P02b-2 | P02b-1 (the stored type identity and the row's identity, both written at enqueue), P02b-2 (unresolvable, retired and unsubscribed types) | Stored identities and resolution outcomes Accepted; resolver/readiness runtime [implemented](#p02b-2-step-3--hosted-lifecycle-and-observability-2026-10-12); Active durable subscriptions pending P02b-3 | -| G6 | When the unit-of-work owner is not an `AuditLogBehavior`, who performs the four owner-side audit acts and enters the outermost audit frame — and does a state-mutating delivery declare its own intents, keyed by handler type? | The transport opens the outermost frame beside its transaction; a delivery declares its own intents and an unregistered handler is refused as an unregistered request is | [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md): bounded ADR-0040 nested-failure supersession; separately enumerated additive UoW read-seam amendment; clarifying ADR-0044/0033 common-owner amendments. G1-dependent production delivery/inbox implementation remains P02b-3 | P02b-1, P02b-3 | Producer/common owner delivered; delivery/inbox contract Accepted, implementation pending P02b-3 | -| G7 | For a dead-letter transition: what class, what `OperationType`, what slug, on which transaction, in what order against the state change, with what failure posture — and may a second non-MediatR caller reach `IAuditStore.WriteStandaloneAsync`? | MUST `security-event`; producer `platform.outbox.dead_letter` and reserved P3 `platform.delivery.dead_letter`; fenced terminal transition and audit INSERT on the same short transaction through a narrow fifth write seam; failed audit retries terminalization without publication | Accepted ADR-0046 + bounded ADR-0044 audit-grant replacement, separate additive fifth-writer seam amendment and ADR-0033 same-transaction clarification under this acceptance; `DeclareOffPath` registration and Audit Coverage baseline | P02b-2 (the producer dead-letter row its criteria assert), P02b-3 | Accepted — 2026-10-12; producer terminal audit [implemented](#p02b-2-step-2--durable-transitions-and-terminal-audit-2026-10-12); subscriber terminal audit pending P02b-3 | -| G8 | Does the dispatcher's `BYPASSRLS` use carry a per-invocation `security-event` row, or is it exempt — and on what written terms? | Exempt, on three terms: the bypass is grant-bounded and code-path-confined, dispatch attempts are logged rather than audited, and the dead-letter transitions are the audited events | Accepted ADR-0046 explicitly replaces the routine-dispatch coverage rule and bounds the ADR-0003 grant extension; dated disclosures under this acceptance, not an ADR-0041 correction | P02b-2 | Accepted — 2026-10-12; confined dispatcher exception/census [implemented](#p02b-2-step-4--enforcement-and-closeout-2026-10-12); local closeout complete; [PR handoff](#p02b-2-pr-handoff-2026-10-12) owns CI/review/merge state | +| G5 | Does the stored event identity stay the assembly-qualified type name with declared compatibility mappings, and what happens to an unresolvable type, a retired type and an event with no subscriber? And what is the **row's** identity — a surrogate `id` plus an `event_id` column, or `EventId` as the key — given that two documents list an `event_id` the canonical DDL does not have? | Keep the type identity, add a bounded resolver reading through `PayloadJsonOptions` and declared mappings; dead-letter a retired type at once rather than spending its attempt budget; reconcile the row's column list in one direction | Stored identities: [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md) bounded supersession of ADR-0032 Sub-decision 12 and the Phase 02b deliverables table in Implementation Notes, with a dated supersession disclosure under this acceptance; Database Standards alignment. ADR-0046 accepts declared resolution, Pending holds and intentional absence; dispatch runtime remains P02b-2 | P02b-1 (the stored type identity and the row's identity, both written at enqueue), P02b-2 (unresolvable, retired and unsubscribed types) | Stored identities and resolution outcomes Accepted; resolver/readiness runtime [implemented](#p02b-2-step-3--hosted-lifecycle-and-observability-2026-10-12); both real durable subscriptions Active in P02b-3 Step 5 | +| G6 | When the unit-of-work owner is not an `AuditLogBehavior`, who performs the four owner-side audit acts and enters the outermost audit frame — and does a state-mutating delivery declare its own intents, keyed by handler type? | The transport opens the outermost frame beside its transaction; a delivery declares its own intents and an unregistered handler is refused as an unregistered request is | [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md): bounded ADR-0040 nested-failure supersession; separately enumerated additive UoW read-seam amendment; clarifying ADR-0044/0033 common-owner amendments. G1-dependent production delivery/inbox implementation remains P02b-3 | P02b-1, P02b-3 | Producer/common owner delivered; delivery/inbox contract Accepted; common owner and execution implemented by P02b-3 Steps 2–3 | +| G7 | For a dead-letter transition: what class, what `OperationType`, what slug, on which transaction, in what order against the state change, with what failure posture — and may a second non-MediatR caller reach `IAuditStore.WriteStandaloneAsync`? | MUST `security-event`; producer `platform.outbox.dead_letter` and reserved P3 `platform.delivery.dead_letter`; fenced terminal transition and audit INSERT on the same short transaction through a narrow fifth write seam; failed audit retries terminalization without publication | Accepted ADR-0046 + bounded ADR-0044 audit-grant replacement, separate additive fifth-writer seam amendment and ADR-0033 same-transaction clarification under this acceptance; `DeclareOffPath` registration and Audit Coverage baseline | P02b-2 (the producer dead-letter row its criteria assert), P02b-3 | Accepted — 2026-10-12; producer terminal audit [implemented](#p02b-2-step-2--durable-transitions-and-terminal-audit-2026-10-12); subscriber terminal audit implemented by P02b-3 Step 4 | +| G8 | Does the dispatcher's `BYPASSRLS` use carry a per-invocation `security-event` row, or is it exempt — and on what written terms? | Exempt, on three terms: the bypass is grant-bounded and code-path-confined, dispatch attempts are logged rather than audited, and the dead-letter transitions are the audited events | Accepted ADR-0046 explicitly replaces the routine-dispatch coverage rule and bounds the ADR-0003 grant extension; dated disclosures under this acceptance, not an ADR-0041 correction | P02b-2 | Accepted — 2026-10-12; confined dispatcher exception/census [implemented](#p02b-2-step-4--enforcement-and-closeout-2026-10-12); complete and merged; [merge closeout](#p02b-2-merge-and-closeout-2026-10-13) owns CI/review/merge state | | G9 | Which record decides the background-job runtime — its storage under the closed four-role model, the single tenant-context writer, the job frame's transaction owner, the enqueue-site rule *and who may enqueue whose job type*, the retry contract (whose class, slug and posture adopt **G7**'s), the tenantless platform-job class a purge needs, the ADR-0035 classification, the queue grammar, and the package pins with their licence verdict? | One record answering all of it; a job reaches the database only through `ISender`, which leaves ADR-0040's closed setter set at eight | **ADR-0047** + a dated ADR-0003 Amendment for the storage + a dated **ADR-0036 Amendment** (or an erratum) if the job-path writer is not the `JobActivator` its § Rules names in the closed four-caller set, then Standards 05 and Standards 20 | P02b-4 | Open | | G10 | Does the active tenant travel as a scalar `tenant_id` claim, a `memberships` array, or both — and what value does `UserId` hold? | The scalar claim, whose 02b source is an admin-set user attribute; a second UUID-valued `user_id` attribute for the actor, with `sub` staying Keycloak's subject | Phase-doc: [ADR-0036 § What is out of scope](../decisions/0036-tenant-resolution-trusted-inputs.md#what-is-out-of-scope-and-what-is-not) hands the shape to this phase by name. Reconcile `architecture/13`'s array sketch in the same change | P02b-5 | Open | | G11 | Which [ADR-0036 matrix](../decisions/0036-tenant-resolution-trusted-inputs.md#the-reconciliation-matrix) rows resolve in this phase, given that row 10 consults a reader that denies everyone — and what row covers a signature-valid token carrying no `tenant_id` claim on a tenant host? | Either mint `organization_id` scoped to the host's organization, which makes row 10 need no membership read, or declare the organization host's authenticated 404 and test it as the expected outcome | A dated **ADR-0036** amendment plus errata beside § Staging across packets and the 2026-09-02 erratum, both of which assign row 10 to this phase without noting that its `M covers (T, O)` term is denied until Phase 03 — which groups it with rows 7 and 14 rather than with 6 and 9 | P02b-5 | Open | | G12 | Which Keycloak client does the BFF exchange the code with, where does sign-in happen across two tenant hosts, and how do the client scope, the mappers and the seed users reach a Keycloak database that has already consumed the import? Native HTTPS on retained `*.learnstack.local` hosts, with manual hosts/trust setup, is delivered by Phase 02d and re-verified by P02b-0 | A confidential BFF client with one redirect URI per seed host, retaining native HTTPS and `Secure` cookies; and an idempotent reconciler inside `make seed` — a mapper added only to the import reaches no existing workstation | Phase-doc + the realm JSON + `scripts/seed.sh` + `infra/keycloak/README.md`; a new ADR only if the reconciler becomes ADR-0004's `IIdentityProvider` | P02b-0 re-verifies [Phase 02d G32](phase-02d-walking-skeleton.md#the-decision-register), ADR-0053/0054 and ADR-0055; P02b-5 (which client the BFF redeems the code with: the API pins `azp` against it), P02b-6 | Open | | G13 | What is the BFF session — its custody of the refresh token, its store, its idle and absolute lifetimes, its refresh serialisation under concurrent requests, its terminal `invalid_grant` behaviour, its CSRF control and its logout — and which document owns the access-token lifetime? | The `HttpOnly` cookie Security Standards already describes, encrypted by the session adapter, read only by server code, with single-flight refresh and one authority for the TTL | Phase-doc + one reconciled [Security Standards § Authentication](../standards/11-security.md#authentication); a **new ADR** only for a server-side session store | P02b-6 (which document owns the access-token lifetime: the realm export and its reconciler write it), P02b-7 | Open | | G14 | What bounds the request types reachable over HTTP while `AuthorizationBehavior` is `return next()`, and what is the shape of the token-keyed rate-limit stage behind a shared BFF connection? | An enumerated read-only routable set held by a catalogue-registered rule, and a limiter stage after `UseAuthentication` keyed on the validated subject. The early anonymous stage already runs before authentication: P02d-5 delivers 60/min per canonical direct or authenticated-hop visitor IP plus 600/min peer acquisitions/allocations under ADR-0053 and ADR-0054. Exhausted known visitors refuse before peer debit; unknown visitor allocation remains peer-gated. These remain IP/peer quotas, not token quotas; a request they reject never reaches the token stage. Re-verify the [P5 remediation decision](phase-02d-walking-skeleton.md#p02d-5-external-review-remediation-2026-10-09) before choosing authenticated accounting, preserving pre-classification protection against unknown-host abuse | Phase-doc + catalogue registration; a dated ADR-0032 Amendment for a deny-by-default step instead | P02b-5 (the token-keyed stage and the early stage's treatment of validated callers); G34's anonymous-hop policy and ADR-0054's bounded accounting replacement are Accepted and delivered in P02d-5 | Open | -| G15 | What initiates a replay in this phase? | Targeted internal subscriber replay in P02b-3; producer/operator reset and fencing require a separate Phase 03 contract | Phase-doc; the URL space is its own decision record, per ADR-0036 | — (answered in § The consumer side; recorded so the question travels with the deferral) | Accepted — internal subscriber replay P02b-3; operator contract Phase 03 | +| G15 | What initiates a replay in this phase? | Targeted internal subscriber replay in P02b-3; producer/operator reset and fencing require a separate Phase 03 contract | Phase-doc; the URL space is its own decision record, per ADR-0036 | — (answered in § The consumer side; recorded so the question travels with the deferral) | Accepted — internal subscriber replay implemented by P02b-3 Step 4; operator contract Phase 03 | | G16 | What is `LS0002`'s rule name, which project trees does it run over, what does it inspect, and what escalates it to Error? | The name and scope recorded the way Amendment 1 recorded `LS0001`, with the scope reaching the assemblies that actually handle tokens | A dated **ADR-0032** amendment + a catalogue entry + an `AnalyzerReleases` row | P02b-8 | Open | -| G17 | What are the label sets for the event and job metrics, is the tenant axis permitted, and is it spelled `tenant` or `tenant_id`? | One note in the document that owns metric names, no raw tenant label on a per-event series, and an oldest-eligible-row age metric separate from the pending count | [Observability Standards § Required Metrics](../standards/10-observability.md#required-metrics) edit | P02b-2 (the first series ships there, and a label added afterwards touches every emitter already written) | Event metrics Accepted — 2026-10-12; runtime/guards [implemented](#p02b-2-step-4--enforcement-and-closeout-2026-10-12); local closeout complete; [PR handoff](#p02b-2-pr-handoff-2026-10-12) owns CI/review/merge state; job metrics pending P02b-4 | +| G17 | What are the label sets for the event and job metrics, is the tenant axis permitted, and is it spelled `tenant` or `tenant_id`? | One note in the document that owns metric names, no raw tenant label on a per-event series, and an oldest-eligible-row age metric separate from the pending count | [Observability Standards § Required Metrics](../standards/10-observability.md#required-metrics) edit | P02b-2 (the first series ships there, and a label added afterwards touches every emitter already written) | Event metrics Accepted — 2026-10-12; runtime/guards [implemented](#p02b-2-step-4--enforcement-and-closeout-2026-10-12); complete and merged; [merge closeout](#p02b-2-merge-and-closeout-2026-10-13) owns CI/review/merge state; job metrics pending P02b-4 | | G18 | May an integration-event payload or a job argument carry personal data, what may `last_error` contain, and do the four new durable stores enter the erasure scope? | Identifiers only, a sanitized and length-bounded failure descriptor, and the stores named in the erasure inventory | [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md) decides the new producer payload/NULL JSON metadata policy and application byte cap; a dated ADR-0038 cross-reference under this acceptance. [Data Protection § Right to Erasure](../architecture/23-data-protection.md#right-to-erasure-right-to-be-forgotten), P02b-8 and Phase 11 retain erasure ownership | P02b-1 (the payload rule: the first payload ships there), P02b-2 (the `last_error` rule and its DDL bound), P02b-8 (the erasure inventory) | Payload and failure descriptors Accepted and implemented; erasure inventory open P02b-8 | -| G19 | Where do the sample flow's publisher and consumer live — a fixture pair in the integration assembly, or a real event a shipped module consumes? | A real event between two shipped domain modules, because a test-assembly-only consumer leaves every module-scoped sweep with zero subjects | [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md) and the P02b-1 package; reconcile Tenancy/Customization/Audit specs under this acceptance. ADR-0046 accepts distinct subscription deduplication and envelope delivery scope; P02b-3 implements the two effects and Seeder handoff. The provisioning publisher must prove enqueue from its transaction announcement while ambient context remains unresolved | P02b-1 | Flow and delivery contract Accepted; subscriptions/Seeder handoff pending P02b-3 | +| G19 | Where do the sample flow's publisher and consumer live — a fixture pair in the integration assembly, or a real event a shipped module consumes? | A real event between two shipped domain modules, because a test-assembly-only consumer leaves every module-scoped sweep with zero subjects | [Accepted ADR-0057](../decisions/0057-transactional-event-production-and-completion.md) and the P02b-1 package; reconcile Tenancy/Customization/Audit specs under this acceptance. ADR-0046 accepts distinct subscription deduplication and envelope delivery scope; P02b-3 implements the two effects and Seeder handoff. The provisioning publisher must prove enqueue from its transaction announcement while ambient context remains unresolved | P02b-1 | Flow and delivery contract Accepted; subscriptions/semantic Seeder handoff implemented by P02b-3 Step 5 | | G20 | What does an anonymous request to a **routed non-public** request type on a live tenant host answer — and does the catalogued `Backend_RequiresJwt_OnAllAuthenticatedRoutes` narrow to a surface outside tenant resolution? A second half rides with it: the shipped assertion middleware says that from this phase the refusal code differs by caller, `tenant_mismatch` for an authenticated one and `not_found` for an anonymous one, written by the middleware itself | The matrix wins on a tenant host — [row 2](../decisions/0036-tenant-resolution-trusted-inputs.md#the-reconciliation-matrix) answers 404, byte-identical to an unknown host's, and the 401 rule narrows to the surface where a 401 discloses nothing | The dated **ADR-0036** amendment G11 needs + a catalogue edit to that rule's Asserts line | P02b-5 | Open | ### P02b-0 entry verification (2026-10-12) @@ -2066,14 +2081,1374 @@ assertion (one executed, one failed); the corrected case passes. New-head CI remains pending in this local receipt; the PR description owns live checks and artifact-backed evidence after publication. +### P02b-2 merge and closeout (2026-10-13) + +The maintainer merged [PR #30](https://github.com/HodeTech/LearnStack/pull/30) at +2026-10-13 07:43:06 UTC. GitHub records final PR source +`4b4e3e40cf9147262934652c4eb9621258356f4c` and squash merge +`6a277770e46a7f4ea3574d9d6891d1802cb8fa90`, whose parent is previous main +`b1f6f21cab556ee7ba497dfc0d64e51b8cf1dad8`. Source and squash merge have the +identical tree `c3d02e3368365a108cb8321ccb3fcbc9cec2e575`. Development incorporates +the main merge without switching branches or rewriting published history. + +All seven required checks pass in the +[final PR run](https://github.com/HodeTech/LearnStack/actions/runs/38358520310) +for that source; CodeRabbit also reports success. The tested synthetic merge is +`d347a60a36fc0001656c3d95d7551bc35d818b91`. Verified artifacts record: + +- **3,409 backend passes**, zero failures/skips: 1,847 unit, 287 architecture, + four contract, 175 Docker-free integration and 1,096 Docker integration cases. + The snapshot-interval regression passes on that exact source. +- **1,616 guarded frontend passes**: 1,556 web and 60 SDK, zero + failures/skips/todos; frontend lint, typecheck and builds also pass. +- Exactly 27 matching Lighthouse JSON/HTML report pairs across nine URLs, nine + screenshots and matching source/checkout identities and report digests. + Automated accessibility is 1 throughout, CLS is 0 and normalized initial JS + gzip is 106,357–106,552 bytes. Proof finishes in 631.094 seconds and owned + cleanup passes; timing medians remain advisory. +- OpenAPI base/head specifications are byte-identical, with SHA-256 + `69333252a95d4179e319512e5f45db2781d12eb003d405a509fa047d64e8e012`. + +These are final PR-head results, not a claim about a new post-merge or local +full-stack run. **P02b-2 is closed.** All four steps and verified review fixes +are delivered, including recovery-index alignment, credential confinement, +terminal COMMIT reconciliation and interval-bound observation. Two independent +review rounds per step and correction bundle remain recorded above. Earlier +dated receipts retain their pre-merge wording; all Accepted ADR files remain +unchanged by this closeout. Phase 02b remains in progress. + +**Next: P02b-3 decision pass.** Reverify the Accepted G1/G6/G7 subscriber +contract and G19 handoff against the merged code, then plan delivery and its +proofs before implementation. P02b-3 owns durable acceptance, shared inbox and +independent delivery ledger, subscriber retries/terminal MUST audit, targeted +internal replay, both real Customization defaults subscriptions and semantic +key/version Seeder handoff. The effects remain `card@1` content type and +`plain@1` taxonomy, preserving existing authored/lifecycle state. Required +production subscriptions remain Pending until that boundary is implemented; +producer dispatch does not claim durable consumer effects. + +Background jobs, API authentication, realm reconciliation, BFF sessions and +phase-exit enforcement remain P02b-4–8 in the +[packet table](#packets-and-decision-gates). Operator recovery permissions/routes +remain Phase 03; broker adapters and production readiness remain Phase 11 on +their recorded triggers. This closeout starts no later packet and makes no new +architectural decision. + +**Documentation-only closeout validation.** Five local Release corpus controls +and seven ADR lifecycle/workflow controls pass, zero failures/skips. The 14 +changed Markdown files resolve 1,577 relative links and 333 fragments without +errors; added prose wrapping and whitespace checks pass. All prior P02b +delivery/review receipts and every ADR file are byte-preserved. Runtime suites +are not rerun locally for this documentation-only change; the final PR artifacts +above supply their evidence. + +### P02b-3 decision package (2026-10-13) + +**Accepted — 2026-10-13, @cemil.** The maintainer approves the complete package +and six implementation steps, each with commit and two independent review rounds. +Acceptance precedes runtime implementation; it claims no delivered subscriber. +The audit baseline is `development` at +`652004290c9696e65e68817f223839d4a0f02137`, after PR #30's verified merge. +Three independent read-only audits covered the accepted contracts, storage/ +privilege/owner lifecycle, and Customization/Seeder convergence. Their findings +were checked against current source; they report inspection, not passing tests. + +[ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md#subscriber-acceptance-and-inbox-ownership) +already accepts G1/G6/G7's shared transport-owned inbox, stable subscriptions, +durable acceptance, independent retries, atomic terminal audit and targeted +internal replay. [ADR-0057](../decisions/0057-transactional-event-production-and-completion.md#first-useful-production-flow) +accepts both create-only default effects and the Seeder handoff. This package +concretizes the storage, runtime and enforcement details those records assign to +P02b-3. It does not reopen their decisions or authorize jobs/authentication. + +#### Document and code review + +The review follows the roadmap entry reading, then ADR-0003/0006/0010/0017/0023/ +0033/0036/0038/0040/0042/0043/0044/0046/0057, applicable Backend, Database, +Testing, Observability, Security, Documentation, Audit, Infrastructure and +Architecture-test standards, architecture 03/09/10/15/29/31/32, the Tenancy/ +Customization/Audit specs and matrices, and event/seed/migration/test skills. +The actual producer, bus, registry, transaction/audit coordinators, terminal +witness/store, Customization writers/readers and Seeder are the implementation +premises. Acceptance history remains the authority for any proposed ADR erratum. + +| Verified premise | Required P02b-3 action | +|---|---| +| `InProcessEventBus` invokes and constructs handlers immediately; it owns no durable acceptance or inbox | Replace that publication outcome with durable acceptance; execute only through the independent delivery owner | +| `OutboxEventRegistry` owns stable IDs/readiness; the handler registry carries only concrete handler metadata | Preserve one manifest identity source; bind each Active ID to exactly its declared contract and handler | +| Common owner completion drains events, writes MUST audit and commits without a delivery completion participant | Add a closed delivery-specific final fence/gate while preserving request ownership and all existing announcement methods | +| `EventTerminalTransition` and the fifth audit seam accept only producer terminalization | Extend the restricted actual-row witness and operation census to the subscriber terminal writer | +| Customization's current readers exclude deleted rows and discover by ID or Active key | Add owner-local, tenant-protected concept history across every revision/status, including soft deletion | +| Register uniqueness protects one revision, not concurrent initialization versus another revision | Serialize initialization and concept registration on the same transaction-owned semantic key | +| Seeder requires fixed built-in IDs; its counts assume separate register/publish acts | Resolve built-ins semantically, preserve warm IDs, share initialization, and update inventory/fixtures without weakening authored-ID checks | +| Direct delivery production has no incoming-event causation source | Carry a trusted delivery entry through nested producer frames; keep tenant/org authority in the announcement | +| Existing aggregate/audit request censuses omit integration handlers | Add nonempty production-consumer coverage and planted violations before marking consumer rules Implemented | +| Existing bus logs raw exceptions and adds exception details to spans | Replace transport-owned diagnostics with bounded codes before enabling production subscriptions | + +These gaps are expected P02b-3 work; they are not retroactive P02b-2 failures. +Required production manifests stay Pending until acceptance, execution and Seeder +handoff are ready together. Test-owned Active contracts exercise earlier steps. + +#### Accepted storage and authority details + +1. **Two shared infrastructure tables.** `event_deliveries` retains one copy of + the validated producer envelope per `(tenant_id, subscription_id, event_id)`. + `inbox_messages` holds the completion identity for that same tuple, with no + surrogate UUID. Neither depends on a live producer row. A producer purge + cannot remove a pending delivery's payload. No third dead-letter store/topic + or arbitrary metadata is added. Retained copies and their retention dependency + enter the P02b-8 inventory; actual purging remains Phase 11. +2. **RLS and classes.** The ledger is tenant-owned and organization-scoped because + its envelope carries organization metadata and payload. Use the canonical + tenant-and-organization policy, ENABLE and FORCE, with explicit WITH CHECK. + The inbox is tenant-wide identity machinery with no payload/org authority. + Both use explicit tenant predicates in application SQL. No foreign key to + `outbox_messages` or cascade from business tables controls retention. The + executor restores the stored organization before reading the envelope; + an unmarked event never widens a present organization to null. +3. **Closed grant budget.** `learnstack_app` receives RLS-protected acceptance + INSERT/SELECT and only the ledger completion-column UPDATE. Inbox claim is an + INSERT of an immutable completion marker inside the uncommitted business + transaction; uniqueness serializes duplicates before construction. Only owner + success makes that marker durable; rollback removes it. Its timestamp records + the claim; ledger completion records business completion. Successful app-role + completion retains token/expiry as inert provenance because it cannot UPDATE + those fields. Completed rows are excluded from acquisition, recovery, + terminalization and replay. The bypass scheduler + receives column SELECT for tenant/subscription/event identity, organization, + declared type/topic/partition, occurrence, attempts, availability, lease, + completion, terminal/replay state and bounded diagnostics; UPDATE covers only + scheduling/status columns. It receives no payload or causal-envelope SELECT, + no ledger INSERT/DELETE and no inbox privilege. Application INSERT grants name + acceptance columns only, excluding caller-supplied scheduling/terminal state. + The existing Audit-chain + INSERT-only grant is reused; no fifth role or business-table grant is added. +4. **Canonical detail before code.** Standards 05 owns complete DDL, constraints, + exact column grants and matching pending/recovery/terminal indexes. A new + Platform-chain migration implements it; historical migrations stay intact. + Identity, subscription grammar, nonnegative attempts, lease pairing, + mutually exclusive completion/terminal state and the 256-byte error cap have + CHECKs. Completion and terminal UPDATE predicates match the actual grants; + tests run them as the real roles, not merely inspect grant text. No P3 role + purges these stores; Phase 11 must decide the minimal retention grants and + ordering before installing its purger. +5. **Accepted envelope is immutable.** Publication validates the envelope and + Active manifest, restores its scope, and inserts every required delivery in + one app-role acceptance transaction without constructing business handlers. + An existing identical identity/envelope is idempotent; a conflicting envelope + refuses acceptance without overwriting the first copy. Canonical serialized + data, including organization and causal metadata, determines equality, not + raw `jsonb` whitespace. An existing accepted subset can be completed safely. + An uncertain acceptance COMMIT leaves the producer retryable; subscriber + execution failures never flow back into the producer's retry budget. + +Scoped application completion is a constrained call path, not a claim that column +grants sandbox arbitrary compromised app SQL. Likewise, the terminal audit's +tenant binding is the row witness, not BYPASSRLS GUCs. No tenant-status lookup is +added as an asynchronous hold/drop policy; Phase 02c retains that decision. + +#### Accepted execution, retries and replay + +Each attempt owns a fresh app-role scope. Announce the validated routing tenant/ +organization before resolving a module context, read and revalidate the retained +envelope by its claimed identity/token under RLS, then claim the inbox before +handler construction. Completed duplicates skip construction. Effective actor is +SystemActor; original causal actor and incoming EventId remain trusted producer +metadata. Ordinary execution refuses privileged subscriptions. Phase 03's +erasure execution contract remains separate. + +The outer audit coordinator classifies the declared handler before construction; +nested commands retain their own classified intents. No module writes audit SQL. +The common transaction owner drains domain events, writes pending MUST audit, +performs the delivery completion UPDATE with tuple/token/unexpired database lease, +and atomically admits physical COMMIT against a sticky completion gate. A failed +fence or a gate closed by cancellation poisons and rolls back all work. The gate +arbitrates Open -> Closed versus Open -> Committing; cancellation after commit +admission is an unknown/Indeterminate outcome, not guaranteed rollback. Durable +inbox/ledger state decides recovery, including an uncertain sixth-attempt COMMIT. +This adds no IUnitOfWork session setter, retargeting or independent business commit. + +| Policy | Accepted initial bound | +|---|---| +| Host and acquisition | API-owned subscriber worker; one just-in-time normal claim; reuse the confined keyed dispatcher source | +| Normal claims | Six per subscription/event, including claims whose execution outcome is unknown | +| Five retry delays | 1 s, 5 s, 30 s, 5 min, 1 h, using database availability time | +| SQL / terminal transaction | 5 s | +| Complete execution wait | 10 s, including construction, handler and owned cleanup; timeout closes the commit gate | +| Lease | 30 s, with validated execution/transition margin; no renewal promise | +| Idle poll / snapshot interval | 200 ms / 5 s; successful work does not force a scan | +| Progress / stale health | 45 s | +| Cancellation-ignoring work | At most four retained tasks; block their local partition, unhealthy while retained, stop acquisition at capacity | +| Shutdown | One total 30 s wait; live task resources remain owned until actual completion or process exit | + +The worker retains and observes cancellation-ignoring tasks with their live scope; +it does not dispose a running callback or promise managed-code termination. +This is one additional in-process subscriber worker in the existing API host; +it neither creates a new host nor changes the producer's existing bounds. Its +four-task and 30 s bounds apply to this worker, not an added whole-host shutdown +guarantee. All stop waits also observe the host's shared shutdown cancellation. +If scope disposal faults or times out after acknowledged business COMMIT, the +completed ledger/inbox remain definitive; cleanup cannot undo or rerun that +effect. Unknown completion reconciles durable state. Overrunning cleanup remains +task-owned and consumes capacity until actual disposal finishes. +Recovery replaces expired ownership before transitions and spends no additional +attempt. Ordinary failed/unknown outcomes use independent backoff. Expired claim +six requests terminalization without a seventh invocation. Requests for terminal +state are never normal execution candidates. Audit refusal retries terminalization +alone. Closed reasons distinguish invalid envelope/subscription from exhaustion; +descriptors expose bounded stage/category codes, never raw exception text. + +Subscriber terminalization uses the same fifth audit seam and a subscriber witness +minted only from its successful fenced UPDATE result. It binds actual tuple, +organization, token, attempts, reason and live connection/transaction. The +Tenancy catalogue/matrix registers `platform.delivery.dead_letter` as off-path +MUST/SecurityEvent when this writer ships. Subscriber terminal audit uses the +validated attempt trace and bounded event/subscription/reason/attempt metadata; +it does not grant the bypass scheduler causal-envelope or payload access merely +to reuse the producer witness. Business execution retains original causality. + +Internal replay takes exactly one declared tenant/subscription/event tuple. It +admits an Active ordinary subscription and completed terminal ledger row only; +live claims, pending terminalization, unknown/retired/Pending subscriptions and +completed business deliveries are refused. The fenced reset clears that row's +terminal request/timestamp, attempts and bounded error, makes it due and invalidates +old ownership. It never deletes/resets an inbox or republishes a broadcast. +The scheduler later invokes only that subscription. A completed inbox still +short-circuits business construction if an earlier uncertain attempt committed. +No replay HTTP route, permission grant or operator producer reset ships here. + +Standards 10 owns subscriber metric names/dimensions and the registered, unmapped +health check. Keep no tenant/event/partition/trace labels; subscription/event-type +labels are finite declared-registry values only. Snapshot gauges perform no SQL; +refresh failure preserves the prior snapshot and reports stale/unhealthy state. +Producer and subscriber attempt/failure/terminal/dedup signals remain distinct. +The API's `/healthz` and anonymous public contract remain unchanged. + +#### Accepted defaults and Seeder handoff + +The two stable identities remain +`customization.initialize-content-type-default.v1` (`card@1` content type) and +`customization.initialize-taxonomy-default.v1` (`plain@1` level taxonomy). +Each effect owns exactly one root. New roots are validated, created and made +Active in one transaction/save, with a Customization-owned UUIDv7, one generation +increment and one classified initialization MUST audit. Shared owner-local +initialization commands let the Seeder use the same logic without a worker, +dispatcher credential, raw SQL or new tenant-context setter. Event handlers +coordinate only their own effect through those commands; they do not combine roots. + +A transaction-owned semantic concept lock serializes each initializer with the +ordinary registration writers for `(tenant, aggregate-kind, key)`. Under that +lock, any revision in Draft/Active/Deprecated or soft-deleted history means +preserve, not initialize. A successor-only concept also counts as touched. +Whichever writer commits first defines whether the concept was untouched; +initialization never overwrites, republishes, revives or fills a missing revision +of an already touched concept. Parameterized owner-local history reads retain +explicit tenant predicates and RLS; no query-filter bypass is introduced. +Same-key locks do not create a cross-tenant authority path. + +The initialization operation classifies preserved/no-op attempts explicitly; +it changes no root or generation. A new delivery may record an initialization +attempt without entity changes; a completed duplicate constructs no handler and +creates no second business audit. Seeder pre-verifies completed built-ins before +sending an initializer, so a completed rerun remains write/audit-neutral. +Fresh inventory counts reflect one initialization audit/bump per built-in, +not the old two register/publish acts. That is the uncontended creation count; +separately classified preserved attempts after a seed/delivery race may add an +audit without a generation bump. Tests distinguish creation from no-op attempts, +and count the actual deduplicated delivery identities rather than asserting that +every asynchronous interleaving has the Seeder-only audit total. Historical +receipts are not recounted. + +Built-in Seeder verification discovers semantic key/version, then validates exact +Active state and built-in shape while retaining the stored ID. Existing matching +non-demo IDs pass. Preserved Draft, deprecated, deleted, successor-only or +mismatched definitions remain untouched; if they cannot satisfy the declared +demo inventory, seeding refuses with a bounded mismatch instead of claiming +completion. Tenant-authored definitions/courses/lessons retain exact declared +IDs and exact pins. Consumer and fresh Seeder races converge in both winning +orders; an unexpected database failure is not a blind retry/overwrite. + +#### Decision vehicles and documentation actions + +No new architecture decision is proposed: these details fit ADR-0046/0057's +explicitly delegated P02b-3 boundary. If implementation disproves that fit, stop +and prepare a new decision for maintainer review before changing the contract. +Acceptance of this package authorizes the following already-required changes, +with original Accepted bodies/metadata and historical receipts preserved: + +- **ADR-0023:** verify the inbox minting-list wording at its acceptance commit; + add the ADR-0041 inline erratum and its own dated amendment, because inbox + identity reuses the producer EventId rather than minting a surrogate. This + corrects the listed table, not the UUIDv7 decision. +- **ADR-0044:** append the additive capture-exclusion clarification for the + shared inbox and delivery ledger; add subscriber fifth-seam/operation navigation + already authorized by ADR-0046. Explicit business intents remain audited. +- **ADR-0046/0038/0057:** dated delivery/navigation notes when their P3 obligations + actually ship; no premature completion claim or replacement of earlier bodies. +- **Mutable carriers:** canonical Inbox/ledger DDL and grant matrix in Standards + 05; lifecycle/diagnostics/audit rules in 02/09/10/11/18/20; canonical tests in 21; + architecture 03/09/10/15/29/31/32; glossary; module specs/matrices; event/seed/test + skills; Seeder inventory/fixtures and `BuiltInCustomizations` creator remarks. + Reconcile only touched claims; do not duplicate an ADR or RLS template. +- **Hub pointer:** verify the sibling repository's per-module inbox dependency + and coordinate its documentation pointer before P3 closeout, as ADR-0046 requires. + The current sibling `docs/roadmap/README.md` P02c-3 row still names per-module + inbox tables; replace only that dependency with a link to ADR-0046's shared + transport-owned inbox contract when coordinating the documentation change. + No Hub runtime, HTTPS contract or branch change is authorized by this package. + +At acceptance, Standards 21 registers the following additional canonical rules +as Registered, with their behavioural/structural owners; no passing enforcement +is claimed before code: + +- `Integration_Event_Publication_Is_Durable_Acceptance` — real-role acceptance, + immutable duplicate/collision handling, partial/unknown COMMIT and no construction. +- `Delivery_Retries_And_Replay_Are_Subscription_Scoped` — independent budgets, + terminalization, completion gates, final-claim recovery and targeted replay. +- `Delivery_Terminal_Audit_Identity_Is_Row_Bound` — restricted subscriber witness + census and atomic matching/mismatched identity/transaction proofs. +- `Delivery_Execution_And_Observation_Are_Bounded` — actual hosted execution, + retained resources, shutdown, closed diagnostics and snapshot/health emitters. +- `Customization_Default_Initialization_Preserves_Touched_Concepts` — one-root + useful creation, all lifecycle preservation and serialized registration races. +- `Seeder_BuiltIns_Use_Semantic_Identity` — generated/warm IDs, exact shape and + authored IDs, no-worker operation, rerun neutrality and real delivery races. + +Extend the existing `Integration_Event_Handlers_Use_InboxGuard` and +`Integration_Event_Handler_Restores_Tenant_Context` obligations rather than +inventing alternate names for their claim/owner/scope assertions. Extend the +existing credential confinement census only for the named subscriber scheduler/ +store; business handlers and application execution never receive bypass access. + +#### Six accepted implementation steps + +Each step commits on `development`, then completes two fresh independent review +rounds. Verify findings against current code, fix valid findings, validate and +commit fixes before the next round/step. A clean review needs no empty commit. +Continue automatically unless a real blocker or new maintainer decision arises. + +1. **Accepted detail, schema and privileges.** Record approval before runtime; + register canonical proof obligations, reconcile the named ADR instruments, + implement Platform migration/table classes/RLS/grants/indexes and test + forward/down/reapply, actual role permissions and tenant/org isolation. + Production subscriptions remain Pending. +2. **Durable acceptance and inbox primitives.** Replace immediate broadcast + execution with validated transactional acceptance; implement immutable envelope + collision/duplicate semantics and transport-owned inbox claims. Prove partial + and uncertain acceptance, independent identities, concurrent duplicates and + construction-free publication with real PostgreSQL. +3. **Business owner and completion gate.** Implement fresh application scopes, + restoration before resolution, classification, incoming-event causation, + common owner completion and sticky commit arbitration. Prove business/inbox/ + outbox/MUST-audit/ledger atomicity, canceled/expired/stale attempts, no-listener + metadata, owner cleanup and unknown COMMIT without activating production defaults. +4. **Subscriber scheduling, terminalization and replay.** Implement independent + leases/attempts/recovery, witnessed terminal MUST audit, targeted reset, bounded + task ownership, minimal diagnostics and snapshot health/metrics. Prove sibling + independence, sixth-claim recovery, late completion, audit rollback, identity + substitutions, safe shutdown and replay fences using the actual worker. +5. **Useful consumers and convergent seed.** Add the two one-root initializers, + concept serialization/history reads and semantic verification; migrate Seeder + and fixtures, then activate both real subscriptions together. Prove untouched + creation, every preservation lifecycle, warm IDs, strict authored pins, repeated + seeding and concurrent fresh seed/delivery in both winning orders. +6. **Enforcement and PR closeout.** Complete production-handler/call-path censuses + with planted clean/dirty controls, full Release build/format/backend regression, + affected frontend/launcher tests and real demo/Lighthouse proof. Reconcile all + current-state docs and accurate test counts; open the PR only after every step + and both reviews finish. Verify all seven required CI checks at the exact head; + never claim merge or delivery of P02b-4–8. + +#### Decisive acceptance proofs + +- Actual app-role tenant/org isolation and real bypass column grants: scheduler + payload/inbox reads and app retry/reset writes fail; allowed acceptance and + completion succeed. Completed inbox rows persist across replay and producer purge. +- Two same-module subscriptions and concurrent duplicates yield one business + effect each. A duplicate waits for the first transaction; rollback permits a + new claim. Handler construction failure is independent of acceptance success. +- Fenced ledger completion, business changes, produced events and MUST audit all + roll back on a stale/expired token or closed gate. Kill the fence/gate in a + controlled mutant to prove sensitivity, including timeout persistence failure + followed by an ignored-token late callback. +- Real lost-COMMIT acknowledgement recovery, including the final subscriber + claim, does not repeat committed business work or terminal audit. Cancellation + after commit admission never claims certain rollback. Postcommit disposal fault + and timeout controls retain durable completion, prevent effect replay and keep + live cleanup resources safely owned. +- Subscriber terminal draft tenant/org/event/subscription/operation/transaction + substitution rolls back both transition and audit; matching org-scoped work + commits. Audit refusal never makes terminal-requested work executable again. +- Targeted replay changes one terminal row only; completed sibling/inbox rows + and producer attempts stay unchanged. Stale pre-replay callbacks cannot complete. +- Nonempty production censuses cover every module layer and the actual handler, + manifest, owner, inbox, terminal-witness and credential sites. Planted direct/ + interface/helper bypasses and clean controls distinguish enforcement from an + empty or narrowed subject set. Extend one-root and audit-registration coverage + to integration handlers; do not only count request handlers. +- Logs/spans/descriptors/artifacts reject planted payload, secret, raw type and + exception values. Finite telemetry labels, prior-snapshot retention, exact + refresh cadence and no SQL gauge callbacks have positive/negative controls. +- Runtime proof includes actual TenantCreated backlog activation, both defaults, + ordinary app-role isolation and warm/fresh Seeder races. Test-only subscribers + alone cannot close the packet's useful-flow requirement. + +#### Decision-package reviews and validation + +Two fresh, sequential, read-only reviews examined the proposed working-tree +package against baseline `65200429` and the current accepted contracts/code: + +- **Round 1 — GPT-6-astra, high:** Approve, no Blocker/Major. Clarifications for + postcommit disposal, inert completed-row lease provenance and creation/no-op + audit counts were verified and incorporated before round 2. +- **Round 2 — GPT-6.1-sol, high:** Approve, no Blocker/Major/Minor. Independently + checked the completion/grant model, defaults/Seeder policy and ADR-0023's + acceptance-time natural inbox identity. It found no additional decision gap. + +Root validation passes all five `CorpusConsistencyTests` and seven ADR-status +controls, with zero failures/skips. The changed phase document has 184 checked +relative links and 98 fragments, with no missing targets/anchors. Added prose +passes the 88-column check with documented link/table exceptions; whitespace is +clean. Every Accepted ADR and all earlier P02b delivery/review receipts remain +byte-preserved. This is documentation validation, not a P02b-3 runtime test run. +The package remains Proposed until explicit maintainer approval; that approval +and the registered obligations precede the first runtime implementation commit. + +#### Maintainer acceptance (2026-10-13) + +The maintainer's explicit answer, "Paketi onaylıyorum; implementasyona geç", +accepts this package after both decision-package reviews. Their preceding receipt +records the pre-approval state. The six steps and two-round loop are authorized +on development. G1/G6/G7/G19 keep their accepted architectural answers; concrete +storage, authority, execution/replay, default and Seeder details now have this +accepted vehicle. Catalogue additions remain Registered until their proofs ship. +The decision acceptance, ADR-0023 erratum and ADR-0044 capture clarification are +committed before the first P02b-3 runtime change. + +### P02b-3 Step 1 — shared storage and privileges (2026-10-13) + +**Complete; both independent review rounds complete.** Decision acceptance +`63363ac` precedes this runtime change. EF generated Platform migration +`20261013083903_add_event_delivery_and_inbox`; its raw infrastructure tables add +no EF entity or model snapshot change. Standards 05 owns the complete applied +DDL, table classes, constraints, total-order indexes and exact column grants. + +The org-scoped ledger retains a subscription envelope independently of producer +retention. The tenant-wide inbox uses the producer event identity and subscription +without a surrogate; database guards refuse UPDATE, DELETE and TRUNCATE even as +the owner. No P3 role receives purge privileges. The app role accepts only immutable +envelope columns and completes only `processed_at`; the scheduler receives routing/ +status columns and nine scheduling UPDATE columns, excluding payload, causal +metadata, inbox access and business completion. Completed rows retain inert lease +provenance and are excluded from every scheduler index. Real subscriptions remain +Pending; this step claims no acceptance or business-delivery runtime. + +`EventDeliveryMigrationTests` exercises 73 real-PostgreSQL cases: permitted SQL, +exact grants and forbidden capabilities, tuple identity/ASCII grammar/state, +immutable guards with removed-trigger dirty controls, matching index/order plans, +producer-retention independence, canonical-document catalog parity and disposable +Down/reapply. The existing all-table fixture now seeds both stores for both tenants, +including distinct ledger organizations, and marks fixture ledger canaries complete. +The general tenant-isolation, foreign-write and grant sweeps include them. +Ledger-specific organization and scope-hatch proofs are recorded below. +Standalone Platform migration proofs install Tenancy's shared organization guard +first, matching the documented deployment sequence; historical migrations are intact. + +Validation before the implementation commit: Release build with `CI=true` has +zero warnings/errors; 164 focused schema/migration integration cases and all 287 +architecture cases pass, zero failures/skips. The seven ADR-status controls pass. +Format and whitespace validation pass; 1,180 relative links and 286 fragments +resolve without errors. Earlier receipts and Accepted ADR text remain intact. +Unit/frontend/runtime-worker suites are +not rerun for this schema-only step. Subsequent steps own acceptance, execution, +subscriber scheduling, useful defaults and full packet regression. + +#### Step 1 review round 1 + +Fresh GPT-6-astra/high reviewed every file in `63363ac..ad57a1b` through the +standards and code-review lenses. It requested changes for one missing required +proof: existing cross-organization/scope-hatch behavior tests name other tables, +so they do not establish the new ledger's organization boundary. Root independently +confirmed this gap; the current migration predicates were not found defective. + +The correction adds app-role ledger read, exact/null/mismatched acceptance and +fenced completion tests, with and without the tenant read hatch. Five live leased +canaries distinguish tenant-wide, own/sibling organization and foreign-tenant rows. +Own-scope controls remain writable. Disposable policy mutants prove a missing +organization read term leaks the sibling, a read hatch copied into WITH CHECK +admits forbidden acceptance, and removing only the restrictive UPDATE guard +changes zero matched rows into the remaining WITH CHECK's RLS refusal. The last +control does not claim a write bypasses both protections. No production policy or +grant changes are needed. The correction passes 125 focused ledger/organization +integration cases, zero failures/skips, with `CI=true` Release. Review round 2 +follows the correction commit. + +#### P02b-3 Step 1 review round 2 and closeout + +Fresh GPT-6-sol/high independently reviewed all 13 files in `63363ac..fe6f9c7`, +including the correction rather than only its receipt. Verdict: Approve, no +Blocker/Major/Minor. It verified the real-role organization proofs and weakened +policy controls, exact grants, RLS, immutable tuple inbox, ordered indexes, +canonical DDL parity, reversal and documentation scope. It made no edits and +inspected the tests; it did not independently rerun the reported suites. + +Root's correction run passes 125 ledger/organization integration cases, including +all 83 current `EventDeliveryMigrationTests`, zero failures/skips. Earlier Step 1 +verification passed the 164-case schema/migration group and 287 architecture cases. +The correction changes tests and their record only; format, documentation and +whitespace checks pass. Implementation commits are `6b1efef` and `ad57a1b`; +`fe6f9c7` closes the verified round-one proof gap. Step 2 follows automatically +under the accepted package; production subscriptions remain Pending. + +### P02b-3 Step 2 — durable acceptance and inbox primitives (2026-10-13) + +**Implemented; both independent review rounds complete.** `InProcessEventBus` now +validates the concrete canonical contract and required Active manifest, restores +publication context before creating a fresh application scope, and commits all +subscription envelope copies in one announced transaction. It constructs no +business handler. Exact identity plus canonical immutable fields distinguishes +idempotent duplicates from conflicting or inaccessible organization copies. +Matching subsets complete; a later collision rolls back earlier new copies. +Uncertain acceptance COMMIT remains producer-retryable. Only explicit intentional +absence may acknowledge without deliveries; Pending/unknown/retired contracts +refuse rather than silently reach zero handlers. + +`IInboxGuard`/`PostgresInboxGuard` claim only the tuple on the live, writable, +unpoisoned announced app-role transaction. PostgreSQL uniqueness waits for a +competing transaction: commit yields an existing identity; rollback permits a new +claim. The primitive owns neither connection nor commit. Business completion, +classification, independent execution and timeout gates remain Steps 3–4; +production default subscriptions remain Pending until Step 5's complete path. + +`DurableEventAcceptanceTests` and `InboxGuardTests` exercise 36 PostgreSQL cases: +construction-free acceptance, immutable-field collisions and JSON key-order +equivalence, matching/conflicting subsets, independent publishers and forced +unique-key waits in both commit/rollback orders, uncertain COMMIT acknowledgement, +actual tenant/org scope, fresh owner transactions, cancellation and inbox admission. +The fixture uses the production AsyncLocal accessor; a forced blocked publication +shares its provider with another tenant's independent successful publication. +Their blocker observers own finite cancellation deadlines. Root corrected the +initial duplicate test's shared process serializer before claiming SQL concurrency. +The canonical durable-acceptance catalogue rule is Implemented; the larger handler/ +inbox ownership and production census rules remain Registered. + +The former 32 immediate-handler bus unit cases no longer describe publish success. +Eight publication unit cases cover refusal boundaries, canonical runtime routing, +context before scope construction and restoration, intentional absence and queued +cancellation. Actual handler construction, invocation, internal handler dispatch, +per-subscription scopes, causal actor/trace, foreign cancellation, async faults and +cleanup are explicit Step 3/4 obligations; this step claims none of those execution +proofs. Existing producer worker timing/cleanup proofs use a disclosed scoped +publication wrapper around the real acceptance adapter. It deliberately models +cancellation-ignoring publication and real DI cleanup ownership. A separate +unwrapped-worker case proves durable acceptance without business construction. + +#### Step 2 validation before review + +The final `CI=true` Release solution build reports zero warnings/errors. The unit +suite passes 1,823 cases and architecture passes 287, with no failures/skips. +The 36 acceptance/inbox cases plus all 18 producer-worker cases pass together +(54 Docker cases). Foundation's canonical manifest registration replaces two +old Persistence-only composition fixture assumptions; the fixtures now exercise +the actual registration without opening a database or adding a second manifest. +The focused composition/corpus run also passes seven cases. Seven ADR-status +controls, format verification and whitespace checks pass. These are local receipts; +no PR-head CI or subscriber-owner delivery is claimed. Both fresh review receipts +follow below before Step 3 begins. + +#### Step 2 review round 1 + +A fresh GPT-6-astra/high reviewer examined every file in `80a3261..78992fc` and +approves with zero Blocker/Major findings. It independently passes all 36 new +acceptance/inbox Docker cases; the other local receipts above were inspected, +not rerun. One Minor catalogue link still named the renamed immediate-handler +worker proof. Root verifies the method rename and updates the label to the actual +acceptance proof. No runtime change is needed. Documentation links/fragments, +historical-record preservation and whitespace checks pass after the correction. +The second fresh review follows the correction commit. + +#### Step 2 review round 2 and closeout + +A fresh GPT-6.1-sol/high reviewer independently examines all changed files in +`80a3261..bce0bc7` and approves with zero Blocker/Major findings. It independently +passes 36 acceptance/inbox, 18 producer-worker, ten focused publication/composition +unit and all 287 architecture cases, with zero failures/skips. It reports one +Minor XML-comment grammar error; root verifies and removes the extra word. +Documentation/fragment, historical-preservation and whitespace checks pass. +Implementation is `78992fc`; `bce0bc7` corrects the verified round-one reference. +No runtime fixes were required. Step 3 follows automatically under the accepted +package; production subscriptions remain Pending. + +### P02b-3 Step 3 — business owner and completion gate (2026-10-13) + +**Complete; both independent review rounds complete.** +The closed prepared-owner entry announces the routing tenant/org and binds the +actual frame, unit, transaction and generation before retained-envelope loading. +Full validated event context replaces any inherited caller context before module +resolution. Inbox claim precedes classification/handler construction. A duplicate +skips business classification/construction and completes under the same final fence. + +Requests and deliveries share the common owner body: successful-save domain drain, +producer invariants, MUST audit, exact-row lock and fresh expiry check, sticky gate +admission, then physical COMMIT. No delivery opens a raw frame and calls the old +entry as a joiner. Incoming-event causation reaches nested production; immediate +domain-event identity takes precedence. SystemActor remains effective while closed +audit metadata preserves original causal actor and incoming event identity in both +writing paths. Shared audit fallback/reconciliation diagnostics suppress raw +exceptions only in event context; ordinary HTTP diagnostics remain unchanged. + +The one-use completion gate distinguishes cancellation before admission, unknown +COMMIT after admission and acknowledged durability. Npgsql retains successful +transaction disposal for owned cleanup, so an acknowledged commit reaches its +receipt before disposal faults or hangs. Cleanup before reuse and stale-frame +protection remain enforced. The executor retains its callback, frame and scope +through actual cleanup; scheduling deadlines/capacity/shutdown remain Step 4. +Ordinary versus privileged execution is explicit manifest metadata; P3 supplies +no privileged executor. Retained Active work may execute despite a Pending sibling, +while the producer's whole-event Pending hold remains unchanged. + +Production defaults remain Pending. Two fresh sequential review receipts follow +before Step 4 begins. + +#### Step 3 validation before review + +The `CI=true` Release solution build has zero warnings/errors. All 1,847 unit and +289 architecture cases pass, zero failures/skips. The 33 new delivery execution +and four physical Npgsql cleanup cases pass together as 37 Docker cases. They +exercise constructor/invocation/null-task failures, caller versus foreign +cancellation, SystemActor and original causal metadata before construction and +through cleanup, direct/nested/domain-event production, atomic five-leg rollback, +duplicate short-circuiting, ownership loss after saved work, expiry behind an +unchanged row lock, lost COMMIT acknowledgement at attempts one/six, and cleanup +faults/hangs after acknowledged durability. Unit proofs separately exercise the +closed capability/gate, fallback Activity, causal audit composition, capture +exclusions and actual nonempty bounded audit-failure diagnostics. + +An isolated source copy passes all six gate/token/expiry controls before mutation. +Removing gate refusal admits a late callback when its token is deliberately not +canceled; the corresponding test fails. Removing only the final expiry check +fails both expired-work and blocked-lock-expiry cases while the token/gate controls +remain green. Removing token equality fails its ownership-loss case while the +other two controls remain green. These are compiled behavioral mutations, not +source-string assertions; production source is never mutated for this experiment. +The restored isolated baseline passes all six controls after the mutations. +Required format verification, whitespace, seven ADR-status controls and the +1,623-link/372-fragment corpus sweep pass. The broader integration regression, +excluding the long frontend rendering proof, is still running at this commit; +its outcome must be recorded before this step closes. Neither this local +receipt nor the held production manifests claims an active production consumer +or PR-head CI. Step 4 retains independent scheduling, timeout and replay proofs; +Steps 5/6 retain actual defaults, Seeder convergence and full packet closeout. + +#### Step 3 review round 1 + +Two fresh complementary reviewers inspect `81ab105..77890ea` against the accepted +package and current code. GPT-6-astra/high approves the runtime/security path with +zero Blocker/Major/Minor findings. It independently passes 22 gate/audit-metadata/ +diagnostic unit and two owner-census cases; integration/mutation evidence is +inspected, not rerun. GPT-6.1-sol/high independently passes those 22 unit, all 37 +new PostgreSQL integration and two owner-census cases, zero failures/skips. It +approves with one Minor: the catalogue's updated 196-method total omits P02b-3 +Step 3 from its delivery attribution. Root verifies the two new methods and adds +that provenance; no runtime correction is needed. + +The broader integration run finishes after the implementation receipt: all 1,373 +additional cases pass, zero failures/skips. This excludes the 37 new cases already +run and `PublicServerRenderingTests`, whose long frontend proof remains in the +full Step 6 regression. Together these runs pass 1,410 integration cases. The +catalogue correction and review record pass link/fragment, historical-preservation +and whitespace checks. Fresh round 2 follows the correction commit. + +#### Step 3 review round 2 and closeout + +Two new reviewers independently inspect `81ab105..5a8b85d`, including the verified +round-one correction. GPT-6-sol/high approves the runtime/security/concurrency +path with zero actionable findings and independently passes 22 focused unit and +37 integration cases. A fresh GPT-6-astra/high evidence reviewer approves with +zero Blocker/Major/Minor findings and independently passes those 22 unit, 37 +integration and two owner-census cases. All reruns have zero failures/skips. +Full-suite and compiled-mutation receipts are inspected rather than rerun by +these reviewers. The initially selected evidence reviewer could not start because +its model was at capacity; it supplies no review evidence and was replaced by +the fresh reviewer named above. + +Implementation is `77890ea`; `5a8b85d` records the broader regression and fixes the +verified catalogue attribution. No runtime review fix is required. Earlier +Accepted ADR text and historical delivery records remain intact. Step 4 follows +automatically under the accepted package; no new decision or privilege expansion +is needed for its independent subscriber scheduler. + +### P02b-3 Step 4 — subscriber scheduling, terminalization and replay (2026-10-13) + +**Complete; both independent review rounds and the verified correction complete.** +The API-owned subscriber worker shares the existing confined keyed data source, +with no new role, migration or credential reader. Its store claims one exact +tenant/subscription/event tuple, selects routing/status columns only, and uses +the existing ordered indexes. Pending and privileged targets are held by exact +type/subscription pair; an accepted Active sibling remains runnable. Normal +claims spend independent attempts, expired claims recover without execution or +another attempt, and the sixth claim never authorizes a seventh invocation. + +The subscriber terminal writer extends the restricted actual-row witness and +fifth audit seam. Tenancy registers `platform.delivery.dead_letter` with that +writer; the terminal UPDATE and matching MUST/SecurityEvent audit share one +transaction. Targeted internal replay admits one completed terminal Active +ordinary tuple, invalidates its old ownership and never resets an inbox or +republishes a broadcast. Operator HTTP/permissions remain Phase 03. + +Task-owned controlled execution closes its sticky gate before asynchronous +cancellation callbacks, retains the actual application scope through those +callbacks and cleanup, and rejects control reuse before acquiring cleanup +ownership. Acknowledged durability survives faulted or overlong cleanup; retained +tasks continue to block their local partitions and consume the four-task capacity. +The shutdown wait uses one budget and the host token. Snapshot refresh follows +the configured cadence even after failure; gauges read immutable memory only. +Finite event/subscription labels and bounded diagnostics add no envelope read to +the scheduler. The private delivery health check leaves `/healthz` unchanged. + +Real-role store/worker tests and exact production-capability censuses with planted +controls pass local validation. Production defaults remain Pending until Step 5's +initializers and Seeder handoff are ready. Final test and two fresh sequential +review receipts follow before that step begins. + +#### Implementation validation + +The CI-equivalent Release build passes with zero warnings/errors. The final +candidate passes 1,855 unit, 298 architecture and 238 focused integration cases, +with zero failures/skips. The integration set includes 45 new store and 21 new +worker cases plus 172 producer/acceptance/inbox/owner/role regressions. TRX files +record each run; Docker-dependent tests use actual PostgreSQL and the guarded +application or confined scheduler role appropriate to the boundary. + +The final replay control first terminalizes four real rows, then resets one; +terminal neighbours differing in tenant, subscription or event remain byte-for-byte +unchanged. Business execution also carries a distinct human causal actor while +effective audit actor remains SystemActor. No-listener trace proof asserts the +actual source has no listeners before checking a fresh canonical attempt ID. + +Final format verification, all seven ADR-status controls, all five corpus guards +and whitespace checks pass. The changed-document sweep checks 1,818 relative +links and 398 fragments with zero missing targets/anchors; added prose passes +the 88-column check. Earlier Accepted ADR bodies and dated P02b receipts remain +byte-preserved. + +Three catalogue rules become Implemented with their behavioural/structural owners. +The catalogue now records 205 architecture methods and 196 Implemented rules, +120 in the architecture assembly and 76 elsewhere. Consumer-handler censuses, +production activation and defaults/Seeder proofs remain Steps 5–6. Earlier dated +receipts retain their original counts and pending status. + +#### Step 4 review round 1 + +Two fresh read-only reviewers inspected `3411424..262d4b2`: + +- **Runtime/security — GPT-6-astra, high:** one verified Major. Subscriber + execution and durable acceptance used the same singleton/raw partition lane; + a retained handler or postcommit cleanup could delay producer acceptance and + consume producer timeout/capacity. Its independent rerun passed 66 focused + PostgreSQL and eight focused unit cases, showing the missing combined scenario. +- **Proofs/docs — GPT-6-sol, high:** Approve, no Blocker/Major/Minor/Suggestion. + It inspected production capability censuses, actual-role/worker tests and + documentation/history without independently rerunning the suites. + +Root verified the shared lane and separates only the internal execution lane +with a fixed `delivery:` prefix. Validated event partitions remain canonical UUIDs; +consumer serialization and raw scheduler partition holds remain intact. A live +subscriber cannot delay a new same-partition publication's durable acceptance. +Two real-bus/PostgreSQL regressions hold the actual handler or acknowledged-COMMIT +cleanup and accept a second same-partition event before releasing the first. +In an isolated source copy, restoring the shared raw lane makes both cases fail; +no production source or branch is mutated for that control. This implements the +accepted independent failure domains and introduces no new policy or ADR. +Correction validation passes the Release build with zero warnings/errors, 1,855 +unit cases, 41 affected architecture cases and 101 store/worker/executor PostgreSQL +cases, with zero failures/skips. Both isolated restored-baseline cases pass again. +Format, whitespace and the 1,818-link/398-fragment document sweep pass; the next +fresh review round follows the correction commit. + +#### Step 4 review round 2 and closeout + +Two new read-only reviewers independently inspected `3411424..14d5f93`, including +the complete Step 4 implementation and round 1 correction: + +- **Runtime/security — GPT-6-astra, high:** Approve, zero + Blocker/Major/Minor/Suggestion. Independently passed 101 PostgreSQL cases, + 16 lifecycle/observation/commit-gate unit cases and 36 architecture cases, + with zero failures/skips. It verified lane separation, fresh-clock fences, + row-bound terminal audit, exact replay and actual resource ownership. +- **Proofs/docs — GPT-6.1-sol, high:** Approve, zero + Blocker/Major/Minor/Suggestion. Independently passed 101 PostgreSQL cases, + 41 architecture cases including all five corpus guards, and eight unit cases, + with zero failures/skips. It verified the corrected binary's new tests/lane, + planted census controls, catalogue counts and preserved historical records. + +Both reviews used existing Release artifacts without independently rebuilding or +rerunning the isolated mutant. Their receipts distinguish actual reruns from +inspection of root's earlier build/format/link/mutant evidence. Neither changed +files or GitHub. The correction is committed at `14d5f93`; root's restored mutant +baseline passes both cases after the old-lane mutant failed both. Step 4 is closed; +Step 5 follows automatically with useful consumers and semantic Seeder handoff. + +### P02b-3 Step 5 — useful defaults and semantic seed handoff (2026-10-13) + +**Complete; both independent review rounds complete.** +Step 4 is closed before these changes. This step implements the accepted pair of +one-root initializers and shared transaction-owned concept serialization, then +moves built-in seed verification to semantic identity. Authored identities and +exact pins retain their existing contract. Both production subscriptions activate +together only after the initializer and Seeder paths are ready. + +The initial implementation supplies two Off-classified subscriber coordinators +and two MUST-classified initializer commands. Both production manifests are Active +and bind the discovered handlers exactly. Each untouched concept is published +before its single save, with one generated UUIDv7 identity, generation bump and +resource audit. Every existing lifecycle/revision is preserved under the same +transaction-owned concept lock used by ordinary registration. + +Seeder discovers the exact Active built-in by key, verifies its full shape and +retains the discovered identity. Authored identities/pins remain strict. Fresh +seed inventory retains all definitions/content but uses six customization +generation bumps per tenant and 86 audit rows, replacing the earlier two-step +register/publish treatment of four built-ins. The P02d seed receipts remain +historical evidence of their original behavior. + +Initial validation: full CI-mode Release build has zero warnings/errors; all +1,862 unit and 298 architecture cases pass. Focused app-role integration passes +24 initialization cases, 83 Seeder/audit/projection/manifest cases and eight real +API delivery cases, all with zero failures/skips. The real flow covers stored +backlog, fresh provisioning, audit-failure rollback/recovery, duplicate acceptance +and Seeder/subscriber races for both concepts in both winning orders. Test-owned +database gates coordinate real writes without a production registration hook. +The race observer uses the app role so PostgreSQL exposes its own role's lock +waits; the earlier owner-role observer could not see app-role activity and failed +the fixture, rather than detecting a product defect. The corrected full flow +passes all eight cases. The existing publication concurrency suite also passes +all ten cases, and the guarded frontend run passes 1,616 cases with zero +skips/todos. Two fresh sequential review receipts follow before Step 6. + +#### Step 5 review round 1 + +Three fresh read-only reviewers examined `94d7d6b..d53ae27` before the second +round. GPT-6-astra xhigh reviewed production correctness, isolation and races; +GPT-6.1-sol xhigh reviewed proof quality and Seeder convergence; both approve with +no findings. They independently reran 92 integration cases and 100 focused +integration/unit cases respectively, all passing with zero skips. These overlap +root's receipts and are independent reruns, not additional distinct test cases. + +GPT-6-sol high identified four mutable Architecture 31 passages still describing +real defaults as Pending, and the standards-check skill still describing inbox +acquisition as pending. Root verified both against the Active bindings and owner +path, corrected those current-state carriers and extended the phasing table to +Step 5. The historical ADR and Step 1–4 receipts remain unchanged. Root's updated +document check passes 2,209 relative links and 480 fragments with no errors; +the reviewer independently passes all five corpus guards. No runtime correction +was needed. Missing full-packet ADR delivery notes are not a Step 5 finding: +Step 6 owns them after production-census enforcement and final regression. + +#### Step 5 review round 2 + +Fresh GPT-6-astra high and GPT-6.1-sol high reviewers independently reviewed +`94d7d6b..0302d8a`, including implementation and the R1 correction. Both approve; +no runtime finding remains. The runtime reviewer independently passes 92 focused +integration and 97 unit cases, with zero failures/skips. The proof/corpus reviewer +independently passes 92 integration cases and all five corpus guards, including +the final correction below. These are overlapping independent reruns. + +The second reviewer found the Pending-policy catalogue entry still naming the +renamed production-composition test and calling actual subscriptions Pending. +Root verified the surviving test and explicit Pending fixture registrations, +then corrected the reference and separated the policy proof from current Active +bindings. The reviewer rechecked that correction and the 198/120/78 catalogue +counts with no open finding. Release binaries are unchanged. Root's document +check has zero link/fragment/width errors and preserves earlier receipts and ADR +text. Step 5 is closed. Step 6 follows with production censuses, remaining current +carriers, full regression, demo/Lighthouse and PR-head CI verification. + +### P02b-3 Step 6 — production enforcement and PR closeout (2026-10-13) + +**Implementation and both independent review rounds complete.** +[PR #31](https://github.com/HodeTech/LearnStack/pull/31) is open for maintainer +review; merge remains pending. +Step 5 is closed at `ac7bbee` before these changes. This step completes nonempty +production handler/manifest and call-path controls, extends aggregate/audit +coverage to integration handlers, reconciles current carriers and performs full +Release regression plus the real shared demo/Lighthouse lifecycle. The earlier +five delivery/review receipts remain unchanged. + +The authorized Hub dependency pointer is reconciled in the existing sibling +`docs/hub-agent-handoff` branch at `dd243a0`: its P02c-3 dependency now links to +ADR-0046's shared transport-owned inbox. This is a local documentation-only commit; +no Hub runtime, HTTPS surface, branch change, push or PR is part of that update. +The Hub track remains frozen under its existing owner decision. + +#### Step 6 implementation and local proof scope + +The canonical inbox rule now discovers concrete integration handlers in every +production assembly/layer, compares inherited/explicit event contracts with the +actual API's two scoped Active bindings, and checks typed construction, +invocation, generic DI, helper, async and delegate capabilities. Missing assembly, +handler, registration and actual invocation subjects fail the same predicates. +The lifecycle companion pins exact typed unit/frame and raw transaction +completion owners, including site multiplicity and missing-owner controls. +These checks do not infer arbitrary reflection, raw SQL transaction control or +control-flow dominance; the app-role behavioural suites own execution order and +atomicity. They introduce no production runtime hook. + +Audit discovery now fails on assembly/type load errors and classifies each +concrete handler, rather than its shared event. Actual API/Seeder catalogues must +include both Off coordinators even when their operation entries are unchanged; +an omitted-handler control and integration-only module control can fail. +The aggregate constructor-port census includes inherited integration handlers, +fused ports and separate-root controls. It does not infer nested ISender +request dataflow; the real default effects supply the one-root runtime proof. + +The canonical restored-context test checks constructor/callback timing, inbox, +prior context cleanup, effective SystemActor and separate causal human identity. +It also reads the committed outgoing trace, tenant/org and exact nonempty +partition. A new test-only HTTP controller sends the existing provisioning +command through the real API pipeline and production outbox, then reads the +committed quartet from a separate app-role connection. This proves producer +tracing, not authorization or admission of a new production endpoint. + +Self-check corrected test-harness overload/fixture lifetime issues without +weakening assertions. The first full architecture run reported the new partial +HTTP test file missing a source-visible Docker trait; the runtime class already +inherited it from its other part. The method now declares the trait explicitly. +No failing or partial run is counted as passed evidence. + +Local Release proof receipts: + +- Build: zero warnings/errors; solution-wide format verification passed. +- Unit: 1,862; contract: 4; full integration: 1,550, all passed with zero skips. + The final integration receipt includes the new HTTP and strengthened delivery + quartet proofs; its subsequent source change adds only the explicit method + trait. The final full architecture rerun passes all 311 cases, zero skips; + the source convention requires the shared `RequiresDocker` constants. The + combined backend receipts total 3,727 passing cases. Catalogue self-checks + confirm 218 architecture methods and 200 Implemented rules: 121 in that + assembly and 79 elsewhere. Five focused HTTP/context/composition controls + also pass on the final candidate build. +- The unchanged frontend passed its guarded 1,616-case suite, lint and typecheck + in Step 5; those receipts remain applicable, without claiming a Step 6 rerun. +- A fresh isolated development clone at `ac7bbee` ran the full shared demo. Step 6 + changes only tests/docs, so its production runtime is identical. The proof + finished in 580,109 ms with 27 JSON/HTML report pairs, nine URLs and three runs + each: maximum normalized gzip JS 106,573 bytes, maximum CLS 0 and every + automated accessibility score 1. Nine timing advisories remain warnings, not + production p95 claims. Cleanup passed and every originally running external + container was restored. This local receipt is not an exact final-PR-head CI + claim; the required CI job reruns the actual head. +- Relative document links/fragments and 88-column prose checks pass. Earlier + P02b delivery/review receipts and Accepted ADR bodies are byte-preserved; + authorized dated delivery notes are additive. ADR parser/disclosure tests: 7. + +#### Step 6 review round 1 + +Three fresh independent agents reviewed `ac7bbee..82623d8`: GPT-6-astra/xhigh +covered typed invocation/lifecycle capabilities; GPT-6.1-sol/xhigh covered +aggregate/audit discovery, actual roots and HTTP/delivery proofs; GPT-6-sol/high +covered the corpus and governance. Independent runs passed eight capability +cases, twenty aggregate/audit cases and five actual root/HTTP/context cases. +The whole backend/demo receipts above were inspected, not independently rerun. + +They found four enforcement gaps: a same-simple-name integration handler could +hide inside the provisioning exception; a handler could inherit implementation +from a base without the interface; a closed generic helper could hide typed DI +resolution; and EF transaction completion APIs could bypass the typed lifecycle +census. Root read the actual predicates and independently reran both compiled +scratch reproductions, confirming dirty cases were missed and matched controls +were detected. No current production-runtime defect was demonstrated. + +The fix pins the exact provisioning Type and adds independently failing name +collision controls. The capability fixes map inherited implementations, follow +closed generic helper specialization to known DI capabilities and enumerate EF +transaction/facade completion APIs. Existing Facts gain compiled dirty/clean +controls without increasing counts. Root's corrected Release build has zero +warnings/errors and all nine focused guard cases pass, including independent +same-name and recursive/ordinary generic controls. Rerunning the original +compiled capability probe now detects both inherited/generic misses and every +EF completion case, while the actual production baseline remains clean. The +first fix build caught instance-only probe methods that used no instance state; +their instance task fields preserve the intended inherited call shape without +an analyzer waiver. Full architecture passes all 311 cases with zero skips; +solution-wide format verification and document checks pass. The fix changes +tests/docs only; prior full unit/contract/integration and demo receipts retain +their explicitly recorded scope. +Current README, processor and skill guidance also retained delivered-as-pending +claims; those verified carriers are corrected. The new ADR-0046 delivery note +stays only in its dated Amendment 3, with mutable-index navigation; no new top +navigation is inserted into the Accepted body. + +#### Step 6 review round 2 and local closeout + +Three new independent agents reviewed the full `ac7bbee..84995d3` Step 6 range: +GPT-6-astra/xhigh covered capability/lifecycle enforcement; GPT-6.1-sol/xhigh +covered audit/aggregate/root/tracing proofs; GPT-6-sol/high covered the corpus. +The technical slices approve without an actionable finding. Independent runs +pass eight capability cases, twenty aggregate/audit cases and five real root/ +HTTP/delivery cases, all with zero skips. A separate compiled probe detects +seven additional dirty generic/inherited/async/iterator/closure/local-function/ +permutation shapes and leaves two unused-handler-argument controls clean. + +The corpus reviewer approves with minor current-pointer corrections in +architecture 10/15, Standards 20 and the producer catalogue entry. Root verified +and fixed them, preserving Step 5 as dated evidence and linking current status +to this record; the reviewer rechecked the edits. A final glossary clarification +states context precedes business construction/DI resolution, while metadata +registry discovery happens earlier. Root verified the actual executor ordering. +Five corpus consistency cases and link/fragment checks pass; these final fixes +change documents only. No open verified finding remains from either round. + +All six P02b-3 steps completed their implementation commits and both independent +review rounds. The local full-suite, focused fix and demo receipts retain the +explicit scopes above; no final-head CI success is inferred from them. The PR +handoff below will own the live seven-check verification and maintainer review. +This completes P02b-3 implementation and local review, not a merge or the whole +Phase 02b. P02b-4–8 remain unimplemented; the next packet is P02b-4's background- +job decision pass, including the reserved ADR-0047 before runtime work. + +### P02b-3 PR handoff (2026-10-13) + +[PR #31](https://github.com/HodeTech/LearnStack/pull/31) compares `development` +with `main`. The six-step implementation and both independent review rounds +are complete; initial enforcement is `82623d8`, verified first-round fixes are +`84995d3`, and local review/document closeout is `a8e9c9d`. The +[Step 6 record](#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) +owns scoped local validation, independent receipts and corrected findings. +This handoff changes documents only and introduces no new implementation scope. + +The [live PR checks](https://github.com/HodeTech/LearnStack/pull/31/checks) own +exact-head CI results. All seven required contexts remain active with strict +up-to-date protection: backend, frontend, secret scan, meta, backend integration, +OpenAPI diff and public demo/Lighthouse. The initial published head started CI; +this documentation receipt receives its own final-head run. No pending or +superseded run is counted as success. Maintainer review and merge remain pending. + +P02b-4's background-job decision pass is next after this packet's review. It +must draft and obtain approval for the reserved ADR-0047 before runtime work. +Authentication, realm reconciliation, BFF/session and phase exit remain +P02b-5–8; Phase 02b is not complete. + +### P02b-3 external review follow-up (2026-10-15) + +The external PR #31 review compares `6a277770..683ea537`. Findings are review +inputs, not authority; each was checked against current code, Accepted ADR-0046 +and the approved P02b-3 package. Earlier dated receipts and Accepted ADR bodies +remain unchanged. No new runtime decision, migration, role, endpoint or replay +permission is introduced by this follow-up. + +Verified corrections: + +- Live P02b-2 pointers now name its merge closeout; P02b-3 status points to Step 6 + and the PR handoff. Shared handler/audit remarks, canonical Inbox prose, + glossary, decision index and review guidance describe delivered behavior. +- Failed snapshot reads no longer advance the successful refresh timestamp. + Recovery is tested after restoring SELECT without advancing the fake clock. + A normal claim's uncertain completion records `execute:unknown:commit`; + expired-lease recovery retains `recovery:unknown:expired`. +- The existing lifecycle census now enumerates actual tenant-announcement calls + through both the interface and concrete adapter, including inherited, async + and delegate shapes. Missing real calls and planted extra callers fail it; + ADR-0040's eight logical setter paths remain unchanged. +- Catalogue discovery reads the declared Status block rather than any bold + Implemented mention elsewhere. A Registered entry with misleading Implemented + prose is a negative control. The scheduling-index proof reads actual production + eligibility/admission constants and includes time, lease, locking and LIMIT. +- Removed the duplicate executor registration. Added actual-store wrong-kind + controls and producer terminal audit cases for absent versus invalid causal IDs. + Composition/factory remarks and seed-audit count explanation are aligned. + +Verified dispositions for the remaining review items: + +| Review item | Disposition and current evidence | +|---|---| +| Health should tolerate one retained overrun | Rejected: the approved P3 runtime table and Standards 10 require unhealthy while any callback/cleanup overrun remains. Capacity four separately stops acquisition; ordinary running work is not this count | +| SQL NULL causal IDs should invalidate a valid original trace | Rejected: optional absence is valid; only empty IDs are invalid. Real terminal regressions now pin both outcomes; audit prose explicitly distinguishes absence | +| Unclassified delivery log should include raw handler type | Kept minimized: existing delivery proofs prohibit handler type text. The scheduler's correlated attempt already names finite subscription/event labels and outcome; no raw stored type or exception detail is added | +| Queue snapshot lacks a dedicated aggregate index | No schema change: the exact count/minimum covers all unfinished ledger states at the approved cadence, under the SQL deadline. Claim-index applicability is now tested against actual selection. This is not a claim of constant-time counting; Phase 11's existing retention/scale work owns aggregate cost tuning | +| Initializers need ordinary AggregateConflictException handling | No defect: initialization and registration share the concept lock, with both winning orders tested. Unexpected ID/constraint faults abort and follow subscriber retry | +| Initializer exception should include Result.Error detail | Kept bounded: actual consumers throw fixed effect-specific failures, tested on refusal. Raw details do not belong in delivery diagnostics | +| No-change initializer audit lacks a business snapshot | Accepted behavior: preservation attempts remain classified MUST successes without mutation or entity snapshots; creation separately requires captured Active state | +| Preserved non-Active built-in causes untested Seeder divergence | Rejected: Draft/Deprecated/deleted and successor/history tests preserve every business field, then refuse the exact Active demo postcondition; the seed skill already documents it | +| Seed skill implies old built-in registration | Rejected: its write-order step explicitly uses separate shared initialization commands and distinguishes authored registration/publication | +| Concept-history SQL duplicates mapping names | No refactor: owner-local constant SQL intentionally reads tombstones through explicit tenant predicates and RLS; actual schema/history/isolation/race tests cover it | +| Five-second acceptance/inbox command timeouts | Match the approved fixed SQL bound. No shared mutable configuration contract is promised | +| Per-subscription acceptance statements | Two declared subscriptions; separate equality reads deliberately obtain a fresh READ COMMITTED snapshot after a concurrent conflicting insert | +| Repeated writable-announcement checks | Deliberate boundary validation: detect changed transaction, context generation and tenant/org before SQL and commit | +| Retained envelope needs an owner trigger | Existing exact column grants implement the approved immutability boundary and have real-role forbidden-write controls. Inbox's owner trigger is a separate marker contract; no new owner-level policy is inferred | +| Capture exclusions name currently unmapped SQL machinery | Intentional ADR-0044 Amendment 11 policy; mapped probe types exercise actual interceptor exclusions even though production storage uses SQL | +| Integration friend assembly exposes all internals | C# friendship is assembly-wide; it enables actual internal execution/lifecycle proofs without exposing production APIs | +| DeliveryAttemptTests name, worker-test title and options binding | Naming/configuration suggestions only: the tests deliberately cover attempt cancellation control and exactly one OutboxProcessor; dispatcher options use approved fixed bounds, not a promised configuration section | +| Lifecycle/owner Facts need separate canonical rows | Existing Integration_Event_Handlers_Use_InboxGuard entry already names both companions. The expanded announcement proof is recorded there; no duplicate rule is added | +| Producer/subscriber rules share TerminalOffenders | Intentional common witness authority, with distinct producer and subscriber dirty controls; not a missing independent failure path | +| Bare First/Single in planted controls | Missing fixtures fail loudly rather than pass. No absent-subject success was reproduced; production subject and signature checks remain explicit | +| Nonempty/derived-capability assertions | Existing required-type, method, signature and exact-owner checks fail before mutation loops when a production subject disappears | +| Ignored consumer ActivityContext.TryParse result | Immutable envelope construction and the retained-envelope reader validate the trace before this call | +| Internal delivery partition lane needs explanation | Existing comment explains acceptance/execution separation and canonical UUID collision avoidance; the approved local-ordering bound remains | +| Subscription grammar copies need consolidation | Equivalent current contract, already tested through registry, storage and witness admission; optional refactor, no divergent behavior demonstrated | +| CancellationToken.None after commit admission | Accepted cancellation/unknown-outcome tradeoff; do not replace the approved sticky gate and reconciliation contract | +| Decision-index amendment numbering gaps | Selected navigation links do not enumerate all historical amendments; complete dated sections remain in each ADR | +| Built-in enumeration, meter factory and health Degraded suggestions | Optional abstractions/observability policy changes; no demonstrated current defect in the approved two-default, binary-health contract | +| Earlier P02d seed receipts and ADR-0046 top navigation | Historical evidence stays intact. P02b-3 Step 5 owns current generated built-in identities and semantic verification; current index links the preserved ADR-0046 Amendment 3 | +| Replay/privileged execution and future strict ordering | Existing Phase 03/new-decision triggers remain explicit; no operator entry point or privileged subscription is activated | +| Retention, inbox-trigger purge exception and erasure inventory | Existing Phase 11 retention/grant decision and P02b-8 inventory obligations remain; this follow-up does not grant purge or bypass access | + +The original source head `683ea537` passed all seven required CI contexts in +[run 38424753309](https://github.com/HodeTech/LearnStack/actions/runs/38424753309). +Downloaded exact-head artifacts corroborate 1,862 unit, 311 architecture, +4 contract and 1,550 integration tests (3,727 backend; zero failed/skipped), +1,556 web plus 60 SDK tests, and 27 admitted browser/Lighthouse report pairs. +Maximum normalized gzip JS was 106,551 bytes, maximum CLS 0, all automated +accessibility scores 1; the proof finished in 621,330 ms with successful cleanup. +These are evidence for the reviewed head, not this correction's final-head CI. +Correction validation: Release solution build has zero warnings/errors; +solution-wide format and diff checks pass. All 1,862 unit, 311 architecture, +4 contract and 176 Docker-free integration tests pass, with zero skips. Focused +real PostgreSQL validation passes 73 worker/store/terminal cases plus the actual +candidate-index case. Seven ADR parser cases and the document link/fragment, +prose-width and historical-record checks pass. At this initial receipt the +catalogue counted 218 methods and 200 Implemented rules (121 architecture, +79 elsewhere); the round-2 correction below records the accurate current count. + +The first full unit run exposed a cross-test metric listener race (one observed +failure counted another test's same-name meter). The corrected test filters its +owned Meter by reference, disposes its provider and verifies an unrelated meter +really emits while contributing zero to its count; the actual store must still +count exactly one. The focused case and corrected full unit suite pass. +Fresh review rounds and correction-head CI remain pending at this initial receipt. + +#### External follow-up review round 1 + +Two fresh independent agents reviewed `683ea537..586ed75`. GPT-6-astra/xhigh +reviewed runtime, security, isolation, lifecycle and performance, independently +passing 72 focused PostgreSQL integration cases and three architecture cases. +GPT-6.1-sol/xhigh reviewed guards, tests, governance and all changed documents, +independently passing eight architecture cases, 18 recorder unit cases and the +real PostgreSQL candidate-index case. All runs have zero failures/skips. + +The runtime review found no actionable issue. The proof/corpus review found two +Minor wording errors: the new machinery-write prohibition inadvertently required +all business handlers to send commands, although the Accepted classified direct +one-root path remains valid; and Security retained an obsolete one-setter count. +Root verified both against ADR-0044 Amendment 8/ADR-0057 and current transport +calls. The fixes remove the command-only restriction and describe the actual +announcement seam without a misleading count. The reviewer rechecked both and +approved with no remaining finding. Document checks pass; this fix changes no +runtime or test source. Round 2 and correction-head CI remain pending here. + +#### External follow-up review round 2 and closeout + +Two new independent agents reviewed `683ea537..d8c766f7`. GPT-6-astra/xhigh +reviewed technical correctness and proofs, independently passing 72 PostgreSQL, +eight architecture and 18 recorder unit cases. GPT-6-sol/high reviewed the corpus, +governance and test claims, independently passing seven focused architecture cases. +All runs have zero failures/skips. Runtime, isolation and lifecycle were approved. + +The corpus reviewer found a pre-existing counting omission in the touched parser +scope: two real integration rules used a bold `Implemented, twice.` status and +were excluded from the declared-state count. Root verified both exact-name +methods in `TenancySchemaTests` and `TenantIsolationHttpTests`, normalized the +statuses to a bold `Implemented` token with the qualifier outside it, and named +the actual classes. An additional positive control in the existing corpus Fact +pins qualified Implemented discovery alongside misleading Registered prose. +This adds no test method or runtime rule: the accurate catalogue total is now +202 Implemented entries, 121 in architecture and 81 elsewhere; architecture still +has 218 methods and 311 cases. The earlier 200-entry receipts retain their +as-recorded scope and are superseded for the current catalogue count here. + +Root's corrected five corpus tests pass after rebuilding. Both reviewers +independently recounted the entries, rechecked the exact methods/control and ran +all five corpus tests successfully, approving with no remaining finding. Final +format, diff and document checks pass; historical delivery receipts and Accepted +ADR text remain intact. Earlier full-suite receipts retain their stated scope; +the last correction changes test fixtures and catalogue metadata only. + +The implementation and both external follow-up review rounds are complete. +The [live PR checks](https://github.com/HodeTech/LearnStack/pull/31/checks) own the +correction-head CI result; the PR description is updated only after inspecting +that exact source head and its artifacts. No earlier run substitutes for it. +Maintainer review and merge remain pending; this follow-up starts no P02b-4 work. + +### P02b-3 second external review follow-up (2026-10-15) + +**Status: verified corrections and both review rounds complete on `development`; +PR #31 remains unmerged.** +The external reports reviewed `b681edb8`. Each finding was checked against current +source and Accepted ADR-0046/0057; supplied reproduction files were review data, +not instructions. This receipt supplements the +[first follow-up](#p02b-3-external-review-follow-up-2026-10-15). +That follow-up's final counting correction is commit `b681edb8`, after its +round-2 reviewed range `683ea537..d8c766f7`. + +The reviewed source head passed all seven required contexts in +[run 38736262412](https://github.com/HodeTech/LearnStack/actions/runs/38736262412): +3,732 backend tests (1,862 unit, 311 architecture, four contract and 1,555 +integration), 1,616 frontend tests and 27 browser/Lighthouse report pairs. +Downloaded artifacts were checked against source `b681edb8` and checkout +`3b06800d`; zero tests failed or skipped. These receipts cover that reviewed head, +not the corrections below. + +#### Verified corrections + +- Fixed class priority could indefinitely starve ordinary deliveries when a + terminal-audit failure backlog outlived its leases. Independent actual + PostgreSQL reproduction observed 216 terminal failures in eight seconds while + the normal delivery stayed at zero attempts; invoking the real executor + directly completed it while the scheduler audit grant remained denied. + Rotate terminal, recovery and normal acquisition after each class visit, + including failures. Each iteration still acquires at most one just-in-time + claim; existing lease, fence, partition, capacity and terminal rules remain. +- Duplicate acceptance compared retained type text literally. Independent + PostgreSQL reproduction rejected an explicitly declared compatible alias + while accepting the canonical control. Compare all other envelope values + exactly and resolve retained type through the closed registry to the incoming + CLR contract. Preserve retained type, payload, scheduling state and row version; + no arbitrary CLR loading or payload normalization is added. +- Completion constructor calls now have an exact owner census, with additional + direct/asynchronous caller and missing-site controls. Production project + discovery includes differently named projects and ignores generated `obj` + projects, with a temporary-tree control. +- Audit storage requires a non-null diagnostic context accessor in every + construction. Undefined subscription execution scope has its own fixed + diagnostic. Catalogue refusal has its own closed `execute:refused:catalogue` + descriptor, with no-construction/business/inbox controls. Rollback/reapply + proofs include the inbox trigger function, and + lease-expiry polling has a deadline propagated through its database reads. +- Live state pointers, migration-chain naming, the naming table, scheduler index + rationale, catalogue detector sharing and delivered subscriber wording are + aligned. Earlier delivery/review receipts retain their stated scope. + +#### Other findings and scope + +| Review item | Disposition and current evidence | +|---|---| +| Unknown-COMMIT failure counter | Accepted process observation, not durable business accounting; Standards 10 explicitly documents that an unknown COMMIT may have succeeded | +| Startup should log exception types/details | Existing minimized dispatcher diagnostics deliberately omit provider/exception type text; current fixed failures and health remain bounded | +| Additional completion owner assertion | Already enforced by prepared-owner creation and `RequireLive`, including before fencing; no reachable missing check | +| Suppress post-acknowledgement disposal errors | Deliberately pinned by `UnitOfWorkCommitCleanupTests` and `DeliveryCommitGateTests`; delivery interprets definitive commit evidence. Changing ordinary request response policy needs a separate design | +| Acceptance round trips | Two current subscriptions; the separate equality statement intentionally gets a fresh READ COMMITTED snapshot after a conflicting INSERT | +| Duplicated SystemActor derivation | Not present: the coordinator derives it; producer envelope creation consumes captured causal context | +| Add a 200-character subscription ID limit | No such bound exists in the Accepted grammar/package; the cited 150-character cap is specific to killswitch keys. Finite registered IDs remain trusted configuration; no new numeric contract or migration is inferred | +| Initializer conflict catch | Shared concept lock plus any-history refusal handles both registration orders; unexpected ID/constraint faults must roll back | +| Taxonomy verification needs set equality | Unique stored band keys plus equal count and complete membership already establish unordered equality; real shape/history mismatch controls remain | +| Two stores must use separate pools | Existing contention proof promises two stores and uses distinct live connections/transactions; separate pools/processes are not its claim | +| Raise shortened fixture deadlines | No reproduced deadline defect; keep the intentionally shortened valid test bounds and controlled shutdown clock | +| Health should count ordinary awaited callbacks | The published count tracks retained overruns/cleanup; the ordinary serial await does not publish it. Any retained task remains unhealthy, with acquisition stopping separately at capacity four | +| Additional commit-owner abstraction/refactors | Exact declared owners remain intentional; no new helper, outcome router or diagnostics abstraction is needed for these corrections | +| Internal replay needs a production audit writer | Still unwired; Phase 03 owns operator permission/HTTP and its classified audit path. No new replay entry point is activated | +| Friendship, cross-cutting composition and N envelope copies | Existing explicit internal proof access and composition are unchanged; the accepted package already inventories per-subscription retained copies for P02b-8/Phase 11 | +| Further policy parity/concurrency/census optimization | Existing real-role policy, independent-connection contention and planted production controls remain; no missing behavior was demonstrated beyond the corrected discovery/constructor gaps | + +Correction validation: Release solution build has zero warnings/errors; full +format and diff checks pass. All 1,864 unit, 312 architecture, four contract and +176 Docker-free integration cases pass with zero skips. Focused integration +passes 31 acceptance, 23 actual-worker and 113 store/audit/migration/composition +cases. The latter includes three Docker-free composition cases already covered +by the full Docker-free receipt; these are not extra distinct tests. + +The fixed-priority and literal-type-equality mutants each fail their new +PostgreSQL regression. Original source was restored, the solution rebuilt and +both regressions rerun successfully. Seven ADR parser cases pass; document +checks validate 2,822 relative links and 631 fragments, prose width, historical +receipts and earlier Accepted ADR text. The catalogue counts 219 methods and +202 Implemented rules (121 architecture, 81 elsewhere). + +The first architecture run correctly rejected the moved claim calls because its +exact privileged-consumer list still named `ExecuteAsync`. The corrected census +requires the private `ClaimNextAsync(string[], CancellationToken)` signature +instead, with an extra-overload control. It grants no whole-type exemption; the +full corrected architecture suite passes. + +Two fresh independent review rounds and correction-head CI remain pending at +this initial receipt. Earlier CI evidence does not substitute for that head. + +#### Second follow-up review round 1 + +Three fresh agents reviewed `b681edb8..92e296ed`. GPT-6-astra/xhigh reviewed +runtime, transaction/cancellation ownership, fairness and diagnostics; it passed +23 worker, 25 diagnostic/registry unit and 40 dispatcher architecture cases. +GPT-6.1-sol/xhigh reviewed guards, proofs, governance and corpus consistency, +independently passing all 312 architecture cases. GPT-6-sol/high reviewed alias +acceptance, concurrency, RLS and audit composition; it passed 31 acceptance, +23 worker and 25 focused unit cases. All runs have zero failures/skips. + +All three approve with no verified actionable finding. The older prefix-scoped +request/handler discovery scans are separately documented scopes, not a claim +of this shared production census. No new rule or deferred decision is silently +introduced. Round 2 and correction-head CI remain pending here. + +#### Second follow-up review round 2 + +Two new agents reviewed `b681edb8..a95ae2ec`. GPT-6-astra/xhigh reviewed runtime, +acceptance, isolation and audit composition, independently passing 163 focused +integration, 43 unit and 41 architecture cases. GPT-6.1-sol/xhigh reviewed proofs, +corpus, ownership controls and governance, independently passing all 312 +architecture and 25 focused unit cases. All runs have zero failures/skips. +Both approve with no verified actionable finding against that correction scope. + +The runtime reviewer separately identified the unchanged producer worker's same +fixed class-priority pattern. This concern was not counted as a verified finding +against the reviewed diff. The adjacent correction below owns its PostgreSQL +verification and any resulting implementation/review evidence; the delivery and +alias approvals do not substitute for reviewing that change. + +#### Adjacent producer fairness correction + +**Status: correction, validation and both independent review rounds complete.** The +producer worker's strict terminal/recovery/normal priority reproduced the same +starvation in real PostgreSQL. With 100 terminal requests and only the scheduler +role's audit INSERT denied, the regression passed its lease-cycle failure check +then failed at the independent normal-publication/recovery deadline. The baseline +producer source was unchanged. The fixed worker uses the same bounded rotating +acquisition as the subscriber; its exact private credential-consumer signature +and extra-overload control replace the prior call-site requirement. + +Accepted ADR-0046 requires continued unrelated eligible work and does not require +strict claim-class priority. This is an implementation correction, with no new +ADR, grant, lease, attempt, terminal-audit or publication-ownership contract. +The regression requires actual unwrapped transport acceptance, no business +construction, unchanged recovery attempts and every failed terminal request +remaining intact without terminal audit while the grant stays denied. + +The unchanged baseline fails the new progress regression after its first +lease-cycle check. After correction the full actual producer suite passes +19 cases, zero failed/skipped; the new regression passes in 10.634 seconds. +Release solution build has zero warnings/errors. All 1,864 unit, 312 +architecture, four contract and 176 Docker-free integration cases pass again; +full format/diff and document checks pass. Two fresh independent reviews of +this additional producer correction remain pending at this initial receipt. + +Producer correction review round 1 used two fresh agents against +`a95ae2ec..bd51a974`. GPT-6-astra/xhigh passed 80 producer/store integration, +40 architecture and 20 ownership/bounds/observation unit cases. +GPT-6.1-sol/high passed all 19 producer and 40 architecture cases. Both approve +without a verified actionable finding; no fix was needed before round 2. + +Two new agents independently reviewed the same frozen correction in round 2. +GPT-6.1-sol/xhigh passed 80 producer/store integration, 40 architecture and +20 focused unit cases; GPT-6-astra/high passed all 19 producer and all 312 +architecture cases. Both approve with no verified actionable finding. All +independent runs have zero failures/skips; neither round relies on earlier +scope approvals to cover the producer change. + +The verified delivery/alias corrections and the adjacent producer correction are +complete, each with both fresh independent review rounds. Final document/diff +checks pass; the earlier receipts retain their exact as-of scope. The +[live PR checks](https://github.com/HodeTech/LearnStack/pull/31/checks) own +correction-head CI; its exact source and artifacts must be inspected before the +PR description records success. Maintainer review and merge remain pending; +this correction starts no P02b-4 work. + ### Durable outbox dispatch P02b-1 Steps 2/3 implement `IOutbox`, replace Packet 3's step-7 shell with an enrollment/invariant guard, and supply owner-side domain collection and the first publisher. P02b-2 delivers hosted producer dispatch and production-census -enforcement; all four steps and both independent review rounds are locally -complete. The [PR handoff](#p02b-2-pr-handoff-2026-10-12) owns current CI/review/merge -state; consumer delivery remains P02b-3. +enforcement; all four steps and both independent review rounds are complete and +merged. The [merge closeout](#p02b-2-merge-and-closeout-2026-10-13) owns final +verification. P02b-3 delivers independent subscriber execution and enforcement; +its [Step 6 record](#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) +owns local validation and the [PR handoff](#p02b-3-pr-handoff-2026-10-13) owns CI/review status. The table shipped in Packet 6. A handler calls `IOutbox.EnqueueAsync` and the row commits on the same **transaction** @@ -2083,7 +3458,7 @@ formulation [ADR-0033](../decisions/0033-audit-durability-model.md) withdrew. [A accepts immediate INSERT through that connection/transaction. Step 7 is an enrollment/invariant guard; the transaction owner drains domain events before MUST audit and commit. Raw transactions without an enrolled producer frame -cannot enqueue. Production delivery wiring remains P02b-3. +cannot enqueue. P02b-3 delivers production acceptance and subscriber execution wiring. #### Process topology @@ -2107,8 +3482,8 @@ Dapr trigger verbatim — so it stays with Phase 11. the pattern `Platform_DataSource_Resolved_Only_By_PlatformAdminScope` already holds. `LearnStack_OutboxAdmin_Role_OnlyUsedBy_OutboxProcessor` is **Implemented** with Step 4's nonempty credential caller census and passing planted controls; - all four steps and both independent review rounds are locally complete. - The [PR handoff](#p02b-2-pr-handoff-2026-10-12) owns CI/review/merge state. + all four steps and both independent review rounds are complete and merged. + The [merge closeout](#p02b-2-merge-and-closeout-2026-10-13) owns final verification. - The three carriers that say the credential lives in "the worker host" — [Database Standards § Database roles](../standards/05-database.md#database-roles), `architecture/15` and `.env.example` — are derived cells, not a topology decision; @@ -2177,7 +3552,7 @@ every terminal path clears transaction-scoped source/collector state exactly onc #### The consumer side -**Accepted contract; implementation pending P02b-3.** Construction-free manifests +**Delivered by P02b-3 Steps 1–6 under Accepted ADR-0046.** Construction-free manifests carry stable versioned subscription IDs, compatible aliases and required Pending/Active readiness. Active publication durably accepts every matching delivery without running business handlers. The independent executor owns retry @@ -2212,7 +3587,7 @@ failure, terminal and lease-loss diagnostics, immutable backlog/eligible-age/ deferred/heartbeat snapshots and registered `outbox` health. No metric has a tenant label; event-type labels come only from the bounded registry. Required Pending rows, completed and terminal rows are excluded from normal eligibility. -P02b-3 owns independent subscriber terminal/dedup signals and minimizes the old +P02b-3 delivers independent subscriber terminal/dedup signals and minimizes the old transport diagnostics. Separate producer/subscriber failure signals describe separate budgets. Health is registered, not mapped; `/healthz` stays liveness and Phase 11 owns `/readyz` and production alert windows. @@ -2475,17 +3850,13 @@ The audit pipeline lit up in [ADR-0033](../decisions/0033-audit-durability-model.md). This phase adds the entry points it does not reach, and both of Packet 9's hand-offs land here. -- **The delivery path has no transaction or audit frame today.** +- **The delivery path owns its transaction and audit frame.** [ADR-0040](../decisions/0040-ambient-unit-of-work.md#consumers-do-not-have-a-request-and-still-need-all-of-this) - assigns `Begin → SetTenantContext → handler → Commit` to this phase; the current - transport restores only the ambient accessor and resolves a fresh handler scope. - Adding that transaction alone would leave an audit gap: - only `AuditLogBehavior` declares intents, only `TransactionBehavior` flushes them, and - the catalogue is keyed by request type — so a consumer that writes its own aggregate - through its `DbContext`, which is the shape the corpus teaches as canonical, has its - changes would be captured into a buffer that holds no intent naming its type. - **No row would be written and nothing would fail.** That is the window - [ADR-0033](../decisions/0033-audit-durability-model.md) says must not exist. + assigns `Begin → SetTenantContext → handler → Commit` to this phase. P02b-3 + prepares the app-role transaction, restores validated event context, claims the + inbox and enters the common audit owner before handler construction. Handler + identity is classified even for a direct business write; an unregistered handler + is refused. This closes the old accessor-only transport's missing audit frame. **G6's common-owner contract is Accepted under ADR-0057**; P02b-3 implements it in production delivery under ADR-0046. Its proof is a MUST-classified direct-write consumer, @@ -2495,19 +3866,16 @@ points it does not reach, and both of Packet 9's hand-offs land here. [ADR-0033 Amendment 2 § 2](../decisions/0033-audit-durability-model.md) and [ADR-0044 § 4](../decisions/0044-audit-write-path.md) both say only the owning frame writes, signals the boundary, reconciles and clears, and a joiner does none of it. - ADR-0057 accepts common entry coordination for **G6**. The remaining P02b-3 - production wiring must repair this baseline: `AuditLogBehavior` takes its owner + ADR-0057 accepts common entry coordination for **G6**. `AuditLogBehavior` takes + its owner flag from an audit-frame counter rather than from unit-of-work ownership, and [Audit Subsystem](../architecture/31-audit-subsystem.md) records that split as deliberate for the HTTP path — at step 3 no transaction is open to answer the - question. The two coincide today only because the pipeline is the only thing that - opens a transaction with a command inside it. A transport-opened transaction breaks - the coincidence: the inner command becomes a unit-of-work joiner while still being the - outermost audit frame, so it reconciles before the delivery commits, writes its MUST - row standalone as `failed` for a change that then commits, and clears the buffer the - owner would have flushed. ADR-0044/0033 now clarify the common owner; P02b-1 - proves it through a - controlled direct fixture before production transport wiring in P02b-3. + question. P02b-3 explicitly opens the outer audit frame around subscriber execution; + nested commands join it and cannot reconcile or clear before delivery completion. + The shared transaction owner flushes MUST rows before commit, then the outer audit + owner alone reconciles and clears. P02b-1's controlled fixture and P02b-3's actual + consumer and direct-write proofs enforce ADR-0044/0033's common ownership. - **The discriminating case.** The pair this phase used to name — a consumer's MUST-class command recording `success` on the delivery's transaction, and a rolled-back delivery recording it standalone as `failed` — passes under **both** @@ -2520,8 +3888,8 @@ points it does not reach, and both of Packet 9's hand-offs land here. short confined-role transaction. `WriteEventTerminalAsync` validates its restricted `EventTerminalTransition` witness against the actual fenced UPDATE and draft identity. Audit failure rolls back state and retries terminalization, - without another publication. The corresponding subscriber operation remains - P02b-3. Register each planned off-path slug only with its writer. + without another publication. P02b-3 Step 4 delivers the corresponding subscriber + operation and its off-path registration with the writer. - **Routine dispatcher bypass — Accepted ADR-0046.** The confined credential, exact grants, bounded diagnostics and atomic terminal audit explicitly exempt routine polls/claims/acknowledgements/retries from per-invocation security audit. @@ -2533,12 +3901,11 @@ points it does not reach, and both of Packet 9's hand-offs land here. actor the finding on that tier, so the realm integration above populates it from the validated token's user — the observable that also proves the assertion middleware runs after authentication. -- **Causal actor and causation.** The envelope carries both and the outbox row has - columns for both, and neither reaches a consumer's audit row or a follow-up outbox - row. ADR-0038 promises the causal actor "is retained separately as causal audit - metadata" and nothing retains it. Which metadata keys a composed row may carry is part - of the Accepted **G6** producer/common-owner contract; production delivery - supplies the trusted incoming event in P02b-3. +- **Causal actor and causation.** P02b-3 supplies the validated incoming event ID + to consumer audit intents and follow-up outbox production; causal actor remains + separate from effective SystemActor. Nested commands inherit these identifiers, + while a newly drained domain event supplies its own causation ID. The Accepted + **G6** producer/common-owner contract owns these bounded metadata keys. - **Classification for the operations this phase introduces** — the sample flow's command, the first authenticated read, login, logout and refresh — registered in their owning module's catalogue source and matrix before their handlers ship, because an @@ -2663,7 +4030,8 @@ command. The phase retains zero skips, per assembly. delivery ledger: stable subscription identity, readiness, durable acceptance, attempts, cancellation, terminal transitions and targeted replay. Canonical inbox/ledger - DDL remains P02b-3 work; acceptance claims no consumer runtime. + DDL and consumer runtime are delivered by P02b-3; acceptance of the ADR alone + did not claim implementation. - **ADR-0047 Accepted** — the background-job runtime, with the dated ADR-0003 Amendment its storage decision owes. - **The dated amendments the register names**, each Accepted before the code it governs: @@ -2717,7 +4085,8 @@ command. The phase retains zero skips, per assembly. - The registered, unmapped `outbox` health check and accepted producer event metrics in [Observability Standards](../standards/10-observability.md#required-metrics), - including bounded immutable snapshots. Subscriber signals remain P02b-3. + including bounded immutable snapshots. P02b-3 Step 4 delivers the separate + subscriber signals and registered, unmapped `delivery` health check. - `UseAuthentication` and `UseAuthorization` at their two pinned positions, the claim signal populated at the resolver's reserved site, one JWT authority for `/api/v1/*` with its full validation contract written into @@ -2796,9 +4165,12 @@ shape depends on an open gate say so, and are written when that gate is Accepted P02b-2-owned dispatcher criteria below are met by the [Step 4 local closeout](#p02b-2-step-4--enforcement-and-closeout-2026-10-12). -The combined HTTP/event-delivery tracing criterion still requires P02b-3 -consumer execution; the whole phase is not complete. The -[PR handoff](#p02b-2-pr-handoff-2026-10-12) owns CI/review/merge state. +P02b-3's +[Step 6 proofs](#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) +also cover the combined HTTP/event-delivery tracing criterion: actual committed +rows match the enqueuing activity, announced tenant/org and exact nonempty +partition. P02b-4–8 remain unimplemented; the whole phase is not complete. The +[merge closeout](#p02b-2-merge-and-closeout-2026-10-13) owns final verification. - An aggregate change and its outbox row commit together and roll back together, asserted as `learnstack_app`. An enqueue whose event `TenantId` differs from the diff --git a/docs/standards/02-backend-coding.md b/docs/standards/02-backend-coding.md index 88513222..989b544d 100644 --- a/docs/standards/02-backend-coding.md +++ b/docs/standards/02-backend-coding.md @@ -331,6 +331,13 @@ decides what happens *inside* the steps ADR-0033 named; the order is untouched: `IAuditStateCapture.Clear()`; the outermost audit coordinator clears capture in its `finally` ([ADR-0044 § 4](../decisions/0044-audit-write-path.md)). + Under [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md), + `DeliveryExecutor` enters this same coordinator before constructing a concrete + subscriber. Each handler type is classified, even an `Off` coordinator whose + nested commands own resource audits. Modules never commit the delivery frame; + the common owner drains events, writes MUST audit, applies the final fence + and admits completion through the timeout gate. + **Fail-closed is narrower than it reads.** An **in-transaction** MUST-class write that fails rolls the operation back and answers `503 audit_unavailable`. A **standalone** MUST-class write that fails changes diff --git a/docs/standards/05-database.md b/docs/standards/05-database.md index 9cbb2dc6..87869b89 100644 --- a/docs/standards/05-database.md +++ b/docs/standards/05-database.md @@ -41,8 +41,9 @@ PostgreSQL schema, EF Core, and migration conventions. - Schema migrations live with the owning module. With [P02d-1 complete](../roadmap/phase-02d-walking-skeleton.md#delivery-record-p02d-1), -five migration chains exist: Tenancy, shared persistence, Customization, Audit and -Education. They contain twenty-one data tables and five migration-history tables. +five migration chains exist: Tenancy, Platform, Customization, Audit and +Education. P02b-3 Step 1 brings their inventory to twenty-three data tables and +five migration-history tables. ## Naming @@ -62,6 +63,7 @@ Education. They contain twenty-one data tables and five migration-history tables | Functions | `fn_` | `fn_set_updated_at` | | Read-model tables | `public__` | `public_education_course_summaries` | | Outbox | `outbox_messages` (global) | | +| Shared inbox / delivery ledger | Tuple primary key; [canonical DDL](#inbox) | `inbox_messages` / `event_deliveries` | `learnstack_` prefix reserved for system-wide objects (roles, extensions). @@ -539,8 +541,8 @@ migration states which one its table is. | Class | Rule | Tables | |---|---|---| -| **Tenant-owned, org-scoped** | The full template above: `ENABLE` + `FORCE`, one permissive policy `AND`-ing the tenant term with the organization term, explicit `WITH CHECK`, **and** the two `AS RESTRICTIVE` `UPDATE` / `DELETE` guards | any domain table carrying `organization_id`, plus `tenant_settings` — the only org-scoped table in the Packet 6 set — and `audit_log`, which Packet 9 added ([ADR-0044 § 9](../decisions/0044-audit-write-path.md)); P02d-1 Step 2 adds `courses`, `lessons` and both translation tables | -| **Tenant-owned, tenant-wide** | The same shape with the organization half of the predicate omitted, and therefore **no** restrictive guards — there is no organization to guard | `organizations`, `tenant_domains`, `tenant_locales`, `tenant_feature_flags`, `platform_entitlement_cache`, `idempotency_keys`, `outbox_messages`, `tenant_content_types`, `tenant_level_taxonomies`, `tenant_level_taxonomy_items`, `customization_generations`, and `audit_config`, which Packet 9 adds ([ADR-0044 Amendment 1](../decisions/0044-audit-write-path.md)) | +| **Tenant-owned, org-scoped** | The full template above: `ENABLE` + `FORCE`, one permissive policy `AND`-ing the tenant term with the organization term, explicit `WITH CHECK`, **and** the two `AS RESTRICTIVE` `UPDATE` / `DELETE` guards | any domain table carrying `organization_id`, plus `tenant_settings` — the only org-scoped table in the Packet 6 set — `event_deliveries` (P02b-3 Step 1), and `audit_log`, which Packet 9 added ([ADR-0044 § 9](../decisions/0044-audit-write-path.md)); P02d-1 Step 2 adds `courses`, `lessons` and both translation tables | +| **Tenant-owned, tenant-wide** | The same shape with the organization half of the predicate omitted, and therefore **no** restrictive guards — there is no organization to guard | `organizations`, `tenant_domains`, `tenant_locales`, `tenant_feature_flags`, `platform_entitlement_cache`, `idempotency_keys`, `outbox_messages`, `inbox_messages` (P02b-3 Step 1), `tenant_content_types`, `tenant_level_taxonomies`, `tenant_level_taxonomy_items`, `customization_generations`, and `audit_config`, which Packet 9 adds ([ADR-0044 Amendment 1](../decisions/0044-audit-write-path.md)) | | **Tenant-owned, self-keyed** | Identical, except the tenant term is `id = …` because the row's primary key *is* the tenant id | `tenants` | | **Platform-scoped** | `ENABLE` + `FORCE`, and role-qualified per-command policies: the read is widened by an explicitly declared non-tenant predicate, and writes are either tenant-keyed or reserved to `learnstack_platform` | `platform_host_to_tenant`, `platform_killswitches` | @@ -995,6 +997,8 @@ privileges implicitly. | `platform_host_to_tenant` | `SELECT, INSERT, UPDATE, DELETE` | `SELECT, INSERT, UPDATE, DELETE` | — | | `platform_killswitches` | `SELECT` | `SELECT, INSERT, UPDATE, DELETE` | — | | `idempotency_keys` | `SELECT, INSERT, UPDATE` | `SELECT, DELETE` | — | +| `inbox_messages` | `SELECT`, `INSERT (tenant_id, subscription_id, event_id)` | — | — | +| `event_deliveries` | `SELECT`, acceptance-column `INSERT`, `UPDATE (processed_at)`; exact lists [below](#inbox) | — | routing/status-column `SELECT`, scheduling/status-column `UPDATE`; exact lists [below](#inbox) | | `outbox_messages` | `SELECT, INSERT` | `SELECT, DELETE` | `SELECT`, `UPDATE (processed_at, attempts, last_error, available_after, locked_by, locked_until, terminal_reason, dead_lettered_at)` (P02b-2 Step 1) | | `tenant_content_types` | `SELECT, INSERT, UPDATE, DELETE` | `SELECT` | — | | `tenant_level_taxonomies` | `SELECT, INSERT, UPDATE, DELETE` | `SELECT` | — | @@ -1013,9 +1017,10 @@ and exact-grant proofs; both independent review rounds are complete in the Outbox status grants land in the Platform chain. The audit INSERT grant lands in a new Audit-chain migration after `audit_log` exists, never in initdb. No audit SELECT/UPDATE/DELETE, business access or role membership is -added. P02b-3 separately enumerates column-level scheduling grants on its delivery -ledger; it grants no envelope/payload SELECT to the bypass scheduler and only -RLS-protected acceptance/completion permissions to `learnstack_app`. +added. P02b-3 Step 1 implements the exact [Inbox/ledger grants](#inbox). The bypass +scheduler has no payload or causal-envelope SELECT and no business-completion +UPDATE; `learnstack_app` holds RLS-protected acceptance/completion permissions. +Neither store grants access to `learnstack_platform` or `PUBLIC`. The four Education grants are **implemented in P02d-1 Step 2** and checked against the applied schema. Root UPDATE supports publication and concurrency; @@ -1252,8 +1257,10 @@ changes `xmin` while leaving `row_version` intact. connections — needs both inserts to carry the *same* id, which a server-side default cannot give them. - `inbox_messages` is **not** one of them despite being append-only: its key - is the producing envelope's `EventId`, which the producer minted app-side. - Generating a second id there would defeat the deduplication the table is. + is `(tenant_id, subscription_id, event_id)`, and `event_id` reuses the + producing envelope's app-minted `EventId`. `event_deliveries` uses the same + tuple without a surrogate. Generating another event ID would defeat + deduplication; [Inbox](#inbox) owns both canonical definitions. - `idempotency_keys` is not one either: it is addressed by `(tenant_id, key)` and has no surrogate id at all. - **Never `gen_random_uuid()`.** It is a real function and produces a **v4** @@ -1346,7 +1353,8 @@ changes `xmin` while leaving `row_version` intact. foreign key crossing two migration chains. P02d-1's Education triggers also depend on Tenancy's shared `fn_organization_id_immutable`, including its no-`id` correction ([ADR-0003 Amendment 6](../decisions/0003-tenant-isolation-defense-in-depth.md#amendment-6--insert-scope-and-parent-mirrors-2026-09-14)). - Education adds no cross-chain foreign key. A run reaching a dependent chain first + P02b-3's Platform-chain ledger trigger reuses the same function, without adding + a cross-chain foreign key. A run reaching a dependent chain first fails on a clean database with a missing relation or function. Alphabetical order produces exactly that run — `Modules/Audit` sorts before `Modules/Tenancy` — so `make migrate` names the Tenancy chain ahead of the glob that finds the rest. The @@ -1360,8 +1368,9 @@ changes `xmin` while leaving `row_version` intact. order: Tenancy precedes every dependent chain. Independent chains may differ in relative order: the fixture applies Customization before Audit, while the recipe discovers Audit first. Neither depends on the other. - Reversal removes the dependent Education and Audit schema before reversing Tenancy; - restoring the old shared function while Education still uses it is not a valid + Reversal removes the dependent Platform, Education and Audit schema before reversing + Tenancy; restoring the old shared function while a ledger or Education trigger + still uses it is not a valid rollback order. Fresh application, reversed application and reapplication belong to the migration-chain integration proof. @@ -1400,7 +1409,7 @@ fenced transitions and atomic terminal auditing; both review rounds passed. Step 3 implements hosted execution and metrics; both independent review rounds are complete. Step 4 enforcement and local closeout are complete; all four steps completed both -independent review rounds. The [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) +independent review rounds. The [merge closeout](../roadmap/phase-02b-events-auth.md#p02b-2-merge-and-closeout-2026-10-13) owns CI/merge status. Existing migrations remain immutable; a new Platform-chain migration replaces free-form legacy `last_error` with `migration:legacy:redacted` before its byte CHECK. @@ -1528,11 +1537,12 @@ P02b-3 owns inbox/delivery runtime. that enqueued the row — [ADR-0032 § Sub-decision 12](../decisions/0032-exception-handling-logging-and-observability.md) makes it contractual. It is nullable because a tenant-wide event has no organization, not -because it is optional on an org-scoped one. The outbox row is the **only** carrier of -organization context across a transport boundary: the consumer runs in a fresh scope and -restores `ITenantContext` from the envelope, so an organization that was never persisted -here cannot be restored there. It is deliberately **not** indexed and **not** part of the -table's RLS predicate — `outbox_messages` is a tenant-wide table whose policy carries no +because it is optional on an org-scoped one. The producer persists organization +context for the transport; the [delivery ledger](#inbox) retains it in each accepted +envelope copy. The consumer runs in a fresh scope and restores `ITenantContext` +from that envelope, so an organization that was never persisted cannot be +restored. The outbox column is deliberately **not** indexed and **not** part of +its RLS predicate — `outbox_messages` is a tenant-wide table whose policy carries no organization term, and the dispatcher reads the column rather than filtering on it. `correlation_id` is `text NOT NULL` (not `uuid`): it stores the **full W3C `traceparent` @@ -1565,22 +1575,206 @@ and its MUST audit atomically. See ## Inbox -**Accepted contract; DDL/grants pending P02b-3.** One shared tenant-owned -`inbox_messages` deduplicates by `(tenant_id, subscription_id, event_id)`. -The transport's uncommitted claim precedes handler construction; completed inbox, -business state, produced outbox, MUST audit and fenced ledger completion share -one announced app-role transaction. Rollback removes a new claim; completed rows -are never reset by targeted replay. A separate delivery ledger owns attempts, -leases, terminal/replay state and retained validated envelopes. - -P02b-3 supplies canonical DDL, table-class assignments, exact RLS/grants and -ADR-0023's minting-list correction before runtime. It first chooses durable -envelope copies/references and their retention dependency; P02b-8 inventories -every retained store for erasure. The bypass scheduler receives only enumerated -identity/routing/lease/status column SELECT and status UPDATE, never payload -SELECT; the app role receives tenant-RLS-protected acceptance/completion, -including the named completion-column UPDATE, never cross-tenant scheduling. -The canonical RLS templates remain in [Table classes](#table-classes). +**P02b-3 Steps 1–4 deliver the shared schema, durable acceptance, inbox claims, +business completion and independent subscriber scheduling.** +[ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md#subscriber-acceptance-and-inbox-ownership) +and the [accepted P02b-3 package](../roadmap/phase-02b-events-auth.md#accepted-storage-and-authority-details) +define transport-owned deduplication and independent subscription delivery. +`event_deliveries` retains one immutable envelope copy per +`(tenant_id, subscription_id, event_id)`; `inbox_messages` retains that tuple's +immutable completion marker without a surrogate ID. Neither references the +producer or a business table. Producer deletion cannot remove pending delivery +payload or deduplication history. P02b-8 inventories these copies for erasure; +Phase 11 owns retention purge and its minimal grants. No P3 role receives purge +privileges on either table, including `learnstack_platform`. + +The cross-tenant scheduler indexes intentionally begin with availability time, +then occurrence time and event ID, to support ADR-0046's global selection order. +They are the explicit exception to the ordinary tenant-first index convention; +application isolation remains enforced by the policies and grants below. + +The ledger is **tenant-owned, organization-scoped** and follows the canonical +[organization policy and write guards](#tenant-owned-and-organization-scoped-tables), +including organization immutability. The inbox is **tenant-owned, tenant-wide**: +it carries identity only, without payload or organization authority. Both are +raw SQL infrastructure, without EF entities or query filters; application SQL +carries explicit tenant predicates and executes on the announced app-role +transaction. Tenancy's shared `fn_organization_id_immutable` must exist before +this Platform migration, as in the sanctioned migration sequence. + +The inbox timestamp records the uncommitted claim, using database time. The +transport inserts it before handler construction; only a successful owner +transaction makes it durable alongside business effects, produced outbox, MUST +audit and fenced ledger completion. Rollback removes a new claim; targeted replay +never resets completed inbox rows. Database triggers refuse inbox UPDATE, DELETE +and TRUNCATE even as the owner; Phase 11 must explicitly decide any retention +exception. Only `learnstack_app` updates the ledger's `processed_at`. +Completion retains the paired lease token/expiry as inert provenance, because +app grants cannot clear them. Every scheduler index and selection excludes +completed rows. Terminal completion clears ownership. Replay count/timestamp +are paired, and subscription IDs follow the manifest's ASCII +`{module}.{subscription}.v{positive-version}` grammar. + +Canonical DDL and exact column grants: + +```sql +CREATE TABLE event_deliveries ( + tenant_id uuid NOT NULL, + subscription_id text NOT NULL, + event_id uuid NOT NULL, + organization_id uuid NULL, + occurred_at timestamptz NOT NULL, + correlation_id text NOT NULL, + causation_id uuid NULL, + actor_user_id uuid NULL, + type text NOT NULL, + topic text NOT NULL, + partition_key text NOT NULL, + payload jsonb NOT NULL, + processed_at timestamptz NULL, + attempts int NOT NULL DEFAULT 0, + last_error text NULL, + available_after timestamptz NOT NULL DEFAULT now(), + locked_by text NULL, + locked_until timestamptz NULL, + terminal_reason text NULL, + dead_lettered_at timestamptz NULL, + replay_count int NOT NULL DEFAULT 0, + replayed_at timestamptz NULL, + CONSTRAINT pk_event_deliveries PRIMARY KEY (tenant_id, subscription_id, event_id), + CONSTRAINT ck_event_deliveries_identity CHECK ( + tenant_id NOT IN ('00000000-0000-0000-0000-000000000000'::uuid, + '00000000-0000-7000-8000-000000000002'::uuid) + AND event_id <> '00000000-0000-0000-0000-000000000000'::uuid + AND (organization_id IS NULL OR + organization_id <> '00000000-0000-0000-0000-000000000000'::uuid) + AND (causation_id IS NULL OR causation_id <> '00000000-0000-0000-0000-000000000000'::uuid) + AND (actor_user_id IS NULL OR actor_user_id <> '00000000-0000-0000-0000-000000000000'::uuid)), + CONSTRAINT ck_event_deliveries_subscription CHECK ( + subscription_id COLLATE "C" ~ '^[a-z][a-z0-9]*\.[a-z][a-z0-9-]*\.v[1-9][0-9]*$'), + CONSTRAINT ck_event_deliveries_attempts CHECK (attempts >= 0), + CONSTRAINT ck_event_deliveries_replay CHECK ( + replay_count >= 0 AND ((replay_count = 0 AND replayed_at IS NULL) + OR (replay_count > 0 AND replayed_at IS NOT NULL))), + CONSTRAINT ck_event_deliveries_last_error_size CHECK ( + last_error IS NULL OR octet_length(last_error) <= 256), + CONSTRAINT ck_event_deliveries_lease CHECK ( + (locked_by IS NULL AND locked_until IS NULL) OR + (locked_by IS NOT NULL AND locked_until IS NOT NULL + AND locked_by COLLATE "C" ~ '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$' + AND locked_by <> '00000000-0000-0000-0000-000000000000')), + CONSTRAINT ck_event_deliveries_terminal_reason CHECK ( + terminal_reason IS NULL OR terminal_reason IN ( + 'unknown_type', 'retired_type', 'invalid_envelope', + 'invalid_subscription', 'retry_exhausted')), + CONSTRAINT ck_event_deliveries_state CHECK ( + (processed_at IS NULL OR (dead_lettered_at IS NULL AND terminal_reason IS NULL)) + AND (dead_lettered_at IS NULL OR + (processed_at IS NULL AND terminal_reason IS NOT NULL AND locked_by IS NULL))) +); + +CREATE INDEX ix_event_deliveries_tenant_id_organization_id + ON event_deliveries (tenant_id, organization_id); +CREATE INDEX ix_event_deliveries_pending + ON event_deliveries (available_after, occurred_at, event_id, tenant_id, subscription_id) + WHERE processed_at IS NULL AND dead_lettered_at IS NULL + AND terminal_reason IS NULL AND attempts < 6 AND locked_by IS NULL; +CREATE INDEX ix_event_deliveries_recovery + ON event_deliveries (available_after, occurred_at, event_id, tenant_id, subscription_id) + WHERE processed_at IS NULL AND dead_lettered_at IS NULL + AND terminal_reason IS NULL AND (locked_by IS NOT NULL OR attempts >= 6); +CREATE INDEX ix_event_deliveries_terminal + ON event_deliveries (available_after, occurred_at, event_id, tenant_id, subscription_id) + WHERE processed_at IS NULL AND dead_lettered_at IS NULL AND terminal_reason IS NOT NULL; + +ALTER TABLE event_deliveries ENABLE ROW LEVEL SECURITY; +ALTER TABLE event_deliveries FORCE ROW LEVEL SECURITY; +CREATE POLICY event_deliveries_isolation ON event_deliveries + USING ( + tenant_id = NULLIF(current_setting('app.tenant_id', true), '')::uuid + AND (organization_id IS NULL + OR organization_id = NULLIF(current_setting('app.organization_id', true), '')::uuid + OR current_setting('app.scope', true) = 'tenant')) + WITH CHECK ( + tenant_id = NULLIF(current_setting('app.tenant_id', true), '')::uuid + AND ((organization_id IS NULL + AND NULLIF(current_setting('app.organization_id', true), '') IS NULL) + OR organization_id = NULLIF(current_setting('app.organization_id', true), '')::uuid)); +CREATE POLICY event_deliveries_org_write_guard ON event_deliveries + AS RESTRICTIVE FOR UPDATE + USING ((organization_id IS NULL + AND NULLIF(current_setting('app.organization_id', true), '') IS NULL) + OR organization_id = NULLIF(current_setting('app.organization_id', true), '')::uuid); +CREATE POLICY event_deliveries_org_delete_guard ON event_deliveries + AS RESTRICTIVE FOR DELETE + USING ((organization_id IS NULL + AND NULLIF(current_setting('app.organization_id', true), '') IS NULL) + OR organization_id = NULLIF(current_setting('app.organization_id', true), '')::uuid); +CREATE TRIGGER tg_event_deliveries_organization_id_immutable + BEFORE UPDATE ON event_deliveries + FOR EACH ROW EXECUTE FUNCTION fn_organization_id_immutable(); + +CREATE TABLE inbox_messages ( + tenant_id uuid NOT NULL, + subscription_id text NOT NULL, + event_id uuid NOT NULL, + recorded_at timestamptz NOT NULL DEFAULT now(), + CONSTRAINT pk_inbox_messages PRIMARY KEY (tenant_id, subscription_id, event_id), + CONSTRAINT ck_inbox_messages_identity CHECK ( + tenant_id NOT IN ('00000000-0000-0000-0000-000000000000'::uuid, + '00000000-0000-7000-8000-000000000002'::uuid) + AND event_id <> '00000000-0000-0000-0000-000000000000'::uuid), + CONSTRAINT ck_inbox_messages_subscription CHECK ( + subscription_id COLLATE "C" ~ '^[a-z][a-z0-9]*\.[a-z][a-z0-9-]*\.v[1-9][0-9]*$') +); +ALTER TABLE inbox_messages ENABLE ROW LEVEL SECURITY; +ALTER TABLE inbox_messages FORCE ROW LEVEL SECURITY; +CREATE POLICY inbox_messages_isolation ON inbox_messages + USING (tenant_id = NULLIF(current_setting('app.tenant_id', true), '')::uuid) + WITH CHECK (tenant_id = NULLIF(current_setting('app.tenant_id', true), '')::uuid); + +CREATE FUNCTION fn_inbox_messages_immutable() RETURNS trigger +LANGUAGE plpgsql AS $$ +BEGIN + RAISE EXCEPTION 'inbox_messages is immutable (attempted % as %)', TG_OP, current_user + USING ERRCODE = 'insufficient_privilege'; +END; +$$; +CREATE TRIGGER tg_inbox_messages_immutable + BEFORE UPDATE OR DELETE ON inbox_messages + FOR EACH ROW EXECUTE FUNCTION fn_inbox_messages_immutable(); +CREATE TRIGGER tg_inbox_messages_no_truncate + BEFORE TRUNCATE ON inbox_messages + FOR EACH STATEMENT EXECUTE FUNCTION fn_inbox_messages_immutable(); + +GRANT SELECT ON event_deliveries, inbox_messages TO learnstack_app; +GRANT INSERT (tenant_id, subscription_id, event_id, organization_id, occurred_at, + correlation_id, causation_id, actor_user_id, type, topic, partition_key, payload) + ON event_deliveries TO learnstack_app; +GRANT UPDATE (processed_at) ON event_deliveries TO learnstack_app; +GRANT INSERT (tenant_id, subscription_id, event_id) ON inbox_messages TO learnstack_app; +GRANT SELECT (tenant_id, subscription_id, event_id, organization_id, occurred_at, + type, topic, partition_key, processed_at, attempts, last_error, + available_after, locked_by, locked_until, terminal_reason, + dead_lettered_at, replay_count, replayed_at) + ON event_deliveries TO learnstack_outbox_admin; +GRANT UPDATE (attempts, last_error, available_after, locked_by, locked_until, + terminal_reason, dead_lettered_at, replay_count, replayed_at) + ON event_deliveries TO learnstack_outbox_admin; +``` + +The app acceptance INSERT excludes all scheduling/completion/terminal columns; +inbox INSERT excludes the database-owned timestamp. The scheduler cannot read +payload, correlation, causation or original actor; it cannot insert/delete a +ledger row, access the inbox, mutate routing/envelope fields or fake business +completion. Column UPDATE permits row locking without table-wide UPDATE. +The three scheduling indexes share total order +`(available_after, occurred_at, event_id, tenant_id, subscription_id)` and their +eligibility predicates. Recovery handles expired ownership or exhausted attempts; +terminal requests remain outside normal acquisition. Runtime adds database-time +lease/due predicates and manifest/local partition admission. The primary key +covers tenant-only reads; the full tenant/organization index covers RLS branches, +including tenant-wide rows. ## Idempotency diff --git a/docs/standards/09-error-handling.md b/docs/standards/09-error-handling.md index 7fa7ff6e..cd5babbf 100644 --- a/docs/standards/09-error-handling.md +++ b/docs/standards/09-error-handling.md @@ -564,10 +564,12 @@ exception types (`LiveKit.NET.LiveKitException`, `Stripe.StripeException`, ## Outbox -**Accepted ADR-0046; P02b-2 producer runtime and enforcement locally complete.** -All four steps completed both independent review rounds. The [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) -owns CI/merge status. Subscriber delivery/retries and inbox -remain P02b-3. +**Accepted ADR-0046; P02b-2 producer runtime and enforcement complete and merged.** +All four steps completed both independent review rounds. The [merge closeout](../roadmap/phase-02b-events-auth.md#p02b-2-merge-and-closeout-2026-10-13) +owns CI/merge status. P02b-3 Steps 1–4 implement durable acceptance, shared inbox, +independent subscriber execution/retries and terminal audit. Step 5 activates both +real default +subscriptions and implements the shared initializer/Seeder handoff. - Each committed normal claim increments producer `attempts` once, including an unknown crash outcome. Six total claims use retry delays 1 s, 5 s, 30 s, 5 min @@ -583,9 +585,9 @@ remain P02b-3. - Persist only closed stage/reason/category codes, optional allowlisted SQLSTATE and validated attempt traceparent in `last_error`, at most 256 UTF-8 bytes. Dispatcher logs/traces likewise exclude exception text/stacks, payloads, raw - stored type and connection data. Existing transport exception telemetry is a - P02b-3 reconciliation obligation before Active production consumers. -- After P02b-3 durable acceptance, subscriber business failure spends only its + stored type and connection data. P02b-3 removes the former raw transport exception + telemetry; subscriber execution and observation retain bounded diagnostics. +- After durable acceptance, subscriber business failure spends only its independent ledger budget. Failed/unknown acceptance still spends producer attempts. Subscriber retry invokes one internal delivery executor, never a broadcast; operator recovery routes and producer reset/fencing are Phase 03. diff --git a/docs/standards/10-observability.md b/docs/standards/10-observability.md index 40826599..f5e924e9 100644 --- a/docs/standards/10-observability.md +++ b/docs/standards/10-observability.md @@ -85,11 +85,12 @@ telemetry listener, including the Seeder. Enqueue persists the full traceparent; transaction announcement supplies tenant/organization while trusted execution supplies correlation, causation and optional causal actor. Domain translation allocates a new integration EventId and retains the domain EventId as causation. -Future delivery-originated production uses the trusted incoming event identity, +Delivery-originated production uses the trusted incoming event identity, with effective `UserId.SystemActor` and causal human retained separately. No caller-supplied header establishes causation; JSON `metadata` stays SQL NULL. P02b-1 implements this producer metadata boundary in both API and Seeder. -Production delivery context restoration remains P02b-3 work. +P02b-3 Step 3 implements restoration and incoming-event causation in the common +delivery owner; Step 5 activates both real defaults and semantic Seeder handoff. The propagation primitive is **W3C `traceparent`** end to end. Every cross-boundary write (outbox row, Hangfire payload, Hub envelope) carries it; @@ -154,6 +155,7 @@ Auto-instrument: - MediatR commands and queries. - Hangfire job invocations. - Outbox dispatcher attempts (P02b-2 Step 3; no batch span). +- Subscriber execution/terminal attempts (P02b-3 Step 4; no batch span). - Valkey client calls. - SeaweedFS/S3 SDK calls. - LiveKit provider calls. @@ -233,8 +235,15 @@ backend treat business rejections as system failures. | `learnstack_outbox_oldest_eligible_age_seconds` | gauge | — | | `learnstack_outbox_deferred_count` | gauge | — | | `learnstack_outbox_heartbeat_age_seconds` | gauge | — | -| `learnstack_inbox_dedup_total` | counter | module, event_type (P02b-3 reserved) | -| `learnstack_inbox_deadletter_total` | counter | module, event_type (P02b-3 reserved) | +| `learnstack_delivery_execute_duration_seconds` | histogram | event_type, subscription | +| `learnstack_delivery_failed_total` | counter | event_type, subscription | +| `learnstack_delivery_pending_count` | gauge | — | +| `learnstack_delivery_oldest_eligible_age_seconds` | gauge | — | +| `learnstack_delivery_deferred_count` | gauge | — | +| `learnstack_delivery_lease_lost_total` | counter | — | +| `learnstack_delivery_heartbeat_age_seconds` | gauge | — | +| `learnstack_inbox_dedup_total` | counter | event_type, subscription | +| `learnstack_inbox_deadletter_total` | counter | event_type, subscription | | `learnstack_job_duration_seconds` | histogram | job_name | | `learnstack_job_failed_total` | counter | job_name | | `learnstack_provider_request_duration_seconds` | histogram | provider, operation, outcome | @@ -254,10 +263,19 @@ backend treat business rejections as system failures. | `learnstack_entitlement_unresolved_total` | counter | — | Outbox metrics above are implemented in P02b-2 Step 3 under ADR-0046; -both independent review rounds are complete. Inbox counters remain P02b-3 and job -metrics P02b-4. `event_type` comes only from the -finite declared registry plus fixed unknown/retired/invalid categories. Never -label event metrics by tenant, event, partition, subscription or trace identity. +both independent review rounds are complete. The subscriber/inbox inventory is +accepted by the [P02b-3 package](../roadmap/phase-02b-events-auth.md#p02b-3-decision-package-2026-10-13); +Step 4 implements it with real worker, finite-label and memory-only observation +proofs. Job metrics remain P02b-4. +`event_type` comes only from the finite declared registry plus fixed +unknown/retired/invalid categories. `subscription` comes only from declared stable +subscription IDs plus fixed `invalid`; a well-formed undeclared ID is still +`invalid`. Never label event metrics by tenant, event, partition or trace identity. +Subscriber failure, deduplication and terminal counters are distinct from producer +publication counters; one subscriber's failure is not a producer failure. +Counters describe observed attempts and fenced transition refusals, not a durable +accounting ledger. An unknown COMMIT can already have completed business work; +its refused later fence does not prove that business work failed or should repeat. Pending count includes all unprocessed, non-dead-lettered rows, including delayed, leased, terminal-requested and manifest-deferred work. Oldest eligible age uses @@ -272,9 +290,12 @@ health bound. Failure retains the prior snapshot and reports unhealthy. The [P02b-2 bounds](../roadmap/phase-02b-events-auth.md#runtime-bounds-and-observation) own deadlines and task limits. Dispatcher-owned logs/spans use validated correlation and bounded outcomes, without exception text/stacks, payload, raw -stored type or connection data. P02b-3 reconciles existing transport/consumer -exception telemetry before Active consumers; no current transport/consumer -minimization is claimed. +stored type or connection data. P02b-3 Steps 3–4 minimize transport-owned +execution/audit diagnostics before activating production consumers. Original +causal metadata is restored only in the app-role executor; the bypass scheduler +emits a fresh validated attempt trace without reading payload or causal-envelope +columns. No raw exception details, stored type/subscription text or payload enters +its diagnostics. Cache `cache.name` is a governed, low-cardinality family from the Standards 20 inventory (`hub:host-map`, `hub:entitlement`, `identity:permissions`, @@ -312,10 +333,12 @@ touched. |---|---|---| | Audit write path | `audit` | The most recent MUST-class standalone audit write failed and no later one has succeeded | | Outbox dispatcher (P02b-2 Step 3) | `outbox` | No actual progress or observation refresh for 45 s, any retained callback overrun, or terminal-audit failure | +| Subscriber dispatcher (P02b-3 Step 4) | `delivery` | No actual progress or observation refresh for 45 s, any retained callback/cleanup overrun, or terminal-audit failure | The `audit` check is registered in Phase 02a Packet 9; P02b-2 Step 3 registers -`outbox`, with both independent review rounds complete. -Both are **not mapped**: `/healthz` stays +`outbox`, with both independent review rounds complete. P02b-3 Step 4 adds +`delivery`; its delivery record owns current validation and review status. +All three are **not mapped**: `/healthz` stays a liveness probe, because a process that cannot write audit rows is still worth leaving alive. The readiness surface that reads the check, and the deployment-level backstop that stops serving past a configured unhealthy window, are demand-gated to diff --git a/docs/standards/11-security.md b/docs/standards/11-security.md index 6c84817a..d45a005c 100644 --- a/docs/standards/11-security.md +++ b/docs/standards/11-security.md @@ -247,7 +247,7 @@ ADR-0046 accepts this P02b-2 boundary. Step 1 implements source/role guards; both review rounds are complete. Step 3 implements mandatory worker startup resolution and connectivity checks; both independent review rounds are complete. Step 4 credential/publication enforcement and local closeout are complete; all -four steps completed both independent review rounds. The [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) +four steps completed both independent review rounds. The [merge closeout](../roadmap/phase-02b-events-auth.md#p02b-2-merge-and-closeout-2026-10-13) owns CI/merge status. The named keyed, lazy dispatcher source belongs only to the API host. Configuration validates the exact username without connecting or @@ -266,10 +266,12 @@ Routine confined poll/claim/mark receives ADR-0046's bounded audit exception; terminal state is MUST/SecurityEvent and atomic with its row-bound witness audit. INSERT-only and GUCs cannot tenant-bind a bypass credential. Real mismatch controls must roll back terminal state and audit together. Existing platform-admin entry -audit and gate stay unchanged. P02b-3 opens separate app-role business scopes, -restores validated tenant/organization and announces RLS context before reading -the envelope and constructing a handler. The ordinary executor refuses privileged -subscriptions until their Phase 03 execution contract exists. +audit and gate stay unchanged. P02b-3 Steps 3–4 implement separate app-role +business scopes, restore validated tenant/organization and announce RLS context +before reading the envelope and constructing a handler. The subscriber scheduler +shares the confined source for enumerated routing/status columns; it receives no +payload, causal-envelope or inbox SELECT privilege. The ordinary executor refuses +privileged subscriptions until their Phase 03 execution contract exists. ## Tenant Context @@ -372,7 +374,7 @@ yet: | Setter | Transaction | Why it is not `TransactionBehavior` | |---|---|---| | `TransactionBehavior` → `TransactionExecutionCoordinator` | ambient | — the existing general announcement path; common execution adds no authority | -| The integration-event transport, per delivery | ambient — it opens it | There is no MediatR request: `InProcessEventBus` invokes the handler directly, so no behavior runs. It opens the ambient transaction itself, from the delivery's `EventTenantContext` | +| The integration-event transport, per delivery | ambient — it opens it | `DeliveryExecutor` uses the common transaction owner and validated retained envelope before restoring `EventTenantContext`, claiming inbox and constructing the handler. `InProcessEventBus` owns durable acceptance only; nested commands join the prepared business transaction | | `IOrganizationScopeValidator` | its own short read-only one | The organization assertion is validated in the request edge, before the pipeline reaches step 6 ([ADR-0036](../decisions/0036-tenant-resolution-trusted-inputs.md)) | | `IIdempotencyStore` (durable) | its own short one | A claim is taken **before** the pipeline reaches step 6 ([ADR-0037](../decisions/0037-idempotency-key-contract.md)) | | `IAuditStore.WriteStandaloneAsync` | its own short one | An audit row that must survive the rollback of the operation it describes cannot share that operation's transaction ([ADR-0033](../decisions/0033-audit-durability-model.md)) | @@ -418,25 +420,27 @@ leak — but an empty result set arriving from production is an outage, so the `SET LOCAL` pair on this transaction before any command a module `DbContext` issues runs on it. -**Keyed on the transaction, not on the table**, and marked by one setter rather than -eight — both narrower than an earlier wording here, and both for reasons the shipped -mechanism makes plain. Matching `[TenantOwned]` table names would put a parser between -every query and the database to decide something the transaction already answers. -And of the eight paths, only `TransactionBehavior` through -`TransactionExecutionCoordinator` and the sanctioned `IUnitOfWork` announcement -methods marks the ambient transaction today: two do not exist in code yet — -the durable idempotency store, and the integration-event transport, which matters most, -because it is the other setter that *opens* the ambient transaction and must therefore -announce it when Phase 02b lands it — and the five that do, `OrganizationScopeValidator`, +**Keyed on the transaction, not on the table**, and marked through the ambient +unit's announcement seam. Matching `[TenantOwned]` table names would put a parser +between every query and the database to decide something the transaction already +answers. +Of the eight logical paths, the MediatR transaction owner and P02b-3's integration +transport mark the ambient transaction through sanctioned `IUnitOfWork` +announcement methods. Durable acceptance and prepared subscriber execution both +announce before their first app-role command; the production call-site census +closes these implementations without adding a ninth logical setter. The durable +idempotency store remains unimplemented. The five other paths, +`OrganizationScopeValidator`, the two standalone audit writers and the two cached-projection loaders, issue raw `NpgsqlCommand`s on connections of their own, which EF interception never sees. `CachedHostToTenantResolver` is not in this set at all: it sets `app.resolving_host`, not `app.tenant_id`. That is also why the exemption list is empty, and why `PlatformAdminScope`, whose `BYPASSRLS` connection announces no tenant by design, is invisible to the guard by construction rather than by a hand-written exception. The concern the earlier wording had — that naming -`TransactionBehavior` alone would reject the writes the idempotency store and the audit -store legitimately make on their own short transactions — is real, and is answered by the -guard reading a marker rather than a behavior's name: a setter marks the transaction it +`TransactionBehavior` alone would reject legitimate transport and audit writes, +and the future idempotency store's writes on its own short transaction — is real, +and is answered by the guard reading a marker rather than a behavior's name: +a setter marks the transaction it opens if and only if it reaches EF. The audit store's writers have landed and reach none; the durable idempotency store will mark its own if it does. It throws `TenantContextMissingException` when it has not, which diff --git a/docs/standards/18-audit-coverage.md b/docs/standards/18-audit-coverage.md index 10fdceb0..4bee2178 100644 --- a/docs/standards/18-audit-coverage.md +++ b/docs/standards/18-audit-coverage.md @@ -130,7 +130,12 @@ joins them catalogue → matrix and matrix → catalogue, each reading a cell as the list of slugs it holds. Neither can see a request type nobody registered when another request shares its slug, which is why [`Every_Shipped_Request_Is_Registered`](21-architecture-tests-catalogue.md#every_shipped_request_is_registered) -asks the catalogue for every request type with a handler. +asks the catalogue for every request type with a handler and every concrete +integration-event handler type, including `Off` coordinators. Two handlers for +one event are two subjects; classifying the shared event type covers neither. +P02b-3 Step 6 checks both API and Seeder catalogues against fail-loud discovery +and an omitted-`Off` control whose operation entries remain unchanged. A module +with only integration handlers still requires its own matrix. ### The join @@ -159,14 +164,14 @@ will raise them — which this standard requires it to do. `tenancy.entitlement.refresh` and the two tenant-assertion keys of § Baseline Coverage, and Audit's `audit.redaction.apply` and `audit.purge.apply`. They sit outside the request-type join in **both** directions and register by slug rather than by type — the - four whose writer ships through `DeclareOffPath` today, the other five when their + five whose writer ships through `DeclareOffPath` today, the other four when their writer lands. P02b-2 Step 2 registers the shipped producer terminal writer; - subscriber terminal registration remains planned for P02b-3. + P02b-3 Step 4 adds subscriber terminal registration with its writer. - **One operation, one row.** A slug is classified in exactly one row across every matrix. A copy is a second answer that passes every comparison the moment it agrees, so the matrix → catalogue direction refuses it, and the catalogue → matrix direction compares every row that carries a slug rather than the first. -- **A row can carry both markers**, and five do. `tenancy.killswitch.toggle` is +- **A row can carry both markers**, and four do. `tenancy.killswitch.toggle` is `(off-path)` because every write runs inside `EnterPlatformAdminScope(reason)` rather than through the pipeline, and `(planned)` because Packet 9 ships the table, the policies and the read path but no writer — @@ -392,8 +397,9 @@ Rules: commands keep their own intents without a duplicate business row. Modules see none of this plumbing; the MUST floor and standalone refusal posture survive. - The interceptor captures **every** `ChangeTracker` entry in state `Added`, `Modified` or - `Deleted`, minus a named exclusion list — `OutboxMessage` and `IdempotencyKey` - (machinery) and `AuditEntry` / `AuditConfig` (the audit tables themselves). The + `Deleted`, minus a named exclusion list — `OutboxMessage`, `IdempotencyKey`, + `InboxMessage` and `EventDelivery` (machinery) and `AuditEntry` / `AuditConfig` + (the audit tables themselves; [ADR-0044 Amendment 11](../decisions/0044-audit-write-path.md#amendment-11--shared-delivery-capture-exclusions-2026-10-13)). The "`AuditableEntity<>` descendants only" predicate is **withdrawn**: it was blind to the seven plain classes the modules ship — `PlatformHostMapping`, `TenantLocale`, `TenantFeatureFlag`, `PlatformEntitlement` and `PlatformKillswitch` in Tenancy, @@ -509,11 +515,12 @@ table above is the policy those deliverables implement, not a description of tod transaction. Validate declared MUST/SecurityEvent operation and matching draft before INSERT; mismatch rolls back both writes. No transaction is opened or committed by the seam. `PostgresAuditStore` remains the sole audit SQL writer. - Tenancy owns off-path `platform.outbox.dead_letter` (P02b-2) and reserved - `platform.delivery.dead_letter` (P02b-3); register only with each writer. + Tenancy owns off-path `platform.outbox.dead_letter` (P02b-2) and + `platform.delivery.dead_letter` (P02b-3 Step 4); each is registered with its writer. Outcome success records terminal state, not publication. Invalid causal metadata - is omitted and a fresh validated attempt trace is used. Closed terminal values - live in bounded metadata; `AuditEntryDraft.Reason` remains for platform entry. + is omitted and a fresh validated attempt trace is used. Absent optional causal + IDs are valid; SQL NULL alone does not invalidate a stored trace. Closed terminal + values live in bounded metadata; `AuditEntryDraft.Reason` remains for platform entry. Audit refusal retries terminalization without delivery. Because the bypass role ignores RLS, witness/call-path proof binds identity, not announced GUCs. - Entering `EnterPlatformAdminScope(reason)` writes that row through @@ -567,8 +574,8 @@ observes neither a rollback nor a `42501`. - [`OperationType_Enum_Matches_Catalog`](21-architecture-tests-catalogue.md#operationtype_enum_matches_catalog) — the enum and § Operation Types carry the same seven members. - [`Every_Shipped_Request_Is_Registered`](21-architecture-tests-catalogue.md#every_shipped_request_is_registered) - — every request type with a handler in any backend assembly is in the catalogue, `Off` - included. + — every request type with a handler and every concrete integration handler in + any backend assembly is in the catalogue, `Off` included. - [`Every_TenantOwned_Command_HasAuditCoverage`](21-architecture-tests-catalogue.md#every_tenantowned_command_hasauditcoverage) — every catalogue entry has a matrix row with the same slug, class and type. - [`Every_Matrix_Row_Whose_Command_Exists_Is_Registered`](21-architecture-tests-catalogue.md#every_matrix_row_whose_command_exists_is_registered) diff --git a/docs/standards/20-infrastructure-stack.md b/docs/standards/20-infrastructure-stack.md index a10105e7..b14683ab 100644 --- a/docs/standards/20-infrastructure-stack.md +++ b/docs/standards/20-infrastructure-stack.md @@ -81,10 +81,13 @@ Rules that follow from this: written against the port and holds for the default implementation too. If a rule only makes sense for the vendor adapter, it is a rule about the adapter and belongs in that adapter's section, not in module guidance. -- **`InProcessEventBus` is a first-class transport, not a stub.** It uses - `IIntegrationEventHandler` and restores tenant context before handler resolution. - Durable acceptance and the shared transport-owned `IInboxGuard` remain P02b-3 - work under [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md#subscriber-acceptance-and-inbox-ownership). +- **`InProcessEventBus` is a first-class transport, not a stub.** P02b-3 Step 2 + implements durable acceptance for declared `IIntegrationEventHandler` + subscriptions, restoring context before its fresh application scope. Publication + constructs no handler. The shared transport-owned `IInboxGuard` claim primitive + and Step 3's business owner/executor are implemented; Step 4 implements + independent scheduling, terminalization and replay. Step 5 activates both real + defaults with semantic Seeder handoff under [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md#subscriber-acceptance-and-inbox-ownership). Phase 11's adapter must preserve that consumer contract. - **A demand-gated block is not "deferred".** It qualifies only with all four of: a port, a working default implementation, an owning phase, and a written trigger. If a @@ -187,8 +190,9 @@ adapter replaces `InProcessEventBus`. (`learnstack.hub.entitlement`, `learnstack.hub.custom-domain.activated`). - Consumers implement `IIntegrationEventHandler`. Accepted ADR-0046 assigns shared inbox claim/completion to the transport, before handler construction. - `Integration_Event_Handlers_Use_InboxGuard` is Registered for P02b-3's nonempty - production census and real atomicity/duplicate proofs; handlers do not own the guard. + [`Integration_Event_Handlers_Use_InboxGuard`](21-architecture-tests-catalogue.md#integration_event_handlers_use_inboxguard) + is Implemented by P02b-3's nonempty production census and real + atomicity/duplicate proofs; handlers do not own the guard. - Construction-free readiness declares Pending, Active or intentional absence. Any required Pending subscription holds its contract/aliases before acquisition and validation; Active declarations require actual handler registration. @@ -451,16 +455,37 @@ Rules: - P02b-3 implements the shared transport-owned inbox keyed by tenant/subscription/ event. Its claim precedes handler construction; business state, inbox completion, produced outbox, MUST audit and fenced ledger completion share the app-role owner. - Independent delivery ledgers retain validated envelopes and own subscriber - retries/terminal/replay. Targeted replay invokes one internal executor, never - broadcast; the outbox processor remains the only production publisher. + The shared delivery ledger retains one validated envelope copy per tuple and + owns independent retries/terminal/replay. Step 4's scheduler uses the confined + bypass source for routing/status only; business execution uses `learnstack_app`. + Exact Pending/privileged type/subscription pairs are held while accepted Active + siblings can run. Targeted replay resets only a completed terminal Active + ordinary tuple with no lease; the scheduler later invokes that executor, never + broadcast. It never resets a completed inbox; the outbox processor remains the + only production publisher. - Operator routes, permissions and producer reset/fencing belong to Phase 03. Brokers and retention purge remain demand-gated Phase 11. P02b-2 Step 3 implements hosted producer dispatch, with both independent review rounds complete; Step 4 enforcement and local closeout are complete; all four steps completed - both independent review rounds. The [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) - owns CI/merge status. Inbox and - effectively-once consumer delivery remain P02b-3. + both independent review rounds. The [merge closeout](../roadmap/phase-02b-events-auth.md#p02b-2-merge-and-closeout-2026-10-13) + owns CI/merge status. P02b-3 Steps 1–4 implement shared storage, durable acceptance, + transactional execution and independent scheduling/terminalization/replay. Its + real defaults are Active with Step 5 initialization and semantic Seeder handoff. The + [Step 5 record](../roadmap/phase-02b-events-auth.md#p02b-3-step-5--useful-defaults-and-semantic-seed-handoff-2026-10-13) + records that step's validation and reviews. The + [Step 6 record](../roadmap/phase-02b-events-auth.md#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) + owns current enforcement, validation and review/PR status. +- Controlled subscriber tasks retain their actual app-role scope through + cancellation callbacks and cleanup. Timeout closes the sticky commit gate + before invoking callbacks. Acknowledged durability survives cleanup faults; + retained tasks block their local execution partition and consume bounded capacity. + Acceptance uses a separate serializer lane and never waits for subscriber cleanup. + Shutdown observes one total finite budget and the host token. +- Subscriber observation emits the nine instruments defined by + [Observability Standards](10-observability.md#required-metrics), with finite + declared event/subscription labels. Gauges read immutable memory snapshots; + failed refresh preserves stale data and makes health unhealthy. The `delivery` + check is registered and unmapped; `/healthz` remains liveness-only. Full deep dive: [15-event-and-outbox.md](../architecture/15-event-and-outbox.md). @@ -667,8 +692,10 @@ Its restricted witness comes only from the actual successful fenced terminal UPD The store validates declared operation/class/type, row identity and live caller transaction, then INSERTs without opening/committing a transaction. Existing four writes retain their contracts. The terminal UPDATE and audit commit together, or -both roll back. Tenancy owns producer terminal audit in P02b-2 and subscriber -terminal audit in P02b-3. See +both roll back. Tenancy owns producer terminal audit in P02b-2 and +`platform.delivery.dead_letter` in P02b-3 Step 4. The subscriber witness uses only +actual routing/terminal identity and a validated attempt trace; it adds no +causal-envelope or payload read to the bypass scheduler. See [Audit Coverage](18-audit-coverage.md#required-behaviours). ## Background Jobs and Hangfire diff --git a/docs/standards/21-architecture-tests-catalogue.md b/docs/standards/21-architecture-tests-catalogue.md index 87e1e4de..c13f9b3e 100644 --- a/docs/standards/21-architecture-tests-catalogue.md +++ b/docs/standards/21-architecture-tests-catalogue.md @@ -15,9 +15,12 @@ (producer lifecycle, payload and production-census proofs implemented), [ADR-0046 Durable Event Delivery and Recovery](../decisions/0046-durable-event-delivery-and-recovery.md) (P02b-2 transition, terminal-witness and hosted runtime proofs implemented; -all four steps completed both independent review rounds and local closeout; -the [Step 4 record](../roadmap/phase-02b-events-auth.md#p02b-2-step-4--enforcement-and-closeout-2026-10-12) -owns CI/merge status; P02b-3 consumer proofs remain pending). The +all four steps completed both independent review rounds; packet complete and merged; +the [merge closeout](../roadmap/phase-02b-events-auth.md#p02b-2-merge-and-closeout-2026-10-13) +owns CI/merge status; P02b-3 Steps 1–6 implement consumer runtime, defaults and +production-census enforcement; its +[Step 6 record](../roadmap/phase-02b-events-auth.md#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) +owns validation, review and PR closeout status). The catalogue grows as subsequent ADRs and phases land their tests; per-test ownership stays with the originating ADR / standard. @@ -110,12 +113,12 @@ not implemented is the failure mode this column exists to prevent. ### Implemented today -**194 test methods run in +**219 test methods run in [`backend/tests/LearnStack.Tests.Architecture`](../../backend/tests/LearnStack.Tests.Architecture),** shipped by [Phase 01](../roadmap/phase-01-repository-tooling.md), [Phase 02a Packets 2–3](../roadmap/phase-02a-kernel-tenancy.md), Packet 4, Packet 6, Packet 7, -Packet 8, Packet 9, Packet 10, P02d-1, P02d-2, P02d-3, P02d-4 Steps 1–3, P02b-1 -and P02b-2 Steps 2–4. +Packet 8, Packet 9, Packet 10, P02d-1, P02d-2, P02d-3, P02d-4 Steps 1–3, P02b-1, +P02b-2 Steps 2–4 and P02b-3 Steps 3–6. Methods are not rows: a `[Theory]` is one row and many cases, and most rows pair a rule with the companion assertion that stops it passing vacuously. @@ -142,8 +145,8 @@ two fifths of its subject is the defect this section is about. It also refuses a test class that exists nowhere, because otherwise a renamed or deleted file drops its entries out of the subject instead of failing. -**192 rules in this catalogue are Implemented, and 118 of them are in that assembly.** -The other 74 are no less binding, and most could not live there. The table says where +**202 rules in this catalogue are Implemented, and 121 of them are in that assembly.** +The other 81 are no less binding, and most could not live there. The table says where and why, and deliberately carries no per-row count: those are the numbers nothing recomputes, and the first version of this table claimed "three rules" for a suite that holds ten. @@ -174,6 +177,14 @@ Every rule in this document carries its own **Status** line, and that line is th One rule, one identifier. When a document, skill, or comment uses a superseded spelling, the fix is to replace it with the canonical name — not to add a row here. +### Terminal audit witness rules + +`Outbox_Terminal_Audit_Identity_Is_Row_Bound` and +`Delivery_Terminal_Audit_Identity_Is_Row_Bound` share the `TerminalOffenders` +architecture detector because both use the same restricted witness/writer seam. +Distinct producer and subscriber planted controls and real-row integration +proofs enforce their respective identities; these remain two catalogue rules. + ### Tenant isolation **Canonical: `Every_TenantOwned_Entity_HasFilterAndRlsPolicy`** — one rule covering the @@ -1311,10 +1322,12 @@ otherwise). #### `Cross_Aggregate_Writes_Are_Confined_To_Tenant_Provisioning` -- **Asserts:** no type implementing `IRequestHandler<,>`, `IRequestHandler<>` or - `INotificationHandler<>` can reach more than one **distinct aggregate root** through +- **Asserts:** no type implementing `IRequestHandler<,>`, `IRequestHandler<>`, + `INotificationHandler<>` or `IIntegrationEventHandler<>` can reach more than one + **distinct aggregate root** through its constructor parameters' `IAggregateWriteStore` derivations, except the - single handler on a literal allow-list — `ProvisionTenantCommandHandler`, which writes + single exact assembly/type identity — Tenancy's `ProvisionTenantCommandHandler`, + which writes `Tenant` and its default `Organization` in one transaction per ADR-0042. - **Source:** [ADR-0042](../decisions/0042-tenant-provisioning-cross-aggregate-transaction.md); [Architecture Standards § Aggregate Ownership](01-architecture-standards.md). @@ -1340,8 +1353,14 @@ otherwise). `Cross_Aggregate_Census_Catches_Fused_Separate_Notification_And_Internal_Constructors` exercises the same production predicate with fused/two-port request and notification handlers, a non-public constructor, and same-root/non-handler negative controls. + P02b-3 Step 6 adds + `Cross_Aggregate_Census_Catches_Integration_Handlers_And_Inherited_Contracts` for + fused/separate integration ports, inherited contracts and a same-simple-name + integration offender tested independently of the other dirty subjects. The + actual provisioning type alone is the clean exception control. - **What it does not catch:** a write routed through a helper that itself holds two - ports, or through a second `DbContext` reached indirectly — the same limit + ports, arbitrary nested `ISender.Send` sequences, or a second `DbContext` reached + indirectly — the same limit [§ What a structural test proves](#what-a-structural-test-proves--and-what-it-does-not) states for every structural rule. The binding control is that the allow-list has one entry and growing it is a reviewed diff. @@ -1616,7 +1635,8 @@ catalogue as the carrier of their status — so all three are Packet 10's. - **Source:** [Database Standards § Soft Delete](05-database.md). - **Type:** xUnit over the EF model (`LearnStack.Tests.Architecture`, `PersistenceConventionTests`). **Kind:** structural. -- **Status:** **Implemented** (Packet 9, fifth review). Its companion, +- **Status:** **Implemented** (`PersistenceConventionTests`, Packet 9, fifth review). + Its companion, `The_Soft_Delete_Index_Sweep_Can_Actually_Fail`, runs the predicate over a probe context carrying one unique index of each shape — counting deleted rows, partial, containing the primary key, and on a table with no `deleted_at` — and expects exactly the first. @@ -2060,9 +2080,9 @@ diagnostic above row security, not the boundary, as its own note says. - **Source:** ADR-0003 Amendment 3 § Test requirement. - **Type:** **integration** test (Testcontainers + PostgreSQL), not an architecture test. **Kind:** behavioural. -- **Status:** **Implemented, twice.** The schema-level case is Packet 6 step 4 +- **Status:** **Implemented**, twice. The schema-level case is Packet 6 step 4 (`TenancySchemaTests`); the request-level one is Packet 7 step 11 - (`Database/TenantIsolationHttpTests`), which drives it through + (`TenantIsolationHttpTests`), which drives it through `HostClassificationMiddleware`, `TenantResolverMiddleware`, the announcement and the EF query filters, with the host header as the only input and no stubbed `ITenantContext`. The schema-level case moved forward because that class's own @@ -2084,11 +2104,11 @@ diagnostic above row security, not the boundary, as its own note says. - **Source:** ADR-0003 Amendment 3 § Test requirement; [05-database.md](05-database.md). - **Type:** **integration** test (Testcontainers + PostgreSQL). **Kind:** behavioural. -- **Status:** **Implemented, twice.** Packet 6 step 4 (`TenancySchemaTests`): a +- **Status:** **Implemented**, twice. Packet 6 step 4 (`TenancySchemaTests`): a `[Theory]` over every table carrying a `WITH CHECK` for the `INSERT` half, and `Reassigning_An_Owned_Row_To_Another_Tenant_Is_Rejected` for the `UPDATE` half, because `WITH CHECK` guards both and a rule covering one leaves the other open. Packet 7 step 11 - (`Database/TenantIsolationHttpTests`) issues the write through a request: raw SQL on + (`TenantIsolationHttpTests`) issues the write through a request: raw SQL on the ambient connection, so no query filter is in front of it and only `WITH CHECK` can refuse. The first version of that case asserted merely that an anonymous POST failed, and passed against a **deleted endpoint** and against a database with every policy @@ -2319,11 +2339,10 @@ because the filters hold, and removing both turns all five red. - **Status:** **Implemented** (`TenantContextGuardTests`, Packet 7 step 8). - **Phase:** 02a (Packet 7). - **Note:** the marker is a flag on `NpgsqlUnitOfWork`, read through the seam member - ADR-0040 Amendment 5 adds. **Only one of the eight sanctioned setters stamps it**, and - that is the honest count: `TransactionBehavior` via `SetTenantContextAsync`. Of the - other seven, **two do not exist in code yet** — the durable idempotency store, and the - integration-event transport, which is the one other setter that *opens* the ambient - transaction and will have to announce it when Phase 02b lands it — and the five that do, + ADR-0040 Amendment 5 adds. The MediatR transaction owner and P02b-3's transport + stamp it through `SetTenantContextAsync`; acceptance and prepared execution + announce their own ambient transaction before app-role commands. The durable + idempotency store remains unimplemented. The five other logical setters, `OrganizationScopeValidator`, the two standalone audit writers and the two cached projection loaders (`AuditConfig` and the tenant flags), issue raw `NpgsqlCommand`s on connections of their own, which EF interception cannot see, so they need neither a mark @@ -2589,9 +2608,12 @@ which decides identity, multiplicity, capture and classification; #### `Every_Shipped_Request_Is_Registered` -- **Asserts:** every request type with a handler — a closed `IRequestHandler<,>` or - `IRequestHandler<>` implemented in any project under `backend/src` — is registered in the - merged audit catalogue, `Off` included. An unclassified request fails the build rather +- **Asserts:** every request type with a closed `IRequestHandler<,>` or + `IRequestHandler<>`, and every concrete `IIntegrationEventHandler<>` implemented + in any project under `backend/src`, is registered in the merged audit catalogue, + `Off` included. Delivery classification keys the concrete handler, not its event; + sibling handlers remain independent. Discovery fails loudly on incomplete loads. + An unclassified request fails the build rather than defaulting to silence; at runtime, reaching step 3 unclassified is `audit_unclassified_operation`, and there is no `RequestKind.Other`. - **Why it is keyed on the request type.** `AuditLogBehavior` looks a registration up by @@ -2601,7 +2623,9 @@ which decides identity, multiplicity, capture and classification; `tenancy.organization.create` registered through provisioning, both directions stayed green, and the running system would have refused every `CreateOrganizationCommand`. - **Companion:** `The_Request_Sweep_Can_Actually_Fail` runs the predicate over exactly that - catalogue and requires it to name the missing request. + catalogue and requires it to name the missing request. P02b-3 Step 6 adds + handler-specific missing/Off/event-only and inherited-contract controls in + `AuditCoverageIntegrationTests.cs`. - **The catalogue it reads is the discovered one — so the roots are held to it.** This rule constructs every `IAuditCatalogSource` by reflection, which is what lets it see a source nobody listed and what makes it blind to the registrations: deleting the API's @@ -2784,8 +2808,9 @@ which decides identity, multiplicity, capture and classification; #### `Every_Module_With_An_Aggregate_Or_A_Request_Has_A_Matrix` -- **Asserts:** every module that ships an aggregate root or a request type — an - `IAggregateRoot<>` implementation, or a request with a handler, whose namespace is +- **Asserts:** every module shipping an aggregate root, request type or concrete + integration handler — an `IAggregateRoot<>`, request with a handler or + `IIntegrationEventHandler<>`, whose namespace is `LearnStack.Modules..…` — has `docs/modules//audit.md`. - **Why it exists beside the rule above.** `Every_Module_Has_An_AuditCoverage_Matrix` walks the spec directories that exist, so a module that ships code with no spec directory at all @@ -2797,6 +2822,7 @@ which decides identity, multiplicity, capture and classification; the Audit module itself ships aggregates and no request. - **Companion:** `The_Module_Sweep_Can_Actually_Fail` feeds the discovery an aggregate root in a module with no spec directory and no request, and requires the predicate to name it. + P02b-3 Step 6 adds `Integration_Only_Module_Requires_An_Audit_Matrix`. - **Source:** [ADR-0044 Amendment 4 § 3](../decisions/0044-audit-write-path.md); [13-documentation.md § Per-Module Specifications](13-documentation.md). - **Type:** xUnit + reflection + file scan. **Kind:** structural. @@ -3194,18 +3220,24 @@ Introduced by [Phase 02b](../roadmap/phase-02b-events-auth.md). #### `Integration_Event_Handler_Restores_Tenant_Context` -- **Asserts:** when an outbox consumer dispatches an integration event, - the inner handler scope has `ITenantContext.IsResolved == true` before - business code runs. Verifies the envelope-to-context restoration. -- **Source:** ADR-0032 § Sub-decision 12; Phase 02b deliverable. -- **Type:** integration test (Testcontainers). Runs against - `InProcessEventBus` — which invokes the shared handler interface and restores - tenant context - ([ADR-0035](../decisions/0035-demand-gated-infrastructure.md)), so the assertion does - not wait on the Dapr adapter. Consumer delivery/inbox deduplication remains - P02b-3 work. **Kind:** behavioural. -- **Status:** **Registered.** -- **Phase:** 02b. +- **Asserts:** the independent delivery executor restores resolved tenant and + organization before business construction, keeps effective SystemActor separate + from human causation, and restores the prior context after actual cleanup. The + real app-role transaction contains inbox, business, produced outbox, MUST audit + and fenced ledger completion; refusal/rollback controls preserve isolation. +- **Source:** ADR-0032 § Sub-decision 12 and + [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md#subscriber-acceptance-and-inbox-ownership). +- **Type:** actual `DeliveryExecutor` integration (Testcontainers), with constructor, + callback, prior-context, failure and independent real-default controls. + `InProcessEventBus` performs durable acceptance and constructs no business handler. + **Kind:** behavioural. +- **Status:** **Implemented** + (`DeliveryExecutionTests.Integration_Event_Handler_Restores_Tenant_Context`). + The [HTTP producer proof](../../backend/tests/LearnStack.Tests.Integration/Database/OutboxProducerHttpTests.cs) + also checks the real HTTP producer's stored trace, + announced tenant/org and exact nonempty partition; the delivery proof checks + the same quartet for its outgoing event. Actual production consumers are Active. +- **Phase:** 02b, P02b-3. #### `Integration_Event_Handlers_Use_InboxGuard` @@ -3221,7 +3253,33 @@ Introduced by [Phase 02b](../roadmap/phase-02b-events-auth.md). superseding ADR-0006/0010's handler-called guard assertion. - **Type:** architecture call-path census and integration tests (Testcontainers). **Kind:** structural + behavioural. -- **Status:** **Registered** — transport-owned inbox/delivery remains unimplemented. +- **Status:** **Implemented** (`OutboxDispatchConventionTests`, with + `DeliveryExecutionTests`, `CustomizationDefaultDeliveryTests` and + `DeliveryDispatchStoreTests`). Step 6's + [handler/manifest census](../../backend/tests/LearnStack.Tests.Architecture/IntegrationDeliveryBoundaryTests.cs) + requires all production assemblies/layers and the two exact scoped Active + bindings. It detects typed construction, inherited implementation/virtual-slot + invocation, DI resolution through statically bound generic helpers, async and + delegate bypasses, with missing-subject and dirty/clean controls. Generic type + and method argument membership states bound recursive helper traversal. + The [lifecycle census](../../backend/tests/LearnStack.Tests.Architecture/IntegrationDeliveryLifecycleBoundaryTests.cs) + pins typed unit/frame and database-transaction completion sites, including + known EF transaction/facade APIs, and their multiplicity. It also pins the + three `SetTenantContextAsync` call sites in the ordinary owner, prepared delivery + owner and durable acceptance, with missing-site and extra interface/concrete/ + inherited/async/delegate caller controls. These implement two of ADR-0040's + eight logical setter paths rather than adding a new path. These are bounded + static capability checks, not arbitrary + reflection, raw SQL transaction analysis or branch-order proofs; the real + PostgreSQL tests own inbox-before-construction and transaction atomicity. + The owner census also pins completion construction to the executor's owned + path, with extra direct/asynchronous and missing-constructor-site controls. + Earlier structural companions remain + `OutboxDispatchConventionTests.Delivery_owner_and_inbox_capabilities_are_closed` + and + [Delivery_owner_census_rejects_extra_direct_delegate_async_and_inherited_capabilities](../../backend/tests/LearnStack.Tests.Architecture/DeliveryOwnerBoundaryTests.cs). + They enumerate all production assemblies, prepared-owner minting/consumption, + inbox callers, completion subtypes and common-owner fence/commit capabilities. - **Phase:** 02b, P02b-3. #### `Integration_Events_Inherit_From_IntegrationEventBase` @@ -3311,7 +3369,7 @@ Introduced by [Phase 02b](../roadmap/phase-02b-events-auth.md). - **Status:** **Implemented** (`DomainEventProductionTests.The_owner_drains_FIFO_across_sources_saves_delete_and_detach_without_duplicates`, `TenantCreatedProducerTests`, `ProducerExecutionTests`); real save qualification, FIFO/bounds, owner/common-audit atomicity and cleanup. -- **Phase:** 02b, P02b-1; production delivery remains P02b-3 after G1. +- **Phase:** 02b, P02b-1; P02b-3 Steps 1–5 deliver production subscriber execution. #### `Domain_Event_Handlers_Are_Outbox_Only` @@ -3348,7 +3406,8 @@ Introduced by [Phase 02b](../roadmap/phase-02b-events-auth.md). #### `OutboxProcessor_NeverBlocks_OnSingleMessageFailure` -- **Asserts:** ordinary publication failure permits unrelated eligible work to +- **Asserts:** ordinary publication failure and a sustained terminal-audit failure + backlog permit unrelated normal publication and expired-claim recovery to progress. A callback ignoring cancellation retains its live task-owned scope, blocks its own partition and makes health unhealthy; unrelated keys progress within the approved four-task bound. Saturation stops acquisition, late results @@ -3359,7 +3418,9 @@ Introduced by [Phase 02b](../roadmap/phase-02b-events-auth.md). - **Status:** **Implemented** (`OutboxProcessorTests`, `OutboxPublishAttemptTests`): actual hosted ordinary-failure, retained-scope/cleanup, partition hold, four-task saturation and finite-shutdown proofs; hostile synchronous construction and - cancellation callbacks retain ownership through completion. + cancellation callbacks retain ownership through completion. A real PostgreSQL + audit-grant refusal backlog outliving its leases preserves terminal rollback + while independent durable acceptance and recovery continue. - **Phase:** 02b. #### `Outbox_Claim_IsHeld_Until_Dispatch_Completes` @@ -3455,11 +3516,13 @@ Introduced by [Phase 02b](../roadmap/phase-02b-events-auth.md). #### `Outbox_Pending_Subscriptions_Preserve_Events` -- **Asserts:** Real TenantCreated rows and aliases remain unchanged through repeated +- **Asserts:** TenantCreated rows and aliases assigned Pending manifests remain + unchanged through repeated Pending polls, even with malformed stored payloads: no acquisition, attempts, validation or handler construction. Declared intentional absence is a distinct tested acknowledgement policy. Test-owned Active contracts exercise the shipped - publish/mark path; real subscriptions stay Pending until P02b-3. + publish/mark path. P02b-3 Step 5 activates both real production subscriptions; + Pending-manifest fixtures retain the policy proof. - **Source:** [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md); [P02b-2 proof obligations](../roadmap/phase-02b-events-auth.md#decisive-proof-obligations). - **Type:** integration tests (Testcontainers) and unit tests of production @@ -3469,10 +3532,11 @@ Introduced by [Phase 02b](../roadmap/phase-02b-events-auth.md). `OutboxProductionCompositionTests`, `OutboxEventRegistryTests`), with these proofs: [OutboxProcessorTests.Real_provisioning_rows_remain_Pending_across_hosted_polls_without_construction_or_mutation](../../backend/tests/LearnStack.Tests.Integration/Database/OutboxProcessorTests.cs), [OutboxDispatchStoreTests.Real_TenantCreated_contracts_and_aliases_are_held_before_any_payload_validation](../../backend/tests/LearnStack.Tests.Integration/Database/OutboxDispatchStoreTests.cs), - [OutboxProductionCompositionTests.Real_provisioning_contract_remains_pending_for_both_distinct_default_consumers](../../backend/tests/LearnStack.Tests.Unit/Api/Composition/OutboxProductionCompositionTests.cs), + [OutboxProductionCompositionTests.Real_provisioning_contract_activates_both_distinct_default_consumers_with_exact_handler_bindings](../../backend/tests/LearnStack.Tests.Unit/Api/Composition/OutboxProductionCompositionTests.cs), and [OutboxEventRegistryTests.Active_manifest_and_discovered_handlers_must_match_in_both_directions](../../backend/tests/LearnStack.Tests.Unit/Infrastructure/Messaging/OutboxEventRegistryTests.cs) - prove real Pending holds and composition refusal. Test-owned Active publication - runs in [OutboxProcessorTests.A_poison_first_row_terminalizes_with_MUST_audit_and_the_following_real_handler_progresses](../../backend/tests/LearnStack.Tests.Integration/Database/OutboxProcessorTests.cs); + prove declared Pending holds, actual Active bindings and composition refusal. + Test-owned Active publication + runs in [OutboxProcessorTests.A_poison_first_row_terminalizes_with_MUST_audit_and_following_acceptance_progresses](../../backend/tests/LearnStack.Tests.Integration/Database/OutboxProcessorTests.cs); [OutboxProcessorTests.Explicit_intentional_absence_acknowledges_without_constructing_a_handler](../../backend/tests/LearnStack.Tests.Integration/Database/OutboxProcessorTests.cs) proves the distinct intentional-absence policy. - **Phase:** 02b, P02b-2. @@ -3506,6 +3570,108 @@ Introduced by [Phase 02b](../roadmap/phase-02b-events-auth.md). Focused verification and both fresh fix reviews pass at `606362b`. - **Phase:** 02b, P02b-2. +#### `Integration_Event_Publication_Is_Durable_Acceptance` + +- **Asserts:** Publish success means every required stable subscription has an + immutable, durably accepted envelope without business-handler construction. + Identical duplicates are idempotent, conflicting copies are refused and + partial/uncertain acceptance retries safely without spending subscriber budgets. +- **Source:** [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md#subscriber-acceptance-and-inbox-ownership) + and the [P02b-3 package](../roadmap/phase-02b-events-auth.md#p02b-3-decision-package-2026-10-13). +- **Type:** real PostgreSQL/app-role integration and construction controls. + **Kind:** behavioural. +- **Status:** **Implemented** (`DurableEventAcceptanceTests`, `InboxGuardTests`, + `OutboxProcessorTests`, P02b-3 Step 2). Includes independent publisher transactions, + blocked duplicate commit/rollback, immutable collisions, dropped COMMIT response, + constructor refusal controls and the actual producer worker's durable handoff. +- **Phase:** 02b, P02b-3. + +#### `Delivery_Retries_And_Replay_Are_Subscription_Scoped` + +- **Asserts:** Independent subscription attempts/backoff/leases, terminal requests + and replay preserve sibling and producer outcomes. Actual completion is fenced + with the business transaction and sticky timeout gate; final-claim unknown + COMMIT recovers from atomic inbox/ledger state. Replay resets one terminal + ledger pair only, never its completed inbox or a healthy sibling. +- **Source:** [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md#subscriber-acceptance-and-inbox-ownership) + and the [P02b-3 package](../roadmap/phase-02b-events-auth.md#p02b-3-decision-package-2026-10-13). +- **Type:** real PostgreSQL worker/owner integration, controlled cancellation, + lost-COMMIT and state-transition mutants. **Kind:** behavioural. +- **Status:** **Implemented** (`DeliveryDispatchStoreTests`, `DeliveryProcessorTests`, + `DeliveryExecutionTests`, P02b-3 Step 4). Real-role tests cover independent + retries, final-claim recovery, exact-tuple replay with terminal sibling controls, + unchanged-row lock expiry, lost COMMIT and postcommit cleanup ownership. +- **Phase:** 02b, P02b-3. + +#### `Delivery_Terminal_Audit_Identity_Is_Row_Bound` + +- **Asserts:** The successful fenced subscriber terminal UPDATE alone mints a + restricted witness binding actual tenant/org/subscription/event/token and live + transaction. Mismatched draft identities/operation/class/type/transaction and + failed audit roll back both writes; uncertain COMMIT never duplicates terminal + audit. The scheduler has no retained payload/causal-envelope SELECT privilege. +- **Source:** [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md#atomic-terminal-audit-and-privilege-boundary) + and [ADR-0044 Amendment 11](../decisions/0044-audit-write-path.md#amendment-11--shared-delivery-capture-exclusions-2026-10-13). +- **Type:** nonempty production mint/writer census with dirty/clean controls and + real-role atomic transition integration. **Kind:** structural + behavioural. +- **Status:** **Implemented** (`OutboxDispatchConventionTests`, + `DeliveryDispatchStoreTests`, P02b-3 Step 4). Exact producer/subscriber mint + and writer censuses have planted dirty/clean controls; real-role tests pin draft + substitutions, audit rollback, matching witnesses and uncertain-COMMIT reconciliation. +- **Phase:** 02b, P02b-3. + +#### `Delivery_Execution_And_Observation_Are_Bounded` + +- **Asserts:** Actual API subscriber worker, deadlines, ignored cancellation and + disposal retain safe task-owned resources within the accepted capacity/shutdown + bounds. Postcommit cleanup cannot undo/repeat a durable business effect. Closed + diagnostics/finite labels, exact snapshot cadence and prior-snapshot retention + preserve truthful registered health; gauge callbacks execute no SQL. +- **Source:** [ADR-0046](../decisions/0046-durable-event-delivery-and-recovery.md#diagnostics-and-execution-bounds) + and the [P02b-3 package](../roadmap/phase-02b-events-auth.md#p02b-3-decision-package-2026-10-13). +- **Type:** production registration/observation/call-path census with planted + dirty/clean controls, unit probes and real worker integration. + **Kind:** structural + behavioural. +- **Status:** **Implemented** (`OutboxDispatchConventionTests`, + `DeliveryProcessorTests`, `DeliveryObservationTests`, `DeliveryAttemptTests`, + P02b-3 Step 4). The actual worker proves retained callbacks/cleanup, capacity, + shutdown and refresh cadence. Production capability and memory-observation + censuses have planted controls; + literal-SQL checks cover exact declared writers/helpers, not arbitrary dataflow. +- **Phase:** 02b, P02b-3. + +#### `Customization_Default_Initialization_Preserves_Touched_Concepts` + +- **Asserts:** Both actual default subscriptions create one untouched root each + with owner-minted UUIDv7, one generation bump and MUST audit. Any Draft, Active, + Deprecated, successor-only or soft-deleted concept is preserved. Semantic-key + serialization makes competing registration and initialization converge without + overwrite, republish or resurrection; each effect has independent inbox state. +- **Source:** [ADR-0057](../decisions/0057-transactional-event-production-and-completion.md#first-useful-production-flow) + and the [P02b-3 package](../roadmap/phase-02b-events-auth.md#p02b-3-decision-package-2026-10-13). +- **Type:** actual consumer integration, lifecycle/race/atomic-audit controls. + **Kind:** behavioural. +- **Status:** **Implemented** — `CustomizationDefaultInitializationTests`, + `CustomizationDefaultDeliveryTests` (P02b-3 Step 5): app-role lifecycle, + ordinary-registration races, backlog/fresh delivery and atomic audit proofs. +- **Phase:** 02b, P02b-3. + +#### `Seeder_BuiltIns_Use_Semantic_Identity` + +- **Asserts:** Seeder discovers and exactly verifies built-ins by semantic + key/version while preserving warm/generated IDs and authored exact IDs/pins. + It uses the ordinary app-role pipeline without a worker or dispatcher credential; + reruns are neutral, conflicting lifecycle/shape refuses without overwrite, and + fresh seeding/delivery races converge in both winning orders. +- **Source:** [ADR-0057](../decisions/0057-transactional-event-production-and-completion.md#first-useful-production-flow) + and the [P02b-3 package](../roadmap/phase-02b-events-auth.md#p02b-3-decision-package-2026-10-13). +- **Type:** actual Seeder/module/worker integration with warm-ID and race controls. + **Kind:** behavioural. +- **Status:** **Implemented** — `SeederTests`, + `CustomizationDefaultDeliveryTests` (P02b-3 Step 5): generated/warm identity, + exact-shape/lifecycle refusal, neutral reruns and both Seeder/delivery race orders. +- **Phase:** 02b, P02b-3. + ### Demand-gated: lands with its adapter These rules are agreed but cannot be written until the technology they constrain is diff --git a/docs/standards/README.md b/docs/standards/README.md index 211483d4..ae021000 100644 --- a/docs/standards/README.md +++ b/docs/standards/README.md @@ -67,7 +67,7 @@ bookkeeping pass. ### Honest status today -The table below describes the implementation as of 2026-10-11: +The table below describes the implementation as of 2026-10-13: [Phase 02a](../roadmap/phase-02a-kernel-tenancy.md) and P02d-1–6 are complete and merged. P02d-5's ingress, configured server caller, dynamic entry and frontend guards are delivered through PR #26; the @@ -121,9 +121,9 @@ included. | 15 | [Performance](15-performance.md) | **Active** | P02d-7 delivers the always-running `public demo + lighthouse` job under ADR-0056: nine seeded pages, 27 audits, actual-page/source/census controls and hard normalized gzip/CLS/automated-accessibility checks. Timing medians are advisory. The [Step 4 record](../roadmap/phase-02d-walking-skeleton.md#p02d-7-step-4--ci-enforcement-and-phase-exit-2026-10-11) owns real CI and required-check receipts. Production p95/field telemetry/load tests remain [Phase 11](../roadmap/phase-11-production-hardening.md). | | 16 | [Accessibility](16-accessibility.md) | **Active** | P6 delivers catalog/course/lesson and controlled-state semantics, all 31 active recommended jsx-a11y rules at error severity with per-rule actual-config dirty/clean controls, and real product HTML/Flight proofs. The [packet closeout](../roadmap/phase-02d-walking-skeleton.md#p02d-6-packet-closeout-2026-10-10) records keyboard/focus/reflow/contrast observations and the maintainer's passing VoiceOver smoke. This is enforcement for delivered public-page scope, not a full WCAG audit. P02d-7 adds hard Lighthouse automated accessibility and a narrow library browser walkthrough; full Playwright Test/axe remains [Phase 06](../roadmap/phase-06-renderer-admin-studio.md). | | 17 | [Code Review](17-code-review.md) | **Active** | Applied to every pull request merged so far; the zero-tolerance blocker list is in live use. | -| 18 | [Audit Coverage](18-audit-coverage.md) | **Active** | Packet 9 lit the write path under [ADR-0033](../decisions/0033-audit-durability-model.md) and [ADR-0044](../decisions/0044-audit-write-path.md): classification at step 3, MUST rows on the business transaction before `COMMIT`, `audit_log` / `audit_config` with their append-only layers, and the catalogue ↔ matrix join enforced in both directions and per request type by `AuditCoverageTests`. Retention and partitioning are [Phase 11](../roadmap/phase-11-production-hardening.md)'s. | +| 18 | [Audit Coverage](18-audit-coverage.md) | **Active** | Packet 9 lit the write path under [ADR-0033](../decisions/0033-audit-durability-model.md) and [ADR-0044](../decisions/0044-audit-write-path.md): classification at step 3, MUST rows on the business transaction before `COMMIT`, `audit_log` / `audit_config` with their append-only layers, and the catalogue ↔ matrix join enforced in both directions and per request/concrete integration-handler identity by `AuditCoverageTests`; actual API/Seeder catalogue parity includes Off coordinators. Retention and partitioning are [Phase 11](../roadmap/phase-11-production-hardening.md)'s. | | 19 | [Permissions](19-permissions.md) | **Adopted** | No permission key, policy or role exists. Lands in [Phase 03](../roadmap/phase-03-identity-admin.md). | -| 20 | [Infrastructure Stack](20-infrastructure-stack.md) | **Active** | `ISecretProvider` shipped in Packet 3, the foundation ports and their defaults in Packet 5, and the entitlement socket in Packet 9; `DeploymentMode` branching happens once, at the composition root. Packet 10 made the bans this document states mechanical: no module reaches a cache client, a Hub namespace, `audit_log` or `platform_entitlement_cache`, and every entitlement key a call site names is a registry member. The adapters themselves — Dapr, Kafka, Valkey, Vault, APISIX — arrive on [ADR-0035](../decisions/0035-demand-gated-infrastructure.md)'s triggers, which is the model rather than a gap in it. | +| 20 | [Infrastructure Stack](20-infrastructure-stack.md) | **Active** | `ISecretProvider` shipped in Packet 3, the foundation ports and their defaults in Packet 5, and the entitlement socket in Packet 9; `DeploymentMode` branching happens once, at the composition root. P02b-2 delivers producer dispatch; P02b-3 Steps 1–5 add durable acceptance, shared inbox and independently scheduled default consumers. The [Step 6 record](../roadmap/phase-02b-events-auth.md#p02b-3-step-6--production-enforcement-and-pr-closeout-2026-10-13) owns current enforcement and review status. Packet 10 made the bans this document states mechanical: no module reaches a cache client, a Hub namespace, `audit_log` or `platform_entitlement_cache`, and every entitlement key a call site names is a registry member. The adapters themselves — Dapr, Kafka, Valkey, Vault, APISIX — arrive on [ADR-0035](../decisions/0035-demand-gated-infrastructure.md)'s triggers, which is the model rather than a gap in it. | | 21 | [Architecture Tests Catalogue](21-architecture-tests-catalogue.md) | **Active** | The catalogue's own § Implemented today carries the counts, and `The_Catalogue_Counts_Its_Own_Rules` recomputes them, so this row does not keep a third copy. Rules run in the architecture assembly and beside it in the unit, integration and frontend suites — each where it can actually fail, against an applied schema, a real host or a real ESLint configuration. `Every_Implemented_Rule_Names_A_Test_That_Exists` holds each Implemented entry to a method of that name. `No_Architecture_Test_Is_Skippable` governs backend suites; P02d-5 delivers G38(d)'s guarded frontend runner and planted skip/todo, empty-report and omitted-workspace controls. | P02d-4's [accepted decision package](../roadmap/phase-02d-walking-skeleton.md#p02d-4-accepted-answers) diff --git a/frontend/apps/web/src/test/fixtures/demo-identities.json b/frontend/apps/web/src/test/fixtures/demo-identities.json index cb2b090e..9902f051 100644 --- a/frontend/apps/web/src/test/fixtures/demo-identities.json +++ b/frontend/apps/web/src/test/fixtures/demo-identities.json @@ -87,12 +87,8 @@ { "kind": "id", "value": "01930000-0000-7000-8000-0000000000a2" }, { "kind": "id", "value": "01930000-0000-7000-8000-0000000000b1" }, { "kind": "id", "value": "01930000-0000-7000-8000-0000000000b2" }, - { "kind": "id", "value": "01930000-0000-7000-8000-0000000000c1" }, - { "kind": "id", "value": "01930000-0000-7000-8000-0000000000c2" }, { "kind": "id", "value": "01930000-0000-7000-8000-0000000000c3" }, { "kind": "id", "value": "01930000-0000-7000-8000-0000000000c4" }, - { "kind": "id", "value": "01930000-0000-7000-8000-0000000000d1" }, - { "kind": "id", "value": "01930000-0000-7000-8000-0000000000d2" }, { "kind": "id", "value": "01930000-0000-7000-8000-0000000000d3" }, { "kind": "id", "value": "01930000-0000-7000-8000-0000000000d4" }, { "kind": "id", "value": "01930000-0000-7000-8000-0000000000e1" },