Skip to content

[Feature] F5.3: Deduplication, durable ingestion, recovery, and operational health #8229

Open
2 of 2 issues completed

Description

@sfmskywalker

Program: #8194
Parent: #8200
Planning ID: F5.3

Outcome and scope

Test duplicate/reordered events, durable acknowledgment, recovery, reconciliation, and health visibility; document actual delivery guarantees.

Acceptance criteria

  • Deliver the specified capability and demonstrate each behavior above with targeted checks or documented research evidence.
  • Preserve existing package/activity identities and supported contracts, or provide an explicit migration.
  • Document setup, limitations, dependencies and verification results.

Readiness and execution — 2026-10-08

The execution ownership prerequisite #8651/#8652, main promotion #8655/#8657 and offline Admission task #8658 are accepted. #8660 merged as a866284727cce19068b417a908bbe4a2573815d5; main and the shared program branch were verified at its exact reviewed/tested tree. PostgreSQL admission, one-shot authority, guarded local execution/continuation and bootstrap are implemented and verified within the declared isolated-host scope. #8661 under #8219 is now the sole active Task, supplying the seven-package prerequisite for the later transport work. This feature remains open for its remaining transport, outbound, live pilot and operational acceptance; #8194 owns the queue.

Explicit fixture settings and default-deny grants support offline verification. Owner-selected live numeric settings, delegation/operator authority and credentials still gate sandbox activation. Real Socket Mode ack/reconnect/restart, legacy Watch/CreateMessage compatibility, durable outbound uncertainty, live same-thread reply and operational acceptance remain separate required outcomes. This feature remains open until those behaviors are demonstrated; no live Slack setup or delivery is claimed.

Follow AGENTS.md, existing test/ conventions and dated ADRs under doc/adr/. Scope is Elsa 3 unless explicitly a portability assessment. Repository consolidation includes Core, all Extensions, and Studio; release boundaries remain explicit and independent.

Hierarchy: native parent/sub-issue relationships verified on 2026-09-23. Explicit blocking dependencies are configured separately; Markdown links remain references. See #8194.

Approved prerequisite design and active Task

Reviewed admission and execution-boundary design defined the bounded isolated-host contract implemented by #8658. Exact-head implementation and hosted acceptance now supersedes the earlier queued/design-only status. This accepts neither live Socket Mode transport nor outbound delivery; their separate criteria below remain required.

Generic management/import/alteration and distributed/Proto.Actor adapters are unsupported by the first isolated-host design, not acceptance gates for the initial Socket Mode pilot. Any extension needs a concrete use case and separate scope/design review. The Socket Mode adapter/reconnect/ack proof, additive legacy compatibility, outbound-write uncertainty, live threaded reply and operational acceptance remain required. Owner-selected live retention/capacity settings, delegation, operator ownership and actual app/workspace/credentials remain activation gates; explicit fixture values suffice for offline contract verification. Reuse the existing exact-instance grant and credential infrastructure; no implicit outbound grant or new credential store is authorized.

Activity

  1. added
    elsa 3This issue is specific to Elsa 3
    enhancementNew feature or request
    prio highIs on the roadmap for the near-future
    on Sep 23, 2026
  2. sfmskywalker commented on Oct 7, 2026

    @sfmskywalker
    MemberAuthor

    Source-grounded next ready leaf: #8651 preserves execution-cycle tracking across intermediate checkpoints and final owner writes. It is a native child of this Feature and a prerequisite for the proposed #8236 Socket Mode restart proof. The lead reviewed the scheduler/commit/runner paths and later writes in alteration, cancellation and legacy host callers; the task contains explicit compatibility and hosted acceptance requirements.

    #8651 is Todo / Agent Ready / Pending, unassigned. #8649 remains the sole active implementation task. This Feature remains open: durable ingress, deduplication/recovery, provider behavior and operational health still require their own complete acceptance.

  3. sfmskywalker commented on Oct 7, 2026

    @sfmskywalker
    MemberAuthor

    Accepted through merged PR #8652, program merge 23b852ad6262c4bf07b9f311090c892dbecd740b at 2026-10-07T20:33:44Z. Lead verified the merge tree exactly equals reviewed head 5dee771bb7bb497147a47d71aab6a39e5275e627 (tree 4f6a2e9200c5d8bf549c29b21d84cf829601b159).

    All required exact-head CI passed, including Packages run 37679237178 (broad unit/integration, component tests and Compile+Pack), Integration program tools, mapped-source package proof and the expanded execution-cycle proof. Independent Elsa 3 source/runtime-evidence review is APPROVE + HIGH; Greptile is 5/5 on that same head; both review threads are resolved. Copilot was requested but no delivered review is claimed. CodeRabbit skipped review for this base branch and is not approval evidence.

    Retained focused evidence: 857 passed, zero skipped, five fixture compiles and twelve production builds across net8.0/net9.0/net10.0. Both lead and independent audits reconciled source/input hashes and artifact digest. This includes the eight actual ProtoActor Run/Cancel trailing-save cases that were absent from the earlier 845-case proof. Failed earlier attempts remain recorded.

    Execution ownership now spans intermediate checkpoints and each explicit owner's final awaited write, including imported ProtoActor saves, and releases on operation unwind. The tests distinguish real lifecycle/drain behavior from controlled mocked persistence failures. This does not implement Slack durable ingestion, restart guarantees, detached background activity semantics, or the separate #8489/#8420 defects. Parent #8229 and program #8194 remain open.

    The merge uses [skip ci] to avoid duplicating the accepted exact-head jobs; no separate merge-head CI run is claimed. Publication/deployment jobs were skipped and no publisher, credentials or environment changed. Final program integration into main remains open. #8651 is closed; #8294 is next.

  4. sfmskywalker commented on Oct 7, 2026

    @sfmskywalker
    MemberAuthor

    Offline admission/start design checkpoint for #8229, cross-referencing #8236 and #8199.

    The identified technical design blockers are resolved at design-review level for the bounded isolated-host scope. This does not mean the admission ledger or guard is implemented, tested or accepted. Scheduling update, 2026-10-08: #8655 is accepted and closed after exact-main verification. #8658 is now the sole active Task implementing this approved bounded design. Design approval is not implementation or runtime acceptance; the historical review bindings and normative text below remain intact.

    The proposed single coherent leaf combines a distinct EF Core/PostgreSQL admission ledger with an enforced Core-neutral execution boundary and one in-process, single-execution-host LocalWorkflowRuntime adapter. It includes durable-before-ack recording, stable provider-event identity, insert/start uncertainty, one-shot initial authority, protected direct pipeline delegates, trusted continuation provenance, pre-effect owned mutation denial, retention/cleanup mechanics and reviewed bootstrap of a pinned workflow plus trusted subscription. Required regression/build evidence is specified below; none has run for this proposed leaf.

    Generic management/import/alteration and distributed/Proto.Actor adapters are unsupported by this first isolated-host design. They are not acceptance gates for the initial Socket Mode pilot; extending support requires a concrete use case and separate scope/design review. Socket Mode transport/reconnect/ack proof, additive Watch/CreateMessage integration and legacy compatibility, durable outbound-write uncertainty, live threaded reply and operational acceptance remain required. Explicit fixture values can verify offline retention/capacity and default-deny authority contracts; owner-selected live values, delegation, operator responsibility, workspace/app/channel provisioning and credential/consent authority gate activation. No numeric policy or credential grant is approved here; the ADR's numeric alternatives remain proposals only.

    Scope clarification, 2026-10-08: the generic-adapter obligation in the earlier wording exceeded the pilot acceptance criteria. The two scope sentences below are corrected; the original review hashes identify the previously approved design documents. Execution safeguards and required pilot/operational outcomes are unchanged.

    Design approval bindings:

    • Conceptual ADR: SHA256 978635ebb5d201f4c9516a13e40ac1f62ee5ca47d3a5ec5011e519eb7e096d9d.
    • Isolated-host guard: independent DESIGN APPROVE/HIGH, SHA256 735902324f633aab0aca09d92156c6c28c34f9fbe8a645b4215dc53264a97276.
    • Guard with bootstrap clarification: independent DESIGN APPROVE/HIGH, SHA256 47226ac25bfd201e0d127e8e833096776b850d9ca79534463d1ea5bda95fbc37.

    Source review binds Core commit 12365b98791b5333fcbe9781abd0346314fc195b. The subsequent 5d79ef9876e8f9148ee8a341a851434cd8bed8bb changes four Python test files; Git-object comparison confirms the reviewed Core/Runtime/API/Alterations/Management production sources are unchanged. This source check supplies no new admission runtime proof.

    The full canonical design text follows. Presentation removes local-only links and updates superseded review-status sentences; normative requirements are unchanged. Underlying local artifact hashes before those editorial changes: ADR ecb2901cf3f5641ebe7c9ce22804cd50257006e1473e11bc4c9b4cc5262695f5, guard/bootstrap 47226ac25bfd201e0d127e8e833096776b850d9ca79534463d1ea5bda95fbc37. Historical source/evidence references remain identified as such.

    Full canonical admission/start ADR

    Durable pre-instance admission with explicit start authority

    Status: Approved offline conceptual design with independently approved isolated-host guard/bootstrap refinements, 2026-10-08. Approval bindings are recorded above. No implementation acceptance, Task creation/activation, Agent Ready claim or live authorization. Explicit fixture values permit offline mechanics within the reviewed scope; live values/authority remain activation gates.

    Context and provider

    Accepted #8236/#8229 require trusted admission, durable-before-ack recording, provider-event deduplication, recovery, withdrawal and separate outbound authorization. #8652 fixes execution ownership; #8654 fixes Mongo definition CAS. Neither supplies this ledger.

    Socket Mode uses refreshed connections, Events API payloads and envelope_id acknowledgments. Socket authentication does not establish Elsa tenant/subscription authority. Envelope identity is transport metadata; HTTP retry descriptions establish no Socket retention horizon. Official Socket Mode reference.

    Recommend a distinct EF Core/PostgreSQL admission provider, reusing Elsa tenant/DbContext/migration and revision-predicate update/transaction conventions. Existing PostgreSQL fixtures exercise independent processes. Do not extend the credential lifecycle store with ingress responsibilities. Additional providers require separate conformance; SQLite local tests do not establish multiworker support.

    One ledger transaction owns authoritative subscription activation/withdrawal epochs, admission transitions and capacity accounting. Workflow insertion remains a separate operation: no shared transaction with workflow/credential stores or runtime is assumed.

    Stable identity

    Resolve trusted tenant/environment, Slack app and workspace/enterprise installation scope, channel, logical subscription and pinned published definition. The installation namespace identifies that logical app/workspace scope, independently of socket connection, credential generation, token rotation or reinstall. Reinstall reconciles the existing namespace; payloads cannot create a new namespace.

    Unique admission key: (tenant, environment, logical-installation, logical-subscription, provider-event-id). Subscription edits, definition revisions and reconnects keep identity; their revisions are pinned data. One event per logical subscription is intentional fan-out, independently authorized for each subscription. Retirement prevents automatic identity reuse/recreation. An explicitly new subscription needs a fresh trusted activation boundary and rejects/quarantines earlier events; it does not implicitly replay history. These semantics require tests across edits, reinstall, retirement and fan-out.

    Store the allocated instance ID, routing/input fingerprint, bounded minimal payload, state/revision, attempt/fence/expiry and audit timestamps. No credential material. Identity conflicts quarantine instead of adopting unrelated rows.

    State machine

    Durable state Meaning and recovery
    Admitted Eligible event and allocated instance ID committed; envelope may be acknowledged. Duplicate returns the same record.
    Creating Fenced insert-only materialization attempt; no execution authority.
    Materialized Matching instance and known completion of the selected creation/notification chain; no start attempt yet.
    StartPreparing Durable one-attempt reservation before permit transaction; no authority. Lost/uncertain owner requires reconciliation.
    StartAuthorized Permit committed; the row itself is never an invocable capability.
    ExecutionObserved Actual runner/persistence observation recorded; suspended or long-running workflows remain nonterminal.
    RecoveryRequired Uncertain creation/notification, permit/start/effect, or conflict; no automatic replay.
    Terminal Explicit Completed, SuppressedBeforeStart or Resolved disposition, subject to terminal conditions below.

    Resolve trusted routing and human-message/loop filters before admission. Commit Admitted before ack. Unknown persistence is not acknowledgement success. Ack loss/redelivery reconciles the same key. Bounded indexed recovery queries expose safe metadata.

    Creating claims use revision/fence predicates. Reconcile uncertain insertion against the allocated ID, tenant, pinned definition and admission marker. A match proves identity only: CreateAsync calls AddAsync then sends a saved notification. It does not prove notifications completed or were effect-free. Unknown notification/materialization outcome becomes RecoveryRequired; never rerun notifications or CreateAsync merely because a matching Pending row exists. Retry absent insertion only when the previous operation cannot still commit, or with the same unique ID under proven insert-only conflict handling; that still does not resolve notification effects. Audit the actual supported host's materialization, activation, middleware and notification chain before enabling it. Unsupported/unaudited chains fail closed.

    One-shot start authority

    After eligibility/fence checks, persist StartPreparing with a unique attempt. A ledger transaction locks/checks the active authoritative subscription epoch and conditionally commits StartAuthorized. Only definitive success of this first commit returns a process-local, nonserializable, one-shot capability. Consume it atomically before entering the runner; failure after consumption never resets it.

    An uncertain commit, readback, process restart, duplicate request or recovery scan returns no invocable capability, even when StartAuthorized or a Pending instance is visible. A duplicate request preserves known durable state; it does not demote Completed or ExecutionObserved. Classify genuinely uncertain permit/start outcomes as RecoveryRequired; never reissue/reset a permit or replay through lease expiry. Definitely pre-attempt materialization work can recover under its own proven rules. A final eligibility read can catch known withdrawal but cannot close the race after that read. Already-issued start authority may still run.

    Fences reject stale ledger claims/finalizations; they do not cancel delayed authorized execution or provider calls. Record runner return and durable trailing writes using #8652 ownership. Return can mean suspension, fault or completion; it is not proof of workflow termination or exactly-once effects. Stronger automatic recovery requires a separately reviewed checkpoint/idempotency contract.

    Execution-boundary closure still required

    A process-local capability alone does not protect the allocated Pending instance from another caller. At accepted object eccc39fe15d2e525af866bd5a2e59f12821de381, src/modules/Elsa.Workflows.Runtime/Services/LocalWorkflowClient.cs:190 accepts Running/Pending instances; existing-instance execution bypasses its activation gate. Dispatcher/resumers reach this path. WorkflowHost.cs:94 and src/modules/Elsa.Http/Middleware/HttpWorkflowsMiddleware.cs:230 call the runner directly.

    First support is an isolated in-process LocalWorkflowRuntime, single execution host, excluding management mutation/import routes and Alterations. Owned control mutations fail closed before effects. Every host accessing its store must enforce the guard. Multi-process PostgreSQL admission proof does not establish clustered workflow execution. Proto.Actor serializes requests and cannot transport this capability; remote/actor and distributed-wrapper adapters require separate review.

    Required guard design: authoritative admission ownership by trusted tenant/environment/instance identity, before WorkflowRunner.RunAsync(WorkflowExecutionContext) sends WorkflowExecuting (src/modules/Elsa.Workflows.Core/Services/WorkflowRunner.cs:207-222). All built-in runner overloads converge there. Bind/consume the capability for exactly that invocation/context; status, properties, bookmark/activity handles are not authority. Preserve Core/provider dependency boundaries. Audit earlier context construction, registration and scheduling effects. Pipeline middleware is too late for runner notifications: direct ExecuteAsync and public Pipeline delegate invocation also need enforced protection (src/modules/Elsa.Workflows.Core/Pipelines/WorkflowExecution/WorkflowExecutionPipeline.cs:22-37). Custom executors/unguarded hosts are unsupported, not implicitly protected.

    Retain ownership protection while startable. Continuation requires ledger-bound committed checkpoint/revision and unconsumed bookmark provenance, never imported state alone. Bind immutable trusted identity/revision and prepared scheduling/delegate values, not only context reference. The guard contract reproduced in the next details block specifies pre-effect mutation denial, earlier callback audit and tests; its isolated-host architecture and bootstrap have independent design approval.

    Acceptance: hold the original capability while competing client/dispatcher/host/HTTP/runner/pipeline paths attempt initial execution; require zero start notifications/activity effects. Repeat with forged request/status, uncertain commit/restart and ownership-cleanup races. Prove real initial suspension, final writes/unwind, then legitimate bookmark resumption without reusing initial authority. Admission plus this guard remains one coherent proposed leaf; scheduling depends on promotion and implementation acceptance remains outstanding.

    Withdrawal coordination

    The first supported host must coordinate authoritative ledger withdrawal before unpublish, disconnect or tenant deletion reports completion, across every supported actual entry point. Fail closed if that coordination is absent; asynchronous notification alone is insufficient. Ledger withdrawal first forbids a later start permit; permit first means already-admitted authority, possibly unknown/in-flight.

    Withdraw first, then perform the external lifecycle mutation. If that later mutation fails, remain withdrawn and expose explicit reconciliation; do not silently reactivate. This orders authority without claiming cross-store atomicity. Separately stored definition/connection checks record revision/time; credential use independently revalidates authorization. Already-authorized starts may still run after withdrawal and require audit/reconciliation, not retroactive cancellation promises. Test every supported lifecycle entry point and both race outcomes before host support is claimed.

    Terminal cleanup and proposed retention

    Completed requires durably terminal workflow state and resolved outstanding operation uncertainty. SuppressedBeforeStart requires no issued capability and no unresolved creation/notification effects. Resolved requires an audited operator disposition establishing quiescence of every outstanding invocable capability/delayed authorized owner, as well as no active execution or outstanding unknown effects. Lease expiry or absence of an active workflow is not proof of that quiescence. Materialized, StartPreparing, StartAuthorized, RecoveryRequired and suspended/long-running ExecutionObserved are nonterminal. Never infer terminality from age, lease expiry, Pending or runner return.

    Set terminal_at once when a valid terminal transition commits; terminal-payload retention starts there. Retain admitted/provider-event timestamps separately. Validate required maximum event age, allowed clock skew and missing/invalid/future/late timestamp disposition. For bounded identity retention, retain keys beyond the accepted age plus skew and enforce late-arrival quarantine even after keys are purged; never freely readmit forgotten old events. Nonexpiring identity retention must be explicitly configured.

    Cleanup conditionally changes/deletes only an eligible terminal row at the expected state/revision and cutoff, preserving dedup identity until its separate horizon. Claim, cleanup, duplicate, capacity reservation/release and admission races require atomic store predicates/accounting; cleanup cannot erase newly active/ambiguous state or release capacity twice. Capacity pressure rejects new admission without false ack and raises health signals. Withdrawal precedes offboarding/namespace retirement, which requires a durable retirement/activation barrier against automatic recreation.

    Owner policy options, all proposals with no default:

    • A, recommended sandbox: terminal payload seven days; minimal identities retained until explicit offboarding/deletion policy. Better late-duplicate protection; ongoing metadata/privacy cost.
    • B: payload seven days; identities ninety days; validated event-age/skew gate quarantines late/unclassifiable arrivals. Bounded storage; legitimate late events may require review.
    • C: immediate terminal-payload deletion; identities thirty days with the same gate. Less content retention and weaker diagnostics.

    Fixtures declare values explicitly; missing policy disables new admission. Workflow/export copies have separate retention. Owner values, deletion authority and capacity limits gate activation, not construction of these contracts.

    Outbound authority and readiness

    Ingress never mints grants. Existing exact-instance manage/inspect/share/use policies remain separate and default-deny. The owner must select the subscription manager/delegating principal, reply connection/binding and operator escalation before sandbox activation. Person/team sharing stays #8523 unless the actual scenario requires it; app credentials imply no reply authorization.

    RecoveryRequired accepts only explicit audited resolution, never automatic effect replay. A later Slack-write slice must persist call-start intent and returned timestamp/unknown outcome; this ledger does not close remote-call/local-commit uncertainty.

    Lead design gates: supported provider/key, audited creation chain, capability/withdrawal integration, required retention/cleanup contract and monotonic unknown states. Live settings are not blanket offline blockers. Revisit for automatic replay, stronger cross-store guarantees, new providers, changed fan-out or person/team delegation. Full transport ack/reconnect, additive legacy migration, live threaded reply and broader #8236/#8229 operations remain required.

    Hosted validation and evidence

    Two independent processes share real PostgreSQL; reuse deterministic barriers and kill/restart fixtures. Cover envelope-independent duplicates, reconnect/reorder/reinstall/fan-out; uncertain admission/insertion/notification/permit; death before/after capability consumption and effect-before-final-write; duplicate/readback/restart never invoking a permit; all lifecycle entry points and withdrawal races; stale fences; scope/revision conflicts; terminality, delayed authorized owner racing operator resolution/cleanup, duplicate permit requests preserving known states, timestamp/skew, cleanup/capacity/admission races; no grant/no credential leakage. Retain sanitized cases, source hashes and actual container/database/process identities; no unexplained skips. No heavy checks ran for this draft.

    Accepted evidence: #8236 comments 6038080278/6038519220; #8229 comment 6046372135; #8345 closure 5861333529; PR #8423; #8523. Exact source object eccc39fe15d2e525af866bd5a2e59f12821de381: src/modules/Elsa.Connections.Credentials.Persistence.EFCore/EFCoreConnectionLifecycleStore.cs; src/modules/Elsa.Workflows.Management/Services/WorkflowInstanceManager.cs:68; test/integration/Elsa.Connections.Credentials.Persistence.PostgreSql.IntegrationTests/{CredentialWorkerProcessTests,WorkerProcessRunner,PostgreSqlConnectionsPersistenceTests}.cs. Graph-first discovery preceded exact-object fallback. Existing tests are precedent, not admission proof.

    Full canonical execution guard and supported bootstrap contract

    Proposed minimum Core execution guard for admission-owned instances

    Offline design, 2026-10-08. Independent DESIGN APPROVE/HIGH covers the isolated-host guard and bootstrap at the hashes recorded above. No implementation acceptance, Task creation/activation or Agent Ready claim; #8655 remains the sole active Task.

    Source-confirmed boundary

    Graph-first discovery, confirmed against explicit object 12365b98791b5333fcbe9781abd0346314fc195b (earlier findings also confirmed at accepted program eccc39fe):

    • src/modules/Elsa.Workflows.Core/Services/WorkflowRunner.cs:207-230: every built-in runner overload converges before WorkflowExecuting/WorkflowStarted, then invokes the workflow pipeline.
    • src/modules/Elsa.Workflows.Core/Pipelines/WorkflowExecution/WorkflowExecutionPipeline.cs:22-37: Pipeline exposes the stored delegate; Setup returns it; ExecuteAsync invokes it.
    • src/modules/Elsa.Workflows.Core/Pipelines/WorkflowExecution/WorkflowExecutionPipelineBuilder.cs:38-45: Build independently returns the composed delegate. Guarding only ExecuteAsync or Setup misses this path.
    • src/modules/Elsa.Workflows.Core/Features/WorkflowsFeature.cs:249-253 registers the built-in pipeline; the matching ShellFeatures registration also needs coverage.
    • src/modules/Elsa.Workflows.Runtime/Services/LocalWorkflowRuntime.cs:34-38 constructs LocalWorkflowClient directly with ActivatorUtilities. Decorating an IWorkflowClient DI registration alone would not intercept this factory.
    • src/modules/Elsa.Workflows.Core/Extensions/WorkflowExecutionContextExtensions.cs:20-35,78-110 exposes root/bookmark scheduling. ScheduleBookmark returns null when its owning activity context is absent; the adapter must reject that result, never fall back to root execution.

    Additional confirmed paths: src/modules/Elsa.Workflows.Runtime/Services/WorkflowCanceler.cs:20,30 constructs/restores context and emits WorkflowCancelling before its pipeline; LocalWorkflowClient.cs:140-186 exposes Cancel/Delete/Import; ObsoleteWorkflowRuntime.cs:213-215 saves a supplied bookmark directly. src/modules/Elsa.Workflows.Api/Endpoints/WorkflowInstances/Import/Endpoint.cs:117-150 bypasses the runtime wrapper and saves imported state and bookmarks. src/modules/Elsa.Alterations/Services/DefaultAlterationRunner.cs:67-94 constructs context, executes alterations and commits/imports state. These are separate boundaries, not covered by the runner hook.

    Recommended hook and adaptation

    Add one optional Core-neutral guard contract with two operations: AuthorizeAsync(context, Runner|DirectPipeline) and an identity-only DemandUnownedAsync(instanceId, cancellationToken) preflight. The latter rejects owned operations that the first adapter does not support, before context/provider callbacks, notifications or writes. Trusted tenant/environment comes from the scoped host, not the request. This necessary new public extension contract has scoped design approval; implementation/signature changes still require review; existing signatures/constructors remain compatible through existing service-provider resolution. Hosts without admission retain behavior. Enabling admission validates the actual supported registrations; lookup uncertainty fails closed.

    Runner calls authorization before WorkflowExecuting. For an owned instance, consume a private exact-context registration bound to an immutable trusted tuple: tenant/environment/instance, admission revision, attempt and pinned definition/content identity. Also bind the prepared graph identity, complete input, scheduler work-item plan and ExecuteDelegate. Reference equality alone is insufficient: Id and graph/delegate/scheduling are mutable. Compare the prepared values at consumption and immediately before internal pipeline dispatch; mismatch denies/records uncertainty without rebinding. No copied context property, supplied status/bookmark or AsyncLocal inheritance grants permission. Contexts/graphs have one trusted owner; concurrent mutation is unsupported, and last-read validation does not claim to close that race.

    For direct pipeline entry, the provider rejects admission-owned instances even if that context has a capability or a runner is already active. Thus a notification handler or concurrent caller cannot steal a context-wide authorization stamp.

    To let the authorized runner execute normally, add an internal built-in pipeline invocation path, called only by the built-in runner after authorization. It invokes the privately held composition directly, without re-entering the public DirectPipeline guard. This is trusted Core control flow, not a public bypass or an ambient permission. An admitted runner must not fall back to an arbitrary/custom pipeline implementation.

    Build the composition once into a small internal pair: raw delegate plus guarded public delegate. Public Build returns only the guarded delegate; built-in Setup retains the pair internally and also returns only the guarded delegate; Pipeline exposes that guarded value; ExecuteAsync invokes it. All public delegates check at invocation time, including delegates cached before an instance was admitted. Reset/Insert/Replace and a later Setup cannot remove the outer guard. The raw delegate is never returned by a public API. This requires builder, pipeline and runner changes, not a general runtime rewrite. Middleware delegates may retain their own inner callbacks: they remain trusted audited host code, not a security boundary against hostile extensions.

    First supported adapter and continuation

    First support is an isolated single-execution-host LocalWorkflowRuntime with an admission-aware IWorkflowRuntime client-factory wrapper. It excludes Workflows.Api management mutation/import routes and Alterations; startup feature/endpoint inventory tests enforce their absence. No other unguarded host shares its store. Enabling the general management API would require explicit per-instance endpoint/bulk guards before writes, including every imported bookmark target ID; it is outside this first scope, not implicitly safe. Forward unowned runtime operations. Owned Run loads authoritative state/pinned graph, constructs/schedules and privately binds the exact context, then calls the built-in runner context overload. Keep #8652 ownership through final writes/unwind; no serialized capability.

    Owned Cancel/Delete/Import/UpdateBookmark/Alteration are explicitly unsupported, including direct service calls, and fail before effects; unowned behavior remains unchanged. Add DemandUnowned preflight to both WorkflowCanceler overloads (before CreateAsync or WorkflowCancelling respectively), LocalWorkflowClient mutation entrypoints before Cancel/Save/Delete, obsolete UpdateBookmark before Save, and DefaultAlterationRunner before context creation. No force/admin bypass or automatic recovery authority is implied. Later support needs a separately reviewed control-operation authority and invocation-lane protocol; already-issued initial authority is not retroactively cancellable.

    Earlier preparation is a trusted-host prerequisite: audit materialization, ActivityRegistry.RegisterAsync (Context.cs:235), descriptor-provider refresh during state restoration and notification handlers as effect-free before authorized entry; cache/metadata updates are allowed, business/provider effects are not. Early owned-ID preflight rejects standard runner preparation overloads before context restoration; the trusted adapter validates its private capability/identity before preparation. Direct public Context.CreateAsync/SetWorkflowGraphAsync can still invoke registry callbacks, so preflight alone is insufficient: unaudited or effectful providers disable this host. Queue/stack scheduling only enqueues. Tests must inject/observe these earlier callbacks and prove no business effects; post-authorization callbacks must not mutate protected identity/plan.

    The initial admission service hands its definite first-commit capability directly to this adapter. Ordinary Run requests cannot mint an initial capability. Legacy/direct-client/HTTP/direct-runner callers without the binding are denied for owned instances; ownership detection does not depend on their supplied state.

    Continuation requires more than persisted Suspended/bookmark state: the ledger retains a trusted committed checkpoint revision/full-state-and-bookmark fingerprint and its unconsumed continuation lineage, recorded only after authorized final writes/unwind definitively succeed. Imports cannot mint or advance this record; unknown checkpoint recording becomes RecoveryRequired. Compare authoritative workflow/bookmark content to that checkpoint, reject mismatches rather than adopting them, serialize the invocation lane and bind a distinct exact-context continuation authorization. Reject a replayed consumed checkpoint/bookmark, including an otherwise valid old imported snapshot. Forged Suspended state, missing bookmark/owner, arbitrary root/activity execution, outstanding owner and RecoveryRequired never qualify. Record continuation uncertainty conservatively; this lane still needs implementation review and does not promise exactly-once effects.

    Ownership protection outlives dedup payload/key cleanup while the instance remains startable. First support requires every host accessing this store to enforce the guard. Remote/Proto.Actor and distributed-wrapper support require separate review; multi-process admission-store proof does not establish clustered workflow execution. Arbitrary custom executors, captured inner middleware delegates, direct activity invocation, replacement service providers or unguarded hosts are outside the supported trusted-host contract.

    Supported offline bootstrap before activation

    The same guarded host has an explicit deployment-controlled bootstrap/configuration path for an allowlisted, predefined workflow artifact and trusted subscription. Offline fixtures provide declared synthetic values; live values require their separate owner approval. Bootstrap runs with admission inactive, after guard/provider registration and isolated-host validation. It never acknowledges events or starts a workflow.

    Validate the artifact's reviewed content fingerprint, fixed definition/version identity and supported activity/provider chain. Insert the definition only when absent; an existing definition must match the pinned identity/content exactly or activation fails. Complete the audited definition publication/indexing chain without execution; disallow autonomous triggers/startup execution in this first artifact. Provision an inactive ledger subscription from trusted host configuration: stable subscription/installation identity, tenant/environment, app/workspace/channel, pinned published workflow identity/fingerprint and explicit admission/retention settings. Payloads cannot provision or select these records. Validate every binding and the pinned stored definition before explicitly activating that ledger revision.

    Bootstrap cannot import workflow instances/bookmarks, mint continuation provenance, overwrite owned records, reset start attempts or erase ownership/dedup protection. Mismatches fail closed, not upsert/adopt. Reconfiguration requires withdrawal and the existing quiescence/reconciliation rules; it preserves stable identity and cannot replay history. No second unguarded host shares the stores. This path grants no credentials or outbound use authority; those policies and live provisioning remain separate.

    Add offline tests for successful declared bootstrap/activation, repeat verification without mutation, identity/content mismatch, premature activation, forbidden instance/bookmark import and owned-record overwrite/reset. Generic management and distributed adapters remain unsupported unless a concrete use case justifies separately reviewed support. Live operational acceptance remains required.

    Required proof before readiness/acceptance

    1. Hold the initial capability: attempt competing client/dispatcher/legacy host/HTTP/direct-runner starts. Require zero WorkflowExecuting/WorkflowStarted notifications, middleware entry and activity effects; the original authorized attempt still succeeds once.
    2. Invoke ExecuteAsync, Pipeline, Setup's returned delegate and a separately built Build delegate. Repeat with cached delegates, Reset/Replace/re-Setup, a capability-bound context and a call made from WorkflowExecuting. Owned instances must be denied before any middleware. Unowned delegates must preserve behavior.
    3. Concurrent and sequential reuse of the exact context/capability, and distinct contexts with the same instance ID, cannot obtain a second initial entry. No copied TransientProperties/Properties/status or forked execution context confers authority.
    4. Real suspension/final writes/unwind records a definite trusted checkpoint, then legitimate bookmark continuation succeeds. Forge Suspended plus bookmarks through runtime import, HTTP import, obsolete bookmark update and direct fixture persistence; no new/replayed continuation authority results. Enabled mutation services deny owned operations before notification/write; excluded management routes/Alterations remain unavailable. Unowned actions still work.
    5. Mutate the bound Id, graph/definition, scheduler plan, complete input or ExecuteDelegate before consumption/pre-pipeline dispatch; reject without rebinding. Exercise earlier registry/materialization/descriptor callbacks and reject unsupported effectful host chains. Unknown permit/checkpoint outcomes, restart/readback, ledger failure, ownership cleanup and lifecycle withdrawal remain conservative; test classic and ShellFeatures wiring.

    The isolated-host guard architecture and bootstrap have scoped independent design approval. The lead still owns scheduling and the combined leaf's readiness; neither implementation nor proof is complete. No heavy execution was performed; these are required future proofs, not passing results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    elsa 3This issue is specific to Elsa 3enhancementNew feature or requestprio highIs on the roadmap for the near-futuretriaged

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions