Deferred: compatibility checks for terminal preservation recovery

.github/workflows/terraform.yml

@@ -73,3 +73,6 @@ jobs:
 
       - name: Yjs configuration tests (mock providers)
         run: terraform test -filter=tests/yjs.tftest.hcl
+
+      - name: Recovery compatibility deployment tests (mock providers)
+        run: terraform test -filter=tests/recovery-compatibility.tftest.hcl

docs/development/recovery-ownership.md

@@ -74,8 +74,110 @@ repair its stored success.
 
 Recovery launches a new durable history through the configured orchestrator
 target. The original methodology, credential binding, repositories, model
-choices and runtime ARN/endpoint remain pinned. Before approving production
-recovery, verify that this launch target and the captured runtime endpoint serve
-the fixed release. Do not change execution pins or patch DynamoDB rows to force
-the new behavior. Starting the recovery and deploying the release are separate
-operator actions.
+choices and runtime ARN/endpoint remain pinned. Starting recovery and deploying
+the release remain separate operator actions.
+
+### Automatic preservation recovery checks
+
+The incomplete terminal section preview now checks **both** execution targets
+before it permits confirmation. Its required capabilities are
+`preserve-integrated-section@1` and `checked-runtime-dispatch@1`, using capability
+schema 1. This check applies to the terminal preservation recovery introduced
+above. Existing running-lane repair, ordinary rewind (which resets affected
+integrated lanes), gate resume, and historical reads keep their existing
+contracts. There is no alternative request field that skips the check: the
+server derives it from authoritative execution, unit-plan and unit state.
+
+The preview reports one of three states:
+
+- **Recovery is available:** both targets support the required behavior.
+- **A deployment update is needed:** the published orchestrator package is not
+  attested by this API deployment, or the original runtime lacks the capability.
+- **Recovery cannot currently be verified:** configuration is missing, a metadata
+  read is denied or unavailable, an endpoint is updating, or a target changes
+  while the check is running. Refresh after the service or deployment settles;
+  an administrator must fix missing read permissions.
+
+The orchestrator observation resolves the configured qualifier with `GetAlias`
+(numeric qualifiers need no alias read), refuses weighted aliases, and reads
+the published version with `GetFunctionConfiguration`. Terraform generates
+`RECOVERY_ORCHESTRATOR_RELEASE` from the actual published **qualified ARN and
+deployment package SHA-256**, plus `lambda/shared/recovery-capabilities.json`.
+Both identity and package hash must match the service observation. A feature
+flag, release label, creation time, or the name `live` cannot certify a target.
+Only the release attested by the current API deployment is admitted; pointing
+an alias at another version requires its matching API deployment as well.
+
+The runtime observation uses the original snapshot through `resolveRuntimeTarget`,
+including the established fallback for legacy records without a snapshot.
+`GetAgentRuntimeEndpoint` resolves the captured endpoint (or `DEFAULT`) to a
+ready live version. Before and after a `recovery-capabilities` invocation it
+compares the endpoint version and update identity. The command reports the
+capability contract and SHA-256 of executable source files in the container's
+protected `agentcore/` and `shared/` directories. It does not read credentials,
+account usage, discover models, open a CLI/model session, access the intent
+workspace, or dispatch business handlers. A fresh metadata-only AgentCore
+session avoids an old warm business session; it expires under normal runtime
+idle policy. This can start a runtime microVM, so it is not a zero-cost HTTP
+health check. No session-stop operation is issued by preview.
+
+Observations are included in the preview token. Confirmation repeats the checks,
+compares the token, and repeats them once more immediately before atomic run
+retirement. Reads have a bounded request deadline and there is no compatibility
+cache. No token retirement, unit/stage reset, artifact archival or execution
+launch occurs on a failed check. Ownership conditions still admit at most one
+concurrent confirmation.
+
+After confirmation, the new durable invocation uses the **observed numeric
+Lambda version**, even if the alias subsequently moves. Its existing `load-meta`
+checkpoint captures the runtime guard; no new durable operation is inserted into
+old replay histories. Every runtime command in this recovery is enclosed in
+`recovery-dispatch`, which checks the observed source digest and capabilities
+before credentials, accounting or business work. A later runtime rollout with
+different source is refused rather than silently continuing recovery on it;
+old containers reject the unknown envelope. Refresh and preview the resulting
+failed recovery before retrying. This preserves the original endpoint and
+does not create, update or repoint runtime endpoints.
+
+An endpoint update can still happen after the last metadata read. A later
+compatible container with identical executable source can serve that endpoint;
+an incompatible/different source fails the guarded dispatch. The guard is a
+code compatibility check, not a mechanism for making mutable AWS endpoints
+immutable. Infrastructure administrators must continue to honor published
+environment revision immutability.
+
+### Mixed-version rollout and verification
+
+1. Deploy the orchestrator package, intents API/IAM, runtime image and frontend.
+   Terraform propagates the package identity to the API; changing shared
+   capability code participates in both Lambda and runtime packaging hashes.
+   Intents needs `lambda:GetAlias`, `lambda:GetFunctionConfiguration`, and
+   `bedrock-agentcore:GetAgentRuntimeEndpoint` on its existing target scopes.
+   No model, credential-read, or endpoint mutation permission is added.
+2. During rollout, a new API refuses unattested orchestrator packages and old
+   runtimes. A new frontend refuses a terminal preview from an old API that
+   omits compatibility. An old frontend talking to the new API cannot bypass
+   server checks. An old API remains unguarded until it is replaced: do not
+   enable terminal recovery while old API workers/deployments still serve it.
+   Existing histories and ordinary supported retries are not migrated.
+3. For a managed environment pinned to an older immutable revision, rebuilding
+   the project's current environment does **not** change the old intent's pin.
+   Review whether its original target can legitimately receive the update.
+   Otherwise retain that history and create a follow-up intent on a supported
+   environment. Never patch execution pins or repoint an immutable revision to
+   make a blocked check pass.
+4. After deployment, an owner/admin opens **Preview recovery** on an eligible
+   test intent. Check both supported observations in the expandable deployment
+   details. Refresh once and verify that no execution records, artifacts or
+   model work changed. No confirmation is needed for this check.
+5. In a disposable test deployment, verify missing read permission and an older
+   captured endpoint produce a blocked preview; restore the deployment. Move a
+   launch alias between preview and confirmation and expect 409 without record
+   mutation. Restore it, refresh, then deliberately confirm only when a real
+   recovery is intended. Confirm numeric-version launch, original environment
+   targeting, preserved integrated units, and normal human approvals.
+
+Deterministic tests cover these decisions with sanitized service responses and
+local stores. They do not certify a currently deployed AWS package, IAM policy,
+endpoint, or model. Deployment and the above post-deployment checks are required
+before making that claim.

frontend/src/pages/IntentView.test.tsx

@@ -1024,6 +1024,13 @@ describe('IntentView', () => {
       preview: {
         previewToken: 'current-preview',
         kind: 'incomplete-section',
+        compatibility: {
+          schemaVersion: 1,
+          status: 'supported',
+          requiredCapabilities: ['preserve-integrated-section@1'],
+          orchestrator: { status: 'supported' },
+          runtime: { status: 'supported' },
+        },
         sectionIndex: 1,
         fromStageId: 'functional-design',
         laneSlugs: ['notification-core'],
@@ -1057,6 +1064,85 @@ describe('IntentView', () => {
     });
   });
 
+  it.each([
+    ['update-required', 'A deployment update is needed'],
+    ['unverified', 'Recovery cannot currently be verified'],
+    ['missing', 'Recovery cannot currently be verified'],
+  ])(
+    'explains %s compatibility and allows a safe refresh before confirming',
+    async (status, heading) => {
+      get.mockResolvedValue({
+        ...baseDetail({ status: 'SUCCEEDED' }),
+        unitCompletion: {
+          state: 'incomplete',
+          total: 2,
+          integrated: 1,
+          skipped: 0,
+          legacyPartial: false,
+          unresolved: [{ sectionIndex: 1, slug: 'remaining', state: 'BLOCKED' }],
+        },
+      });
+      const preview = {
+        previewToken: 'blocked-preview',
+        kind: 'incomplete-section',
+        sectionIndex: 1,
+        fromStageId: 'code',
+        laneSlugs: ['remaining'],
+        preservedSlugs: ['foundation'],
+        rerunSharedStages: [],
+        skeletonProblem: null,
+        skeletonCandidates: [],
+        walkingSkeleton: 'foundation',
+        branch: 'aidlc/feature',
+        approvals: 'Normal approvals apply.',
+        pullRequests: [],
+      };
+      previewRepair.mockResolvedValueOnce({
+        preview: {
+          ...preview,
+          ...(status === 'missing'
+            ? {}
+            : {
+                compatibility: {
+                  schemaVersion: 1,
+                  status,
+                  requiredCapabilities: [],
+                  orchestrator: { status: 'supported' },
+                  runtime: {
+                    status,
+                    message: 'Ask an administrator to update the original environment.',
+                  },
+                },
+              }),
+        },
+      });
+      renderAt();
+      await userEvent.click(await screen.findByRole('button', { name: 'Preview recovery' }));
+      expect(await screen.findByText(heading)).toBeInTheDocument();
+      expect(screen.getByRole('button', { name: 'Confirm and continue' })).toBeDisabled();
+      expect(repair).not.toHaveBeenCalled();
+      previewRepair.mockResolvedValueOnce({
+        preview: {
+          ...preview,
+          previewToken: 'refreshed-preview',
+          compatibility: {
+            schemaVersion: 1,
+            status: 'supported',
+            requiredCapabilities: [],
+            orchestrator: { status: 'supported' },
+            runtime: { status: 'supported' },
+          },
+        },
+      });
+      await userEvent.click(screen.getByRole('button', { name: 'Refresh recovery preview' }));
+      expect(await screen.findByText('Recovery is available')).toBeInTheDocument();
+      expect(screen.getByRole('button', { name: 'Confirm and continue' })).toBeEnabled();
+      repair.mockResolvedValue({});
+      await userEvent.click(screen.getByRole('button', { name: 'Confirm and continue' }));
+      expect(repair).toHaveBeenCalledWith('p1', 'i1', { previewToken: 'refreshed-preview' });
+    },
+  );
+
   it('identifies explicitly accepted partial delivery without offering to undo it', async () => {
     get.mockResolvedValue({
       ...baseDetail({ status: 'SUCCEEDED' }),

frontend/src/pages/IntentView.tsx

@@ -292,7 +292,12 @@ export default function IntentView() {
   };
 
   const handleRepair = async () => {
-    if (!repairPreview) return;
+    if (
+      !repairPreview ||
+      (repairPreview.kind === 'incomplete-section' &&
+        repairPreview.compatibility?.status !== 'supported')
+    )
+      return;
     setRepairing(true);
     setActionError(null);
     try {
@@ -963,6 +968,59 @@ export default function IntentView() {
           </AlertDialogHeader>
           {repairPreview && (
             <div className="max-h-[50vh] space-y-3 overflow-y-auto text-sm">
+              {repairPreview.kind === 'incomplete-section' && (
+                <div className="space-y-2 rounded border p-3" role="status">
+                  <p className="font-medium">
+                    {repairPreview.compatibility?.status === 'supported'
+                      ? 'Recovery is available'
+                      : repairPreview.compatibility?.status === 'update-required'
+                        ? 'A deployment update is needed'
+                        : 'Recovery cannot currently be verified'}
+                  </p>
+                  {repairPreview.compatibility?.status === 'supported' ? (
+                    <p>
+                      Both the recovery service and this intent’s original environment support
+                      preserving integrated work. We check again when you confirm.
+                    </p>
+                  ) : (
+                    <p>
+                      Recovery has made no changes.{' '}
+                      {[
+                        repairPreview.compatibility?.orchestrator.message,
+                        repairPreview.compatibility?.runtime.message,
+                      ]
+                        .filter(Boolean)
+                        .join(' ') ||
+                        'Ask an administrator to deploy the recovery checks, then refresh this preview.'}
+                    </p>
+                  )}
+                  <Button
+                    variant="outline"
+                    disabled={repairing}
+                    onClick={() => void previewRepair()}
+                  >
+                    Refresh recovery preview
+                  </Button>
+                  {repairPreview.compatibility && (
+                    <details>
+                      <summary>Deployment details for administrators</summary>
+                      <dl className="mt-2 break-all">
+                        <dt>Recovery service</dt>
+                        <dd>
+                          {repairPreview.compatibility.orchestrator.functionArn || 'Not verified'}
+                        </dd>
+                        <dt>Original environment endpoint</dt>
+                        <dd>
+                          {repairPreview.compatibility.runtime.agentRuntimeArn || 'Not verified'}{' '}
+                          {repairPreview.compatibility.runtime.qualifier}{' '}
+                          {repairPreview.compatibility.runtime.version &&
+                            `(version ${repairPreview.compatibility.runtime.version})`}
+                        </dd>
+                      </dl>
+                    </details>
+                  )}
+                </div>
+              )}
               <p>
                 <strong>Preserved units:</strong>{' '}
                 {repairPreview.preservedSlugs.join(', ') || 'None integrated yet'}.
@@ -1008,6 +1066,8 @@ export default function IntentView() {
               disabled={
                 repairing ||
                 !repairPreview ||
+                (repairPreview.kind === 'incomplete-section' &&
+                  repairPreview.compatibility?.status !== 'supported') ||
                 (Boolean(repairPreview?.skeletonProblem) && !repairSkeleton)
               }
               onClick={(event) => {

frontend/src/services/intents.ts

@@ -824,6 +824,25 @@ export interface IntentRepairPreview {
   branch: string;
   approvals: string;
   pullRequests: { repository: string; number: string | number; url: string }[];
+  compatibility?: {
+    schemaVersion: number;
+    status: 'supported' | 'update-required' | 'unverified';
+    requiredCapabilities: string[];
+    orchestrator: {
+      status: string;
+      message?: string;
+      functionArn?: string;
+      codeSha256?: string;
+    };
+    runtime: {
+      status: string;
+      message?: string;
+      agentRuntimeArn?: string;
+      qualifier?: string;
+      version?: string;
+      sourceSha256?: string;
+    };
+  };
 }
 
 export interface IntentRepairResult {

lambda/agentcore/http-server.js

@@ -79,6 +79,7 @@ import { commandDefinition } from './command-registry.js';
 import { installProcessGroupShutdown } from './cli/spawn.js';
 import { createCredentialSession } from './credential-session.js';
 import { accountCredentialInvocation } from './invocation-accounting.js';
+import { deployedRecoveryCapabilities, recoveryDispatchPayload } from './recovery-capabilities.js';
 
 const logger = new Logger({ persistentKeys: { component: 'agentcore' } });
 
@@ -114,6 +115,24 @@ export const dispatchInvocation = async ({
   now = () => new Date().toISOString(),
 }) => {
   const command = payload?.command;
+  // Both commands are handled before auth, accounting, busy state or business
+  // handlers. An old image rejects the envelope as an unknown command.
+  if (command === 'recovery-capabilities') {
+    return { statusCode: 200, body: { ok: true, ...deployedRecoveryCapabilities() } };
+  }
+  if (command === 'recovery-dispatch') {
+    const checked = recoveryDispatchPayload(payload);
+    if (!checked) {
+      return {
+        statusCode: 409,
+        body: {
+          error: 'Recovery environment changed. Refresh the recovery preview before retrying.',
+          code: 'recovery_runtime_changed',
+        },
+      };
+    }
+    return dispatchInvocation({ payload: checked, handlers, busy, prepareInvocation, now });
+  }
   if (!command) {
     logger.warn('dispatch rejected: missing command');
     return { statusCode: 400, body: { error: 'missing "command"' } };