diff --git a/docs/concepts/execution.md b/docs/concepts/execution.md index 381f9babb..91af53121 100644 --- a/docs/concepts/execution.md +++ b/docs/concepts/execution.md @@ -69,6 +69,24 @@ Git is deterministic and owned by the engine — **the agent never runs git and For PR-per-unit delivery, DynamoDB is scheduling truth. A unit is complete only after every changed repository confirms the exact ready head is on the intent branch. A partial multi-repository merge preserves merged work, blocks dependents, and enters halt-and-ask. +The lane board calls this outcome **Integrated**. Its stored identifier remains +`MERGED`; it does not mean that the final intent-to-base PR has merged. Counts +use the active unit plan; retained rows from earlier plans remain history. + +A walking skeleton must be a dependency-free unit in the current DAG. A missing +choice receives a dependency-free default before approval. Re-promotion retains +valid captured choices. If the regenerated DAG renames that unit or gives it +dependencies, promotion records the change and offers a new dependency-free +default at the normal fan-out approval. Original answers remain in history. +Execution still rejects invalid choices in an already approved plan. A workflow with `skeleton: off` +still runs all units, without the solo-first ceremony. + +Shared stages and successful completion require every active-plan lane to be +integrated or explicitly omitted by a recorded human skip. Blocked-only work +still enters halt-and-ask. Retry preserves integration; skip also omits the +unintegrated dependent closure; abort fails the run. Partial delivery is presented +separately from complete delivery. + Provider comments never launch agents by themselves. A project member selects comments in the intent's review drawer; the API refetches those comments with fresh credentials, rejects bot/system comments, and queues a bounded, audited feedback batch. The lane revisits its last successful stage, runs its normal sensors, commits and pushes the revision, and posts a summary without resolving provider threads. ## Human gates, park and resume diff --git a/docs/development/recovery-ownership.md b/docs/development/recovery-ownership.md index f7e672899..76f45c6eb 100644 --- a/docs/development/recovery-ownership.md +++ b/docs/development/recovery-ownership.md @@ -31,3 +31,43 @@ Storage must accept a write to persist FAILED state. During a complete outage th worker preserves its structured failure, while durable dispatch and callback reconciliation remain the recovery backstop. These guards protect process state; they do not undo provider requests or Git operations already in flight. + +## Completion boundaries + +New histories capture completion policy 1 in the existing `load-meta` result. +The shared unit-completion evaluator validates the pinned, resolved workflow's +expected sections, current-plan units, repository integration evidence and scoped +human omissions. Section fan-in and final success use the same evaluator against +persisted process state. Missing rows, unfinished work and partial integrations +cannot pass; missing provenance, read failures and ambiguous legacy omissions +remain unverified. Serial workflows and sections excluded by approved workflow +scope/composition remain valid. Pre-deploy `MERGED` rows without an +`integrationResults` attribute retain their supported integrated meaning, including +multi-repository retries and rewinds past completed sections. New integrations +always write the attribute; explicit empty, invalid or incomplete evidence never +receives this legacy exemption. PR-per-unit integration still requires its +per-repository PR records. Unpinned historical serial intents without unit work +keep their success presentation without unit-completion diagnostics. Cancelled +executions and recorded aborts from the current run are classified as aborted +before workflow-pin resolution, without implying that their work was integrated. +Missing or truncated execution reads remain unverified. + +Required integration writes are not best-effort. A lost response is reconciled +from persisted state; an unrecorded integration stops the lane before dependent +work can proceed. Final status and its completion receipt commit together under +run, status, META revision and unit-plan conditions. Initial publication rechecks +completion before opening a PR. These checks never rerun models or repair rows. +Marker-absent histories (policy 0) retain their saved operation sequence and +terminal step. The intermediate completion implementation was not released and +has no separate replay path. Saved historical successes are never rewritten. + +A terminal success with unfinished or unverified work is shown honestly in the +intent detail. Accepted omissions remain partial delivery, not execution defects. +This release does not add automatic preservation-and-continuation recovery for +finished incomplete intents. Existing orphan-lane/PR-review repair and ordinary +rewind remain supported; rewind resets affected lanes, including integrated work, +and clears omission receipts for the work explicitly being retried. Original +answers and events remain historical. Preserve runtime and methodology pins. + +Deploy the runtime, orchestrator, intents API and frontend from the same release. +No automatic production audit, status migration or runtime repinning is performed. diff --git a/frontend/src/components/intent/UnitLaneBoard.test.tsx b/frontend/src/components/intent/UnitLaneBoard.test.tsx index 9eb125d6e..88a0232ef 100644 --- a/frontend/src/components/intent/UnitLaneBoard.test.tsx +++ b/frontend/src/components/intent/UnitLaneBoard.test.tsx @@ -95,7 +95,7 @@ describe('UnitLaneBoardView', () => { // (>=1) rather than uniqueness; state labels are unique per fixture. expect(screen.getAllByText('auth').length).toBeGreaterThanOrEqual(1); expect(screen.getByText('reporting')).toBeInTheDocument(); - expect(screen.getByText('Merged')).toBeInTheDocument(); + expect(screen.getByText('Integrated')).toBeInTheDocument(); expect(screen.getByText('Building')).toBeInTheDocument(); expect(screen.getByText('Blocked')).toBeInTheDocument(); expect(screen.getByText('Pending')).toBeInTheDocument(); @@ -115,7 +115,7 @@ describe('UnitLaneBoardView', () => { it('summarizes only the states that are present', () => { renderView(); - expect(screen.getByText('1 merged')).toBeInTheDocument(); + expect(screen.getByText('1 integrated')).toBeInTheDocument(); expect(screen.getByText('1 building')).toBeInTheDocument(); expect(screen.getByText('1 blocked')).toBeInTheDocument(); expect(screen.getByText('1 pending')).toBeInTheDocument(); @@ -137,7 +137,7 @@ describe('UnitLaneBoardView', () => { expect(screen.queryByText(/failed/)).not.toBeInTheDocument(); expect(screen.queryByText('legacy')).not.toBeInTheDocument(); // real units still counted - expect(screen.getByText('1 merged')).toBeInTheDocument(); + expect(screen.getByText('1 integrated')).toBeInTheDocument(); }); it('renders the live stream only when a running stage instance is provided', () => { diff --git a/frontend/src/components/intent/UnitLaneBoard.tsx b/frontend/src/components/intent/UnitLaneBoard.tsx index 6ba6c59f3..cb995206a 100644 --- a/frontend/src/components/intent/UnitLaneBoard.tsx +++ b/frontend/src/components/intent/UnitLaneBoard.tsx @@ -56,7 +56,7 @@ const STATE_LABEL: Record = { PR_READY: 'Ready to merge', ADDRESSING_FEEDBACK: 'Fixing feedback', MERGING: 'Merging', - MERGED: 'Merged', + MERGED: 'Integrated', BLOCKED: 'Blocked', FAILED: 'Failed', WAITING_INPUT: 'Waiting for input', @@ -402,6 +402,7 @@ function UnitCard({ // ── Presentational board (pure — tested in isolation) ──────────────────────── export interface UnitLaneBoardViewProps { + intentBranch?: string | null; unitPlan: IntentUnitPlan; units: IntentUnit[]; /** Live transcript tail per unit slug (only running units need an entry). */ @@ -418,6 +419,7 @@ export interface UnitLaneBoardViewProps { } export function UnitLaneBoardView({ + intentBranch, unitPlan, units, streamsBySlug, @@ -449,10 +451,8 @@ export function UnitLaneBoardView({ // lambda/shared/v2-process-store.js:1000), so counting over all detail.units // would report states for units that aren't on the board. const planSlugs = useMemo(() => { - const fromBatches = (unitPlan.batches ?? []).flat(); - const slugs = fromBatches.length > 0 ? fromBatches : (unitPlan.units ?? []).map((u) => u.slug); - return new Set(slugs); - }, [unitPlan.batches, unitPlan.units]); + return new Set((unitPlan.units ?? []).map((u) => u.slug)); + }, [unitPlan.units]); // State-count chips — only states actually present among plan units, in a // stable order. @@ -480,6 +480,16 @@ export function UnitLaneBoardView({ return ( +

+ Integrated units are on the intent branch + {intentBranch ? ( + <> + {' '} + {intentBranch} + + ) : null} + . The final pull request is reviewed and merged separately. +

Units @@ -708,6 +718,7 @@ export function UnitLaneBoard({ return ( <> { ).not.toBeInTheDocument(); }); }); + +describe('terminal completion diagnostics without terminal recovery', () => { + it.each(['incomplete', 'unknown', 'partial'] as const)( + 'shows %s evidence without offering preservation recovery', + async (state) => { + get.mockResolvedValue({ + ...baseDetail({ status: 'SUCCEEDED' }), + unitCompletion: { + state, + total: 2, + integrated: 1, + skipped: state === 'partial' ? 1 : 0, + unresolved: + state === 'partial' + ? [] + : [ + { + sectionIndex: 1, + slug: 'remaining', + state: 'BLOCKED', + reason: 'unit_unfinished', + }, + ], + legacyPartial: false, + }, + }); + renderAt(); + expect( + await screen.findByText( + state === 'incomplete' + ? 'Incomplete work' + : state === 'unknown' + ? 'Completion unverified' + : 'Partial delivery', + ), + ).toBeInTheDocument(); + expect( + screen.queryByRole('button', { name: /Preview recovery|Confirm and continue/ }), + ).not.toBeInTheDocument(); + expect(repair).not.toHaveBeenCalled(); + }, + ); +}); diff --git a/frontend/src/pages/IntentView.tsx b/frontend/src/pages/IntentView.tsx index 14a57ecbe..4ff6cb4f1 100644 --- a/frontend/src/pages/IntentView.tsx +++ b/frontend/src/pages/IntentView.tsx @@ -342,6 +342,8 @@ export default function IntentView() { } const intent = detail.intent; + const completion = detail.unitCompletion; + const terminalCompletion = intent.status === 'SUCCEEDED' ? completion?.state : null; const laneWaits = deriveLaneWaits(detail.stages, gates); const recoveryWaits = Object.values(laneWaits).filter((wait) => wait.kind === 'recovery'); const needsLaneRepair = @@ -417,7 +419,13 @@ export default function IntentView() { {TERMINAL_STATUSES.has(intent.status) && ( - {intent.status} + {terminalCompletion === 'incomplete' + ? 'Incomplete work' + : terminalCompletion === 'unknown' + ? 'Completion unverified' + : terminalCompletion === 'partial' + ? 'Partial delivery' + : intent.status} )} @@ -588,6 +596,33 @@ export default function IntentView() {
)} + {terminalCompletion && terminalCompletion !== 'complete' && ( +
+

+ {terminalCompletion === 'partial' + ? 'Partial delivery was explicitly accepted. Integrated work and the recorded omissions are preserved.' + : terminalCompletion === 'unknown' + ? 'Completion could not be verified. Review the recorded work and available evidence before publishing.' + : 'The recorded success has unfinished current-plan work. Review the remaining units before publishing.'} +

+

+ {completion?.integrated} of {completion?.total} units verified integrated;{' '} + {completion?.skipped} explicitly omitted. +

+ {completion?.unresolved.slice(0, 12).map((unit) => ( +

+ Section {unit.sectionIndex}/{unit.slug}: {unit.reason ?? unit.state} +

+ ))} + {terminalCompletion !== 'partial' && ( +

+ Automatic continuation of finished incomplete intents is not supported. Ordinary + rewind reruns the affected work. +

+ )} +
+ )} + {needsLaneRepair && (
diff --git a/frontend/src/services/intents.ts b/frontend/src/services/intents.ts index fc6966b45..bf14e8fb9 100644 --- a/frontend/src/services/intents.ts +++ b/frontend/src/services/intents.ts @@ -867,6 +867,20 @@ export interface IntentDetail { /** Composer sessions (scope/grid proposals + authoritative validation). */ composes?: ComposeSession[]; unitPlan?: IntentUnitPlan | null; + unitCompletion?: { + state: 'complete' | 'partial' | 'incomplete' | 'unknown' | 'aborted'; + reason?: string | null; + total: number; + integrated: number; + skipped: number; + unresolved: { + sectionIndex: number; + slug: string; + state: UnitState | 'UNKNOWN' | 'MISSING'; + reason?: string; + }[]; + legacyPartial: boolean; + } | null; units?: IntentUnit[]; unitPrs?: IntentUnitPr[]; feedbackBatches?: IntentFeedbackBatch[]; diff --git a/lambda/agentcore/commands/promote-units.js b/lambda/agentcore/commands/promote-units.js index 698dbc0a9..5bf4730c9 100644 --- a/lambda/agentcore/commands/promote-units.js +++ b/lambda/agentcore/commands/promote-units.js @@ -34,6 +34,7 @@ import { createGraphWriter, closeGraphSource } from '../mcp/graph-writer.js'; import { parseBoltDag } from '../../shared/v2-sensor-contract.js'; +import { effectiveSkeleton, skeletonProblem } from '../../shared/unit-completion.js'; const DAG_ARTIFACT_TYPE = 'unit-of-work-dependency'; @@ -122,20 +123,39 @@ export const promoteUnits = async (payload, deps) => { // batch (batches are sorted). The fan-out approval can override. // Consistent: a stale read would drop decisions the fan-out approval saved. const existingPlan = await store.getUnitPlan(executionId, { consistentRead: true }); + const priorSkeleton = existingPlan?.walkingSkeleton; + const priorProblem = + priorSkeleton == null ? null : skeletonProblem({ units, batches }, priorSkeleton); + const walkingSkeleton = priorProblem + ? effectiveSkeleton({ units, batches }) + : effectiveSkeleton({ units, batches, walkingSkeleton: priorSkeleton }); const plan = await store.putUnitPlan({ executionId, units, batches, sourceArtifactId: artifact.id ?? null, producedByStageInstanceId: stageInstanceId, - // Preserve previously captured human decisions across a re-promotion — - // a rewind re-produces the DAG, not the humans' skip/skeleton/autonomy - // answers (the unit-DAG stage's gate re-confirms them on re-approval). + // Keep valid choices across re-promotion. A regenerated DAG can rename + // or add dependencies to the old skeleton; choose a valid default BEFORE + // the normal fan-out approval, preserving the original answer in history. skipMatrix: existingPlan?.skipMatrix ?? {}, - walkingSkeleton: existingPlan?.walkingSkeleton ?? batches[0]?.[0] ?? null, + walkingSkeleton, autonomyMode: existingPlan?.autonomyMode ?? null, }); - const sync = await store.syncUnitRows({ executionId, units, batches, sectionIndexes }); + const sync = await store.syncUnitRows({ + executionId, + units, + batches, + sectionIndexes, + }); + if (priorProblem) + await event( + 'v2.units.skeleton_reselected', + `${priorProblem} Regenerated DAG default: ${walkingSkeleton}; review it at fan-out approval.`, + ); + // Execution still rejects invalid choices in an already approved plan. + const selectionProblem = skeletonProblem(plan); + if (selectionProblem) await event('v2.units.decisions_invalid', selectionProblem); // 4. Traceability mirror (never blocks promotion outcome — the DDB truth // is already written; a mirror failure is recorded and visible). @@ -164,6 +184,7 @@ export const promoteUnits = async (payload, deps) => { unitCount: units.length, batchCount: batches.length, walkingSkeleton: plan.walkingSkeleton, + selectionProblem, sync, mirror, }; diff --git a/lambda/agentcore/test/completion-store.test.js b/lambda/agentcore/test/completion-store.test.js new file mode 100644 index 000000000..b7f6e0705 --- /dev/null +++ b/lambda/agentcore/test/completion-store.test.js @@ -0,0 +1,184 @@ +import { randomUUID } from 'node:crypto'; +import { afterAll, beforeAll, expect, it } from 'vitest'; +import { TransactWriteCommand } from '@aws-sdk/lib-dynamodb'; +import { createProcessStore } from '../../shared/v2-process-store.js'; +import { finalizeUnitCompletion, unitPlanIdentity } from '../../shared/unit-completion.js'; +import { createV2Table, deleteV2Table, makeDdb } from './helpers/v2-table.js'; + +const tableName = `completion-${randomUUID()}`; +let db; +let store; +beforeAll(async () => { + db = makeDdb(); + await createV2Table(db.client, tableName); + store = createProcessStore({ ddb: db.doc, tableName }); +}); +afterAll(async () => { + await deleteV2Table(db.client, tableName); + db.client.destroy(); +}); +const plan = { stages: [{ stageId: 'build', parallelSection: 1 }] }; +const setup = async ({ + state = 'MERGED', + projectId = randomUUID(), + repos = [], + prStrategy = 'intent-pr', +} = {}) => { + const executionId = randomUUID(); + await store.createExecution({ + executionId, + intentId: executionId, + projectId, + startedAt: '2026-10-01T00:00:00.000Z', + workflowId: 'workflow', + workflowVersion: 1, + repos, + prStrategy, + methodologyPins: { STAGE: { build: { tenantId: 'SYSTEM', version: 1 } } }, + }); + await store.updateExecution({ executionId, status: 'RUNNING', orchestratorRunId: 'run' }); + await store.putUnitPlan({ executionId, units: [{ slug: 'a', dependsOn: [] }], batches: [['a']] }); + await store.syncUnitRows({ + executionId, + units: [{ slug: 'a', dependsOn: [] }], + sectionIndexes: [1], + }); + await store.updateUnitState({ executionId, sectionIndex: 1, slug: 'a', state }); + return executionId; +}; + +it('commits success and returns the same decision when a durable response is lost', async () => { + const executionId = await setup(); + const args = { store, executionId, runId: 'run', plan }; + const result = await finalizeUnitCompletion(args); + expect(result).toMatchObject({ state: 'complete', integrated: 1 }); + const first = await store.getExecution(executionId); + expect(first.status).toBe('SUCCEEDED'); + expect(await finalizeUnitCompletion(args)).toEqual(result); + expect((await store.getExecution(executionId)).updatedAt).toBe(first.updatedAt); +}); + +it('checks the persisted unitSlug repository rows before accepting an integrated unit', async () => { + const executionId = await setup({ repos: ['one', 'two'], prStrategy: 'pr-per-unit' }); + for (const repository of ['one', 'two']) { + await store.createUnitPr({ + executionId, + sectionIndex: 1, + slug: 'a', + repository, + provider: 'github', + sourceBranch: 'unit', + targetBranch: 'intent', + state: repository === 'one' ? 'MERGED' : 'PARTIALLY_MERGED', + }); + } + expect(await finalizeUnitCompletion({ store, executionId, runId: 'run', plan })).toMatchObject({ + state: 'incomplete', + integrated: 0, + unresolvedCount: 1, + }); + expect((await store.getUnit(executionId, 1, 'a')).state).toBe('MERGED'); +}); + +it('permits serial execution without inventing a unit plan', async () => { + const executionId = randomUUID(); + await store.createExecution({ executionId, intentId: executionId, projectId: 'serial' }); + await store.updateExecution({ executionId, status: 'RUNNING', orchestratorRunId: 'run' }); + expect( + await finalizeUnitCompletion({ store, executionId, runId: 'run', plan: { stages: [] } }), + ).toMatchObject({ state: 'complete', total: 0 }); + expect(await store.getUnitPlan(executionId)).toBeNull(); +}); + +it('preserves rows and records an actionable failure instead of false success', async () => { + const executionId = await setup({ state: 'BLOCKED' }); + const before = await store.getUnit(executionId, 1, 'a'); + expect(await finalizeUnitCompletion({ store, executionId, runId: 'run', plan })).toMatchObject({ + state: 'incomplete', + unresolvedCount: 1, + }); + expect(await store.getExecution(executionId)).toMatchObject({ + status: 'FAILED', + failure: { code: 'completion_inconsistent' }, + }); + expect(await store.getUnit(executionId, 1, 'a')).toEqual(before); +}); + +it.each(['run', 'plan', 'meta'])( + 'rejects a concurrent %s change between read and terminal write', + async (change) => { + const executionId = await setup(); + let raced = false; + const racing = createProcessStore({ + tableName, + ddb: { + send: async (command) => { + if (command instanceof TransactWriteCommand && !raced) { + raced = true; + if (change === 'plan') + await store.putUnitPlan({ + executionId, + units: [{ slug: 'replacement', dependsOn: [] }], + batches: [['replacement']], + }); + else + await store.updateExecution({ + executionId, + ...(change === 'run' + ? { orchestratorRunId: 'replacement' } + : { currentStage: 'new-attempt' }), + }); + } + return db.doc.send(command); + }, + }, + }); + expect( + await finalizeUnitCompletion({ store: racing, executionId, runId: 'run', plan }), + ).toEqual({ state: 'retired' }); + expect((await store.getExecution(executionId)).status).toBe('RUNNING'); + }, +); + +it('detects a plan replaced after the section checkpoint without re-running any stage', async () => { + const executionId = await setup(); + expect( + await finalizeUnitCompletion({ + store, + executionId, + runId: 'run', + plan, + expectedUnitPlanIdentity: unitPlanIdentity({ units: [{ slug: 'prior' }] }), + }), + ).toMatchObject({ state: 'unknown', reason: 'unit_plan_changed_after_section' }); +}); + +it('bounds completion reads and never calls a truncated prefix complete', async () => { + const executionId = await setup(); + expect(await store.getCompletionRecords(executionId, { maxRecords: 2 })).toMatchObject({ + truncated: true, + units: [], + }); +}); + +it('reconciles a committed final transaction whose response was lost', async () => { + const executionId = await setup(); + let lost = false; + const responseLost = createProcessStore({ + tableName, + ddb: { + send: async (command) => { + const result = await db.doc.send(command); + if (command instanceof TransactWriteCommand && !lost) { + lost = true; + throw new Error('response lost after commit'); + } + return result; + }, + }, + }); + expect( + await finalizeUnitCompletion({ store: responseLost, executionId, runId: 'run', plan }), + ).toMatchObject({ state: 'complete' }); + expect((await store.getExecution(executionId)).status).toBe('SUCCEEDED'); +}); diff --git a/lambda/agentcore/test/promote-units.test.js b/lambda/agentcore/test/promote-units.test.js index 747ae234c..44cf946a8 100644 --- a/lambda/agentcore/test/promote-units.test.js +++ b/lambda/agentcore/test/promote-units.test.js @@ -62,6 +62,23 @@ beforeEach(() => { }; }); +it.each(['checkout', 'renamed-unit', ''])( + 'replaces unsupported old skeleton %s before approving a regenerated DAG', + async (walkingSkeleton) => { + store.getUnitPlan.mockResolvedValue({ walkingSkeleton, autonomyMode: 'gated' }); + const result = await promoteUnits(basePayload, deps); + expect(result.ok).toBe(true); + expect(result.walkingSkeleton).toBe('auth'); + expect(result.selectionProblem).toBeFalsy(); + expect(store.putUnitPlan).toHaveBeenCalledWith( + expect.objectContaining({ walkingSkeleton: 'auth', autonomyMode: 'gated' }), + ); + expect(store.appendEvent).toHaveBeenCalledWith( + expect.objectContaining({ type: 'v2.units.skeleton_reselected' }), + ); + }, +); + describe('pickCurrentArtifact', () => { it('prefers non-superseded rows and the newest timestamp', () => { const rows = [ diff --git a/lambda/intents/index.js b/lambda/intents/index.js index d0468c0e5..c278b2038 100644 --- a/lambda/intents/index.js +++ b/lambda/intents/index.js @@ -40,6 +40,12 @@ import { GRANT_AUTONOMY_OPTION, constructionAutonomyApplies, } from '../shared/construction-autonomy.js'; +import { + unitCompletion, + assessExecutionCompletion, + activePlanUnits, + hasUnitSkipDecision, +} from '../shared/unit-completion.js'; import { SessionReleaseHandoffError, stopSessions } from '../shared/runtime-session.js'; import { composeSessionIdFor, @@ -4246,7 +4252,7 @@ export const handler = async (event, context) => { }); } - if (intentId && httpMethod === 'DELETE') { + if (intentId && httpMethod === 'DELETE' && path?.endsWith(`/intents/${intentId}`)) { const meta = await store.getExecution(intentId); if (!meta || meta.projectId !== projectId) { return response(404, { error: 'Intent not found' }); @@ -4421,8 +4427,12 @@ export const handler = async (event, context) => { 'ADDRESSING_FEEDBACK', 'MERGING', ]); - const prUnits = (records.units ?? []).filter( + const activeUnits = records.unitPlan?.units?.length + ? activePlanUnits(records.unitPlan, records.units) + : (records.units ?? []); + const prUnits = activeUnits.filter( (unit) => + !hasUnitSkipDecision(unit, records.unitPlan) && activeReviewStates.has(unit.state) && (records.unitPrs ?? []).some( (pr) => @@ -4465,8 +4475,11 @@ export const handler = async (event, context) => { 'FAILED', 'BLOCKED', ]); - const repairUnits = (records.units ?? []).filter( - (unit) => Number(unit.sectionIndex) === sectionIndex && repairableStates.has(unit.state), + const repairUnits = activeUnits.filter( + (unit) => + Number(unit.sectionIndex) === sectionIndex && + !hasUnitSkipDecision(unit, records.unitPlan) && + repairableStates.has(unit.state), ); const repairSlugs = [...new Set(repairUnits.map((unit) => unit.slug))]; if (repairSlugs.length === 0) { @@ -4984,18 +4997,30 @@ export const handler = async (event, context) => { // already superseded by the first cleanup step, with no action capable // of waking it. FAILED remains rewindable, so a repeated request can // safely finish an interrupted reset. - await store.updateExecution({ - executionId: intentId, - projectId, - status: 'FAILED', - fromStatus: priorStatus, - startedAt: meta.startedAt, - pendingHumanTaskId: null, - resumeRequired: null, - failureReason: 'rewind_in_progress', - completedAt: null, - orchestratorRunId: `retired-${randomBytes(8).toString('hex')}`, - }); + try { + await store.updateExecution({ + executionId: intentId, + projectId, + status: 'FAILED', + fromStatus: priorStatus, + startedAt: meta.startedAt, + pendingHumanTaskId: null, + resumeRequired: null, + failureReason: 'rewind_in_progress', + completedAt: null, + ifOrchestratorRunId: meta.orchestratorRunId, + orchestratorRunId: `retired-${randomBytes(8).toString('hex')}`, + }); + } catch (error) { + if (error?.name === 'ConditionalCheckFailedException') { + return response(409, { + error: + 'Execution changed while preparing rewind. Refresh and review the restart point again.', + code: 'execution_changed', + }); + } + throw error; + } // Retire a parked run first so the woken orchestrator exits quietly (its // gate is superseded) instead of racing the relaunch. await retireParkedRun(intentId, `rewound to ${fromStageId}`); @@ -5063,7 +5088,7 @@ export const handler = async (event, context) => { sectionIndex, slug, state: 'PENDING', - fields: { failureReason: null, blockedOn: null }, + fields: { failureReason: null, blockedOn: null, completionSkip: null }, }) .catch((err) => logger.error('Unit lane reset failed', err, { sectionIndex, slug })), ); @@ -5306,11 +5331,23 @@ export const handler = async (event, context) => { }); // Retire only after every affected artifact was durably snapshotted. - await store.updateExecution({ - executionId: intentId, - fromStatus: meta.status, - orchestratorRunId: `retired-${randomBytes(8).toString('hex')}`, - }); + try { + await store.updateExecution({ + executionId: intentId, + fromStatus: meta.status, + ifOrchestratorRunId: meta.orchestratorRunId, + orchestratorRunId: `retired-${randomBytes(8).toString('hex')}`, + }); + } catch (error) { + if (error?.name === 'ConditionalCheckFailedException') { + return response(409, { + error: + 'Execution changed while preparing recompose. Refresh and review the stage selection again.', + code: 'execution_changed', + }); + } + throw error; + } await retireParkedRun(intentId, `recomposed from ${fromStage.stageId}`); await stopRuntimeSessions(intentId, meta); for (const stageInstanceId of resetIds) { @@ -5470,6 +5507,9 @@ export const handler = async (event, context) => { if (!records.meta || records.meta.projectId !== projectId) { return response(404, { error: 'Intent not found' }); } + const completion = ['SUCCEEDED', 'FAILED', 'CANCELLED'].includes(records.meta.status) + ? await assessExecutionCompletion({ records, loadPlan: intentMethodology.loadPlan }) + : unitCompletion(records); const artifacts = await fetchArtifacts(g, intentId); const pullRequests = await fetchPullRequests(g, intentId); const stagesByInstance = new Map( @@ -5541,6 +5581,7 @@ export const handler = async (event, context) => { // Unit lanes (docs/v2-parallel.md WP4): the promoted UNITPLAN snapshot // + the live UNIT lane rows, so the UI can render the lane board and // attribute per-unit stage instances. Both null/empty pre-promotion. + unitCompletion: completion, unitPlan: mapUnitPlan(records.unitPlan), units: (records.units ?? []).map(mapUnit), unitPrs: (records.unitPrs ?? []).map(mapUnitPr), diff --git a/lambda/intents/test/intents.test.js b/lambda/intents/test/intents.test.js index bb7ccb7bc..43d135594 100644 --- a/lambda/intents/test/intents.test.js +++ b/lambda/intents/test/intents.test.js @@ -1,3 +1,6 @@ +import { __durableHandler as runOrchestrator } from '../../v2-orchestrator/index.js'; +import { createProcessStore } from '../../shared/v2-process-store.js'; +import { unitPlanIdentity } from '../../shared/unit-completion.js'; import { beforeAll, beforeEach, afterAll, describe, it, expect, vi } from 'vitest'; import { randomUUID } from 'node:crypto'; import { readFileSync } from 'node:fs'; @@ -3142,6 +3145,51 @@ describe('POST /recompose — in-flight reshape', () => { expect(events.map((e) => e.eventType)).toContain('v2.execution.recomposed'); }); + it.each(['ownership race', 'storage failure'])( + 'stops recompose after a retirement %s', + async (failure) => { + const sub = `u-${randomUUID()}`; + const projectId = await seedV2Project(sub); + seedRecomposeFixtures(); + const intent = await seedRun(sub, projectId, { + metaOver: { orchestratorRunId: 'original-run' }, + rows: [ + { stageInstanceId: 'si-analyze', stageId: 'analyze', state: 'SUCCEEDED' }, + { stageInstanceId: 'si-optional', stageId: 'optional', state: 'WAITING_FOR_HUMAN' }, + ], + }); + const pk = `EXEC#${intent.id}`; + seedGate(intent.id, 'keep-pending', { status: 'pending' }); + if (failure === 'ownership race') { + archiveArtifactsSpy.mockImplementationOnce(async () => { + procStore.get(keyOf(pk, 'META')).orchestratorRunId = 'concurrent-run'; + return []; + }); + } else { + ddbMock + .on(UpdateCommand, { + Key: { pk, sk: 'META' }, + ExpressionAttributeValues: { ':ifOrid': 'original-run' }, + }) + .rejects(new Error('storage unavailable')); + } + const result = await recomposeReq(sub, projectId, intent.id, { + composedGrid: { analyze: 'EXECUTE', optional: 'EXECUTE', build: 'SKIP' }, + }); + expect(result.statusCode).toBe(failure === 'ownership race' ? 409 : 500); + if (failure === 'ownership race') + expect(JSON.parse(result.body).code).toBe('execution_changed'); + expect(procStore.get(keyOf(pk, 'HUMAN#keep-pending')).status).toBe('pending'); + expect(procStore.get(keyOf(pk, 'STAGE#si-optional'))).toMatchObject({ + state: 'WAITING_FOR_HUMAN', + attempt: 0, + }); + expect(procStore.get(keyOf(pk, 'META')).composedGrid).toBeFalsy(); + expect(agentcoreMock.commandCalls(StopRuntimeSessionCommand)).toHaveLength(0); + expect(orchestratorInvokes()).toHaveLength(0); + }, + ); + it('freezes the past: a stage that ran cannot flip to SKIP', async () => { const sub = `u-${randomUUID()}`; const projectId = await seedV2Project(sub); @@ -5593,6 +5641,20 @@ describe('DELETE /projects/{id}/intents/{intentId}', () => { .next(); }; + it('does not dispatch unknown nested DELETE routes as intent deletion', async () => { + const sub = `u-${randomUUID()}`; + const projectId = await seedV2Project(sub); + const intent = JSON.parse((await createIntent(sub, projectId)).body); + const result = await handler({ + httpMethod: 'DELETE', + path: `/projects/${projectId}/intents/${intent.id}/unknown`, + pathParameters: { projectId, intentId: intent.id }, + ...claims(sub), + }); + expect(result.statusCode).toBeGreaterThanOrEqual(400); + expect(procStore.get(keyOf(`EXEC#${intent.id}`, 'META'))).toBeTruthy(); + }); + it('deletes a DRAFT intent (no Neptune anchor yet) — 204 and the META row is gone', async () => { const sub = `u-${randomUUID()}`; const projectId = await seedV2Project(sub); @@ -6481,6 +6543,25 @@ describe('POST /rewind', () => { // ── WP4: the unit dimension on the API surface (docs/v2-parallel.md) ───────── describe('WP4 — unit lanes in the detail DTO', () => { + it('keeps an unpinned serial SUCCEEDED intent free of unit completion diagnostics', async () => { + const sub = `u-${randomUUID()}`; + const projectId = await seedV2Project(sub); + const intent = JSON.parse((await createIntent(sub, projectId)).body); + setStatus(intent.id, { status: 'SUCCEEDED', methodologyRelease: null, methodologyPins: null }); + const result = await handler({ + httpMethod: 'GET', + path: `/projects/${projectId}/intents/${intent.id}`, + pathParameters: { projectId, intentId: intent.id }, + ...claims(sub), + }); + expect(result.statusCode).toBe(200); + expect(JSON.parse(result.body)).toMatchObject({ + intent: { status: 'SUCCEEDED' }, + unitCompletion: null, + unitPlan: null, + units: [], + }); + }); it('surfaces unitPlan + units and lane attribution on stages/gates/events', async () => { const sub = `u-${randomUUID()}`; const projectId = await seedV2Project(sub); @@ -6667,14 +6748,234 @@ describe('WP4 — rewind expands per-unit stage instances', () => { .next(); }; - const repair = (sub, projectId, intentId) => + const repair = (sub, projectId, intentId, input = {}) => handler({ httpMethod: 'POST', path: `/projects/${projectId}/intents/${intentId}/repair`, pathParameters: { projectId, intentId }, + body: JSON.stringify(input), ...claims(sub), }); + const falseSuccess = async () => { + const sub = `u-${randomUUID()}`; + const projectId = await seedV2Project(sub); + seedSectionPlan(); + const intent = JSON.parse((await createIntent(sub, projectId)).body); + await seedIntentAnchor(intent.id); + setStatus(intent.id, { + status: 'SUCCEEDED', + orchestratorRunId: 'finished-run', + branch: 'aidlc/feature', + }); + seedStageRow(intent.id, 'units-gen'); + seedStageRow(intent.id, 'cg', 'foundation', 'SUCCEEDED', 1); + seedStageRow(intent.id, 'bt'); + const pk = `EXEC#${intent.id}`; + procStore.set(keyOf(pk, 'UNITPLAN'), { + pk, + sk: 'UNITPLAN', + executionId: intent.id, + units: [ + { slug: 'foundation', dependsOn: [] }, + { slug: 'feature', dependsOn: ['foundation'] }, + ], + batches: [['foundation'], ['feature']], + walkingSkeleton: 'feature', + autonomyMode: 'autonomous', + }); + for (const [slug, state] of [ + ['foundation', 'MERGED'], + ['feature', 'BLOCKED'], + ['old-plan-unit', 'PENDING'], + ]) { + procStore.set(keyOf(pk, `UNIT#S1#${slug}`), { + pk, + sk: `UNIT#S1#${slug}`, + type: 'Unit', + executionId: intent.id, + sectionIndex: 1, + slug, + state, + branch: `aidlc/feature--${slug}`, + }); + } + await g + .addV('PullRequest') + .property('id', `pr-${intent.id}`) + .property('repository', 'owner/repo') + .property('pr_number', 533) + .property('pr_url', 'https://github.com/owner/repo/pull/533') + .property('branch', 'aidlc/feature') + .addE('HAS_PR') + .from_(gremlin.process.statics.V().has('Intent', 'id', intent.id)) + .next(); + sourceControlOperationHandler = (request) => + request.operation === 'pr-status' ? { state: 'open', sourceBranch: 'aidlc/feature' } : null; + return { sub, projectId, intent, pk }; + }; + + it.each(['rewind', 'start'])( + 'completes publication after %s without rebuilding pre-deploy multi-repo integrated units', + async (action) => { + const { sub, projectId, intent, pk } = await falseSuccess(); + const repos = ['owner/repo', 'owner/second']; + setStatus(intent.id, { status: action === 'start' ? 'FAILED' : 'SUCCEEDED', repos }); + const plan = procStore.get(keyOf(pk, 'UNITPLAN')); + plan.walkingSkeleton = 'foundation'; + procStore.get(keyOf(pk, 'UNIT#S1#feature')).state = 'MERGED'; + seedStageRow(intent.id, 'cg', 'feature', 'SUCCEEDED', 1); + const before = ['foundation', 'feature'].map((slug) => + structuredClone(procStore.get(keyOf(pk, `UNIT#S1#${slug}`))), + ); + expect(before.every((row) => !Object.hasOwn(row, 'integrationResults'))).toBe(true); + const result = await handler({ + httpMethod: 'POST', + path: `/projects/${projectId}/intents/${intent.id}/${action}`, + pathParameters: { projectId, intentId: intent.id }, + body: JSON.stringify(action === 'rewind' ? { fromStageId: 'bt' } : {}), + ...claims(sub), + }); + expect(result.statusCode).toBe(202); + const event = JSON.parse( + Buffer.from(orchestratorInvokes().at(-1).args[0].input.Payload).toString(), + ); + const doc = DynamoDBDocumentClient.from( + new (await import('@aws-sdk/client-dynamodb')).DynamoDBClient({}), + ); + const store = createProcessStore({ ddb: doc, tableName: 'v2-proc-test' }); + const runtime = vi.fn(async () => ({ ok: true, state: 'SUCCEEDED' })); + const openPr = vi.fn(async ({ repo }) => ({ + number: 42, + url: `https://example.test/${repo}/pull/42`, + })); + const ctx = { + logger: { info() {}, error() {}, debug() {} }, + step: async (_name, fn) => fn(), + createCallback: async (name) => { + if (name.startsWith('await-')) { + const humanTaskId = name.slice('await-'.length); + await store.answerHumanTask({ + executionId: intent.id, + humanTaskId, + status: 'answered', + answer: { decision: 'approve' }, + answeredBy: sub, + }); + return [Promise.resolve({ answer: { decision: 'approve' } }), `cb-${name}`]; + } + return [Promise.resolve({ ok: true, state: 'SUCCEEDED' }), `cb-${name}`]; + }, + runInChildContext: async (_name, fn) => fn(ctx), + promise: { + allSettled: async (_name, promises) => Promise.allSettled(promises), + race: async (_name, promises) => Promise.race(promises), + }, + }; + const completed = await runOrchestrator(event, ctx, { + store, + loadPlan: (args) => planSpy.actual({ ddb: doc, tableName: 'v2-blocks-test', ...args }), + invokeRuntime: runtime, + stopSession: async () => ({ stopped: true }), + broadcast: async () => {}, + comparePrBranches: async () => ({ status: 'ahead' }), + openPr, + }); + expect(completed, JSON.stringify(completed)).toMatchObject({ ok: true }); + expect((await store.getExecution(intent.id)).status).toBe('SUCCEEDED'); + expect(openPr).toHaveBeenCalledTimes(2); + expect(runtime.mock.calls.filter(([p]) => p.unitSlug || p.command === 'merge-lane')).toEqual( + [], + ); + expect( + ['foundation', 'feature'].map((slug) => procStore.get(keyOf(pk, `UNIT#S1#${slug}`))), + ).toEqual(before); + doc.destroy(); + }, + ); + it.each(['ownership race', 'storage failure'])( + 'stops rewind after a retirement %s', + async (failure) => { + const { sub, projectId, intent, pk } = await falseSuccess(); + seedGate(intent.id, 'keep-pending', { status: 'pending' }); + if (failure === 'ownership race') { + archiveArtifactsSpy.mockImplementationOnce(async () => { + procStore.get(keyOf(pk, 'META')).orchestratorRunId = 'concurrent-run'; + return []; + }); + } else { + ddbMock + .on(UpdateCommand, { + Key: { pk, sk: 'META' }, + ExpressionAttributeValues: { ':ifOrid': 'finished-run' }, + }) + .rejects(new Error('storage unavailable')); + } + const result = await handler({ + httpMethod: 'POST', + path: `/projects/${projectId}/intents/${intent.id}/rewind`, + pathParameters: { projectId, intentId: intent.id }, + body: JSON.stringify({ fromStageId: 'cg' }), + ...claims(sub), + }); + expect(result.statusCode).toBe(failure === 'ownership race' ? 409 : 500); + if (failure === 'ownership race') + expect(JSON.parse(result.body).code).toBe('execution_changed'); + expect(procStore.get(keyOf(pk, 'HUMAN#keep-pending')).status).toBe('pending'); + expect(procStore.get(keyOf(pk, `STAGE#${siOf('cg', 'foundation', 1)}`))).toMatchObject({ + state: 'SUCCEEDED', + attempt: 0, + }); + expect(procStore.get(keyOf(pk, 'UNIT#S1#foundation')).state).toBe('MERGED'); + expect(agentcoreMock.commandCalls(StopRuntimeSessionCommand)).toHaveLength(0); + expect(orchestratorInvokes()).toHaveLength(0); + }, + ); + + it('ordinary rewind explicitly clears a prior omission without changing the answered gate', async () => { + const { sub, projectId, intent, pk } = await falseSuccess(); + procStore.get(keyOf(pk, 'UNIT#S1#feature')).completionSkip = { + humanTaskId: 'old-halt', + planIdentity: 'old-plan', + }; + seedGate(intent.id, 'old-halt', { status: 'answered' }); + procStore.get(keyOf(pk, 'HUMAN#old-halt')).answer = { decision: 'skip' }; + const result = await handler({ + httpMethod: 'POST', + path: `/projects/${projectId}/intents/${intent.id}/rewind`, + pathParameters: { projectId, intentId: intent.id }, + body: JSON.stringify({ fromStageId: 'cg' }), + ...claims(sub), + }); + expect(result.statusCode).toBe(202); + expect(procStore.get(keyOf(pk, 'UNIT#S1#feature'))).toMatchObject({ + state: 'PENDING', + completionSkip: null, + }); + expect(procStore.get(keyOf(pk, 'HUMAN#old-halt'))).toMatchObject({ + status: 'answered', + answer: { decision: 'skip' }, + }); + }); + it.each(['complete', 'partial', 'incomplete'])( + 'keeps terminal %s work outside ordinary repair', + async (state) => { + const { sub, projectId, intent, pk } = await falseSuccess(); + const unit = procStore.get(keyOf(pk, 'UNIT#S1#feature')); + if (state === 'complete') unit.state = 'MERGED'; + if (state === 'partial') + unit.completionSkip = { + humanTaskId: 'accepted', + planIdentity: unitPlanIdentity(procStore.get(keyOf(pk, 'UNITPLAN'))), + }; + const before = structuredClone([...procStore.entries()]); + for (const input of [{}, { preview: true }]) + expect((await repair(sub, projectId, intent.id, input)).statusCode).toBe(409); + expect([...procStore.entries()]).toEqual(before); + expect(orchestratorInvokes()).toHaveLength(0); + }, + ); + it('repairs orphaned parallel waits while preserving merged units', async () => { const sub = `u-${randomUUID()}`; const projectId = await seedV2Project(sub); @@ -6966,6 +7267,7 @@ describe('WP4 — rewind expands per-unit stage instances', () => { status: 'RUNNING', durableExecutionArn: 'arn:aws:lambda:eu-central-1:123:durable-execution:old', }); + seedStageRow(intent.id, 'units-gen'); seedStageRow(intent.id, 'cg', 'asset-containment', 'WAITING_FOR_HUMAN', 1); procStore.set(keyOf(`EXEC#${intent.id}`, 'UNIT#S1#asset-containment'), { pk: `EXEC#${intent.id}`, diff --git a/lambda/shared/test/unit-completion.test.js b/lambda/shared/test/unit-completion.test.js new file mode 100644 index 000000000..548164585 --- /dev/null +++ b/lambda/shared/test/unit-completion.test.js @@ -0,0 +1,331 @@ +import { describe, it, expect, vi } from 'vitest'; +import { + effectiveSkeleton, + skeletonProblem, + unitCompletion, + unitPlanIdentity, + completionSections, + finalizeUnitCompletion, + assessExecutionCompletion, +} from '../unit-completion.js'; + +const unitPlan = { + units: [ + { slug: 'foundation', dependsOn: [] }, + { slug: 'feature', dependsOn: ['foundation'] }, + ], + batches: [['foundation'], ['feature']], + walkingSkeleton: 'feature', +}; +const records = () => ({ + meta: { status: 'SUCCEEDED', orchestratorRunId: 'run1' }, + unitPlan, + units: [ + { slug: 'foundation', sectionIndex: 1, state: 'MERGED' }, + { slug: 'feature', sectionIndex: 1, state: 'BLOCKED' }, + { slug: 'old-plan-unit', sectionIndex: 1, state: 'PENDING' }, + ], +}); + +describe('active-plan completion', () => { + it('counts active work and detects false success regardless of the terminal META status', () => { + expect(unitCompletion(records())).toMatchObject({ + state: 'incomplete', + total: 2, + integrated: 1, + skipped: 0, + }); + }); + it('requires an exact-plan human skip receipt for partial delivery', () => { + const input = records(); + input.units[1].completionSkip = { + humanTaskId: 'halt', + planIdentity: unitPlanIdentity(unitPlan), + }; + expect(unitCompletion(input).state).toBe('partial'); + input.unitPlan = { + ...unitPlan, + units: [...unitPlan.units, { slug: 'new-work', dependsOn: [] }], + }; + expect(unitCompletion(input).state).toBe('incomplete'); + }); + it('keeps legacy accepted partial delivery distinct without trusting a prior run decision', () => { + const input = records(); + input.unitPlan = { ...unitPlan, promotedAt: '2026-10-01T00:00:00.000Z' }; + input.humanTasks = [ + { + humanTaskId: 'eg-halt-s1-r0-run1', + orchestratorRunId: 'run1', + status: 'approved', + answer: { decision: 'skip' }, + sectionIndex: 1, + prompt: 'Lane failure in section 1\n- feature: failed', + createdAt: '2026-10-02T00:00:00.000Z', + }, + ]; + expect(unitCompletion(input)).toMatchObject({ state: 'partial', legacyPartial: true }); + input.meta.orchestratorRunId = 'run2'; + expect(unitCompletion(input).state).toBe('incomplete'); + }); + it('does not apply one legacy skip to other sections or unrelated unfinished units', () => { + const input = records(); + input.humanTasks = [ + { + humanTaskId: 'eg-halt-s1-r0-run1', + orchestratorRunId: 'run1', + status: 'answered', + answer: 'skip', + sectionIndex: 1, + }, + ]; + expect(unitCompletion(input, [1]).state).toBe('unknown'); + expect(unitCompletion(input, [2])).toMatchObject({ state: 'incomplete', skipped: 0, total: 2 }); + }); + it('honors an explicit dependent omission if its per-row receipt write was interrupted', () => { + const input = records(); + input.units[0].state = 'FAILED'; + input.unitPlan = { ...unitPlan, promotedAt: '2026-10-01T00:00:00.000Z' }; + input.humanTasks = [ + { + humanTaskId: 'eg-halt-s1-r0-run1', + orchestratorRunId: 'run1', + status: 'answered', + answer: 'skip', + createdAt: '2026-10-02T00:00:00.000Z', + sectionIndex: 1, + prompt: `- foundation: failed +- skip: explicitly omit these lanes and all their unintegrated dependents from delivery`, + }, + ]; + expect(unitCompletion(input, [1])).toMatchObject({ state: 'partial', skipped: 2 }); + input.humanTasks[0].prompt = `- foundation: failed +- skip: continue without them (their work will be missing downstream)`; + expect(unitCompletion(input, [1])).toMatchObject({ state: 'unknown', skipped: 1 }); + }); + it('uses expected sections even when every row or an entire section is missing', () => { + const sections = completionSections({ + stages: [ + { stageId: 'design', parallelSection: 1 }, + { stageId: 'review' }, + { stageId: 'implement', parallelSection: 2 }, + ], + }); + expect(sections).toEqual([1, 2]); + expect(unitCompletion({ ...records(), units: [] }, sections)).toMatchObject({ + state: 'incomplete', + total: 4, + integrated: 0, + }); + const input = records(); + input.units[1].state = 'MERGED'; + expect(unitCompletion(input, sections)).toMatchObject({ + state: 'incomplete', + integrated: 2, + total: 4, + }); + expect(unitCompletion(input, [1])).toMatchObject({ state: 'complete', total: 2 }); + expect(unitCompletion(input, [])).toMatchObject({ state: 'complete', total: 0 }); + }); + it.each(['PENDING', 'BLOCKED', 'RUNNING', 'FAILED', 'PARTIALLY_MERGED'])( + 'does not resolve an active %s row', + (state) => { + const input = records(); + input.units[1].state = state; + expect(unitCompletion(input, [1])).toMatchObject({ state: 'incomplete', integrated: 1 }); + }, + ); + it('allows serial/no-unit plans but never a missing required unit plan', () => { + expect(unitCompletion({}, [])).toMatchObject({ state: 'complete', total: 0 }); + expect(unitCompletion({}, [1])).toMatchObject({ + state: 'unknown', + reason: 'unit_plan_missing', + }); + expect(unitCompletion({ ...records(), units: [] }).state).toBe('unknown'); + }); + it('checks each repository and preserves the distinction between unknown and partial integration', () => { + const input = records(); + input.meta.repos = ['one', 'two']; + input.units[1].state = 'MERGED'; + expect(unitCompletion(input, [1]).state).toBe('complete'); + for (const row of input.units) + row.integrationResults = [ + { repo: 'one', merged: true }, + { repo: 'two', merged: 'up_to_date' }, + ]; + expect(unitCompletion(input, [1]).state).toBe('complete'); + input.units[1].integrationResults[1].merged = false; + expect(unitCompletion(input, [1])).toMatchObject({ state: 'incomplete', integrated: 1 }); + input.meta.prStrategy = 'pr-per-unit'; + input.unitPrs = input.units.flatMap((row) => + ['one', 'two'].map((repository) => ({ + slug: row.slug, + sectionIndex: 1, + repository, + state: repository === 'one' ? 'MERGED' : 'UNCHANGED', + })), + ); + expect(unitCompletion(input, [1]).state).toBe('complete'); + input.unitPrs[3].state = 'PARTIALLY_MERGED'; + expect(unitCompletion(input, [1]).state).toBe('incomplete'); + }); + it.each([ + [[], 'unknown'], + [[{ repo: 'one', merged: true }], 'unknown'], + [ + [ + { repo: 'one', merged: true }, + { repo: 'two', merged: false }, + ], + 'incomplete', + ], + [null, 'unknown'], + [{}, 'unknown'], + ['invalid', 'unknown'], + [[null], 'unknown'], + [[{}], 'unknown'], + ])( + 'does not treat explicit invalid or incomplete evidence %j as a pre-deploy row', + (evidence, state) => { + const input = records(); + input.meta.repos = ['one', 'two']; + input.units[1].state = 'MERGED'; + input.units[1].integrationResults = evidence; + expect(unitCompletion(input, [1])).toMatchObject({ state, integrated: 1 }); + }, + ); + it('omits unit diagnostics only for legacy unpinned executions without unit work', async () => { + const meta = { status: 'SUCCEEDED', orchestratorRunId: 'old-run' }; + const loadPlan = async () => ({ valid: true, plan: { stages: [{ parallelSection: 1 }] } }); + expect(await assessExecutionCompletion({ records: { meta, units: [] }, loadPlan })).toBeNull(); + expect( + await assessExecutionCompletion({ records: { ...records(), meta }, loadPlan }), + ).toMatchObject({ state: 'unknown', reason: 'legacy_workflow_unpinned' }); + for (const pin of [ + { methodologyPins: { STAGE: {} } }, + { completionValidation: { runId: 'old-run', expectedSections: [1] } }, + ]) { + expect( + await assessExecutionCompletion({ + records: { meta: { ...meta, ...pin }, units: [] }, + loadPlan, + }), + ).toMatchObject({ state: 'unknown', reason: 'unit_plan_missing' }); + } + }); + it.each(['CANCELLED', 'recorded-abort'])( + 'classifies unpinned %s work as aborted before loading workflow definitions', + async (kind) => { + const input = records(); + input.meta.status = kind === 'CANCELLED' ? 'CANCELLED' : 'FAILED'; + if (kind === 'recorded-abort') { + input.meta.failure = { code: 'section_aborted' }; + input.humanTasks = [ + { orchestratorRunId: 'run1', status: 'answered', answer: { decision: 'abort' } }, + ]; + } + const before = structuredClone(input); + const loadPlan = vi.fn(async () => { + throw new Error('workflow pin unavailable'); + }); + expect(await assessExecutionCompletion({ records: input, loadPlan })).toMatchObject({ + state: 'aborted', + reason: 'execution_aborted', + integrated: 1, + total: 2, + unresolved: [expect.objectContaining({ slug: 'feature', state: 'BLOCKED' })], + }); + expect(loadPlan).not.toHaveBeenCalled(); + expect(input).toEqual(before); + }, + ); + it.each(['stale-run', 'missing-run', 'unrelated-failure', 'pending-answer'])( + 'keeps legacy evidence unknown for a %s abort decision', + async (kind) => { + const input = records(); + input.meta.status = 'FAILED'; + input.meta.failure = { + code: kind === 'unrelated-failure' ? 'stage_failed' : 'section_aborted', + }; + input.humanTasks = [ + { + orchestratorRunId: kind === 'stale-run' ? 'prior-run' : 'run1', + status: kind === 'pending-answer' ? 'pending' : 'answered', + answer: 'abort', + }, + ]; + if (kind === 'missing-run') { + delete input.meta.orchestratorRunId; + delete input.humanTasks[0].orchestratorRunId; + } + expect(await assessExecutionCompletion({ records: input, loadPlan: vi.fn() })).toMatchObject({ + state: 'unknown', + reason: 'legacy_workflow_unpinned', + }); + }, + ); + it('retains missing/truncated read semantics before cancellation and handles no-unit cancellation', async () => { + const input = records(); + input.meta.status = 'CANCELLED'; + const loadPlan = vi.fn(); + expect( + await assessExecutionCompletion({ records: { ...input, meta: null }, loadPlan }), + ).toMatchObject({ state: 'unknown', reason: 'execution_missing' }); + expect( + await assessExecutionCompletion({ records: { ...input, truncated: true }, loadPlan }), + ).toMatchObject({ state: 'unknown', reason: 'completion_read_limit' }); + expect( + await assessExecutionCompletion({ + records: { ...input, unitPlan: null, units: [] }, + loadPlan, + }), + ).toMatchObject({ state: 'aborted', total: 0, integrated: 0 }); + expect(loadPlan).not.toHaveBeenCalled(); + }); + it('does not guess section mappings for historical slug-only rows', () => { + const input = records(); + input.units[0].sectionIndex = null; + expect(unitCompletion(input, [1]).state).toBe('unknown'); + }); + it('reports a missing state as unknown without inventing an execution status', () => { + const input = records(); + delete input.units[1].state; + expect(unitCompletion(input, [1])).toMatchObject({ + state: 'unknown', + reason: 'unit_state_missing', + unresolved: [{ slug: 'feature', state: 'UNKNOWN' }], + }); + }); + it('fails closed when completion records were bounded or could not be read', async () => { + expect(unitCompletion({ ...records(), truncated: true }, [1]).state).toBe('unknown'); + await expect( + finalizeUnitCompletion({ + store: { + getExecution: async () => ({ status: 'RUNNING', orchestratorRunId: 'r' }), + getCompletionRecords: async () => { + throw new Error('unavailable'); + }, + }, + executionId: 'e', + runId: 'r', + plan: { stages: [] }, + }), + ).rejects.toThrow('unavailable'); + }); + it('counts missing lane rows as unfinished', () => { + const input = records(); + input.units = []; + expect(unitCompletion(input, [1])).toMatchObject({ + state: 'incomplete', + total: 2, + integrated: 0, + }); + }); +}); + +it('defaults only an absent decision and respects skeleton-off ceremonies', () => { + expect(effectiveSkeleton({ ...unitPlan, walkingSkeleton: null })).toBe('foundation'); + expect(effectiveSkeleton(unitPlan)).toBe('feature'); + expect(skeletonProblem(unitPlan)).toContain('depends on foundation'); + expect(skeletonProblem(unitPlan, undefined, [{ policy: { skeleton: 'off' } }])).toBeNull(); + expect(skeletonProblem({ ...unitPlan, walkingSkeleton: 'foundation' })).toBeNull(); +}); diff --git a/lambda/shared/unit-completion.js b/lambda/shared/unit-completion.js new file mode 100644 index 000000000..beb294789 --- /dev/null +++ b/lambda/shared/unit-completion.js @@ -0,0 +1,377 @@ +// Execution truth, not provider PR state. MERGED means integrated into the +// intent branch. A deliberately omitted lane needs a recorded human decision. +import { createHash } from 'node:crypto'; +import { repoUrl } from './repo-provider.js'; + +export const unitPlanIdentity = (plan) => + createHash('sha256') + .update( + JSON.stringify({ + units: plan?.units ?? [], + sourceArtifactId: plan?.sourceArtifactId ?? null, + producedByStageInstanceId: plan?.producedByStageInstanceId ?? null, + }), + ) + .digest('hex'); + +export const skeletonCeremonyDisabled = (stages = []) => + stages.find((stage) => stage.policy?.skeleton)?.policy?.skeleton === 'off'; + +export const skeletonCandidates = (plan) => { + const roots = new Set( + (plan?.units ?? []).filter((unit) => !(unit.dependsOn ?? []).length).map((unit) => unit.slug), + ); + return [ + ...new Set([...(plan?.batches ?? []).flat(), ...(plan?.units ?? []).map((u) => u.slug)]), + ].filter((slug) => roots.has(slug)); +}; + +// Preserve a captured decision, including an invalid one, so approval/execution +// can explain and reject it. Only an absent decision receives a safe default. +export const effectiveSkeleton = (plan) => + plan?.walkingSkeleton ?? skeletonCandidates(plan)[0] ?? null; + +export const skeletonProblem = (plan, skeleton = effectiveSkeleton(plan), stages = []) => { + if (skeletonCeremonyDisabled(stages)) return null; + const unit = plan?.units?.find((item) => item.slug === skeleton); + if (!unit) return `Walking skeleton "${skeleton}" is not in the current unit plan.`; + if (unit.dependsOn?.length) { + return `Walking skeleton "${skeleton}" depends on ${unit.dependsOn.join(', ')}. It must run first, alone; select a dependency-free unit (${skeletonCandidates(plan).join(', ')}).`; + } + return null; +}; + +export const dependentClosure = (plan, slugs) => { + const selected = new Set(slugs); + let changed = true; + while (changed) { + changed = false; + for (const unit of plan.units ?? []) { + if (!selected.has(unit.slug) && unit.dependsOn?.some((slug) => selected.has(slug))) { + selected.add(unit.slug); + changed = true; + } + } + } + return [...selected]; +}; + +export const hasUnitSkipDecision = (unit, plan) => + unit?.state !== 'MERGED' && + Boolean(unit?.completionSkip?.humanTaskId) && + unit.completionSkip.planIdentity === unitPlanIdentity(plan); + +export const activePlanUnits = (plan, units = []) => { + const slugs = new Set((plan?.units ?? []).map((unit) => unit.slug)); + return units.filter((unit) => slugs.has(unit.slug)); +}; + +export const completionSections = (plan) => [ + ...new Set((plan?.stages ?? []).map((stage) => stage.parallelSection).filter((s) => s != null)), +]; + +export const unverifiedCompletion = (reason) => ({ + state: 'unknown', + reason, + total: 0, + integrated: 0, + skipped: 0, + unresolved: [], + legacyPartial: false, +}); + +// Legacy halt prompts listed the exact failed lanes. Their old "without them" +// wording did not explicitly authorize every blocked dependent, much less a +// different section. Missing scope or plan provenance is ambiguity, not consent. +const legacyOmission = (records, sectionIndex, slug) => { + let ambiguous = false; + for (const gate of records.humanTasks ?? []) { + if ( + !gate.orchestratorRunId || + gate.orchestratorRunId !== records.meta?.orchestratorRunId || + !/^eg-halt-s\d/.test(gate.humanTaskId ?? '') || + !['approved', 'answered'].includes(gate.status) || + (gate.answer?.decision ?? gate.answer?.choice ?? gate.answer) !== 'skip' + ) + continue; + const section = gate.sectionIndex ?? Number(gate.humanTaskId.match(/^eg-halt-s(\d+)/)?.[1]); + if (section !== sectionIndex) continue; + const named = [ + ...(gate.prompt ?? '').matchAll( + /^- ([^:\n]+): (?:failed|FAILED|BLOCKED|PENDING|RUNNING|undefined)$/gm, + ), + ].map((match) => match[1]); + const planTime = Date.parse(records.unitPlan?.promotedAt); + const gateTime = Date.parse(gate.createdAt); + if (!named.length || !Number.isFinite(planTime) || !Number.isFinite(gateTime)) { + ambiguous = true; + continue; + } + if (gateTime < planTime) continue; + if (named.includes(slug)) return 'skipped'; + if (dependentClosure(records.unitPlan, named).includes(slug)) { + // Policy 1 made the dependent scope explicit in the accepted prompt. + // Its per-row receipt loop can fail partway through; preserve the rest + // of that recorded decision without broadening the older "without them". + if ( + gate.prompt.includes( + '- skip: explicitly omit these lanes and all their unintegrated dependents from delivery', + ) + ) + return 'skipped'; + ambiguous = true; + } + } + return ambiguous ? 'unknown' : null; +}; + +const integrationProblem = (records, row) => { + const repos = (records.meta?.repos ?? []).map(repoUrl); + const prs = (records.unitPrs ?? []).filter( + (pr) => pr.sectionIndex === row.sectionIndex && (pr.unitSlug ?? pr.slug) === row.slug, + ); + if (prs.some((pr) => !['MERGED', 'UNCHANGED'].includes(pr.state))) { + return 'repository_integration_incomplete'; + } + if (records.meta?.prStrategy === 'pr-per-unit') { + if (repos.some((repo) => !prs.some((pr) => pr.repository === repo))) { + return 'repository_evidence_missing'; + } + } else if (Object.hasOwn(row, 'integrationResults')) { + // Only an absent attribute identifies a pre-deploy MERGED row. New writes + // always persist this attribute, including an empty result array. + if ( + !Array.isArray(row.integrationResults) || + row.integrationResults.some((r) => !r || typeof r.repo !== 'string' || !r.repo) + ) + return 'repository_evidence_invalid'; + if (row.integrationResults.some((r) => r.merged !== true && r.merged !== 'up_to_date')) { + return 'repository_integration_incomplete'; + } + if (repos.some((repo) => !row.integrationResults.some((r) => r.repo === repo))) { + return 'repository_evidence_missing'; + } + } + return null; +}; + +export const unitCompletion = (records, sectionIndexes) => { + if (records.truncated) return unverifiedCompletion('completion_read_limit'); + const plan = records.unitPlan; + if (!plan?.units?.length) { + return sectionIndexes?.length + ? unverifiedCompletion('unit_plan_missing') + : sectionIndexes + ? { ...unverifiedCompletion(null), state: 'complete' } + : null; + } + const rows = activePlanUnits(plan, records.units); + const sections = sectionIndexes ?? [...new Set(rows.map((unit) => unit.sectionIndex))]; + if (!sections.length && sectionIndexes === undefined) { + return unverifiedCompletion('expected_sections_unknown'); + } + if ( + sections.some((s) => s == null) || + (sections.length && rows.some((r) => r.sectionIndex == null)) + ) { + return unverifiedCompletion('legacy_section_mapping_unknown'); + } + const outcomes = sections.flatMap((sectionIndex) => + plan.units.map((unit) => { + const row = rows.find( + (item) => item.slug === unit.slug && item.sectionIndex === sectionIndex, + ); + const problem = row?.state === 'MERGED' ? integrationProblem(records, row) : null; + const skipped = hasUnitSkipDecision(row, plan); + const legacy = + !skipped && row?.state !== 'MERGED' + ? legacyOmission(records, sectionIndex, unit.slug) + : null; + return { + sectionIndex, + slug: unit.slug, + state: row?.state ?? (row ? 'UNKNOWN' : 'MISSING'), + integrated: row?.state === 'MERGED' && !problem, + skipped: skipped || legacy === 'skipped', + legacyPartial: legacy === 'skipped', + reason: + problem ?? + (legacy === 'unknown' + ? 'legacy_skip_scope_unknown' + : row + ? row.state + ? 'unit_unfinished' + : 'unit_state_missing' + : 'unit_row_missing'), + unknown: + problem === 'repository_evidence_missing' || + problem === 'repository_evidence_invalid' || + legacy === 'unknown' || + Boolean(row && !row.state), + }; + }), + ); + const unresolved = outcomes.filter((unit) => !unit.integrated && !unit.skipped); + return { + state: unresolved.length + ? unresolved.some((unit) => unit.unknown) + ? 'unknown' + : 'incomplete' + : outcomes.some((unit) => unit.skipped) + ? 'partial' + : 'complete', + total: outcomes.length, + integrated: outcomes.filter((unit) => unit.integrated).length, + skipped: outcomes.filter((unit) => unit.skipped).length, + unresolved, + legacyPartial: outcomes.some((unit) => unit.legacyPartial), + reason: unresolved.find((unit) => unit.unknown)?.reason ?? unresolved[0]?.reason ?? null, + }; +}; + +export const completionMessage = (completion) => { + const work = completion.unresolved + .slice(0, 12) + .map((unit) => `section ${unit.sectionIndex}/${unit.slug}: ${unit.reason ?? unit.state}`) + .join('; '); + return `${completion.integrated}/${completion.total} units verified integrated, ${completion.skipped} explicitly omitted. ${ + work || completion.reason || 'Completion evidence unavailable' + }${completion.unresolved.length > 12 ? '; further units in diagnostics' : ''}. Work is preserved. Review completion diagnostics. Work remains preserved; ordinary rewind reruns affected stages.`; +}; + +// Called inside one versioned durable step. The final decision and its status +// commit together. A lost step response returns that receipt without rewriting +// the historical decision. Run, META revision and promoted plan are fenced by +// the process-store transaction; this function never writes unit/stage rows. +export const finalizeUnitCompletion = async ({ + store, + executionId, + runId, + plan, + expectedUnitPlanIdentity = null, +}) => { + const saved = await store.getExecution(executionId, { consistentRead: true }); + if ( + saved?.orchestratorRunId === runId && + ['SUCCEEDED', 'FAILED'].includes(saved.status) && + saved.completionValidation?.runId === runId + ) + return saved.completionValidation; + const records = await store.getCompletionRecords(executionId); + const meta = records.meta; + if (!meta || meta.orchestratorRunId !== runId) return { state: 'retired' }; + if (['SUCCEEDED', 'FAILED'].includes(meta.status) && meta.completionValidation?.runId === runId) + return meta.completionValidation; + if (meta.status !== 'RUNNING') return { state: 'retired' }; + const sections = completionSections(plan); + const completion = + expectedUnitPlanIdentity && unitPlanIdentity(records.unitPlan) !== expectedUnitPlanIdentity + ? unverifiedCompletion('unit_plan_changed_after_section') + : unitCompletion(records, sections); + const succeeded = ['complete', 'partial'].includes(completion.state); + const validation = { + ...completion, + unresolved: completion.unresolved.slice(0, 50), + unresolvedCount: completion.unresolved.length, + version: 1, + runId, + expectedSections: sections, + unitPlanIdentity: unitPlanIdentity(records.unitPlan), + message: succeeded ? null : completionMessage(completion), + }; + try { + await store.updateExecution({ + executionId, + projectId: meta.projectId, + startedAt: meta.startedAt, + ifOrchestratorRunId: runId, + fromStatus: 'RUNNING', + ifUpdatedAt: meta.updatedAt, + ifUnitPlan: records.unitPlan, + status: succeeded ? 'SUCCEEDED' : 'FAILED', + completedAt: new Date().toISOString(), + pendingHumanTaskId: null, + failureReason: validation.message, + failure: succeeded + ? null + : { + code: + completion.state === 'unknown' ? 'completion_unverified' : 'completion_inconsistent', + message: validation.message, + }, + completionValidation: validation, + }); + return validation; + } catch (error) { + if (error?.name === 'ConditionalCheckFailedException') return { state: 'retired' }; + // The transaction may have committed before its response (or the store's + // follow-up read) was lost. Reconcile the saved decision before reporting + // failure; never overwrite a committed success because its response failed. + const current = await store + .getExecution(executionId, { consistentRead: true }) + .catch(() => null); + if ( + current?.orchestratorRunId === runId && + ['SUCCEEDED', 'FAILED'].includes(current.status) && + current.completionValidation?.runId === runId + ) + return current.completionValidation; + throw error; + } +}; + +export const assessExecutionCompletion = async ({ records, loadPlan }) => { + if (!records.meta) return unverifiedCompletion('execution_missing'); + if (records.truncated) return unverifiedCompletion('completion_read_limit'); + const capturedSections = + records.meta.completionValidation?.runId === records.meta.orchestratorRunId + ? records.meta.completionValidation?.expectedSections + : null; + const aborted = + records.meta.status === 'CANCELLED' || + (records.meta.failure?.code === 'section_aborted' && + Boolean(records.meta.orchestratorRunId) && + (records.humanTasks ?? []).some( + (gate) => + gate.orchestratorRunId === records.meta.orchestratorRunId && + ['answered', 'approved'].includes(gate.status) && + (gate.answer?.decision ?? gate.answer?.choice ?? gate.answer) === 'abort', + )); + if (aborted) { + // Cancellation is an execution decision, not proof of completed work. It + // does not require resolving the historical workflow; preserve the unit + // evidence we do have without treating unfinished units as integrated. + const completion = unitCompletion( + records, + Array.isArray(capturedSections) ? capturedSections : undefined, + ); + return { + ...(completion ?? unverifiedCompletion(null)), + state: 'aborted', + reason: 'execution_aborted', + }; + } + // Old workflow versions can still reference LATEST stage definitions. A + // fresh resolution cannot prove the sections that old execution walked. + if ( + !records.meta.methodologyRelease && + !records.meta.methodologyPins?.STAGE && + !Array.isArray(capturedSections) + ) { + // Legacy serial executions have no unit work to diagnose. This exception + // cannot hide a current receipt's expected sections or a pinned unit plan. + if (!records.unitPlan?.units?.length && !records.units?.length) return null; + return unverifiedCompletion('legacy_workflow_unpinned'); + } + let resolved; + try { + resolved = await loadPlan(records.meta); + } catch { + return unverifiedCompletion('pinned_plan_read_failed'); + } + if (!resolved?.valid || !Array.isArray(resolved.plan?.stages)) { + return unverifiedCompletion('pinned_plan_unverified'); + } + const completion = unitCompletion(records, capturedSections ?? completionSections(resolved.plan)); + return completion; +}; diff --git a/lambda/shared/v2-process-store.js b/lambda/shared/v2-process-store.js index 4255b9613..936b1b8b7 100644 --- a/lambda/shared/v2-process-store.js +++ b/lambda/shared/v2-process-store.js @@ -133,6 +133,23 @@ const retryTransactionConflicts = (client) => ({ }, }); +const snapshotCondition = (row, fields) => { + if (!row) return { ConditionExpression: 'attribute_not_exists(pk)' }; + const names = {}; + const values = {}; + const conditions = fields.map((field, index) => { + names[`#s${index}`] = field; + if (row[field] === undefined) return `attribute_not_exists(#s${index})`; + values[`:s${index}`] = row[field]; + return `#s${index} = :s${index}`; + }); + return { + ConditionExpression: `attribute_exists(pk) AND ${conditions.join(' AND ')}`, + ExpressionAttributeNames: names, + ...(Object.keys(values).length ? { ExpressionAttributeValues: values } : {}), + }; +}; + const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { if (!client) throw new Error('createProcessStore requires a DynamoDB DocumentClient'); const ddb = retryTransactionConflicts(client); @@ -140,6 +157,47 @@ const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { const now = () => (clock ? clock() : new Date().toISOString()); const nextId = () => (ids ? ids() : randomUUID()); + // Used by recovery's retired owner as well as active section decisions. + // The token check and row mutation must commit together. + const updateForRun = async (executionId, runId, params, ifUnitPlan) => { + const { ReturnValues: _returnValues, ...update } = params; + await transact([ + { + ConditionCheck: { + TableName: table(), + Key: executionMetaKey(executionId), + ConditionExpression: 'orchestratorRunId = :run', + ExpressionAttributeValues: { ':run': runId }, + }, + }, + ...(ifUnitPlan === undefined + ? [] + : [ + { + ConditionCheck: { + TableName: table(), + Key: unitPlanKey(executionId), + ...snapshotCondition(ifUnitPlan, [ + 'units', + 'sourceArtifactId', + 'producedByStageInstanceId', + 'promotedAt', + ]), + }, + }, + ]), + { Update: update }, + ]); + const { Item } = await ddb.send( + new GetCommand({ + TableName: table(), + Key: params.Key, + ConsistentRead: true, + }), + ); + return Item ?? null; + }; + const transact = async (TransactItems) => { try { await ddb.send(new TransactWriteCommand({ TransactItems })); @@ -358,8 +416,22 @@ const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { return Item ?? null; }; - const deleteWorkflowCheckpoint = async (executionId) => { + const deleteWorkflowCheckpoint = async (executionId, { ifOrchestratorRunId } = {}) => { if (!executionId) throw new Error('deleteWorkflowCheckpoint requires executionId'); + if (ifOrchestratorRunId) { + await transact([ + { + ConditionCheck: { + TableName: table(), + Key: executionMetaKey(executionId), + ConditionExpression: 'orchestratorRunId = :run', + ExpressionAttributeValues: { ':run': ifOrchestratorRunId }, + }, + }, + { Delete: { TableName: table(), Key: workflowCheckpointKey(executionId) } }, + ]); + return null; + } const { Attributes } = await ddb.send( new DeleteCommand({ TableName: table(), @@ -381,6 +453,8 @@ const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { status, fromStatus = null, ifOrchestratorRunId = null, + ifUpdatedAt, + ifUnitPlan, ifPendingHumanTaskId, ownership = null, additionalWrites = [], @@ -397,6 +471,7 @@ const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { startedAt, completedAt, failureReason, + completionValidation, failure, startedBy, starterName, @@ -500,6 +575,10 @@ const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { sets.push('completedAt = :ca'); values[':ca'] = completedAt; } + if (completionValidation !== undefined) { + sets.push('completionValidation = :completionValidation'); + values[':completionValidation'] = completionValidation; + } if (failureReason !== undefined) { sets.push('failureReason = :fr'); values[':fr'] = failureReason; @@ -758,6 +837,10 @@ const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { conditions.push('orchestratorRunId = :ifOrid'); params.ExpressionAttributeValues[':ifOrid'] = ifOrchestratorRunId; } + if (ifUpdatedAt !== undefined) { + conditions.push('updatedAt = :ifUpdatedAt'); + params.ExpressionAttributeValues[':ifUpdatedAt'] = ifUpdatedAt; + } if (ifResumeRequiredFor !== null) { conditions.push('resumeRequired.humanTaskId = :ifRrh'); params.ExpressionAttributeValues[':ifRrh'] = ifResumeRequiredFor; @@ -789,18 +872,40 @@ const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { ...scopeActivityWrites(table(), [row], { includePlatform: true }), ); } + const planWrites = + ifUnitPlan === undefined + ? [] + : [ + { + ConditionCheck: { + TableName: table(), + Key: unitPlanKey(executionId), + ...snapshotCondition(ifUnitPlan, [ + 'units', + 'batches', + 'sourceArtifactId', + 'producedByStageInstanceId', + 'walkingSkeleton', + 'skipMatrix', + 'autonomyMode', + 'promotedAt', + 'updatedAt', + ]), + }, + }, + ]; if (ownership) { await writeStageAttempt({ executionId, ownership, - writes: [...additionalWrites, ...credentialWrites, { Update: params }], + writes: [...additionalWrites, ...planWrites, ...credentialWrites, { Update: params }], }); // Transactions cannot return ALL_NEW; read the committed row instead. return getExecution(executionId, { consistentRead: true }); } - if (additionalWrites.length) { + if (additionalWrites.length || planWrites.length) { const { ReturnValues: _returnValues, ...update } = params; - await transact([...additionalWrites, ...credentialWrites, { Update: update }]); + await transact([...additionalWrites, ...planWrites, ...credentialWrites, { Update: update }]); return getExecution(executionId, { consistentRead: true }); } if (credentialBinding !== undefined) { @@ -1412,6 +1517,7 @@ const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { humanTaskId, supersededBy = null, ownership = null, + ifOrchestratorRunId = null, }) => { const ts = now(); try { @@ -1444,6 +1550,7 @@ const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { await writeStageAttempt({ executionId, ownership, writes: [{ Update: input }] }); return getHumanTask(executionId, humanTaskId, { consistentRead: true }); } + if (ifOrchestratorRunId) return updateForRun(executionId, ifOrchestratorRunId, input); const { Attributes } = await ddb.send(new UpdateCommand(input)); return Attributes; } catch (e) { @@ -1647,6 +1754,7 @@ const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { preservePendingCodeCommitRefs = false, loopBackId = null, countsAgainstCap = true, + ifOrchestratorRunId = null, }) => { const existing = await getStage(executionId, stageInstanceId, { consistentRead: Boolean(loopBackId), @@ -1721,6 +1829,7 @@ const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { : { ReturnValues: 'ALL_NEW' }), }; if (!loopBackId) { + if (ifOrchestratorRunId) return updateForRun(executionId, ifOrchestratorRunId, stageUpdate); const { Attributes } = await ddb.send(new UpdateCommand(stageUpdate)); return Attributes; } @@ -2131,6 +2240,56 @@ const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { return items; }; + // Completion does not need outputs, model prompts, metrics or graph state. + // Bound every read, including a single unusually large execution. A limit is + // explicitly unverified; callers must never treat the prefix as a clean run. + const getCompletionRecords = async ( + executionId, + { maxRecords = 2000, includeStages = false } = {}, + ) => { + if (!Number.isInteger(maxRecords) || maxRecords < 2 || maxRecords > 10000) { + throw new Error('Completion record limit must be 2..10000'); + } + const meta = await getExecution(executionId, { consistentRead: true }); + const unitPlan = await getUnitPlan(executionId, { consistentRead: true }); + const records = { + meta, + unitPlan, + units: [], + unitPrs: [], + humanTasks: [], + stages: [], + truncated: false, + }; + let remaining = maxRecords - 2; + for (const [field, prefix] of [ + ['units', 'UNIT#'], + ['unitPrs', 'UNITPR#'], + ['humanTasks', 'HUMAN#'], + ...(includeStages ? [['stages', 'STAGE#']] : []), + ]) { + let cursor; + do { + const page = await ddb.send( + new QueryCommand({ + TableName: table(), + KeyConditionExpression: 'pk = :pk AND begins_with(sk, :prefix)', + ExpressionAttributeValues: { ':pk': executionPk(executionId), ':prefix': prefix }, + ConsistentRead: true, + Limit: Math.min(remaining + 1, 100), + ExclusiveStartKey: cursor, + }), + ); + const items = page.Items ?? []; + records[field].push(...items.slice(0, remaining)); + if (items.length > remaining) return { ...records, truncated: true }; + remaining -= items.length; + cursor = page.LastEvaluatedKey; + } while (cursor); + } + return records; + }; + const listActiveExecutions = async ({ limit = Infinity } = {}) => { const items = []; let ExclusiveStartKey; @@ -2275,11 +2434,12 @@ const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { // All unit lane rows for an execution. SK prefix 'UNIT#' (exact — a bare // 'UNIT' would also match UNITPLAN). Paginated: the orchestrator's lane // scheduler reads this — a dropped page would make lanes invisible. - const listUnits = async (executionId) => { + const listUnits = async (executionId, { consistentRead = false } = {}) => { const items = await queryAll(ddb, { TableName: table(), KeyConditionExpression: 'pk = :pk AND begins_with(sk, :p)', ExpressionAttributeValues: { ':pk': executionPk(executionId), ':p': 'UNIT#' }, + ...(consistentRead ? { ConsistentRead: true } : {}), }); return items.toSorted(bySk); }; @@ -2377,6 +2537,8 @@ const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { state, fromStates = null, fields = {}, + ifOrchestratorRunId = null, + ifUnitPlan, }) => { if (!UNIT_STATES.includes(state)) throw new Error(`invalid unit state: ${state}`); const ts = now(); @@ -2426,6 +2588,8 @@ const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { }); } try { + if (ifOrchestratorRunId) + return await updateForRun(executionId, ifOrchestratorRunId, params, ifUnitPlan); const { Attributes } = await ddb.send(new UpdateCommand(params)); return Attributes; } catch (e) { @@ -2492,6 +2656,7 @@ const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { repository, state, fromStates = null, + ifOrchestratorRunId = null, fields = {}, }) => { if (state !== undefined && !UNIT_PR_STATES.includes(state)) { @@ -2542,6 +2707,7 @@ const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { params.ExpressionAttributeValues[`:from${index}`] = value; }); } + if (ifOrchestratorRunId) return updateForRun(executionId, ifOrchestratorRunId, params); try { const { Attributes } = await ddb.send(new UpdateCommand(params)); return Attributes; @@ -2870,6 +3036,7 @@ const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { skipMatrix, walkingSkeleton, autonomyMode, + ifOrchestratorRunId = null, }) => { const sets = ['updatedAt = :ts']; const values = { ':ts': now() }; @@ -2885,16 +3052,16 @@ const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { sets.push('autonomyMode = :am'); values[':am'] = autonomyMode; } - const { Attributes } = await ddb.send( - new UpdateCommand({ - TableName: table(), - Key: unitPlanKey(executionId), - UpdateExpression: `SET ${sets.join(', ')}`, - ExpressionAttributeValues: values, - ConditionExpression: 'attribute_exists(pk)', - ReturnValues: 'ALL_NEW', - }), - ); + const params = { + TableName: table(), + Key: unitPlanKey(executionId), + UpdateExpression: `SET ${sets.join(', ')}`, + ExpressionAttributeValues: values, + ConditionExpression: 'attribute_exists(pk)', + ReturnValues: 'ALL_NEW', + }; + if (ifOrchestratorRunId) return updateForRun(executionId, ifOrchestratorRunId, params); + const { Attributes } = await ddb.send(new UpdateCommand(params)); return Attributes; }; @@ -3206,9 +3373,13 @@ const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { // seconds and outputs are the bulk of a long run's partition. '#' is 0x23; // '$' (0x24) is the next code point, so sk < 'OUTPUT#' and sk >= 'OUTPUT$' // partition the SK space exactly around the OUTPUT# prefix. - const getExecutionRecords = async (executionId, { includeOutputs = true } = {}) => { + const getExecutionRecords = async ( + executionId, + { includeOutputs = true, consistentRead = false } = {}, + ) => { const base = { TableName: table(), + ...(consistentRead ? { ConsistentRead: true } : {}), ExpressionAttributeValues: { ':pk': executionPk(executionId) }, }; const Items = includeOutputs @@ -3351,6 +3522,7 @@ const createProcessStore = ({ ddb: client, tableName, clock, ids } = {}) => { appendOutput, getOutputs, listProjectExecutions, + getCompletionRecords, listActiveExecutions, listStaleActiveExecutions, patchExecutionConfig, diff --git a/lambda/v2-orchestrator/index.js b/lambda/v2-orchestrator/index.js index 5d8f5a9ea..df4122511 100644 --- a/lambda/v2-orchestrator/index.js +++ b/lambda/v2-orchestrator/index.js @@ -73,6 +73,15 @@ import { } from '../shared/construction-autonomy.js'; import { broadcastToIntentChannel } from '../shared/ws-fanout.js'; import { resolveRuntimeTarget } from '../shared/runtime-target.js'; +import { + activePlanUnits, + skeletonProblem, + finalizeUnitCompletion, + completionSections, + unitCompletion, + completionMessage, + unitPlanIdentity, +} from '../shared/unit-completion.js'; import { sdkTarget, stopSession as stopSharedSession } from '../shared/runtime-session.js'; import { awaitEngineGate, @@ -415,9 +424,14 @@ const handler = async (event, ctx, deps = defaultDeps()) => { return { ok: false, reason: 'not_a_start' }; } - const meta = await ctx.step('load-meta', () => - store.getExecution(executionId, { consistentRead: true }), - ); + const meta = await ctx.step('load-meta', async () => { + const loaded = await store.getExecution(executionId, { + consistentRead: true, + }); + // Version behavior in the existing first checkpoint. Old histories have no + // marker and retain their operation sequence; repair creates a new history. + return loaded ? { ...loaded, completionPolicyVersion: 1 } : null; + }); if (!meta) return { ok: false, reason: 'execution_not_found' }; const runtimeTarget = resolveRuntimeTarget(meta, RUNTIME_ARN()); const stopIntentSession = (sessionId) => stopSession(sessionId, runtimeTarget); @@ -1161,6 +1175,7 @@ const handler = async (event, ctx, deps = defaultDeps()) => { publishCheckpoint, ids: { projectId, intentId, executionId }, runId: null, // stamped below once minted + completionPolicyVersion: meta.completionPolicyVersion ?? 0, intentBranch: meta.branch, cloneBase: { repos: meta.repos ?? [], @@ -1723,7 +1738,9 @@ const handler = async (event, ctx, deps = defaultDeps()) => { sectionIndex: fanoutSection.index, unitPlan: unitPlanForGate, sectionStages: fanoutSection.stages, - skeleton: defaultSkeletonFor(unitPlanForGate), + skeleton: defaultSkeletonFor(unitPlanForGate, { + legacy: !meta.completionPolicyVersion, + }), }), ] : []), @@ -2181,12 +2198,26 @@ const handler = async (event, ctx, deps = defaultDeps()) => { }); const effective = { walkingSkeleton: - overrides.walkingSkeleton ?? defaultSkeletonFor(unitPlanForGate), + overrides.walkingSkeleton ?? + defaultSkeletonFor(unitPlanForGate, { + legacy: !meta.completionPolicyVersion, + }), skipMatrix: overrides.skipMatrix ?? unitPlanForGate.skipMatrix ?? {}, }; + const problem = skeletonProblem( + unitPlanForGate, + effective.walkingSkeleton, + fanoutSection.stages, + ); + if (meta.completionPolicyVersion && problem) return { ...effective, problem }; try { - await store.updateUnitPlanDecisions({ executionId, ...effective }); - } catch { + await store.updateUnitPlanDecisions({ + executionId, + ...effective, + ...(meta.completionPolicyVersion ? { ifOrchestratorRunId: runId } : {}), + }); + } catch (error) { + if (meta.completionPolicyVersion) throw error; /* promotion defaults stand when the patch write is lost */ } if (overrides.rejected.length) { @@ -2204,6 +2235,8 @@ const handler = async (event, ctx, deps = defaultDeps()) => { return effective; }, ); + if (decisions.problem) + return await fail('invalid_walking_skeleton', decisions.problem); await emitEvent( ctx, `fanout-approved-${stage.stageId}-${round}`, @@ -2376,24 +2409,64 @@ const handler = async (event, ctx, deps = defaultDeps()) => { // Terminal success — CAS on the ownership token: a retired run (relaunched // under a new runId while this one was still unwinding) exits quietly. The // verdict is the step result so a durable replay sees it too. - const ownedFinish = await ctx.step('finish-succeeded', async () => { + let ownedFinish; + if (meta.completionPolicyVersion >= 1) { + let completion; try { - await store.updateExecution({ - executionId, - projectId, - status: 'SUCCEEDED', - startedAt: meta.startedAt, - completedAt: nowIso(), - failureReason: null, - failure: null, - ...(runId ? { ifOrchestratorRunId: runId } : {}), - }); - return true; - } catch (e) { - if (e?.name === 'ConditionalCheckFailedException') return false; - throw e; + completion = await ctx.step('finish-completion-v1', () => + finalizeUnitCompletion({ + store, + executionId, + runId, + plan: planResult.plan, + expectedUnitPlanIdentity: sectionToolkit.completionUnitPlan + ? unitPlanIdentity(sectionToolkit.completionUnitPlan) + : null, + }), + ); + } catch { + return fail( + 'completion_unverified', + 'Completion state could not be read or committed. Work is preserved; review diagnostics before retrying.', + null, + { fromStatus: 'RUNNING' }, + ); } - }); + if (completion.state === 'retired') return { ok: false, reason: 'retired', intentId }; + if (!['complete', 'partial'].includes(completion.state)) { + await emitEvent( + ctx, + 'completion-failed-event-v2', + 'v2.execution.failed', + completion.message, + ); + return { + ok: false, + reason: + completion.state === 'unknown' ? 'completion_unverified' : 'completion_inconsistent', + intentId, + }; + } + ownedFinish = true; + } else + ownedFinish = await ctx.step('finish-succeeded', async () => { + try { + await store.updateExecution({ + executionId, + projectId, + status: 'SUCCEEDED', + startedAt: meta.startedAt, + completedAt: nowIso(), + failureReason: null, + failure: null, + ...(runId ? { ifOrchestratorRunId: runId } : {}), + }); + return true; + } catch (e) { + if (e?.name === 'ConditionalCheckFailedException') return false; + throw e; + } + }); if (!ownedFinish) { logger.info('retired run skipped terminal success write'); return { ok: false, reason: 'retired', intentId }; @@ -2405,9 +2478,45 @@ const handler = async (event, ctx, deps = defaultDeps()) => { // intent branch onto the base branch, via the source-control service (the // v1 unmerged-branch guard lives inside createPullRequest). PR problems // never un-succeed the run — every outcome is a loud timeline event. A - const prResults = await ctx.step('open-pr', async () => - openIntentPrs({ - openPr, + const prResults = await ctx.step('open-pr', async () => { + const verifyCompletion = async () => { + const records = await store.getCompletionRecords(executionId); + const receipt = records.meta?.completionValidation; + if ( + records.meta?.orchestratorRunId !== runId || + records.meta?.status !== 'SUCCEEDED' || + receipt?.runId !== runId || + !['complete', 'partial'].includes(receipt?.state) || + receipt?.unitPlanIdentity !== unitPlanIdentity(records.unitPlan) + ) + throw new Error( + 'Execution ownership, plan or completion decision changed before publication', + ); + const completion = unitCompletion(records, completionSections(planResult.plan)); + if (!['complete', 'partial'].includes(completion.state)) + throw new Error(completionMessage(completion)); + }; + if (meta.completionPolicyVersion >= 1) { + try { + await verifyCompletion(); + } catch { + return [ + { + eventType: 'v2.pr.failed', + summary: + 'Final publication withheld: completion evidence is incomplete or unavailable. Work is preserved; review completion diagnostics.', + }, + ]; + } + } + return openIntentPrs({ + openPr: + meta.completionPolicyVersion >= 1 + ? async (input) => { + await verifyCompletion(); + return openPr(input); + } + : openPr, comparePrBranches, store, meta, @@ -2415,8 +2524,8 @@ const handler = async (event, ctx, deps = defaultDeps()) => { runId, applicationUrl, log: (m) => logger.info(m), - }), - ); + }); + }); for (let i = 0; i < prResults.length; i++) { const r = prResults[i]; await emitEvent(ctx, `pr-event-${i}`, r.eventType, r.summary); @@ -2898,15 +3007,17 @@ const openIntentPrs = async ({ // rows, or a pre-WP3 run, just omits the section). let unitLines = []; try { - const units = (await store.listUnits?.(executionId)) ?? []; + const allUnits = (await store.listUnits?.(executionId)) ?? []; + const unitPlan = await store.getUnitPlan?.(executionId); + const units = unitPlan ? activePlanUnits(unitPlan, allUnits) : allUnits; const unmerged = units.filter((u) => u.state && u.state !== 'MERGED'); if (units.length) { unitLines = [ '', - `Units: ${units.length} total, ${units.length - unmerged.length} merged.`, + `Units: ${units.length} total, ${units.length - unmerged.length} integrated into the intent branch.`, ...unmerged.map( (u) => - `- ⚠️ unit \`${u.slug}\` NOT merged (${u.state}${u.failureReason ? `: ${u.failureReason}` : ''}) — its branch is preserved`, + `- ⚠️ unit \`${u.slug}\` NOT integrated (${u.state}${u.failureReason ? `: ${u.failureReason}` : ''}) — its branch is preserved`, ), ]; } diff --git a/lambda/v2-orchestrator/section.js b/lambda/v2-orchestrator/section.js index 7fd9ede09..12da433c1 100644 --- a/lambda/v2-orchestrator/section.js +++ b/lambda/v2-orchestrator/section.js @@ -51,6 +51,16 @@ import processKeysPkg from '../shared/v2-process-keys.js'; import { normalizeChangedFileProvenance } from '../shared/changed-file-provenance.js'; +import { + dependentClosure, + effectiveSkeleton, + hasUnitSkipDecision, + skeletonCeremonyDisabled, + skeletonProblem, + unitPlanIdentity, + unitCompletion, + completionMessage, +} from '../shared/unit-completion.js'; import { stageIsNoopForUnit } from '../shared/unit-kind-pruning.js'; import { buildIntentAttribution } from './pr-attribution.js'; import { bindGateCallback, unparkGate } from './gate-callback.js'; @@ -394,14 +404,14 @@ export const validateFanoutOverrides = (answer, { slugs, sectionStages, bySlug } // it is (still) a known unit, else the first slug of the first topological // wave. Shared by the fan-out approval (index.js) and the section runner so // the gate presents exactly what the section would run. -export const defaultSkeletonFor = (unitPlan) => { - const bySlug = new Map((unitPlan?.units ?? []).map((u) => [u.slug, u])); - const waves = ( - unitPlan?.batches?.length ? unitPlan.batches : [(unitPlan?.units ?? []).map((u) => u.slug)] - ).map((w) => w.filter((slug) => bySlug.has(slug))); - return unitPlan?.walkingSkeleton && bySlug.has(unitPlan.walkingSkeleton) +export const defaultSkeletonFor = (unitPlan, { legacy = false } = {}) => { + if (!legacy) return effectiveSkeleton(unitPlan); + const slugs = new Set((unitPlan?.units ?? []).map((unit) => unit.slug)); + return slugs.has(unitPlan?.walkingSkeleton) ? unitPlan.walkingSkeleton - : (waves.flat()[0] ?? null); + : ((unitPlan?.batches?.length ? unitPlan.batches.flat() : [...slugs]).find((slug) => + slugs.has(slug), + ) ?? null); }; // The fan-out summary appended to the unit-DAG stage's validation gate (A2 @@ -419,7 +429,12 @@ export const fanoutGateAddendum = ({ sectionIndex, unitPlan, sectionStages, skel `- ${u.slug}${(u.dependsOn ?? []).length ? ` (depends on ${(u.dependsOn ?? []).join(', ')})` : ''}`, ), '', - `Walking skeleton: ${skeleton} (runs first, solo, with a mandatory review gate).`, + skeletonCeremonyDisabled(sectionStages) + ? 'Walking-skeleton ceremony is disabled: all units run with the normal stage and batch gates.' + : `Walking skeleton: ${skeleton} (runs first, solo, with a mandatory review gate).`, + ...(skeletonProblem(unitPlan, skeleton, sectionStages) + ? [skeletonProblem(unitPlan, skeleton, sectionStages)] + : []), `Per-unit stages: ${sectionStages.map((s) => s.stageId).join(', ')}.`, `Skip matrix (CONDITIONAL stages skipped per unit): ${ Object.keys(unitPlan.skipMatrix ?? {}).length @@ -457,18 +472,21 @@ export const runParallelSection = async (segment, toolkit) => { unitPrProvider = null, applicationUrl, runId = null, + completionPolicyVersion = 1, // Derive-time enrichment mode ('off'|'llm'), snapshotted on META — rides // the lane derive-artifacts dispatches like the once-per-workflow hook. deriveEnrichment = 'off', attachments = [], } = toolkit; const { executionId, intentId, projectId } = ids; + const checkedCompletion = completionPolicyVersion >= 1; const sk = `s${segment.index}`; // Consistent: the fan-out approval patched this plan's decisions just before. const unitPlan = await ctx.step(`load-unit-plan-${sk}`, () => store.getUnitPlan(executionId, { consistentRead: true }), ); + if (checkedCompletion) toolkit.completionUnitPlan = unitPlan; if (!unitPlan || (unitPlan.units ?? []).length === 0) { return await fail( 'unit_plan_missing', @@ -480,6 +498,16 @@ export const runParallelSection = async (segment, toolkit) => { unitPlan.batches?.length ? unitPlan.batches : [unitPlan.units.map((u) => u.slug)] ).map((w) => w.filter((slug) => bySlug.has(slug))); const laneOrder = waves.flat(); + if (checkedCompletion) { + const problem = skeletonProblem(unitPlan, undefined, segment.stages); + if (problem) return fail('invalid_walking_skeleton', problem); + if (laneOrder.length !== bySlug.size || new Set(laneOrder).size !== bySlug.size) { + return fail( + 'invalid_unit_waves', + 'Every current-plan unit must occur exactly once in the scheduling waves.', + ); + } + } // ── fan-out decisions (A2 rules 2/7/8) ──────────────────────────────────── // The human approved the fan-out at the unit-DAG stage's validation gate @@ -487,7 +515,7 @@ export const runParallelSection = async (segment, toolkit) => { // UNITPLAN via updateUnitPlanDecisions. The plan row read above (a durable // step) is therefore the section's scheduling truth — no second gate here. const decisions = { - walkingSkeleton: defaultSkeletonFor(unitPlan), + walkingSkeleton: defaultSkeletonFor(unitPlan, { legacy: !checkedCompletion }), skipMatrix: unitPlan.skipMatrix ?? {}, }; await emitEvent( @@ -504,6 +532,7 @@ export const runParallelSection = async (segment, toolkit) => { const withIntegrationLock = makeMergeLock(); // slug → 'MERGED' | 'FAILED' | 'BLOCKED' — settled lane states (this run). const laneState = new Map(); + const skipped = new Set(); // slug → in-flight lane DurablePromise (wavefront cross-lane coordination). const lanePromises = new Map(); const usesUnitPrs = prStrategy === 'pr-per-unit'; @@ -523,8 +552,28 @@ export const runParallelSection = async (segment, toolkit) => { // gate are not replayed and dependency waits can continue from durable DDB // truth rather than the empty in-memory map. const persistedUnits = await ctx.step(`load-unit-states-${sk}`, () => - store.listUnits(executionId), + store.listUnits(executionId, { consistentRead: checkedCompletion }), ); + if (checkedCompletion) { + let initial; + try { + initial = await ctx.step(`load-completion-v1-${sk}`, async () => { + const records = await store.getCompletionRecords(executionId); + if (records.meta?.orchestratorRunId !== runId) return { state: 'retired' }; + return unitCompletion(records, [segment.index]); + }); + } catch { + return fail( + 'completion_unverified', + 'Existing unit integration evidence could not be read. Work is preserved.', + ); + } + if (initial.state === 'retired') return { ok: false, reason: 'retired', intentId }; + // Validate recorded repository evidence before releasing dependencies. + // Unfinished lanes may execute; uncertain or contradictory integration may not. + if (initial.state === 'unknown' || initial.unresolved.some((unit) => unit.state === 'MERGED')) + return fail('completion_unverified', completionMessage(initial)); + } const persistedBySlug = new Map( (persistedUnits ?? []) .filter((row) => Number(row.sectionIndex) === Number(segment.index)) @@ -538,6 +587,14 @@ export const runParallelSection = async (segment, toolkit) => { ) { laneState.set(row.slug, 'MERGED'); } + if ( + checkedCompletion && + Number(row.sectionIndex) === Number(segment.index) && + bySlug.has(row.slug) && + hasUnitSkipDecision(row, unitPlan) + ) { + skipped.add(row.slug); + } } const callProvider = async (method, args) => { @@ -834,7 +891,10 @@ export const runParallelSection = async (segment, toolkit) => { sectionIndex: segment.index, slug, state: 'MERGED', - fields: { mergedAt: true }, + fields: { + mergedAt: true, + ...(checkedCompletion ? { integrationResults: [] } : {}), + }, }); }); return { slug, state: 'MERGED' }; @@ -1530,6 +1590,7 @@ export const runParallelSection = async (segment, toolkit) => { state: 'MERGED', fields: { mergedAt: true, + ...(checkedCompletion ? { integrationResults: [] } : {}), integrationOwner: false, blockedReason: null, }, @@ -1660,6 +1721,8 @@ export const runParallelSection = async (segment, toolkit) => { let depState = laneState.get(dep); if (lanePromises.has(dep)) { const settled = await lanePromises.get(dep).catch(() => ({ state: 'FAILED' })); + if (checkedCompletion && settled?.state === 'TERMINAL') + return { slug, state: 'TERMINAL', value: settled.value }; depState = settled?.state; } if (depState !== 'MERGED') { @@ -2005,6 +2068,7 @@ export const runParallelSection = async (segment, toolkit) => { intentId, executionId, unitSlug: slug, + ...(checkedCompletion ? { sectionIndex: segment.index } : {}), repos: cloneBase.repos, unitBranch, intentBranch, @@ -2073,22 +2137,59 @@ export const runParallelSection = async (segment, toolkit) => { stageId: 'merge-lane', reason: merge?.reason ?? 'merge_failed', detail: merge?.detail ?? null, + integrationResults: merge?.results, }); } - await laneCtx.step(`unit-merged-${sk}-${slug}${rTag}`, async () => { - try { - await store.updateUnitState({ - executionId, - sectionIndex: segment.index, - slug, - state: 'MERGED', - fromStates: ['MERGING', 'RUNNING'], - fields: { mergedAt: true }, - }); - } catch { - /* tolerated */ - } - }); + const integrationRecorded = await laneCtx.step( + `unit-merged-${sk}-${slug}${rTag}`, + async () => { + try { + const saved = await store.updateUnitState({ + executionId, + sectionIndex: segment.index, + slug, + state: 'MERGED', + fromStates: ['MERGING', 'RUNNING'], + ...(checkedCompletion ? { ifOrchestratorRunId: runId, ifUnitPlan: unitPlan } : {}), + fields: { + mergedAt: true, + ...(checkedCompletion + ? { + integrationResults: (merge.results ?? []).map(({ repo, merged, reason }) => ({ + repo, + merged: merged ?? false, + ...(reason ? { reason } : {}), + })), + } + : {}), + }, + }); + if (checkedCompletion && !saved) + throw new Error('Unit integration write was not accepted'); + return true; + } catch (error) { + if (!checkedCompletion) return true; // frozen legacy behavior + const records = await store.getCompletionRecords(executionId); + if ( + records.meta?.orchestratorRunId !== runId || + unitPlanIdentity(records.unitPlan) !== unitPlanIdentity(unitPlan) + ) + return false; + const row = records.units.find( + (item) => item.sectionIndex === segment.index && item.slug === slug, + ); + const completion = unitCompletion(records, [segment.index]); + if ( + row?.state === 'MERGED' && + !completion.unresolved.some((item) => item.slug === slug) + ) + return true; + throw error; + } + }, + ); + if (integrationRecorded === false) + return { slug, state: 'TERMINAL', value: { ok: false, reason: 'retired', intentId } }; await emitEvent( laneCtx, `unit-merged-event-${sk}-${slug}${rTag}`, @@ -2114,7 +2215,12 @@ export const runParallelSection = async (segment, toolkit) => { } }; - const laneFailed = async (laneCtx, slug, round, { stageId, reason, detail }) => { + const laneFailed = async ( + laneCtx, + slug, + round, + { stageId, reason, detail, integrationResults }, + ) => { const rTag = round > 0 ? `-r${round}` : ''; await laneCtx.step(`unit-failed-${sk}-${slug}${rTag}`, async () => { try { @@ -2123,7 +2229,21 @@ export const runParallelSection = async (segment, toolkit) => { sectionIndex: segment.index, slug, state: 'FAILED', - fields: { failureReason: `${stageId}: ${reason}${detail ? ` (${detail})` : ''}` }, + ...(checkedCompletion ? { ifOrchestratorRunId: runId, ifUnitPlan: unitPlan } : {}), + fields: { + failureReason: `${stageId}: ${reason}${detail ? ` (${detail})` : ''}`, + ...(checkedCompletion && integrationResults + ? { + integrationResults: integrationResults.map( + ({ repo, merged, reason: repoReason }) => ({ + repo, + merged: merged ?? false, + ...(repoReason ? { reason: repoReason } : {}), + }), + ), + } + : {}), + }, }); } catch { /* lane bookkeeping must never mask the failure */ @@ -2168,9 +2288,6 @@ export const runParallelSection = async (segment, toolkit) => { return { terminal }; }; - const blocked = () => - [...laneState.entries()].filter(([, s]) => s === 'BLOCKED').map(([slug]) => slug); - // Halt-and-ask (stage-protocol.md): wait for all lanes (done by the caller), // preserve successes (pushed work + MERGED lanes), ask retry / skip / abort. // An unparseable answer RE-ASKS (fresh gate, one human answer per attempt) @@ -2179,6 +2296,12 @@ export const runParallelSection = async (segment, toolkit) => { // deterministic fallback is abort (audited), never an unbounded hot loop. // Returns 'retry' | 'skip' | a terminal value (abort/retired). const haltAndAsk = async (round, failedSlugs, idSuffix = '') => { + const unresolvedStates = [...laneState.entries()] + .filter(([slug, state]) => state !== 'MERGED' && !skipped.has(slug)) + .map(([, state]) => state); + const failedCount = unresolvedStates.filter((state) => state === 'FAILED').length; + const blockedCount = unresolvedStates.filter((state) => state === 'BLOCKED').length; + const otherCount = unresolvedStates.length - failedCount - blockedCount; const detailLines = failedSlugs.map((slug) => { const row = laneState.get(slug); return `- ${slug}: ${row === 'FAILED' ? 'failed' : row}`; @@ -2188,14 +2311,14 @@ export const runParallelSection = async (segment, toolkit) => { name: `halt-${sk}${idSuffix}-r${round}${attempt ? `-a${attempt}` : ''}`, sectionIndex: segment.index, prompt: [ - `Lane failure in section ${segment.index} (${failedSlugs.length} unit(s) failed, ${blocked().length} blocked, ${ + `Unresolved work in section ${segment.index} (${failedCount} failed, ${blockedCount} blocked${otherCount ? `, ${otherCount} other unresolved` : ''}, ${ [...laneState.values()].filter((s) => s === 'MERGED').length - } merged).`, + } integrated).`, ...detailLines, '', 'All pushed work is preserved on the unit branches. Choose:', - '- retry: re-run the failed lanes (and their blocked dependents) in place', - '- skip: continue without them (their work will be missing downstream)', + '- retry: re-run unresolved lanes (and their blocked dependents) in place; integrated work is preserved', + '- skip: explicitly omit these lanes and all their unintegrated dependents from delivery', '- abort: fail the run', ...(attempt ? [ @@ -2232,11 +2355,11 @@ export const runParallelSection = async (segment, toolkit) => { return { terminal: await fail( 'section_aborted', - `section ${segment.index}: units [${failedSlugs.join(', ')}] failed; human aborted`, + `section ${segment.index}: units [${failedSlugs.join(', ')}] unresolved; human aborted`, ), }; } - return { choice }; + return { choice, humanTaskId: halt.gate.humanTaskId }; } }; @@ -2272,7 +2395,7 @@ export const runParallelSection = async (segment, toolkit) => { initialSlugs, { tag, idSuffix = '', feedbackTaskId = null, revive = false }, ) => { - let toRun = initialSlugs; + let toRun = initialSlugs.filter((slug) => !skipped.has(slug)); let round = 0; // A retry round re-runs the previous round's FAILED lanes and revives // their BLOCKED dependents. Unbounded by design — each round is one @@ -2284,16 +2407,46 @@ export const runParallelSection = async (segment, toolkit) => { laneOpts: { idSuffix, feedbackTaskId, revive }, }); if (terminal) return terminal; - const failed = toRun.filter((s) => laneState.get(s) === 'FAILED'); + const failed = toRun.filter((s) => + checkedCompletion + ? laneState.get(s) !== 'MERGED' && !skipped.has(s) + : laneState.get(s) === 'FAILED', + ); if (failed.length === 0) return null; - const decision = await haltAndAsk(round, failed, idSuffix); + const decisionSuffix = checkedCompletion ? `${tag}${idSuffix}` : idSuffix; + const decision = await haltAndAsk(round, failed, decisionSuffix); if (decision.terminal) return decision.terminal; if (decision.choice === 'skip') { + if (checkedCompletion) { + const omitted = dependentClosure(unitPlan, failed).filter( + (slug) => laneState.get(slug) !== 'MERGED' && !skipped.has(slug), + ); + await ctx.step(`record-lane-skips-${sk}${decisionSuffix}-r${round}`, async () => { + for (const slug of omitted) { + const saved = await store.updateUnitState({ + executionId, + sectionIndex: segment.index, + slug, + state: laneState.get(slug) ?? 'BLOCKED', + ifOrchestratorRunId: runId, + fields: { + completionSkip: { + humanTaskId: decision.humanTaskId, + planIdentity: unitPlanIdentity(unitPlan), + }, + }, + }); + if (!saved) throw new Error('Unit skip decision lost execution ownership'); + } + return omitted; + }); + omitted.forEach((slug) => skipped.add(slug)); + } await emitEvent( ctx, - `skip-lanes-${sk}${idSuffix}-r${round}`, + `skip-lanes-${sk}${checkedCompletion ? decisionSuffix : idSuffix}-r${round}`, 'v2.units.lanes_skipped', - `Continuing section ${segment.index} without [${failed.join(', ')}] (human skip); dependents stay blocked`, + `Continuing section ${segment.index} without [${failed.join(', ')}] and their unintegrated dependents (human skip); their work is omitted from delivery`, { sectionIndex: segment.index }, ); return null; @@ -2314,7 +2467,7 @@ export const runParallelSection = async (segment, toolkit) => { }); } round += 1; - toRun = [...failed, ...toRun.filter((s) => laneState.get(s) === 'BLOCKED')]; + toRun = [...new Set([...failed, ...toRun.filter((s) => laneState.get(s) === 'BLOCKED')])]; } }; @@ -2327,8 +2480,7 @@ export const runParallelSection = async (segment, toolkit) => { // policy, which only exists in release mode — so an unpinned run takes the // pre-Phase-6 path byte for byte. const skeleton = decisions.walkingSkeleton; - const skeletonCeremonyOff = - (segment.stages.find((stage) => stage.policy?.skeleton)?.policy?.skeleton ?? null) === 'off'; + const skeletonCeremonyOff = skeletonCeremonyDisabled(segment.stages); const skeletonAlreadyApproved = laneState.get(skeleton) === 'MERGED' && CONSTRUCTION_AUTONOMY_MODES.includes(unitPlan.autonomyMode); @@ -2414,14 +2566,51 @@ export const runParallelSection = async (segment, toolkit) => { // ── phase 2: autonomy ladder (A2 rule 9), then the remaining lanes ─────── const remaining = laneOrder.filter( - (s) => (skeletonCeremonyOff || s !== skeleton) && laneState.get(s) !== 'MERGED', + (s) => + (skeletonCeremonyOff || s !== skeleton) && laneState.get(s) !== 'MERGED' && !skipped.has(s), ); + const validateCompletion = async () => { + if (checkedCompletion) { + let completion; + try { + completion = await ctx.step(`section-completion-v1-${sk}`, async () => { + const records = await store.getCompletionRecords(executionId); + if (records.meta?.orchestratorRunId !== runId) return { state: 'retired' }; + if (unitPlanIdentity(records.unitPlan) !== unitPlanIdentity(unitPlan)) + return { + state: 'unknown', + reason: 'unit_plan_changed_after_section', + integrated: 0, + total: unitPlan.units.length, + skipped: 0, + unresolved: [], + }; + return unitCompletion(records, [segment.index]); + }); + } catch { + return fail( + 'completion_unverified', + `Section ${segment.index} completion evidence could not be read. Work is preserved.`, + ); + } + if (completion.state === 'retired') return { ok: false, reason: 'retired', intentId }; + return ['complete', 'partial'].includes(completion.state) + ? null + : fail( + completion.state === 'unknown' ? 'completion_unverified' : 'section_incomplete', + completionMessage(completion), + ); + } + return null; // Marker-absent histories retain the pre-check operation sequence. + }; if (remaining.length === 0) { + const incomplete = await validateCompletion(); + if (incomplete) return incomplete; await emitEvent( ctx, `fan-in-${sk}`, 'v2.units.fan_in', - `Section ${segment.index} complete: ${[...laneState.values()].filter((s) => s === 'MERGED').length}/${laneOrder.length} unit(s) merged`, + `Section ${segment.index} resolved: ${[...laneState.values()].filter((s) => s === 'MERGED').length}/${laneOrder.length} unit(s) integrated${skipped.size ? `; ${skipped.size} explicitly omitted by human decision` : ''}`, { sectionIndex: segment.index }, ); return null; @@ -2455,8 +2644,13 @@ export const runParallelSection = async (segment, toolkit) => { /* META mirror is best-effort; the UNITPLAN write below is the truth */ } try { - await store.updateUnitPlanDecisions({ executionId, autonomyMode: parsed }); - } catch { + await store.updateUnitPlanDecisions({ + executionId, + autonomyMode: parsed, + ...(checkedCompletion ? { ifOrchestratorRunId: runId } : {}), + }); + } catch (error) { + if (checkedCompletion) throw error; /* tolerated — the step result governs this run */ } return parsed; @@ -2544,14 +2738,14 @@ export const runParallelSection = async (segment, toolkit) => { } // ── fan-in bookkeeping (A2 rule 5) ──────────────────────────────────────── + const incomplete = await validateCompletion(); + if (incomplete) return incomplete; const merged = [...laneState.values()].filter((s) => s === 'MERGED').length; await emitEvent( ctx, `fan-in-${sk}`, 'v2.units.fan_in', - `Section ${segment.index} complete: ${merged}/${laneOrder.length} unit(s) merged${ - merged < laneOrder.length ? ' (some lanes skipped after failure — see halt decisions)' : '' - }`, + `Section ${segment.index} resolved: ${merged}/${laneOrder.length} unit(s) integrated${skipped.size ? `; ${skipped.size} explicitly omitted by human decision` : ''}`, { sectionIndex: segment.index }, ); return null; diff --git a/lambda/v2-orchestrator/test/completion-section.test.js b/lambda/v2-orchestrator/test/completion-section.test.js new file mode 100644 index 000000000..9b3748733 --- /dev/null +++ b/lambda/v2-orchestrator/test/completion-section.test.js @@ -0,0 +1,206 @@ +import { randomUUID } from 'node:crypto'; +import { afterAll, beforeAll, expect, it, vi } from 'vitest'; +import { createProcessStore } from '../../shared/v2-process-store.js'; +import { createV2Table, deleteV2Table, makeDdb } from '../../agentcore/test/helpers/v2-table.js'; +import { runParallelSection } from '../section.js'; + +const tableName = `completion-section-${randomUUID()}`; +let db; +let store; +beforeAll(async () => { + db = makeDdb(); + await createV2Table(db.client, tableName); + store = createProcessStore({ ddb: db.doc, tableName }); +}); +afterAll(async () => { + await deleteV2Table(db.client, tableName); + db.client.destroy(); +}); +const setup = async (fault) => { + const executionId = randomUUID(); + const runId = 'run'; + const units = [ + { slug: 'foundation', dependsOn: [] }, + { slug: 'dependent', dependsOn: ['foundation'] }, + ]; + await store.createExecution({ + executionId, + intentId: executionId, + projectId: 'project', + repos: ['one', 'two'], + }); + await store.updateExecution({ executionId, status: 'RUNNING', orchestratorRunId: runId }); + await store.putUnitPlan({ + executionId, + units, + batches: [['foundation'], ['dependent']], + autonomyMode: 'autonomous', + }); + await store.syncUnitRows({ executionId, units, sectionIndexes: [1] }); + if (fault === 'legacy') + for (const { slug } of units) + await store.updateUnitState({ + executionId, + sectionIndex: 1, + slug, + state: 'MERGED', + fields: { mergedAt: true }, + }); + if (fault === 'contradictory') + await store.updateUnitState({ + executionId, + sectionIndex: 1, + slug: 'foundation', + state: 'MERGED', + fields: { + integrationResults: [ + { repo: 'one', merged: true }, + { repo: 'two', merged: false }, + ], + }, + }); + const events = []; + const executeStage = vi.fn(async () => ({ state: 'SUCCEEDED' })); + const invokeRuntime = vi.fn(async (payload) => + payload.command === 'merge-lane' + ? { + ok: true, + results: + fault === 'empty-results' + ? [] + : [ + { repo: 'one', merged: true }, + { repo: 'two', merged: true }, + ], + } + : { ok: true }, + ); + const ctx = { + logger: { error() {} }, + step: async (_name, fn) => fn(), + runInChildContext: async (_name, fn) => fn(ctx), + promise: { allSettled: async (_name, promises) => Promise.allSettled(promises) }, + createCallback: async (name) => { + const humanTaskId = name.replace(/^await-/, ''); + await store.answerHumanTask({ + executionId, + humanTaskId, + status: 'answered', + answer: 'abort', + answeredBy: 'operator', + ifOrchestratorRunId: runId, + }); + return [Promise.resolve({ answer: 'abort' }), `cb-${name}`]; + }, + }; + const guarded = { + ...store, + updateUnitState: async (input) => { + if (input.slug === 'foundation' && input.state === 'MERGED') { + if (fault === 'write-failed') throw new Error('required integration write failed'); + if (fault === 'lost-response') { + await store.updateUnitState(input); + throw new Error('response lost'); + } + if (fault === 'retired') { + await store.updateExecution({ executionId, orchestratorRunId: 'successor' }); + return store.updateUnitState(input); + } + } + return store.updateUnitState(input); + }, + }; + const toolkit = { + ctx, + store: guarded, + runId, + completionPolicyVersion: 1, + ids: { executionId, intentId: executionId, projectId: 'project' }, + intentBranch: 'intent', + cloneBase: { repos: ['one', 'two'] }, + intentSessionId: 'session', + maxParallelUnits: 2, + invokeRuntime, + executeStage, + stopSession: async () => {}, + broadcast: async () => {}, + emitEvent: async (_ctx, _name, type, summary) => events.push({ type, summary }), + fail: async (reason, detail) => ({ ok: false, reason, detail }), + stageInstanceIdFor: (stage, slug) => `${stage}-${slug}`, + }; + return { + executionId, + events, + executeStage, + invokeRuntime, + run: () => + runParallelSection( + { index: 1, stages: [{ stageId: 'build', policy: { skeleton: 'off' } }] }, + toolkit, + ), + }; +}; + +it('does not release dependent work when the required MERGED write fails', async () => { + const world = await setup('write-failed'); + expect(await world.run()).toMatchObject({ ok: false, reason: 'section_aborted' }); + expect(world.executeStage.mock.calls.map(([, , options]) => options.unitSlug)).toEqual([ + 'foundation', + ]); + expect(world.invokeRuntime.mock.calls.filter(([p]) => p.command === 'merge-lane')).toHaveLength( + 1, + ); + expect((await store.getUnit(world.executionId, 1, 'foundation')).state).toBe('FAILED'); + expect((await store.getUnit(world.executionId, 1, 'dependent')).state).toBe('BLOCKED'); + expect(world.events.some((event) => event.type === 'v2.units.fan_in')).toBe(false); +}); +it('reconciles a lost integration response without rerunning stage or merge', async () => { + const world = await setup('lost-response'); + expect(await world.run()).toBeNull(); + expect(world.executeStage).toHaveBeenCalledTimes(2); + expect(world.invokeRuntime.mock.calls.filter(([p]) => p.command === 'merge-lane')).toHaveLength( + 2, + ); + for (const row of await store.listUnits(world.executionId)) { + expect(row).toMatchObject({ + state: 'MERGED', + integrationResults: [ + { repo: 'one', merged: true }, + { repo: 'two', merged: true }, + ], + }); + } +}); +it('does not let contradictory persisted repository evidence unblock dependencies', async () => { + const world = await setup('contradictory'); + expect(await world.run()).toMatchObject({ ok: false, reason: 'completion_unverified' }); + expect(world.executeStage).not.toHaveBeenCalled(); + expect(world.invokeRuntime).not.toHaveBeenCalled(); +}); +it('does not mark successor work failed when the integration owner is retired', async () => { + const world = await setup('retired'); + expect(await world.run()).toMatchObject({ ok: false, reason: 'retired' }); + expect((await store.getExecution(world.executionId)).orchestratorRunId).toBe('successor'); + expect((await store.getUnit(world.executionId, 1, 'foundation')).state).toBe('MERGING'); + expect((await store.getUnit(world.executionId, 1, 'dependent')).state).toBe('PENDING'); +}); + +it('persists explicit empty integration evidence instead of producing another legacy row', async () => { + const world = await setup('empty-results'); + expect(await world.run()).toMatchObject({ ok: false }); + expect(await store.getUnit(world.executionId, 1, 'foundation')).toMatchObject({ + integrationResults: [], + }); +}); + +it('hydrates pre-deploy multi-repo MERGED rows from the real store without rebuilding units', async () => { + const world = await setup('legacy'); + expect(await world.run()).toBeNull(); + expect(world.executeStage).not.toHaveBeenCalled(); + expect(world.invokeRuntime).not.toHaveBeenCalled(); + expect( + (await store.listUnits(world.executionId)).every( + (row) => row.state === 'MERGED' && !Object.hasOwn(row, 'integrationResults'), + ), + ).toBe(true); +}); diff --git a/lambda/v2-orchestrator/test/construction-autonomy-gates.test.js b/lambda/v2-orchestrator/test/construction-autonomy-gates.test.js index cf79faa1d..bf3a050d5 100644 --- a/lambda/v2-orchestrator/test/construction-autonomy-gates.test.js +++ b/lambda/v2-orchestrator/test/construction-autonomy-gates.test.js @@ -5,7 +5,7 @@ // exactly the gates it opened before the mode existed. import { beforeEach, describe, expect, it, vi } from 'vitest'; -import { __durableHandler } from '../index.js'; +import { __durableHandler } from './fixtures/completion-handler.js'; const makeCtx = (over = {}) => { const stageCallbacks = new Map(); diff --git a/lambda/v2-orchestrator/test/final-completion.test.js b/lambda/v2-orchestrator/test/final-completion.test.js new file mode 100644 index 000000000..d054406d1 --- /dev/null +++ b/lambda/v2-orchestrator/test/final-completion.test.js @@ -0,0 +1,132 @@ +import { expect, it, vi } from 'vitest'; +import { __durableHandler } from '../index.js'; +import { unitPlanIdentity } from '../../shared/unit-completion.js'; + +const setup = () => { + const meta = { + executionId: 'intent', + intentId: 'intent', + projectId: 'project', + status: 'RUNNING', + workflowId: 'workflow', + workflowVersion: 1, + scope: 'feature', + repos: ['owner/repo'], + branch: 'intent-branch', + startedAt: '2026-10-01T00:00:00.000Z', + }; + const unitPlan = { + units: [ + { slug: 'a', dependsOn: [] }, + { slug: 'b', dependsOn: ['a'] }, + ], + batches: [['a'], ['b']], + autonomyMode: 'autonomous', + }; + const units = unitPlan.units.map(({ slug }) => ({ sectionIndex: 1, slug, state: 'MERGED' })); + const records = { meta, unitPlan, units: structuredClone(units) }; + const events = []; + const store = { + getExecution: async () => meta, + updateExecution: vi.fn(async (input) => { + Object.assign(meta, input); + return { ...meta }; + }), + getUnitPlan: async () => unitPlan, + listUnits: async () => units, // scheduler's earlier state, independent of final read + getCompletionRecords: vi.fn(async () => records), + appendEvent: async (event) => { + events.push(event.type); + }, + }; + const deps = { + store, + loadPlan: async () => ({ + valid: true, + plan: { stages: [{ stageId: 'build', parallelSection: 1 }] }, + }), + invokeRuntime: vi.fn(async () => ({ ok: true })), + stopSession: async () => ({ stopped: true }), + broadcast: async () => {}, + comparePrBranches: async () => ({ status: 'ahead' }), + openPr: vi.fn(async () => ({ number: 1, url: 'https://example.test/pr/1' })), + }; + const ctx = { + step: async (name, fn) => + name.startsWith('section-completion-v1-') || name.startsWith('load-completion-v1-') + ? { state: 'complete', unresolved: [] } + : fn(), + logger: { info() {}, error() {} }, + }; + return { + deps, + records, + events, + run: () => + __durableHandler( + { + action: 'start', + intentId: 'intent', + executionId: 'intent', + }, + ctx, + deps, + ), + }; +}; + +it.each(['BLOCKED', 'RUNNING', 'FAILED', 'PENDING', 'missing'])( + 'withholds both success and final PR for an independent final %s observation', + async (state) => { + const world = setup(); + if (state === 'missing') world.records.units = []; + else world.records.units[1].state = state; + const result = await world.run(); + expect(result).toMatchObject({ ok: false, reason: 'completion_inconsistent' }); + expect(world.records.meta.status).toBe('FAILED'); + expect(world.events).not.toContain('v2.execution.succeeded'); + expect(world.deps.openPr).not.toHaveBeenCalled(); + expect(world.deps.invokeRuntime.mock.calls.some(([p]) => p.command === 'run-stage-start')).toBe( + false, + ); + }, +); + +it('publishes complete work and plan-bound approved omissions', async () => { + for (const skipped of [false, true]) { + const world = setup(); + if (skipped) { + world.records.units[1].state = 'BLOCKED'; + world.records.units[1].completionSkip = { + humanTaskId: 'approved-skip', + planIdentity: unitPlanIdentity(world.records.unitPlan), + }; + } + expect(await world.run()).toMatchObject({ ok: true }); + expect(world.records.meta.completionValidation.state).toBe(skipped ? 'partial' : 'complete'); + expect(world.deps.openPr).toHaveBeenCalledOnce(); + } +}); + +it('fails visibly without publication when final reads fail', async () => { + const world = setup(); + world.deps.store.getCompletionRecords.mockRejectedValue(new Error('DynamoDB unavailable')); + expect(await world.run()).toMatchObject({ ok: false, reason: 'completion_unverified' }); + expect(world.records.meta.status).toBe('FAILED'); + expect(world.deps.openPr).not.toHaveBeenCalled(); + expect(world.events).not.toContain('v2.execution.succeeded'); +}); + +it('rechecks publication after saved success without rewriting that decision', async () => { + const world = setup(); + let reads = 0; + world.deps.store.getCompletionRecords.mockImplementation(async () => { + if (++reads === 2) world.records.units[1].state = 'BLOCKED'; + return world.records; + }); + await world.run(); + expect(world.deps.openPr).not.toHaveBeenCalled(); + expect(world.events).toContain('v2.pr.failed'); + expect(world.records.meta.completionValidation.state).toBe('complete'); + expect(world.records.meta.status).toBe('SUCCEEDED'); +}); diff --git a/lambda/v2-orchestrator/test/fixtures/completion-handler.js b/lambda/v2-orchestrator/test/fixtures/completion-handler.js new file mode 100644 index 000000000..2b2c92277 --- /dev/null +++ b/lambda/v2-orchestrator/test/fixtures/completion-handler.js @@ -0,0 +1,23 @@ +import { __durableHandler as handler } from '../../index.js'; +import { repoUrl } from '../../../shared/repo-provider.js'; +import { withCompletionReads } from './completion-store.js'; + +// Upgrade the existing control-flow fakes to the persisted completion read +// contract. Production logic still runs; real conditional stores and deliberately +// incomplete snapshots are covered by the completion tests. +export const __durableHandler = (event, ctx, deps) => + handler(event, ctx, { + ...deps, + store: withCompletionReads(deps.store), + invokeRuntime: async (...args) => { + const result = await deps.invokeRuntime(...args); + const [payload] = args; + if (payload.command === 'merge-lane' && result?.ok && result.results === undefined) { + return { + ...result, + results: (payload.repos ?? []).map((repo) => ({ repo: repoUrl(repo), merged: true })), + }; + } + return result; + }, + }); diff --git a/lambda/v2-orchestrator/test/fixtures/completion-store.js b/lambda/v2-orchestrator/test/fixtures/completion-store.js new file mode 100644 index 000000000..bcb285df3 --- /dev/null +++ b/lambda/v2-orchestrator/test/fixtures/completion-store.js @@ -0,0 +1,56 @@ +// Stateful completion reads for the orchestration fakes. Existing tests own +// their store methods and assertions; these reads assemble the writes those +// fakes accepted. Conditional-write races use DynamoDB Local separately. +import { vi } from 'vitest'; + +export const withCompletionReads = (store) => { + const calls = {}; + for (const name of ['updateExecution', 'updateUnitState', 'createUnitPr', 'updateUnitPr']) { + if (!store[name]) continue; + if (!store[name].mock) { + const original = store[name]; + store[name] = vi.fn((...args) => original(...args)); + } + calls[name] = () => + store[name].mock.calls + .filter((_, index) => store[name].mock.settledResults?.[index]?.type !== 'rejected') + .map(([input]) => input); + } + store.getCompletionRecords = async (executionId) => { + const loaded = await store.getExecution(executionId); + const meta = { ...loaded }; + for (const input of calls.updateExecution?.() ?? []) { + for (const field of ['status', 'orchestratorRunId', 'completionValidation']) { + if (input[field] !== undefined) meta[field] = input[field]; + } + } + const units = new Map( + ((await store.listUnits?.(executionId)) ?? []).map((row) => [ + `${row.sectionIndex}:${row.slug}`, + { ...row }, + ]), + ); + for (const input of calls.updateUnitState?.() ?? []) { + const key = `${input.sectionIndex}:${input.slug}`; + units.set(key, { ...units.get(key), ...input, ...input.fields }); + } + const unitPrs = new Map( + ((await store.listUnitPrs?.(executionId)) ?? []).map((row) => [ + `${row.sectionIndex}:${row.unitSlug ?? row.slug}:${row.repository}`, + { ...row }, + ]), + ); + for (const input of [...(calls.createUnitPr?.() ?? []), ...(calls.updateUnitPr?.() ?? [])]) { + const key = `${input.sectionIndex}:${input.slug}:${input.repository}`; + unitPrs.set(key, { ...unitPrs.get(key), ...input, ...input.fields }); + } + return { + meta, + unitPlan: (await store.getUnitPlan?.(executionId)) ?? null, + units: [...units.values()], + unitPrs: [...unitPrs.values()], + humanTasks: [], + }; + }; + return store; +}; diff --git a/lambda/v2-orchestrator/test/fixtures/dependent-skeleton-plan.json b/lambda/v2-orchestrator/test/fixtures/dependent-skeleton-plan.json new file mode 100644 index 000000000..4c42d3665 --- /dev/null +++ b/lambda/v2-orchestrator/test/fixtures/dependent-skeleton-plan.json @@ -0,0 +1,83 @@ +{ + "units": [ + { + "dependsOn": [], + "slug": "notifications-data-infra", + "kind": null + }, + { + "dependsOn": ["notifications-data-infra"], + "slug": "notification-core", + "kind": "service" + }, + { + "dependsOn": ["notification-core", "notifications-data-infra"], + "slug": "execution-event-capture", + "kind": "service" + }, + { + "kind": "service", + "dependsOn": ["notification-core", "notifications-data-infra"], + "slug": "notifications-api" + }, + { + "dependsOn": ["notifications-api"], + "kind": "ui", + "slug": "fe-inbox" + }, + { + "dependsOn": ["notification-core", "notifications-api", "fe-inbox"], + "kind": "service", + "slug": "realtime-user-channel" + }, + { + "dependsOn": ["realtime-user-channel", "fe-inbox"], + "kind": "ui", + "slug": "fe-realtime-alerts" + }, + { + "dependsOn": [ + "notification-core", + "notifications-data-infra", + "notifications-api", + "fe-inbox" + ], + "kind": "service", + "slug": "email-setup" + }, + { + "dependsOn": [ + "email-setup", + "execution-event-capture", + "realtime-user-channel", + "fe-realtime-alerts" + ], + "kind": "service", + "slug": "email-immediate" + }, + { + "dependsOn": ["email-immediate", "execution-event-capture"], + "kind": "service", + "slug": "email-digest" + }, + { + "slug": "notification-observability", + "dependsOn": ["email-digest", "execution-event-capture", "notification-core"], + "kind": null + } + ], + "batches": [ + ["notifications-data-infra"], + ["notification-core"], + ["execution-event-capture", "notifications-api"], + ["fe-inbox"], + ["email-setup", "realtime-user-channel"], + ["fe-realtime-alerts"], + ["email-immediate"], + ["email-digest"], + ["notification-observability"] + ], + "walkingSkeleton": "notification-core", + "autonomyMode": "autonomous", + "skipMatrix": {} +} diff --git a/lambda/v2-orchestrator/test/fixtures/load-meta-before-completion.js b/lambda/v2-orchestrator/test/fixtures/load-meta-before-completion.js new file mode 100644 index 000000000..d34d6ea06 --- /dev/null +++ b/lambda/v2-orchestrator/test/fixtures/load-meta-before-completion.js @@ -0,0 +1,30 @@ +// Frozen load-meta body from main 4fd4558ae8b064d8d99efcbd905d38b0b853c27a, +// also used by the deployed pre-fix orchestrator. Do not modernize this fixture. +export const loadMetaBeforeCompletion = (store, executionId) => + store.getExecution(executionId, { consistentRead: true }); + +// Frozen finish-succeeded body from main c4b6afff3147be23c567492b3a5ea9e692af7e0f. +export const finishSucceededBeforeCompletion = async ({ + store, + executionId, + projectId, + meta, + runId, +}) => { + try { + await store.updateExecution({ + executionId, + projectId, + status: 'SUCCEEDED', + startedAt: meta.startedAt, + completedAt: new Date().toISOString(), + failureReason: null, + failure: null, + ...(runId ? { ifOrchestratorRunId: runId } : {}), + }); + return true; + } catch (e) { + if (e?.name === 'ConditionalCheckFailedException') return false; + throw e; + } +}; diff --git a/lambda/v2-orchestrator/test/gate-findings.test.js b/lambda/v2-orchestrator/test/gate-findings.test.js index 8f10a0309..a02179567 100644 --- a/lambda/v2-orchestrator/test/gate-findings.test.js +++ b/lambda/v2-orchestrator/test/gate-findings.test.js @@ -8,7 +8,7 @@ import { beforeEach, describe, expect, it, vi } from 'vitest'; import { Logger } from '@aws-lambda-powertools/logger'; -import { __durableHandler } from '../index.js'; +import { __durableHandler } from './fixtures/completion-handler.js'; const makeCtx = (over = {}) => { const stageCallbacks = new Map(); diff --git a/lambda/v2-orchestrator/test/lane-gate-resume.test.js b/lambda/v2-orchestrator/test/lane-gate-resume.test.js index cb61bf3ef..25fe1b9d7 100644 --- a/lambda/v2-orchestrator/test/lane-gate-resume.test.js +++ b/lambda/v2-orchestrator/test/lane-gate-resume.test.js @@ -1,5 +1,5 @@ import { describe, it, expect, vi } from 'vitest'; -import { __durableHandler } from '../index.js'; +import { __durableHandler } from './fixtures/completion-handler.js'; const makeCtx = () => { const stageCallbacks = new Map(); diff --git a/lambda/v2-orchestrator/test/learnings-ritual.test.js b/lambda/v2-orchestrator/test/learnings-ritual.test.js index d158fb8a5..dcf2405a7 100644 --- a/lambda/v2-orchestrator/test/learnings-ritual.test.js +++ b/lambda/v2-orchestrator/test/learnings-ritual.test.js @@ -7,7 +7,7 @@ // and a failure to record a learning never costs the run its approval. import { beforeEach, describe, expect, it, vi } from 'vitest'; -import { __durableHandler } from '../index.js'; +import { __durableHandler } from './fixtures/completion-handler.js'; import { buildEventRow } from '../../shared/v2-process-keys.js'; // Timeline fixtures go through the SAME row builder the process store persists diff --git a/lambda/v2-orchestrator/test/orchestrator-iam-policy.test.js b/lambda/v2-orchestrator/test/orchestrator-iam-policy.test.js new file mode 100644 index 000000000..ecc6c331f --- /dev/null +++ b/lambda/v2-orchestrator/test/orchestrator-iam-policy.test.js @@ -0,0 +1,25 @@ +import { readFile } from 'node:fs/promises'; +import { describe, expect, it } from 'vitest'; + +const terraformPath = new URL('../../../terraform/modules/api/lambda/main.tf', import.meta.url); + +describe('v2-orchestrator IAM role policy', () => { + it('grants ConditionCheckItem on the process table so ownership fences can transact-write', async () => { + // updateForRun, deleteWorkflowCheckpoint and finalizeUnitCompletion + // (lambda/shared/v2-process-store.js) fence the run and the unit plan with + // ConditionCheck entries inside TransactWriteItems. DynamoDB authorizes + // each one as dynamodb:ConditionCheckItem; without it the transaction is + // denied (not a conditional failure) and the intent ends FAILED. + const terraform = await readFile(terraformPath, 'utf8'); + const orchestratorPolicy = terraform.match( + /resource "aws_iam_role_policy" "v2_orchestrator" \{([\s\S]*?)^\}/m, + )?.[1]; + expect(orchestratorPolicy).toBeTruthy(); + + const processTableAction = orchestratorPolicy.match( + /# v2 process table[\s\S]*?Action\s*=\s*\[([^\]]*)\]/, + )?.[1]; + expect(processTableAction).toBeTruthy(); + expect(processTableAction).toContain('"dynamodb:ConditionCheckItem"'); + }); +}); diff --git a/lambda/v2-orchestrator/test/orchestrator-replay.test.js b/lambda/v2-orchestrator/test/orchestrator-replay.test.js index f873e2cd0..daf42f0c1 100644 --- a/lambda/v2-orchestrator/test/orchestrator-replay.test.js +++ b/lambda/v2-orchestrator/test/orchestrator-replay.test.js @@ -4,9 +4,13 @@ import { LocalDurableTestRunner, WaitingOperationStatus, } from '@aws/durable-execution-sdk-js-testing'; -import { __durableHandler } from '../index.js'; +import { __durableHandler } from './fixtures/completion-handler.js'; import { awaitEngineGate } from '../section.js'; import { awaitEngineGate as previousEngineGate } from './fixtures/engine-gate-before-recovery.js'; +import { + loadMetaBeforeCompletion, + finishSucceededBeforeCompletion, +} from './fixtures/load-meta-before-completion.js'; // --------------------------------------------------------------------------- // REAL replay coverage for the orchestrator's async stage flow (WP1). @@ -292,6 +296,135 @@ const completeStage = async (runner, opName, result) => { }; describe('orchestrator on the real durable runner (replay semantics)', () => { + it.each(['before', 'after'])( + 'replays a frozen marker-absent checkpoint %s terminal success', + async (boundary) => { + const world = makeWorld({ stages: [{ stageId: 'a' }] }); + let upgraded = false; + let finishCalls = 0; + const names = []; + const handler = withDurableExecution((event, ctx) => { + let loadedMeta; + let runId; + const wrapped = new Proxy(ctx, { + get: (target, key) => + key === 'step' + ? async (name, fn, options) => { + names.push(name); + if (name === 'open-pr' && boundary === 'after') { + const [pause] = await target.createCallback('test-after-legacy-success'); + await pause; + } + const result = await target.step( + name, + !upgraded && name === 'load-meta' + ? () => loadMetaBeforeCompletion(world.deps.store, event.executionId) + : !upgraded && name === 'finish-succeeded' + ? () => { + finishCalls++; + return finishSucceededBeforeCompletion({ + store: world.deps.store, + executionId: event.executionId, + projectId: loadedMeta.projectId, + meta: loadedMeta, + runId, + }); + } + : fn, + options, + ); + if (name === 'load-meta') loadedMeta = result; + if (name === 'mint-run-id') runId = result.runId; + return result; + } + : typeof target[key] === 'function' + ? target[key].bind(target) + : target[key], + }); + return __durableHandler(event, wrapped, world.deps); + }); + const runner = new LocalDurableTestRunner({ handlerFunction: handler }); + const completion = runner.run({ + payload: { action: 'start', intentId: 'i1', executionId: 'i1' }, + }); + const stage = await runner + .getOperation('stage-cb-a') + .waitForData(WaitingOperationStatus.STARTED); + if (boundary === 'before') upgraded = true; + await stage.sendCallbackSuccess(JSON.stringify({ ok: true, state: 'SUCCEEDED' })); + if (boundary === 'after') { + const pause = await runner + .getOperation('test-after-legacy-success') + .waitForData(WaitingOperationStatus.STARTED); + upgraded = true; + await pause.sendCallbackSuccess(JSON.stringify({ ok: true })); + } + const result = await completion; + expect(result.getResult()).toMatchObject({ ok: true }); + expect(names).not.toContain('finish-completion-v1'); + expect(world.statusWrites.filter((status) => status === 'SUCCEEDED')).toHaveLength(1); + expect(finishCalls).toBe(boundary === 'after' ? 1 : 0); + expect(world.statusWrites).not.toContain('FAILED'); + }, + ); + it('resumes a frozen pre-fix load-meta checkpoint without silently upgrading completion policy', async () => { + const world = makeWorld({ stages: [{ stageId: 'a' }] }); + let upgraded = false; + const loadedVersions = []; + let firstLoadCalls = 0; + let markSuspended; + const suspended = new Promise((resolve) => { + markSuspended = resolve; + }); + const handler = withDurableExecution((event, ctx) => { + const wrapped = new Proxy(ctx, { + get: (target, key) => + key === 'step' + ? async (name, fn, options) => { + const result = await target.step( + name, + !upgraded && name === 'load-meta' + ? async () => { + firstLoadCalls++; + return loadMetaBeforeCompletion(world.deps.store, event.executionId); + } + : fn, + options, + ); + if (name === 'load-meta') loadedVersions.push(result.completionPolicyVersion ?? 0); + return result; + } + : typeof target[key] === 'function' + ? target[key].bind(target) + : target[key], + }); + return __durableHandler(event, wrapped, world.deps); + }); + const runner = new LocalDurableTestRunner({ + handlerFunction: async (...input) => { + const result = await handler(...input); + if (result.Status === 'PENDING') { + upgraded = true; + markSuspended(); + } + return result; + }, + }); + const completion = runner.run({ + payload: { action: 'start', intentId: 'i1', executionId: 'i1' }, + }); + await suspended; + await completeStage(runner, 'stage-cb-a', { ok: true, state: 'SUCCEEDED' }); + const execution = await completion; + expect(execution.getResult()).toMatchObject({ ok: true }); + expect(execution.getInvocations().length).toBeGreaterThan(1); + expect(firstLoadCalls).toBe(1); + expect(new Set(loadedVersions)).toEqual(new Set([0])); + expect(world.invokes.filter((payload) => payload.command === 'run-stage-start')).toHaveLength( + 1, + ); + }); + it('resumes a stage gate after its unpark committed without a checkpoint', async () => { const world = makeWorld({ stages: [{ stageId: 'a', stageInstanceId: 'si-a' }] }); const meta = { ...META }; diff --git a/lambda/v2-orchestrator/test/orchestrator.test.js b/lambda/v2-orchestrator/test/orchestrator.test.js index 856d40533..6e9191ac6 100644 --- a/lambda/v2-orchestrator/test/orchestrator.test.js +++ b/lambda/v2-orchestrator/test/orchestrator.test.js @@ -1,5 +1,7 @@ import { describe, it, expect, beforeEach, vi } from 'vitest'; -import { __durableHandler } from '../index.js'; +import { __durableHandler } from './fixtures/completion-handler.js'; +import { promoteUnits } from '../../agentcore/commands/promote-units.js'; +import dependentSkeletonPlan from './fixtures/dependent-skeleton-plan.json'; // The orchestrator's control flow is driven through an injected `deps` bag and a // fake DurableContext — no real AWS/Neptune. This isolates the sequencing logic @@ -134,6 +136,7 @@ beforeEach(() => { putTrackerSync: vi.fn(async (args) => args), failRunningStageAttempt: vi.fn(async () => null), listUnits: vi.fn(async () => []), + updateUnitPlanDecisions: vi.fn(async () => ({})), getUnit: vi.fn(async () => null), }, loadPlan: vi.fn(async () => ({ @@ -1879,6 +1882,10 @@ describe('WP5 — parallel sections: lanes, skeleton, ladder, halt-and-ask', () // Complete the missing lane instance → rewind proceeds straight to bt. rows[planStageInstanceId('aidlc-v2@1', 'cg', 'billing', 1)] = doneRow; + deps.store.listUnits = vi.fn(async () => [ + { sectionIndex: 1, slug: 'auth', state: 'MERGED' }, + { sectionIndex: 1, slug: 'billing', state: 'MERGED' }, + ]); invokes.length = 0; ctx = makeCtx(); deps.invokeRuntime = makeRuntime(ctx, sectionScript); @@ -2090,6 +2097,9 @@ describe('PR per unit delivery', () => { ); expect(invokes.map((payload) => payload.command)).toContain('reconcile-lane'); expect(invokes.map((payload) => payload.command)).toContain('refresh-intent'); + expect(unitStates.find((row) => row.state === 'MERGED').fields).toMatchObject({ + integrationResults: [], + }); expect(invokes).toContainEqual( expect.objectContaining({ command: 'record-unit-pr', @@ -2116,6 +2126,16 @@ describe('PR per unit delivery', () => { ); }); + it('records the evidence attribute for an unchanged unit PR integration', async () => { + configure({ statusFor: async () => null }); + deps.unitPrProvider.compare.mockResolvedValue({ status: 'identical' }); + expect(await start()).toMatchObject({ ok: true }); + expect(unitStates.find((row) => row.state === 'MERGED').fields).toMatchObject({ + integrationResults: [], + }); + expect(deps.unitPrProvider.createDraft).not.toHaveBeenCalled(); + }); + it('parks one callback for a long unchanged PR wait without growing durable operations', async () => { const originalCreateCallback = ctx.createCallback; let resolvePrWait; @@ -2518,6 +2538,161 @@ describe('WP5 — engine-gate decisions', () => { const start = () => __durableHandler({ action: 'start', intentId: 'i1', executionId: 'i1' }, ctx, deps); + it('rejects the observed dependent skeleton before building, shared checks or final PR creation', async () => { + unitPlanRow = structuredClone(dependentSkeletonPlan); + const result = await start(); + expect(result).toMatchObject({ ok: false, reason: 'invalid_walking_skeleton' }); + expect(stageStarts().filter((payload) => payload.unitSlug)).toEqual([]); + expect(stageStarts().some((payload) => payload.stageId === 'bt')).toBe(false); + expect(deps.openPr).not.toHaveBeenCalled(); + expect( + deps.store.updateExecution.mock.calls.some(([input]) => input.status === 'SUCCEEDED'), + ).toBe(false); + expect(unitPlanRow.walkingSkeleton).toBe('notification-core'); + }); + + it.each([ + ['renamed-unit', false], + ['billing', false], + ['billing', true], + ])( + 'plain approval executes a regenerated DAG after old skeleton %s becomes invalid (ceremony off: %s)', + async (oldSkeleton, ceremonyOff) => { + unitPlanRow.walkingSkeleton = oldSkeleton; + if (ceremonyOff) { + const plan = SECTION_PLAN(); + plan.plan.stages + .filter((stage) => stage.parallelSection === 1) + .forEach((stage) => { + stage.policy = { skeleton: 'off' }; + }); + deps.loadPlan.mockResolvedValue(plan); + } + const graph = { + lookupArtifacts: async () => [ + { + id: 'regenerated', + content: + '```yaml\nunits:\n - name: auth\n depends_on: []\n - name: billing\n depends_on: [auth]\n```', + }, + ], + mirrorUnitDag: async () => ({}), + }; + deps.invokeRuntime = makeRuntime(ctx, async (payload) => { + if (payload.command !== 'promote-units') return sectionScript(payload); + return promoteUnits(payload, { + store: { + ...deps.store, + putUnitPlan: async (plan) => { + unitPlanRow = plan; + return plan; + }, + syncUnitRows: async () => ({ created: [], preserved: [], orphaned: [] }), + }, + openGraph: async () => ({}), + createWriter: () => graph, + }); + }); + const result = await start(); + expect(result.ok).toBe(true); + expect(unitPlanRow.walkingSkeleton).toBe('auth'); + if (ceremonyOff) + expect( + deps.store.getHumanTask.mock.calls.some(([, id]) => id.startsWith('eg-skeleton')), + ).toBe(false); + expect( + invokes + .filter((payload) => payload.command === 'merge-lane') + .map((payload) => payload.unitSlug), + ).toEqual(['auth', 'billing']); + expect( + deps.store.updateExecution.mock.calls.some(([input]) => input.status === 'SUCCEEDED'), + ).toBe(true); + }, + ); + + it('does not execute a different default when an approved fan-out decision cannot be saved', async () => { + deps.store.updateUnitPlanDecisions.mockRejectedValue(new Error('decision store unavailable')); + const result = await start(); + expect(result.ok).toBe(false); + expect(stageStarts().filter((payload) => payload.unitSlug)).toEqual([]); + expect(deps.openPr).not.toHaveBeenCalled(); + }); + + it('integrates all eleven units with the dependency-free control, preserving the recovered foundation', async () => { + unitPlanRow = { + ...structuredClone(dependentSkeletonPlan), + walkingSkeleton: 'notifications-data-infra', + }; + deps.store.listUnits.mockResolvedValue([ + { slug: 'notifications-data-infra', sectionIndex: 1, state: 'MERGED' }, + { slug: 'old-plan-unit', sectionIndex: 1, state: 'PENDING' }, + ]); + const result = await start(); + expect(result.ok).toBe(true); + const integrations = invokes.filter((payload) => payload.command === 'merge-lane'); + expect(integrations).toHaveLength(10); + expect(integrations.some((payload) => payload.unitSlug === 'notifications-data-infra')).toBe( + false, + ); + expect(stageStarts().some((payload) => payload.stageId === 'bt')).toBe(true); + expect( + deps.store.appendEvent.mock.calls + .map(([event]) => event) + .find((event) => event.type === 'v2.units.fan_in').summary, + ).toContain('11/11'); + }); + + it.each([false, true])( + 'counts disjoint halt outcomes and prevents unresolved completion (failed prerequisite: %s)', + async (failedPrerequisite) => { + unitPlanRow = { + units: [ + ...(failedPrerequisite ? [{ slug: 'prerequisite', dependsOn: [] }] : []), + { + slug: 'blocked', + dependsOn: [failedPrerequisite ? 'prerequisite' : 'missing-prerequisite'], + }, + ], + batches: [failedPrerequisite ? ['prerequisite', 'blocked'] : ['blocked']], + walkingSkeleton: 'blocked', + autonomyMode: 'autonomous', + }; + const plan = SECTION_PLAN(); + plan.plan.stages + .filter((stage) => stage.parallelSection === 1) + .forEach((stage) => { + stage.policy = { skeleton: 'off' }; + }); + deps.loadPlan.mockResolvedValue(plan); + const answered = gateReads({ 'eg-halt': { answer: { decision: 'abort' } } }); + deps.store.getHumanTask = vi.fn(async (executionId, id) => { + if ( + id.startsWith('eg-halt') && + !deps.store.createHumanTask.mock.calls.some(([gate]) => gate.humanTaskId === id) + ) + return null; + return answered(executionId, id); + }); + if (failedPrerequisite) + deps.invokeRuntime = makeRuntime(ctx, (payload) => + payload.stageId === 'cg' && payload.unitSlug === 'prerequisite' + ? { ok: false, state: 'FAILED', reason: 'sensor_blocked' } + : sectionScript(payload), + ); + const result = await start(); + expect(result).toMatchObject({ ok: false, reason: 'section_aborted' }); + expect(stageStarts().some((payload) => payload.stageId === 'bt')).toBe(false); + expect(deps.openPr).not.toHaveBeenCalled(); + const halt = deps.store.createHumanTask.mock.calls + .map(([gate]) => gate) + .find((gate) => gate.humanTaskId.startsWith('eg-halt')); + expect(halt.prompt).toContain( + `${failedPrerequisite ? 1 : 0} failed, 1 blocked, 0 integrated`, + ); + }, + ); + it('halt-and-ask RETRY re-runs the failed lane under fresh round identities and completes the run', async () => { let cgBillingAttempts = 0; deps.store.getHumanTask = gateReads({ 'eg-halt': { answer: { decision: 'retry' } } }); @@ -2630,6 +2805,19 @@ describe('WP5 — engine-gate decisions', () => { const eventCalls = deps.store.appendEvent.mock.calls.map((c) => c[0]); expect(eventCalls.some((e) => e.type === 'v2.units.lanes_skipped')).toBe(true); expect(eventCalls.find((e) => e.type === 'v2.units.fan_in')?.summary).toContain('1/2'); + expect( + unitStates.find((unit) => unit.slug === 'billing' && unit.fields?.completionSkip), + ).toMatchObject({ + fields: { + completionSkip: { + humanTaskId: expect.stringMatching(/^eg-halt/), + planIdentity: expect.any(String), + }, + }, + }); + expect(eventCalls.find((e) => e.type === 'v2.units.fan_in')?.summary).toContain( + 'explicitly omitted', + ); }); it('AUTONOMOUS wavefront: a failed lane BLOCKS its dependents while independents finish', async () => { @@ -2660,7 +2848,7 @@ describe('WP5 — engine-gate decisions', () => { // c never dispatched a stage — it blocked on b. expect(stageStarts().some((p) => p.unitSlug === 'c')).toBe(false); const cStates = unitStates.filter((u) => u.slug === 'c').map((u) => u.state); - expect(cStates).toEqual(['BLOCKED']); + expect([...new Set(cStates)]).toEqual(['BLOCKED']); expect(unitStates.find((u) => u.slug === 'c' && u.state === 'BLOCKED').fields).toMatchObject({ blockedOn: 'b', }); @@ -2862,8 +3050,8 @@ describe('WP5 — engine-gate decisions', () => { it('an uninterpretable halt-and-ask answer RE-ASKS; the follow-up choice is honored', async () => { deps.store.getHumanTask = gateReads({ - 'eg-halt-s1-r0-a1': { answer: { decision: 'skip' } }, - 'eg-halt-s1-r0': { answer: { decision: 'garbage' } }, + 'eg-halt-s1-w1-r0-a1': { answer: { decision: 'skip' } }, + 'eg-halt-s1-w1-r0': { answer: { decision: 'garbage' } }, }); deps.invokeRuntime = makeRuntime(ctx, (payload) => { if (payload.command === 'init-ws') return { ok: true }; @@ -3026,6 +3214,21 @@ describe('WP6 — PR opened on SUCCEEDED (intent-pr)', () => { __durableHandler({ action: 'start', intentId: 'i1', executionId: 'i1' }, ctx, deps); const events = () => deps.store.appendEvent.mock.calls.map((c) => c[0]); + it('counts only current-plan units in the final PR summary, retaining older rows', async () => { + const rows = [ + { slug: 'auth', state: 'MERGED' }, + { slug: 'billing', state: 'MERGED' }, + { slug: 'retired-plan-unit', state: 'PENDING' }, + ]; + deps.store.listUnits.mockResolvedValue(rows); + deps.store.getUnitPlan = vi.fn(async () => UNIT_PLAN()); + expect((await start()).ok).toBe(true); + const body = deps.openPr.mock.calls[0][0].body; + expect(body).toContain('2 total, 2 integrated into the intent branch'); + expect(body).not.toContain('retired-plan-unit'); + expect(rows).toHaveLength(3); + }); + it('opens one PR per repo from the intent branch with execution provenance in the body', async () => { deps.store.getExecution = vi.fn(async () => ({ ...META, @@ -3063,7 +3266,7 @@ describe('WP6 — PR opened on SUCCEEDED (intent-pr)', () => { 'created by [AI-DLC](https://aidlc.example.test/space/p1/intent/i1)', ); expect(deps.openPr.mock.calls[0][0].body).toContain('strategy: intent-pr'); - expect(deps.openPr.mock.calls[0][0].body).toContain('2 total, 2 merged'); + expect(deps.openPr.mock.calls[0][0].body).toContain('2 total, 2 integrated'); const opened = events().filter((e) => e.type === 'v2.pr.opened'); expect(opened).toHaveLength(2); expect(opened[0].summary).toContain('https://github.com/o/r/pull/7'); @@ -3265,9 +3468,9 @@ describe('WP6 — PR opened on SUCCEEDED (intent-pr)', () => { deps.openPr = vi.fn(async () => ({ prUrl: 'https://x/pr/1', prNumber: 1 })); await start(); const body = deps.openPr.mock.calls[0][0].body; - expect(body).toContain('3 total, 1 merged'); - expect(body).toContain('unit `billing` NOT merged (FAILED: merge-lane: markers_remain)'); - expect(body).toContain('unit `checkout` NOT merged (BLOCKED)'); + expect(body).toContain('3 total, 1 integrated'); + expect(body).toContain('unit `billing` NOT integrated (FAILED: merge-lane: markers_remain)'); + expect(body).toContain('unit `checkout` NOT integrated (BLOCKED)'); }); it('an identical branch records v2.pr.skipped without calling the provider create API', async () => { diff --git a/lambda/v2-orchestrator/test/release-coexistence-simulation.test.js b/lambda/v2-orchestrator/test/release-coexistence-simulation.test.js index 239927928..3e30714c9 100644 --- a/lambda/v2-orchestrator/test/release-coexistence-simulation.test.js +++ b/lambda/v2-orchestrator/test/release-coexistence-simulation.test.js @@ -19,7 +19,7 @@ import { methodologyReleasePinFromManifest, } from '../../shared/release-resolver.js'; import { loadExecutionPlan } from '../../shared/v2-workflow-plan.js'; -import { __durableHandler } from '../index.js'; +import { __durableHandler } from './fixtures/completion-handler.js'; vi.spyOn(Logger.prototype, 'info').mockImplementation(() => {}); diff --git a/lambda/v2-orchestrator/test/section-integration.test.js b/lambda/v2-orchestrator/test/section-integration.test.js index c130e249e..9833e49b5 100644 --- a/lambda/v2-orchestrator/test/section-integration.test.js +++ b/lambda/v2-orchestrator/test/section-integration.test.js @@ -7,7 +7,7 @@ import { LocalDurableTestRunner, WaitingOperationStatus, } from '@aws/durable-execution-sdk-js-testing'; -import { __durableHandler } from '../index.js'; +import { __durableHandler } from './fixtures/completion-handler.js'; import { initLane, mergeLane } from '../../agentcore/commands/lane.js'; import { resolveConflict } from '../../agentcore/commands/resolve-conflict.js'; import { runGit, commitAndPushAll, fetchOrigin } from '../../agentcore/git-engine.js'; @@ -295,7 +295,9 @@ const makeWorld = ({ remote, unitPlan, fileFor, beforeStage = null, conflictAgen // wait for the row to appear, flip it, complete its durable callback. const answerEngineGate = async (world, prefix, patch = { status: 'answered' }) => { let id; - for (let i = 0; i < 400 && !id; i++) { + // Real Git and durable callbacks can exceed 10s under concurrent test load. + // Keep the wait bounded within the existing 120s test deadline. + for (let i = 0; i < 1200 && !id; i++) { id = [...world.gates.keys()].find((k) => k.startsWith(prefix)); if (!id) await new Promise((r) => setTimeout(r, 25)); } @@ -455,7 +457,7 @@ describe('WP8 fixture 2 — forced merge conflict → halt-and-ask SKIP (real gi : { name: 'shared.txt', content: `${slug} version\n` }, beforeStage: async (payload, w) => { if (payload.unitSlug !== 'c') return; - for (let i = 0; i < 400 && !w.unitStates.includes('b:MERGED'); i++) { + for (let i = 0; i < 1200 && !w.unitStates.includes('b:MERGED'); i++) { await new Promise((r) => setTimeout(r, 25)); } }, @@ -492,6 +494,7 @@ describe('WP8 fixture 2 — forced merge conflict → halt-and-ask SKIP (real gi 'c:RUNNING', 'c:MERGING', 'c:FAILED', + 'c:FAILED', // Human omission receipt retains the original failure state. ]); const cFailed = world.events.find((e) => e.type === 'v2.unit.failed' && e.unitSlug === 'c'); expect(cFailed?.summary).toContain('markers_remain'); @@ -558,7 +561,7 @@ describe('WP6 fixture 3 — conflict RESOLVED by the resolution stage (real git) : { name: 'shared.txt', content: `${slug} version\n` }, beforeStage: async (payload, w) => { if (payload.unitSlug !== 'c') return; - for (let i = 0; i < 400 && !w.unitStates.includes('b:MERGED'); i++) { + for (let i = 0; i < 1200 && !w.unitStates.includes('b:MERGED'); i++) { await new Promise((r) => setTimeout(r, 25)); } }, diff --git a/lambda/v2-orchestrator/test/skeleton-switch.test.js b/lambda/v2-orchestrator/test/skeleton-switch.test.js index 179e104d7..5878f5ec9 100644 --- a/lambda/v2-orchestrator/test/skeleton-switch.test.js +++ b/lambda/v2-orchestrator/test/skeleton-switch.test.js @@ -17,7 +17,7 @@ import { LocalDurableTestRunner, WaitingOperationStatus, } from '@aws/durable-execution-sdk-js-testing'; -import { __durableHandler } from '../index.js'; +import { __durableHandler } from './fixtures/completion-handler.js'; vi.setConfig({ testTimeout: 60_000 }); diff --git a/lambda/v2-orchestrator/test/stage-loopback.test.js b/lambda/v2-orchestrator/test/stage-loopback.test.js index ff86c68ef..215b24d5e 100644 --- a/lambda/v2-orchestrator/test/stage-loopback.test.js +++ b/lambda/v2-orchestrator/test/stage-loopback.test.js @@ -6,7 +6,7 @@ // outside release mode, and the recommendation behavior at the cap. import { beforeEach, describe, expect, it, vi } from 'vitest'; -import { __durableHandler } from '../index.js'; +import { __durableHandler } from './fixtures/completion-handler.js'; import { LOOP_BACK_RECORDED_EVENT } from '../../shared/stage-loopback.js'; import { buildStageRow } from '../../shared/v2-process-keys.js'; diff --git a/lambda/v2-orchestrator/test/stage-skip.test.js b/lambda/v2-orchestrator/test/stage-skip.test.js index ee90dd4b1..710c87090 100644 --- a/lambda/v2-orchestrator/test/stage-skip.test.js +++ b/lambda/v2-orchestrator/test/stage-skip.test.js @@ -1,5 +1,5 @@ import { describe, it, expect, beforeEach, vi } from 'vitest'; -import { __durableHandler } from '../index.js'; +import { __durableHandler } from './fixtures/completion-handler.js'; // Stage-skipping behaviors of the durable orchestrator (shared/stage-skip.js): // - intent-level skips (META.skipStageIds) ride load-plan, get SKIPPED audit diff --git a/terraform/modules/api/lambda/main.tf b/terraform/modules/api/lambda/main.tf index 585179ea3..f2d5ad9bb 100644 --- a/terraform/modules/api/lambda/main.tf +++ b/terraform/modules/api/lambda/main.tf @@ -3368,8 +3368,10 @@ resource "aws_iam_role_policy" "v2_orchestrator" { }, { # v2 process table: drive execution + stage + gate state. + # ConditionCheckItem: run-ownership and unit-plan fences are + # ConditionCheck entries inside TransactWriteItems. Effect = "Allow" - Action = ["dynamodb:GetItem", "dynamodb:PutItem", "dynamodb:UpdateItem", "dynamodb:Query"] + Action = ["dynamodb:GetItem", "dynamodb:PutItem", "dynamodb:UpdateItem", "dynamodb:Query", "dynamodb:ConditionCheckItem"] Resource = [ var.v2_executions_table_arn, "${var.v2_executions_table_arn}/index/*",