[mattdevdba]

Issue #, if available: Fixes #637

Description of changes:

Problem

The orchestrator runs init-ws as one synchronous durable step, so the clone, intent-branch publish, Intent anchor and state seed all have to finish inside one orchestrator Lambda invocation. A large repository goes past the 900-second function timeout, and the durable retry runs init-ws again against a half-written checkout. Separately, the intent page shows a claimed run that is still CREATED after 2 minutes as "Run stalled — never started", offering a restart that races the clone. Details are in #637.

Solution

• init-ws-start (AgentCore): runs the existing init-ws logic as a background job. It establishes the callback heartbeat before cloning, keeps heartbeating while the clone runs, and completes the callback with init-ws's own { ok, reason, detail } result, including when init-ws throws. A duplicate dispatch of the same attempt is acknowledged without a second clone, and a different attempt is refused while a clone is still writing the workspace. Registered in agent-command-registry.js and http-server.js.
• Orchestrator: initWorkspace replaces the inline init-ws step. New runs run init-ws-probe, createCallback('init-ws-cb'), then init-ws-dispatch, mirroring runStage. The callback has a 2-hour timeout and a 15-minute heartbeat timeout, and every failure returns the same { ok: false, reason } shape, so the caller's init_ws_failed path is unchanged.
• Replay compatibility: the mode is an initWsMode field in the memoized mint-run-id result. A run that checkpointed before this change has no such field and replays exactly the original init-ws step. No existing operation is renamed or reordered.
• Older environment revisions: a published revision pins the AgentCore image it was built from. The probe sends init-ws-start without a callbackId, so it starts no job. A current image refuses it with missing_callback_id on HTTP 200. An older image answers unknown command (HTTP 400 through AgentCore), and the run keeps the synchronous step before any callback exists. A probe transport failure is retried like any durable step, not taken as an older image.
• Intent page: mapIntent returns orchestratorStartedAt. IntentView keeps the 2-minute stall threshold for runs no orchestrator has claimed. For claimed runs it waits until the init callback window has passed (2 h 10 min).
• Docs: docs/concepts/execution.md describes the callback, its limits and the fallbacks.

Testing

• New init-ws-start.test.js covers the probe refusal, the accept while the clone runs, heartbeat set-up and continuation, failure delivery (including a throwing init-ws), duplicate and conflicting dispatches, the busy tracker, and routing through dispatchInvocation (including the 400 an image without the handler returns).
• Orchestrator tests cover the callback path, a probe transport failure, a refused dispatch, heartbeat expiry, a malformed callback body, a failure delivered through the callback, the durable operations recorded for a new run, and replay of a run claimed before initWsMode existed (orchestrator.test.js, orchestrator-replay.test.js, section-integration.test.js, skeleton-switch.test.js).
• IntentView.test.tsx covers a claimed run that is still initializing, an unclaimed hand-off after 2 minutes, and a claimed run after the callback window.
• npx vitest run --project=agentcore --project=v2-orchestrator --project=intents --project=shared on Node 24.22.1: 4,727 tests pass. Two timing tests in files this PR doesn't touch (resolve-conflict.test.js, bedrock-iam-shell.test.js) timed out under the full parallel run and pass when run alone.
• npm --prefix frontend test: 976 pass. EnvironmentsTab.test.tsx failed once under load and passes alone (28/28); this PR doesn't touch it. npm --prefix frontend run build passes.
• npm run lint, format:check, dep:check and secretlint pass.
• The same change runs as a patch in our own deployment with repositories that take longer than the Lambda timeout to clone.

Impact

• New runs initialize workspaces through a callback, so a clone is no longer bounded by the orchestrator's 900-second timeout.
• Runs already in flight at deployment replay the original step. Environment revisions built before this change keep the synchronous step until they are rebuilt.
• There are no Terraform changes. The orchestrator and the AgentCore image can be deployed in either order.
• The intent API response gains orchestratorStartedAt.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[eipasteur]

[P2] Concurrent dispatches can start two init jobs — lambda/agentcore/commands/init-ws-start.js:63–90

Two dispatches arriving while the first heartbeat is pending both see an empty activeJobs entry and launch initWs against the same workspace. I reproduced this with identical and different callback IDs: both requests are accepted and two jobs start. The second job also overwrites the tracked entry, which the first job’s cleanup can remove while the second is still running.

I think this is worth addressing before merge, since it allows concurrent clones and branch operations in the same workspace. One approach could be to reserve the entry before awaiting the heartbeat, let identical callbacks share the acceptance result, and make cleanup conditional on the job still owning that entry.

A test with overlapping initial heartbeats would help cover this case; the current tests wait for the first acceptance before sending the second request.

[mattdevdba]

Thanks @eipasteur, good catch. Addressed in d148021, along the lines you suggested:

• The job is now reserved in activeJobs before the initial heartbeat is awaited, so an overlapping dispatch always sees it.
• A duplicate for the same callback awaits and shares the first dispatch's acceptance result (including a failed initial heartbeat); a different callback is refused with init_job_already_running.
• Cleanup (both the failed-heartbeat path and the job's finally) only releases the entry if this job still owns it.

New tests cover overlapping initial heartbeats with identical and different callback IDs, a failed initial heartbeat shared with the duplicate, and cleanup not removing another job's entry. Three of them fail against the previous implementation. The agentcore suite, npm run lint and npm run format:check pass.

[jeromevdl]

[P2] A dispatch retry after the init job finished is treated as a new job: lambda/agentcore/commands/init-ws-start.js (finally → release)

d148021 covers duplicates that overlap the initial heartbeat or the running job, but the activeJobs entry is released when the job completes. init-ws-dispatch uses the default step retry, which retries every error. If the dispatch call throws after the container already accepted it (lost response, SDK timeout) and the job finishes before the retry arrives, the retry finds no entry and starts over. It then sends an initial heartbeat to a callback that is already completed:

• If the service rejects that heartbeat, the dispatch returns init_ws_callback_heartbeat_failed and initWorkspace fails the run with init_ws_failed, although init succeeded.
• If the service accepts it, a second initWs runs in the workspace while stage 1 starts.

This is unlikely (it needs a lost response plus a fast init), and run-stage-start has the same gap. Still, both outcomes are wrong. Suggestion: keep completed callback ids for a bounded time and answer a same-callback retry with alreadyCompleted instead of starting a new job. Regression test: complete a job, dispatch the same callbackId again, and assert that initWs ran once and the response is a success.

Nit: the delivery-failure and heartbeat-failure logs don't include callbackId. It can be recovered from the durable execution history, but having it in the log makes the "complete it manually" comment practical.

[github-actions]

Complexity and test coverage

Automated quality report using code complexity and test coverage to highlight potential hot spots in this PR.
Review 🟣 and 🔴 items first: consider simplifying the code, improving test coverage, or both. This report is advisory.

	File / function	Complexity	Δ	Branch coverage	Δ	Uncovered branches	Depending files
🟠	lambda/v2-orchestrator/index.js:387 — handler	345 → 346	+1	86.5% → 86.6%	+0.1	44 → 45	0
🟠	frontend/src/pages/IntentView.tsx:109 — IntentView	78 → 79	+1	89.7% → 89.8%	+0.1	19 → 19	n/a

🟢 Improving · 🟠 Needs attention · 🔴 High risk · 🟣 Highest risk

— = unavailable · n/a = not applicable · * = incomplete tests

CI run and full report artifacts

[mattdevdba]

Thanks @jeromevdl, agreed. Addressed in 534b1fe: a finished job's callback id is kept for an hour, and a same-callback dispatch retry is answered { ok: true, accepted: true, alreadyCompleted: true }, with no second initWs and no heartbeat to the completed callback. The orchestrator then reads the already-delivered verdict. A different callback still starts a new job. Regression test: complete a job, redispatch the same callbackId, and assert one initWs and a success; it fails against d148021. The log records now include callbackId. I've left run-stage-start's matching gap out of this PR to keep it focused and can follow up separately.

I also rebased onto main: the tests from #534 needed the init-ws-start container behaviour, which caused the CI failure.