feat(core): hand a long Code Mode build to a durable job that reaches its turn as it ends

[nickruigrok]

What and why

GRA-429, split from GRA-349. A Code Mode build that takes longer than its run's 30 s budget now returns a durable job ID. The job keeps going after the transient Worker and its lease end, and how it ended reaches the turn that submitted it, never another turn and never a closed one.

Security-relevant (leases and authority): needs a person's review before merge. The threat model is written out in build-jobs.ts and in the design comment on GRA-429.

Mechanism: no new framework. The job is a second internal workflow (build) on the existing engine and dispatcher, next to upload extraction. Its record is a new chat_build_jobs table in the Workspace DO, next to chat_code_runs. The migration only adds that table.

API

• env.build.submit(app, { kind: "check" } | { kind: "propose", message }) returns { job, status, revision }.
  • It is admitted against the run's lease, inside the DO's own step.
  • It binds the job to the draft's revision, the run's turn and the run's grant epoch (when the run opened).
  • A chat can have at most 3 jobs running at once.
  • Each job takes one of the draft's checks this turn, as an in-call check or propose does (takeCheck). It settles that check against the turn's counter when it ends, and its result then carries the real failedInARow. So jobs can't get the agent past maxFailedChecks. If the turn has already ended when the job finishes, nothing is settled: the counter belonged to that turn, and the next question resets it anyway.
  • A running record whose instance is errored, terminated, or complete without its last step, or that has no instance a minute after it was submitted, is failed with a generic error. Both submit and job do this, which frees its place and settles its check. A failed status read counts as unknown, not dead, so one transient engine error doesn't fail the call.
  • A chat keeps the records of its latest 100 jobs; running ones are never pruned. They aren't counted in the chat's transcript limit, which bounds what the model reads; the row count and the bounded results bound what they hold.
  • Internal instances (build and extraction) get the deployment's run retention when created.
• env.build.job(id) returns the job's status, its result (what check or propose returns), its receipt (the version proposed) and any error.
  • It answers only for jobs of the same chat, and only while the person still builds the job's App (appFor(..., "builder")). A later execution, with its own new ID, follows the job; the module is never replayed.
  • It also records a receipt for an ended job that committed before it ended.

Completion

• When the job ends, the Workspace DO pushes a system message (no result data, only IDs and the outcome) into the queue of the turn that submitted it.
• Pi's getSteeringMessages and getFollowUpMessages drain that queue between steps, never between a code step and its result.
• The queue is keyed by a random per-turn ID and dropped when the turn ends. A closed turn, a later turn, or a turn after a DO restart never gets the message; the result stays on the job record.

Authority

• Every step reloads the chat's agent and person and checks again what an in-call build checks: the agent's write on Apps, the person's membership and builder role, and restricted mode.
• Every step, and the commit right before it lands (through proposeDraft's admit hook), reads revocations since the grant epoch. Finding one ends the job as revoked.
• finish admits and authorizes a completed job once more: grant, membership, builder role and restricted mode. So a job revoked at the very end, or whose person lost the App meanwhile, hands on no result.
• Cancelling the turn ends that turn's jobs, and deleting the chat ends all of its jobs; the instances are terminated too.
• Once its turn has ended, a job can be stopped only by deleting the chat, short of a revocation. I chose not to add a cancel-by-job-ID: it would be more surface for little gain, since a job is bounded by its steps and its draft's revision and changes nothing live. This is recorded on GRA-429.
• If submit can't start the instance, the job stores a generic internal.unexpected error; the engine's own message goes only to the log.

Idempotency

• Steps are journaled by the engine.
• The commit step records the tree it is about to commit. If the step runs again after the commit landed, it finds the version parent + 1 with that tree, proposed from this chat, and takes it as its receipt instead of committing again.
• A job's commit records the job's ID in proposedBy, and that version is adopted as the receipt only if it carries this job's ID. So a commit of an identical tree from the same chat is never recorded as this job's.
• That lookup, and recording the receipt, happen before the step asks to be admitted. So a commit that landed keeps its version even when the retry is then refused and the job ends revoked or cancelled.
• finish ends only a running job.

Out of scope, recorded on the issue

• resolveDependencies stays in-call: its effect is already admitted right before it commits (feat(core): record every Code Mode run, and close its lease on revocation #389).
• On plain workerd there is still no Workflows engine. This has the same on-prem limitation as uploads, noted in engine.ts.

How it was tested

apps/core/test/build-jobs.test.ts has 17 tests, in workerd against the real engine, DO, model loop and D1. Only the model provider is faked. The tests hold a job inside its check step by holding its screen-build cache read on env.FILES, so each scenario is deterministic:

• Outlives its run. The run is recorded completed while the job runs. The job then completes with version 1. The system message lands right after the answer the turn gave meanwhile, and the model's last request carries it.
• Closed and later turns are never told; the later turn makes exactly one model request.
• Revocation mid-job. Another of the agent's permissions is revoked while the job checks. The job ends revoked with no result and no version, and the refusal is audited.
• Cancel. Stopping the turn ends the job as cancelled, and nothing is committed.
• Restart mid-commit. restart({ from: "commit" }) after the commit landed gives one version and one app.committed event.
• Revoked before the commit runs again. The same restart, with a revocation between the two attempts: the job ends revoked with version: 1.
• Repair loop. Five failing job checks in one turn report failedInARow 1 to 5. After that, a sixth submit (check or propose) and an in-call check are all refused with app.checks_exhausted.
• Cap and slot freeing. Four concurrent submits start 3 jobs and refuse 1 with agent.jobs_exhausted. Terminating one job's instance behind its record's back fails that job (internal.unexpected), and a fifth submit then starts.
• Builder role lost mid-job. The job's result is withheld: failed, role.forbidden, result: null.
• Restricted mode mid-job. The chat turns restricted while the job checks: no version, and the job fails with permission.restricted.
• Pruning. With one job still running and 105 ended, the chat keeps the newest 100 ended records and the running one, which is the oldest. Ties within the same millisecond are ordered by when each record was written.
• job() after losing the App. Once the person no longer builds the App, job() refuses with role.forbidden, and the job record is kept.
• Rights lost while the files are stored. The test holds proposeDraft's tree write, then the person loses the builder role (or the chat turns restricted). The job fails with role.forbidden (or permission.restricted), and no version is created. The commit's admit callback now rechecks the person's build rights and restricted mode, and the draft's revision, right before the commit.
• Draft written meanwhile. A write to the draft while a check job checks fails it with app.conflict and no result. A write while a proposal stores its files fails it with app.conflict and no version.
• Another chat. env.build.job(id) from another chat of the same person and agent reads agent.job_not_found.

Mutation checks. Each mutation was applied on its own and reverted; each failed the test named:

Mutation	Test that fails
Deliver to any running turn	Closed and later turns
Drop the event	Outlives its run
Unwire Pi's steering and follow-up	Outlives its run
Remove the admissions before the commit	Revocation mid-job
Cancelling a turn leaves its jobs	Cancel
Take no check when opening a job	Repair loop
Move receipt reconciliation behind admission (removed from before it)	Restart mid-commit, and Revoked before the commit runs again
Never fail dead jobs	Cap and slot freeing
No authorization in finish	Builder role lost mid-job
No restricted check	Restricted mode mid-job
Never prune	Pruning
Prune running jobs too	Pruning
job() without the builder check	job() after losing the App
Commit callback without the authorization recheck	Both rights-lost-while-storing tests
Commit callback without the draft recheck	Draft written while the files are stored
No draft recheck after a check job's check	Draft written while a check job checks

Regression. agent.test.ts, agent-builds.test.ts, chats.test.ts, agent-instructions.test.ts and uploads.test.ts pass: 181 tests, 3 workers. vp check and vp run check:migrations pass. The branch is rebased on current main.

Checklist

• vp check and targeted vp test pass
• Schema changes are expand-only (no drops or renames in the same release)
• New external calls go through connect; new model calls through the model gateway (none added)
• No secrets, client names or client configuration

[nickruigrok]

@greptileai review

[greptile-apps]

Confidence Score: 5/5

[Critical impact] The reviewed changes appear safe to merge; both previous findings are addressed.

Summary

This PR lets a chat hand a long check or proposal to a durable build job. The job keeps its result and tells only the turn that submitted it.

• Code Mode can hand slow checks and proposals to durable build jobs.
• Each chat keeps a durable record of its build jobs.
• Only the submitting turn hears when its build job ends.

Diagram

sequenceDiagram
  participant Run as Code run
  participant Workspace as Workspace
  participant Job as Build workflow
  participant App as App versions
  participant Turn as Submitting turn
  Run->>Workspace: Open job against run lease
  Run->>Job: Start durable workflow
  Job->>Workspace: Read authority and draft revision
  Job->>Job: Check draft files
  Job->>Workspace: Recheck draft revision
  opt Passing proposal without receipt
    Job->>App: Store files
    Job->>Workspace: Recheck rights, draft, and admission
    Job->>App: Commit version
    Job->>Workspace: Record receipt
  end
  Job->>Workspace: Authorize completed result and end job
  opt Submitting turn still open
    Workspace->>Turn: Queue job outcome
  end

Loading

_{Reviews (2) · Last reviewed commit: "fix(core): recheck a build job's rights ..." · Reviewed by Greptile}

[nickruigrok]

@greptileai review