Skip to content

feat(core): record every Code Mode run, and close its lease on revocation - #389

Merged
nickruigrok merged 5 commits into
mainfrom
feat/code-mode-execution-leases
Oct 11, 2026
Merged

nickruigrok merged 5 commits into
mainfrom
feat/code-mode-execution-leases

Conversation

@nickruigrok

@nickruigrok nickruigrok commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Third PR of the GRA-349 stack, on top of #387. Security-relevant: it changes how long the agent's code holds authority. It needs a person's review before merge.

Threat model (written before the code)

Each row is a way stale or stolen authority could act, and what stops it now.

# Failure or attack What stops it Test
T1 Code keeps running after its run ended (cancelled, past the wall clock, or detached work after it returned) and calls an API The lease closes as the run ends; every call is counted and checked against it first (requireOpenRun) existing cancel test; new wall-clock write test
T2 A call admitted while the lease was open writes after the lease closed (a slow write that races the end of the run) Admitted again when the write commits. Draft writes, discards and App creates check the lease inside the Workspace object's write transaction. Commits elsewhere (propose, permission and dependency requests, connection requests and calls, USER.md saves) call admitEffect right before the effect. DO-level fence test
T3 A permission of the agent is revoked while code is under way Every call and every write admission checks for any revocation of the agent's permissions since the run opened (one indexed D1 read). The first one that finds it closes the lease as revoked, and the run ends revoked whatever its code returned. Its result isn't handed to the model, since it may rest on what was revoked. revocation test
T4 A core restart mid-run leaves a usable lease, or the module runs again Leases live only in memory. After a restart the run is unknown and answers ended. The record is marked interrupted on wake. Turns are never replayed, so a follow-up is a new run with a new ID. restart test (the object is aborted with ctx.abort)
T5 Forged or guessed run IDs, or one chat's run reaching another chat The run ID is a random UUID in props the code can't read or change, and leases are keyed <chat>/<run> existing env/import tests; unknown-run case
T6 Flooding the audit log by calling after the end One refusal is audited per run. The call that finds a revocation audits it, and later calls don't. existing 100-call test; revocation test
T7 Erasing what already happened Effects admitted before the close keep their receipts (pending actions, proposed versions). Nothing is rolled back or described as undone. (by construction)

Accepted residual: revoking any permission of the org agent ends every run of that agent under way, not only runs that used it. A run lasts at most 30 s, so this errs safe and costs little. It follows the posture of allowing and auditing, without per-permission bookkeeping.

What changes

  • chat_code_runs, a new Workspace DO SQLite table. It is the chat's existing persistence, not a separate Code Mode store. Each row holds:
    • the run ID, tool call ID, SHA-256 and length of the module (never the module itself);
    • agent (@earendil-works/pi-agent-core@<exact version>, imported from the package; pi-ai is pinned to the same version in the catalog) and the compatibility date;
    • runtime and cpu_limit;
    • status (running, then the terminal status, or interrupted) and its times.
    • Rows are deleted with the chat. The migration is 0008_chat_code_runs; it is new, so this is a hard cut.
  • D2. Only RUNTIME_PROFILE=cloudflare records enforced. Anything else, including unset or a typo, records cpu_limit_unsupported. Tests, vp run dev and the on-prem smoke config set workerd. The CPU proof is GRA-382's. Follow-up: the console must set RUNTIME_PROFILE=cloudflare on Cloudflare deployments. Until it does, their records say cpu_limit_unsupported, which errs safe.
  • Leases. They now carry openedAt. New revokeCodeRun and admitFromCodeRun close a lease and check one without counting a call. Draft writes and discards now name their run: saveDraft(..., runId) and dropDraft(..., { runId }). A proposal's follow-through drop goes by { revision }, because its commit was already admitted. takeCreate(..., runId) also names its run.
  • New error code agent.run_revoked.

Evidence

New tests in apps/core/test/agent.test.ts ("chat agent code run leases"), through the real Workspace DO, Worker Loader, Pi loop and D1:

  • The record: call ID, source hash and length, the exact Pi version, compatibility date, workerd with cpu_limit_unsupported, and status completed.
  • The DO fence:
    • a write while the run is open is admitted;
    • after closeCodeRun (wall_expired), a write and a discard are refused with agent.run_ended;
    • after revokeCodeRun, a write and a create are refused with agent.run_revoked;
    • an unknown run is refused;
    • the revoked run's close returns revoked;
    • the draft holds only the admitted write, and the records read wall_expired and revoked.
  • Revocation mid-run: the code waits, an admin revokes the agent's grant, and the next call is refused. The tool result is Error (revoked), the record says revoked, and exactly one agent.run_revoked audit refusal is written.
  • Restart mid-run: ctx.abort("restart") while the code waits. The turn fails, and on wake the record says interrupted and the run's ID answers ended. 6 s later there is still one tool call, no result, one run record and one model request: nothing ran again.

In apps/core/test/agent-builds.test.ts: code that waits 2 s past its 1 s wall-clock limit and then writes the draft is refused once (agent.run_ended), and the draft stays empty.

Commands:

  • vp check: passes.
  • vp test run test/agent.test.ts test/agent-connections.test.ts test/chat-connections.test.ts test/agent-instructions.test.ts test/agent-knowledge.test.ts test/chats.test.ts test/memory.test.ts test/run-fixes.test.ts test/packages.test.ts test/dependencies.test.ts: 184 passed.
  • vp test run -c vite.screens.config.ts test/agent-builds.test.ts test/preview-repairs.test.ts test/app-preview.test.ts test/agent-apps.test.ts: 46 passed.

Security review fixes (cc14ff5)

M1. A revocation with no later call went unnoticed. As a run ends, #endCodeRun now reads revocations once more. If it finds one, it closes the run as revoked and records a single agent.run_revoked (method: executeCode).

M2. A revoked run's logs still reached the model. A revoked run now hands on neither its result nor its logs. In #390, output that no watcher has had yet is dropped when the lease is revoked.

M3. Any revoked row of the org agent ended every run. revokedSince now counts only revoked permissions with granted_at IS NOT NULL AND (chat_id IS NULL OR chat_id = <this chat>). A declined request that was never granted, or a revocation in another chat, leaves the run alone. The comparison is revoked_at >= opened_at.

  • I added no skew margin. With a 2 s margin, a run opened just after an unrelated revoke ended as revoked; the existing held-outcome connection test showed it. Both timestamps come from Workers clocks.

S1. saveDraft, dropDraft({ runId }) and takeCreate now run admitEffect first, and keep the lease check inside their own transaction.

S2. admitEffect takes three steps in this order:

  • a cheap lease read, to get openedAt and fail fast;
  • the D1 revocation read;
  • the lease check, last, right before the effect.

S3.

  • connections.call and memory.saveUser record their admitEffect refusals with admitEffect(env, scope, method). Calls wrapped in auditedCall are recorded by it.
  • A refusal is recorded only by the call that closed the lease (revocationOf returns closed or closedBefore), which fixes the double audit.
  • Remaining case: a call admitted at its start, then refused at commit because another call had already revoked the lease, is recorded by auditedCall too.
  • The admitEffect in memory.saveUser stays. It is the commit-time check, which requireOpenRun at call start doesn't cover.

S4. New tests:

  • M1: the code calls once and waits, an admin revokes, and the code returns without another call. The result and logs are withheld, the status is revoked, and one refusal is recorded.
  • M3: a declined, never-granted request and another chat's revocation, stored mid-run, leave the run completed.
  • admitEffect end to end, using a slow outside dependency: every read of core's files is delayed 2 s against a 1 s run limit.
    • A proposal whose run expired while it was checked commits nothing, and build.propose is refused as agent.run_ended.
    • A draft write let in at once outlives its run while it reads the App's files, and is refused at commit.
  • Mutation checks:
    • an admitEffect that does nothing fails the proposal test;
    • dropping the granted and chat filters fails the M3 test;
    • dropping the end-of-run revocation read fails the M1 test.

S5. RUNTIME_PROFILE now fails closed (see D2 above). There is a test of how each profile value maps.

S6. New migration 0056_permissions_revoked_idx adds permissions(subject_type, subject_id, status, revoked_at).

Nits:

  • recordSources refuses a revoked lease with agent.run_revoked.
  • interrupted is a CodeRunStatus, and the statuses live in code-run-statuses.ts.
  • chat_code_runs (migration 0008, regenerated since it isn't on main) gains a CHECK on status, a CHECK on tool_call_id length (1 to 200, cut to fit before insert), and a partial index WHERE status = 'running'. A chat keeps its latest 1000 records.
  • Not recorded: pi-ai's version, which exports no package.json to read (it is pinned to pi-agent-core's version in the catalog), and the workerd build version, which the runtime doesn't expose.
  • RUNTIME_PROFILE is added to the platform's reserved binding names.

Evidence at the top of the stack:

  • vp check: passes.
  • vp test run on agent, connections, chat-connections, instructions, knowledge, chats, memory, run-fixes, permissions, packages and dependencies: 220 passed.
  • vp test run -c vite.screens.config.ts on agent-builds, preview-repairs, app-preview and agent-apps: 47 passed.

Delta review fixes (d14824f)

Must-fix 1, fail open on T1: the end-of-run revocation read could throw before the lease closed. Must-fix 2, regression: the lease stayed open through that D1 round trip after wall expiry or cancel.

  • #endCodeRun now closes the lease and records the status synchronously and unconditionally first. Only then does it read revocations.
  • If a revocation is found, the record is revised to revoked, one agent.run_revoked is recorded, and the result and logs are withheld.
  • A read that throws counts as revoked (fail closed). It is logged as agent.revocation_read_failed.

New tests (only the revocation query is faulted, through a database proxy; everything else is real):

  • A throwing revocation read: the result and logs are withheld (Error (revoked)), the record says revoked, and the lease reads ended.
  • A call during the end-of-run read (the read is slowed 2 s; the run ends at its 1 s wall clock while the code waits, then calls): refused once as agent.run_ended, with the run recorded as wall_expired.
  • Mutation check: against the previous #endCodeRun, both tests fail.

Nits:

  • A comment now states that cross-machine clock skew is the accepted residual of the margin-free comparison.
  • The admitEffect doc now says refusals are recorded per refused call.
  • A comment ties granted_at IS NOT NULL to how permissions become active: both grant paths set it, so only a request declined before it was granted lacks it.

Evidence at the top of the stack:

  • vp check: passes.
  • Core test files: 226 passed (11 files).
  • Screen-config test files: 47 passed (4 files).
  • smoke:code-mode: passes.

Greptile fix (1b7fd82): admission right before the write

admitEffect ran before slow preparation, not right before the write, so work that outlasted the run, or a revocation meanwhile, still let the write land after the lease closed. I confirmed this with a probe for dependency resolution.

Each write helper now takes an admission callback and calls it right before its batch, after any slow preparation. The early check stays as a cheap refusal. Where the late check runs:

Helper Late check runs
resolveDependencies after resolveGraph and inspectGraph, before proposeResolved
proposeDraft after storeTree, before auditedBatch
saveUserMemory as writeVersion's lastCheck
createApp, createFromBlueprint, requestPermission, askInChat before their batch

Unchanged:

  • saveDraft, dropDraft and takeCreate already check the lease inside the Workspace transaction.
  • connections.call is admitted right before CONNECT.call.

New tests, each slowing an outside step past a 1 s run limit:

Slowed step Result
the registry metadata fetch no dependency request or lock is stored, and the refusal is recorded as build.resolveDependencies
the upload of the version's tree no pending version
the knowledge reads no USER.md is saved, and one memory.saveUser refusal is recorded

Mutation check: removing the late admission from any one of the three helpers makes its test fail.

Evidence:

  • vp check: passes.
  • Agent, connections, chat-connections, knowledge, chats, memory, permissions, dependencies, packages, app-blueprints and apps tests: 236 passed.
  • agent-builds and agent-apps (screen config): 34 passed.

@nickruigrok
nickruigrok force-pushed the feat/code-mode-module-contract branch 2 times, most recently from 17e3dfd to 711c4a5 Compare October 11, 2026 03:06
@nickruigrok
nickruigrok force-pushed the feat/code-mode-execution-leases branch 2 times, most recently from cc14ff5 to d14824f Compare October 11, 2026 03:28
@nickruigrok
nickruigrok force-pushed the feat/code-mode-module-contract branch from b4ae7c2 to e41f072 Compare October 11, 2026 03:54
@nickruigrok
nickruigrok force-pushed the feat/code-mode-execution-leases branch from d14824f to b04298a Compare October 11, 2026 04:01
@nickruigrok
nickruigrok force-pushed the feat/code-mode-execution-leases branch 2 times, most recently from dbde2ff to 210f56e Compare October 11, 2026 11:32
Base automatically changed from feat/code-mode-module-contract to main October 11, 2026 11:43
…tion

Each run opens with a record in the chat: its tool call, module hash and
length, the exact agent version, the runtime and whether it enforces the
CPU limit (plain workerd records cpu_limit_unsupported), then how it
ended, or interrupted after a restart. A revocation of any of the agent
permissions after the run opened closes its lease. Writes are admitted
against the lease as they commit, not only when the call starts: draft
writes, discards and creates in the same Workspace transaction, the rest
right before the effect.
…y those

A run reads revocations once more as it ends, so a revocation its code
never called after still ends it as revoked, recorded once. A revoked
run hands on neither its result nor its logs. Only a revoked permission
that was granted and held where the run acts counts: a declined request
or another chat permission no longer ends it. Draft writes, discards
and creates are admitted with the revocation read too; admission reads
revocations first and the lease last; refusals of connection calls and
USER.md saves are recorded. A runtime is taken to enforce the CPU limit
only when it says RUNTIME_PROFILE=cloudflare. Run records constrain
their status and call ID and keep a chat latest 1000, and an index
serves the revocation read.
A run lease now closes at once as the run ends, before anything else
can fail or wait, so nothing the code still does is admitted after.
Only then are revocations read; one found revises the run to revoked
and withholds its result, and a read that fails counts as one.
@nickruigrok
nickruigrok force-pushed the feat/code-mode-execution-leases branch from 210f56e to d33b9d7 Compare October 11, 2026 11:44
@nickruigrok
nickruigrok marked this pull request as ready for review October 11, 2026 11:44
@nickruigrok

Copy link
Copy Markdown
Contributor Author

@greptileai review

@greptile-apps

greptile-apps Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Critical impact] The changes since the previous review appear safe to merge, and the previous write-admission finding remains fixed.

Summary

Records Code Mode runs and checks their leases before calls and writes. Revoked runs hand neither results nor logs to the model.

  • Revoking a permission closes the affected run and withholds its result.
  • Each Code Mode run leaves a record of what ran and how it ended.
  • A run must still be open when its writes commit.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Open lease and record run] --> B[Run code]
  B --> C[Check lease and revocations before calls]
  C --> D[Prepare write]
  D --> E[Check admission before write]
  B --> F[Close lease and record outcome]
  F --> G[Read revocations once more]
  G --> H{Revoked or read failed?}
  H -->|Yes| I[Record revoked and withhold result and logs]
  H -->|No| J[Return outcome]
Loading

Reviews (5) · Last reviewed commit: "test(core): revoke only after the code's..." · Reviewed by Greptile

Comment thread apps/core/src/agent-builds.ts
…w preparation

The lease was checked before resolving dependencies, proposing a draft,
saving USER.md, creating an App, asking for a permission or a connection,
but what came after (fetching packages, storing the version files,
preparing the document) could outlast the run, and the write then
landed after its lease closed. Each of these now takes an admission and
calls it right before its batch; the early check stays as a cheap
refusal.
@nickruigrok

Copy link
Copy Markdown
Contributor Author

@greptileai review

@nickruigrok

Copy link
Copy Markdown
Contributor Author

The dependency-write finding is fixed at 1b7fd82: resolveDependencies (packages/resolve.ts:879) now calls admit() after resolveGraph/inspectGraph and right before proposeResolved. proposeDraft and saveUserMemory were fixed the same way. A test covers each by slowing the fetch past a 1 s run limit (dependencies, agent-builds and agent tests), and removing the late admission from any helper fails its test. The thread is resolved with that evidence.

@nickruigrok

Copy link
Copy Markdown
Contributor Author

@greptileai review

1 similar comment
@nickruigrok

Copy link
Copy Markdown
Contributor Author

@greptileai review

@nickruigrok

Copy link
Copy Markdown
Contributor Author

@greptileai review

@nickruigrok
nickruigrok merged commit 2588d54 into main Oct 11, 2026
9 checks passed
@nickruigrok
nickruigrok deleted the feat/code-mode-execution-leases branch October 11, 2026 12:49
nickruigrok added a commit that referenced this pull request Oct 11, 2026
…get (#390)

Fourth PR of the GRA-349 stack, on top of #389. It adds bounded output
and output that streams before the tool completes (section 21.1, step
5).

## What changes
- **One output budget:** 64 KiB of UTF-8, shared by logs, streamed
progress, result and error.
- 1 KiB is kept for how the run ended, so a flood of logs never hides
the error.
- Cut text ends with an explicit marker: `… (cut: over 65536 bytes of
output)`.
  - This replaces the 32 K-character model view.
- Text is cut with `TextEncoder.encodeInto`, so a cut never lands inside
a character.
- **Streaming:**
- The harness sends each logged line to core's callback in order, one
after another, and waits for every line to arrive before a call returns.
The code never sees the callback.
- The callback is handed to `verify` and to `run` separately, because an
RPC stub only works in the call that brought it.
- Core checks each line like the run's result (a string, bounded) and
drops it once the run has ended.
- Each line becomes a Pi tool update (`onUpdate`; Pi ignores updates
after the step settles). The turn hands it to the Workspace DO.
- The DO takes a line only while the run's lease is open. It counts and
bounds the lines itself (1,000 lines, 128 KiB of raw text a run),
whatever the harness claims.
- A line is shown only after a read of revocations that **began after
the line arrived** finds none. Those reads happen at most once a second
a run and fail closed. So nothing logged after a revocation is ever
shown, even if the code makes no further call.
- The shown text is the whole output redacted at once, then cut at the
budget less the reserve, so a credential split across lines is caught.
Watchers get it through `ChatUpdate.output`, which grows incrementally
like `partial` and is left out when unchanged.
- The output stays until the step's result is stored, so it doesn't
flicker away first. A revocation, or the end of the turn, drops it at
once.
- **Redaction.** Bearer and basic credentials, `sk-`/`pk-`/`rk-` keys,
JWTs, PEM private keys and AWS access keys become `[redacted]`. This
applies to every streamed line and to what the model reads and the
transcript stores. A run holds no secret of its own; this guards against
an outside system's data carrying one.
- **Correlation.** The tool result's `details` now carry the run ID
along with the status, tying the transcript to the run's record.
- **Web.**
- `ChatView.output` holds the running step's output, built with
`applyOutput`.
- The code step shows an "Output" section while it runs, replaced by its
result.
  - Translations are added for nl, de, fr and es.

## Evidence
New tests in `apps/core/test/agent.test.ts` ("chat agent code output"),
through the real DO watch, Loader and Pi loop:
- **Streaming:**
- before the step's result message reaches the watcher, it has seen
`first\nkey [redacted]` and then `…\nafter the wait`;
  - the output goes back to `null` once the step ends;
  - the model's text is redacted the same way;
- a line sent through `codeRunOutput` after the run closed shows
nowhere.
- **Budget:** a flood of 5,000 lines of two-byte characters followed by
a throw:
- the final text is at most 64 KiB, is cut with the marker and still
ends with `Error (threw): Error: the end`;
  - the streamed output never passes 64 KiB less the 1 KiB reserve.
- **Pure helpers:** byte cuts with `é` and emoji, and the redaction
patterns.
- **Updated tests:** the existing flood and forged-prototype tests now
check the 64 KiB budget and the new marker.

In `apps/web/src/chat/follow-chat.test.ts`: the output grows, starts
afresh for another step, and drops when no step runs.

Commands:
- `vp check`: passes.
- `vp test run test/agent.test.ts test/chats.test.ts
test/run-fixes.test.ts test/agent-connections.test.ts`: 90 passed.
- `vp test run -c vite.screens.config.ts test/agent-builds.test.ts
test/agent-apps.test.ts`: 32 passed.
- In apps/web, `vp test run src/chat/follow-chat.test.ts
src/locales/locales.test.ts`: 12 passed.
- `vp build` (web, React Compiler): passes.


## Restacked on the #387 and #389 review fixes (daf0cb4)
- The harness's streaming is merged into the hardened harness. The
output callback, captured `Promise.prototype.then` and
`Promise.resolve`, and the in-order delivery are unchanged, and every
outcome is still returned rather than thrown.
- A lease closed by a revocation drops output that no watcher has had
yet. The model gets no logs of a revoked run (#389, M2).


## Review fixes (c7d3cdd)
**Must-fix: output kept streaming after a mid-run revocation.** A line
is now shown only once a read of revocations begun after it arrived
finds none. Those reads run at most once a second a run, through the
same `revokedSince` read, and fail closed. A revocation found there:
- closes the lease as `revoked`;
- drops the output, shown or pending;
- records one `agent.run_revoked` (`method: output`);
- streams nothing more.

**Should-fix:**
- **Redaction now also covers** Stripe `sk_`/`pk_`/`rk_` live and test
keys, GitHub `ghp_`/`gho_`/`ghs_`/`ghu_`/`ghr_`/`github_pat_`, Slack
`xox[abpr]-`, Google `AIza`, and the value of `api_key`, `apikey`,
`token`, `secret`, `password` or `authorization` written as `name=value`
or `name: value`. The docstring lists exactly what is covered and says
it is a net, not a guarantee.
- **The stream redacts like the model view:** core keeps each run's
lines and redacts all of them as one text.
- **Core counts lines and raw bytes itself**, and doesn't trust the
harness's totals.

**Nits:**
- The output stays until the step's result is stored (no flicker).
- `output` is left out of an update when it didn't change (`applyOutput`
keeps what was shown).
- `aria-live` is dropped.

**New tests:**
- **Revocation with no call:** the code logs, waits and logs again with
no call; the admin revokes in between. No line logged after the revoke
ever reaches the chat, and the run ends `revoked`.
- Mutation check: with the stream's revocation read disabled, this test
fails.
- **Clearing when a call finds the revocation** (`revokeCodeRun`): the
shown output goes to `null` before the step's result.
- **A PEM private key logged line by line** never shows its body in any
streamed text, and the model reads `Logs:\n[redacted]\ndone`.
- **1,500 lines sent straight to `codeRunOutput`** show as 1,000.
- **The redaction table covers the new shapes.** The token-shaped
fixtures are put together at runtime, so secretlint finds no token in
the repository.

Evidence (top of the stack):
- `vp check`: passes.
- Core: 230 passed.
- Screen-config tests: 47 passed.
- Web: 12 passed, and the web build passes.


## Delta review fix (2815b1e)
**Must-fix: quadratic JWT redaction.** The pattern started at `\b`, and
a flood like `eyJ-eyJ-…` has a word boundary at every step. Redacting
about 128K characters of that took about 4 s, inside the Workspace
object, which re-redacts output about once a second.
- The pattern now starts at `(?<![\w-])`. A real JWT still matches.
- A new test redacts floods of about 128K characters for every
credential shape, each with tails of `""`, `"x"`, `"-"` and `"."`. All
must finish within 250 ms.
- With the old pattern, that test fails (13.8 s). With the fix, it
passes. No other pattern was slow under the same floods.

**Nits:**
- A revocation read that fails is now recorded with its own reason,
`agent.revocation_read_failed`, both by the output check and at the end
of a run.
- New tests for the output check:
- **Failed read:** nothing is shown, the run ends `revoked`, and one
`agent.revocation_read_failed` is recorded.
- **Slow read:** the read takes 3 s, longer than the run, so the line is
never shown before the step's result, which shows it.

Evidence: `vp check` passes; `vp test run test/agent.test.ts
test/chats.test.ts` passes (103).
nickruigrok added a commit that referenced this pull request Oct 11, 2026
… its turn as it ends (#409)

## What and why

GRA-429, split from GRA-349. A Code Mode build that takes longer than
its run's 30 s budget now returns a durable job ID. The job keeps going
after the transient Worker and its lease end, and how it ended reaches
the turn that submitted it, never another turn and never a closed one.

**Security-relevant (leases and authority): needs a person's review
before merge.** The threat model is written out in `build-jobs.ts` and
in the design comment on GRA-429.

**Mechanism: no new framework.** The job is a second internal workflow
(`build`) on the existing engine and dispatcher, next to upload
`extraction`. Its record is a new `chat_build_jobs` table in the
Workspace DO, next to `chat_code_runs`. The migration only adds that
table.

**API**
- `env.build.submit(app, { kind: "check" } | { kind: "propose", message
})` returns `{ job, status, revision }`.
  - It is admitted against the run's lease, inside the DO's own step.
- It binds the job to the draft's revision, the run's turn and the run's
grant epoch (when the run opened).
  - A chat can have at most 3 jobs running at once.
- Each job takes one of the draft's checks this turn, as an in-call
`check` or `propose` does (`takeCheck`). It settles that check against
the turn's counter when it ends, and its result then carries the real
`failedInARow`. So jobs can't get the agent past `maxFailedChecks`. If
the turn has already ended when the job finishes, nothing is settled:
the counter belonged to that turn, and the next question resets it
anyway.
- A `running` record whose instance is errored, terminated, or complete
without its last step, or that has no instance a minute after it was
submitted, is failed with a generic error. Both `submit` and `job` do
this, which frees its place and settles its check. A failed status read
counts as unknown, not dead, so one transient engine error doesn't fail
the call.
- A chat keeps the records of its latest 100 jobs; running ones are
never pruned. They aren't counted in the chat's transcript limit, which
bounds what the model reads; the row count and the bounded results bound
what they hold.
- Internal instances (build and extraction) get the deployment's run
retention when created.
- `env.build.job(id)` returns the job's status, its result (what `check`
or `propose` returns), its receipt (the version proposed) and any error.
- It answers only for jobs of the same chat, and only while the person
still builds the job's App (`appFor(..., "builder")`). A later
execution, with its own new ID, follows the job; the module is never
replayed.
- It also records a receipt for an ended job that committed before it
ended.

**Completion**
- When the job ends, the Workspace DO pushes a system message (no result
data, only IDs and the outcome) into the queue of the turn that
submitted it.
- Pi's `getSteeringMessages` and `getFollowUpMessages` drain that queue
between steps, never between a code step and its result.
- The queue is keyed by a random per-turn ID and dropped when the turn
ends. A closed turn, a later turn, or a turn after a DO restart never
gets the message; the result stays on the job record.

**Authority**
- Every step reloads the chat's agent and person and checks again what
an in-call build checks: the agent's `write` on Apps, the person's
membership and builder role, and restricted mode.
- Every step, and the commit right before it lands (through
`proposeDraft`'s admit hook), reads revocations since the grant epoch.
Finding one ends the job as `revoked`.
- `finish` admits and authorizes a completed job once more: grant,
membership, builder role and restricted mode. So a job revoked at the
very end, or whose person lost the App meanwhile, hands on no result.
- Cancelling the turn ends that turn's jobs, and deleting the chat ends
all of its jobs; the instances are terminated too.
- **Once its turn has ended, a job can be stopped only by deleting the
chat, short of a revocation.** I chose not to add a cancel-by-job-ID: it
would be more surface for little gain, since a job is bounded by its
steps and its draft's revision and changes nothing live. This is
recorded on GRA-429.
- If `submit` can't start the instance, the job stores a generic
`internal.unexpected` error; the engine's own message goes only to the
log.

**Idempotency**
- Steps are journaled by the engine.
- The commit step records the tree it is about to commit. If the step
runs again after the commit landed, it finds the version `parent + 1`
with that tree, proposed from this chat, and takes it as its receipt
instead of committing again.
- A job's commit records the job's ID in `proposedBy`, and that version
is adopted as the receipt only if it carries this job's ID. So a commit
of an identical tree from the same chat is never recorded as this job's.
- That lookup, and recording the receipt, happen before the step asks to
be admitted. So a commit that landed keeps its version even when the
retry is then refused and the job ends revoked or cancelled.
- `finish` ends only a running job.

**Out of scope, recorded on the issue**
- `resolveDependencies` stays in-call: its effect is already admitted
right before it commits (#389).
- On plain workerd there is still no Workflows engine. This has the same
on-prem limitation as uploads, noted in `engine.ts`.

## How it was tested

`apps/core/test/build-jobs.test.ts` has 17 tests, in workerd against the
real engine, DO, model loop and D1. Only the model provider is faked.
The tests hold a job inside its check step by holding its screen-build
cache read on `env.FILES`, so each scenario is deterministic:
- **Outlives its run.** The run is recorded `completed` while the job
runs. The job then completes with version 1. The system message lands
right after the answer the turn gave meanwhile, and the model's last
request carries it.
- **Closed and later turns** are never told; the later turn makes
exactly one model request.
- **Revocation mid-job.** Another of the agent's permissions is revoked
while the job checks. The job ends `revoked` with no result and no
version, and the refusal is audited.
- **Cancel.** Stopping the turn ends the job as `cancelled`, and nothing
is committed.
- **Restart mid-commit.** `restart({ from: "commit" })` after the commit
landed gives one version and one `app.committed` event.
- **Revoked before the commit runs again.** The same restart, with a
revocation between the two attempts: the job ends `revoked` with
`version: 1`.
- **Repair loop.** Five failing job checks in one turn report
`failedInARow` 1 to 5. After that, a sixth `submit` (check or propose)
and an in-call `check` are all refused with `app.checks_exhausted`.
- **Cap and slot freeing.** Four concurrent submits start 3 jobs and
refuse 1 with `agent.jobs_exhausted`. Terminating one job's instance
behind its record's back fails that job (`internal.unexpected`), and a
fifth submit then starts.
- **Builder role lost mid-job.** The job's result is withheld: `failed`,
`role.forbidden`, `result: null`.
- **Restricted mode mid-job.** The chat turns restricted while the job
checks: no version, and the job fails with `permission.restricted`.
- **Pruning.** With one job still running and 105 ended, the chat keeps
the newest 100 ended records and the running one, which is the oldest.
Ties within the same millisecond are ordered by when each record was
written.
- **`job()` after losing the App.** Once the person no longer builds the
App, `job()` refuses with `role.forbidden`, and the job record is kept.
- **Rights lost while the files are stored.** The test holds
`proposeDraft`'s tree write, then the person loses the builder role (or
the chat turns restricted). The job fails with `role.forbidden` (or
`permission.restricted`), and no version is created. The commit's admit
callback now rechecks the person's build rights and restricted mode, and
the draft's revision, right before the commit.
- **Draft written meanwhile.** A write to the draft while a check job
checks fails it with `app.conflict` and no result. A write while a
proposal stores its files fails it with `app.conflict` and no version.
- **Another chat.** `env.build.job(id)` from another chat of the same
person and agent reads `agent.job_not_found`.

**Mutation checks.** Each mutation was applied on its own and reverted;
each failed the test named:

| Mutation | Test that fails |
| --- | --- |
| Deliver to any running turn | Closed and later turns |
| Drop the event | Outlives its run |
| Unwire Pi's steering and follow-up | Outlives its run |
| Remove the admissions before the commit | Revocation mid-job |
| Cancelling a turn leaves its jobs | Cancel |
| Take no check when opening a job | Repair loop |
| Move receipt reconciliation behind admission (removed from before it)
| Restart mid-commit, and Revoked before the commit runs again |
| Never fail dead jobs | Cap and slot freeing |
| No authorization in `finish` | Builder role lost mid-job |
| No restricted check | Restricted mode mid-job |
| Never prune | Pruning |
| Prune running jobs too | Pruning |
| `job()` without the builder check | `job()` after losing the App |
| Commit callback without the authorization recheck | Both
rights-lost-while-storing tests |
| Commit callback without the draft recheck | Draft written while the
files are stored |
| No draft recheck after a check job's check | Draft written while a
check job checks |

**Regression.** `agent.test.ts`, `agent-builds.test.ts`,
`chats.test.ts`, `agent-instructions.test.ts` and `uploads.test.ts`
pass: 181 tests, 3 workers. `vp check` and `vp run check:migrations`
pass. The branch is rebased on current main.

## Checklist

- [x] `vp check` and targeted `vp test` pass
- [x] Schema changes are expand-only (no drops or renames in the same
release)
- [x] New external calls go through connect; new model calls through the
model gateway (none added)
- [x] No secrets, client names or client configuration
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant