agent_run subject: company Agent code execution with a private run namespace (Team chat)

[Optale-ops]

Engine: agent_run subject for company Agent code execution

Branch agent-run-subject off e9ad47c (Optale-ops/code-interpreter). Head: e5f925fa5054b8943f83145fb5b25015edd2ce4f. Commits:

• 2f5c98a test: personal parity fixture; golden generated on the unchanged e9ad47c tree
• 850dbae feat: agent_run subject
• 3a0b16b fix: Security review of 850dbae (ENGINE-REVIEW-850dbae.md), P1 and P3; P2 proven live
• e5f925f fix: Foundation review of 3a0b16b (ci20-3a0b16b.json): owner-checked delete operation (P1-1), parity header maps (P1-2), typed test assertions (P2)

Contract: native-chat-backend/agent-code-identity/PROPOSED-CONTRACT-2026-10-04-v2.md (sha256 41d50608…). Security conditions C1-C6: security-agent-code-identity-2026-10-04/CONTRACT-REVIEW-v2.md.
Exact diffs for Security (/home/thor/optale-artifacts/agent-mcp-clone/agent-code-identity/engine-diff/):

• full engine-agent-run-subject-e9ad47c7..e5f925f.diff (sha256 449546df65879dc7fbd8270c335a6e7ef48c89dc1fd920c30889aa5747caf1d7)
• interdiff engine-agent-run-subject-3a0b16b..e5f925f.interdiff (sha256 6b29393da9458076676b12d66d02facd1e518acde37cef2eb86b1e508eabb6da)
• previous full engine-agent-run-subject-e9ad47c7..3a0b16b.diff (sha256 bb703cd7ba11789d9adbff390f706da069099527efa7c187f2aa2bbb41f1667c)
• interdiff engine-agent-run-subject-850dbae..3a0b16b.interdiff (sha256 e43fe15a769ef6e1282e80c2f7548ebca60fec6cda6e18641c872fd3892e7885)
• first review round: engine-agent-run-subject-e9ad47c7..850dbae.diff (sha256 f4be4b52…819ad0)

Foundation review of 3a0b16b: changes in e5f925f

• P1-1, mixed-version deletion. In 3a0b16b an owner-bound delete was a plain DELETE plus X-CodeAPI-Owner-Expect; an e9ad47c file_server ignores that header and deletes. Now:
  • The file_server has a separate operation, POST /sessions/:sid/objects/:fid/owner-delete (file-server.ts:763). It refuses a request without a well-formed expected owner (400) and an object with no stored binding or a different one (403). It answers 200 {outcome:'deleted'} or 200 {outcome:'absent'}, 500 on storage error. It never deletes on "no expectation".
  • The api sends owner-bound deletes only there (service/router.ts:979). Only an explicit 2xx deleted (→ 200) or absent (→ 404) counts; every other answer, including an old file_server's 404/405, is a refusal (403; 500 stays 500) and the bytes stay. No branch falls back to the plain delete.
  • The plain DELETE route on the file_server is back to its e9ad47c form. Personal deletes and cache-present agent_run deletes keep using it, unchanged.
  • Tests: AR mixed version: against a file server without the owner-checked operation, an expired-cache deletion is refused and deletes nothing (cross-tenant token and owner token both 403 on an old file server, bytes present, no plain DELETE sent; on the new file server the other tenant gets 403 and the owner 200); AR after the 24 h session cache expires … asserts the call is POST …/owner-delete and no plain DELETE; AR an object with no binding ….
  • Live, KS-7 side pair (api e5f925f; tenant A token from the pair's tenant-bound key, tenant B token from the unbound lab key naming the same Agent, run and object; session cache deleted in Redis):
    • e9ad47c file_server (ks7-engine-svc-production:e9ad47c76): tenant B 403, tenant A 403, statObject shows the bytes still present. Record side-pair/out/mixed-delete-e5f925f-oldfs.json (sha256 c70e7579c9729c7489ce33f562bd52cd9f40043f06e1bf2e3f96de10cce9ee32).
    • e5f925f file_server: tenant B 403 with bytes present; tenant A 200 and statObject not found; repeat 404; owner-delete without an expected owner 400 with bytes present; an unbound object 403 with bytes present. Record side-pair/out/mixed-delete-e5f925f-newfs.json (sha256 6b232a2120bfe57e2a5b1c3816b03d0b54374e4a5eb898cb38dc458107aa8500).
• Reverse gaps (fail closed). A new api with an old file_server, or a new file_server behind an old gateway, stores agent_run objects without a binding (the old file_server ignores the header; the old gateway never sends it). After the session cache expires, nothing can delete those objects through the binding path: an old file_server has no owner-delete operation, and the new one refuses unbound objects. Confirmed by AR reverse gap: an object stored without a binding (old file server or old gateway) cannot be deleted after expiry and live: the upload through the new api onto the e9ad47c file_server stored the object with no binding (oldfs record). Such objects need cleanup after a full upgrade by other means; they are never deleted for the wrong owner.
• P1-2, CI Bun 1.3.14. The parity test's order-sensitive comparison sorts the forwarded HTTP header maps (the only unordered input) on both sides; the golden is unchanged since 2f5c98a. Verified under oven/bun:1.3.14: parity, agent-run and egress-gateway tests pass (81/81).
• P2, test assertions. agent-run tests assert CodeApiJwtAuthError.reason (malformed_claims, tenant_not_allowed) against a baseline that verifies, plus statuses and side effects; no error wording is pinned.

Security review of 850dbae: changes in 3a0b16b

• P1, agent_run only on the hardened path. apiKeyAuth refuses every agent_run token with 401 (Invalid bearer token, logged reason agent_run_unavailable) unless CODEAPI_HARDENED_SANDBOX_MODE=true and CODEAPI_INTERNAL_SERVICE_TOKEN is configured (middleware/auth.ts:196). All routes are covered because the check runs before dispatch. This keeps output owner bindings coming only from the gateway's sealed grant. Personal tokens are unaffected. Test: AR agent_run availability (hardened sandbox + internal service auth) › agent_run is refused with 401 unless hardened mode and internal service auth are both on; personal is unaffected; PP unchanged.
• P3, single upload kind=user. /upload and /upload/batch refuse agent_run kind=user with 403 even without an id, before parsing (service/router.ts:494,717). Test: AR C1: upload kind=user is refused for agent_run.
• P2, live on a hardened side pair (KS-7 agentrun-pair, hardened mode on, own egress gateway and file_server, 3a0b16b service images, e9ad47c runner with the patched rootfs bind-mounted). Record: /home/thor/optale-artifacts/agent-mcp-clone/agent-code-identity/side-pair/out/agent-3a0b16b.json (sha256 9a86fa8ae1dc349f28362e790ccc4ec6fd130ed5bd71113dfe208161cabc9caf).
  • An agent_run execute wrote agent-output.txt. Its MinIO codeapi-owner equals ownerBindingValue computed from the raw tenant, Agent and run ids (outputBindingIsFromRawIds: true), not the masked labels.
  • The Redis session:<sid> key was deleted to force expiry. After that, run R2's file_delete token got 403 and the active-run token got 403.
  • Run R1's file_delete token got 200. statObject then returned NotFound, file_server returned 404, and a repeat delete returned 404.
  • A forged owner header sent straight to file_server got 400, binding unchanged. All other refusals were as in round one.

Paths below are relative to service/src unless marked api/. Tests: agent-run.test.ts (AR), personal-parity.test.ts (PP), egress-gateway.test.ts (GW), api/src/execution-manifest-request.test.ts (RM).

Claims, grammar, keys (contract §1)

• principal_source: 'agent_run' takes its own validator: auth/librechat-jwt.ts:616 (branch), :688 validateAgentRunClaims. Old engines still refuse it (only librechat_jwt/openid_reuse there; side-pair proof below: 401).
• sub = canonical lowercase 24-hex (agent-run.ts AGENT_ID_PATTERN), run_id = canonical lowercase UUID (RUN_ID_PATTERN), role must be AGENT, tenant_id required regardless of CODEAPI_TENANT_ISOLATION_STRICT and colon/whitespace-free (librechat-jwt.ts:698), no borrowed human claims (:669 HUMAN_ONLY_CLAIMS: org_id, service_id, external_user_id, chc_user_id, plan_id). Values are refused, never normalized.
• feat(auth): bind a JWKS key to the tenants it may sign for #18 bound-key enforcement unchanged and applies to agent_run (librechat-jwt.ts:819).
• Same iss/aud/alg/kid/TTL; issuer/audience/window checks shared and order-preserved (assertTokenWindow).
• Tests: AR agent_run claim grammar › all six cases (well-formed; tenant required strict on/off; malformed values refused; feat(auth): bind a JWKS key to the tenants it may sign for #18 bound tenant; personal file_delete refused; Agent-looking personal sub stays personal).

Private namespace (contract §2)

• <ns>:agent-run:<A>:<R> built only from the verified principal: agent-run.ts:31 agentRunSessionKey; session-key.ts:72 branch, :144 resolveAgentRunSessionKey, :125 output bucket.
• Tests: AR collision: a user token with kind=agent id=<agentId|runId|agent-run:A:R> cannot touch Agent objects; the Agent cannot read a user bucket.

C1 upload kinds

• session-key.ts:144-170: kind=agent requires id == signed run_id (else 403); kind=skill keeps the shared key; kind=user refused (403). :173 resolveUploadSessionKey: agent_run kind=skill requires read_only=true. Router uses it in /upload and /upload/batch (service/router.ts, resolveUploadSessionKey(req, sessionKeyInput, readOnly)), 403 surfaced for single and batch.
• Sandbox outputs: output bucket is always the private key (session-key.ts:125); sessionAuth refuses agent_run DELETE outside kind=agent (middleware/auth.ts:308), so shared Skill objects stay read-only.
• File refs on /exec: resolver 403 mapped to FileRefAuthorizationError(403,'subject_refused') (service/file-authorization.ts).
• Tests: AR C1: upload kind=agent id=<signed run> …, C1: upload kind=agent with an id other than the signed run is refused, C1: upload kind=user is refused for agent_run, C1: upload kind=skill needs read_only=true …, exec with a kind=user file reference is refused for agent_run.

C3 deletion-only tokens

• file_delete = {storage_session_id, file_id} parsed strictly (librechat-jwt.ts:741); a personal token carrying it is malformed (:641).
• Edge check in apiKeyAuth before any handler: middleware/auth.ts:103 fileDeleteRequestMatches (method DELETE, exact path /files/<sid>/<fid>, query exactly kind=agent&id=<run_id>), enforced at :201 → 403.
• Tests: AR every route other than the signed DELETE target is refused before dispatch (/exec, /exec/programmatic initial and continuation, upload, upload/batch, download, files list, metadata, other target, other run, kind=user/skill, extra query, trailing slash, wrong route; asserts no job and no file-server write), the signed target is deleted while the session cache is present.

C4 owner binding

• Binding = agent_run.<sha256([agent_run, tenant, A, R])> (agent-run.ts:75), stored as MinIO object metadata X-Amz-Meta-Codeapi-Owner so it lives exactly as long as the bytes.
• Written only by internal callers from the verified principal, signed per object (HMAC keyed from the internal service token over sid/fid + binding): api at upload (service/router.ts ownerBindingHeaders, private kind=agent only), gateway for sandbox outputs from the grant (egress-gateway.ts agentRunOutputOwnerHeaders). The gateway never forwards sandbox headers; client headers on /upload are ignored.
• file_server stores a binding only if signed for that exact object; a forged/replayed header refuses the PUT (file-server.ts PUT route, ownerBindingFromHeader, agent-run.ts:113).
• Deletion after the 24 h session cache: sessionAuth (middleware/auth.ts:319) lets only a file_delete token with an absent cache entry proceed with ownerBindingExpectation; the file server deletes only if the stored binding equals it, 403 if absent/mismatched, 404 only if the bytes are absent, 500 on storage error (file-server.ts DELETE). Router maps agent_run 404/403 explicitly (service/router.ts DELETE). Present-but-different cache entry stays 403. Ordinary read/exec gets no exception.
• Tests: AR owner binding (C4) › only a binding signed for this exact object is accepted; C4: a client-supplied owner header on upload is ignored …; C4: a PUT with a forged owner header for another run cannot change the binding; that run cannot delete; after the 24 h session cache expires the binding authorizes deletion; repeat is not-found; an object with no binding is never reported deleted through the binding path. GW agent_run output PUT carries the grant owner binding and drops a sandbox-supplied one, personal output PUT carries no owner binding.

C5 identity consumers (no userId fallback)

Consumer	Change	Test
auth/principal.ts:30	Discriminated AgentRunPrincipal (no userId by type); applyPrincipal (:52) writes agentRun, no userId/planId	AR auth context and execution identity carry the Agent subject and no user id
getPrincipalOrReject auth/principal.ts:115	Agent branch requires tenant/Agent/run, else 401	AR routes
execution-identity.ts	AgentRunExecutionIdentity; getExecutionIdentity never builds an Agent from a user id	same
sessionAuth middleware/auth.ts:261	Agent branch, no userId requirement	AR read/collision tests
output bucket session-key.ts:125 (+ programmatic call sites)	private key	AR exec: outputs go to the private key …
continuation equality service/replay-state.ts continuationSubjectMatches (used by replay and blocking paths in programmatic-router.ts)	kind + Agent + run + tenant + context hash; user never matches Agent state	AR continuation equality compares kind, Agent, run, tenant and context hash, never a sub string
programmatic-router.ts:371 fallback → replay-state.ts replaySessionKey	Agent state needs its persisted key; legacy fallback personal-only	AR replay output key: …
rate limit middleware/limits.ts keyGenerator	<ns>:agent:<A> per Agent, not per run	AR rate-limit bucket is stable per Agent …
runtime session runtime-session/id.ts	scope (ns, kind, A, R, hint), injective vs personal material	AR runtime session scope separates runs, Agents and users whatever the hint
manifest execution-manifest-claims.ts, grant egress-grant.ts, execution-manifest.ts, api/src/execution-manifest.ts	principal_source=agent_run, agent_id, run_id, no user_id; ambiguous subjects refused; masked labels agent:/run:	AR manifest and grant …, a grant naming both …; RM agent_run manifest subject
workers.ts:185, sandbox-backend/types.ts, lambda-microvm.ts:823, runtime-session/registry.ts	agentRun instead of canonicalUserId; registry record carries principal_source/agent_id/run_id	AR exec job-data assertions
plan limits (router upload sizes, payload.ts:35, preamble.ts:800,860)	plan_id refused in agent grammar → planLimits.default; no new cap	AR grammar (plan_id is not accepted) + identity test (req.planId undefined)
logs: middleware/auth.ts 33-41/50-65/243, service/router.ts 76-79/110-112/203-206/461/577, programmatic-router.ts, file-server.ts:282, limits.ts, request-error-logger.ts, gateway audit	subject kind + hashed tenant/Agent/run; private session keys → agent-run#<hash>; query strings dropped for agent_run (also for refused tokens declaring agent_run)	AR withoutRawIdentityInLogs on upload, read, exec, delete, expired-cache delete, refused tokens

C6 personal parity

• PP personal identity consumers, routes and log lines are byte-identical to the pre-change engine: fixed Ed25519 seed, clock, jti/UUIDs; token bytes (sha256), verification result, auth context, execution identity, planId, sessionKeys (user, personal kind=agent with run uuid and public key, skill), output bucket, rate-limit key, runtime-session id, manifest + grant claims, replay state, continuation outcomes; plus personal HTTP routes (upload/batch/skill/metadata/download/files/exec/delete and refusals) with Redis writes, forwarded file-server headers, job data and every log line. Golden generated on e9ad47c in commit 2f5c98a; compared both deep-equal and serialized (key order).
• Live: see side-pair proof.

Side-pair proof (KS-7, project agentrun-pair, api 127.0.0.1:27411, own redis/minio/secrets/JWKS)

See the lane report for results; records in /home/ubuntu/agentrun-pair/out/ on KS-7 and copied to /home/thor/optale-artifacts/agent-mcp-clone/agent-code-identity/side-pair/out/.

• Runner: rootfs bundle build reproduces the deployed runner byte-for-byte (index.js sha256 0170a988… both); the new runner image swaps only /sandbox_api/.build/index.js (e735d566…). Deployment must rebuild the runner normally (api/src/execution-manifest.ts changed); an unchanged runner refuses agent_run manifests (fail-closed).

Deploy requirements

• Rebuild and deploy every service together from this head (a mixed version refuses agent_run post-expiry deletes rather than deleting for the wrong owner, but agent_run needs every service at this head): api, service-worker, file_server, egress_gateway, tool_call_server and sandbox-runner. The runner rebuild is required because api/src/execution-manifest.ts changed. An unchanged runner refuses agent_run manifests, which fails closed but breaks every company execution.
• agent_run requires CODEAPI_HARDENED_SANDBOX_MODE=true and CODEAPI_INTERNAL_SERVICE_TOKEN on the api (P1). The gateway and file_server need the same internal service token, because it keys the owner binding. Without them, every agent_run token gets 401; personal tokens are unaffected.
• Order: deploy the engine on AX41 (Foundation) and KS-7 (Agent/MCP, shared restart coordinated with the eval labs) before the Console signer branch ships. Personal behaviour is unchanged, so the engine can go first.
• The wrong-bound-tenant check needs a JWKS key with a tenants list (KS-7's lab key has none).

Side-pair results (2026-10-04, KS-7)

• Old engine (e9ad47c images): personal CLI install (exit 0) + execute_code (exit 0) + output download 200; agent_run token on /files, /upload, /exec → 401 Invalid bearer token.
• New engine, round one (850dbae images + runner with swapped bundle): personal record identical to old after id/timestamp normalization except the order of the sandbox's output file list (sorted: identical). Upload/output session keys unchanged.
• agent_run: upload kind=agent id=R1 200 (private key, owner metadata written); skill read_only 200; skill without read_only 403; kind=user 403; id=R2 403; execute_code exit 0, output in private key with gateway-written binding; read/list own output 200; user token kind=agent id=//agent-run:A:R 403; user sub= kind=user 403; R2 read/exec 403; wrong bound tenant 401; missing tenant 401; deletion token off-target 403 ×3; on target (cache present) 200, bytes gone (404); forced cache expiry: active token 403, R2 deletion token 403, R1 deletion token 200, bytes gone, repeat 404; personal token with file_delete 401; forged owner header PUT straight to file_server 400, stored binding unchanged.
• Logs: requests made with agent_run tokens logged no raw Agent/run/tenant-key values. The only raw Agent id hits were the personal-token collision refusals, whose (unchanged, personal) log line prints the requesting user's own expected key built from their query/sub.