feat(ai): add the ai resource with an OpenAI-compatible gateway by ItamarZand88 · Pull Request #203 · alienplatform/alien

ItamarZand88 · 2026-07-24T23:00:22Z

Summary

Adds an ai resource that gives a workload an OpenAI-compatible endpoint for model inference, in two modes. By default it's keyless: for the host cloud's own models — Bedrock on AWS, Vertex on GCP, Foundry on Azure — a loopback gateway signs each request with the workload's ambient cloud identity, so there are no model API keys to store. A workload can instead bring its own provider key (e.g. OpenAI), in which case the client talks to that provider directly with the vault-resolved key and the gateway isn't involved.

What happens on the keyless path, when a workload declares an ai binding for one of the host cloud's models:

A loopback AI gateway starts next to the app and its URL is placed in the environment; the app points a normal OpenAI-compatible client at it.
The gateway translates each OpenAI-style request into the target cloud's native model call — Bedrock InvokeModel, Vertex rawPredict, or the Foundry Anthropic endpoint — and signs it with the workload's ambient cloud credentials. ← the heart of the change
The response, including streamed tokens, is translated back to the OpenAI/Anthropic shape and returned to the app.

For a cloud's own models, this changes AI access from managing model API keys to using the cloud account the workload already runs in.

What I did

Add a gateway engine (Rust) that translates OpenAI-shaped requests to each cloud's model API and signs them with ambient credentials.
Add the ai resource to the SDK, plus the controllers and per-cloud setup that grant a workload access to that cloud's models.
Support a bring-your-own-key mode: the same ai binding can point at an external provider with the workload's own key (vault-resolved, redacted in logs), which bypasses the gateway and calls the provider directly.
Scope the ambient model-invocation grant to inference only: no deployment or data-plane writes.
Ship the gateway as one standalone binary everywhere: containers run it as their entrypoint, the SDK spawns it on first use and reads back the URL it prints, and compiled single-file Workers embed the binary and extract and spawn it at runtime.
Make getAvailableModels report real availability: the gateway probes each catalog model with a one-token request under the workload's own inference credential and lists only what the cloud actually has enabled (a 429 counts as enabled but rate-limited; 400/401/403/404 drop the model). Each entry carries provider and displayName so an app can build a model picker, and the example README documents each model's one-time enablement step per cloud.
Type the client surface: chat.completions.create and responses.create return OpenAI's own result types (a types-only dev dependency), so callers read .choices[0].message.content with no cast.
Add an ai-quickstart-ts example built around "list the available models, then pick one".

Files touched

crates/alien-ai-gateway — the gateway engine (router, model catalog, availability probe, per-cloud translation + signing) and the alien-ai-gateway launcher binary.
packages/ai-gateway — the TypeScript wrapper: resolves and spawns the gateway binary, and the ambient vs BYO-key connection resolution.
crates/alien-core, packages/core, packages/sdk — the ai resource type, the model catalog with per-model activation metadata, and the SDK surface (plus embedding the gateway binary into compiled Workers via packages/package-layout and crates/alien-build).
crates/alien-infra — the AI controllers and runtime wiring.
crates/alien-terraform + crates/alien-cloudformation — the per-cloud setup emitters and their snapshots.
crates/alien-permissions/permission-sets/ai/* — the ai/invoke grant (plus heartbeat/management/provision).
examples/ai-quickstart-ts — a runnable example.

How I tested

Manually: From a deployed test worker, real Claude requests went through the gateway with no API keys configured — Bedrock (AWS) and the Foundry Anthropic endpoint (Azure) both returned completions, including streaming. I also ran the gateway on my machine against real Bedrock and hit /v1/models: 36 of the 42 AWS catalog models came back listed and enriched, and the 6 dropped were Claude ids that account/region cannot serve.
Compiled Worker: bun build --compile of a Worker that calls ai() — the embedded gateway binary is extracted and spawned from inside the single-file binary and serves a loopback URL (the package-layout compile smoke).
Unit tests: cargo nextest across the workspace (CI-fast filter), including the availability-filter suite (mock upstream: an enabled model is kept and enriched, a 403 model is dropped, a 500 keeps the model and re-probes on the next call, a second call is served from cache); SDK vitest + biome; terraform validate on the AI setup-emitter output and the CloudFormation snapshot suite; generated-schema drift check clean. The live Bedrock test reads the shared test env and runs in the credentialed job.
E2E (ran green on real Bedrock): the comprehensive TypeScript app lists every model getAvailableModels returns and invokes each one, failing if a listed model is not invocable. That run passed after it caught a real filter bug: Bedrock answers 400 "The provided model identifier is invalid" for a model the account cannot address, and I had been counting 400 as available, so it now drops the model (400 joins 401/403/404) and the list only ever contains invocable models.
Anything I couldn't test: Vertex (GCP) live inference is pending a model-garden entitlement on the test project — unrelated to this code; the GCP emitter path is covered by snapshots + terraform validate.

One non-AI fix rides along

The cloud e2e for this PR overflowed its test thread's stack during deploy. It is a pre-existing issue, independent of the AI code: the deployment reconcile polls a deep async chain (executor to controllers to the cloud SDKs' tower/hyper/rustls futures) that needs about 3MB in a debug build, over tokio's 2MB default thread stack. Release binaries fit 2MB and are unaffected. RUST_MIN_STACK=8MB for dev/test threads via .cargo/config.toml fixes it, matching the headroom alien-worker-runtime already gives its runtime. It rides along here because this PR's own e2e could not run without it.

I also ran a security review on the diff. What it checked:

Over-scoped model grant. ai/invoke grants inference only — AWS bedrock:InvokeModel* (+ mantle CreateInference scoped to project/default, not project/*); a GCP custom role of just aiplatform.endpoints.predict/explain (not roles/aiplatform.user, which carries deploy + dataset-export); Azure "Cognitive Services OpenAI User" (data-plane only). No deployment writes or data reads. crates/alien-permissions/permission-sets/ai/invoke.jsonc.
Availability probe surface. The /v1/models probe sends a one-token inference request signed with that same inference-only grant; it adds no control-plane permission, cannot enable or subscribe anything, and its logs carry only the model id and status, never request or response bodies. crates/alien-ai-gateway/src/availability.rs.
Credential exposure. On the keyless path the gateway signs with the workload's ambient cloud identity — no key is stored. On the BYO-key path the provider key is a vault-resolved secret: its Debug impl redacts it (crates/alien-core/src/bindings/ai.rs) and it's resolved into the worker environment, not logged or held in control-plane state. crates/alien-ai-gateway/src/config.rs.
Off-host reachability. The gateway binds the loopback interface and hands the app its URL out of band, so it isn't reachable from other hosts. crates/alien-ai-gateway/src/lib.rs.
BYO-key providers. An external (bring-your-own-key) provider is explicitly not served by the ambient-credential gateway path. crates/alien-ai-gateway/src/config.rs.

Nothing turned up.

Continues #178 (same commits, rebased on main). That PR closed automatically when this branch was renamed.

ItamarZand88 · 2026-07-26T10:24:10Z

@greptile-apps

greptile-apps · 2026-07-26T10:27:34Z

Greptile Summary

Adds an OpenAI-compatible AI resource with ambient-cloud and bring-your-own-key modes.

Introduces a Rust loopback gateway for Bedrock, Vertex AI, and Azure Foundry, including request translation, signing, streaming, and model-availability probing.
Adds AI resource types, SDK bindings, provisioning controllers, inference permissions, packaging, release workflows, and a TypeScript quickstart.
Updates the dev and stable release pipelines to build, package, qualify, and publish the gateway wrapper and platform binaries.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the previously reported dev-package availability, request-body limit, qualification checkout and publication, dry-run publication, and final-release dependency issues are addressed in the current code.

Important Files Changed

Filename	Overview
crates/alien-ai-gateway/src/router/mod.rs	Adds the gateway routing, translation, structured errors, and a tested 32 MB request-body limit across all body-reading routes.
crates/alien-ai-gateway/src/availability.rs	Adds credential-backed model probes and caches only fully resolved availability results.
packages/ai-gateway/src/gateway.ts	Adds lazy gateway process startup, readiness handling, and lifecycle management for ambient AI bindings.
packages/ai-gateway/src/loader.ts	Resolves embedded, packaged, overridden, or locally built gateway binaries for supported platforms.
.github/workflows/publish-npm-dev.yml	Builds and publishes immutable dev versions of the gateway wrapper and platform packages alongside the SDK.
.github/workflows/release.yml	Qualifies gateway artifacts separately, publishes them only during publication, handles dry runs without registry writes, and makes final release creation depend on gateway publication.
packages/sdk/package.json	Adds the AI gateway as an SDK dependency participating in immutable version rewriting.
crates/alien-permissions/permission-sets/ai/invoke.jsonc	Adds cloud-specific model-inference permissions for ambient gateway requests.

Sequence Diagram

sequenceDiagram
  participant App as Workload
  participant SDK as Alien SDK
  participant Gateway as Loopback AI Gateway
  participant Cloud as Host Cloud Model API

  App->>SDK: Create AI client
  alt Ambient cloud credentials
    SDK->>Gateway: OpenAI-compatible request
    Gateway->>Cloud: Translate and sign native request
    Cloud-->>Gateway: Native response or token stream
    Gateway-->>SDK: OpenAI-compatible response
  else Bring-your-own key
    SDK->>Cloud: Direct provider request with resolved key
    Cloud-->>SDK: Provider response
  end
  SDK-->>App: Typed completion response

_{Reviews (9): Last reviewed commit: "fix(ci): keep a dry-run release from pub..." | Re-trigger Greptile}

ItamarZand88 · 2026-07-26T12:26:50Z

@greptile-apps

ItamarZand88 · 2026-07-26T16:09:36Z

@greptile-apps

ItamarZand88 · 2026-07-26T21:36:36Z

@greptile-apps

ItamarZand88 · 2026-07-26T21:56:46Z

@greptile-apps

ItamarZand88 · 2026-07-26T22:04:37Z

@greptile-apps

ItamarZand88 · 2026-07-26T22:31:57Z

@greptile-apps

Introduce built-in AI: an `ai` resource that provisions keyless, in-account model access through an embedded gateway. Adds the Rust gateway engine and napi addon, the TypeScript SDK wrapper, the per-cloud controllers and setup emitters (AWS Bedrock, GCP Vertex, Azure Foundry), the ai/invoke permission sets, and quickstart examples.

- The napi error envelope now carries httpStatusCode and hint, so a gateway startup failure (e.g. BindingConfigInvalid at http_status_code 400) is no longer flattened to a 500 when it crosses into JS. - parseSse preserves the malformed-chunk cause as a structured error source instead of folding it into the message string. - Name the condition the anthropic-beta merge test enforces rather than the old bug it guards against.

…ration test /v1/models now probes every catalog model against the mock upstream, and an unmatched probe 404s and drops the model, so the Claude catalog assertion needs the InvokeModel path answered.

The deployment reconcile polls a deep async chain on a single stack: the executor drives each resource controller, which drives the cloud SDKs' tower/hyper/rustls futures. In debug builds those frames are un-inlined and the chain needs about 3MB, over tokio's 2MB default thread stack, so the alien-test cloud e2e (push_/pull_) overflows its libtest test thread and aborts during initial setup. Release binaries are unaffected (inlined frames fit the 2MB default, confirmed by running the same deploy in release), so this only sizes dev/test threads via a cargo [env] entry. 8MB matches the headroom alien-worker-runtime already gives its runtime for the same deep-async reason.

The availability probe counted 400 as "enabled", assuming it proved the endpoint authed and only rejected the minimal probe body. The cloud e2e disproved that: Bedrock answers "The provided model identifier is invalid" with 400 for a model the account cannot address, so qwen3-coder-next and claude-mythos-5 were listed by getAvailableModels and then failed on the first real call. The probe body is minimal and well formed, so a 400 is about the model rather than the request. Classifying it as unavailable keeps the contract the e2e asserts: every listed model is invocable. A model that only ever answers 400 now drops out of the list instead of being advertised and breaking at runtime.

Draining the child's stdio with resume() keeps the pipe sockets referenced, so a process that calls ai() once and returns never exits. Unref both stdio sockets alongside the child.

The native controller enables aiplatform.googleapis.com at runtime, but a Frozen/Terraform GCP deploy relies on this activation being injected here, or the first Vertex call fails with SERVICE_DISABLED.

Hold the metadata cache lock across the IMDS fetch so a burst of concurrent probes (the /v1/models fan-out authorizes every model at once) collapses to one refresh instead of stampeding the metadata service.

providerBaseUrl defaulted any provider other than "anthropic" to OpenAI's endpoint, and the projected BYO key is attached as a bearer token there, so a typo or unlisted provider shipped the customer's key to OpenAI. Resolve only the providers we have a base URL for and throw otherwise; the ALIEN_AI_LOCAL_BASE_URL override still allows any OpenAI-compatible provider.

The BYO-Anthropic model picker returned retired claude-3-5-* ids. Return the current-generation aliases api.anthropic.com serves instead.

router.rs had grown to hold the transport, per-cloud URL dispatch, three provider adapters, the Bedrock body normalization, and the event-stream codec in one file. Move each provider's request path into its own module under router/ (bedrock/vertex/foundry) plus the event-stream decoder (eventstream), keeping mod.rs as the transport and dispatch. Behavior-preserving; the shared integration test harness stays in mod.rs.

The gateway is a pure proxy, so the upstream enforces its own body size, but axum's 2 MB default rejected large legitimate requests (base64 vision images, long tool-heavy conversations) with 413 before they ever left us. Disable the limit on the router.

reqwest has no default request timeout, so a hung instance metadata service would wedge /v1/models forever — and holding the single-flight lock across the fetch makes that block every waiting caller, not just one. Cap the fetch at 10s (the endpoint is link-local and answers in well under a second) so a real hang fails fast instead of hanging.

Anthropic's OpenAI-compatible host serves /v1/chat/completions but not the Responses API, so a BYO-Anthropic responses.create() returned an opaque 404. Detect the Anthropic native base and throw a clear reason before the request; an ALIEN_AI_LOCAL_BASE_URL override resolves elsewhere and is not blocked.

The three new AI ImportData types were never registered in the snapshot map, so the pinned schemas had no AI entry and the test passed without covering them.

Claude over classic InvokeModel signs with bedrock, not bedrock-mantle; bedrock-mantle serves the Responses API. The doc had the two the other way round.

Two defects in the same manifest. The wrapper version had drifted from the Cargo workspace and every sibling npm package, so validate-release-version.mjs would reject the next release and the loader would look for a prebuild tag that is never published. Separately, the OpenAI result types reach the emitted client.d.ts, so `openai` cannot be a devDependency alone: a consumer typechecking against us gets TS2307 unless skipLibCheck hides it. Declaring it an optional peer dependency matches how packages/testing declares its own optional peers, and needs no lockfile change.

…ror shape Disabling the limit outright let any colocated process make the gateway buffer, parse, and reserialize an arbitrarily large payload. 32 MB clears the largest upstream request limit we serve (Bedrock InvokeModel, 25 MB) so the upstream still owns rejecting its own oversized payloads. axum answers its own body-limit rejection as bare plain text, which would leave this the one gateway failure a caller cannot parse as JSON, so a wrapper extractor maps it to ErrorData::RequestTooLarge.

The sdk depends on @alienplatform/ai-gateway at runtime, but the dev release neither versioned nor packed it, so the sdk kept an unresolvable workspace:^ range and the smoke install rejected it before anything could publish. The channel now builds the launcher for each platform alongside the napi addon and publishes the six prebuild packages ahead of the wrapper that pins them, so an ambient AI call resolves a binary instead of exhausting the loader. The publish loop is an allowlist of globs, so it now fails when a packed package matches none of them. Also unroots the hardcoded core version in the test beside it, which stopped matching once core left 1.14.x.

The reason named a specific target account and its pending grant status. The other live suites state only the grant the test needs.

The gating matrix requires every registered emitter type to be policy-refused or proven to render gated, and the AI emitters are registered on all three clouds, so the branch cannot merge without choosing. The TS builder joins ResourceBuilder for the gate, the type joins the gateability manifest, and the matrix gains ai fixtures plus per-cloud snapshots. The dedicated tests pin what matters on a declined deploy: every inference grant rides the gate through its own emission path (the ai emitter role policies on AWS, the service-account emitter binding as a gated contribution on GCP, the ai emitter role assignments on Azure), and the cognitive account and its model deployments carry it too.

The generated module README documents the deployment_input_values output; the ai minimal renders predate that text. Documentation-only drift, no resource changes.

Bundled cloud Workers run in the versioned Worker base and package only their bundle and source map, so the launcher the ai() client spawns has to live in the base image, the same way the bindings addon does. The loader resolves ./alien-ai-gateway from the bundle working directory before the prebuild route, which a bundle cannot use (no node_modules). Compiled host-process workloads keep the embedded route. The e2e build jobs compile the musl launcher alongside alien-worker-runtime so the base image job can stage it per arch.

A cloud Worker is bundled, not compiled, since the runtime moved into the versioned base image. The bootstrap still emitted the SDK native bridge, which registers both embedded pieces, and outside `bun build --compile` the gateway import degrades to the literal asset path `./alien-ai-gateway.bin` that no bundle packages. Registering it won the loader race against the launcher the base image stages, so every ai() call in a Worker failed on an unextractable embedded path. A bindings-only bridge leaves the gateway to the staged launcher and stops the bundle depending on an asset that staging documents as best-effort. The e2e handler reports the unsanitized error so a cloud-only failure can name itself.

The job built and published the gateway packages in one step during qualification, so an abandoned or amended release left permanent npm versions that were never promoted, and the release could finalize without them because nothing depended on it. Split it the way bindings already is: qualification packs the prebuilds and the wrapper into a qualified artifact and publishes nothing, publish resolves that artifact from the qualification run and publishes prebuilds before the wrapper that pins them. The stable channel now waits for that publish, so a released sdk cannot reference a gateway package that was never pushed.

main released 3.1.0 while this branch was open. The wrapper is not in the release bump list until it merges, so its version is carried by hand; validate-release-version.mjs holds it against the Cargo workspace.

ItamarZand88 · 2026-07-27T10:00:36Z

@greptile-apps

The split dropped the dry-run branch the single job used to carry, leaving the publish step to reach the registry on a validation run. A published version cannot be withdrawn, so the flag is explicit here rather than resting on an upstream job failing first.

ItamarZand88 · 2026-07-27T10:23:37Z

@greptile-apps

greptile-apps Bot reviewed Jul 26, 2026

View reviewed changes

Comment thread packages/sdk/package.json

Comment thread crates/alien-ai-gateway/src/router/mod.rs Outdated

ItamarZand88 force-pushed the itamar/alien-39-ai-gateway-resource branch 2 times, most recently from a768563 to f5d5ed4 Compare July 26, 2026 11:59

greptile-apps Bot reviewed Jul 26, 2026

View reviewed changes

Comment thread .github/workflows/publish-npm-dev.yml

ItamarZand88 force-pushed the itamar/alien-39-ai-gateway-resource branch from f5d5ed4 to 7b0f0a2 Compare July 26, 2026 16:09

ItamarZand88 had a problem deploying to e2e-tests July 26, 2026 17:32 — with GitHub Actions Failure

ItamarZand88 had a problem deploying to e2e-tests July 26, 2026 17:40 — with GitHub Actions Failure

ItamarZand88 had a problem deploying to e2e-tests July 26, 2026 18:55 — with GitHub Actions Failure

ItamarZand88 had a problem deploying to e2e-tests July 26, 2026 19:03 — with GitHub Actions Failure

ItamarZand88 force-pushed the itamar/alien-39-ai-gateway-resource branch from 97c930b to 688a649 Compare July 26, 2026 19:38

ItamarZand88 had a problem deploying to e2e-tests July 26, 2026 19:43 — with GitHub Actions Failure

ItamarZand88 force-pushed the itamar/alien-39-ai-gateway-resource branch from 688a649 to 9a1d264 Compare July 26, 2026 20:06

ItamarZand88 had a problem deploying to e2e-tests July 26, 2026 20:10 — with GitHub Actions Failure

ItamarZand88 had a problem deploying to e2e-tests July 26, 2026 20:17 — with GitHub Actions Failure

ItamarZand88 had a problem deploying to e2e-tests July 26, 2026 21:07 — with GitHub Actions Failure

ItamarZand88 deployed to e2e-tests July 26, 2026 21:14 — with GitHub Actions Active

greptile-apps Bot reviewed Jul 26, 2026

View reviewed changes

Comment thread .github/workflows/release.yml Outdated

ItamarZand88 force-pushed the itamar/alien-39-ai-gateway-resource branch from 5802108 to 612f0dd Compare July 26, 2026 22:04

greptile-apps Bot reviewed Jul 26, 2026

View reviewed changes

Comment thread .github/workflows/release.yml Outdated

ItamarZand88 force-pushed the itamar/alien-39-ai-gateway-resource branch from 612f0dd to 4f20319 Compare July 26, 2026 22:31

ItamarZand88 added 2 commits July 27, 2026 12:32

ItamarZand88 added 25 commits July 27, 2026 12:33

test(ai): answer the Claude availability probe in the two-cloud integ…

631ad71

…ration test /v1/models now probes every catalog model against the mock upstream, and an unmatched probe 404s and drops the model, so the Claude catalog assertion needs the InvokeModel path answered.

fix(ai): let one-shot hosts exit by unref-ing the gateway stdio

c287fe6

Draining the child's stdio with resume() keeps the pipe sockets referenced, so a process that calls ai() once and returns never exits. Unref both stdio sockets alongside the child.

fix(ai): enable aiplatform on the GCP Terraform path

d1c4ef6

The native controller enables aiplatform.googleapis.com at runtime, but a Frozen/Terraform GCP deploy relies on this activation being injected here, or the first Vertex call fails with SERVICE_DISABLED.

fix(ai): refresh the ambient bearer token single-flight

d618be7

Hold the metadata cache lock across the IMDS fetch so a burst of concurrent probes (the /v1/models fan-out authorizes every model at once) collapses to one refresh instead of stampeding the metadata service.

fix(ai): point BYO-Anthropic defaults at active models

f4eeec4

The BYO-Anthropic model picker returned retired claude-3-5-* ids. Return the current-generation aliases api.anthropic.com serves instead.

test(ai): cover the ai import-data types in the schema snapshot

2d2f921

The three new AI ImportData types were never registered in the snapshot map, so the pinned schemas had no AI entry and the test passed without covering them.

docs(ai): correct the AWS signing-service names in the creds module doc

af74585

Claude over classic InvokeModel signs with bedrock, not bedrock-mantle; bedrock-mantle serves the Responses API. The doc had the two the other way round.

chore(ci): tighten the ai-gateway test filter comment

3239828

chore(ai): trim internal detail from a live-test ignore reason

ef8c977

The reason named a specific target account and its pending grant status. The other live suites state only the grant the test needs.

test(ai): refresh the ai render snapshots for the registration doc

058e953

The generated module README documents the deployment_input_values output; the ai minimal renders predate that text. Documentation-only drift, no resource changes.

chore(ai): follow the workspace to 3.1.0

e37599c

main released 3.1.0 while this branch was open. The wrapper is not in the release bump list until it merges, so its version is carried by hand; validate-release-version.mjs holds it against the Cargo workspace.

ItamarZand88 force-pushed the itamar/alien-39-ai-gateway-resource branch from 4f20319 to e37599c Compare July 27, 2026 09:39

greptile-apps Bot reviewed Jul 27, 2026

View reviewed changes

Comment thread .github/workflows/release.yml

Uh oh!

Conversation

ItamarZand88 commented Jul 24, 2026

Summary

What I did

Files touched

How I tested

One non-AI fix rides along

Uh oh!

ItamarZand88 commented Jul 26, 2026

Uh oh!

greptile-apps Bot commented Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Sequence Diagram

Uh oh!

Uh oh!

Uh oh!

ItamarZand88 commented Jul 26, 2026

Uh oh!

Uh oh!

ItamarZand88 commented Jul 26, 2026

Uh oh!

ItamarZand88 commented Jul 26, 2026

Uh oh!

Uh oh!

ItamarZand88 commented Jul 26, 2026

Uh oh!

ItamarZand88 commented Jul 26, 2026

Uh oh!

Uh oh!

ItamarZand88 commented Jul 26, 2026

Uh oh!

ItamarZand88 commented Jul 27, 2026

Uh oh!

Uh oh!

ItamarZand88 commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

greptile-apps Bot commented Jul 26, 2026 •

edited

Loading