diff --git a/src/content/changelog/agents/2026-10-14-createai-model-providers.mdx b/src/content/changelog/agents/2026-10-14-createai-model-providers.mdx new file mode 100644 index 00000000000..9ab15467551 --- /dev/null +++ b/src/content/changelog/agents/2026-10-14-createai-model-providers.mdx @@ -0,0 +1,56 @@ +--- +title: createAI, one model provider for the AI SDK and Pi AI +description: The Agents SDK's createAI calls Workers AI, the AI Gateway catalog, dynamic routes, and vendor models through the AI binding, from the AI SDK or Pi AI. It replaces the binding mode of workers-ai-provider and ai-gateway-provider. +products: + - agents + - workers +date: 2026-10-14 +--- + +import { TypeScriptExample } from "~/components"; + +The Agents SDK now ships [`createAI`](/agents/models/), one model provider for Workers AI and AI Gateway over the Worker's `AI` binding. There are two imports, with the same options: + +- `agents/models/ai-sdk` for the [AI SDK](/agents/models/ai-sdk/). It replaces the `AI` binding mode of `workers-ai-provider` and `ai-gateway-provider`. +- `agents/models/pi-ai` for [Pi AI](/agents/models/pi-ai/) and the [Pi harness](/agents/harnesses/pi/). It gives Pi AI the same binding-based support. + +The [OpenCode harness](/agents/harnesses/opencode/) uses the AI SDK provider through `agents/models/opencode`. + + + +```ts +import { createAI } from "agents/models/ai-sdk"; +import { streamText } from "ai"; + +export default { + async fetch(request: Request, env: Env) { + const ai = createAI({ binding: env.AI, id: "my-gateway" }); + + const result = streamText({ + model: ai("anthropic/claude-opus-5.5", { + fallback: ["@cf/zai-org/glm-4.7-flash"], + }), + prompt: "Write a haiku about Durable Objects", + }); + + return result.toTextStreamResponse(); + }, +}; +``` + + + +With `createAI` you get: + +- **Every model by id.** Workers AI `@cf/` ids, any model in the [AI Gateway catalog](/ai/models/) such as `anthropic/claude-opus-5.5`, and [dynamic routes](/ai-gateway/features/dynamic-routing/) as `dynamic/`. +- **Vendor model objects.** Pass a model from the vendor's own package, such as `ai(anthropic("claude-opus-5-5"))`. The request goes unchanged to AI Gateway's [provider-native endpoint](/ai-gateway/usage/providers/). +- **No API tokens.** Every request goes through the `AI` binding. AI Gateway bills through [Unified Billing](/ai-gateway/features/unified-billing/) or a [stored key](/ai-gateway/configuration/bring-your-own-keys/). `byokAlias` picks a stored key, `zdr` sets Zero Data Retention, and `vendorKey: "request"` sends a key you hold. +- **Gateway options.** Caching, logging, metadata, timeouts, and retries on the provider, the model, or the call. +- **Fallback.** Try models in order if one fails before it answers. Mix ids and model objects. +- **Stream resume.** A catalog or dynamic route stream that drops mid-answer resumes, with nothing lost or repeated. +- **Gateway metadata and typed errors.** Each result carries the log id, cache status, run id, and the model that answered. Failures throw `CloudflareAIError`. +- **Other model types.** Embeddings, images, speech, transcription, and reranking from Workers AI in the AI SDK provider, and images in Pi AI. + +`createAI` does not use AI Gateway's deprecated [Universal Endpoint](/ai-gateway/usage/universal/). Once `createAI` covers everything `workers-ai-provider` and `ai-gateway-provider` do, those packages will be deprecated. + +For more information, refer to [Models](/agents/models/). diff --git a/src/content/docs/agents/harnesses/opencode.mdx b/src/content/docs/agents/harnesses/opencode.mdx index 06bd00bd040..35824d3ea1c 100644 --- a/src/content/docs/agents/harnesses/opencode.mdx +++ b/src/content/docs/agents/harnesses/opencode.mdx @@ -1,6 +1,6 @@ --- title: OpenCode -description: Run OpenCode 2.0 in a plain Durable Object with the Agents SDK, Workers AI models, bundled plugins, and durable sessions. +description: Run OpenCode 2.0 in a plain Durable Object with the Agents SDK, Workers AI and AI Gateway models, bundled plugins, and durable sessions. pcx_content_type: overview sidebar: order: 11 @@ -18,7 +18,7 @@ import { [OpenCode](https://opencode.ai/v2/docs/build/sdk/) is an extensible agent harness. The Agents SDK's `OpenCodeHarness` runs its Workerd SDK inside a plain Durable Object. Refer to [OpenCode's Cloudflare SDK documentation](https://opencode.ai/v2/docs/build/sdk/cloudflare/) for the embedded host API. -OpenCode owns the agent loop, transcript, tools, and session recovery. `OpenCodeHarness` supplies SQLite storage and wakes the object after an interrupted run. The `agents/models/opencode` provider connects OpenCode to Workers AI through your `AI` binding. +OpenCode owns the agent loop, transcript, tools, and session recovery. `OpenCodeHarness` supplies SQLite storage and wakes the object after an interrupted run. The `agents/models/opencode` provider connects OpenCode to Workers AI models, [AI Gateway](/ai-gateway/) catalog models, and dynamic routes through your `AI` binding. sqlite[("SQLite: opencode_ tables")] host <--> plugins["Bundled plugins"] end - host --> ai["Workers AI binding"] + host --> ai["AI binding"] plugins --> mcp["Remote MCP server"] plugins --> workspace["External execution environment"] ``` @@ -52,7 +52,7 @@ flowchart LR | Capability | Support | | ---------------------------------------------- | -------------------------------------------------------- | | Durable sessions and interrupted-turn recovery | OpenCode persists state; Lifecycle jobs wake the object. | -| Workers AI models | Register the `agents/models/opencode` plugin. | +| Workers AI and AI Gateway models | Register the `agents/models/opencode` plugin. | | Custom tools, agents, and skills | Bundle OpenCode plugins with your Worker. | | Remote Model Context Protocol (MCP) servers | Register remote servers through a plugin. | | Live events and durable event replay | Use `session.events()` and `session.log()`. | @@ -63,9 +63,9 @@ Install the Agents SDK and OpenCode packages: -Use `agents` version 0.28.1 or later. The OpenCode packages are optional peer dependencies, required by this integration, with a supported range of `^2.0.24`. +Use the latest `agents` release. The OpenCode packages are optional peer dependencies, required by this integration, with a supported range of `^2.0.24`. -Add the Workers AI binding, a SQLite-backed Durable Object, and `nodejs_compat` to your Wrangler configuration: +Add the `AI` binding, a SQLite-backed Durable Object, and `nodejs_compat` to your Wrangler configuration: @@ -172,12 +172,26 @@ export default { -Pass the factory's `storage` to `OpenCodeWorkerd.create()` so OpenCode uses the prefixed database. Register `this.ai.plugin` to make the Workers AI models available. `models: { fetch: false }` skips OpenCode's external model catalog refresh; the provider registers models you name with `this.ai()`. +Pass the factory's `storage` to `OpenCodeWorkerd.create()` so OpenCode uses the prefixed database. Register `this.ai.plugin` to add a provider named "Cloudflare" (id `cloudflare`) to OpenCode. `models: { fetch: false }` skips OpenCode's external model catalog refresh. The provider registers the models you name with `this.ai()`. The Lifecycle manages host startup. The factory runs again when the harness reopens an idle host, so it must support repeated calls. The `"demo"` object name is for this example. In an application, choose an object name based on an authenticated user or conversation. +## Choose a model + +`this.ai()` takes the same model ids as the [AI SDK provider](/agents/models/ai-sdk/), and accepts its model options: + +```ts +this.ai("@cf/moonshotai/kimi-k2.7-code"); // Workers AI +this.ai("anthropic/claude-opus-5.5"); // AI Gateway catalog +this.ai("dynamic/support"); // AI Gateway dynamic route +``` + +A catalog id uses the [model catalog](/ai/models/)'s spelling, such as `anthropic/claude-opus-5.5`. A dynamic route belongs to one gateway, so pass that gateway's `id` to `createAI`. An id that is not one of these three forms throws a `TypeError` on the line that names it. Streams from catalog models and dynamic routes [resume after a dropped connection](/agents/models/ai-sdk/#resume-a-dropped-stream). + +Each call returns OpenCode's model reference, `{ providerID, id }`. Pass it to `defaults.model` or `session.setModel()`. To set how OpenCode budgets a model's context, pass `limit: { context, output }`. The default is 128,000 context tokens and 8,192 output tokens. + ## Send prompts `harness.prompt()` submits to the root session and waits for completion. It returns `status`, optional `text` or `reason`, and the active transcript as `messages`. Check `status`: `"done"` means answered; `"unanswered"` means failed, interrupted, withdrawn, or not found. diff --git a/src/content/docs/agents/models/ai-sdk.mdx b/src/content/docs/agents/models/ai-sdk.mdx index 5f628c0ab15..222cf4c03e3 100644 --- a/src/content/docs/agents/models/ai-sdk.mdx +++ b/src/content/docs/agents/models/ai-sdk.mdx @@ -1,7 +1,7 @@ --- title: AI SDK pcx_content_type: how-to -description: Use createAI from agents/models/ai-sdk to call Workers AI models and route third-party AI SDK models through AI Gateway. +description: Use createAI from agents/models/ai-sdk to call Workers AI models, AI Gateway catalog models, dynamic routes, and vendor AI SDK models through the AI binding. tags: - AI sidebar: @@ -19,6 +19,22 @@ import { `agents/models/ai-sdk` is an [AI SDK](https://ai-sdk.dev/) provider for Workers AI and AI Gateway. `createAI` returns a full `ProviderV4`, so its models work anywhere the AI SDK takes a model: `generateText`, `streamText`, tools, structured output, and [Think](/agents/harnesses/think/). +The provider takes two kinds of model. Both go through the `AI` binding: + +```ts +// A string runs on env.AI.run. +ai("@cf/zai-org/glm-4.7-flash"); // Workers AI +ai("anthropic/claude-opus-5.5"); // AI Gateway catalog +ai("dynamic/support"); // AI Gateway dynamic route + +// A vendor's own model object goes to AI Gateway's provider-native endpoint. +ai(anthropic("claude-opus-5-5")); +``` + +A string needs no vendor package and no vendor key. A model object keeps the vendor's own request builder and parser. + +`createAI` replaces the `AI` binding mode of `workers-ai-provider` (`createWorkersAI({ binding })`) and `ai-gateway-provider` (`createAiGateway({ binding })`). Those packages will be deprecated once `createAI` covers everything they do. + :::caution[Beta] This provider is in beta. Its API may change in a minor release. ::: @@ -29,7 +45,7 @@ The provider needs `ai@^7` and `@ai-sdk/provider@^4`. Both are optional peer dep -Add a vendor's AI SDK provider only if you call that vendor's models, for example `@ai-sdk/anthropic` or `@ai-sdk/openai`. +Add a vendor's AI SDK provider only if you pass that vendor's model objects, for example `@ai-sdk/anthropic` or `@ai-sdk/openai`. Catalog ids do not need one. Add the `AI` binding to your Wrangler configuration: @@ -71,11 +87,61 @@ export default { -Any string that is not a `@cf/` id throws a `TypeError` that names the provider package to install. +A string that is not a Workers AI id, a catalog id, or a dynamic route throws a `TypeError` that names the three forms. + +## Call an AI Gateway catalog model + +Pass an `/` id from the [model catalog](/ai/models/). No vendor package or key is needed: + + + +```ts +import { createAI } from "agents/models/ai-sdk"; +import { generateText } from "ai"; + +export default { + async fetch(request: Request, env: Env) { + const ai = createAI({ binding: env.AI, id: "my-gateway" }); + + const { text } = await generateText({ + model: ai("anthropic/claude-opus-5.5"), + prompt: "Say hello in three words", + }); + + return Response.json({ text }); + }, +}; +``` + + + +Use the catalog's spelling of the id, not the vendor's. AI Gateway looks the model up by its exact catalog name, so `anthropic/claude-opus-5.5` works and `anthropic/claude-opus-5-5` does not. Other examples are `openai/gpt-5.5` and `cloudflare/auto`. + +The request goes through `env.AI.run()` to AI Gateway. The provider sends it in the OpenAI chat completions format, and AI Gateway converts it to and from the format of the provider that serves the model. The gateway also handles: + +- **Credentials.** [Unified Billing](/ai-gateway/features/unified-billing/) bills the model to your Cloudflare account. A key you [store on the gateway](/ai-gateway/configuration/bring-your-own-keys/) is used instead when one is configured. +- **Providers.** A catalog model can have several providers. AI Gateway picks one and retries across them. +- **Errors.** An unknown model, an unpaid account, or an upstream failure throws a `CloudflareAIError`, the same type as a Workers AI failure. + +Unified Billing requires [authentication](/ai-gateway/configuration/authentication/) on the gateway. Without it, the request fails with error `2049`. + +`reasoningEffort` is sent as `reasoning_effort`. `chatTemplateKwargs` is a Workers AI option, so the provider drops it with a warning. -## Call a third-party model +## Call a dynamic route -Build the model with the vendor's own AI SDK provider, then pass it to `ai()`. The request goes through AI Gateway, which holds the credential, so the vendor's `apiKey` is a placeholder: +An [AI Gateway dynamic route](/ai-gateway/features/dynamic-routing/) is named `dynamic/`. The route you configured on the gateway picks the model, and can apply conditions, percentages, rate and budget limits, and its own fallbacks: + +```ts +const ai = createAI({ binding: env.AI, id: "my-gateway" }); + +await generateText({ model: ai("dynamic/support"), prompt: "Hi" }); +``` + +A route belongs to one gateway, so set the `id` of the gateway you created it on. The route answers in the chat completions format whatever model it picks. Like a catalog model, it drops `chatTemplateKwargs` with a warning. A dynamic route is a language model only. The embedding, image, speech, transcription, and reranking methods reject it. + +## Call a vendor's model object + +Build the model with the vendor's own AI SDK provider, then pass it to `ai()`. AI Gateway holds the credential, so the vendor's `apiKey` is a placeholder: @@ -92,7 +158,7 @@ export default { const openai = createOpenAI({ apiKey: "cloudflare" }); const claude = await generateText({ - model: ai(anthropic("claude-opus-4-8")), + model: ai(anthropic("claude-opus-5-5")), prompt: "Say hello in three words", }); @@ -108,21 +174,51 @@ export default { -Use the vendor's own model id. The provider does not rewrite ids, so `claude-opus-4.8` reaches Anthropic as written and Anthropic rejects it. +The vendor's package builds the request and parses the response. The provider replaces the model's `fetch`, and sends the same request, with the same method, headers, and body, through the binding to AI Gateway's [provider-native endpoint](/ai-gateway/usage/providers/) for that vendor: + +```txt +https://workers-binding.ai/ai-gateway/gateways/{gateway}/{provider}/{endpoint} +``` + +The provider takes the gateway provider from the request host. For example, `api.anthropic.com` is `anthropic` and `api.openai.com` is `openai`. A host that AI Gateway has no provider for throws a `TypeError`. Gateway options travel as `cf-aig-*` headers. Multipart bodies, such as file uploads for transcription, work. + +Use the vendor's own model id, such as `claude-opus-5-5`. The vendor's package reads model capabilities from it. Providers on [Unified Billing](/ai-gateway/features/unified-billing/) answer with no key of yours. For any other provider, [store a key on the gateway](/ai-gateway/configuration/bring-your-own-keys/). Until you do, the vendor's own error comes back unchanged. `ai()` clones the model for each call and never changes the object you passed. If you add middleware, wrap the model with `ai()` first: ```ts -wrapLanguageModel({ model: ai(anthropic("claude-opus-4-8")), middleware }); +wrapLanguageModel({ model: ai(anthropic("claude-opus-5-5")), middleware }); ``` A model that is already wrapped in middleware hides the settings the provider needs, so `ai()` rejects it. +### Unified billing and BYOK + +By default, the provider removes the vendor's credential header (`authorization`, `x-api-key`, `x-goog-api-key`, or `api-key`) so AI Gateway supplies the key. To use a key you hold, give it to the vendor's provider and set `vendorKey: "request"` on the model: + +```ts +const anthropic = createAnthropic({ apiKey: env.ANTHROPIC_API_KEY }); +const model = ai(anthropic("claude-opus-5-5"), { vendorKey: "request" }); +``` + +`vendorKey` belongs to the model you set it on. A fallback model does not inherit it, so one chain can mix a model on your key with a model on Unified Billing. + +`byokAlias` picks a [stored key by alias](/ai-gateway/configuration/bring-your-own-keys/) when the gateway holds several keys for one provider. The provider sends it as `cf-aig-byok-alias` for catalog models and model objects. + +`zdr` turns [Zero Data Retention](/ai-gateway/features/unified-billing/#zero-data-retention-zdr) on (`true`) or off (`false`) for one Unified Billing request, whatever the gateway's own setting. The provider sends it as `cf-aig-zdr`. It applies to model objects only, because catalog models and dynamic routes do not read it. + +```ts +const model = ai(anthropic("claude-opus-5-5"), { + byokAlias: "production", + zdr: true, +}); +``` + ## Stream, call tools, and return structured output -Both kinds of model work with every AI SDK call: +Every kind of model works with every AI SDK call: @@ -173,6 +269,24 @@ export default { +## Resume a dropped stream + +A stream from a catalog model or a dynamic route resumes after a dropped connection. You do not need to configure anything: + +```ts +const result = streamText({ + model: ai("anthropic/claude-opus-5.5"), + prompt: "Write a haiku about Durable Objects", +}); +// If the connection drops, result.textStream keeps going. +``` + +AI Gateway names each streamed run in the `cf-aig-run-id` header and keeps its events. If the connection drops mid-stream, the provider reconnects through the binding and replays the run from the last complete event. Your stream continues with nothing lost or repeated. The run keeps going on the gateway while the connection is down, so the resumed stream continues the same answer. + +The provider gives up and throws a `CloudflareAIError` after five drops in one stream, or once the gateway no longer holds the run. The gateway keeps a run for a few minutes after it ends. Aborting the call never resumes it. + +Workers AI `@cf/` streams and model objects have no run id, so they do not resume. The run id is in `providerMetadata.cloudflare.runId`. + ## Use it in Think Return a model from `getModel()`: @@ -221,7 +335,7 @@ Attach `metadata` to the gateway log entry. Set it once on the provider and add const ai = createAI({ binding: env.AI, metadata: { app: "support-bot" } }); await generateText({ - model: ai(anthropic("claude-opus-4-8")), + model: ai(anthropic("claude-opus-5-5")), prompt: "Summarize this ticket", providerOptions: { cloudflare: { metadata: { userId: "u_123" }, eventId: "ticket-42" }, @@ -233,19 +347,21 @@ await generateText({ ## Fall back to another model -`fallback` lists models to try in order if the primary model fails before it produces any output. Mix Workers AI ids and model objects: +`fallback` lists models to try in order if the primary model fails before it produces any output. A fallback model can be a Workers AI id, a catalog id, a dynamic route, or a model object: ```ts -const model = ai(anthropic("claude-opus-4-8"), { - fallback: [ai(openai.responses("gpt-5-mini")), "@cf/zai-org/glm-4.7-flash"], +const model = ai(anthropic("claude-opus-5-5"), { + fallback: ["openai/gpt-5.5", "dynamic/support", "@cf/zai-org/glm-4.7-flash"], }); ``` +A model object in the list goes to the provider-native endpoint, like `ai(model)`. + A request that the vendor rejects falls back. So does a stream that fails before its first output part. Once a model has produced output, its answer is final, errors included. `providerMetadata.cloudflare.model` names the model that answered. -Each fallback model uses the chain's gateway options. The chain's `id`, `cacheTtl`, and the other gateway options win over a fallback model's own, and `metadata` merges. Headers and Workers AI options a fallback model was built with stay its own. +Each fallback model uses the chain's gateway options. The chain's `id`, `cacheTtl`, and the other gateway options win over a fallback model's own, and `metadata` merges. The `headers`, Workers AI options, and `vendorKey` a fallback model was built with stay its own. -Fallback runs in your Worker today. +Fallback runs in your Worker and moves across models. Within one catalog model, AI Gateway already retries across that model's providers, and a dynamic route runs its own fallback on the gateway. ## Set options @@ -256,7 +372,7 @@ Set gateway options on the provider, on a model, or per call under `providerOpti const ai = createAI({ binding: env.AI, id: "prod", metadata: { app: "demo" } }); // Model -const model = ai(anthropic("claude-opus-4-8"), { cacheTtl: 3600 }); +const model = ai("anthropic/claude-opus-5.5", { cacheTtl: 3600 }); // Call await generateText({ @@ -272,25 +388,28 @@ The provider removes `providerOptions.cloudflare` before the call reaches a vend Gateway options can be given flat or nested: `{ cacheTtl: 60 }` and `{ gateway: { cacheTtl: 60 } }` are the same. -| Option | Type | Description | -| -------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------ | -| `id` | `string` | Gateway id. Defaults to `"default"`, which is created the first time you use it. | -| `gateway` | `string \| GatewayOptions` | The gateway id, or the gateway options nested. | -| `skipCache` | `boolean` | Bypass the gateway cache. | -| `cacheTtl` | `number` | Cache lifetime in seconds. | -| `cacheKey` | `string` | Cache key to use instead of the one derived from the request. | -| `collectLog` | `boolean` | Whether the gateway stores the request and response in its log. | +| Option | Type | Description | +| -------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------ | +| `id` | `string` | Gateway id. Defaults to `"default"`, which is created the first time you use it. | +| `gateway` | `string \| GatewayOptions` | The gateway id, or the gateway options nested. | +| `skipCache` | `boolean` | Bypass the gateway cache. | +| `cacheTtl` | `number` | Cache lifetime in seconds. | +| `cacheKey` | `string` | Cache key to use instead of the one derived from the request. | +| `collectLog` | `boolean` | Whether the gateway stores the request and response in its log. | | `metadata` | `Record` | Values attached to the gateway log entry. | -| `eventId` | `string` | Groups several requests into one gateway event. | -| `requestTimeoutMs` | `number` | Timeout for the upstream request. | -| `retries` | `{ maxAttempts, retryDelayMs, backoff }` | Gateway retry policy for the upstream call. `maxAttempts` is 1 to 5. | -| `fallback` | `(WorkersAIModelId \| LanguageModelV4)[]` | Models to try in order. Model and call only. | -| `headers` | `Record` | Extra request headers, merged last. Model and call only. | -| `sessionAffinity` | `string` | Workers AI only. Sends requests with the same key to one replica for prefix caching. | -| `reasoningEffort` | `"low" \| "medium" \| "high" \| null` | Workers AI only. `null` turns reasoning off on models that allow it. | -| `chatTemplateKwargs` | `Record` | Workers AI only. Sent as `chat_template_kwargs`, for example `{ enable_thinking: false }`. | - -For a vendor's model, ask for reasoning with the AI SDK's `reasoning` call option or the vendor's own `providerOptions`. +| `eventId` | `string` | Groups several requests into one gateway event. | +| `requestTimeoutMs` | `number` | Timeout for the upstream request. | +| `retries` | `{ maxAttempts, retryDelayMs, backoff }` | Gateway retry policy for the upstream call. `maxAttempts` is 1 to 5. | +| `byokAlias` | `string` | A stored BYOK key, by alias. Catalog models and model objects. | +| `zdr` | `boolean` | Zero Data Retention on or off for a Unified Billing request. Model objects only. | +| `vendorKey` | `"gateway" \| "request"` | `"request"` forwards the vendor key on the request. Model objects only, per model. | +| `fallback` | `(CloudflareLanguageModelId \| LanguageModelV4)[]` | Models to try in order. Model and call only. | +| `headers` | `Record` | Extra request headers, merged last. Model and call only. | +| `sessionAffinity` | `string` | Workers AI only. Sends requests with the same key to one replica for prefix caching. | +| `reasoningEffort` | `"low" \| "medium" \| "high" \| null` | Sent as `reasoning_effort`. `null` turns reasoning off on models that allow it. | +| `chatTemplateKwargs` | `Record` | Workers AI only. Sent as `chat_template_kwargs`, for example `{ enable_thinking: false }`. | + +`reasoningEffort` applies to Workers AI models, catalog models, and dynamic routes. When it is not set, the AI SDK's `reasoning` call option maps onto it. For a model object, ask for reasoning with the AI SDK's `reasoning` call option or the vendor's own `providerOptions`. The provider ignores `reasoningEffort` and `chatTemplateKwargs` on a model object and adds a warning. ## Read gateway metadata @@ -298,13 +417,13 @@ Every result, and every stream's `finish` part, carries `providerMetadata.cloudf ```ts const { providerMetadata } = await generateText({ - model: ai(anthropic("claude-opus-4-8")), + model: ai(anthropic("claude-opus-5-5")), prompt: "hi", }); providerMetadata?.cloudflare; // { -// model: "claude-opus-4-8", +// model: "claude-opus-5-5", // provider: "anthropic", // gateway: "default", // logId: "01M1KZWN069WWNPC18V05NKHSS", @@ -313,24 +432,24 @@ providerMetadata?.cloudflare; // } ``` -| Field | Description | -| ------------- | -------------------------------------------------------------- | -| `model` | The model that answered, after any fallback. | -| `gateway` | The gateway id the request went through. | -| `provider` | The gateway provider, for a vendor's model. | -| `logId` | The gateway log entry, from `cf-aig-log-id`. | -| `eventId` | The gateway event, from `cf-aig-event-id`. | -| `requestId` | The gateway request id, from `cf-aig-request-id`. | -| `traceId` | The trace this request belongs to, from `cf-aig-trace-id`. | -| `runId` | From `cf-aig-run-id`, when the gateway sends one. | -| `cacheStatus` | `HIT` or `MISS`, from `cf-aig-cache-status`. | -| `step` | The step of a gateway route that answered, from `cf-aig-step`. | +| Field | Description | +| ------------- | ------------------------------------------------------------------------------- | +| `model` | The model that answered, after any fallback. | +| `gateway` | The gateway id the request went through. | +| `provider` | The gateway provider, for a model object. | +| `logId` | The gateway log entry, from `cf-aig-log-id`. | +| `eventId` | The gateway event, from `cf-aig-event-id`. | +| `requestId` | The gateway request id, from `cf-aig-request-id`. | +| `traceId` | The trace this request belongs to, from `cf-aig-trace-id`. | +| `runId` | The streamed run, from `cf-aig-run-id`, for a catalog model or a dynamic route. | +| `cacheStatus` | `HIT` or `MISS`, from `cf-aig-cache-status`. | +| `step` | The step of a gateway route that answered, from `cf-aig-step`. | Only `model` and `gateway` are always present. Check for the other fields before you use them. ## Handle errors -The provider throws `CloudflareAIError`, which extends the AI SDK's `APICallError`. The AI SDK's retry loop reads its `isRetryable` flag: +The provider throws `CloudflareAIError`, which extends the AI SDK's `APICallError`, for Workers AI failures, AI Gateway failures, failed stream resumes, and a fallback chain where every model failed. The AI SDK's retry loop reads its `isRetryable` flag: ```ts import { CloudflareAIError } from "agents/models/ai-sdk"; @@ -347,7 +466,7 @@ try { `code` is one of `"auth"`, `"rate-limit"`, `"not-found"`, `"bad-request"`, `"provider-error"`, `"gateway-error"`, or `"unknown"`. -For a vendor's model, `CloudflareAIError` covers the gateway's own failures, such as a missing gateway or an unpaid account. The vendor's own errors, such as an unknown model id or a rate limit, come from the vendor's provider in its own error type. +For a model object, `CloudflareAIError` covers the gateway's own failures, such as a missing gateway or an unpaid account. The vendor's own errors, such as an unknown model id or a rate limit, come from the vendor's provider in its own error type. ## Use other modalities @@ -400,7 +519,7 @@ Model-specific settings go under `providerOptions.cloudflare`: An option a model does not take is listed in the result's `warnings` instead of being sent. For example, `flux-1-schnell` takes only a prompt and `steps`. Image generation makes one image per request, so `generateImage` with a larger `n` sends parallel requests. -To route a vendor's own embedding, image, speech, or transcription model through AI Gateway, pass it to `ai.routed(model)`. `ai.routed()` takes the gateway options, `headers`, and `sessionAffinity`. +These methods take Workers AI `@cf/` ids only, including in `fallback`. To route a vendor's own embedding, image, speech, or transcription model through AI Gateway, pass it to `ai.routed(model)`. It goes to the provider-native endpoint, like `ai(model)`. `ai.routed()` takes the gateway options, `headers`, `sessionAffinity`, and `vendorKey`. ## Use a provider registry @@ -411,7 +530,7 @@ import { createProviderRegistry, generateText } from "ai"; const registry = createProviderRegistry({ cloudflare: ai }); await generateText({ - model: registry.languageModel("cloudflare:@cf/zai-org/glm-4.7-flash"), + model: registry.languageModel("cloudflare:anthropic/claude-opus-5.5"), prompt: "Hi", }); @@ -420,17 +539,20 @@ globalThis.AI_SDK_DEFAULT_PROVIDER = createAI({ binding: env.AI }); await generateText({ model: "@cf/zai-org/glm-4.7-flash", prompt: "Hi" }); ``` -A string still has to be a `@cf/` id. Reach a vendor's model with a model object. +A string can be a Workers AI id, a catalog id, or a dynamic route. ## Limitations -- AI Gateway's universal request carries JSON only. A multipart body, such as a file upload sent as `FormData`, throws a `TypeError`. - `@cf/deepgram/nova-3` and `@cf/deepgram/flux` are not supported. They throw a `CloudflareAIError` before any request is sent. Use `@cf/openai/whisper-large-v3-turbo` for transcription. -- `reasoningEffort`, `chatTemplateKwargs`, and `sessionAffinity` apply to Workers AI models only. +- `chatTemplateKwargs` and `sessionAffinity` apply to Workers AI models only. +- `zdr` applies to model objects only. +- Workers AI streams and model object streams do not resume after a dropped connection. ## Related resources - [pi-ai provider](/agents/models/pi-ai/) - [Model catalog](/ai/models/) +- [Dynamic routing](/ai-gateway/features/dynamic-routing/) +- [AI Gateway provider-native endpoints](/ai-gateway/usage/providers/) - [AI Gateway Worker binding methods](/ai-gateway/usage/worker-binding-methods/) - [Unified Billing](/ai-gateway/features/unified-billing/) diff --git a/src/content/docs/agents/models/index.mdx b/src/content/docs/agents/models/index.mdx index d62bfd3ae84..95650775572 100644 --- a/src/content/docs/agents/models/index.mdx +++ b/src/content/docs/agents/models/index.mdx @@ -1,7 +1,7 @@ --- title: Models pcx_content_type: overview -description: Call Workers AI and third-party models through AI Gateway from the AI SDK or Pi AI with one createAI provider over the AI binding. +description: Call Workers AI, AI Gateway catalog models, dynamic routes, and vendor models from the AI SDK or Pi AI with one createAI provider over the AI binding. tags: - AI sidebar: @@ -22,7 +22,7 @@ The Agents SDK includes model providers for [Workers AI](/workers-ai/) and [AI G | `agents/models/ai-sdk` | [AI SDK](https://ai-sdk.dev/) v7 | `generateText`, `streamText`, Think, and any AI SDK consumer | | `agents/models/pi-ai` | [Pi AI](https://github.com/earendil-works/pi/tree/main/packages/ai) | The [Pi harness](/agents/harnesses/pi/), and Pi AI's `stream` | -Both take the same options, route through the same gateway, and handle Workers AI the same way. +Both take the same options, route through the same gateway, and handle Workers AI the same way. The [OpenCode harness](/agents/harnesses/opencode/) has a third, `agents/models/opencode`, built on the AI SDK provider. :::caution[Beta] The model providers are in beta. Their APIs may change in a minor release. @@ -30,19 +30,31 @@ The model providers are in beta. Their APIs may change in a minor release. ## How a model is chosen -The providers keep one model catalog up to date: Workers AI. Every other vendor's models come from that vendor's own package, and the provider routes them through AI Gateway. +Name a model with a string, or pass a model object built by the vendor's own package: -- **A Workers AI model** is a `@cf/` id, such as `ai("@cf/zai-org/glm-4.7-flash")`. It runs through `env.AI.run()`. A compatibility layer turns each model's response into the standard OpenAI chat completions shape. -- **A third-party model** is a model object built by the vendor's provider, such as `ai(anthropic("claude-opus-4-8"))`. The vendor's code builds the request and parses the response. The provider swaps the transport, so the request goes through AI Gateway with `env.AI.gateway(id).run()`. +```ts +ai("@cf/zai-org/glm-4.7-flash"); // Workers AI +ai("anthropic/claude-opus-5.5"); // AI Gateway catalog +ai("dynamic/support"); // AI Gateway dynamic route +ai(anthropic("claude-opus-5-5")); // the vendor's own model object +``` + +- **A Workers AI model** is a `@cf/` id. It runs through `env.AI.run()`. A compatibility layer turns each model's response into the standard OpenAI chat completions shape. +- **An AI Gateway catalog model** is an `/` id from the [model catalog](/ai/models/), such as `anthropic/claude-opus-5.5` or `openai/gpt-5.5`. It also runs through `env.AI.run()`, which sends it to AI Gateway. The gateway picks a provider and bills the call. Use the catalog's spelling, not the vendor's: `anthropic/claude-opus-5.5`, not `claude-opus-5-5`. +- **A [dynamic route](/ai-gateway/features/dynamic-routing/)** is `dynamic/`. The route you configured on the gateway picks the model. +- **A model object** comes from the vendor's provider package, such as `anthropic("claude-opus-5-5")`. The vendor's code builds the request and parses the response. The provider sends that request unchanged through the binding to AI Gateway's [provider-native endpoint](/ai-gateway/usage/providers/) for that vendor. -The provider does not keep third-party model ids, wire formats, or thinking settings. When a vendor ships a new model, update the vendor's package. +The providers do not use the deprecated [Universal Endpoint](/ai-gateway/usage/universal/). ## What every model gets -- **No API tokens in your Worker.** Requests go through the `AI` binding. AI Gateway holds third-party credentials, through [Unified Billing](/ai-gateway/features/unified-billing/) or a key you [store on the gateway](/ai-gateway/configuration/bring-your-own-keys/). -- **Gateway options.** Caching, logging, metadata, timeouts, and retries are options on the provider, the model, or the call. The `default` gateway is created the first time you use it. -- **Fallback.** List models to try in order if the first fails before it produces output. Mix Workers AI and third-party models freely. -- **Gateway metadata on every result.** Each response carries the gateway log id, cache status, and the model that actually answered. +- **No API tokens in your Worker.** Requests go through the `AI` binding. AI Gateway holds third-party credentials, through [Unified Billing](/ai-gateway/features/unified-billing/) or a key you [store on the gateway](/ai-gateway/configuration/bring-your-own-keys/). Unified Billing requires [authentication](/ai-gateway/configuration/authentication/) on the gateway. +- **Gateway options.** Caching, logging, metadata, timeouts, retries, and BYOK key aliases are options on the provider, the model, or the call. The `default` gateway is created the first time you use it. +- **Fallback.** List models to try in order if the first fails before it produces output. Mix Workers AI ids, catalog ids, dynamic routes, and model objects. +- **Stream resume.** A stream from a catalog model or a dynamic route resumes after a dropped connection. You do not need to configure anything. +- **Gateway metadata on every result.** Each response carries the gateway log id, cache status, run id, and the model that actually answered. + +`createAI` from `agents/models/ai-sdk` replaces the `AI` binding mode of `workers-ai-provider` (`createWorkersAI({ binding })`) and `ai-gateway-provider` (`createAiGateway({ binding })`). Those packages will be deprecated once `createAI` covers everything they do. ## Configure the binding diff --git a/src/content/docs/agents/models/pi-ai.mdx b/src/content/docs/agents/models/pi-ai.mdx index 3a27c2096a5..0e45a457ff9 100644 --- a/src/content/docs/agents/models/pi-ai.mdx +++ b/src/content/docs/agents/models/pi-ai.mdx @@ -1,7 +1,7 @@ --- title: Pi AI pcx_content_type: how-to -description: Use createAI from agents/models/pi-ai to run Workers AI and third-party models through AI Gateway with Pi AI, Pi Durable, and the Pi harness. +description: Use createAI from agents/models/pi-ai to run Workers AI models, AI Gateway catalog models, dynamic routes, and Pi AI model objects through the AI binding with Pi AI, Pi Durable, and the Pi harness. tags: - AI sidebar: @@ -76,7 +76,7 @@ export default { `ai.stream()` returns Pi AI's `AssistantMessageEvent` stream. `ai.completeSimple()` and `ai.streamSimple()` take Pi AI's simple options, such as a `reasoning` level and `maxTokens`: ```ts -const model = ai("anthropic/claude-opus-4.8"); +const model = ai("anthropic/claude-opus-5.5"); const context = { messages: [{ role: "user", content: "Hi", timestamp: Date.now() }], }; @@ -92,30 +92,37 @@ await ai.completeSimple(model, context, { reasoning: "low" }); ## Name a model -There are three ways to name a model: +There are four ways to name a model: ```ts import { createAI } from "agents/models/pi-ai"; import { getBuiltinModel } from "@earendil-works/pi-ai/providers/all"; -const ai = createAI({ binding: env.AI }); +const ai = createAI({ binding: env.AI, id: "my-gateway" }); // 1. A Workers AI model, by its @cf/ id. ai("@cf/zai-org/glm-4.7-flash"); -// 2. A model from pi-ai's Cloudflare AI Gateway catalog, by /. -ai("anthropic/claude-opus-4.8"); +// 2. An AI Gateway dynamic route, by dynamic/. +ai("dynamic/support"); + +// 3. A model from Pi AI's Cloudflare AI Gateway catalog, by /. +ai("anthropic/claude-opus-5.5"); ai("openai/gpt-5.4"); -// 3. Any pi-ai model object, from any pi-ai registry. -ai(getBuiltinModel("anthropic", "claude-opus-4-8")); +// 4. Any Pi AI model object, from any Pi AI registry. +ai(getBuiltinModel("anthropic", "claude-opus-5-5")); ``` Any other string throws a `TypeError`. The provider does not guess what a prefix means. -The second and third forms keep Pi AI's model metadata, including `api`, `compat`, `thinkingLevelMap`, cost, context window, and maximum tokens. That metadata decides how each vendor is asked to think. For example, Claude Opus gets `thinking: { type: "adaptive" }` and GPT-5 gets `reasoning.effort`. `ai(model)` does not change the object you pass. +A [dynamic route](/ai-gateway/features/dynamic-routing/) runs on `env.AI.run()` under the gateway you created it on, so set that gateway's `id`. The route picks the model and answers in the chat completions format. It takes the same metadata defaults and overrides as a Workers AI id. + +A catalog id uses the gateway's spelling (`anthropic/claude-opus-5.5`). A model object from a vendor registry uses the vendor's spelling (`claude-opus-5-5`). -Third-party models go through AI Gateway, which holds the credential through [Unified Billing](/ai-gateway/features/unified-billing/) or a [key stored on the gateway](/ai-gateway/configuration/bring-your-own-keys/). Your Worker holds no provider key. +The third and fourth forms keep Pi AI's model metadata, including `api`, `compat`, `thinkingLevelMap`, cost, context window, and maximum tokens. That metadata decides how each vendor is asked to think. For example, Claude Opus gets `thinking: { type: "adaptive" }` and GPT-5 gets `reasoning.effort`. `ai(model)` does not change the object you pass. + +Third-party models go through AI Gateway, which holds the credential through [Unified Billing](/ai-gateway/features/unified-billing/) or a [key stored on the gateway](/ai-gateway/configuration/bring-your-own-keys/). Your Worker holds no provider key. Unified Billing requires [authentication](/ai-gateway/configuration/authentication/) on the gateway. ## Use it with the Pi harness @@ -133,12 +140,12 @@ models.setProvider(ai.provider); // defaults: { model: ai("@cf/moonshotai/kimi-k2.7-code") } // For one session: -await session.setModel(ai("anthropic/claude-opus-4.8")); +await session.setModel(ai("anthropic/claude-opus-5.5")); ``` -A Pi session stores only its model's provider and id, then looks the model up in the `Models` registry. Use the `@cf/` and `/` forms there: both belong to the `cloudflare` provider (`CLOUDFLARE_PROVIDER_ID`) that you registered. A model object keeps its vendor's provider id, which this registry does not know. Options passed to `ai(model, options)` do not reach a session. Set them on `createAI` instead. +A Pi session stores only its model's provider and id, then looks the model up in the `Models` registry. Use the `@cf/`, `dynamic/`, and `/` forms there. They belong to the `cloudflare` provider (`CLOUDFLARE_PROVIDER_ID`) that you registered. A model object keeps its vendor's provider id, which this registry does not know. Options passed to `ai(model, options)` do not reach a session. Set them on `createAI` instead. -The provider's model list is every Workers AI model Pi AI knows, plus every `/` in Pi AI's AI Gateway catalog, so `models.getModel("cloudflare", "anthropic/claude-opus-4.8")` resolves. +The provider's model list is every Workers AI model Pi AI knows, plus every `/` in Pi AI's AI Gateway catalog, so `models.getModel("cloudflare", "anthropic/claude-opus-5.5")` resolves. ## Cache, tag, and time out requests @@ -147,17 +154,20 @@ Gateway options go on `createAI`, on a model, or on a call. They can be flat or ```ts const ai = createAI({ binding: env.AI, id: "prod", cacheTtl: 60 }); -const model = ai("anthropic/claude-opus-4.8", { +const model = ai("anthropic/claude-opus-5.5", { cacheTtl: 3600, cacheKey: "greeting-v1", metadata: { team: "growth" }, eventId: "turn-17", requestTimeoutMs: 30_000, retries: { maxAttempts: 3, backoff: "exponential", retryDelayMs: 500 }, + byokAlias: "production", }); ``` -The options are the same as the [AI SDK provider's options](/agents/models/ai-sdk/#set-options): `id`, `skipCache`, `cacheTtl`, `cacheKey`, `collectLog`, `metadata`, `eventId`, `requestTimeoutMs`, `retries`, `fallback`, `headers`, `sessionAffinity`, `reasoningEffort`, and `chatTemplateKwargs`. +The options are the same as the [AI SDK provider's options](/agents/models/ai-sdk/#set-options): `id`, `skipCache`, `cacheTtl`, `cacheKey`, `collectLog`, `metadata`, `eventId`, `requestTimeoutMs`, `retries`, `byokAlias`, `zdr`, `fallback`, `headers`, `sessionAffinity`, `reasoningEffort`, and `chatTemplateKwargs`. `vendorKey` is AI SDK only. + +`byokAlias` picks a [stored key by alias](/ai-gateway/configuration/bring-your-own-keys/) when the gateway holds several keys for one provider. The provider sends it as `cf-aig-byok-alias` for catalog models and model objects. `zdr` turns [Zero Data Retention](/ai-gateway/features/unified-billing/#zero-data-retention-zdr) on or off for a Unified Billing request. It applies only to model objects outside Pi AI's gateway catalog, because catalog models do not read it. Pi AI's own call options map onto them. `headers` merge last, `metadata` joins the gateway metadata, `sessionId` sets session affinity, and `reasoning` maps through the model's own `thinkingLevelMap`. A call option wins over a model option, which wins over `createAI`. @@ -168,12 +178,12 @@ A Pi AI model is plain data. Per-model options are stored on it under symbol key ## Fall back to another model ```ts -ai(getBuiltinModel("anthropic", "claude-opus-4-8"), { +ai(getBuiltinModel("anthropic", "claude-opus-5-5"), { fallback: ["@cf/zai-org/glm-4.7-flash"], }); ``` -Fallback models can be Workers AI ids, `/` strings, or model objects. The provider tries them in order. It moves on when a stream ends with an `error` before producing any output. Once a model has produced output, its answer is final. +Fallback models can be Workers AI ids, dynamic routes, `/` strings, or model objects. The provider tries them in order. It moves on when a stream ends with an `error` before producing any output. Once a model has produced output, its answer is final. The message that answers has a `cloudflare-fallback` diagnostic listing the attempts that failed, and `message.model` names the model that answered. @@ -196,20 +206,32 @@ A failed request ends the stream with an `error` event. Its message has a `cloud The gateway's own failures, such as a missing gateway or an unpaid account, become a `CloudflareAIError`. Errors from the vendor come back as the vendor sent them. +## Resume a dropped stream + +A stream from a catalog model or a dynamic route resumes after a dropped connection. You do not need to configure anything. + +AI Gateway names each streamed run in the `cf-aig-run-id` header and keeps its events. If the connection drops, the provider reconnects through the binding and replays the run from the last complete event. The stream continues with nothing lost or repeated. + +The provider gives up after five drops in one stream, or once the gateway no longer holds the run. The gateway keeps a run for a few minutes after it ends. Aborting the call never resumes it. Workers AI `@cf/` streams and model objects outside the catalog have no run id, so they do not resume. The run id is on the `cloudflare` diagnostic as `runId`. + ## How requests are routed -The provider routes by the model's Pi AI `api`: +The provider routes every request through the `AI` binding. It has no HTTP transport and needs no API token: + +| Model | `api` | Transport | +| ---------------------------------------------------------- | -------------------- | -------------------------------------------------------------- | +| `@cf/` models and dynamic routes | `cloudflare-ai` | `env.AI.run()`, with Cloudflare's compatibility layer | +| Anthropic models from Pi AI's gateway catalog | `anthropic-messages` | `env.AI.run()` to AI Gateway, in Anthropic Messages format | +| OpenAI models from Pi AI's gateway catalog | `openai-responses` | `env.AI.run()` to AI Gateway, in OpenAI Responses format | +| Any other model object, such as Anthropic, OpenAI, or Groq | its own `api` | The binding's `fetch` to AI Gateway's provider-native endpoint | + +A model from Pi AI's Cloudflare AI Gateway catalog runs on `env.AI.run()` under its catalog id, in the model's native format. AI Gateway converts nothing, so Pi AI's own stream processors read the answer. The gateway picks the provider and bills through Unified Billing or a key stored on the gateway. -| Model | `api` | Transport | -| ----------------------------- | -------------------- | ----------------------------------------------------- | -| `@cf/` models | `cloudflare-ai` | `env.AI.run()`, with Cloudflare's compatibility layer | -| Anthropic, from any registry | `anthropic-messages` | AI Gateway universal request to `v1/messages` | -| OpenAI, from any registry | `openai-responses` | AI Gateway universal request to `v1/responses` | -| Groq, DeepSeek, xAI, and more | `openai-completions` | AI Gateway universal request to `v1/chat/completions` | +Any other model object goes to AI Gateway's [provider-native endpoint](/ai-gateway/usage/providers/) through the binding's `fetch`. The provider takes the gateway provider from the model's `baseUrl` host. Pi AI's own converter builds the vendor's request body, and Pi AI parses the vendor's response stream. -For a third-party model, Pi AI's own converter builds the vendor's request body. The provider sends it to AI Gateway, and Pi AI parses the vendor's response stream. The provider never rewrites the model's `api`, `provider`, or `id`, so thinking signatures and encrypted reasoning items work on the next turn. +Either way, no credential header leaves your Worker. The provider never rewrites the model's `api`, `provider`, or `id`, so thinking signatures and encrypted reasoning items work on the next turn. -These four `api` values are the only ones the provider routes. Models from several Pi AI registries use other values and cannot be routed yet, including `mistral-conversations`, `google-generative-ai`, `google-vertex`, `amazon-bedrock`, `azure-openai-responses`, and `openai-codex`. Such a model's stream ends with an `error` event that names its `api`. +The provider routes four `api` values: `cloudflare-ai`, `anthropic-messages`, `openai-responses`, and `openai-completions`. Models from several Pi AI registries use other values and cannot be routed yet, including `mistral-conversations`, `google-generative-ai`, `google-vertex`, `amazon-bedrock`, `azure-openai-responses`, and `openai-codex`. Such a model's stream ends with an `error` event that names its `api`. So does a model whose `baseUrl` host AI Gateway has no provider for. On the `openai-completions` wire, the provider sends reasoning only as OpenAI's `reasoning_effort`. A model whose `compat.thinkingFormat` is anything else, such as `deepseek`, `qwen`, or `openrouter`, has its reasoning level dropped. A `cloudflare-compat` diagnostic on the message records the drop. Use Pi AI's own provider for that vendor if you need its thinking format. @@ -233,4 +255,5 @@ const result = await ai.generateImages(images, { - [AI SDK provider](/agents/models/ai-sdk/) - [Pi harness](/agents/harnesses/pi/) - [Model catalog](/ai/models/) +- [Dynamic routing](/ai-gateway/features/dynamic-routing/) - [Pi AI](https://github.com/earendil-works/pi/tree/main/packages/ai)