diff --git a/public/images/custom-model/1.png b/public/images/custom-model/1.png deleted file mode 100644 index 5e0b9df4..00000000 Binary files a/public/images/custom-model/1.png and /dev/null differ diff --git a/public/images/custom-model/2.png b/public/images/custom-model/2.png deleted file mode 100644 index a18cca66..00000000 Binary files a/public/images/custom-model/2.png and /dev/null differ diff --git a/public/images/custom-model/3.png b/public/images/custom-model/3.png deleted file mode 100644 index fdec51af..00000000 Binary files a/public/images/custom-model/3.png and /dev/null differ diff --git a/public/images/custom-model/4.png b/public/images/custom-model/4.png deleted file mode 100644 index 51c947b4..00000000 Binary files a/public/images/custom-model/4.png and /dev/null differ diff --git a/public/images/custom-model/5.png b/public/images/custom-model/5.png deleted file mode 100644 index b598e78b..00000000 Binary files a/public/images/custom-model/5.png and /dev/null differ diff --git a/public/images/custom-model/6.png b/public/images/custom-model/6.png deleted file mode 100644 index b5c229e8..00000000 Binary files a/public/images/custom-model/6.png and /dev/null differ diff --git a/public/images/docs/evaluation/guides/custom-models/configure-custom-form.png b/public/images/docs/evaluation/guides/custom-models/configure-custom-form.png new file mode 100644 index 00000000..135a77f1 Binary files /dev/null and b/public/images/docs/evaluation/guides/custom-models/configure-custom-form.png differ diff --git a/public/images/docs/evaluation/guides/custom-models/custom-model-list.png b/public/images/docs/evaluation/guides/custom-models/custom-model-list.png new file mode 100644 index 00000000..157e4d36 Binary files /dev/null and b/public/images/docs/evaluation/guides/custom-models/custom-model-list.png differ diff --git a/public/images/docs/evaluation/guides/custom-models/openai-form.png b/public/images/docs/evaluation/guides/custom-models/openai-form.png new file mode 100644 index 00000000..0d697da3 Binary files /dev/null and b/public/images/docs/evaluation/guides/custom-models/openai-form.png differ diff --git a/public/images/docs/evaluation/guides/custom-models/provider-dropdown.png b/public/images/docs/evaluation/guides/custom-models/provider-dropdown.png new file mode 100644 index 00000000..60f9c125 Binary files /dev/null and b/public/images/docs/evaluation/guides/custom-models/provider-dropdown.png differ diff --git a/src/pages/docs/evaluation/guides/custom-models.mdx b/src/pages/docs/evaluation/guides/custom-models.mdx index b362c630..15374a1b 100644 --- a/src/pages/docs/evaluation/guides/custom-models.mdx +++ b/src/pages/docs/evaluation/guides/custom-models.mdx @@ -1,141 +1,60 @@ --- title: "Use custom models" -description: "Use your own or third-party models for evaluations in Future AGI via supported providers or a custom API endpoint with full configuration control." +description: "Bring your own LLM as the evaluator, from a provider or a custom endpoint" --- -## About +Evaluations need a model to act as the [evaluator](/docs/evaluation/concepts/evaluator-models): to read each response and decide whether it passes, fails, or scores in a range. A **custom model** lets you bring your own LLM as that evaluator instead of a Future AGI model, when a model of yours knows your domain better, when inference has to stay in a specific cloud or region, or when you want eval costs tracked against a model you already pay for. Once added, it appears in the model dropdown wherever you configure an eval. -Evaluations need a model to act as the evaluator: to read each response and decide whether it passes, fails, or scores within a range. Custom models let you bring your own LLM as the evaluator instead of using Future AGI's built-in models. +Two ways to connect: -This matters when you have a model that knows your domain better, when you need inference to stay within a specific cloud provider or region, or when you want to track evaluation costs against a model you already pay for. +- **From a provider**: a direct integration with Open AI, AWS Bedrock, AWS Sagemaker, Vertex AI, or Azure +- **Custom endpoint**: any model behind an HTTP API, including self-hosted, fine-tuned, or proxy deployments -Once you add a custom model, it appears in the model dropdown everywhere evaluations are configured : datasets, simulations, custom evals, and eval groups. +## Add a model -Two ways to connect: +Go to **Settings → AI Providers**, open the **Custom model** tab, and click **Create custom model**. -- **From a provider**: Direct integration with OpenAI, AWS Bedrock, AWS SageMaker, Vertex AI, or Azure. Recommended for reliability and simpler credential management. -- **Custom endpoint**: Connect any model behind an HTTP API, including self-hosted, fine-tuned, or proxy deployments. +AI Providers page on the Custom model tab, listing existing custom models with masked credentials and edit, delete, and copy icons, with a callout pointing at the Create custom model button top right +*The Custom model tab lists everything you've added so far; edit, delete, or copy any entry from here* - -Learn how to define eval rules that use your model: [Create custom evals](/docs/evaluation/guides/custom-evals). - +An **Add Model** drawer opens with two options: **From model Provider** or **Configure Custom Model**. ---- -## When to use +### From a provider -- **Control cost and compliance**: Bring your own LLM and set token costs so evaluation spend is tracked. Keep inference in your chosen region or provider for compliance. -- **Evaluate with a fine-tuned or internal model**: Run evals with a model tuned on your domain or hosted in-house by connecting it via the custom endpoint option. -- **Unify evals across providers**: Add multiple models and use the same eval templates against each to compare quality or cost. -- **Proxy or third-party APIs**: Connect any API-compatible endpoint when it is not one of the built-in providers. +With **From model Provider** selected, pick a provider from the **Model Provider** dropdown: Open AI, AWS Bedrock, AWS Sagemaker, Vertex AI, or Azure. ---- -## How to - -Choose how you want to connect your model: - - - - Direct integration with **OpenAI**, **AWS Bedrock**, **AWS SageMaker**, **Vertex AI**, or **Azure**. Follow the steps below. - - - - In your project, go to model configuration (e.g. **Settings** or **Models**) and choose to add a model **from a provider**. Select your provider; each has its own form (see tabs below). - - - - Configure your OpenAI API key and model; set a custom name and token costs for cost tracking. - ![OpenAI](/images/custom-model/1.png) - - - Connect via AWS credentials; choose a Bedrock model, set name and token costs. - ![AWS Bedrock](/images/custom-model/2.png) - - - Use your SageMaker endpoint; add name and token costs for evaluations. - ![AWS SageMaker](/images/custom-model/3.png) - - - Integrate with Google Cloud Vertex AI; configure model, name, and token costs. - ![Vertex AI](/images/custom-model/4.png) - - - Connect your Azure OpenAI or other Azure model; set name and token costs. - ![Azure](/images/custom-model/5.png) - - - - - Fill in the provider-specific authentication and options (e.g. API key, region, endpoint) in the form for your provider. - - - Give the model a **custom name** so you can recognise it in the model dropdown. Enter **input** and **output token cost per million tokens** so Future AGI can compute cost when running evaluations. - - - Save the model; it will appear in the model dropdown when you add or run custom evaluations. - - - - - Connect any model behind an API endpoint: self-hosted, fine-tuned, or third-party. Use this when integrating endpoints that are not one of the supported providers. - - - - In your project, go to model configuration (e.g. **Settings** or **Models**) and choose **Configure custom model** (or **Add custom model**) to open the form. - ![Add custom model](/images/custom-model/6.png) - - - **Model name**: a friendly identifier (e.g. `mistral-rag-prod`) so you can recognise it in selectors and reports. **API base URL**: the endpoint Future AGI will call (e.g. `https://api.my-model-server.com/v1`). Required for evaluations, RAG, and agent calls. - - - Enter **input token cost per million tokens** and **output token cost per million tokens** so Future AGI can compute cost and show usage analytics (e.g. `1.50` for input, `2.00` for output). - - - If your API needs extra headers or parameters (e.g. `Authorization: Bearer ...`), use **Add custom configuration** and add **Custom key** and **Custom value** pairs. Use this for auth, multi-tenant routing, or provider-specific options. - - - Save the model; it will appear in the model dropdown when you add or run custom evaluations. - - - - - +Model Provider dropdown open, listing Open AI, AWS Bedrock, AWS Sagemaker, Vertex AI, and Azure, with a callout pointing at Open AI +*Five supported providers, each with its own credential form* ---- +Fill in the provider's form: a **Model Name** to recognize it later, **Input** and **Output Token Cost Per Million Tokens** for cost tracking, and the provider's credentials, an API key for Open AI, region and access keys for Bedrock or Sagemaker, a service account for Vertex AI, endpoint and key for Azure. A **Form** and **JSON** toggle lets you fill the fields individually or paste a raw config. -## Field reference +Add Model drawer with Open AI selected as the Model Provider, showing Model Name, Input and Output Token Cost Per Million Tokens, a Form/JSON toggle, API Key, and an optional Base URL field, with a callout pointing at the API Key field +*Open AI's form: model name, token costs, API key, and an optional base URL* -Fields you may see when adding a model (from a provider or custom). **Applies to** indicates which flow uses the field. +### Custom endpoint -| Field | Applies to | About | Example | -| --- | --- | --- | --- | -| **Model name** / **Custom name** | Both | Friendly name for the model in Future AGI; shown in selectors and reports. | `mistral-rag-prod`, `my-openai-gpt4` | -| **Input token cost per million tokens** | Both | Cost of input tokens per 1M tokens; used for cost tracking and analytics. | `1.50` | -| **Output token cost per million tokens** | Both | Cost of output tokens per 1M tokens; used with input cost for total cost. | `2.00` | -| **Provider-specific fields** (auth, region, model ID, etc.) | From providers | Vary by provider (e.g. API key, region). See provider tabs in Step 1. | | -| **API base URL** | Custom model | Endpoint Future AGI calls for your model (evaluations, RAG, agent calls). | `https://api.my-model-server.com/v1` | -| **Add custom configuration** (Custom key & value) | Custom model | Custom headers or params (e.g. auth). Key/value pairs. | **Key:** `Authorization` **Value:** `Bearer sk-...` | +Select **Configure Custom Model** instead to connect any model behind an HTTP API. Fill in the **Model Name**, the token costs, and the **API Base URL**, the endpoint Future AGI calls. Anything else the endpoint needs, an auth header, a tenant ID, a routing parameter, goes under **Custom Configuration** as key/value pairs, and **Add more configuration** adds another pair. ---- +Add Model drawer with Configure Custom Model selected, showing required Model Name, Input and Output Token Cost Per Million Tokens, and API Base URL fields, plus an Add Custom Configuration section with Custom Key and Custom Value inputs and an Add more configuration button, with a callout pointing at the API Base URL field +*A custom endpoint needs an API base URL; everything else it needs goes in Custom Configuration* + +## Save the model + +Click **Add Custom model**. The model joins the Custom model list, and it now shows up in the model dropdown wherever you pick an evaluator, like when you [create a custom eval](/docs/evaluation/guides/custom-evals). -## Next Steps +## Keep exploring - - Run a single eval from the UI or SDK. + + Run an eval with your model on any surface - - Define eval rules and select your custom model. - - - Run multiple evals together as a group. + + Write an eval rule your model applies - Built-in models available for evals. - - - Run evals automatically in your pipeline. + The built-in models you can use instead - - How evaluation fits into the platform. + + Turn eval scores into a merge gate