Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file removed public/images/custom-model/1.png
Binary file not shown.
Binary file removed public/images/custom-model/2.png
Binary file not shown.
Binary file removed public/images/custom-model/3.png
Binary file not shown.
Binary file removed public/images/custom-model/4.png
Binary file not shown.
Binary file removed public/images/custom-model/5.png
Binary file not shown.
Binary file removed public/images/custom-model/6.png
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
147 changes: 33 additions & 114 deletions src/pages/docs/evaluation/guides/custom-models.mdx
Original file line number Diff line number Diff line change
@@ -1,141 +1,60 @@
---
title: "Use custom models"
description: "Use your own or third-party models for evaluations in Future AGI via supported providers or a custom API endpoint with full configuration control."
description: "Bring your own LLM as the evaluator, from a provider or a custom endpoint"
---

## About
Evaluations need a model to act as the [evaluator](/docs/evaluation/concepts/evaluator-models): to read each response and decide whether it passes, fails, or scores in a range. A **custom model** lets you bring your own LLM as that evaluator instead of a Future AGI model, when a model of yours knows your domain better, when inference has to stay in a specific cloud or region, or when you want eval costs tracked against a model you already pay for. Once added, it appears in the model dropdown wherever you configure an eval.

Evaluations need a model to act as the evaluator: to read each response and decide whether it passes, fails, or scores within a range. Custom models let you bring your own LLM as the evaluator instead of using Future AGI's built-in models.
Two ways to connect:

This matters when you have a model that knows your domain better, when you need inference to stay within a specific cloud provider or region, or when you want to track evaluation costs against a model you already pay for.
- **From a provider**: a direct integration with Open AI, AWS Bedrock, AWS Sagemaker, Vertex AI, or Azure
- **Custom endpoint**: any model behind an HTTP API, including self-hosted, fine-tuned, or proxy deployments

Once you add a custom model, it appears in the model dropdown everywhere evaluations are configured : datasets, simulations, custom evals, and eval groups.
## Add a model

Two ways to connect:
Go to **Settings → AI Providers**, open the **Custom model** tab, and click **Create custom model**.

- **From a provider**: Direct integration with OpenAI, AWS Bedrock, AWS SageMaker, Vertex AI, or Azure. Recommended for reliability and simpler credential management.
- **Custom endpoint**: Connect any model behind an HTTP API, including self-hosted, fine-tuned, or proxy deployments.
<img src="/images/docs/evaluation/guides/custom-models/custom-model-list.png" alt="AI Providers page on the Custom model tab, listing existing custom models with masked credentials and edit, delete, and copy icons, with a callout pointing at the Create custom model button top right" style={{ borderRadius: '5px' }} />
*The Custom model tab lists everything you've added so far; edit, delete, or copy any entry from here*

<Tip>
Learn how to define eval rules that use your model: [Create custom evals](/docs/evaluation/guides/custom-evals).
</Tip>
An **Add Model** drawer opens with two options: **From model Provider** or **Configure Custom Model**.

---
## When to use
### From a provider

- **Control cost and compliance**: Bring your own LLM and set token costs so evaluation spend is tracked. Keep inference in your chosen region or provider for compliance.
- **Evaluate with a fine-tuned or internal model**: Run evals with a model tuned on your domain or hosted in-house by connecting it via the custom endpoint option.
- **Unify evals across providers**: Add multiple models and use the same eval templates against each to compare quality or cost.
- **Proxy or third-party APIs**: Connect any API-compatible endpoint when it is not one of the built-in providers.
With **From model Provider** selected, pick a provider from the **Model Provider** dropdown: Open AI, AWS Bedrock, AWS Sagemaker, Vertex AI, or Azure.

---
## How to

Choose how you want to connect your model:

<Tabs sync={false}>
<Tab title="From providers" icon="plug">
Direct integration with **OpenAI**, **AWS Bedrock**, **AWS SageMaker**, **Vertex AI**, or **Azure**. Follow the steps below.

<Steps>
<Step title="Open model configuration and choose provider">
In your project, go to model configuration (e.g. **Settings** or **Models**) and choose to add a model **from a provider**. Select your provider; each has its own form (see tabs below).

<Tabs sync={false}>
<Tab title="OpenAI">
Configure your OpenAI API key and model; set a custom name and token costs for cost tracking.
![OpenAI](/images/custom-model/1.png)
</Tab>
<Tab title="AWS Bedrock">
Connect via AWS credentials; choose a Bedrock model, set name and token costs.
![AWS Bedrock](/images/custom-model/2.png)
</Tab>
<Tab title="AWS SageMaker">
Use your SageMaker endpoint; add name and token costs for evaluations.
![AWS SageMaker](/images/custom-model/3.png)
</Tab>
<Tab title="Vertex AI">
Integrate with Google Cloud Vertex AI; configure model, name, and token costs.
![Vertex AI](/images/custom-model/4.png)
</Tab>
<Tab title="Azure">
Connect your Azure OpenAI or other Azure model; set name and token costs.
![Azure](/images/custom-model/5.png)
</Tab>
</Tabs>
</Step>
<Step title="Enter provider credentials and settings">
Fill in the provider-specific authentication and options (e.g. API key, region, endpoint) in the form for your provider.
</Step>
<Step title="Set custom name and token costs">
Give the model a **custom name** so you can recognise it in the model dropdown. Enter **input** and **output token cost per million tokens** so Future AGI can compute cost when running evaluations.
</Step>
<Step title="Save">
Save the model; it will appear in the model dropdown when you add or run custom evaluations.
</Step>
</Steps>
</Tab>
<Tab title="Custom model (advanced)" icon="server">
Connect any model behind an API endpoint: self-hosted, fine-tuned, or third-party. Use this when integrating endpoints that are not one of the supported providers.

<Steps>
<Step title="Open custom model configuration">
In your project, go to model configuration (e.g. **Settings** or **Models**) and choose **Configure custom model** (or **Add custom model**) to open the form.
![Add custom model](/images/custom-model/6.png)
</Step>
<Step title="Enter model name and API base URL">
**Model name**: a friendly identifier (e.g. `mistral-rag-prod`) so you can recognise it in selectors and reports. **API base URL**: the endpoint Future AGI will call (e.g. `https://api.my-model-server.com/v1`). Required for evaluations, RAG, and agent calls.
</Step>
<Step title="Set token costs">
Enter **input token cost per million tokens** and **output token cost per million tokens** so Future AGI can compute cost and show usage analytics (e.g. `1.50` for input, `2.00` for output).
</Step>
<Step title="Add custom configuration (optional)">
If your API needs extra headers or parameters (e.g. `Authorization: Bearer ...`), use **Add custom configuration** and add **Custom key** and **Custom value** pairs. Use this for auth, multi-tenant routing, or provider-specific options.
</Step>
<Step title="Save">
Save the model; it will appear in the model dropdown when you add or run custom evaluations.
</Step>
</Steps>

</Tab>
</Tabs>
<img src="/images/docs/evaluation/guides/custom-models/provider-dropdown.png" alt="Model Provider dropdown open, listing Open AI, AWS Bedrock, AWS Sagemaker, Vertex AI, and Azure, with a callout pointing at Open AI" style={{ borderRadius: '5px' }} />
*Five supported providers, each with its own credential form*

---
Fill in the provider's form: a **Model Name** to recognize it later, **Input** and **Output Token Cost Per Million Tokens** for cost tracking, and the provider's credentials, an API key for Open AI, region and access keys for Bedrock or Sagemaker, a service account for Vertex AI, endpoint and key for Azure. A **Form** and **JSON** toggle lets you fill the fields individually or paste a raw config.

## Field reference
<img src="/images/docs/evaluation/guides/custom-models/openai-form.png" alt="Add Model drawer with Open AI selected as the Model Provider, showing Model Name, Input and Output Token Cost Per Million Tokens, a Form/JSON toggle, API Key, and an optional Base URL field, with a callout pointing at the API Key field" style={{ borderRadius: '5px' }} />
*Open AI's form: model name, token costs, API key, and an optional base URL*

Fields you may see when adding a model (from a provider or custom). **Applies to** indicates which flow uses the field.
### Custom endpoint

| Field | Applies to | About | Example |
| --- | --- | --- | --- |
| **Model name** / **Custom name** | Both | Friendly name for the model in Future AGI; shown in selectors and reports. | `mistral-rag-prod`, `my-openai-gpt4` |
| **Input token cost per million tokens** | Both | Cost of input tokens per 1M tokens; used for cost tracking and analytics. | `1.50` |
| **Output token cost per million tokens** | Both | Cost of output tokens per 1M tokens; used with input cost for total cost. | `2.00` |
| **Provider-specific fields** (auth, region, model ID, etc.) | From providers | Vary by provider (e.g. API key, region). See provider tabs in Step 1. | |
| **API base URL** | Custom model | Endpoint Future AGI calls for your model (evaluations, RAG, agent calls). | `https://api.my-model-server.com/v1` |
| **Add custom configuration** (Custom key & value) | Custom model | Custom headers or params (e.g. auth). Key/value pairs. | **Key:** `Authorization` **Value:** `Bearer sk-...` |
Select **Configure Custom Model** instead to connect any model behind an HTTP API. Fill in the **Model Name**, the token costs, and the **API Base URL**, the endpoint Future AGI calls. Anything else the endpoint needs, an auth header, a tenant ID, a routing parameter, goes under **Custom Configuration** as key/value pairs, and **Add more configuration** adds another pair.

---
<img src="/images/docs/evaluation/guides/custom-models/configure-custom-form.png" alt="Add Model drawer with Configure Custom Model selected, showing required Model Name, Input and Output Token Cost Per Million Tokens, and API Base URL fields, plus an Add Custom Configuration section with Custom Key and Custom Value inputs and an Add more configuration button, with a callout pointing at the API Base URL field" style={{ borderRadius: '5px' }} />
*A custom endpoint needs an API base URL; everything else it needs goes in Custom Configuration*

## Save the model

Click **Add Custom model**. The model joins the Custom model list, and it now shows up in the model dropdown wherever you pick an evaluator, like when you [create a custom eval](/docs/evaluation/guides/custom-evals).

## Next Steps
## Keep exploring

<CardGroup cols={2}>
<Card title="Evaluate via Platform &amp; SDK" icon="play" href="/docs/evaluation/guides/running-evaluations">
Run a single eval from the UI or SDK.
<Card title="Running Evaluations" icon="play-circle" href="/docs/evaluation/guides/running-evaluations">
Run an eval with your model on any surface
</Card>
<Card title="Create custom evals" icon="wand-magic-sparkles" href="/docs/evaluation/guides/custom-evals">
Define eval rules and select your custom model.
</Card>
<Card title="Eval groups" icon="table-rows" href="/docs/evaluation">
Run multiple evals together as a group.
<Card title="Create a custom eval" icon="wand-magic-sparkles" href="/docs/evaluation/guides/custom-evals">
Write an eval rule your model applies
</Card>
<Card title="Evaluator models" icon="robot" href="/docs/evaluation/concepts/evaluator-models">
Built-in models available for evals.
</Card>
<Card title="CI/CD pipeline" icon="code" href="/docs/evaluation/guides/cicd">
Run evals automatically in your pipeline.
The built-in models you can use instead
</Card>
<Card title="Evaluation overview" icon="chart-line" href="/docs/evaluation">
How evaluation fits into the platform.
<Card title="Evaluate in CI/CD" icon="code-branch" href="/docs/evaluation/guides/cicd">
Turn eval scores into a merge gate
</Card>
</CardGroup>