diff --git a/CLAUDE.md b/CLAUDE.md index 88c36121..1e24d6bc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -29,7 +29,7 @@ API changes update `docs/APIS.md` in the same commit (rule 1). Tool changes upda - **assistant-ui**: `useLangGraphRuntime` from `@assistant-ui/react-langgraph` (NOT `useChatRuntime`). Full-page ``, not ``. See `assistant-ui` skill. - **Env specifics** (full table in `README.md`): chat models in `backend/model.ts` carry `modelKwargs: { reasoning_split: true }` — strip when switching providers. `DENO_DEPLOY_TOKEN` personal tokens (prefix `ddp_`) need `DENO_DEPLOY_ORG`; without it, `execute_code` is unregistered and `write_code` is the fallback. -- **Backend graph**: two compiled graphs — `agent` (`backend/agent.ts`) and `background_agent` (`backend/background-agent.ts`). Adding a sub-agent: update the enum in `state.ts` AND `router-agent-node.ts` AND `agent.ts` routeToSubAgent union (or parsing throws — see [[router-decision-schema-duplicated]]). +- **Backend graph**: two compiled graphs — `agent` (`backend/agent.ts`) and `backgroundAgent` (`backend/background-agent.ts`). Adding a sub-agent: update the enum in `state.ts` AND `router-agent-node.ts` AND `agent.ts` routeToSubAgent union (or parsing throws — see [[router-decision-schema-duplicated]]). - **State persistence**: checkpointer is runner-chosen — `langgraphjs dev` uses `InMemorySaver`, `langgraphjs start` uses `PostgresSaver` (`backend/checkpointer.ts`). `POST /api/threads` calls `langGraphClient.threads.create(...)` to register the new id with the dev server's in-process store (no-op in prod). - **Web3**: trade flow is fully SIMULATED regardless of wallet connectivity — `place_crypto_order` auto-funds Mock Coin on first trade. - **Observability**: see `docs/OBSERVABILITY.md`. Gotchas: `parent_message_id` backfill window; `FORBIDDEN` regex in `bulkInsertSpans` (fail-closed); FK cascade from `threads`; retention via `OBSERVABILITY_RETENTION_DAYS` (default 30) + `scripts/cleanup-observability.ts`. diff --git a/README.md b/README.md index 01260e44..1953e2a8 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ ## Features - **Streaming chat UI** powered by assistant-ui's `Thread` component. -- **LangGraph backend** running **two compiled graphs** side-by-side: a chat graph (`agent` — router + sub-agents + `triggerBackgroundAgent`) and a turn-end side-effect graph (`background_agent` — `touchLastMessage` + `summarize`). The chat stream doesn't block on background work; `triggerBackgroundAgentNode` HTTP-dispatches via the SDK and returns immediately. +- **LangGraph backend** running **two compiled graphs** side-by-side: a chat graph (`agent` — router + sub-agents + `triggerBackgroundAgent`) and a turn-end side-effect graph (`backgroundAgent` — `touchLastMessage` + `summarize`). The chat stream doesn't block on background work; `triggerBackgroundAgentNode` HTTP-dispatches via the SDK and returns immediately. - **Persistent threads and checkpoints** in Postgres — closing the tab doesn't lose context. - **Cross-conversation memory**: the model calls `save_memory` to persist durable user facts (RFC 6902 patches against `[userId, "memory"] main`); a recall middleware prepends `` (profile + auth overlay) and `` (compressed Q&A history) blocks to the SystemMessage on every invoke. Long threads stay readable via a store-anchored `threadSummarizeNode` trigger that compresses K-turn windows into `SummaryEntry` rows. The Memory settings tab lets users review and delete. See [docs/MEMORY.md](docs/MEMORY.md). - **Self-hosted**: runs on a single VPS with Docker Compose, no SaaS lock-in. @@ -25,6 +25,7 @@ - **Knowledge Base & Hybrid Search**: ingestion across seven source kinds (PDF, image, plain text, markdown, DOCX, XLSX, PPTX) plus pasted URLs — PDF / image go through vision OCR, Office formats are parsed structurally by `officeparser` with embedded images extracted to R2, text and markdown skip straight to chunking — followed by text chunking, embedding, entity / relationship / theme extraction, and pgvector-backed three-leg RRF (Keyword, Vector, Tag) hybrid search combined with semantic Reranking (Cohere/Jina), `@` mention resolution (with automatic fallback to full markdown when chunks are not ready), dynamic budget scaling, and iterative search. See [docs/KNOWLEDGE_BASE.md](docs/KNOWLEDGE_BASE.md). - **Per-LLM-call credit quota**: every successful call is metered against a UTC-aligned rolling-window cap read from `role.creditLimit` / `role.windowHours`. Enforcement lives at the `/api/[..._path]` proxy — when the cap is hit, the proxy synthesizes a `show_credit_card` SSE stream and the chat UI renders the credit-limit-reached card inline. The call log backs a per-user history (Settings → Credits) and an admin-managed rate config. See [docs/CREDIT.md](docs/CREDIT.md). - **Admin console**: a single `/admin` page with three tabs — Providers (registry + encrypted API keys + per-model rates), Roles (credit caps + window length), Users (role assignment, ban with immediate session revoke, delete). The first admin is bootstrapped via `INITIAL_ADMIN_EMAIL`. See [docs/ADMIN.md](docs/ADMIN.md). +- **Agent Evaluation Studio** (under `/admin/eval`): per-agent Benchmark Studio, Rubric Editor, Online Executions (per-agent paginated waterfall of production + judge runs), and LLM-as-a-Judge orchestration via the dedicated `evalAgent` graph. 11 agent IDs are seeded with domain-tailored rubrics + English benchmark datasets, including `judgeByLLM` (the judge itself) and three KB sub-agents (`pageToMarkdown`, `chunkExtract`, `chunkAlignment`) whose ingest runs surface under their own agent cards. `eval-judge` and `eval-benchmark` thread kinds split judge vs synthetic activity. See [docs/EVALUATION.md](docs/EVALUATION.md). ## Tech stack @@ -145,11 +146,12 @@ backend/ weather-agent.ts weatherAgent compiled subgraph crypto-agent.ts cryptoAgent compiled subgraph code-agent.ts codeAgent compiled subgraph + eval-agent.ts evalAgent compiled evaluation graph (LLM-as-a-Judge) node/ call-model-node.ts "agent" node — calls the model, appends AI reply rename-thread-agent-node.ts "renameThreadAgent" — generates + persists the title router-agent-node.ts "routerAgent" — picks weatherAgent / chatAgent / cryptoAgent / codeAgent - trigger-background-agent-node.ts "triggerBackgroundAgent" — SDK runs.create to background_agent + trigger-background-agent-node.ts "triggerBackgroundAgent" — SDK runs.create to backgroundAgent thread-summarize-node.ts "summarize" — compresses K-turn window into a SummaryEntry tool/ LangChain tools bound to the agent web-search.ts search_web — Jina Search (s.jina.ai/{query}) @@ -159,6 +161,8 @@ backend/ recall.ts loadMemory / getCachedMemory (LRU max 1000, 60s TTL); extractUserId / extractThreadId template.ts buildSystemMessageWithMemory (mustache + ) + trimMessagesForInvoke profile-size.ts assertProfileSize — guard before the store write (NFR-003) + eval/ + callback.ts EvalCallbackHandler — captures LLM run executions for evaluation observability/ callback-collector.ts CapturingHandler — buffers spans per runId, persists on End hook @@ -181,6 +185,9 @@ lib/ adapter.ts RemoteThreadListAdapter for assistant-ui validators.ts Zod API body schemas memory/ Memory module — queries (getMemoryDoc / putMemoryDoc / getAuthInfo / writeSummary / getThreadSummaries / getRecentThreadSummaries / deleteThreadSummaries) + validators (RFC 6902 patches, SummaryEntry) + merge (mergeMemory + getStoreKeys) + constants + format + eval/ Agent Evaluation Studio & A/B testing module + schema.ts Drizzle tables (prompt_template, prompt_variant, prompt_variant_assignment, eval_run, eval_feedback, eval_rubric, eval_benchmark, eval_judgment) + queries.ts seedInitialPrompts (100% English per-agent Rubrics & Benchmarks + rubric_default) + CRUD (run / feedback / judgment / benchmark / getRunsByAgentPage with PG row-value cursor) observability/ Observability module schema.ts Drizzle table (observability_spans) queries.ts bulkInsertSpans / getSpansByThreadId / markRunningAsFailed / deleteSpansByThreadId @@ -225,7 +232,7 @@ tests/ Vitest (NODE_ENV=test → reads .env.test) drizzle.config.ts Drizzle-kit config (uses @next/env to load .env) vitest.config.ts Vitest config (NODE_ENV=test → reads .env.test) -langgraph.json LangGraph CLI config (registers BOTH graphs: agent + background_agent) +langgraph.json LangGraph CLI config (registers BOTH graphs: agent + backgroundAgent) .env.example Template (committed) .env.local Local dev secrets (gitignored) .env.test Test secrets (gitignored) @@ -245,6 +252,11 @@ Owned by `lib//schema.ts` (re-exported from `db/schema.ts`). See [`docs/ - **`role`** — per-tier credit cap (`credit_limit`, `window_hours`). Seeded with `guest`, `user`, `admin` (migration `0003`). - **`provider`** — LLM registry (encrypted API keys + per-model rates). Seeded with `default` (migration `0003`). - **`credit_usage_log`** — append-only per-LLM-call log. Source of truth for the cap check + the user-facing history. Composite index `(user_id, created_at)` covers both workloads. +- **`eval_rubric`** — per-agent evaluation criteria definitions for LLM-as-a-Judge (keyed directly by `id = "rubric_${agentId}"`, plus a generic `rubric_default` seeded by `seedInitialPrompts`). +- **`eval_benchmark`** — admin-defined offline benchmark test cases per agent (`id`, `agent`, `title`, `input_prompt`, `expected_output`) with denormalized `latest_judgment_id` / `latest_run_at` / `latest_run_status` / `latest_score` so the Benchmark Datasets table can render "Last Result" inline. +- **`eval_judgment`** — assessment scores and reasoning outputs emitted by `evalAgent`, with `judge_thread_id` tracking AI Judge execution traces. +- **`prompt_template`, `prompt_variant`, `prompt_variant_assignment`** — system-prompt versioning + sticky A/B traffic splitting (admin-managed weights; sticky by `(userId, agent)`). +- **`eval_run`, `eval_feedback`** — turn-level execution record + user/admin 1-5 rating (👍/👎). The chat UI sends the assistant message id; `lib/eval/queries.ts:submitFeedback` resolves it via `eval_run.parent_message_id` and silently no-ops when no match — the thumbs-up button never crashes on a dangling FK. ### 2. LangGraph checkpoints @@ -331,7 +343,8 @@ Test database stays isolated from dev — never put production-like data in `lan - [`docs/INTERRUPT.md`](docs/INTERRUPT.md) — interrupt-driven tool flows (ask_location, connect_wallet, place_crypto_order, get_order_status) — the two runtime paths the cards can take. - [`docs/AUTH.md`](docs/AUTH.md) — operator guide for the auth layer: env vars, OAuth app setup, Resend, role mechanism, `INITIAL_ADMIN_EMAIL` bootstrap, troubleshooting. - [`docs/ATTACHMENTS.md`](docs/ATTACHMENTS.md) — chat attachments backed by Cloudflare R2: direct-upload architecture, key convention, lazy-register on missing env, `Content-Disposition` XSS guard, `messageId`-deferred decision. -- [`docs/DB.md`](docs/DB.md) — database schema (Better Auth + `threads` + `attachments` + `role` + `provider` + `credit_usage_log`), ownership model, indexes. Source of truth: `db/migrations/0000_*.sql`. +- [`docs/DB.md`](docs/DB.md) — database schema (Better Auth + `threads` + `attachments` + `role` + `provider` + `credit_usage_log` + `prompt_*` / `eval_*`), ownership model, indexes. Source of truth: `db/migrations/0000_*.sql`. +- [`docs/EVALUATION.md`](docs/EVALUATION.md) — Agent Evaluation Studio + A/B testing: prompt versioning, sticky traffic splitting, per-agent Benchmark Datasets, LLM-as-a-Judge orchestration, `threads.kind = 'eval-judge' | 'eval-benchmark'` discriminator, full `/api/eval/*` route map. - [`docs/CREDIT.md`](docs/CREDIT.md) — per-LLM-call credit cap: UTC-aligned rolling window, proxy-level enforcement, callback-level recording, `show_credit_card` UI affordance, `credit_usage_log` audit trail. - [`docs/ADMIN.md`](docs/ADMIN.md) — admin console: `/admin` Providers / Roles / Users tabs, AES-256-GCM API-key encryption, last-admin + default-provider guards, secrets handling. - [`docs/PROVIDERS.md`](docs/PROVIDERS.md) — DB-backed chat-model registry: `getChatModel` / `getChatModelFromDB` / `invalidateModelCache`, LRU + 60s cross-process TTL tradeoff, seeded `default` row, env fallback. diff --git a/app/admin/admin-tabs.tsx b/app/admin/admin-tabs.tsx index ec11331e..9f71d34c 100644 --- a/app/admin/admin-tabs.tsx +++ b/app/admin/admin-tabs.tsx @@ -1,120 +1,16 @@ "use client"; -import { useEffect, useState, useTransition } from "react"; import { useRouter, useSearchParams } from "next/navigation"; - -import { Loader2 } from "lucide-react"; - import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs"; -import { Button } from "@/components/ui/button"; -import { Input } from "@/components/ui/input"; -import { Badge } from "@/components/ui/badge"; -import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"; -import { Checkbox } from "@/components/ui/checkbox"; -import { Separator } from "@/components/ui/separator"; -import { Switch } from "@/components/ui/switch"; -import { - Dialog, - DialogContent, - DialogDescription, - DialogFooter, - DialogHeader, - DialogTitle, -} from "@/components/ui/dialog"; -import { FormDialog } from "@/components/ui/form-dialog"; -import { - Select, - SelectContent, - SelectItem, - SelectTrigger, - SelectValue, -} from "@/components/ui/select"; -import { Tooltip, TooltipContent, TooltipProvider, TooltipTrigger } from "@/components/ui/tooltip"; -import { Avatar, AvatarImage, AvatarFallback } from "@/components/ui/avatar"; -import { toast } from "sonner"; - -type PublicProviderApiKey = { name: string }; -type ModelKind = "chat" | "ocr" | "embed" | "extract" | "rerank"; -type PublicModel = { - name: string; - enabled: boolean; - inputPer1k: number; - outputPer1k: number; - // ponytail: server-side default ["chat"] means an older row may not - // carry the field at all. Fall back to chat at render time so the - // table shows a value for every model, regardless of seed-vs-new. - kind?: ModelKind[]; -}; -type PublicProviderRow = { - id: string; - name: string; - enabled: boolean; - baseUrl: string; - apiKeys: PublicProviderApiKey[]; - models: PublicModel[]; - createdAt: string; - updatedAt: string; -}; -type RoleRow = { - id: string; - name: string; - creditLimit: number | null; - windowHours: number; - createdAt: string; - updatedAt: string; -}; +import { ProvidersPanel } from "@/app/admin/providers/providers-client"; +import { RolesPanel } from "@/app/admin/roles/roles-client"; +import { UsersPanel } from "@/app/admin/users/users-client"; +import { ObservabilitySheet } from "@/components/observability/sheet"; +import { ObservabilitySheetProvider } from "@/components/observability/sheet-context"; +import { EvalDashboardClient } from "@/app/admin/eval/eval-dashboard-client"; +import type { PublicProviderRow, RoleRow, UserRow } from "@/app/admin/types"; -// ponytail: module-scope constant so add-mode `initialKind ?? ["chat"]` -// keeps the same array reference across renders — `useEffect` dep -// comparison uses Object.is, and a fresh literal each render would loop -// forever (Next dev StrictMode catches it as "Maximum update depth"). -const DEFAULT_KIND: ModelKind[] = ["chat"]; -// ponytail: server-side join shape — user + role name inlined so the -// table can render "Admin" / "User" labels without a second round-trip. -// `roleName` is null when the FK points at a missing role (defensive — -// FK prevents it, but a leftJoin can still null it on race). -type UserRow = { - id: string; - name: string | null; - email: string; - image?: string | null; - emailVerified: boolean; - roleId: string; - roleName: string | null; - banned: boolean; - createdAt: string; - updatedAt: string; - todayCredits?: number; - todayInputTokens?: number; - todayOutputTokens?: number; - todayTokens?: number; - totalCredits?: number; - totalInputTokens?: number; - totalOutputTokens?: number; - totalTokens?: number; -}; - -async function jsonFetch( - url: string, - init?: RequestInit, -): Promise<{ ok: boolean; status: number; body: T | { code: string; message?: string } }> { - const res = await fetch(url, { - ...init, - headers: { "Content-Type": "application/json", ...init?.headers }, - }); - const body = (await res.json().catch(() => ({}))) as T | { code: string; message?: string }; - return { ok: res.ok, status: res.status, body }; -} - -function errMsg(body: unknown): string { - if (typeof body === "object" && body !== null) { - const m = (body as { message?: string }).message; - const c = (body as { code?: string }).code; - if (m) return m; - if (c) return c; - } - return "request failed"; -} +export type { PublicProviderRow, RoleRow, UserRow }; export function AdminTabs({ providers, @@ -129,7 +25,7 @@ export function AdminTabs({ const searchParams = useSearchParams(); const tabParam = searchParams.get("tab"); const activeTab = - tabParam && ["providers", "roles", "users"].includes(tabParam) ? tabParam : "providers"; + tabParam && ["providers", "roles", "users", "eval"].includes(tabParam) ? tabParam : "providers"; const handleTabChange = (val: string) => { const params = new URLSearchParams(searchParams.toString()); @@ -143,6 +39,7 @@ export function AdminTabs({ Providers Roles Users + Eval & A/B @@ -153,1791 +50,14 @@ export function AdminTabs({ - - ); -} - -function ProvidersPanel({ initial }: { initial: PublicProviderRow[] }) { - const router = useRouter(); - const [adding, setAdding] = useState(false); - - return ( -
-
-
-

Providers

-

- LLM providers, their base URL, models, and API keys. -

-
- -
- - {initial.length === 0 ? ( - - - No providers yet. - - - ) : ( - initial.map((p) => ) - )} - - setAdding(false)} - onSaved={() => { - setAdding(false); - router.refresh(); - }} - /> -
- ); -} - -function ProviderCard({ provider }: { provider: PublicProviderRow }) { - const router = useRouter(); - const [pending, start] = useTransition(); - const [confirmingDelete, setConfirmingDelete] = useState(false); - const [editing, setEditing] = useState(false); - - const confirmRemove = () => { - start(async () => { - const r = await jsonFetch(`/api/admin/providers/${encodeURIComponent(provider.id)}`, { - method: "DELETE", - }); - if (!r.ok) { - toast.error(errMsg(r.body)); - return; - } - setConfirmingDelete(false); - toast.success("provider deleted"); - router.refresh(); - }); - }; - - return ( - // ponytail: muted background for disabled cards so the list separates - // "live" from "kept around but off" at a glance. - - -
-
-
- {provider.name} - - {provider.enabled ? "Enabled" : "Disabled"} - -
- - {provider.id} - · - {new Date(provider.createdAt).toLocaleDateString("en-CA")} - -
-
- - {/* ponytail: the default provider is the seed row — at least - one provider must always exist for the system to boot, - so we hard-disable Delete on id="default". The backend - enforces the same rule (DELETE /[id]/route returns 409 - for default) so a tampered client can't slip past. */} - {provider.id === "default" ? ( - - - - - - - - Default provider — at least one is required. - - - ) : ( - - )} -
-
- - -
- - Base URL - - {provider.baseUrl} -
- - - - - - - setEditing(false)} - onSaved={() => { - setEditing(false); - router.refresh(); - }} - /> - - !open && setConfirmingDelete(false)}> - - - Delete this provider? - - “{provider.name}” and all of its models and API keys will be removed. This cannot be - undone. - - - - - - - - - - ); -} - -function ModelsSection({ providerId, models }: { providerId: string; models: PublicModel[] }) { - const router = useRouter(); - const [pending, start] = useTransition(); - const [pendingDelete, setPendingDelete] = useState(null); - const [editing, setEditing] = useState(null); - const [adding, setAdding] = useState(false); - - const confirmRemove = () => { - if (!pendingDelete) return; - const target = pendingDelete; - start(async () => { - const r = await jsonFetch( - `/api/admin/providers/${encodeURIComponent(providerId)}/models/${encodeURIComponent(target.name)}`, - { method: "DELETE" }, - ); - if (!r.ok) { - toast.error(errMsg(r.body)); - return; - } - setPendingDelete(null); - router.refresh(); - }); - }; - - return ( -
-
-

Models

- -
-
- - - - - - - - - - - - {models.length === 0 ? ( - - - - ) : ( - models.map((m) => ( - - - - - - - - - )) - )} - -
NameStatusKindInput / 1kOutput / 1k -
- No models configured. -
{m.name} - - {m.enabled ? "Enabled" : "Disabled"} - - -
- {(m.kind ?? ["chat"]).map((k) => ( - - {k.charAt(0).toUpperCase() + k.slice(1)} - - ))} -
- 		{m.inputPer1k}{m.outputPer1k} -
- - -
-
-
- - !open && setPendingDelete(null)} - > - - - Delete this model? - - “{pendingDelete?.name}” will be removed from this provider. This cannot be undone. - - - - - - - - - - {editing ? ( - setEditing(null)} - onSaved={() => { - setEditing(null); - router.refresh(); - }} - /> - ) : null} - - setAdding(false)} - onSaved={() => { - setAdding(false); - router.refresh(); - }} - /> -
- ); -} - -function ModelDialog( - props: - | { - mode: "add"; - open: boolean; - providerId: string; - pending: boolean; - onClose: () => void; - onSaved: () => void; - } - | { - mode: "edit"; - open?: boolean; - model: PublicModel; - providerId: string; - pending: boolean; - onClose: () => void; - onSaved: () => void; - }, -) { - const isEdit = props.mode === "edit"; - const initialName = isEdit ? props.model.name : ""; - const initialEnabled = isEdit ? props.model.enabled : true; - const initialIn = isEdit ? String(props.model.inputPer1k) : "0.001"; - const initialOut = isEdit ? String(props.model.outputPer1k) : "0.002"; - // ponytail: backend defaults kind to ["chat"] on POST when omitted, - // so the add-mode local seed mirrors the server contract and we don't - // need to send `kind` in the body. Edit-mode seeds from existing row. - const initialKind: ModelKind[] = isEdit ? (props.model.kind ?? DEFAULT_KIND) : DEFAULT_KIND; - - const [name, setName] = useState(initialName); - const [enabled, setEnabled] = useState(initialEnabled); - const [inputPer1k, setInputPer1k] = useState(initialIn); - const [outputPer1k, setOutputPer1k] = useState(initialOut); - const [kind, setKind] = useState(initialKind); - const [saving, start] = useTransition(); - - useEffect(() => { - setName(initialName); - setEnabled(initialEnabled); - setInputPer1k(initialIn); - setOutputPer1k(initialOut); - setKind(initialKind); - }, [initialName, initialEnabled, initialIn, initialOut, initialKind]); - - const toggleKind = (k: ModelKind, checked: boolean) => { - setKind((prev) => { - const next = checked ? Array.from(new Set([...prev, k])) : prev.filter((x) => x !== k); - // ponytail: at least one kind must stay selected — unchecking the - // last box is a no-op so the user can never save a model with - // kind = [] (which the API rejects with 400 anyway). - return next.length === 0 ? prev : next; - }); - }; - - const save = () => { - if (!name.trim()) { - toast.error("model name required"); - return; - } - const inp = Number(inputPer1k); - const out = Number(outputPer1k); - if (!Number.isFinite(inp) || inp < 0 || !Number.isFinite(out) || out < 0) { - toast.error("rates must be non-negative numbers"); - return; - } - const path = isEdit - ? `/api/admin/providers/${encodeURIComponent(props.providerId)}/models/${encodeURIComponent(props.model.name)}` - : `/api/admin/providers/${encodeURIComponent(props.providerId)}/models`; - const method = isEdit ? "PATCH" : "POST"; - const body = isEdit - ? { name: name.trim(), enabled, inputPer1k: inp, outputPer1k: out, kind } - : { name: name.trim(), enabled, inputPer1k: inp, outputPer1k: out, kind }; - start(async () => { - const r = await jsonFetch(path, { method, body: JSON.stringify(body) }); - if (!r.ok) { - toast.error(errMsg(r.body)); - return; - } - toast.success(isEdit ? "model updated" : "model added"); - props.onSaved(); - }); - }; - - return ( - !o && props.onClose()} - title={isEdit ? "Edit model" : "Add model"} - description={ - isEdit - ? `${props.model.name} — rename takes effect on Save; collision with another model on this provider is rejected.` - : "Register a new model for this provider. The name is the JSONB-array key; you can rename it later from the Edit dialog." - } - submitLabel={isEdit ? "Save" : "Add"} - pending={saving || props.pending} - onSubmit={save} - onCancel={props.onClose} - > -
- - - - -
- Kind -
- {(["chat", "ocr", "embed", "extract", "rerank"] as const).map((k) => ( - - ))} -
-

- At least one kind is required — chat for general inference, ocr for PDF vision, embed - for KB chunk vectors, extract for structured-output triples from chunks, rerank for - secondary retrieval reranking. -

-
-
- - ); -} - -function KeysSection({ providerId, keys }: { providerId: string; keys: PublicProviderApiKey[] }) { - const router = useRouter(); - const [pending, start] = useTransition(); - const [pendingDelete, setPendingDelete] = useState(null); - const [editing, setEditing] = useState(null); - const [adding, setAdding] = useState(false); - - const confirmRemove = () => { - if (!pendingDelete) return; - const target = pendingDelete; - start(async () => { - const r = await jsonFetch(`/api/admin/providers/${encodeURIComponent(providerId)}/keys`, { - method: "DELETE", - body: JSON.stringify({ name: target.name }), - }); - if (!r.ok) { - toast.error(errMsg(r.body)); - return; - } - setPendingDelete(null); - router.refresh(); - }); - }; - - return ( -
-
-

API keys

- -
-
- - - - - - - - {keys.length === 0 ? ( - - - - ) : ( - keys.map((k) => ( - - - - - )) - )} - -
Key -
- No API keys configured. -
{k.name} -
- - -
-
-
- - !open && setPendingDelete(null)} - > - - - Delete this API key? - - “{pendingDelete?.name}” will be removed from this provider. Requests using this key - will start failing immediately. This cannot be undone. - - - - - - - - - - {editing ? ( - setEditing(null)} - onSaved={() => { - setEditing(null); - router.refresh(); - }} - /> - ) : null} - - setAdding(false)} - onSaved={() => { - setAdding(false); - router.refresh(); - }} - /> -
- ); -} - -function KeyDialog( - props: - | { - mode: "add"; - open: boolean; - providerId: string; - pending: boolean; - onClose: () => void; - onSaved: () => void; - } - | { - mode: "edit"; - open?: boolean; - keyEntry: PublicProviderApiKey; - providerId: string; - pending: boolean; - onClose: () => void; - onSaved: () => void; - }, -) { - const isEdit = props.mode === "edit"; - - const [plain, setPlain] = useState(""); - const [saving, start] = useTransition(); - - // ponytail: re-seed `plain` on each dialog re-open so the new secret - // field starts blank (the edit-mode hint says "required to update", - // so a leftover from a previous attempt is misleading). - useEffect(() => { - setPlain(""); - // eslint-disable-next-line react-hooks/exhaustive-deps - }, [isEdit ? props.mode === "edit" && props.keyEntry?.name : props.mode === "add" && props.open]); - - const save = () => { - const trimmedPlain = plain.trim(); - if (!trimmedPlain) { - toast.error("key value required"); - return; - } - start(async () => { - // ponytail: both add and edit send plaintext only — the display - // name is derived server-side from the ciphertext tail via - // deriveKeyName (POST) or stays the same on PATCH (which keeps - // the path-bound keyName and only re-encrypts the blob). The - // backend's optional `name` field is left for SQL-direct edits - // and is intentionally not exposed in the UI. - const path = isEdit - ? `/api/admin/providers/${encodeURIComponent(props.providerId)}/keys/${encodeURIComponent(props.keyEntry.name)}` - : `/api/admin/providers/${encodeURIComponent(props.providerId)}/keys`; - const method = isEdit ? "PATCH" : "POST"; - const r = await jsonFetch(path, { - method, - body: JSON.stringify({ plaintext: trimmedPlain }), - }); - if (!r.ok) { - if (r.status === 409) { - toast.error("a key with this tail already exists"); - } else { - toast.error(errMsg(r.body)); - } - return; - } - toast.success(isEdit ? "key updated" : "key added"); - props.onSaved(); - }); - }; - - return ( - !o && props.onClose()} - title={isEdit ? "Edit API key" : "Add API key"} - description={ - isEdit - ? "Paste the new secret — the same key entry is re-encrypted in place. The display name (derived from the ciphertext) updates automatically." - : "Paste the key value to add it. The display name is derived from the ciphertext — both the value and the visible name are produced by the same secret." - } - submitLabel={isEdit ? "Save" : "Add"} - pending={saving || props.pending} - onSubmit={save} - onCancel={props.onClose} - > -
- - {isEdit ? ( -

- Display name {props.keyEntry.name} is derived from - the key value — paste the new secret to regenerate. -

- ) : null} -
- - ); -} - -function RolesPanel({ initial }: { initial: RoleRow[] }) { - const router = useRouter(); - const [adding, setAdding] = useState(false); - - return ( -
-
-
-

Roles

-

- Roles set the per-window credit cap for users. A blank credit limit means unlimited. -

-
- -
- -
- - - - - - - - - - - {initial.map((r) => ( - - ))} - -
IDNameCredit limitWindow (h) -
-
- - setAdding(false)} - onSaved={() => { - setAdding(false); - router.refresh(); - }} - /> -
- ); -} - -function RoleRowView({ role }: { role: RoleRow }) { - const router = useRouter(); - const [pending, start] = useTransition(); - const [editing, setEditing] = useState(false); - const [confirmingDelete, setConfirmingDelete] = useState(false); - - const remove = () => { - start(async () => { - const r = await jsonFetch(`/api/admin/roles/${encodeURIComponent(role.id)}`, { - method: "DELETE", - }); - if (!r.ok) { - toast.error(errMsg(r.body)); - return; - } - setConfirmingDelete(false); - toast.success("role deleted"); - router.refresh(); - }); - }; - - return ( - - {role.id} - {role.name} - - {role.creditLimit === null ? "unlimited" : role.creditLimit} - - {role.windowHours} - -
- - -
- - !open && setConfirmingDelete(false)} - > - - - Delete this role? - - “{role.name}” will be removed. Users on this role fall back to the default. This - cannot be undone. - - - - - - - - - - setEditing(false)} - onSaved={() => { - setEditing(false); - router.refresh(); - }} - /> - - ); -} - -function RoleDialog( - props: - | { - mode: "add"; - open: boolean; - onClose: () => void; - onSaved: () => void; - } - | { - mode: "edit"; - open?: boolean; - role: RoleRow; - onClose: () => void; - onSaved: () => void; - }, -) { - const isEdit = props.mode === "edit"; - const initialName = isEdit ? props.role.name : ""; - const initialLimit = isEdit - ? props.role.creditLimit === null - ? "" - : String(props.role.creditLimit) - : ""; - const initialHours = isEdit ? String(props.role.windowHours) : "24"; - - const [id, setId] = useState(isEdit ? props.role.id : ""); - const [name, setName] = useState(initialName); - const [limit, setLimit] = useState(initialLimit); - const [hours, setHours] = useState(initialHours); - const [saving, start] = useTransition(); - - useEffect(() => { - setName(initialName); - setLimit(initialLimit); - setHours(initialHours); - }, [initialName, initialLimit, initialHours]); - - const save = () => { - if (!name.trim()) { - toast.error("name required"); - return; - } - if (!isEdit && !id.trim()) { - toast.error("id required"); - return; - } - if (!isEdit && !/^[a-z][a-z0-9_-]*$/.test(id.trim())) { - toast.error("id must be lowercase alphanumeric / dash / underscore"); - return; - } - const h = Number(hours); - if (!Number.isInteger(h) || h < 1) { - toast.error("windowHours must be a positive integer"); - return; - } - const limitNum = limit.trim() === "" ? null : Number(limit); - if (limitNum !== null && (!Number.isFinite(limitNum) || limitNum < 0)) { - toast.error("creditLimit must be a non-negative number or blank for unlimited"); - return; - } - start(async () => { - let r; - if (isEdit) { - r = await jsonFetch(`/api/admin/roles/${encodeURIComponent(props.role.id)}`, { - method: "PATCH", - body: JSON.stringify({ name: name.trim(), creditLimit: limitNum, windowHours: h }), - }); - } else { - r = await jsonFetch("/api/admin/roles", { - method: "POST", - body: JSON.stringify({ - id: id.trim(), - name: name.trim(), - creditLimit: limitNum, - windowHours: h, - }), - }); - } - if (!r.ok) { - toast.error(errMsg(r.body)); - return; - } - toast.success(isEdit ? "role updated" : "role created"); - props.onSaved(); - }); - }; - - return ( - !o && props.onClose()} - title={isEdit ? `Edit role: ${props.role.id}` : "Add role"} - description={ - isEdit - ? "ID is the FK identifier and can’t be changed here. Delete + recreate to rename." - : "Roles set the per-window credit cap for users. A blank credit limit means unlimited." - } - submitLabel={isEdit ? "Save" : "Add"} - pending={saving} - onSubmit={save} - onCancel={props.onClose} - > -
- - - - -
- - ); -} - -function UsersPanel({ initial, roles }: { initial: UserRow[]; roles: RoleRow[] }) { - const total = initial.length; - const admins = initial.filter((u) => u.roleId === "admin").length; - const banned = initial.filter((u) => u.banned).length; - - const todayCredits = initial.reduce((sum, u) => sum + (u.todayCredits ?? 0), 0); - const todayInputTokens = initial.reduce((sum, u) => sum + (u.todayInputTokens ?? 0), 0); - const todayOutputTokens = initial.reduce((sum, u) => sum + (u.todayOutputTokens ?? 0), 0); - - const totalCredits = initial.reduce((sum, u) => sum + (u.totalCredits ?? 0), 0); - const totalInputTokens = initial.reduce((sum, u) => sum + (u.totalInputTokens ?? 0), 0); - const totalOutputTokens = initial.reduce((sum, u) => sum + (u.totalOutputTokens ?? 0), 0); - - return ( -
-
-

Users

-

- Registered accounts, their role, and ban status. Banning immediately revokes every active - session for that user — they’re signed out on the next request. -

-
- -
- - - -
- -
- - - - - - - - - - - - {initial.length === 0 ? ( - - - - ) : ( - initial.map((u) => ) - )} - -
UserRoleStatusToday's UsageTotal Usage -
- No users yet. -
-
-
- ); -} - -function UserBreakdownCard({ - total, - admins, - banned, -}: { - total: number; - admins: number; - banned: number; -}) { - const regular = Math.max(0, total - admins - banned); - const adminPct = total > 0 ? (admins / total) * 100 : 0; - const regularPct = total > 0 ? (regular / total) * 100 : 0; - const bannedPct = total > 0 ? (banned / total) * 100 : 0; - - return ( - - -
-
- Users -
- {total} registered -
- -
- {total} - users -
- -
- - - -
- {total > 0 ? ( - <> -
-
-
- - ) : ( -
- )} -
- - -
- admins: {admins} ({adminPct.toFixed(1)}%) -
-
- active: {regular} ({regularPct.toFixed(1)}%) -
-
- banned: {banned} ({bannedPct.toFixed(1)}%) -
- - - - -
-
- - Admin: {admins} -
-
- - Active: {regular} -
-
- - Banned: {banned} -
-
-
- - - ); -} - -function UsageStatCard({ - title, - credits, - inputTokens, - outputTokens, -}: { - title: string; - credits: number; - inputTokens: number; - outputTokens: number; -}) { - const totalTokens = inputTokens + outputTokens; - const inputPct = totalTokens > 0 ? (inputTokens / totalTokens) * 100 : 0; - const outputPct = totalTokens > 0 ? 100 - inputPct : 0; - - return ( - - -
-
- {title} -
- - {totalTokens.toLocaleString()} tok - + +
+ + + +
- -
- - {credits.toLocaleString(undefined, { - minimumFractionDigits: 2, - maximumFractionDigits: 2, - })} - - credit -
- -
- - - -
- {totalTokens > 0 ? ( - <> -
-
- - ) : ( -
- )} -
- - -
- input: {inputTokens.toLocaleString()} tok ({inputPct.toFixed(1)}%) -
-
- output: {outputTokens.toLocaleString()} tok ({outputPct.toFixed(1)}%) -
- - - - -
-
- - In: {inputTokens.toLocaleString()} -
-
- - Out: {outputTokens.toLocaleString()} -
-
-
- - - ); -} - -function UserRowView({ user, roles }: { user: UserRow; roles: RoleRow[] }) { - const router = useRouter(); - const [pending, start] = useTransition(); - const [editing, setEditing] = useState(false); - const [confirmingDelete, setConfirmingDelete] = useState(false); - - const remove = () => { - start(async () => { - const r = await jsonFetch(`/api/admin/users/${encodeURIComponent(user.id)}`, { - method: "DELETE", - }); - if (!r.ok) { - if (r.status === 409) { - toast.error("cannot delete the last admin"); - } else { - toast.error(errMsg(r.body)); - } - return; - } - setConfirmingDelete(false); - toast.success("user deleted"); - router.refresh(); - }); - }; - - return ( - - -
- - {user.image ? : null} - - {(user.name || user.email).slice(0, 2).toUpperCase()} - - -
- - {user.name ?? "—"} - - - {user.email} - -
-
- - - - {user.roleName ?? user.roleId} - - - - {user.banned ? ( - Banned - ) : user.emailVerified ? ( - Verified - ) : ( - Unverified - )} - - -
- - {(user.todayCredits ?? 0).toLocaleString(undefined, { - minimumFractionDigits: 2, - maximumFractionDigits: 2, - })}{" "} - credit - - - - - - {(user.todayTokens ?? 0).toLocaleString()}{" "} - tok - - - -
input: {(user.todayInputTokens ?? 0).toLocaleString()} tok
-
output: {(user.todayOutputTokens ?? 0).toLocaleString()} tok
- - - -
- - -
- - {(user.totalCredits ?? 0).toLocaleString(undefined, { - minimumFractionDigits: 2, - maximumFractionDigits: 2, - })}{" "} - credit - - - - - - {(user.totalTokens ?? 0).toLocaleString()}{" "} - tok - - - -
input: {(user.totalInputTokens ?? 0).toLocaleString()} tok
-
output: {(user.totalOutputTokens ?? 0).toLocaleString()} tok
- - - -
- - -
- - -
- - !open && setConfirmingDelete(false)} - > - - - Delete this user? - - “{user.email}” and all of their sessions, accounts, and threads will be removed. - This cannot be undone. - - - - - - - - - - setEditing(false)} - onSaved={() => { - setEditing(false); - router.refresh(); - }} - /> - - - ); -} - -type UserDialogProps = { - mode: "edit"; - open: boolean; - user: UserRow; - roles: RoleRow[]; - onClose: () => void; - onSaved: () => void; -}; - -// ponytail: role is a Select (dropdown), never a free-text input — the -// FK target list comes from the roles prop, so the dropdown is the -// authoritative surface and there's no risk of typing a typo that -// 404s on save. Server still validates (404 ROLE_NOT_FOUND) as a -// defense-in-depth in case the role list goes stale. -function UserDialog(props: UserDialogProps) { - const initialRoleId = props.user.roleId; - const initialBanned = props.user.banned; - - const [roleId, setRoleId] = useState(initialRoleId); - const [banned, setBanned] = useState(initialBanned); - const [saving, start] = useTransition(); - - useEffect(() => { - setRoleId(initialRoleId); - setBanned(initialBanned); - }, [initialRoleId, initialBanned]); - - const save = () => { - if (!roleId) { - toast.error("role required"); - return; - } - start(async () => { - const r = await jsonFetch(`/api/admin/users/${encodeURIComponent(props.user.id)}`, { - method: "PATCH", - body: JSON.stringify({ roleId, banned }), - }); - if (!r.ok) { - if (r.status === 409) { - toast.error("cannot demote or ban the last admin"); - } else if (r.status === 404) { - toast.error("role not found"); - } else { - toast.error(errMsg(r.body)); - } - return; - } - toast.success("user updated"); - props.onSaved(); - }); - }; - - return ( - !o && props.onClose()} - title={`Edit user: ${props.user.email}`} - description="Pick a role from the list — FK is validated server-side. Banning immediately revokes every active session for this user." - submitLabel="Save" - pending={saving} - onSubmit={save} - onCancel={props.onClose} - > -
-
- - Email - - {props.user.email} -
- - -
- - ); -} - -// ponytail: provider add + edit share fields (display name, enabled, -// baseUrl). Mode flips the title, submit label, and whether the auto- -// generated id is shown. `id` is the DB primary key — PATCH doesn't -// accept a rename, so the edit dialog locks it. Add mode hides it -// entirely (server picks one from `crypto.randomUUID()`). -type ProviderDialogProps = - | { - mode: "add"; - open: boolean; - onClose: () => void; - onSaved: () => void; - } - | { - mode: "edit"; - open?: boolean; - provider: PublicProviderRow; - pending: boolean; - onClose: () => void; - onSaved: () => void; - }; - -function ProviderDialog(props: ProviderDialogProps) { - const isEdit = props.mode === "edit"; - const initialName = isEdit ? props.provider.name : ""; - const initialEnabled = isEdit ? props.provider.enabled : true; - const initialBaseUrl = isEdit ? props.provider.baseUrl : ""; - - const [name, setName] = useState(initialName); - const [enabled, setEnabled] = useState(initialEnabled); - const [baseUrl, setBaseUrl] = useState(initialBaseUrl); - const [saving, start] = useTransition(); - - useEffect(() => { - setName(initialName); - setEnabled(initialEnabled); - setBaseUrl(initialBaseUrl); - }, [initialName, initialEnabled, initialBaseUrl]); - - const save = () => { - if (!name.trim()) { - toast.error("display name is required"); - return; - } - if (!baseUrl.trim()) { - toast.error("base URL is required"); - return; - } - try { - // ponytail: trust the user but validate the URL parses — admin - // will hit 4xx from the upstream if they paste a typo, but a - // frontend check keeps the error next to the field. - // eslint-disable-next-line no-new - new URL(baseUrl.trim()); - } catch { - toast.error("base URL must be a valid URL"); - return; - } - start(async () => { - let r; - if (isEdit) { - r = await jsonFetch(`/api/admin/providers/${encodeURIComponent(props.provider.id)}`, { - method: "PATCH", - body: JSON.stringify({ - name: name.trim(), - enabled, - baseUrl: baseUrl.trim(), - }), - }); - } else { - // ponytail: id is auto-generated server-side — we send a UUID - // with a "prov_" prefix so the URL space is unmistakable. - // Collisions are astronomically unlikely but POST returns 409 - // with a "DUPLICATE" code if it ever happens; the toast - // surfaces that as a fallback. - const newId = `prov_${crypto.randomUUID().replace(/-/g, "").slice(0, 12)}`; - r = await jsonFetch("/api/admin/providers", { - method: "POST", - body: JSON.stringify({ - id: newId, - name: name.trim(), - enabled, - baseUrl: baseUrl.trim(), - }), - }); - } - if (!r.ok) { - if (r.status === 409) { - toast.error("a provider with this id already exists"); - } else { - toast.error(errMsg(r.body)); - } - return; - } - toast.success(isEdit ? "provider updated" : "provider created"); - props.onSaved(); - }); - }; - - return ( - !o && props.onClose()} - title={isEdit ? "Edit provider" : "Add provider"} - description={ - isEdit - ? `ID ${props.provider.id} is the FK identifier and can’t be changed here. Delete + recreate to rename.` - : "Register a new LLM provider. The id is auto-generated; provide a display name, enabled toggle, and base URL." - } - submitLabel={isEdit ? "Save" : "Add"} - pending={saving || (isEdit ? props.pending : false)} - onSubmit={save} - onCancel={props.onClose} - > -
- {isEdit ? ( -
- - ID - - {props.provider.id} -
- ) : null} - - - -
- + + ); } diff --git a/app/admin/eval/assignments/index.ts b/app/admin/eval/assignments/index.ts new file mode 100644 index 00000000..78090cb1 --- /dev/null +++ b/app/admin/eval/assignments/index.ts @@ -0,0 +1 @@ +export * from "./user-assignments-tab"; diff --git a/app/admin/eval/assignments/user-assignments-tab.tsx b/app/admin/eval/assignments/user-assignments-tab.tsx new file mode 100644 index 00000000..38413636 --- /dev/null +++ b/app/admin/eval/assignments/user-assignments-tab.tsx @@ -0,0 +1,245 @@ +"use client"; + +import React from "react"; +import { Network, RefreshCw, Search, ShieldCheck } from "lucide-react"; +import { Avatar, AvatarFallback, AvatarImage } from "@/components/ui/avatar"; +import { Badge } from "@/components/ui/badge"; +import { Input } from "@/components/ui/input"; +import { + Select, + SelectContent, + SelectItem, + SelectTrigger, + SelectValue, +} from "@/components/ui/select"; +import { + Table, + TableBody, + TableCell, + TableHead, + TableHeader, + TableRow, +} from "@/components/ui/table"; +import { UserAssignment, Variant } from "../types"; + +interface UserAssignmentsTabProps { + assignments: UserAssignment[]; + variants: Variant[]; + userSearch: string; + onSearchChange: (search: string) => void; + onOverrideCohort: (userId: string, cohortLabel: string) => void; + overridingUserId?: string | null; +} + +function getInitials(name?: string, email?: string): string { + if (name && name.trim()) { + const parts = name.trim().split(/\s+/); + if (parts.length >= 2) { + return (parts[0][0] + parts[parts.length - 1][0]).toUpperCase(); + } + return parts[0].slice(0, 2).toUpperCase(); + } + if (email && email.trim()) { + return email.trim().slice(0, 2).toUpperCase(); + } + return "US"; +} + +export function UserAssignmentsTab({ + assignments, + variants, + userSearch, + onSearchChange, + onOverrideCohort, + overridingUserId, +}: UserAssignmentsTabProps) { + // Extract all unique Cohort Labels from available variants + const availableCohorts = Array.from(new Set(variants.map((v) => v.label))); + if (!availableCohorts.includes("default")) { + availableCohorts.unshift("default"); + } + + // Group assignments by userId + const userMap = new Map< + string, + { + userId: string; + userName?: string; + userEmail?: string; + userImage?: string | null; + assignedAt: string; + cohortLabel: string; + items: UserAssignment[]; + } + >(); + + for (const a of assignments) { + const existing = userMap.get(a.userId) || { + userId: a.userId, + userName: a.userName, + userEmail: a.userEmail, + userImage: a.userImage, + assignedAt: a.assignedAt, + cohortLabel: a.variantLabel || "default", + items: [], + }; + existing.items.push(a); + if (a.variantLabel && a.variantLabel !== "default") { + existing.cohortLabel = a.variantLabel; + } + userMap.set(a.userId, existing); + } + + const userList = Array.from(userMap.values()).filter( + (u) => + !userSearch || + (u.userEmail && u.userEmail.toLowerCase().includes(userSearch.toLowerCase())) || + (u.userName && u.userName.toLowerCase().includes(userSearch.toLowerCase())) || + (u.userId && u.userId.toLowerCase().includes(userSearch.toLowerCase())), + ); + + return ( +
+
+
+

User Experiment Variant Assignments

+

+ Inspect or override sticky experiment variant assignments for registered users across + all agent nodes. +

+
+
+ + onSearchChange(e.target.value)} + placeholder="Search user or email..." + className="pl-8 h-9 text-xs" + /> +
+
+ +
+ + + + User Profile + Routing Mechanism + Bound Graph Nodes + Assigned Variant + Assigned Date + + + + {userList.map((u) => { + const isUpdating = overridingUserId === u.userId; + const isOverride = u.cohortLabel.toLowerCase() !== "default"; + + return ( + + +
+ + {u.userImage ? ( + + ) : null} + + {(u.userName || u.userEmail || u.userId).slice(0, 2).toUpperCase()} + + +
+ + {u.userName || "Registered User"} + + + {u.userEmail || u.userId} + +
+
+ + + + {isOverride ? ( + <> + + Admin Override + + ) : ( + <> + + Deterministic Hashed + + )} + + + + + 10 Agent Nodes + + + +
+ + + {isUpdating && ( + + )} +
+ + + {new Date(u.assignedAt).toLocaleDateString("en-CA")} + + + ); + })} + {userList.length === 0 && ( + + + No sticky user assignments recorded yet. Invoking agents will automatically assign + users to cohorts. + + + )} + +
+
+
+ ); +} diff --git a/app/admin/eval/benchmark/add-benchmark-dialog.tsx b/app/admin/eval/benchmark/add-benchmark-dialog.tsx new file mode 100644 index 00000000..702ce932 --- /dev/null +++ b/app/admin/eval/benchmark/add-benchmark-dialog.tsx @@ -0,0 +1,119 @@ +"use client"; + +import React, { useState } from "react"; +import { Button } from "@/components/ui/button"; +import { + Dialog, + DialogContent, + DialogDescription, + DialogFooter, + DialogHeader, + DialogTitle, +} from "@/components/ui/dialog"; +import { Input } from "@/components/ui/input"; +import { Textarea } from "@/components/ui/textarea"; + +interface AddBenchmarkDialogProps { + open: boolean; + onOpenChange: (open: boolean) => void; + agentId: string; + agentName: string; + onAdd: (data: { + agent: string; + title: string; + inputPrompt: string; + expectedOutput?: string; + }) => Promise; + submitting?: boolean; +} + +export function AddBenchmarkDialog({ + open, + onOpenChange, + agentId, + agentName, + onAdd, + submitting = false, +}: AddBenchmarkDialogProps) { + const [title, setTitle] = useState(""); + const [inputPrompt, setInputPrompt] = useState(""); + const [expectedOutput, setExpectedOutput] = useState(""); + + const handleSubmit = async (e: React.FormEvent) => { + e.preventDefault(); + if (!title || !inputPrompt) return; + await onAdd({ + agent: agentId, + title, + inputPrompt, + expectedOutput: expectedOutput || undefined, + }); + setTitle(""); + setInputPrompt(""); + setExpectedOutput(""); + onOpenChange(false); + }; + + return ( + + +
+ + + Add Benchmark Test Case for {agentName} + + + Define a test prompt and optional ground truth expected output to benchmark this agent + node. + + + +
+
+ + setTitle(e.target.value)} + className="text-xs" + required + /> +
+ +
+ +