Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions docs-site/src/content/docs/guides/combos.md
Original file line number Diff line number Diff line change
Expand Up @@ -701,6 +701,25 @@ even when every target supports images — the catalog drops `image` from `input
image-bearing requests are rejected with HTTP 400 before any target is called. `"auto"` (or
omitting the field) keeps the automatic intersection.

In the dashboard, the **Image / multimodal** switch can be enabled whenever every target is a
known catalog row. Members that advertise image input stay unchanged; members that do not are
automatically declared text-only on save (`modelCapabilities[model].inputModalities = ["text"]`)
so the [Vision Sidecar](/guides/sidecars/) describes their images. The switch's hint names the
members that will be enrolled. A member whose modalities are unknown or have no text input (for
example an audio-only model) cannot be covered by the sidecar; it keeps the switch unavailable and
is named in the hint. If the [Vision Sidecar](/guides/sidecars/) is disabled globally, enabling the
switch shows a warning linking to the dashboard: enrollment still saves, but images are replaced
with an omission note and requests continue without image descriptions until the sidecar is
re-enabled. Turning the switch off disables image input for the combo
but keeps those provider declarations, and `PUT /api/combos` accepts the same enrollment as an
optional top-level `visionSidecarTargets` array of exact `{ provider, model }` targets
(request-only; it is never stored on the combo and is rejected while `imageInput` is
Comment thread
x3M3x marked this conversation as resolved.
`"disabled"` when the enrollment list is non-empty). Entries that already accept image input or
are declared without text are rejected with HTTP 400 rather than overwritten, entries the sidecar
already covers are skipped, and a failed save restores the combo and every declaration together.
Removing a member from the combo does not remove its text-only declaration; clear it in the
provider editor if you no longer want it.

## Encrypted v2 sub-agent tasks

There is one important limitation for Codex v2 sub-agents ([issue #92](https://github.com/lidge-jun/opencodex/issues/92)).
Expand Down
6 changes: 6 additions & 0 deletions docs-site/src/content/docs/guides/sidecars.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,12 @@ allow attachments instead of blocking them before the sidecar runs. When
use the `gpt-5.6-luna` fallback. Startup still migrates an explicitly persisted legacy
`gpt-5.4-mini` value to `gpt-5.6-luna`; that migration applies to a stored value, not to an absent
model field.
Saving a multimodal combo from the dashboard enrolls its non-image members automatically: the
Combos page sends them as `visionSidecarTargets` on `PUT /api/combos`, which writes the exact
text-only declaration on the member's provider, so no manual config edit and reload is needed.
Enrollment never overwrites an existing image-capable declaration, and members whose modalities are
unknown or have no text input are not enrolled — the dashboard names them and keeps the combo's
image switch unavailable.
The first-party DeepSeek `deepseek-flash` model is native multimodal (`text` and `image`) and does
not use this sidecar by default. Explicit `noVisionModels` or text-only declarations remain
authoritative. First-party `deepseek-chat`, `deepseek-reasoner`, and `deepseek-v4-flash` remain
Expand Down
69 changes: 63 additions & 6 deletions gui/src/combo-capabilities.ts
Original file line number Diff line number Diff line change
@@ -1,14 +1,71 @@
import type { ComboTarget } from "./combo-workspace-data";
import type { ModelOption } from "./components/combo-workspace-types";

/** Whether every selected target advertises image input (incomplete rows fail closed). */
type ComboImageMemberKind = "vision" | "sidecar" | "blocked" | "missing";

/**
* One combo member's image story. Classification must survive a reload, so it
* reads `visionSidecarConsumer` — the runtime predicate's verdict served on the
* /api/models row — before falling back to the DECLARED modalities
* (`inputModalitiesDeclared`). A model covered only by legacy `noVisionModels`
* has no declaration and its advertised modalities are widened to image, so the
* flag is what keeps it a sidecar member after reload. Rows with no known
* modalities, or modalities without text, cannot be covered and block the combo.
*/
function imageMemberKind(target: ComboTarget, models: ModelOption[]): ComboImageMemberKind {
const provider = target.provider.trim();
const modelId = target.model.trim();
if (!provider || !modelId) return "missing";
const model = models.find((row) => row.provider === provider && row.id === modelId);
if (!model) return "missing";
if (model.visionSidecarConsumer === true) return "sidecar";
const declared = model.inputModalitiesDeclared ?? model.inputModalities;
if (!declared || declared.length === 0) return "blocked";
if (declared.includes("image")) return "vision";
return declared.includes("text") ? "sidecar" : "blocked";
}

/** Whether images can be enabled: every target is known and either images natively or can be declared text-only. */
export function comboImagesSupported(targets: ComboTarget[], models: ModelOption[]): boolean {
if (targets.length === 0) return false;
return targets.every((target) => {
const provider = target.provider.trim();
const modelId = target.model.trim();
if (!provider || !modelId) return false;
const model = models.find((row) => row.provider === provider && row.id === modelId);
return !!model?.inputModalities?.includes("image");
const kind = imageMemberKind(target, models);
return kind === "vision" || kind === "sidecar";
});
}

/**
* Exact targets that need a text-only declaration so the Vision Sidecar covers
* them when the combo accepts images. Deduplicated in submission order.
*/
export function comboVisionSidecarTargets(
targets: ComboTarget[],
models: ModelOption[],
): Array<{ provider: string; model: string }> {
const seen = new Set<string>();
const out: Array<{ provider: string; model: string }> = [];
for (const target of targets) {
if (imageMemberKind(target, models) !== "sidecar") continue;
const provider = target.provider.trim();
const model = target.model.trim();
const key = `${provider}/${model}`;
if (seen.has(key)) continue;
seen.add(key);
out.push({ provider, model });
}
return out;
}

/**
* Exact targets that cannot be covered by the Vision Sidecar: their row is
* known but its modalities are unknown or have no text input. Named in the
* hint so the operator knows which member blocks the image switch.
*/
export function comboImageBlockedTargets(
targets: ComboTarget[],
models: ModelOption[],
): Array<{ provider: string; model: string }> {
return targets
.filter((target) => imageMemberKind(target, models) === "blocked")
.map((target) => ({ provider: target.provider.trim(), model: target.model.trim() }));
}
7 changes: 6 additions & 1 deletion gui/src/combo-workspace-data.ts
Original file line number Diff line number Diff line change
Expand Up @@ -504,9 +504,13 @@ export function draftEquals(a: ComboItem, b: ComboItem): boolean {
});
}

export function toPutBody(item: ComboItem, options: { renameFrom?: string } = {}): {
export function toPutBody(
item: ComboItem,
options: { renameFrom?: string; visionSidecarTargets?: Array<{ provider: string; model: string }> } = {},
): {
id: string;
renameFrom?: string;
visionSidecarTargets?: Array<{ provider: string; model: string }>;
combo: {
targets: ComboTarget[];
strategy: ComboStrategy;
Expand All @@ -526,6 +530,7 @@ export function toPutBody(item: ComboItem, options: { renameFrom?: string } = {}
return {
id: item.id.trim(),
...(options.renameFrom ? { renameFrom: options.renameFrom } : {}),
...(options.visionSidecarTargets?.length ? { visionSidecarTargets: options.visionSidecarTargets } : {}),
combo: {
targets: item.targets.map((target) => ({
provider: target.provider.trim(),
Expand Down
3 changes: 3 additions & 0 deletions gui/src/components/ComboWorkspace.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ export default function ComboWorkspace({
providers,
models,
cataloguedComboIds,
visionEnabled,
loading,
onRefresh,
onSave,
Expand Down Expand Up @@ -224,6 +225,7 @@ export default function ComboWorkspace({
providerQuotaStates={providerQuotaStates}
providers={providers}
models={models}
visionEnabled={visionEnabled}
onBack={() => trySelect(null)}
onSaved={(item) => {
setDetailDirty(false);
Expand Down Expand Up @@ -253,6 +255,7 @@ export default function ComboWorkspace({
providerQuotaStates={providerQuotaStates}
providers={providers}
models={models}
visionEnabled={visionEnabled}
onSaved={(item) => {
setDetailDirty(false);
setSelectedId(item.id);
Expand Down
28 changes: 26 additions & 2 deletions gui/src/components/combo-workspace-controls.tsx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { useState } from "react";
import type { ComboEffort, ComboStrategy, ComboTarget, ProviderQuotaStates } from "../combo-workspace-data";
import { comboImagesSupported } from "../combo-capabilities";
import { comboImageBlockedTargets, comboImagesSupported, comboVisionSidecarTargets } from "../combo-capabilities";
import {
COMBO_EFFORTS,
COMBO_STRATEGIES,
Expand Down Expand Up @@ -93,20 +93,38 @@ export function ComboCapabilities({
models,
imageInput,
reasoningEffortMode,
visionEnabled,
disabled,
onChange,
}: {
targets: ComboTarget[];
models: ModelOption[];
imageInput: "auto" | "disabled";
reasoningEffortMode: "strict" | "adaptive";
/** Vision Sidecar enabled state; undefined = unknown, renders no warning. */
visionEnabled?: boolean;
disabled?: boolean;
onChange: (patch: { imageInput?: "auto" | "disabled"; reasoningEffortMode?: "strict" | "adaptive" }) => void;
}) {
const t = useT();
const imagesSupported = comboImagesSupported(targets, models);
const blockedTargets = comboImageBlockedTargets(targets, models);
// Default: checked (auto) when supported; force off when any target lacks image.
const effectiveOn = imagesSupported && imageInput !== "disabled";
const sidecarTargets = comboVisionSidecarTargets(targets, models);
// Enrollment wording only while image input is enabled: a disabled combo is not
// enrolling anything, so it gets the plain hint instead of save-time promises.
const imageHint = !imagesSupported
? blockedTargets.length > 0
? t("cws.capability.imageInputBlockedHint", {
models: blockedTargets.map(({ provider, model }) => `${provider}/${model}`).join(", "),
})
: t("cws.capability.imageInputUnavailable")
: sidecarTargets.length > 0 && imageInput !== "disabled"
? t("cws.capability.imageInputSidecarHint", {
models: sidecarTargets.map(({ provider, model }) => `${provider}/${model}`).join(", "),
})
: t("cws.capability.imageInputHint");

return (
<section className="cwi-capabilities" aria-label={t("cws.capabilities")}>
Expand All @@ -115,8 +133,14 @@ export function ComboCapabilities({
<div>
<span className="cwi-capability-label">{t("cws.capability.imageInput")}</span>
<p className="muted cwi-capability-hint">
{imagesSupported ? t("cws.capability.imageInputHint") : t("cws.capability.imageInputUnavailable")}
{imageHint}
</p>
{sidecarTargets.length > 0 && effectiveOn && visionEnabled === false ? (
<p className="muted cwi-capability-hint" style={{ color: "var(--danger, #b42318)" }}>
{t("cws.capability.imageInputSidecarDisabled")}{" "}
<a href="#dashboard">{t("nav.dashboard")}</a>
</p>
) : null}
</div>
<Switch
on={effectiveOn}
Expand Down
4 changes: 4 additions & 0 deletions gui/src/components/combo-workspace-detail-panel.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ export function DetailPanel({
providerQuotaStates,
providers,
models,
visionEnabled,
onBack,
onSaved,
onRequestRemove,
Expand All @@ -68,6 +69,8 @@ export function DetailPanel({
providerQuotaStates: ProviderQuotaStates;
providers: ProviderOption[];
models: ModelOption[];
/** Vision Sidecar enabled state; undefined = unknown, renders no warning. */
visionEnabled?: boolean;
onBack?: () => void;
onSaved: (item: ComboItem) => void;
onRequestRemove?: () => void;
Expand Down Expand Up @@ -408,6 +411,7 @@ export function DetailPanel({
models={models}
imageInput={draft.imageInput ?? "auto"}
reasoningEffortMode={draft.reasoningEffortMode ?? "strict"}
visionEnabled={visionEnabled}
disabled={busy}
onChange={(patch) => updateDraft((d) => ({ ...d, ...patch }))}
/>
Expand Down
6 changes: 6 additions & 0 deletions gui/src/components/combo-workspace-types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@ export type ModelOption = {
namespaced?: string;
reasoningEfforts?: string[];
inputModalities?: string[];
/** Operator-declared modalities; beats the (sidecar-widened) catalog view on reload. */
inputModalitiesDeclared?: string[];
/** Authoritative runtime verdict from /api/models: the Vision Sidecar describes this model's images. */
visionSidecarConsumer?: boolean;
};

export type ComboAddIntent = "blank" | "jev-auto";
Expand All @@ -30,6 +34,8 @@ export interface ComboWorkspaceProps {
models: ModelOption[];
/** Combo ids currently present in the live catalog (`provider === "combo"`). */
cataloguedComboIds?: ReadonlySet<string>;
/** Vision Sidecar enabled state from /api/sidecar-settings; undefined = unknown, no warning. */
visionEnabled?: boolean;
loading?: boolean;
onRefresh: () => void;
onSave: (item: ComboItem, isCreate: boolean, renameFrom?: string) => Promise<{ ok: boolean; error?: string }>;
Expand Down
5 changes: 4 additions & 1 deletion gui/src/i18n/de.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3117,8 +3117,11 @@ export const de: Record<TKey, string> = {
"cws.field.defaultEffort": "Standard-Reasoning",
"cws.field.defaultEffortNone": "Keine (Ziel-Standard)",
"cws.field.defaultEffortHint": "Nur verwendet, wenn der Client keinen Reasoning-Aufwand sendet. Optionen sind die Schnittmenge der beworbenen Aufwände der gewählten Ziele.",
"cws.capability.imageInputUnavailable": "Erst verfügbar, wenn jedes gewählte Ziel Bildeingabe unterstützt.",
"cws.capability.imageInputUnavailable": "Wähle zuerst alle Ziele aus dem Katalog — unbekannte Modelle kann der Vision Sidecar nicht abdecken.",
"cws.capability.imageInputHint": "Standardmäßig aktiv, wenn jedes Ziel Bilder unterstützt. Ausschalten für nur Text.",
"cws.capability.imageInputSidecarHint": "Standardmäßig aktiv. {models} wird beim Speichern als text-only deklariert und nutzt den Vision Sidecar für Bilder.",
"cws.capability.imageInputSidecarDisabled": "Der Vision Sidecar ist ausgeschaltet — an {models} gesendete Bilder werden durch einen Hinweistext ersetzt und die Anfrage läuft ohne sie weiter. Aktiviere ihn in den Dashboard-Einstellungen, um Bildbeschreibungen wiederherzustellen.",
"cws.capability.imageInputBlockedHint": "{models} kann vom Vision Sidecar nicht abgedeckt werden: Eingabemodalitäten sind unbekannt oder ohne Text.",
"cws.capability.imageInput": "Bild / multimodal",
"cws.capability.adaptiveEffort": "Adaptive Denkstufen",
"cws.capability.adaptiveEffortHint": "Aus: Ziele ohne Denkstufen-Regelung blenden die Auswahl für die gesamte Kombination aus. An: Solche Ziele bleiben nutzbar, und die Auswahl zeigt weiterhin die Stufen der übrigen Ziele.",
Expand Down
5 changes: 4 additions & 1 deletion gui/src/i18n/en.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3231,8 +3231,11 @@ export const en = {
"cws.field.defaultEffort": "Default reasoning",
"cws.field.defaultEffortNone": "None (target default)",
"cws.field.defaultEffortHint": "Used only when the client omits reasoning effort. Options are the intersection of the selected targets' advertised efforts; targets without catalog effort metadata offer none.",
"cws.capability.imageInputUnavailable": "Unavailable until every selected target supports image input.",
"cws.capability.imageInputUnavailable": "Pick every target from the catalog first — unknown models can't be covered by the Vision Sidecar.",
"cws.capability.imageInputHint": "On by default when every target supports images. Turn off to accept text only.",
"cws.capability.imageInputSidecarHint": "On by default. {models} will be declared text-only on save and use the Vision Sidecar for images.",
"cws.capability.imageInputSidecarDisabled": "The Vision Sidecar is turned off — images sent to {models} are replaced with an omission note and the request continues without them. Enable it in dashboard settings to restore image descriptions.",
"cws.capability.imageInputBlockedHint": "{models} cannot be covered by the Vision Sidecar: its input modalities are unknown or have no text.",
"cws.capability.imageInput": "Image / multimodal",
"cws.capability.adaptiveEffort": "Adaptive reasoning ladder",
"cws.capability.adaptiveEffortHint": "Off: a target with no reasoning control hides the effort picker for the whole combo. On: those targets stay usable and the picker keeps the levels the remaining targets share.",
Expand Down
7 changes: 5 additions & 2 deletions gui/src/i18n/fr.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3109,8 +3109,11 @@ export const fr: Record<TKey, string> = {
"cws.emptyTitle": "Créer votre première combinaison",
"cws.empty.createDesc": "Nommez un modèle virtuel et enchaînez au moins deux services principaux.",
"cws.backToAll": "Retour à toutes les combinaisons",
"cws.capability.imageInputUnavailable": "Indisponible tant que toutes les cibles sélectionnées ne prennent pas en charge les images.",
"cws.capability.imageInputHint": "Activé par défaut lorsque toutes les cibles prennent en charge les images. Désactivez cette option pour n’accepter que du texte.",
"cws.capability.imageInputUnavailable": "Sélectionnez d'abord chaque cible dans le catalogue — les modèles inconnus ne peuvent pas être couverts par le Vision Sidecar.",
"cws.capability.imageInputHint": "Activé par défaut lorsque toutes les cibles prennent en charge les images. Désactivez cette option pour n'accepter que du texte.",
"cws.capability.imageInputSidecarHint": "Activé par défaut. Lors de l’enregistrement, les cibles {models} seront déclarées en mode texte uniquement et utiliseront le Vision Sidecar pour les images.",
"cws.capability.imageInputSidecarDisabled": "Le Vision Sidecar est désactivé — les images envoyées à {models} sont remplacées par une note d’omission et la requête continue sans elles. Activez-le dans les paramètres du tableau de bord pour rétablir les descriptions.",
"cws.capability.imageInputBlockedHint": "{models} ne peut pas être couvert par le Vision Sidecar : ses modalités d’entrée sont inconnues ou sans texte.",
"cws.capability.imageInput": "Images / multimodal",
"cws.capability.adaptiveEffort": "Échelle de raisonnement adaptative",
"cws.capability.adaptiveEffortHint": "Désactivé : une cible sans réglage de raisonnement masque le sélecteur pour toute la combinaison. Activé : ces cibles restent utilisables et le sélecteur conserve les niveaux communs aux autres cibles.",
Expand Down
Loading
Loading