Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
import { describe, expect, it } from "vitest";
import { afterEach, beforeEach, describe, expect, it } from "vitest";

import {
DEFAULT_MODEL,
getLanguageModel,
resolveModelAlias,
resolvePipelineModelSelection,
} from "./modelResolver";
Expand Down Expand Up @@ -42,3 +44,36 @@ describe("pipeline model route resolution", () => {
expect(resolveModelAlias("poolside/laguna-s-2.1:free")).toBe("laguna-s-2.1-free");
});
});

describe("a missing provider key must not break Convex module analysis", () => {
// Convex analyses every backend module on every push, and
// domains/agents/core/coordinatorAgent.ts builds DEFAULT_MODEL ("kimi-k2.6",
// an OpenRouter model) at module scope. When buildLanguageModel threw for an
// unset OPENROUTER_API_KEY, `convex dev` failed the ENTIRE push with
// `InvalidModules: Failed to analyze domains/agents/digestAgent.js`, so a
// reader with only a Gemini key got no backend at all. Construction must
// succeed; the call must still fail.
const savedKey = process.env.OPENROUTER_API_KEY;

beforeEach(() => {
delete process.env.OPENROUTER_API_KEY;
});
afterEach(() => {
if (savedKey === undefined) delete process.env.OPENROUTER_API_KEY;
else process.env.OPENROUTER_API_KEY = savedKey;
});

it("constructs the default OpenRouter model with no key, and fails only when called", async () => {
// `LanguageModel` is `string | LanguageModelV2`; the object branch is the
// one under test, so read it through one local cast rather than four.
const model = getLanguageModel(DEFAULT_MODEL) as any;
expect(model.modelId).toBe(DEFAULT_MODEL);
expect(model.provider).toBe("unconfigured");
await expect(model.doGenerate({})).rejects.toThrow(
/OPENROUTER_API_KEY not configured/,
);
await expect(model.doStream({})).rejects.toThrow(
/OPENROUTER_API_KEY not configured/,
);
});
});
41 changes: 36 additions & 5 deletions backend/convex/domains/agents/mcp_tools/models/modelResolver.ts
Original file line number Diff line number Diff line change
Expand Up @@ -831,6 +831,38 @@ export const LEGACY_ALIASES: Record<string, ApprovedModel> = {
// RESOLVER FUNCTIONS
// ═══════════════════════════════════════════════════════════════════════════

/**
* A LanguageModel that can be *constructed* without the key it would need to
* *run*.
*
* Why this exists, in the order it bites: Convex analyses every backend module
* on every push, and `domains/agents/core/coordinatorAgent.ts` builds its model
* at module scope (`createCoordinatorAgent(DEFAULT_MODEL).asTextAction(...)`),
* because a Convex function has to be a module-level export. `DEFAULT_MODEL` is
* `kimi-k2.6`, an OpenRouter model. So throwing here — at construction — made
* `convex dev` fail with `InvalidModules: Failed to analyze
* domains/agents/digestAgent.js` for anyone without an OpenRouter account, and
* a failed push means *no* backend at all: no chat, no auth, no persistence,
* for a reader who only wanted the Gemini-backed `/redesign/chat` path.
*
* The error is not removed, only moved to the call that actually needs the key.
* `doGenerate`/`doStream` still throw the same sentence, so a run against an
* unconfigured provider fails loudly instead of silently returning something.
*/
function unconfiguredModel(alias: string, requirement: string): LanguageModel {
const fail = async (): Promise<never> => {
throw new Error(`Model "${alias}" requested but ${requirement}`);
};
return {
specificationVersion: "v2",
provider: "unconfigured",
modelId: alias,
supportedUrls: {},
doGenerate: fail,
doStream: fail,
} as unknown as LanguageModel;
}

/**
* Build a LanguageModel instance from a ModelSpec
*/
Expand All @@ -843,18 +875,17 @@ function buildLanguageModel(spec: ModelSpec): LanguageModel {
case "google": {
const googleProvider = getGoogleProvider();
if (!googleProvider) {
throw new Error(
`Google model "${spec.alias}" requested but no Google API key alias is configured`,
return unconfiguredModel(
spec.alias,
"no Google API key alias is configured",
);
}
return googleProvider(spec.sdkId);
}
case "openrouter": {
const openrouter = getOpenRouterProvider();
if (!openrouter) {
throw new Error(
`OpenRouter model "${spec.alias}" requested but OPENROUTER_API_KEY not configured`
);
return unconfiguredModel(spec.alias, "OPENROUTER_API_KEY not configured");
}
return openrouter.chat(spec.sdkId);
}
Expand Down
60 changes: 58 additions & 2 deletions docs/START_HERE.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,13 +32,69 @@ does not ship a local substitute for. Without a deployment URL there is no
database, so there is no product.

To get past that card you need `VITE_CONVEX_URL` pointing at a real Convex
deployment (`npx convex dev` provisions one against a Convex account). Everything
from Step 3 down needs that backend. Steps 1–2 you can read and run without it.
deployment. Everything from Step 3 down needs that backend. Steps 1–2 you can
read and run without it.

Do **not** start with `npm run dev`. That command runs three processes in
parallel and two of them block on credentials you may not have. See
`docs/codebase/CONCERNS.md`, defect D4.

### Standing the backend up — the whole list, in order

This is the part that used to be missing, and it is the reason nine of the
twelve promotion conditions sat at UNVERIFIED: **you cannot observe anything
below Step 2 without doing this.** It needs a Convex account and a Gemini API
key. There is no offline, fixture, or local-backend substitute in this repo.

```bash
npx convex dev --once --configure new --project <yours> --team <yours>
# provisions an isolated DEV deployment and writes CONVEX_DEPLOYMENT,
# VITE_CONVEX_URL and VITE_CONVEX_SITE_URL into .env.local (gitignored).

npx @convex-dev/auth
# generates JWT_PRIVATE_KEY + JWKS and sets SITE_URL on that deployment.
# Without them every sign-in fails with
# "Missing environment variable `JWT_PRIVATE_KEY`" and no journey can run,
# because live research refuses anonymous callers (Step 5).

npx convex env set GEMINI_API_KEY -- "<your key>"
# Step 7 calls Gemini directly. Without this the run fails at the model call.

npx vite --port 4902 --strictPort --host 127.0.0.1
```

Two traps that cost real time here, so they are written down rather than
rediscovered:

- **`@erquhart/convex-oss-stats` imports `@convex-dev/crons` without declaring
it.** `package-lock.json` is gitignored (CONCERNS C5b), so a fresh
`npm install` can resolve a tree where that transitive package is absent and
the very first push dies with
`Could not resolve "@convex-dev/crons/convex.config"`. It is now a direct
dependency for exactly this reason.
- **A missing OPTIONAL model key used to break the whole deploy.** Convex
analyses every backend module on every push, and
`domains/agents/core/coordinatorAgent.ts` builds `DEFAULT_MODEL`
(`kimi-k2.6`, an OpenRouter model) at module scope. Building a model for an
unconfigured provider threw *at construction*, so `convex dev` failed with
`InvalidModules: Failed to analyze domains/agents/digestAgent.js` unless you
had an OpenRouter account — even though `/redesign/chat` never touches
OpenRouter. `modelResolver.ts` now defers that error to the call that needs
the key. You do **not** need `OPENROUTER_API_KEY` to run the primary journey.

### Proving it, without trusting this page

```bash
node scripts/capture-live-journey.mjs --port 4902
```

That drives J1 (ask → stream → answer with sources), J2 (open the permanent
receipt link cold and get the same answer, proven by the latest-run id being
unchanged) and J4 (cancel, honest terminal state, keep working) in a real
browser at 1280 and 375, and reads the durable rows back out of Convex. It
writes `promotion/evidence/live-journey/report.json` plus eight screenshots, and
exits nonzero if any of it stops being true. **It costs real model calls.**

---

## Step 1 — The browser loads the app and decides whether the backend is usable
Expand Down
64 changes: 60 additions & 4 deletions docs/codebase/CONCERNS.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,21 +176,77 @@ version of the reuse ladder.

---

## C7b — MAJOR for a new engineer: the product is unobservable until you stand up a Convex deployment, and the door has three locks, not one

This is the first thing that will happen to you, so it is the first thing you
should read. Every product route renders **"Convex backend not configured"**
until `VITE_CONVEX_URL` points at a real deployment. That is by design — this
repo keeps all durable state in Convex and ships no local substitute — but
until 2026-08-14 the setup instructions covered one of the three things you
actually need, and the other two failed in ways that do not name themselves.

**Reproduce the working path** (needs a Convex account and a Gemini key; the
full version with the traps is `docs/START_HERE.md` → "Before Step 1"):

```bash
npx convex dev --once --configure new --project <yours> --team <yours>
npx @convex-dev/auth # JWT_PRIVATE_KEY + JWKS + SITE_URL
npx convex env set GEMINI_API_KEY -- "<key>"
npx vite --port 4902 --strictPort --host 127.0.0.1
node scripts/capture-live-journey.mjs --port 4902 # drives J1/J2/J4, costs model calls
```

The three locks, in the order they bite:

1. **A missing transitive dependency stops the first push.**
`@erquhart/convex-oss-stats@0.8.2` imports `@convex-dev/crons/convex.config`
and declares it in neither `dependencies` nor `peerDependencies`. With
`package-lock.json` gitignored (C5b), a fresh install can land a tree without
it and `convex dev` dies on `Could not resolve
"@convex-dev/crons/convex.config"`. Fixed by declaring `@convex-dev/crons`
directly; if you see this again, that is what regressed.
2. **A missing OPTIONAL model key used to fail the ENTIRE deploy.** Convex
analyses every backend module on every push.
`domains/agents/core/coordinatorAgent.ts` builds `DEFAULT_MODEL`
(`kimi-k2.6`, OpenRouter) at module scope, and `buildLanguageModel` threw at
construction when `OPENROUTER_API_KEY` was unset — so the push failed with
`InvalidModules: Failed to analyze domains/agents/digestAgent.js` and you got
no backend at all, for a provider `/redesign/chat` never calls. The error now
lives on `doGenerate`/`doStream` instead, so an unconfigured provider fails
the call that needs it rather than the deploy.
Gated by `backend/convex/domains/agents/mcp_tools/models/modelResolver.test.ts`.
3. **Convex Auth needs its own keys, and nothing on screen says so.** Without
`JWT_PRIVATE_KEY`/`JWKS`, sign-in throws `Missing environment variable
'JWT_PRIVATE_KEY'` from the server. You cannot skip this: live research
rejects anonymous accounts (`requirePaidChatUserId`,
`backend/convex/domains/redesign/chatRuns.ts:159`), so the journey is
unreachable signed out.

**What it costs you if you skip it.** Nine of the twelve promotion conditions
are judged on what a browser shows. Tests and typecheck tell you nothing about
them. See `promotion/PROMOTION_LOG.md` iteration 2.

---

## C8 — Documented product defects, not restated here

`promotion/PROMOTION_LOG.md` carries the reproductions:

- **D1** — no product route works without a Convex cloud deployment; there is no
offline or fixture backend for the product surfaces. Narrowed, not closed.
offline or fixture backend for the product surfaces. **Closed 2026-08-14** as
a *blocker*: with a deployment the journeys run end to end
(`promotion/evidence/live-journey/report.json`). The dependency itself is not
a defect, it is the architecture; the setup path is C7b above.
- **D2** — the red typecheck (C1 above).
- **D3** — the graph rail dies permanently if mounted at zero viewport width
(collapsed drawer, `display:none` tab) and never recovers without a reload.
- **D4** — `npm run dev` blocks on interactive credentials; the frontend-only
path is now documented in the README.

Four of the five product journeys are recorded **UNVERIFIED**, not passing, for
the reason in D1: nobody has driven them without a backend, and creating one was
out of scope. Read that word literally — it does not mean they work.
As of 2026-08-14, **J1, J2 and J4 are driven end to end** against a live
deployment by `node scripts/capture-live-journey.mjs`; J3 (inline correction) and
the product half of J5 are still UNVERIFIED. Read UNVERIFIED literally — it does
not mean they work, it means nobody has watched them.

---

Expand Down
1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,7 @@
"@codemirror/lang-markdown": "^6.5.0",
"@convex-dev/agent": "0.2.10",
"@convex-dev/auth": "0.0.80",
"@convex-dev/crons": "^0.2.2",
"@convex-dev/persistent-text-streaming": "0.2.3",
"@convex-dev/polar": "0.6.3",
"@convex-dev/presence": "0.1.2",
Expand Down
Loading
Loading