Skip to content

Stand the backend up, and drive the primary journey on it - #615

Closed
HomenShum wants to merge 1 commit into
mainfrom
codex/live-backend-observability
Closed

Stand the backend up, and drive the primary journey on it#615
HomenShum wants to merge 1 commit into
mainfrom
codex/live-backend-observability

Conversation

@HomenShum

Copy link
Copy Markdown
Owner

Nine of the twelve promotion conditions are judged on what a browser shows, and all nine were UNVERIFIED for one reason: every product route rendered "Convex backend not configured" (defect D1). This provisions an isolated Convex dev deployment, drives J1, J2 and J4 end to end against it, and commits both the captures and the scripts that regenerate them.

Two things were actually broken behind that door

Neither names itself, and both stop a cold clone dead.

  1. @erquhart/convex-oss-stats imports @convex-dev/crons and declares it nowhere — not in dependencies, not in peerDependencies. With package-lock.json gitignored, a fresh install can land a tree without it and the very first convex dev push dies on Could not resolve "@convex-dev/crons/convex.config". Now a direct dependency.

  2. A missing OPTIONAL model key failed the ENTIRE deploy. Convex analyses every backend module on every push; coordinatorAgent.ts builds DEFAULT_MODEL (kimi-k2.6, OpenRouter) at module scope because a Convex function must be a module-level export; and buildLanguageModel threw at construction for an unconfigured provider. So convex dev failed with InvalidModules: Failed to analyze domains/agents/digestAgent.js unless you had an OpenRouter account — for a provider /redesign/chat never calls. The error now lives on doGenerate/doStream, so an unconfigured provider fails the call that needs the key rather than the deploy that does not. Gated by a new modelResolver.test.ts case that deletes the key, constructs the model, and asserts both halves.

What now runs, and what proves it

node scripts/capture-live-journey.mjs --port 4902 → exit 0, 10/10 checks. It asserts, it does not narrate:

  • the surface mounts with data-empty="true" on a live backend;
  • the run streams with a named-stage checklist and a durable data-chat-run-id;
  • the sealed packet's source count and evidence agree, in both directions;
  • the run row reaches complete with 30 ordered redesignChatStreamEvents rows read back out of Convex — so "it streamed" and "it persisted" are separate claims;
  • /redesign/chat/r/<hash> opened cold replays rather than re-runs, proven by getLatestOwnedRun().runId being unchanged;
  • Stop yields an honest terminal state with no fabricated packet, and the session keeps working;
  • a 2-character prompt is rejected by startChat at the trust boundary;
  • zero console errors and zero failed requests, at 1280 and at 375.

Plus scripts/audit-web-quality.mjs (Lighthouse 13.4.1 + axe-core 4.13.0 on the production build) and scripts/review-web-interface-guidelines.mjs with the written review at promotion/evidence/wig-review/REVIEW.md. That review is a review, not a Lighthouse score — the two measure different things and are committed separately.

Scorecard: 1/12 → 7/12, and UNVERIFIED is gone from the table

Conditions 1, 3, 4, 5, 6 and 9 move on committed evidence. Conditions 2, 7, 8, 10 and 11 FAIL on measured numbers, which is the point — running the thing found three new defects:

  • D5 — a rejected prompt shows "The live chat run could not be started." and sends the actual reason only to the console.
  • D6 — LCP 10 827 ms on the production build, plus four Web Interface Guidelines deviations (no h1, no skip link, sub-44px touch targets including the submit button, 14.5px composer font).
  • D7 — the same question returned 3, 2, 1, 0, 1 and 0 grounded sources across six runs.

Setup is written down in docs/START_HERE.md "Before Step 1" and docs/codebase/CONCERNS.md C7b, including both traps, so the next cold reader does not rediscover them.

Not touched: the known apinever typecheck cascade (D2).

🤖 Generated with Claude Code

Nine of the twelve promotion conditions are judged on what a browser shows,
and all nine were UNVERIFIED for one reason: every product route rendered
"Convex backend not configured" (defect D1). This provisions an isolated
Convex dev deployment, drives J1, J2 and J4 end to end against it, and
commits both the captures and the scripts that regenerate them.

Two things were actually broken behind that door, and neither names itself:

- @erquhart/convex-oss-stats imports @convex-dev/crons and declares it in
  neither dependencies nor peerDependencies. With package-lock.json
  gitignored, a fresh install can land a tree without it and the very first
  `convex dev` push dies on "Could not resolve
  @convex-dev/crons/convex.config". Declared directly.

- A missing OPTIONAL model key failed the ENTIRE deploy. Convex analyses
  every backend module on every push; coordinatorAgent.ts builds
  DEFAULT_MODEL (kimi-k2.6, OpenRouter) at module scope because a Convex
  function must be a module-level export; and buildLanguageModel threw at
  construction for an unconfigured provider. So `convex dev` failed with
  "InvalidModules: Failed to analyze domains/agents/digestAgent.js" unless
  you had an OpenRouter account -- for a provider /redesign/chat never
  calls. The error now lives on doGenerate/doStream instead, so an
  unconfigured provider fails the call that needs the key rather than the
  deploy that does not. The same shape is fixed for the google branch, and
  modelResolver.test.ts gates it: construct DEFAULT_MODEL with the key
  deleted, assert construction succeeds and the call still rejects.

New producers, each writing a committed artifact:

- scripts/capture-live-journey.mjs -- J1 (ask, stream, sealed packet),
  J2 (the permanent link replays instead of re-running, proven by
  getLatestOwnedRun().runId being unchanged), J4 (cancel, honest terminal
  state, keep working), the server-side validation error path, and the same
  journey at 375. It also reads the run row and its stream events back out
  of Convex, so "it streamed" and "it persisted" are separate claims.
- scripts/audit-web-quality.mjs -- Lighthouse 13.4.1 + @axe-core/cli 4.13.0
  against the production build under vite preview.
- scripts/review-web-interface-guidelines.mjs -- DOM measurements for the
  Web Interface Guidelines review written up in
  promotion/evidence/wig-review/REVIEW.md. A review, not a score; the two
  measure different things and are committed separately.

Scorecard: 1/12 -> 7/12, and UNVERIFIED is gone from the table. Conditions
1, 3, 4, 5, 6 and 9 move on committed evidence. Conditions 2, 7, 8, 10 and
11 FAIL on measured numbers, which is the point -- three new defects were
found by running the thing: D5 (a rejected prompt shows no reason on
screen), D6 (LCP 10.8 s plus four guideline deviations), D7 (the same
question returned 3, 2, 1, 0, 1 and 0 grounded sources across six runs).

Setup is written down in docs/START_HERE.md "Before Step 1" and
docs/codebase/CONCERNS.md C7b, including both traps above, so the next cold
reader does not rediscover them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@gitguardian

gitguardian Bot commented Aug 14, 2026

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
36087749 Triggered Generic Password a4e9cd1 scripts/capture-live-journey.mjs View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@vercel

vercel Bot commented Aug 14, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
nodebench-ai Ready Ready Preview Aug 14, 2026 3:43am

Request Review

@github-actions

Copy link
Copy Markdown

PR size advisory

This PR adds 19309 lines of substantive change. CONTRIBUTING.md defines a soft limit of ~400 LOC.

If the PR is genuinely cohesive (e.g. an architecture map, a generated migration, a deletion of a dead module), no action is needed. Otherwise consider:

  • splitting into 2-3 PRs along independent concerns
  • pre-discussing the architecture change in a GitHub Discussion before merge

This is advisory — it does not block the merge.

@HomenShum

Copy link
Copy Markdown
Owner Author

Superseded by #616, which carries the same work on a conventionally-named branch with a conventional title, rebased onto main so the expired MCP publish token is no longer in the tree.

@HomenShum HomenShum closed this Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant