Repository navigation
feat(ai): capture model calls with Rudimint - #855
Conversation
When RUDIMINT_KEY is set, every Vercel AI Gateway call goes through Rudimint's gateway, which forwards it unchanged and keeps a copy for evaluation and training. Calls are tagged with their task, the thread as the session, the user, and the workspace, turn and model as metadata. Without the key, calls go straight to Vercel as before.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configuration
📒 Files selected for processing (4)
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to No confirmed issue blocks merging. The meaning of the captured model metadata remains unclear. Pre-merge checks |
  |
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Docstring Coverage |  |
Docstring coverage is 12.50% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 3 files. | Write docstrings for the functions missing them to satisfy the coverage threshold. |
✅ Passed checks (4 passed)
-
-
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled. Title check The title clearly and concisely describes the main change: capturing AI model calls with Rudimint. Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request.
-
- Fix all pre-merge checks with AI
✨ Finishing Touches
-
📝 Generate docstrings
-
-
-
-
- Commit to this branch
- Create a new PR
-
-
-
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
|
React Doctor found no new issues. 🎉 Reviewed by React Doctor for commit |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🔇 Additional comments (1)
src/features/workspaces/ai/chat/chat-endpoint.ts-201-201 (1)
201-201: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
⚠️ Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.Record the compaction model in compaction capture metadata.
When a turn selects a model other than
"gemini",capture("chat-compaction")still records the selected chatmodelId. The compaction call at Line 237 usesAI_CHAT_COMPACTION_MODEL_IDinstead. Pass the compaction model ID to the capture callback so Rudimint records the model used for that call.
ℹ️ Review info
⚙️ Run configuration
- Configuration used: defaults
- Review profile: CHILL
- Plan: Advanced
- Run ID:
1e791028-2d91-4c64-8ff2-5e0603ae724a
📒 Files selected for processing (3)
src/features/workspaces/ai/chat/chat-endpoint.tssrc/features/workspaces/ai/gateway.tssrc/features/workspaces/extraction/providers/gemini-image-markdown.ts
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7ee83938fa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
💡 Codex Reviewthinkex/src/features/workspaces/ai/gateway.ts Line 160 in 7ee8393  Declare RUDIMINT_KEY in Wrangler
When the key is supplied through the documented Infisical or local-dev paths, it still cannot reach this lookup: AGENTS.md reference: AGENTS.md:L7-L9 thinkex/src/features/workspaces/ai/gateway.ts Lines 155 to 156 in 7ee8393 Disclose unconditional Rudimint capture before shipping
Once AGENTS.md reference: AGENTS.md:L10-L12 ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
5 issues found across 3 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/features/workspaces/ai/chat/chat-endpoint.ts">
<violation number="1" location="src/features/workspaces/ai/chat/chat-endpoint.ts:197">
P2: Rudimint captures and retains full request/response content for every chat turn, compaction summary, and title generation, even when the user opted out of analytics content capture. The existing telemetry on the lines above this one gates content behind `analyticsConsent` (`includeContent: analyticsConsent` in `captureAiChatGeneration`), but this new path has no equivalent gate, so opt-out users' conversation content is copied to a third party whenever `RUDIMINT_KEY` is configured. Gate `capture` on `analyticsConsent` (or add an explicit consent/PII-handling step before rollout) rather than relying only on the planned privacy-policy update.</violation>
<violation number="2" location="src/features/workspaces/ai/chat/chat-endpoint.ts:201">
P2: Custom agent: **Flag AI Slop and Fabricated Changes**
Rudimint receives the session's `modelId` for compaction and title calls, even though those calls use `AI_CHAT_COMPACTION_MODEL_ID` and `AI_THREAD_TITLE_GATEWAY_MODEL`. Pass the actual gateway model into each capture so the recorded model matches the call.</violation>
<violation number="3" location="src/features/workspaces/ai/chat/chat-endpoint.ts:237">
P2: Custom agent: **Flag AI Slop and Fabricated Changes**
The new Rudimint capture plumbing is untested: no regression test verifies that chat, compaction, or title calls propagate the task/session/user/metadata headers when `RUDIMINT_KEY` is set. Add a focused gateway/model test that asserts those settings without calling the live Rudimint service.</violation>
</file>
<file name="src/features/workspaces/extraction/providers/gemini-image-markdown.ts">
<violation number="1" location="src/features/workspaces/extraction/providers/gemini-image-markdown.ts:42">
P2: Custom agent: **Flag AI Slop and Fabricated Changes**
The new Rudimint capture for image extraction has no regression test: no test asserts that `task` and the workspace/item metadata reach the configured gateway headers. Add a gateway/provider test covering the `RUDIMINT_KEY` path and these values.</violation>
</file>
<file name="src/features/workspaces/ai/gateway.ts">
<violation number="1" location="src/features/workspaces/ai/gateway.ts:159">
P2: Custom agent: **Flag AI Slop and Fabricated Changes**
The new Rudimint routing and capture headers have no regression test. Add tests through the exported model factory for enabled routing and the unchanged Vercel fallback when `RUDIMINT_KEY` is absent.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Re-trigger cubic
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
- Declare RUDIMINT_KEY in every secrets.required list so the Worker and local dev actually receive it. - Drop modelId from capture metadata; Rudimint records the model each call actually used, which differs for compaction and titles. - Name Rudimint in the privacy policy's providers and AI sections.
There was a problem hiding this comment.
2 issues found across 4 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/routes/privacy.tsx">
<violation number="1" location="src/routes/privacy.tsx:42">
P2: This claims every AI request and response passes through Rudimint, but recording transcription calls Cloudflare Workers AI directly. Limit the disclosure to Vercel AI Gateway requests and note that Rudimint is used when configured.</violation>
</file>
<file name="src/features/workspaces/ai/chat/chat-endpoint.ts">
<violation number="1" location="src/features/workspaces/ai/chat/chat-endpoint.ts:201">
P2: Custom agent: **Flag AI Slop and Fabricated Changes**
The PR description still claims chat, compaction, and title Rudimint metadata includes `model`, but `capture` now sends only `workspaceId` and `traceId` in `x-rudimint-metadata`. Update the table to say model attribution is recorded from the request separately; do not re-add `modelId` to this session metadata.</violation>
</file>
Tip: cubic used a learning from your PR history. Let your coding agent read cubic learnings directly with the cubic MCP.
View guided diff | Re-trigger cubic
1 participant
ThinkEx's model calls aren't recorded anywhere we can evaluate or train on: PostHog keeps metadata, and content only with analytics consent. This routes every model call through Rudimint, which forwards it to Vercel AI Gateway unchanged and keeps a copy of the request and response.
How it works
flowchart LR TX[ThinkEx Worker] -->|same AI SDK request + x-rudimint-* headers| R[api.rudimint.com/vercel] R -->|forwarded unchanged| V[Vercel AI Gateway] V --> P[OpenAI / Anthropic / Google] P --> V --> R -->|streamed back| TX R -.->|copy after the response| S[(Rudimint)]getWorkspaceAiLanguageModelForGatewayModel(src/features/workspaces/ai/gateway.ts). WhenRUDIMINT_KEYis set,createGatewaygets Rudimint's base URL and headers; without it, calls go to Vercel exactly as before. Vercel's key, BYOK routing, fallbacks and streaming are untouched.tasksessionusermetadatachatchat-compactionchat-titleimage-to-markdownEvals and the probe script stay untagged.
Rollout
RUDIMINT_KEY(a key from the ThinkEx tenant in Rudimint) to Infisical for each environment. It's optional on purpose: merging before the key exists changes nothing.Validation
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Summary by CodeRabbit