Skip to content

[Agent Chat feedback] - Step 3: Offer a prefilled feedback card - #3566

Open
zeroliu wants to merge 4 commits into
zeroliu/report-issue-prefillfrom
zeroliu/grill-issue-672
Open

zeroliu wants to merge 4 commits into
zeroliu/report-issue-prefillfrom
zeroliu/grill-issue-672

Conversation

@zeroliu

@zeroliu zeroliu commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Relates to Brevilabs/obsidian-copilot-private#672

Why

When an Agent Chat turn goes wrong, the user usually says so in the chat ("What the fuck? I mean, commit and push. Why do you switch to a different language?"), and that complaint is the best bug report we will ever get: what happened, the user's own words, and how to reproduce it. Today it goes nowhere. Sending it to us means finding Advanced → Report an issue and rewriting the failure from memory, and the agent cannot help because Agent Chat gives agents no Copilot-owned tools.

What

Every Agent Chat backend (Claude, Codex, OpenCode) can now raise a feedback card when the user complains or the agent notices its own mistake. The agent drafts the report; nothing is sent until the user uploads it.

Moment Before After
User complains, or the agent retracts or admits a mistake Complaint stays in the chat A Report this to the Copilot team? card appears under that turn's reply, with a summary and the user's own words. The turn is not paused and no permission prompt appears
User presses Review & report n/a The existing Report an issue dialog opens with the draft as the description (what happened, what the user said, steps to reproduce, backend, model, session, time) and every available attachment ticked; the screenshot is of the Obsidian window showing this chat. Upload works as today, opening the prefilled GitHub issue
Upload lands n/a The card switches to Report sent with the report ID and a link to the issue
User presses Dismiss n/a The card disappears; nothing is sent. An upload that was already in flight still lands, but the card stays dismissed
Agent tries a second card in the same chat n/a Refused, whatever became of the first card
User presses Don't offer again, or turns off Advanced → Offer feedback reports in Agent Chat n/a No more cards; new chats no longer get the tool, and Codex and OpenCode restart to drop the prompt line (offering Reload while a chat is open). Turning it back on works the same way

Non goal

  • Copying Claude Code's feedback UI or /feedback command.
  • A second upload or storage pipeline: reports go through the existing report bundle and upload.
  • Sending anything without the user pressing upload.
  • Feedback cards in legacy (non-agent) Copilot Chat.
  • Triage tooling on our side, such as dashboards or auto-filing issues from reports.
  • Keeping cards across reloading a chat from history: a card lives with the open session.

Screenshot

Card under a turn that also changed a note, below that turn's files-changed card (component gallery):

Feedback card below the files-changed card

Card raised by Claude after the complaint from the issue:

Feedback card under the agent's reply

Review & report opens the existing dialog with the draft filled in and every attachment ticked:

Report an issue dialog prefilled from the card

After upload, the card shows the report ID:

Card showing Report sent with the report ID

Risk

High

Criterion Status Reason
No behavior change, or a cosmetic/copy/docs/config change visible where it renders, or deterministic tests cover the changed behavior ❌ Server, card, session and dialog behavior are covered by unit tests, but whether a model calls the tool cannot be covered by an automated test because it is nondeterministic LLM output
A defect would fail CI or be obvious on first use ❌ A model that stops calling the tool simply shows no card; nothing fails
A revert fully restores prior state, including persisted data ✅ The only persisted addition is an optional boolean setting that older builds drop on load
No auth, permissions, secrets, or input-handling surface changes ❌ Every new and reopened chat now gets the loopback feedback endpoint and its per-session bearer token, so agent-written input reaches it, and the tool is pre-approved so no permission prompt appears
No public API, plugin API, message, or on-disk contract changes ❌ Agents get a new MCP tool and a one-line system-prompt pointer; ACP session new, load and resume now carry Copilot's MCP server
No core-path concurrency, async-lifecycle, or state-machine changes ❌ Opening and resuming a chat now waits for the feedback server to connect, and the server's lifetime follows the session manager
No hot-path behavior lacks deterministic coverage ✅ Session manager tests cover attach, opt-out, a server that cannot start, resume, a reopen that fails, close and shutdown; message tests cover the card's placement, including a streaming reply keeping its viewport height
No new dependency ✅ Dependency manifests are unchanged
Human-only behavior stays in one feature area and surfaces quickly ✅ Confined to Agent Chat; a broken card shows on the next complaint

Review: inspect the offerFeedbackCards settings subscriber in createAgentSessionManager in src/agentMode/index.ts; connectFeedbackServer and its calls in createSession and resumeSessionFromHistory in src/agentMode/session/AgentSessionManager.ts; offerFeedback, dismissFeedbackOffer and markFeedbackReported in src/agentMode/session/AgentSession.ts; findFeedbackCardIndex in src/agentMode/ui/FeedbackCard.tsx; and the feedbackCard wiring in src/agentMode/ui/AgentChatMessages.tsx, line by line. Review the new prompt text: COPILOT_FEEDBACK_POINTER in src/agentMode/backends/shared/agentSystemPrompt.ts. Then run Verification steps 2-7.

Verification

  1. Install this build in a test vault with Claude, Codex and OpenCode configured, and open Agent Chat.
  2. In a new Claude chat send Reply only with the word: hej. Then nothing else., then What the fuck? Why do you answer in Swedish? I asked you in English. This is the third time Copilot switches language on me. A card appears under the second reply, quoting the complaint verbatim, with no permission prompt. Repeat in a Codex chat and an OpenCode chat.
  3. Press Review & report on the card. The dialog's description holds the draft and an Evidence: line naming the backend, model and session; every attachment is ticked. Edit the text, prepare and upload. Your browser opens the prefilled GitHub issue (do not submit it), and the card shows Report sent with the report ID. The bundle's screenshot is of the Obsidian window showing the chat and its card, without the dialog.
  4. In another chat, raise a card and press Dismiss. The card disappears and no report is uploaded.
  5. In the same chat, complain again. No second card appears.
  6. Press Don't offer again on a fresh card in a Codex chat. Advanced → Offer feedback reports in Agent Chat is now off and Codex offers Reload; after reloading, a new chat raises no card after the same complaint.
  7. Vent about your own work in a new chat, for example Ugh, my essay draft is terrible, help me tighten the intro. No card appears.

@vercel

vercel Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
obsidian-copilot-docs Ignored Ignored Oct 15, 2026 4:50am UTC

Request Review

@zeroliu

zeroliu commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-15T05:08:49.557669Z ed57843 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ee97a260f6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

);
options.allowedTools = [
...(options.allowedTools ?? []),
...session.mcpServers.map((server) => `mcp__${server.name}`),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Allow the fully qualified Claude MCP tool

For every Claude session with feedback enabled, the exposed tool is named mcp__obsidian-copilot__send_feedback, but this allow entry is only mcp__obsidian-copilot and therefore matches nothing. The Claude Agent SDK permission docs require either the complete MCP tool name or a pattern after mcp__<server>__; consequently the call falls through to PermissionBridge.canUseTool() and displays the ordinary permission card, contradicting this feature's no-prompt behavior. Use the exact tool name or mcp__${server.name}__*.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changed; this code now lives in #3593. Claude Code permission rules treat a server-level mcp__<server> entry as matching every tool on that server ("mcp__puppeteer matches any tool provided by the puppeteer server"). In the end-to-end trigger runs, the harness rejected every unrelated permission prompt, and Claude still raised the card. That could not happen if send_feedback had hit a prompt.

Comment thread src/agentMode/backends/shared/agentSystemPrompt.ts
Comment thread src/agentMode/session/AgentSessionManager.ts
Comment on lines +162 to +164
let message: JsonRpcRequest;
try {
message = JSON.parse(await readBody(request)) as JsonRpcRequest;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate parsed JSON-RPC values before property access

If an authenticated MCP client sends syntactically valid JSON whose root is null, this cast succeeds and the subsequent message.id access throws. Because the HTTP server invokes handleHttpRequest() with void and no rejection handler, the request is left without a response and the rejection becomes unhandled in the plugin process. Validate that the parsed value is a non-null object before accessing it and return a JSON-RPC invalid-request error for malformed message shapes.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #3593 (2eab740): a body that parses to something other than an object now gets a 400 with JSON-RPC -32600.

Comment thread src/agentMode/session/AgentSession.ts
Comment thread src/agentMode/ui/ReportIssueModal.tsx Outdated
Comment on lines +70 to +73
canCaptureTarget: () => app.workspace.getLeavesOfType(CHAT_AGENT_VIEWTYPE).length > 0,
resolveCaptureTarget: () => {
const leaf = app.workspace.getLeavesOfType(CHAT_AGENT_VIEWTYPE)[0];
if (!leaf) return null;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Capture the chat that opened the feedback report

When more than one Agent Chat leaf is open, including a popout, pressing Review & report on one card still resolves the screenshot target as the first leaf returned by getLeavesOfType(). Because feedback reports preselect the screenshot, preparing and uploading from chat B can therefore send a screenshot of unrelated chat A, including its private note or prompt content, while the attachment list merely says “Screenshot of the Agent Mode pane.” Pass the originating leaf or element from the card instead of selecting an arbitrary global leaf.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Review & report passes its own button as origin, and openReportIssueModal (#3594) captures the Agent leaf that contains that button instead of the first leaf. Covered in FeedbackCard.test.tsx, AgentChatMessages.test.tsx and ReportIssueModal.test.tsx.

@zeroliu

zeroliu commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

End-to-end trigger test: when the card appears, and when it does not

Build ee97a260 in the test vault. Each scenario runs in a fresh chat, so the one-card-per-session rule never hides a result.

Scenario Expected Claude (default) Codex (gpt-6.1-sol) OpenCode (deepseek-v4-pro)
N1 Normal task: explain Zettelkasten, then "Thanks, that's helpful!" No card ✅ 0/2 ✅ 0/2 ✅ 0/1
N2 Venting about own work: "I'm so frustrated with myself, my essay intro is terrible" No card ✅ 0/4 ✅ 0/4 ✅ 0/3
N3 Frustrated at a third party: "My coworker keeps ignoring my emails" No card ✅ 0/4 ✅ 0/4 ✅ 0/3
P1 Complaint: agent answered "hej", user: "Why do you answer in Swedish? … third time Copilot switches language" Card ✅ 2/2 ✅ 2/2 ✅ 1/1
P2 Complaint: missing note, user: "Copilot never finds my notes, this is so frustrating" Card ✅ 2/2 ✅ 1/1 ❌ 0/4

No false positives in 27 no-complaint chats. Complaints were caught 4/4 on Claude and 3/3 on Codex. OpenCode with deepseek-v4-pro caught the language complaint but missed the missing-note complaint in all four runs: it apologises, searches the vault again and answers without raising the card.

In Codex P2 the card appears mid-turn, then Codex in Safe mode asks permission to run the miyo-search script; that unrelated prompt is where the recording ends. The card did not wait for the turn.

Claude: N1, N2, N3, then P1 and P2 each raise a card

feedback-e2e-claude.mp4

Codex: N1, N2, N3, then P1 and P2 each raise a card

feedback-e2e-codex.mp4

OpenCode: N1, N2, N3, P1 raises a card, P2 does not

feedback-e2e-opencode.mp4

@zeroliu

zeroliu commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

Trigger test after rewording the prompt pointer (99766a9)

The previous run showed OpenCode (deepseek-v4-pro) answering "Copilot never finds my notes" by searching again without offering the card. 99766a9 rewords the system-prompt pointer to name that kind of complaint and to say retrying does not replace the card:

When the user complains about you or Copilot (wrong output, failing to find or do something, a repeated problem) or you notice your own mistake, call the obsidian-copilot send_feedback tool once to offer them a bug report card, even if you also retry or fix it. Follow its description.

Same scenarios, same build content, each in a fresh chat:

Scenario Expected Claude (default) Codex (gpt-6.1-sol) OpenCode (deepseek-v4-pro)
N1 Normal task No card ✅ 0/2 ✅ 0/2 ✅ 0/2
N2 Venting about own work No card ✅ 0/3 ✅ 0/3 ✅ 0/3
N3 Frustrated at a third party No card ✅ 0/3 ✅ 0/3 ✅ 0/3
P1 Language complaint Card ✅ 2/2 ✅ 2/2 ✅ 2/2
P2 Missing-note complaint Card ✅ 2/2 ✅ 2/2 ⚠️ 2/5 (was 0/4)

Still no false positives (0 cards in 24 no-complaint chats). Claude and Codex catch every complaint. OpenCode with deepseek-v4-pro now catches the missing-note complaint 2 of 5 times instead of never; the remaining misses are model judgement.

In both Codex P2 runs the card appears first, then Codex keeps going: once it asks the user where the note lives, once it asks to run a shell command. The test runner rejects unrelated shell-permission prompts so a scenario cannot hang.

Claude: N1, N2, N3, P1 card, P2 card (the recording stops once the P2 card is on screen; the reply after it lists other local vault paths)

feedback-e2e-v3b-claude.mp4

Codex: N1, N2, N3, P1 card, P2 card

feedback-e2e-v3-codex.mp4

OpenCode: N1, N2, N3, P1 card, P2 card

feedback-e2e-v3-opencode.mp4

@zeroliu
zeroliu force-pushed the zeroliu/grill-issue-672 branch from 99766a9 to abfc3fa Compare October 13, 2026 01:31
@zeroliu zeroliu changed the title Offer a prefilled feedback card when Agent Chat frustrates the user [Agent Chat feedback] - Step 3: Offer a prefilled feedback card Oct 13, 2026
@zeroliu
zeroliu changed the base branch from master to zeroliu/report-issue-prefill October 13, 2026 01:31
@github-actions github-actions Bot added size/L and removed size/XL labels Oct 13, 2026
@zeroliu
zeroliu added this pull request to stack #3595 October 13, 2026 02:07
@zeroliu
zeroliu force-pushed the zeroliu/grill-issue-672 branch from abfc3fa to 22e6c74 Compare October 13, 2026 02:24
@zeroliu
zeroliu force-pushed the zeroliu/grill-issue-672 branch 2 times, most recently from d3becd0 to b39b5b6 Compare October 14, 2026 21:50
@zeroliu
zeroliu force-pushed the zeroliu/grill-issue-672 branch from b39b5b6 to 24c2e67 Compare October 14, 2026 22:11
zeroliu and others added 4 commits October 14, 2026 21:41
Agents get a plugin-owned send_feedback MCP tool, served over loopback
HTTP with a per-session token, that drafts a report card in the chat.
Review & report opens the existing Report an issue flow prefilled; nothing
is sent until the user uploads. One card per session, and an Advanced
setting turns it off.

Relates to Brevilabs/obsidian-copilot-private#672

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Master's file-edit policy left the full prompt at 2999 bytes, and the
send_feedback pointer adds 287, so the compact-prompt guard moves to 3300.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The report's screenshot is now the whole window the user was working in,
so the card no longer hands its button over to pick a pane.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e with it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zeroliu
zeroliu force-pushed the zeroliu/grill-issue-672 branch from 24c2e67 to ed57843 Compare October 15, 2026 04:50
@zeroliu
zeroliu marked this pull request as ready for review October 15, 2026 05:01

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ed5784324d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agentMode/index.ts
Comment on lines +236 to +239
subscribeToSettingsChange((prev, next) => {
if (prev.agentMode.offerFeedbackCards !== next.agentMode.offerFeedbackCards) {
refreshSystemPromptAffected();
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restart Claude sessions when enabling feedback cards

When a Claude chat was created while feedback cards were disabled, enabling this setting does not activate the feature for that chat. This callback only refreshes descriptors with restartOnSystemPromptChange, while the Claude descriptor sets that flag to false; ClaudeSdkBackendProcess also snapshots both systemPromptAppend and mcpServers when the session opens. The enabled setting therefore has no effect until the user creates or reopens a Claude chat, so this change should explicitly refresh or replace Claude sessions when the setting is enabled.

Useful? React with 👍 / 👎.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant