Skip to content

feat(quizzes): add AI-graded short answers and improve question quality #851

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
mahruschow wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

feat(quizzes): add AI-graded short answers and improve question quality #851

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6ae101f
fix(quizzes): strengthen question and distractor authoring guidance
mahruschow Oct 4, 2026
357b0ed
feat(quizzes): add AI-graded short-answer questions
mahruschow Oct 4, 2026
ed0263b
fix(quizzes): address prompt guidance and snapshot review findings
mahruschow Oct 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/concepts/items.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,11 @@ Every piece of workspace content is a workspace item. Items can be opened in tab
<Accordion title="Folder">
A container for organizing related items. Folders are part of the workspace item tree and can be opened like any other workspace location.
</Accordion>
<Accordion title="Quiz">
A study set with multiple-choice or short-answer questions, saved answer history, feedback, and explanations. Quiz creation asks the AI to emphasize application and reasoning, use plausible distractors based on misconceptions, and keep options comparable in wording and length. Correct multiple-choice answers are shuffled into a stable display order when questions are created.

Choose Short answer when creating a quiz to write responses and receive AI feedback against a model answer and grading criteria. Each successful grading request uses an AI message from your plan. Answers lock after grading; reset quiz progress for a fresh attempt. If grading fails, the response stays in the current tab so it can be retried. Editing a short-answer question invalidates its previous grade.
</Accordion>
</AccordionGroup>

## Supported Uploads
Expand Down
196 changes: 130 additions & 66 deletions src/features/workspaces/components/quizzes/QuizViewer.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import { useCallback, useEffect, useMemo, useState } from "react";

import { Button } from "#/components/ui/button";
import { Skeleton } from "#/components/ui/skeleton";
import { Textarea } from "#/components/ui/textarea";
import { sendComposerPrompt } from "#/features/workspaces/composer/workspace-composer-actions";
import { StudyProgressStrip } from "#/features/workspaces/components/study/StudyProgressStrip";
import { StudyRichText } from "#/features/workspaces/components/study/StudyRichText";
Expand All @@ -21,6 +22,7 @@ import type { QuizQuestion } from "#/features/workspaces/quizzes/quiz-content";
import {
quizViewerQueryOptions,
useRecordQuizAnswer,
useRecordQuizShortAnswer,
useResetQuizStudyProgress,
} from "#/features/workspaces/quizzes/quiz-queries";
import {
Expand All @@ -30,6 +32,7 @@ import {
} from "#/features/workspaces/quizzes/quiz-study-session";
import {
getQuizAnswer,
isQuizAnswerCorrect,
summarizeQuizStudyProgress,
type QuizStudyState,
} from "#/features/workspaces/quizzes/quiz-study-state";
Expand Down Expand Up @@ -137,6 +140,8 @@ function QuizStudySession({
workspaceId: item.workspaceId,
};
const recordAnswer = useRecordQuizAnswer(queryInput);
const recordShortAnswer = useRecordQuizShortAnswer(queryInput);
const [responses, setResponses] = useState<Record<string, string>>({});
const { isPending: isResetting, mutate: resetProgress } = useResetQuizStudyProgress(queryInput);
const startSession = useCallback(
(nextMode: QuizStudyMode, nextShuffled: boolean) => {
Expand Down Expand Up @@ -196,7 +201,7 @@ function QuizStudySession({
itemId: item.id,
...getQuizStudyViewState({
answered: answer !== undefined,
...(answer ? { correct: answer.selectedOptionId === currentQuestion.correctOptionId } : {}),
...(answer ? { correct: isQuizAnswerCorrect(currentQuestion, answer) } : {}),
mode,
progress: quizProgress,
sessionPosition: currentIndex + 1,
Expand Down Expand Up @@ -286,18 +291,21 @@ function QuizStudySession({
if (!currentQuestion) return null;

const graded = answer !== undefined;
const isCorrect = answer?.selectedOptionId === currentQuestion.correctOptionId;
const selectedOptionId = graded ? answer.selectedOptionId : pendingOptionId;
const isCorrect = isQuizAnswerCorrect(currentQuestion, answer);
const selectedOptionId =
answer && "selectedOptionId" in answer ? answer.selectedOptionId : pendingOptionId;
const options = currentQuestion.kind === "multiple_choice" ? currentQuestion.options : [];
const correctOptionId =
currentQuestion.kind === "multiple_choice" ? currentQuestion.correctOptionId : null;
const response = responses[currentQuestion.id] ?? "";
const isLastQuestion = currentIndex >= studyQuestions.length - 1;
// Khan-style: the pending pick, and the correct option once graded, get an
// outline. Wrong picks keep only the filled X — no row chrome — so they
// stay in the hairline list.
const outlinedOptionIds = new Set(
[!graded ? selectedOptionId : null, graded ? currentQuestion.correctOptionId : null].filter(
Boolean,
),
[!graded ? selectedOptionId : null, graded ? correctOptionId : null].filter(Boolean),
);
const lastOption = currentQuestion.options.at(-1);
const lastOption = options.at(-1);

return (
<section
Expand Down Expand Up @@ -338,65 +346,98 @@ function QuizStudySession({
</div>
</div>

<div role="radiogroup" aria-label="Answer options">
{currentQuestion.options.map((option, optionIndex) => {
const isSelected = option.id === selectedOptionId;
const isCorrectOption = option.id === currentQuestion.correctOptionId;
const previousOption = currentQuestion.options[optionIndex - 1];
return (
<div key={option.id}>
<QuizOptionRule
visible={
!outlinedOptionIds.has(option.id) &&
!(previousOption && outlinedOptionIds.has(previousOption.id))
}
/>
<button
type="button"
role="radio"
aria-checked={isSelected}
disabled={graded || recordAnswer.isPending}
className={cn(
"flex w-full items-center gap-3 rounded-xl border border-transparent px-3 py-4 text-left transition-colors",
!graded && "cursor-pointer hover:bg-accent/40",
!graded && isSelected && "border-primary/60 bg-primary/5",
graded && isCorrectOption && "border-emerald-500/60 bg-emerald-500/10",
graded && !isSelected && !isCorrectOption && "opacity-60",
)}
onClick={() =>
setSession((current) => ({ ...current, pendingOptionId: option.id }))
}
>
<span
{currentQuestion.kind === "short_answer" ? (
<div className="space-y-2">
<label htmlFor={`response-${currentQuestion.id}`} className="text-sm font-medium">
Your answer
</label>
<Textarea
id={`response-${currentQuestion.id}`}
rows={6}
maxLength={4_000}
value={answer && "textResponse" in answer ? answer.textResponse : response}
disabled={graded || recordShortAnswer.isPending || isResetting}
onChange={(event) =>
setResponses((current) => ({
...current,
[currentQuestion.id]: event.target.value,
}))
}
/>
{recordShortAnswer.isPending ? (
<p role="status" className="text-sm text-muted-foreground">
Grading your response...
</p>
) : null}
{recordShortAnswer.isError && !graded ? (
<p role="alert" className="text-sm text-destructive">
{recordShortAnswer.error instanceof Error
? recordShortAnswer.error.message
: "Grading failed. Please try again."}
</p>
) : null}
</div>
) : (
<div role="radiogroup" aria-label="Answer options">
{options.map((option, optionIndex) => {
const isSelected = option.id === selectedOptionId;
const isCorrectOption = option.id === correctOptionId;
const previousOption = options[optionIndex - 1];
return (
<div key={option.id}>
<QuizOptionRule
visible={
!outlinedOptionIds.has(option.id) &&
!(previousOption && outlinedOptionIds.has(previousOption.id))
}
/>
<button
type="button"
role="radio"
aria-checked={isSelected}
disabled={graded || recordAnswer.isPending}
className={cn(
"flex size-6 shrink-0 items-center justify-center rounded-full border text-xs font-semibold text-muted-foreground",
!graded &&
isSelected &&
"border-primary bg-primary text-primary-foreground",
graded &&
isCorrectOption &&
"border-emerald-600 bg-emerald-600 text-white dark:border-emerald-500 dark:bg-emerald-500",
graded &&
isSelected &&
!isCorrectOption &&
"border-red-600 bg-red-600 text-white dark:border-red-500 dark:bg-red-500",
"flex w-full items-center gap-3 rounded-xl border border-transparent px-3 py-4 text-left transition-colors",
!graded && "cursor-pointer hover:bg-accent/40",
!graded && isSelected && "border-primary/60 bg-primary/5",
graded && isCorrectOption && "border-emerald-500/60 bg-emerald-500/10",
graded && !isSelected && !isCorrectOption && "opacity-60",
)}
onClick={() =>
setSession((current) => ({ ...current, pendingOptionId: option.id }))
}
>
{graded && isCorrectOption ? (
<Check className="size-3.5" />
) : graded && isSelected ? (
<X className="size-3.5" />
) : (
String.fromCharCode(65 + optionIndex)
)}
</span>
<QuizRichText content={option.text} compact />
</button>
</div>
);
})}
<QuizOptionRule visible={!lastOption || !outlinedOptionIds.has(lastOption.id)} />
</div>
<span
className={cn(
"flex size-6 shrink-0 items-center justify-center rounded-full border text-xs font-semibold text-muted-foreground",
!graded &&
isSelected &&
"border-primary bg-primary text-primary-foreground",
graded &&
isCorrectOption &&
"border-emerald-600 bg-emerald-600 text-white dark:border-emerald-500 dark:bg-emerald-500",
graded &&
isSelected &&
!isCorrectOption &&
"border-red-600 bg-red-600 text-white dark:border-red-500 dark:bg-red-500",
)}
>
{graded && isCorrectOption ? (
<Check className="size-3.5" />
) : graded && isSelected ? (
<X className="size-3.5" />
) : (
String.fromCharCode(65 + optionIndex)
)}
</span>
<QuizRichText content={option.text} compact />
</button>
</div>
);
})}
<QuizOptionRule visible={!lastOption || !outlinedOptionIds.has(lastOption.id)} />
</div>
)}

{graded ? (
<div
Expand Down Expand Up @@ -429,6 +470,14 @@ function QuizStudySession({
/>
</div>
<QuizRichText content={currentQuestion.explanation} compact />
{currentQuestion.kind === "short_answer" && answer && "textResponse" in answer ? (
<div className="mt-3 space-y-2">
<p className="whitespace-pre-wrap break-words text-sm">{answer.feedback}</p>
<p className="text-xs font-medium text-muted-foreground">AI feedback</p>
<p className="text-sm font-medium">Model answer</p>
<QuizRichText content={currentQuestion.modelAnswer} compact />
</div>
) : null}
</div>
) : null}
</div>
Expand All @@ -440,7 +489,7 @@ function QuizStudySession({
onSelect={goTo}
segments={studyQuestions.map((question, index) => {
const questionAnswer = getQuizAnswer(question, studyState);
const questionCorrect = questionAnswer?.selectedOptionId === question.correctOptionId;
const questionCorrect = isQuizAnswerCorrect(question, questionAnswer);
return {
id: question.id,
label: `Question ${index + 1}: ${
Expand Down Expand Up @@ -483,16 +532,31 @@ function QuizStudySession({
) : (
<Button
className="col-start-3 h-11 justify-self-end rounded-xl px-5 text-sm font-medium sm:h-12 sm:px-6 sm:text-base"
disabled={!pendingOptionId || recordAnswer.isPending}
disabled={
isResetting ||
(currentQuestion.kind === "short_answer"
? !response.trim() || recordShortAnswer.isPending
: !pendingOptionId || recordAnswer.isPending)
}
onClick={() => {
if (currentQuestion.kind === "short_answer") {
if (response.trim())
recordShortAnswer.mutate({
questionId: currentQuestion.id,
textResponse: response.trim(),
});
return;
}
if (!pendingOptionId) return;
recordAnswer.mutate({
questionId: currentQuestion.id,
selectedOptionId: pendingOptionId,
});
}}
>
Submit
{currentQuestion.kind === "short_answer" && recordShortAnswer.isPending
? "Grading..."
: "Submit"}
</Button>
)}
</div>
Expand Down
22 changes: 20 additions & 2 deletions src/features/workspaces/components/study/CreateStudyItemDialog.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ const studyItemDialogConfigs = {
title: "Create a quiz",
countLabel: "Number of questions",
prompt: (count: number, locationPhrase: string, topic: string) =>
`Create a quiz with exactly ${count} multiple-choice questions ${locationPhrase}. Cover: ${topic}`,
`Create a quiz with exactly ${count} multiple-choice questions ${locationPhrase}. Cover: ${topic}. Use relevant workspace source material when available; otherwise create the quiz from the requested topic.`,
},
} as const satisfies Record<StudyItemDialogType, unknown>;

Expand All @@ -52,6 +52,7 @@ export function CreateStudyItemDialog({
}) {
const topicId = useId();
const countId = useId();
const formatId = useId();
const config = studyItemDialogConfigs[type];

return (
Expand All @@ -64,11 +65,14 @@ export function CreateStudyItemDialog({
const rawTopic = formData.get("topic");
const topic = typeof rawTopic === "string" ? rawTopic.trim() : "";
const count = Number(formData.get("count"));
const format = formData.get("format");
if (!topic || ![5, 10, 15, 20].includes(count)) return;
if (
!sendComposerPrompt(
workspaceId,
config.prompt(count, describeStudyItemLocation(parentPath), topic),
type === "quiz" && format === "short_answer"
? `Create a quiz with exactly ${count} short-answer questions ${describeStudyItemLocation(parentPath)}. Cover: ${topic}. Use relevant workspace source material when available; otherwise create the quiz from the requested topic.`
: config.prompt(count, describeStudyItemLocation(parentPath), topic),
)
)
return;
Expand All @@ -83,6 +87,20 @@ export function CreateStudyItemDialog({
<DialogDescription>AI will create it in your current chat.</DialogDescription>
</DialogHeader>
<FieldGroup>
{type === "quiz" ? (
<Field orientation="horizontal" className="items-center justify-between">
<FieldLabel htmlFor={formatId}>Question type</FieldLabel>
<NativeSelect
id={formatId}
name="format"
defaultValue="multiple_choice"
size="sm"
>
<NativeSelectOption value="multiple_choice">Multiple choice</NativeSelectOption>
<NativeSelectOption value="short_answer">Short answer</NativeSelectOption>
</NativeSelect>
</Field>
) : null}
<Field>
<FieldLabel htmlFor={topicId}>What should it cover?</FieldLabel>
<Textarea
Expand Down
Loading
Loading