feat: opt-in AI refinement of sample displayName and description - #582
Merged
Conversation
Adds a 'refine_with_ai' workflow_dispatch input (default off). When off, behavior is unchanged: empty displayName is derived from the folder name and empty description is filled by the LLM, existing values untouched. When on, the LLM reviews both displayName and description of every template against its README and rewrites them only when they no longer fit, keeping values that already match. displayName now follows the Foundry Sample Finder convention (short human-friendly Title Case names). Shared LLM plumbing is extracted into callLLMForJson to avoid duplicating the request/parse logic.
The refine path fires one LLM request per template (~90 in a full run); a burst can trip the rate limiter at the sliding-window edge even with ample quota, which previously surfaced as many 429s and dropped fields. Wrap callLLMForJson's request in a retry loop that retries 429/5xx/network errors with jittered exponential backoff, honoring a server Retry-After (capped). Non-retryable 4xx and empty/invalid responses behave as before.
The first AI refine should regenerate every displayName/description without being anchored by the previous catalog's values (the keep-if-fits logic would otherwise preserve them). Adds an opt-in ignore_existing_catalog workflow input (default false) that skips mergeExistingDisplayFields via the IGNORE_EXISTING env, so all fields start empty and are fully regenerated. Normal runs are unchanged and still preserve PM-curated values.
…ne-display-fields
After merging template/dev (PR #580 added fetch-layer retry with delay/parseRetryAfterMs/computeBackoffMs), the LLM retry path's local sleep/retryAfterMsFromResponse duplicated those helpers. Drop the duplicates and reuse the shared delay/parseRetryAfterMs.
Two samples in the same language+framework+protocol group getting the same displayName (e.g. both '05-workflows' -> 'Multi-Agent Workflow') is confusing since the user sees them together. Add detection-only warnDuplicateDisplayNames that flags such same-group collisions in the CI step summary so a PM can disambiguate before merge. Cross-group duplicates are allowed (never seen side by side). The catalog is left unchanged.
Replaces the catalog with the PM's validated final version (gist 353c256): 15 displayName changes verified against each sample's source + deps (MCP/Voice Live capability surfacing), plus the two 'Multi-Agent Workflow' entries disambiguated by framework. Normal (non-refine) sync runs preserve these via mergeExistingDisplayFields, so they persist unless someone explicitly opts into refine_with_ai/ignore_existing_catalog.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds an opt-in toggle to the sample-catalog sync workflow to allow an LLM to refine existing displayName and description values (instead of only filling blanks), with retry/backoff and a shared LLM JSON-call helper in the generator script.
Changes:
- Add
refine_with_ai(andignore_existing_catalog) workflow inputs and pass them to the generator via env vars. - Refactor LLM calling into a shared
callLLMForJsonhelper, add refine-mode prompting fordisplayName+description, and introduce retry/backoff for LLM calls. - Regenerate
samples/hosted-agent/sample-catalog.jsonwith updateddisplayName/descriptionvalues.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
samples/hosted-agent/sample-catalog.json |
Regenerated catalog output with updated commitSha/generatedAt and many refined displayName/description entries. |
.github/workflows/sync-sample-catalog.yml |
Adds opt-in workflow inputs and passes them to the generator as AI_REFINE / IGNORE_EXISTING. |
.github/scripts/generate_sample_catalog.mjs |
Implements refine vs fill-blank paths, shared LLM JSON helper with retries, and duplicate displayName detection. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
1
to
+4
| { | ||
| "commitSha": "84cb8c9a15d7a8bd40e1d80556e00557ea917049", | ||
| "commitSha": "3b98218db7e9a92367dc3e4b6638fd97ac7c9502", | ||
| "repo": "https://github.com/microsoft-foundry/foundry-samples/", | ||
| "generatedAt": "2026-07-23T07:56:05Z", | ||
| "generatedAt": "2026-07-23T07:59:46Z", |
| const usageHint = reasoningTokens !== undefined ? ` (reasoning_tokens=${reasoningTokens})` : ''; | ||
| warn(`LLM returned empty content for ${samplePath} (finish_reason=${finishReason}${usageHint}); description left empty. If finish_reason is "length", raise AZURE_OPENAI_MAX_COMPLETION_TOKENS or set AZURE_OPENAI_REASONING_EFFORT.`); | ||
| // Parse JSON response — strip markdown code fences if present. | ||
| const jsonStr = content.replace(/^```json\s*/, '').replace(/\s*```$/, ''); |
Comment on lines
+751
to
+755
| const DESCRIPTION_GUIDANCE = `A good description: | ||
| - Is one sentence, max 100 characters, plain text (no markdown) | ||
| - Describes what the sample does, not how it is implemented | ||
| - Does NOT include language, protocol, or framework names, or words like "Sample" / "Demo" | ||
| Examples: |
huimiu
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in
refine_with_aitoggle to the Sync Sample Catalog workflow so AI can review and improve bothdisplayNameanddescription— not just fill blanks. Default behavior is unchanged.Behavior
Toggle OFF (default): identical to today — an empty
displayNameis derived from the folder name, an emptydescriptionis generated by the LLM, and any value that already exists (PM-curated or preserved from the previous catalog) is left untouched.Toggle ON (
refine_with_ai: true): the LLM reviews every template's currentdisplayName+descriptionagainst its README and rewrites a field only when it no longer fits the sample's scenario — values that already match are kept verbatim (refinement is not a forced rewrite).displayNamefollows the Foundry Sample Finder convention: short, human-friendly Title Case scenario names (e.g. \Basic Agent, \Foundry Toolbox, \Azure Search RAG) instead of raw folder names.Changes
sync-sample-catalog.yml: newrefine_with_aiboolean input (defaultfalse), passed to the generator asAI_REFINE.generate_sample_catalog.mjs:AI_REFINEflag gating the two paths.callLLMForJsonhelper (removes duplicated request/parse/error handling).refineDisplayFieldsWithLLM(displayName + description, README-aware, keep-if-fits).autoFillDisplayFieldssplit intofillBlankDisplayFields(default) andrefineAllWithLLM(opt-in).Notes
sample-catalog.jsonis regenerated by the workflow and intentionally NOT included.Validation
node --checkpasses. Recommend a manualworkflow_dispatchrun (both toggle states) before merge.