Add E2E test pipeline for azure.ai.agents extension (Tier 0/1)

[v1212]

Status: DRAFT - pending secrets setup and --allow-tool syntax confirmation

Summary

Adds a Copilot CLI-driven E2E test pipeline for the azure.ai.agents extension. Uses the official copilot -p programmatic mode (per GitHub docs).

Architecture (identical to local testing)

Copilot CLI (npm install -g @github/copilot)
  | MCP protocol (stdio, config: ~/.copilot/mcp-config.json)
cli-interactive-tester (MCP server)
  | tmux sessions
azd ai agent CLI (under test)

Workflow

1. Checkout (ref: trangevi/test-scenarios)
2. Setup: Node.js, Go, Python 3.12
3. Install Copilot CLI (npm install -g @github/copilot)
4. Build azd + install extension
5. Install cli-interactive-tester (git clone + pip install -e)
6. Azure Login + gh auth (Tier 1/2 only)
7. Create profile.local.yaml from secrets
8. Configure MCP (write ~/.copilot/mcp-config.json)
9. Run: copilot -p "" --allow-tool='cli-interactive-tester(*)' --no-ask-user
10. Upload results as artifacts
11. Always-run teardown for Tier 2

Trigger

• workflow_dispatch only (tier selector: 0 / 0+1 / 0+1+2)
• No PR trigger until pipeline is validated

Items to confirm

1. --allow-tool syntax for MCP-registered tools (is it cli-interactive-tester(*)?)
2. COPILOT_PAT - who creates the Fine-grained PAT with "Copilot Requests" permission?
3. cli-interactive-tester repo - public or private? If private, need authenticated clone
4. prompt-ci-run.md - needs to be created in scenarios directory (CI-adapted prompt)

Secrets needed

• COPILOT_PAT (Fine-grained PAT with "Copilot Requests" permission)
• AZURE_CLIENT_ID / AZURE_TENANT_ID / AZURE_SUBSCRIPTION_ID
• FOUNDRY_PROJECT_ENDPOINT
• GH_TOKEN (for gh auth + manifest-url scenario)

Validation

All 44 scenarios validated PASS in manual Copilot-driven local run.

[Copilot]

Pull request overview

Adds a new GitHub Actions workflow to run end-to-end (E2E) scenario-based checks for the azure.ai.agents azd extension on PRs and via manual dispatch, intended to cover Tier 0 (offline) and Tier 1 (Azure-authenticated, no provision) scenarios, with a commented Tier 2 placeholder.

Changes:

• Introduces .github/workflows/e2e-ext-azure-ai-agents.yml with Tier 0 and Tier 1 jobs that build azd, install the extension, and execute scenario YAMLs.
• Adds workflow_dispatch input scaffolding for a future Tier 2 live-Azure run (currently commented out).

[glharper]

Review findings

This is a draft targeting wujia/e2e-pipeline-staging, and several items are already flagged by the author (--allow-tool syntax, COPILOT_PAT, repo visibility, prompt-ci-run.md). The notes below are beyond those.

🔴 Concrete bug — azure/login will fail without id-token: write

The Azure Login step uses OIDC federated credentials (client-id/tenant-id/subscription-id, no client-secret). OIDC login requires permissions: id-token: write, but the workflow only declares contents: read. As written, azure/login@v2 cannot fetch the OIDC token and Tier 1/2 runs will fail at the login step. Add id-token: write.

🟠 Convention — authenticated workloads should prefer ADO pipelines

Per cli/azd/AGENTS.md: "Prefer Azure DevOps pipelines for jobs that need secrets or Azure credentials — the team uses internal ADO pipelines for authenticated workloads in this public repo." No other workflow in .github/workflows/ currently uses azure/login — this PR would be the first. The Tier 1/2 path (Azure creds + COPILOT_PAT + GH_TOKEN + FOUNDRY_PROJECT_ENDPOINT) is exactly the kind of authenticated workload the team routes through ADO. Worth confirming with maintainers before this leaves staging. (Tier 0 is offline/no-auth and is a good fit for GitHub Actions.)

🟠 Tier 2 teardown can silently leak Azure resources

cd ~/working/azd-agents-shared/*/ 2>/dev/null && azd down --force --purge || true

• If the glob matches multiple dirs, cd gets multiple args and fails.
• If cd fails (path missing, e.g. setup aborted early), && short-circuits and || true keeps the job green — so no teardown runs and resources keep costing money with no signal. Given the explicit "~$2-5" cost note, consider failing loudly (or emitting a ::warning::) when the expected directory isn't found, and quoting/guarding the glob.

🟡 Minor / hardening

• Secret interpolation in run: scripts. "Create test profile" inlines ${{ secrets.AZURE_SUBSCRIPTION_ID }} / FOUNDRY_PROJECT_ENDPOINT directly into the shell. Prefer passing them via env: (as the Copilot step correctly does for the token) to avoid shell-injection / log-exposure edge cases.
• actions/setup-node@v4 has no node-version — pin one for reproducibility.
• Hardcoded ref: trangevi/test-scenarios (already TODO'd) — must not reach main; it ties the workflow to an unmerged branch.
• PATH across steps — re-exporting cli/azd in each step works, but echo "$DIR" >> $GITHUB_PATH once (per AGENTS.md guidance) is cleaner.
• Trailing blank lines at EOF of the workflow file.
• COPILOT_GITHUB_TOKEN env name — verify the Copilot CLI actually reads this (on the author's confirm list).

✅ Good

• Least-privilege permissions block present (just needs id-token).
• workflow_dispatch-only with tier gating and a Tier 2 cost-confirmation input — appropriately cautious.
• concurrency with cancel-in-progress: false avoids clobbering in-flight Azure resource runs.
• Tier-gated steps (if: contains(inputs.tier, ...)) are consistent and correct.
• prompt-ci-run.md is clear; if: always() artifact upload + teardown are sensible.

Bottom line: Right architecture and appropriately gated, but not mergeable beyond staging yet: fix the id-token: write permission (hard blocker for Tier 1/2), resolve the ADO-vs-Actions convention question for the authenticated tiers, and harden the teardown. The Tier-0-only path is close to ready.

[v1212]

Thanks @glharper, great review! Addressed all items in the latest commit (d08b5c9). Here's the breakdown:

🔴 id-token: write — Fixed. Added to permissions block.

🟠 ADO convention — Totally valid point. I'm aware of the convention in AGENTS.md. This PR's goal is to validate whether this pattern (Copilot driving cli-interactive-tester to exercise azd interactively) works end-to-end in GitHub CI. If the team decides authenticated tiers should move to ADO, I'm fully supportive of that. Happy to discuss in our next sync.

🟠 Teardown glob — Fixed. Now uses explicit guard with ::warning:: annotation instead of silent || true.

🟡 Secret interpolation — Fixed. Secrets now passed via env: block, referenced as ${VAR} in shell.

🟡 Node version — Pinned to node-version: '20'.

🟡 PATH — Switched to >> $GITHUB_PATH in the build step, removed per-step exports.

🟡 Hardcoded ref — Keeping TODO as-is, will update once #8524 merges.

Re: overall direction — This is still a draft targeting a staging branch for all tests with the MCP tool and copilot driven framework. If later it is determined to move tier 1/2 tests to ADO, I would propose to abandon this PR but implement same pipeline in ADO for tier 0/1/2, leaving only tier 0 in GitHub action seems not helpful enough.

[v1212]

https://github.com/v1212/azure-dev/actions/runs/27403167828 validated the pipeline in fork repo, it proved the testing framework working with copilot, mcp tool, and test cases. @glharper