Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
154 commits
Select commit Hold shift + click to select a range
428e680
Update CLM MicroHack model fleet: drop Claude, add gpt-4.1-mini renewal
glejdis Aug 3, 2026
a915134
Update user-journey diagram image
glejdis Aug 3, 2026
49a3280
Update architecture.png with refreshed model fleet diagram
glejdis Aug 3, 2026
c121ff0
Switch renewal agent model to gpt-5.4-nano
glejdis Aug 3, 2026
2c6baae
Sync latest CLM microhack content: add C1 step screenshots + doc refresh
glejdis Aug 3, 2026
5df2083
docs(challenge-01): make Path B local-PDF fallback first-class for no…
glejdis Aug 3, 2026
e6e2f80
docs(challenge-01): make Path B local-PDF fallback first-class for no…
glejdis Aug 3, 2026
6f378e8
Update architecture diagram to gpt-5.4-nano model fleet
glejdis Aug 3, 2026
b5e828e
docs(challenge-01): restore .png step screenshots (keep Path B fallba…
glejdis Aug 3, 2026
2a14f4a
Update user-journey diagram to gpt-5.4-nano model fleet
glejdis Aug 3, 2026
76ec9eb
docs(marketing): update TechTalk_ContractMgmt_AgenticAI deck
glejdis Aug 3, 2026
b803c10
docs(marketing): list full TechTalk_ContractLifecycleManagement deck …
glejdis Aug 3, 2026
7cf9b7d
docs(marketing): add full TechTalk_ContractLifecycleManagement deck (…
glejdis Aug 3, 2026
788bde9
docs(challenge-01): drop placeholder SVGs that now have PNG screenshots
glejdis Aug 4, 2026
7f232da
docs(clm challenge-01): repoint solution-01 image refs to PNG
glejdis Aug 4, 2026
320420b
feat(clm challenge-02): add publish_agent.py to surface agent in Foun…
glejdis Aug 4, 2026
3213a3f
docs(clm challenge-02/03): fix misplaced publish note + clarify it's …
glejdis Aug 4, 2026
b9a8282
Add real challenge-02 screenshots and repoint md to PNG
glejdis Aug 4, 2026
b9d8e34
Remove orphaned challenge-02 step SVG placeholders
glejdis Aug 4, 2026
8c7c3df
Compress TechTalk CLM deck media (517MB source -> 68MB)
glejdis Aug 4, 2026
fea907c
Gate groundedness on groundable rows only (exclude refusal/tool_call)
glejdis Aug 4, 2026
4696c69
Fix gpt-5.4-nano deployment version (2025-04-14 -> 2026-03-17)
glejdis Aug 4, 2026
4df21b2
Fix NLTK regex import block in evaluators.py (Challenge 3 Task 3)
glejdis Aug 4, 2026
8bbc8de
Relaunch eval scripts under PYTHONSAFEPATH to clear NLTK import guard
glejdis Aug 4, 2026
5b9992b
Fix NLTK import-guard crash by preloading eval SDK from a temp cwd
glejdis Aug 4, 2026
6bcb7c0
Preflight empty clm-corpus index before running evaluation
glejdis Aug 4, 2026
391a461
Make Challenge 3 tracing nav version-aware (New + classic Foundry)
glejdis Aug 4, 2026
b84cd40
Add --explain flag to evaluators.py for per-row groundedness diagnosis
glejdis Aug 4, 2026
8b00f12
Enrich groundedness eval context to fix low quality-gate score
glejdis Aug 4, 2026
be73b5d
Calibrate groundedness quality gate to 3.0
glejdis Aug 4, 2026
0d829d2
Add CLM rubric evaluator and make it the Challenge 3 quality gate
glejdis Aug 4, 2026
b25e724
Challenge 4: add MCP server --list smoke test and explain harmless st…
glejdis Aug 5, 2026
01491aa
Challenge 4: move mcp.json to repo-root .vscode so VS Code discovers …
glejdis Aug 5, 2026
336f1db
Challenge 3: add merged step PNGs and point screenshots at them
glejdis Aug 5, 2026
d399390
Challenge 3: retire step placeholder SVGs, point walkthrough at PNGs
glejdis Aug 5, 2026
2d2110a
Challenge 4: clarify that running server.py does not register clm-mcp…
glejdis Aug 5, 2026
5e96e5a
Challenge 4: explain stdio is a per-client subprocess + how to health…
glejdis Aug 5, 2026
8116368
Challenge 4: host the MCP server remotely + consume it from Foundry
glejdis Aug 5, 2026
39f1d17
Challenge 4: add a "Task 4 at a glance" map + which-path hint
glejdis Aug 5, 2026
261cff9
Challenge 4: zero-config MCP deploy (auto-load .env + auto-discover R…
glejdis Aug 5, 2026
3a796e9
Challenge 4: make the terminal MCP client the primary local-consume path
glejdis Aug 5, 2026
c79bcbb
Challenge 4: collapse the three stdio-server callouts into one click-…
glejdis Aug 5, 2026
204e4be
Challenge 4: fold the optional 'Consume it locally' body into a colla…
glejdis Aug 5, 2026
c618404
Challenge 4: clearer stdio 'no output = success' + fix agent_framewor…
glejdis Aug 5, 2026
c65501f
Challenge 4: add sys.executable diagnostic to agent_framework import row
glejdis Aug 5, 2026
bf89613
Challenge 4: make remote MCP (Task 4) a core step, not optional
glejdis Aug 5, 2026
36064f2
Trim challenges to fit 4.5h: Path B default (C1), alerts optional (C5…
glejdis Aug 5, 2026
f1875a4
docs(challenges): simplify structure — Services tables, consistent he…
glejdis Aug 5, 2026
deb6a44
docs(challenge-05): scope 'optional' to alerts in title
glejdis Aug 6, 2026
100682b
docs: add cross-challenge narrative — 'How this fits' bookends + READ…
glejdis Aug 6, 2026
b978eb5
docs(challenge-04): simplify orchestration vs MCP flow
glejdis Aug 6, 2026
2e306ed
fix(challenge-04): MCP server 421 Invalid Host header behind ACA ingress
glejdis Aug 6, 2026
ea760be
fix(challenge-04): self-diagnose remote MCP 421 in orchestrator_mcp
glejdis Aug 6, 2026
d58afbb
docs(challenge-04): name the Foundry MCP agent instead of leaving 'ne…
glejdis Aug 6, 2026
08ac2ce
fix(challenge-05): publish the MCP-backed clm-contract-agent, not the…
glejdis Aug 6, 2026
e4b1e1a
fix(mcp-server): make draft/analyze tools async to fix event-loop crash
glejdis Aug 7, 2026
bbb77bb
docs(challenge-01): remove optional Go Further section
glejdis Aug 7, 2026
87c3067
docs(challenge-01): regenerate resources diagram without Claude/Anthr…
glejdis Aug 7, 2026
cd129ee
docs(challenge-04): flag deploy.ps1 as Windows-only so Codespaces use…
glejdis Aug 7, 2026
d9ee4e0
docs(mcp-server README): flag deploy.ps1 as Windows-only for Codespac…
glejdis Aug 7, 2026
5a78916
docs(challenge-04): add blank line so paragraph isn't folded into the…
glejdis Aug 7, 2026
f702dae
docs(mcp-server README): add blank line so Prereq isn't folded into t…
glejdis Aug 7, 2026
5a813ca
docs(challenge-01): drop the fork step - open the code in a Codespace…
glejdis Aug 7, 2026
74df5f3
docs(challenge-01): simplify Task 4 to one deploy path (azd up) + a s…
glejdis Aug 7, 2026
5c951c9
feat(agents): publish all three CLM specialists to Foundry portal
glejdis Aug 7, 2026
20770b1
docs: publish-all-agents callouts across challenges 2-5
glejdis Aug 7, 2026
dea1c3b
docs: publish-all-agents callouts across challenges 2-5
glejdis Aug 7, 2026
4879998
docs: publish-all-agents callouts across challenges 2-5
glejdis Aug 7, 2026
d2997a1
docs: publish-all-agents callouts across challenges 2-5
glejdis Aug 7, 2026
f67b1d8
docs(challenge-01): align Task 4 with platform pre-provisioning
glejdis Aug 7, 2026
7cce6ab
docs: remove Go Further sections from challenges
glejdis Aug 7, 2026
ce9ea74
docs: remove Go Further sections from challenges
glejdis Aug 7, 2026
8295807
docs: remove Go Further sections from challenges
glejdis Aug 7, 2026
a45c4f7
docs: remove Go Further sections from challenges
glejdis Aug 7, 2026
e1099e2
docs: remove Go Further sections from challenges
glejdis Aug 7, 2026
ae494f8
docs: remove Go Further sections from challenges
glejdis Aug 7, 2026
a52ae19
docs: remove Go Further sections from challenges
glejdis Aug 7, 2026
5516c3e
docs: relabel remaining Go Further references to optional
glejdis Aug 7, 2026
35ea235
docs: relabel remaining Go Further references to optional
glejdis Aug 7, 2026
ec51c63
docs: relabel remaining Go Further references to optional
glejdis Aug 7, 2026
3f6528b
docs: relabel remaining Go Further references to optional
glejdis Aug 7, 2026
4f1dc35
docs: relabel remaining Go Further references to optional
glejdis Aug 7, 2026
a0c6596
Challenge 5: modernize Teams publish flow + fix proactive-alert gaps
glejdis Aug 7, 2026
e62821e
docs(challenge-01): sync CLM challenge-01 simplification from source …
glejdis Aug 7, 2026
dc6977e
docs(challenge-04): fix Part B for current Foundry UI + add walkthrou…
glejdis Aug 7, 2026
0dd3711
docs(challenge-05): document mandatory publish fields + add publish-d…
glejdis Aug 7, 2026
368cd7a
Challenge 6: fix empty red-team scorecard (await Chat-Protocol callback)
glejdis Aug 7, 2026
ce26618
docs(challenge-05): add Publish options (scope) screenshot to Part A
glejdis Aug 7, 2026
67fc686
Challenge 6: cleaner red-team output (single scorecard file, no raw-o…
glejdis Aug 7, 2026
f154b75
Challenge 5: add real publish-flow screenshots (menu, details form, s…
glejdis Aug 7, 2026
496567e
Challenge 5: correct Task 3 to match current portal (Next: Publish op…
glejdis Aug 7, 2026
b590716
Challenge 5: move post-publish dropdown image to Task 4 (Open in Team…
glejdis Aug 7, 2026
2487e36
Challenge 6 Task 4: name each agent (code intake-drafting vs portal c…
glejdis Aug 7, 2026
0466e33
Challenge 4: wire in real screenshots for Tasks 1, 2 and 4 (replace S…
glejdis Aug 7, 2026
a4338b2
Challenge 6 Task 4: add portal Guardrails how-to + PII data-type pick…
glejdis Aug 7, 2026
e5a6e6c
Challenge 6: print per-category red-team scorecard matrix
glejdis Aug 10, 2026
b791d09
Challenge 6/01: drop stale clm-corpus screenshot slot to match source…
glejdis Aug 10, 2026
6fbf75d
Challenge 6: make red-team scorecard export self-healing (fix IsADire…
glejdis Aug 10, 2026
ee60f44
Challenge 6: normalize red_team.py to LF (undo accidental CRLF)
glejdis Aug 10, 2026
6afcf8e
Challenge 6: treat content-filter block as a held guardrail in safety…
glejdis Aug 10, 2026
e7cbd49
Challenge 6 Task 4: clarify guardrail Action = Block, not Annotate
glejdis Aug 10, 2026
5ea4551
Challenge 6 Task 4: explain Protected materials (both boxes, Output+B…
glejdis Aug 10, 2026
47e6ecf
Challenge 6 Task 4: document 'Select agents and models' wizard step +…
glejdis Aug 10, 2026
31ed586
Normalize line endings to match glejdis/microhack-aiagents main
glejdis Aug 10, 2026
8165a96
Challenge 6 Task 4: troubleshoot guardrail 'override base policy' RAI…
glejdis Aug 10, 2026
d164f17
Sync challenge docs + Challenge 6 screenshots from glejdis/microhack-…
glejdis Aug 10, 2026
79cca30
Challenge 6 Task 5: explain CI gate purpose + require enabling the fo…
glejdis Aug 10, 2026
d82cdbe
Challenge 6 Task 5: CI secrets how-to + AADSTS70025 troubleshooting (…
glejdis Aug 10, 2026
25bde12
Challenge 6 Task 5: azd env get-values one-shot + OIDC setup in main …
glejdis Aug 10, 2026
fbd7c21
Challenge 6 Task 5: document azd pipeline config prompts with screens…
glejdis Aug 10, 2026
5a13592
Challenge 6 Task 5: document azd federated-subjects/commit-push + AAD…
glejdis Aug 10, 2026
a1f248d
Challenge 6 Task 5: restructure into required no-op path + optional O…
glejdis Aug 10, 2026
fa7c103
Challenge 6 Task 5: add fork-wide 'enable Actions' step before enabli…
glejdis Aug 10, 2026
3b9cd6b
Make fork-vs-Codespace storyline consistent across challenges
glejdis Aug 10, 2026
4c27259
Challenge 6 Task 5: tighten fork-context wording in OIDC stretch
glejdis Aug 10, 2026
ac7c977
Align Ch1/Ch6 fork story: required CI path is web-UI only, no Codespace
glejdis Aug 10, 2026
0d8b674
Challenge 6 Task 5: remove fragile OIDC stretch, keep 100%-reliable g…
glejdis Aug 11, 2026
7727a42
ci-eval: default to green no-op, require explicit RUN_LIVE_EVAL opt-i…
glejdis Aug 11, 2026
eb47b14
docs(challenge-06): add real ci-eval green no-op screenshot to Task 5
glejdis Aug 11, 2026
81881f8
docs(challenge-06): explain what the CI no-op green check accomplishes
glejdis Aug 11, 2026
2a97e8c
Challenge 1: realign solution-01 walkthrough with the challenge (sync…
glejdis Aug 11, 2026
80d5f2f
docs(challenge-01): add Foundry project portal + project-switcher scr…
glejdis Aug 11, 2026
d02cfb0
docs(challenge-01): space out Task 4 screenshots into a numbered visu…
glejdis Aug 11, 2026
a1b2d60
mirror: sync CLM lab content from glejdis/microhack-aiagents@bd4dac7
glejdis Aug 11, 2026
91f941b
mirror: sync CLM lab content from glejdis/microhack-aiagents@81b1fe1
glejdis Aug 11, 2026
5ccef25
mirror: sync CLM lab content from glejdis/microhack-aiagents@6273751
glejdis Aug 11, 2026
73cff9c
mirror: sync CLM lab content from glejdis/microhack-aiagents@52824e0
glejdis Aug 11, 2026
9b5e926
mirror: sync CLM lab content from glejdis/microhack-aiagents@e713d6e
glejdis Aug 11, 2026
b74ecc1
Challenge 2 Task 5: clarify optional content-safety preview + link to…
glejdis Aug 11, 2026
e6037e4
mirror: sync CLM lab content from glejdis/microhack-aiagents@d9876a1
glejdis Aug 11, 2026
d75a925
Challenge 2 Task 5: name the Foundry portal explicitly in the guardra…
glejdis Aug 11, 2026
f2fa912
Challenge 2: reliable template drafting + progressive per-challenge p…
glejdis Aug 11, 2026
784d789
Challenge 2 Task 6 solution: full guardrail-creation walkthrough with…
glejdis Aug 11, 2026
1b43d55
mirror: sync CLM lab content from glejdis/microhack-aiagents@d8d7ee6
glejdis Aug 11, 2026
baea931
mirror: sync CLM lab content from glejdis/microhack-aiagents@1e41ab4
glejdis Aug 11, 2026
e2f4615
Grant monitoring RBAC so Challenge 3 Monitor dashboard passes access …
glejdis Aug 11, 2026
694141d
Shrink Challenge 3 evaluation scorecard screenshot (75% -> 55%)
glejdis Aug 11, 2026
5cb2d11
Explain the Challenge 3 bake-off (what/why/how-to-read) in challenge …
glejdis Aug 11, 2026
74ebc27
docs(challenge-03): shrink the quality-gate screenshot (75% -> 55%)
glejdis Aug 11, 2026
2d11d93
mirror: sync CLM lab content from glejdis/microhack-aiagents@342e25b
glejdis Aug 11, 2026
6131b84
mirror: sync CLM lab content from glejdis/microhack-aiagents@2495cf1
glejdis Aug 11, 2026
35916ad
mirror: sync CLM lab content from glejdis/microhack-aiagents@8e92667
glejdis Aug 11, 2026
d30ea45
docs(ch03): portal Create-evaluation run flow + rubric-creation images
glejdis Aug 11, 2026
7f41941
docs(ch03): add evaluation-results screenshots (completed run + per-t…
glejdis Aug 11, 2026
73fc625
docs(ch04): add Part A MCP-server deploy screenshot + explanation
glejdis Aug 11, 2026
658536f
docs(ch06): explain the "why" for Task 1 & Task 2 red-team scans
glejdis Aug 11, 2026
23a2295
mirror: sync CLM lab content from glejdis/microhack-aiagents@2f6c415
glejdis Aug 11, 2026
2009269
docs(ch05): "Set up your own bot" flow + fix devtunnel: command not f…
glejdis Aug 11, 2026
7f313b0
docs(challenge-05): sync Task 6 solution with main (app-reg steps + d…
glejdis Aug 11, 2026
d811956
feat(challenge-05): add Azure auto-deploy option for the capture bot
glejdis Aug 11, 2026
416c801
docs(challenge-05): mirror challenge Task 6 Option A/B structure in s…
glejdis Aug 11, 2026
ca2d24a
docs(challenge-05): add deploy.sh output screenshot to solution step …
glejdis Aug 11, 2026
88f91e4
Remove optional Proactive Alerts extension from Challenge 5
glejdis Aug 11, 2026
c5bc313
Sync Challenge 6 with source main 6987c38
glejdis Aug 11, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Keep the MCP server image build context minimal — only requirements.txt and
# src/ are needed to run the server. Everything else (docs, images, walkthrough,
# lab automation, screenshots) is excluded.
*
!requirements.txt
!src

# Prune noise re-included with src/
**/__pycache__
**/*.pyc
**/*.pyo
**/.pytest_cache
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# ==========================================================================
# Foundry CLM Microhack — environment template
# Copy to `.env` (never commit `.env`). Challenge 1's deploy script autofills
# Copy to `.env` (never commit `.env`). For a MicroHack event, paste the values
# from your lab dashboard (Challenge 1, Task 4); self-hosting `azd up` autofills
# most of these for you. Fill the Bot values in Challenge 5.
# ==========================================================================

Expand All @@ -9,20 +10,23 @@
# Example: https://<account>.services.ai.azure.com/api/projects/<project>
AZURE_AI_PROJECT_ENDPOINT=

# --- Model deployments (multi-model fleet) --------------------------------
# Deployment names you created in Challenge 1. Intake & Drafting runs on Claude
# (GA in Microsoft Foundry); Clause & Risk runs on GPT-5.6 Sol; the orchestrator
# + lightweight agent run on GPT. If you skipped Claude (DEPLOY_CLAUDE_MODEL=false
# — no quota/marketplace offer), set MODEL_DRAFTING to gpt-5.4 instead.
# --- Model deployments (multi-model GPT fleet) ----------------------------
# Deployment names you created in Challenge 1. The orchestrator AND the Intake &
# Drafting agent run on gpt-5.4 (the highest-quota flagship in Foundry); Clause &
# Risk runs on GPT-5.6 Sol; the lightweight renewal agent runs on gpt-5.4-nano.
MODEL_ORCHESTRATOR=gpt-5.4
MODEL_DRAFTING=claude-opus-4-8
MODEL_DRAFTING=gpt-5.4
MODEL_CLAUSE_RISK=gpt-5.6-sol
MODEL_RENEWAL=gpt-5-mini
MODEL_RENEWAL=gpt-5.4-nano

# --- Azure AI Search (Foundry IQ knowledge base) --------------------------
AZURE_SEARCH_ENDPOINT=
AZURE_SEARCH_INDEX=clm-corpus
AZURE_SEARCH_CONNECTION_NAME=clm-search
FOUNDRY_IQ_KNOWLEDGE_SOURCE=clm-corpus-ks
FOUNDRY_IQ_KNOWLEDGE_BASE=clm-contracts-kb
FOUNDRY_IQ_CONNECTION_NAME=clm-knowledge-mcp
FOUNDRY_IQ_API_VERSION=2026-05-01-preview

# --- Web grounding (Grounding with Bing Search) — OPTIONAL --------------
# Opt-in. Attaches a web-search tool to the Clause & Risk agent (Ch4) for
Expand Down Expand Up @@ -66,7 +70,7 @@ AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED=true
# with your AAD login (default). Set them to point the judge at a dedicated
# Azure OpenAI deployment (e.g. a cheaper/faster model) or to use an API key.
# AZURE_OPENAI_ENDPOINT=https://<your-aoai>.openai.azure.com/
# AZURE_OPENAI_DEPLOYMENT=gpt-5-mini
# AZURE_OPENAI_DEPLOYMENT=gpt-5.4-nano
# AZURE_OPENAI_API_VERSION=2024-10-21
# AZURE_OPENAI_API_KEY=
# Evaluator batch concurrency. Lower it (or pass --workers) if a shared judge
Expand All @@ -75,11 +79,3 @@ AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED=true

# --- Azure SQL (contract status / renewal dates function tool) ------------
AZURE_SQL_CONNECTION_STRING=

# --- Publish + proactive Teams alerts (Challenge 5) -----------------------
MICROSOFT_APP_ID=
MICROSOFT_APP_PASSWORD=
MICROSOFT_APP_TENANT_ID=
# Captured from an inbound Teams activity; used to push proactive alerts.
TEAMS_SERVICE_URL=
TEAMS_CONVERSATION_ID=
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,8 @@ param(
```

- `deploymentType`: `"resourcegroup"` | `"resourcegroup-with-subscriptionowner"` | `"subscription"`
- `groups`: `["M365-E5-Users"]` for this hack — the CLM scenario needs M365 E5 (Teams publish, SharePoint corpus, proactive alerts). Use `[]` for Azure-only, or `["GHCPUsers"]` for a GitHub Copilot seat.
- `preferredLocation`: comma-separated regions, priority order — swedencentral first for gpt-5.4 + Claude Opus 4.8 availability
- `groups`: `["M365-E5-Users"]` for this hack — the CLM scenario needs M365 E5 (Teams publish, SharePoint corpus). Use `[]` for Azure-only, or `["GHCPUsers"]` for a GitHub Copilot seat.
- `preferredLocation`: comma-separated regions, priority order — swedencentral first for gpt-5.4 availability
- `estimatedDailyCostsUsd`: per-user per-day cost for the lifecycle wizard (Foundry models + AI Search + App Insights)

## Returning Credentials to Users
Expand Down
Original file line number Diff line number Diff line change
@@ -1,62 +1,64 @@
name: ci-eval

# Continuous evaluation gate (Challenge 3 quality + Challenge 6 safety).
# Runs on demand and nightly. Requires Azure secrets to be configured, so it is
# guarded: if AZURE_CLIENT_ID is not set, the job no-ops instead of failing.
on:
workflow_dispatch:
schedule:
- cron: "0 6 * * 1-5" # weekdays 06:00 UTC

permissions:
id-token: write # OIDC federated login to Azure
contents: read

jobs:
eval-gate:
runs-on: ubuntu-latest
env:
AZURE_AI_PROJECT_ENDPOINT: ${{ secrets.AZURE_AI_PROJECT_ENDPOINT }}
MODEL_ORCHESTRATOR: ${{ vars.MODEL_ORCHESTRATOR || 'gpt-5.4' }}
MODEL_DRAFTING: ${{ vars.MODEL_DRAFTING || 'claude-opus-4-8' }}
MODEL_CLAUSE_RISK: ${{ vars.MODEL_CLAUSE_RISK || 'gpt-5.6-sol' }}
MODEL_RENEWAL: ${{ vars.MODEL_RENEWAL || 'gpt-5-mini' }}
AZURE_SEARCH_INDEX: ${{ vars.AZURE_SEARCH_INDEX || 'clm-corpus' }}
PYTHONPATH: src
steps:
- uses: actions/checkout@v4

- name: Skip if Azure isn't configured
id: guard
run: |
if [ -z "${{ secrets.AZURE_CLIENT_ID }}" ]; then
echo "configured=false" >> "$GITHUB_OUTPUT"
echo "::notice::Azure secrets not set — skipping the eval gate. Configure AZURE_CLIENT_ID / AZURE_TENANT_ID / AZURE_SUBSCRIPTION_ID and AZURE_AI_PROJECT_ENDPOINT to enable."
else
echo "configured=true" >> "$GITHUB_OUTPUT"
fi

- uses: actions/setup-python@v5
if: steps.guard.outputs.configured == 'true'
with:
python-version: "3.11"

- name: Install dependencies
if: steps.guard.outputs.configured == 'true'
run: pip install -r requirements.txt "azure-ai-evaluation[redteam]"

- name: Azure login (OIDC)
if: steps.guard.outputs.configured == 'true'
uses: azure/login@v2
with:
client-id: ${{ secrets.AZURE_CLIENT_ID }}
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

- name: Quality gate (Challenge 3)
if: steps.guard.outputs.configured == 'true'
run: python src/evaluators.py --gate 4.0

- name: Safety gate (Challenge 6)
if: steps.guard.outputs.configured == 'true'
run: python src/safety_eval.py --gate 0.1 --safety-evals
name: ci-eval

# Continuous evaluation gate (Challenge 3 quality + Challenge 6 safety).
# Runs on demand and nightly. By default it runs in NO-OP mode (a green check):
# live Azure evaluation is out of scope for the lab, so the gate never attempts
# an Azure login. To run the gates for real, set repo variable RUN_LIVE_EVAL=true
# and add the Azure secrets (AZURE_CLIENT_ID / TENANT_ID / SUBSCRIPTION_ID / ENDPOINT).
on:
workflow_dispatch:
schedule:
- cron: "0 6 * * 1-5" # weekdays 06:00 UTC

permissions:
id-token: write # OIDC federated login to Azure
contents: read

jobs:
eval-gate:
runs-on: ubuntu-latest
env:
AZURE_AI_PROJECT_ENDPOINT: ${{ secrets.AZURE_AI_PROJECT_ENDPOINT }}
MODEL_ORCHESTRATOR: ${{ vars.MODEL_ORCHESTRATOR || 'gpt-5.4' }}
MODEL_DRAFTING: ${{ vars.MODEL_DRAFTING || 'gpt-5.4' }}
MODEL_CLAUSE_RISK: ${{ vars.MODEL_CLAUSE_RISK || 'gpt-5.6-sol' }}
MODEL_RENEWAL: ${{ vars.MODEL_RENEWAL || 'gpt-5.4-nano' }}
AZURE_SEARCH_INDEX: ${{ vars.AZURE_SEARCH_INDEX || 'clm-corpus' }}
PYTHONPATH: src
steps:
- uses: actions/checkout@v4

- name: Decide whether to run the live gate
id: guard
run: |
if [ "${{ vars.RUN_LIVE_EVAL }}" = "true" ] && [ -n "${{ secrets.AZURE_CLIENT_ID }}" ]; then
echo "configured=true" >> "$GITHUB_OUTPUT"
else
echo "configured=false" >> "$GITHUB_OUTPUT"
echo "::notice::Eval gate running in no-op mode (green check). Live Azure evaluation is out of scope for this lab; set repo variable RUN_LIVE_EVAL=true (with Azure secrets) to run the gates for real."
fi

- uses: actions/setup-python@v5
if: steps.guard.outputs.configured == 'true'
with:
python-version: "3.11"

- name: Install dependencies
if: steps.guard.outputs.configured == 'true'
run: pip install -r requirements.txt "azure-ai-evaluation[redteam]"

- name: Azure login (OIDC)
if: steps.guard.outputs.configured == 'true'
uses: azure/login@v2
with:
client-id: ${{ secrets.AZURE_CLIENT_ID }}
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

- name: Quality gate (Challenge 3)
if: steps.guard.outputs.configured == 'true'
run: python src/evaluators.py --gate 3.0

- name: Safety gate (Challenge 6)
if: steps.guard.outputs.configured == 'true'
run: python src/safety_eval.py --gate 0.1 --safety-evals
Original file line number Diff line number Diff line change
@@ -1,44 +1,44 @@
name: links

# Markdown link checker (lychee). Verifies that every relative link and image
# path in the docs resolves on disk and that external links are reachable.
# Relative-link checks are deterministic (they catch renamed/moved files); the
# external check also runs weekly to catch link rot without being noisy on PRs.
on:
push:
branches: ["**"]
paths:
- "**/*.md"
- ".github/workflows/links.yml"
pull_request:
paths:
- "**/*.md"
- ".github/workflows/links.yml"
schedule:
- cron: "0 7 * * 1" # Mondays 07:00 UTC — surface external link rot
workflow_dispatch:

permissions:
contents: read

jobs:
link-check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Check Markdown links
uses: lycheeverse/lychee-action@v2
with:
# Links inside fenced code blocks and mail links are skipped by
# default. 403/429 are accepted because some Microsoft pages
# bot-block or rate-limit crawlers.
args: >-
--no-progress
--accept 200,206,403,429
--max-retries 3
--retry-wait-time 5
"**/*.md"
fail: true
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
name: links
# Markdown link checker (lychee). Verifies that every relative link and image
# path in the docs resolves on disk and that external links are reachable.
# Relative-link checks are deterministic (they catch renamed/moved files); the
# external check also runs weekly to catch link rot without being noisy on PRs.
on:
push:
branches: ["**"]
paths:
- "**/*.md"
- ".github/workflows/links.yml"
pull_request:
paths:
- "**/*.md"
- ".github/workflows/links.yml"
schedule:
- cron: "0 7 * * 1" # Mondays 07:00 UTC — surface external link rot
workflow_dispatch:
permissions:
contents: read
jobs:
link-check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Check Markdown links
uses: lycheeverse/lychee-action@v2
with:
# Links inside fenced code blocks and mail links are skipped by
# default. 403/429 are accepted because some Microsoft pages
# bot-block or rate-limit crawlers.
args: >-
--no-progress
--accept 200,206,403,429
--max-retries 3
--retry-wait-time 5
"**/*.md"
fail: true
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Original file line number Diff line number Diff line change
@@ -1,47 +1,47 @@
name: validate

on:
push:
branches: ["**"]
pull_request:

jobs:
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: "3.11"

- name: Byte-compile all challenge + shared code
run: python -m compileall -q src labautomation

- name: Validate JSON + evaluation dataset
run: |
python - <<'PY'
import json, pathlib, sys
root = pathlib.Path(".")
bad = 0
for p in root.rglob("*.json"):
if "node_modules" in p.parts:
continue
try:
json.loads(p.read_text(encoding="utf-8"))
except Exception as e:
print(f"BAD JSON {p}: {e}"); bad += 1
ds = root / "src" / "data" / "evaluation" / "evaluation_dataset.jsonl"
rows = [l for l in ds.read_text(encoding="utf-8").splitlines() if l.strip()]
for i, line in enumerate(rows, 1):
try:
json.loads(line)
except Exception as e:
print(f"BAD JSONL line {i}: {e}"); bad += 1
print(f"validated {len(rows)} eval rows")
sys.exit(1 if bad else 0)
PY

# Go Further: run the Challenge 3 quality gate on a schedule/PR once Azure
# secrets (AZURE_AI_PROJECT_ENDPOINT, etc.) are configured as repo secrets:
# python src/evaluators.py --gate 4.0
name: validate
on:
push:
branches: ["**"]
pull_request:
jobs:
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Byte-compile all challenge + shared code
run: python -m compileall -q src labautomation
- name: Validate JSON + evaluation dataset
run: |
python - <<'PY'
import json, pathlib, sys
root = pathlib.Path(".")
bad = 0
for p in root.rglob("*.json"):
if "node_modules" in p.parts:
continue
try:
json.loads(p.read_text(encoding="utf-8"))
except Exception as e:
print(f"BAD JSON {p}: {e}"); bad += 1
ds = root / "src" / "data" / "evaluation" / "evaluation_dataset.jsonl"
rows = [l for l in ds.read_text(encoding="utf-8").splitlines() if l.strip()]
for i, line in enumerate(rows, 1):
try:
json.loads(line)
except Exception as e:
print(f"BAD JSONL line {i}: {e}"); bad += 1
print(f"validated {len(rows)} eval rows")
sys.exit(1 if bad else 0)
PY
# Optional: run the Challenge 3 quality gate on a schedule/PR once Azure
# secrets (AZURE_AI_PROJECT_ENDPOINT, etc.) are configured as repo secrets:
# python src/evaluators.py --gate 4.0
Loading
Loading