diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index fe4fb4ac..db500e78 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -23,7 +23,8 @@ say which and how you proved the real crossing. --> - [ ] **`ruff check` + `ruff format --check`** clean - [ ] **`pyright`** clean (0 errors) - [ ] **Full bundle validation** PASS — `scripts/validate-full.sh` → `validation_mode: full` - (the lone mode-advertising ERROR is a documented FALSE POSITIVE — see AGENTS.md — do NOT "fix" it) + (paste `build_check`, `quality_classification.quality_level`, and the selected recipe version; + the final report must agree with machine findings, with no ERROR waivers) ## Real evidence on seams (not mock-only) diff --git a/AGENTS.md b/AGENTS.md index 3e43801f..91737647 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,30 +2,18 @@ Guidance for AI agents and developers working in **this** bundle repository. -## Known validator false positive — do NOT "fix" it +## Mode-reference checks: fix the checker, never waive an ERROR -`validate-bundle-repo` (v3.6.0) reports a mode-advertising **ERROR**: +Use Foundation `validate-bundle-repo` v3.16.1 or later. Earlier validators can +mistake glob/template storage paths ending in `*/context-intelligence` or +`{id}/context-intelligence` for an invocation of the internal mode. The fix +belongs in Foundation's mode-reference matcher, not this bundle's storage +paths or mode visibility. -> `unadvertised_but_referenced`: mode `context-intelligence` (`modes/context-intelligence.md`, -> `advertised: false`) is referenced by name in `context/safe-extraction-patterns.md` -> and `context/agents/session-storage-knowledge.md`. - -**This is a FALSE POSITIVE. Do not act on it.** The flagged occurrences are **not** mode -invocations — they are: - -- **disk paths** — `~/.amplifier/projects/{slug}/sessions/{id}/context-intelligence/` - (the CI storage subdirectory; the `/` before the name is a path separator, not a slash-command), -- **`@mention` prefixes** — `@context-intelligence:context/...`, and -- **skill names** — `context-intelligence-graph-query`, `context-intelligence-session-navigation`. - -The bundle, its on-disk storage subdirectory, its skills, **and** the internal design mode all -share the name `context-intelligence`. The validator's `/` + `name=""` regex cannot -disambiguate them. The **full-mode** validator (see below) re-reads the source files and itself -**confirms this as a false positive — overall verdict PASS**. - -**Therefore:** leave `modes/context-intelligence.md` at `advertised: false` (the mode is correctly -internal), and do **not** remove the path/skill references. The only proper fix, if any, is an -upstream tightening of the validator regex — never a change to this repo. +Keep `modes/context-intelligence.md` at `advertised: false`. Do not delete valid +path references, advertise the internal mode, or treat the resulting ERROR as +a PASS. Select the corrected recipe with `CI_VALIDATE_RECIPE` and rerun the +full validator. Real mode-command references remain errors. ## Running the bundle validator in FULL mode @@ -39,15 +27,40 @@ scripts/validate-full.sh # validates this repo scripts/validate-full.sh # or another bundle repo ``` -It builds a fresh `uv` venv containing the pinned public CLI/Foundation, a prebuilt Core wheel, -`hatchling`, `pyyaml`, and `pip`, then runs **that venv's CLI**. PATH alone is insufficient: -the CLI supplies its own interpreter to recipe shell steps. The venv is removed on exit. -Set `CI_VALIDATE_RECIPE` to an explicit recipe path if multiple Foundation caches exist; -`CI_VALIDATE_VENV`, if supplied, must be a new directory. No validator findings are suppressed. - -Full mode includes the actual `pip wheel` build check. Record each run's mode, build result, -and findings; a successful recipe exit is not itself a validation PASS. Review the documented -mode-name false positive above without suppressing other findings. +It creates a private throwaway `uv` venv with `pip`, `hatchling`, `pyyaml`, a +prebuilt public Core 1.6.1 wheel, and the pinned public `amplifier-app-cli`. +Foundation is overridden in that venv through `uv pip --overrides` to +`f13d08168e14b5bc4720fbb06c40936eb1a7a7d1`; it is not supplied as a second +direct requirement. It then invokes that venv's `amplifier` executable explicitly. +PATH alone is insufficient because the CLI supplies its own interpreter to recipe +shell steps. It preserves the caller's Amplifier settings identity, including +`AMPLIFIER_HOME` when set. + +The default location is a unique, removed-on-exit directory under `TMPDIR` (or +`/tmp`), outside the validation target. Keep `TMPDIR` and any explicit +`CI_VALIDATE_VENV` outside that target: installed dependency skills would otherwise +be scanned as repository source. Set `CI_VALIDATE_VENV` only to a **new** path; an +existing path is refused rather than modified, and a newly claimed explicit path is +also removed on exit. Recipe selection is explicit: one cached Foundation validator +is selected automatically; zero or multiple matches stop before target normalization, +environment creation, or installation. Set `CI_VALIDATE_RECIPE` to a readable recipe +file to choose deliberately, including outside the default `~/.amplifier/cache/` +location. The selected path is printed; this choice does not update settings or caches. + +The wrapper sets `enhance_diagrams: "false"`: diagram validation and deterministic +generation still run, but optional LLM label rewriting does not. Regenerate and +commit stale `bundle.dot` / `bundle.png`; do not suppress the freshness finding. + +The wrapper is a launch/dependency helper, not the full-validator verdict gate. It propagates the +`amplifier tool invoke` exit status unchanged; process exit `0` does **not** mean validation PASS. +User/CI must inspect `env_check.validation_mode`, `build_check.build_tested`, +`build_check.build_success`, and `quality_classification.quality_level` in the +recipe results, plus `final_report`. Full PASS requires full mode, +a successful tested build, and no ERROR findings; a report cannot override +machine findings. The recipe has no structured `overall_verdict` field. +Result parsing remains outside this launch helper. A stale +diagram or validator ERROR must be resolved and the full check rerun; neither +is waived by a successful wrapper exit. ## Testing & what "done" looks like @@ -57,7 +70,7 @@ Run these before calling anything done: uv run pytest # in modules/tool-context-intelligence-query (module suite) uv run pytest # in the repo root (tests/, top-level suite) uv run ruff check . && uv run ruff format --check . && uv run pyright -scripts/validate-full.sh # → validation_mode: full, overall PASS +scripts/validate-full.sh # then inspect env_check, build_check, quality_classification, and final_report ``` **Green unit tests are the FLOOR, not proof of done.** This bundle wires **skills, modes, diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index af785d0c..81d64c0c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,7 +4,7 @@ Thanks for contributing to **amplifier-bundle-context-intelligence**. This repo a few specific conventions — read them before you change the furniture. **Read [`AGENTS.md`](AGENTS.md) first.** It is the authoritative, always-loaded -guidance: the known validator false positive, the full-validation command, and the +guidance: validator version requirements, the full-validation command, and the **seam-awareness** rules that govern how changes to tool/skill/config wiring must be tested. Everything below is the short version. @@ -62,14 +62,24 @@ Before opening a PR that touches bundle structure, run the repo's **full** valid scripts/validate-full.sh ``` -It runs a pinned public CLI from a fresh `uv` venv with Foundation, a prebuilt -Core wheel, `hatchling`, `pyyaml`, and `pip`, so the validator runs at -`validation_mode: full` without compiling Core. The venv is removed on exit. -If multiple Foundation recipes are cached, select one with `CI_VALIDATE_RECIPE`. -The lone -mode-advertising **ERROR** it reports is a **documented FALSE POSITIVE** (a name -collision — see `AGENTS.md`); **do not "fix" it** by advertising the internal mode or -deleting path/skill references. +It runs a pinned public CLI from a fresh `uv` venv with a prebuilt Core 1.6.1 +wheel, `hatchling`, `pyyaml`, and `pip`, so the validator runs at +`validation_mode: full` without compiling Core. The CLI's normal Foundation +dependency is overridden in that private venv to +`f13d08168e14b5bc4720fbb06c40936eb1a7a7d1`; it is not added as a conflicting +direct requirement. The venv is removed on exit, including a newly supplied +`CI_VALIDATE_VENV` path. If zero or multiple Foundation recipes are cached, +selection fails before installation; select a readable one explicitly with +`CI_VALIDATE_RECIPE`. The wrapper disables optional LLM diagram-label +enhancement, not diagram checks. + +Use Foundation's validator v3.16.1 or later for correct mode/path detection +(see `AGENTS.md`). Inspect `env_check.validation_mode`, +`build_check.build_tested`, `build_check.build_success`, and +`quality_classification.quality_level`; `final_report` must agree +with those machine results. Require full mode, a successful tested build, and +no ERROR findings. Process exit `0` alone is not PASS. No ERROR is waived; keep +the internal mode unadvertised and do not delete valid path references. If your change altered bundle structure, regenerate the diagram and commit it (the validator flags `BUNDLE_DOT_STALE` otherwise): diff --git a/bundle.dot b/bundle.dot index 64772875..05a81550 100644 --- a/bundle.dot +++ b/bundle.dot @@ -1,98 +1,98 @@ -// Context Intelligence: a bundle that records what happens during Amplifier sessions and lets you query that history as a searchable graph. digraph context_intelligence { rankdir=LR fontname="Helvetica" fontsize=12 - label="Context Intelligence v0.1.0\nRecords session activity and makes it searchable" + label="context-intelligence v0.1.0 — bundle repo" labelloc=t labeljust=c nodesep=0.6 ranksep=0.7 bgcolor="white" - source_hash="22df839a3f01c6690ee7bbe3d3ee1e24930998da3b5fd81d15e635539722bb16" + source_hash="e2f7810a4d98ce280e62440a0b6b39a6da14ae6d77e9f6fdb25035100e9e4ab1" node [fontname="Helvetica", fontsize=11, style="filled,rounded"] edge [fontname="Helvetica", fontsize=9] - root_context_intelligence [label="Context Intelligence (main entry point) v0.1.0\n0 tools · 0 agents\n~98 tok aggregate", shape=box, fillcolor="#80cbc4", style="filled,rounded,bold", penwidth=2] + root_context_intelligence [label="context-intelligence v0.1.0\n0 tools · 0 agents\n~98 tok aggregate", shape=box, fillcolor="#80cbc4", style="filled,rounded,bold", penwidth=2] subgraph cluster_behaviors { - label="Capability Packages (features you can switch on)" + label="Behaviors" style="filled" fillcolor="#f9f9f9" color="#999999" - beh_context_intelligence_analysis_behavior [label="Analyze Past Sessions\n1 tools\n~359 tok", shape=box, fillcolor="#e0f2f1", style="filled,rounded"] - beh_context_intelligence_design_behavior [label="Design New Query Tools\n1 tools\n~331 tok", shape=box, fillcolor="#e0f2f1", style="filled,rounded"] - beh_context_intelligence_logging_behavior [label="Record Session Activity\n2 tools\n~1761 tok", shape=box, fillcolor="#e0f2f1", style="filled,rounded"] - beh_context_intelligence_navigation_behavior [label="Browse & Search History\n3 tools\n~905 tok", shape=box, fillcolor="#e0f2f1", style="filled,rounded"] - beh_context_intelligence_transcript_behavior [label="Native Transcript Recall\n2 tools\n~215 tok", shape=box, fillcolor="#e0f2f1", style="filled,rounded"] - beh_context_intelligence_behavior [label="Core Bundle Wiring\n~249 tok", shape=box, fillcolor="#e0f2f1", style="filled,rounded"] + beh_context_intelligence_analysis_behavior [label="context-intelligence-analysis-behavior\n1 tools\n~359 tok", shape=box, fillcolor="#e0f2f1", style="filled,rounded"] + beh_context_intelligence_design_behavior [label="context-intelligence-design-behavior\n1 tools\n~331 tok", shape=box, fillcolor="#e0f2f1", style="filled,rounded"] + beh_context_intelligence_logging_behavior [label="context-intelligence-logging-behavior\n2 tools\n~1761 tok", shape=box, fillcolor="#e0f2f1", style="filled,rounded"] + beh_context_intelligence_navigation_behavior [label="context-intelligence-navigation-behavior\n3 tools\n~905 tok", shape=box, fillcolor="#e0f2f1", style="filled,rounded"] + beh_context_intelligence_transcript_behavior [label="context-intelligence-transcript-behavior\n2 tools\n~215 tok", shape=box, fillcolor="#e0f2f1", style="filled,rounded"] + beh_context_intelligence_behavior [label="context-intelligence-behavior\n~249 tok", shape=box, fillcolor="#e0f2f1", style="filled,rounded"] } subgraph cluster_agents { - label="Specialist Assistants (each handles one kind of question)" + label="Agents" style="filled" fillcolor="#f9f9f9" color="#999999" - agt_context_intelligence_design_facilitator [label="Design Conversation Guide\n~149 tok desc", shape=box, fillcolor="#c8e6c9", style="filled,rounded"] - agt_context_intelligence_tool_designer [label="Query Tool Builder\n~149 tok desc", shape=box, fillcolor="#c8e6c9", style="filled,rounded"] - agt_graph_analyst [label="History Graph Analyst\n~131 tok desc", shape=box, fillcolor="#c8e6c9", style="filled,rounded"] - agt_server_data_ops [label="Stored Data Caretaker\n~270 tok desc", shape=box, fillcolor="#c8e6c9", style="filled,rounded"] - agt_session_navigator [label="Local Session Browser\n~143 tok desc", shape=box, fillcolor="#c8e6c9", style="filled,rounded"] + agt_context_intelligence_design_facilitator [label="context-intelligence-design-facilitator\n~149 tok desc", shape=box, fillcolor="#c8e6c9", style="filled,rounded"] + agt_context_intelligence_tool_designer [label="context-intelligence-tool-designer\n~149 tok desc", shape=box, fillcolor="#c8e6c9", style="filled,rounded"] + agt_graph_analyst [label="graph-analyst\n~131 tok desc", shape=box, fillcolor="#c8e6c9", style="filled,rounded"] + agt_server_data_ops [label="server-data-ops\n~270 tok desc", shape=box, fillcolor="#c8e6c9", style="filled,rounded"] + agt_session_navigator [label="session-navigator\n~143 tok desc", shape=box, fillcolor="#c8e6c9", style="filled,rounded"] } subgraph cluster_modules { - label="Building Blocks (the code that does the work)" + label="Modules" style="filled" fillcolor="#f9f9f9" color="#999999" - mod_hook_context_intelligence [label="Activity Recorder\nhook-context-intelligence", shape=box, fillcolor="#bbdefb", style="filled,rounded"] - mod_hook_server_data_ops_lockdown [label="Data Safety Guard\nhook-server-data-ops-lockdown", shape=box, fillcolor="#bbdefb", style="filled,rounded"] - mod_tool_context_intelligence_query [label="History Search\ntool-context-intelligence-query", shape=box, fillcolor="#bbdefb", style="filled,rounded"] - mod_tool_context_intelligence_set_filters [label="Live Ingestion Filter Updates\ntool-context-intelligence-set-filters", shape=box, fillcolor="#bbdefb", style="filled,rounded"] - mod_tool_context_intelligence_transcript [label="Native Transcript Retrieval\ntool-context-intelligence-transcript", shape=box, fillcolor="#bbdefb", style="filled,rounded"] - mod_tool_context_intelligence_upload [label="History Upload\ntool-context-intelligence-upload", shape=box, fillcolor="#bbdefb", style="filled,rounded"] - mod_tool_server_data_ops [label="Stored Data Management\ntool-server-data-ops", shape=box, fillcolor="#bbdefb", style="filled,rounded"] + mod_hook_context_intelligence [label="hook-context-intelligence", shape=box, fillcolor="#bbdefb", style="filled,rounded"] + mod_hook_server_data_ops_lockdown [label="hook-server-data-ops-lockdown", shape=box, fillcolor="#bbdefb", style="filled,rounded"] + mod_tool_context_intelligence_query [label="tool-context-intelligence-query", shape=box, fillcolor="#bbdefb", style="filled,rounded"] + mod_tool_context_intelligence_recover [label="tool-context-intelligence-recover", shape=box, fillcolor="#bbdefb", style="filled,rounded"] + mod_tool_context_intelligence_set_filters [label="tool-context-intelligence-set-filters", shape=box, fillcolor="#bbdefb", style="filled,rounded"] + mod_tool_context_intelligence_transcript [label="tool-context-intelligence-transcript", shape=box, fillcolor="#bbdefb", style="filled,rounded"] + mod_tool_context_intelligence_upload [label="tool-context-intelligence-upload", shape=box, fillcolor="#bbdefb", style="filled,rounded"] + mod_tool_server_data_ops [label="tool-server-data-ops", shape=box, fillcolor="#bbdefb", style="filled,rounded"] } subgraph cluster_legend { - label="Legend — what each colour means" + label="Legend" style="filled" fillcolor="white" color="#cccccc" fontsize=9 - leg_root [label="main entry point", shape=box, fillcolor="#80cbc4", style="filled,rounded,bold", fontsize=9] - leg_behavior [label="capability package", shape=box, fillcolor="#e0f2f1", style="filled,rounded", fontsize=9] - leg_agent [label="specialist assistant", shape=box, fillcolor="#c8e6c9", style="filled,rounded", fontsize=9] - leg_module [label="building block", shape=box, fillcolor="#bbdefb", style="filled,rounded", fontsize=9] - leg_provider [label="AI model connection", shape=box, fillcolor="#e0e0e0", style="filled,rounded", fontsize=9] - leg_context [label="reference text", shape=box, fillcolor="#e1bee7", style="filled,rounded", fontsize=9] - leg_standalone [label="ready-to-run setup", shape=box, fillcolor="#80cbc4", style="filled,rounded", fontsize=9] - leg_experiment [label="experimental", shape=box, fillcolor="#e1bee7", style="filled,rounded", fontsize=9] - leg_ext_cost [label="from another repo\n(adds hidden cost)", shape=box, fillcolor="#80cbc4", style="dashed", color="red", penwidth=2, fontsize=9] - leg_ext_muted [label="from another repo\n(no added cost)", shape=box, fillcolor="#f5f5f5", style="dashed", fontsize=9] + leg_root [label="root bundle", shape=box, fillcolor="#80cbc4", style="filled,rounded,bold", fontsize=9] + leg_behavior [label="behavior", shape=box, fillcolor="#e0f2f1", style="filled,rounded", fontsize=9] + leg_agent [label="agent", shape=box, fillcolor="#c8e6c9", style="filled,rounded", fontsize=9] + leg_module [label="module", shape=box, fillcolor="#bbdefb", style="filled,rounded", fontsize=9] + leg_provider [label="provider", shape=box, fillcolor="#e0e0e0", style="filled,rounded", fontsize=9] + leg_context [label="context", shape=box, fillcolor="#e1bee7", style="filled,rounded", fontsize=9] + leg_standalone [label="standalone", shape=box, fillcolor="#80cbc4", style="filled,rounded", fontsize=9] + leg_experiment [label="experiment", shape=box, fillcolor="#e1bee7", style="filled,rounded", fontsize=9] + leg_ext_cost [label="ext+cost", shape=box, fillcolor="#80cbc4", style="dashed", color="red", penwidth=2, fontsize=9] + leg_ext_muted [label="ext+no-cost", shape=box, fillcolor="#f5f5f5", style="dashed", fontsize=9] } disclaimer [label="Token estimates: ~4 chars/token\nSolid border = local (counted)\nDashed + red = external, hidden cost (not counted)\nDashed + muted = external, no cost\nExcludes: sub-session costs, runtime-dynamic", shape=note, fillcolor="#eceff1", style="filled", fontsize=9] - ext_githttps___github_com_microsoft_amplifier_foundation_main [label="Shared Amplifier Base\namplifier-foundation\n(external, cost)", shape=box, fillcolor="#80cbc4", style="dashed", color="red", penwidth=2] + ext_githttps___github_com_microsoft_amplifier_foundation_main [label="amplifier-foundation\n(external, cost)", shape=box, fillcolor="#80cbc4", style="dashed", color="red", penwidth=2] root_context_intelligence -> ext_githttps___github_com_microsoft_amplifier_foundation_main [style=dashed] root_context_intelligence -> beh_context_intelligence_behavior [label="composes"] - beh_context_intelligence_behavior -> beh_context_intelligence_design_behavior [label="includes", penwidth=0.8] - beh_context_intelligence_behavior -> beh_context_intelligence_logging_behavior [label="includes", penwidth=0.8] - beh_context_intelligence_design_behavior -> beh_context_intelligence_analysis_behavior [label="includes", penwidth=0.8] - beh_context_intelligence_analysis_behavior -> beh_context_intelligence_navigation_behavior [label="includes", penwidth=0.8] + beh_context_intelligence_analysis_behavior -> beh_context_intelligence_navigation_behavior [label="composes"] + beh_context_intelligence_design_behavior -> beh_context_intelligence_analysis_behavior [label="composes"] beh_context_intelligence_logging_behavior -> mod_tool_context_intelligence_set_filters [label="uses", penwidth=0.8] beh_context_intelligence_logging_behavior -> mod_hook_context_intelligence [label="uses", penwidth=0.8] beh_context_intelligence_navigation_behavior -> agt_graph_analyst [label="owns"] beh_context_intelligence_navigation_behavior -> agt_session_navigator [label="owns"] beh_context_intelligence_navigation_behavior -> agt_server_data_ops [label="owns"] beh_context_intelligence_navigation_behavior -> mod_tool_context_intelligence_transcript [label="uses", penwidth=0.8] - beh_context_intelligence_transcript_behavior -> beh_context_intelligence_logging_behavior [label="includes", penwidth=0.8] + beh_context_intelligence_transcript_behavior -> beh_context_intelligence_logging_behavior [label="composes"] beh_context_intelligence_transcript_behavior -> mod_tool_context_intelligence_transcript [label="uses", penwidth=0.8] + beh_context_intelligence_behavior -> beh_context_intelligence_design_behavior [label="composes"] + beh_context_intelligence_behavior -> beh_context_intelligence_logging_behavior [label="composes"] } \ No newline at end of file diff --git a/bundle.png b/bundle.png index 7d1e4f7c..87e9497a 100644 Binary files a/bundle.png and b/bundle.png differ diff --git a/scripts/validate-full.sh b/scripts/validate-full.sh index dcba411f..fd5a55bf 100755 --- a/scripts/validate-full.sh +++ b/scripts/validate-full.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# validate-full.sh — run `validate-bundle-repo` against THIS bundle in FULL mode. +# validate-full.sh — launch `validate-bundle-repo` with full-mode-capable private dependencies. # # WHY THIS EXISTS # -------------- @@ -13,7 +13,19 @@ # # The CLI sets AMPLIFIER_PYTHON for recipe shell steps. PATH alone cannot move # those steps into another venv: the CLI itself must run from the prepared venv. -# Use a public Core wheel, not a Rust source build, for this bundle's validation. +# This script invokes that venv's CLI explicitly, so both its fixed shebang and +# the recipe's `python3` resolve to private dependencies. It uses a public Core +# wheel, not a Rust source build, for this bundle's validation. +# +# This is a launch/dependency helper, not a verdict gate. It propagates the +# `amplifier tool invoke` exit status unchanged; a zero process exit is not a +# validation PASS. User/CI must inspect `env_check.validation_mode`, +# `build_check.build_tested`, `build_check.build_success`, and +# `quality_classification.quality_level`, plus `final_report`. Full PASS +# requires full mode, a successful tested build, no ERROR findings, and a report +# consistent with those machine results. The recipe has no structured +# `overall_verdict` field. Diagram checks and deterministic generation remain +# enabled; optional LLM label enhancement is disabled. # # USAGE # ----- @@ -21,64 +33,94 @@ # REPO_PATH defaults to this bundle's repo root. # # ENV -# CI_VALIDATE_VENV optional NEW venv directory; never overwrite an existing one -# CI_VALIDATE_RECIPE explicit recipe path if more than one Foundation is cached +# CI_VALIDATE_VENV optional NEW venv directory; never overwrite an existing one. +# The directory is removed on exit. +# CI_VALIDATE_RECIPE explicit readable recipe file; otherwise exactly one cached +# Foundation validator must exist. Ambiguity fails before +# environment creation or dependency installation. # # Requires: uv and a cached Foundation recipe (or CI_VALIDATE_RECIPE). -# A fresh venv is removed on exit. The recipe's report, including failures and -# known false positives, is returned unchanged; exit 0 alone is not a PASS. +# Keep TMPDIR and CI_VALIDATE_VENV outside the target repository so installed +# dependency skills are not scanned as source. # set -euo pipefail REPO_PATH="${1:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}" -REPO_PATH="$(cd "$REPO_PATH" && pwd)" +CLI_REF="14dc68eba05bf65b8c6dea28c3a2db93daa12d38" +FOUNDATION_REF="f13d08168e14b5bc4720fbb06c40936eb1a7a7d1" -# Locate the foundation validate-bundle-repo recipe in the Amplifier cache. -# (The bare `amplifier tool invoke` CLI does not resolve the `foundation:` recipe -# namespace, so we pass the cached recipe by absolute path.) -RECIPE="${CI_VALIDATE_RECIPE:-}" -if [[ -z "$RECIPE" ]]; then +# Select before installing: never guess between cached recipe revisions. +# The caller may choose a file explicitly without changing settings or caches. +if [[ -n "${CI_VALIDATE_RECIPE:-}" ]]; then + RECIPE="$CI_VALIDATE_RECIPE" +else shopt -s nullglob recipes=("${HOME}/.amplifier/cache/"amplifier-foundation-*/recipes/validate-bundle-repo.yaml) + shopt -u nullglob + if [[ ${#recipes[@]} -eq 0 ]]; then + echo "!! no cached Foundation validator; set CI_VALIDATE_RECIPE to its recipe file" >&2 + exit 1 + fi if [[ ${#recipes[@]} -ne 1 ]]; then - echo "!! Expected one cached validation recipe; found ${#recipes[@]}. Set CI_VALIDATE_RECIPE." >&2 + echo "!! multiple cached Foundation validators; select one with CI_VALIDATE_RECIPE" >&2 + printf ' %s\n' "${recipes[@]}" >&2 exit 1 fi RECIPE="${recipes[0]}" fi if [[ ! -f "$RECIPE" || ! -r "$RECIPE" ]]; then - echo "!! Validation recipe is not a readable file: $RECIPE" >&2 + echo "!! validation recipe is not a readable file: $RECIPE" >&2 exit 1 fi +# Normalize only after recipe selection, so cache-selection errors remain clear +# even when the target path does not exist. +REPO_PATH="$(cd "$REPO_PATH" && pwd)" + if [[ -n "${CI_VALIDATE_VENV:-}" ]]; then VENV="$CI_VALIDATE_VENV" - # mkdir refuses existing directories and symlinks before uv can touch them. - mkdir -- "$VENV" + # mkdir refuses existing paths and symlinks before uv can touch them. + if ! mkdir -- "$VENV"; then + echo "!! CI_VALIDATE_VENV already exists; refusing to modify it: $VENV" >&2 + exit 1 + fi else - VENV="$(mktemp -d "${TMPDIR:-/tmp}/ci-validate-venv.XXXXXXXX")" + VENV="$(mktemp -d "${TMPDIR:-/tmp}/ci-validate.XXXXXX")" fi + +# Every directory is newly claimed by this invocation and is removed on exit. trap 'rm -rf -- "$VENV"' EXIT export PYTHONNOUSERSITE=1 echo ">> building isolated validation runtime: $VENV" -uv venv --python 3.11 "$VENV" >/dev/null -# The public CLI declares Foundation@main; override that URL rather than +uv venv --python 3.11 --allow-existing "$VENV" >/dev/null +# The public CLI declares Foundation@main; override that dependency rather than # supplying a second, conflicting direct requirement. printf '%s\n' \ - "amplifier-foundation @ git+https://github.com/microsoft/amplifier-foundation@7ad00b359fd5c2ac3ee98436b1b3bccabe6e909d" \ + "amplifier-foundation @ git+https://github.com/microsoft/amplifier-foundation@$FOUNDATION_REF" \ > "$VENV/overrides.txt" -uv pip install --python "$VENV/bin/python" --only-binary amplifier-core \ - --overrides "$VENV/overrides.txt" --quiet \ +uv pip install --python "$VENV/bin/python" --quiet \ + --only-binary amplifier-core \ + --overrides "$VENV/overrides.txt" \ pip hatchling pyyaml "amplifier-core==1.6.1" \ - "amplifier-app-cli @ git+https://github.com/microsoft/amplifier-app-cli@14dc68eba05bf65b8c6dea28c3a2db93daa12d38" -"$VENV/bin/python" -c 'import pip, hatchling, yaml, amplifier_core, amplifier_foundation' + "amplifier-app-cli @ git+https://github.com/microsoft/amplifier-app-cli@$CLI_REF" + +if ! "$VENV/bin/python" -c 'import pip, hatchling, yaml, amplifier_core, amplifier_foundation'; then + echo "!! private validation Python is missing required imports" >&2 + exit 1 +fi +if [[ ! -x "$VENV/bin/amplifier" ]]; then + echo "!! private validation venv did not install an executable amplifier CLI" >&2 + exit 1 +fi + # JSON encoding preserves spaces, quotes, and backslashes in the target path. CONTEXT="$("$VENV/bin/python" -c 'import json,sys; print(json.dumps({"repo_path": sys.argv[1], "enhance_diagrams": "false"}))' "$REPO_PATH")" echo ">> recipe: $RECIPE" echo ">> repo: $REPO_PATH" -echo ">> running validate-bundle-repo in FULL mode ..." +echo ">> launching validate-bundle-repo with full-mode-capable private dependencies ..." +echo ">> require full mode, a successful tested build, no ERROR findings, and a consistent final report; exit 0 is not PASS" PATH="$VENV/bin:$PATH" "$VENV/bin/amplifier" tool invoke recipes operation=execute \ recipe_path="$RECIPE" \ context="$CONTEXT" diff --git a/tests/test_validate_full.py b/tests/test_validate_full.py new file mode 100644 index 00000000..f5e7a8be --- /dev/null +++ b/tests/test_validate_full.py @@ -0,0 +1,436 @@ +"""Regression tests for the isolated full-validator wrapper.""" + +from __future__ import annotations + +import json +import os +import stat +import subprocess +import sys +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).parent.parent +SCRIPT_PATH = REPO_ROOT / "scripts" / "validate-full.sh" +CLI_REF = "14dc68eba05bf65b8c6dea28c3a2db93daa12d38" +FOUNDATION_REF = "f13d08168e14b5bc4720fbb06c40936eb1a7a7d1" + + +def _write_executable(path: Path, content: str) -> None: + path.write_text(content) + path.chmod(path.stat().st_mode | stat.S_IXUSR) + + +def _create_fake_uv(bin_dir: Path) -> None: + _write_executable( + bin_dir / "uv", + """#!/usr/bin/env bash +set -euo pipefail +printf '%s\\n' "$@" >> "$FAKE_UV_ARGS" + +if [[ "$1" == "venv" ]]; then + venv="${!#}" + mkdir -p "$venv/bin" + cat > "$venv/bin/python" <<'PYTHON' +#!/usr/bin/env bash +if [[ "${1:-}" == */bin/amplifier ]]; then + printf '%s\\n' "$1" > "$FAKE_CLI_PATH" + shift + printf 'private-python=%s\\n' "$0" > "$FAKE_CLI_PYTHON" + printf '%s\\n' "$@" > "$FAKE_CLI_ARGS" + printf '%s\\n' "$PATH" > "$FAKE_CLI_ENV_PATH" + printf '%s\\n' "${AMPLIFIER_HOME-__UNSET__}" > "$FAKE_CLI_AMPLIFIER_HOME" + printf '%s\\n' "$PYTHONNOUSERSITE" > "$FAKE_CLI_PYTHONNOUSERSITE" + exit "${FAKE_PRIVATE_CLI_EXIT:-0}" +fi +if [[ "$*" == *"import pip, hatchling, yaml, amplifier_core, amplifier_foundation"* ]]; then + [[ "${FAKE_FAIL_PRIVATE_IMPORTS:-}" != "1" ]] + exit +fi +if [[ "$*" == *"json.dumps"* ]]; then + printf '%s\\n' "$2" > "$FAKE_JSON_CODE" + printf '%s\\n' "$3" > "$FAKE_JSON_INPUT" + printf '%s\\n' "$FAKE_JSON_CONTEXT" + exit +fi +exit 64 +PYTHON + chmod +x "$venv/bin/python" +elif [[ "$1" == "pip" && "${FAKE_OMIT_PRIVATE_CLI:-}" != "1" ]]; then + for ((index = 1; index <= $#; index++)); do + if [[ "${!index}" == "--overrides" ]]; then + next=$((index + 1)) + cat "${!next}" > "$FAKE_OVERRIDE_CONTENTS" + break + fi + done + python="" + for ((index = 1; index <= $#; index++)); do + if [[ "${!index}" == "--python" ]]; then + next=$((index + 1)) + python="${!next}" + break + fi + done + venv="$(dirname "$(dirname "$python")")" + cat > "$venv/bin/amplifier" <<'AMPLIFIER' +#!/usr/bin/env bash +exec "VENV_PYTHON" "$0" "$@" +AMPLIFIER + sed -i "s|VENV_PYTHON|$venv/bin/python|" "$venv/bin/amplifier" + chmod +x "$venv/bin/amplifier" +fi +""", + ) + + +def _create_recipe(home: Path, revision: str = "test") -> Path: + recipe = home / ".amplifier" / "cache" / f"amplifier-foundation-{revision}" / "recipes" + recipe.mkdir(parents=True) + recipe_path = recipe / "validate-bundle-repo.yaml" + recipe_path.write_text("name: validate-bundle-repo\n") + return recipe_path + + +def _environment( + tmp_path: Path, + *, + omit_private_cli: bool = False, + fail_private_imports: bool = False, + caller_amplifier_home: str | None = None, + private_cli_exit: int = 0, +) -> dict[str, str]: + tool_bin = tmp_path / "tool bin" + host_bin = tmp_path / "host bin" + home = tmp_path / "home" + tool_bin.mkdir() + host_bin.mkdir() + _create_fake_uv(tool_bin) + _write_executable( + host_bin / "amplifier", + """#!/usr/bin/env bash +touch "$FAKE_HOST_CLI_SENTINEL" +exit 97 +""", + ) + _create_recipe(home) + + environment = os.environ.copy() + environment.update( + { + "HOME": str(home), + "PATH": f"{tool_bin}:{host_bin}:{environment['PATH']}", + "FAKE_UV_ARGS": str(tmp_path / "uv-args"), + "FAKE_CLI_PATH": str(tmp_path / "cli-path"), + "FAKE_CLI_PYTHON": str(tmp_path / "cli-python"), + "FAKE_CLI_ARGS": str(tmp_path / "cli-args"), + "FAKE_CLI_ENV_PATH": str(tmp_path / "cli-env-path"), + "FAKE_CLI_AMPLIFIER_HOME": str(tmp_path / "cli-amplifier-home"), + "FAKE_CLI_PYTHONNOUSERSITE": str(tmp_path / "cli-pythonnousersite"), + "FAKE_HOST_CLI_SENTINEL": str(tmp_path / "host-cli-called"), + "FAKE_JSON_CODE": str(tmp_path / "json-code"), + "FAKE_JSON_INPUT": str(tmp_path / "json-input"), + "FAKE_OVERRIDE_CONTENTS": str(tmp_path / "override-contents"), + } + ) + environment.pop("AMPLIFIER_HOME", None) + environment.pop("CI_VALIDATE_RECIPE", None) + environment.pop("CI_VALIDATE_VENV", None) + if caller_amplifier_home is not None: + environment["AMPLIFIER_HOME"] = caller_amplifier_home + if omit_private_cli: + environment["FAKE_OMIT_PRIVATE_CLI"] = "1" + if fail_private_imports: + environment["FAKE_FAIL_PRIVATE_IMPORTS"] = "1" + environment["FAKE_PRIVATE_CLI_EXIT"] = str(private_cli_exit) + return environment + + +def test_launches_pinned_private_cli_and_preserves_paths_with_spaces(tmp_path: Path) -> None: + """The wrapper launches private tools without relocating caller settings identity.""" + caller_amplifier_home = str(tmp_path / "caller amplifier home") + environment = _environment(tmp_path, caller_amplifier_home=caller_amplifier_home) + repo_path = tmp_path / 'bundle "quoted" \\ path' + venv_path = tmp_path / "private venv with spaces" + repo_path.mkdir() + expected_context = {"repo_path": str(repo_path), "enhance_diagrams": "false"} + environment["FAKE_JSON_CONTEXT"] = json.dumps(expected_context) + + result = subprocess.run( + [str(SCRIPT_PATH), str(repo_path)], + env={**environment, "CI_VALIDATE_VENV": str(venv_path)}, + capture_output=True, + text=True, + check=False, + ) + + assert result.returncode == 0, result.stderr + assert Path(environment["FAKE_CLI_PATH"]).read_text().strip() == str( + venv_path / "bin" / "amplifier" + ) + assert Path(environment["FAKE_CLI_PYTHON"]).read_text().strip() == ( + f"private-python={venv_path / 'bin' / 'python'}" + ) + assert not Path(environment["FAKE_HOST_CLI_SENTINEL"]).exists() + + uv_args = Path(environment["FAKE_UV_ARGS"]).read_text().splitlines() + assert "--allow-existing" in uv_args + assert "--only-binary" in uv_args + python_targets = [ + uv_args[index + 1] for index, argument in enumerate(uv_args[:-1]) if argument == "--python" + ] + assert str(venv_path / "bin" / "python") in python_targets + assert "pip" in uv_args + assert "hatchling" in uv_args + assert "amplifier-core==1.6.1" in uv_args + # The CLI supplies Core/Foundation through its own dependency closure. + # Repeating Foundation as a direct Git requirement conflicts with its + # tool.uv.sources mapping during a real install. + assert not any(arg.startswith("amplifier-foundation @") for arg in uv_args) + assert not any(arg.startswith("amplifier-core @") for arg in uv_args) + assert FOUNDATION_REF in Path(environment["FAKE_OVERRIDE_CONTENTS"]).read_text() + assert ( + f"amplifier-app-cli @ git+https://github.com/microsoft/amplifier-app-cli@{CLI_REF}" + in uv_args + ) + + cli_args = Path(environment["FAKE_CLI_ARGS"]).read_text().splitlines() + expected_recipe = ( + Path(environment["HOME"]) + / ".amplifier/cache/amplifier-foundation-test/recipes/validate-bundle-repo.yaml" + ) + assert f"recipe_path={expected_recipe}" in cli_args + context = next(argument for argument in cli_args if argument.startswith("context=")) + assert json.loads(context.removeprefix("context=")) == expected_context + assert Path(environment["FAKE_JSON_INPUT"]).read_text().strip() == str(repo_path) + # Exercise the wrapper's actual expression too: fake CLI output alone cannot + # prove that diagram enhancement is disabled or quoted paths survive. + json_result = subprocess.run( + [sys.executable, "-c", Path(environment["FAKE_JSON_CODE"]).read_text(), str(repo_path)], + capture_output=True, + text=True, + check=True, + ) + assert json.loads(json_result.stdout) == expected_context + assert Path(environment["FAKE_CLI_ENV_PATH"]).read_text().splitlines()[0] == ( + f"{venv_path}/bin:{environment['PATH']}" + ) + assert Path(environment["FAKE_CLI_AMPLIFIER_HOME"]).read_text().strip() == caller_amplifier_home + assert Path(environment["FAKE_CLI_PYTHONNOUSERSITE"]).read_text().strip() == "1" + assert not venv_path.exists() + + +def test_propagates_private_cli_failure_status(tmp_path: Path) -> None: + """The helper reports the invoked CLI status rather than creating a validation verdict.""" + environment = _environment(tmp_path, private_cli_exit=7) + venv_path = tmp_path / "private venv" + environment["FAKE_JSON_CONTEXT"] = json.dumps({"repo_path": str(REPO_ROOT)}) + + result = subprocess.run( + [str(SCRIPT_PATH)], + env={**environment, "CI_VALIDATE_VENV": str(venv_path)}, + capture_output=True, + text=True, + check=False, + ) + + assert result.returncode == 7 + assert Path(environment["FAKE_CLI_PATH"]).read_text().strip() == str( + venv_path / "bin" / "amplifier" + ) + assert not venv_path.exists() + + +@pytest.mark.parametrize("cli_exit", [0, 7]) +def test_default_venv_stays_outside_target_and_is_cleaned(tmp_path: Path, cli_exit: int) -> None: + """Installed validator skills must never become default validation inputs.""" + environment = _environment(tmp_path, private_cli_exit=cli_exit) + repo_path = tmp_path / "target repo" + repo_path.mkdir() + temp_root = tmp_path / "private temp" + temp_root.mkdir() + environment["TMPDIR"] = str(temp_root) + environment["FAKE_JSON_CONTEXT"] = json.dumps( + {"repo_path": str(repo_path), "enhance_diagrams": "false"} + ) + + result = subprocess.run( + [str(SCRIPT_PATH), str(repo_path)], + env=environment, + capture_output=True, + text=True, + check=False, + ) + + assert result.returncode == cli_exit, result.stderr + venv = Path(environment["FAKE_CLI_PATH"]).read_text().strip() + venv_path = Path(venv).parent.parent + assert venv_path.parent == temp_root + assert venv_path.name.startswith("ci-validate.") + assert not venv_path.exists() + assert not (repo_path / ".amplifier").exists() + assert not Path(environment["FAKE_HOST_CLI_SENTINEL"]).exists() + + +def test_preserves_an_unset_amplifier_home(tmp_path: Path) -> None: + """The private tools venv must not fabricate an Amplifier settings home.""" + environment = _environment(tmp_path) + venv_path = tmp_path / "private venv" + environment["FAKE_JSON_CONTEXT"] = json.dumps({"repo_path": str(REPO_ROOT)}) + + result = subprocess.run( + [str(SCRIPT_PATH)], + env={**environment, "CI_VALIDATE_VENV": str(venv_path)}, + capture_output=True, + text=True, + check=False, + ) + + assert result.returncode == 0, result.stderr + assert Path(environment["FAKE_CLI_AMPLIFIER_HOME"]).read_text().strip() == "__UNSET__" + + +def test_refuses_an_existing_override_without_clobbering_it(tmp_path: Path) -> None: + """An explicitly supplied existing venv path is never recreated or altered.""" + environment = _environment(tmp_path) + existing_venv = tmp_path / "existing venv" + existing_venv.mkdir() + user_file = existing_venv / "user-file" + user_file.write_text("keep me") + + result = subprocess.run( + [str(SCRIPT_PATH)], + env={**environment, "CI_VALIDATE_VENV": str(existing_venv)}, + capture_output=True, + text=True, + check=False, + ) + + assert result.returncode != 0 + assert "already exists; refusing to modify it" in result.stderr + assert user_file.read_text() == "keep me" + assert not Path(environment["FAKE_UV_ARGS"]).exists() + + +def test_fails_on_missing_private_imports_without_running_host_cli(tmp_path: Path) -> None: + """Missing private dependencies must stop before an accidental host-CLI fallback.""" + environment = _environment(tmp_path, fail_private_imports=True) + venv_path = tmp_path / "private venv" + + result = subprocess.run( + [str(SCRIPT_PATH)], + env={**environment, "CI_VALIDATE_VENV": str(venv_path)}, + capture_output=True, + text=True, + check=False, + ) + + assert result.returncode != 0 + assert "private validation Python is missing required imports" in result.stderr + assert not Path(environment["FAKE_HOST_CLI_SENTINEL"]).exists() + + +def test_fails_without_private_cli_instead_of_falling_back_to_host(tmp_path: Path) -> None: + """A failed private CLI install must not run the host `amplifier` executable.""" + environment = _environment(tmp_path, omit_private_cli=True) + venv_path = tmp_path / "private venv" + + result = subprocess.run( + [str(SCRIPT_PATH)], + env={**environment, "CI_VALIDATE_VENV": str(venv_path)}, + capture_output=True, + text=True, + check=False, + ) + + assert result.returncode != 0 + assert "did not install an executable amplifier CLI" in result.stderr + assert not Path(environment["FAKE_HOST_CLI_SENTINEL"]).exists() + + +@pytest.mark.parametrize("recipe_count", [0, 2]) +def test_requires_unambiguous_recipe_before_any_install(tmp_path: Path, recipe_count: int) -> None: + environment = _environment(tmp_path) + home = Path(environment["HOME"]) + if recipe_count == 0: + ( + home / ".amplifier/cache/amplifier-foundation-test/recipes/validate-bundle-repo.yaml" + ).unlink() + else: + _create_recipe(home, "another") + repo_path = tmp_path / "target repo" + + result = subprocess.run( + [str(SCRIPT_PATH), str(repo_path)], + env=environment, + capture_output=True, + text=True, + timeout=10, + check=False, + ) + + assert result.returncode != 0 + assert "CI_VALIDATE_RECIPE" in result.stderr + assert ("no cached" if recipe_count == 0 else "multiple cached") in result.stderr + assert not repo_path.exists() + assert not Path(environment["FAKE_UV_ARGS"]).exists() + assert not Path(environment["FAKE_CLI_PATH"]).exists() + assert not Path(environment["FAKE_HOST_CLI_SENTINEL"]).exists() + + +@pytest.mark.parametrize("cached_recipe_count", [0, 2]) +def test_explicit_recipe_bypasses_cache_ambiguity(tmp_path: Path, cached_recipe_count: int) -> None: + environment = _environment(tmp_path) + home = Path(environment["HOME"]) + if cached_recipe_count == 0: + ( + home / ".amplifier/cache/amplifier-foundation-test/recipes/validate-bundle-repo.yaml" + ).unlink() + else: + _create_recipe(home, "another") + recipe = tmp_path / "chosen recipe with spaces.yaml" + recipe.write_text("name: explicit-validator\n") + environment["CI_VALIDATE_RECIPE"] = str(recipe) + environment["CI_VALIDATE_VENV"] = str(tmp_path / "private venv") + environment["FAKE_JSON_CONTEXT"] = json.dumps({"repo_path": str(REPO_ROOT)}) + + result = subprocess.run( + [str(SCRIPT_PATH)], + env=environment, + capture_output=True, + text=True, + timeout=10, + check=False, + ) + + assert result.returncode == 0, result.stderr + assert f">> recipe: {recipe}" in result.stdout + cli_args = Path(environment["FAKE_CLI_ARGS"]).read_text().splitlines() + assert f"recipe_path={recipe}" in cli_args + assert not Path(environment["FAKE_HOST_CLI_SENTINEL"]).exists() + + +@pytest.mark.parametrize("kind", ["missing", "directory"]) +def test_invalid_explicit_recipe_never_falls_back(tmp_path: Path, kind: str) -> None: + environment = _environment(tmp_path) + recipe = tmp_path / "invalid recipe.yaml" + if kind == "directory": + recipe.mkdir() + environment["CI_VALIDATE_RECIPE"] = str(recipe) + repo_path = tmp_path / "target repo" + + result = subprocess.run( + [str(SCRIPT_PATH), str(repo_path)], + env=environment, + capture_output=True, + text=True, + timeout=10, + check=False, + ) + + assert result.returncode != 0 + assert "not a readable file" in result.stderr + assert not repo_path.exists() + assert not Path(environment["FAKE_UV_ARGS"]).exists() + assert not Path(environment["FAKE_CLI_PATH"]).exists() diff --git a/tests/test_validate_full_launcher.py b/tests/test_validate_full_launcher.py index d50a3356..feb8dd89 100644 --- a/tests/test_validate_full_launcher.py +++ b/tests/test_validate_full_launcher.py @@ -93,7 +93,7 @@ def test_runs_venv_cli_and_preserves_status_and_json_paths(launcher, status): assert "--only-binary" in calls[1] assert "--overrides" in calls[1] assert ( - "amplifier-foundation@7ad00b359fd5c2ac3ee98436b1b3bccabe6e909d" + "amplifier-foundation@f13d08168e14b5bc4720fbb06c40936eb1a7a7d1" in Path(env["OVERRIDE_COPY"]).read_text() ) assert "amplifier-core==1.6.1" in calls[1] @@ -101,6 +101,7 @@ def test_runs_venv_cli_and_preserves_status_and_json_paths(launcher, status): response = json.loads(Path(env["CLI_RESULT"]).read_text()) context = json.loads(next(arg[8:] for arg in response["args"] if arg.startswith("context="))) assert context["repo_path"] == str(tmp_path / 'repo with "quotes"') + assert context["enhance_diagrams"] == "false" assert response["no_user_site"] == "1" assert not Path(calls[0][-1]).exists()