mosheabr · Oct 28, 2026 · Oct 28, 2026 · Oct 28, 2026 · Oct 28, 2026
@@ -24,3 +24,12 @@ components:
       - omniverse-usd-performance-tuning
       - physical-ai-infrastructure-setup-and-resilient-scaling
       - physical-ai-neural-reconstruction
+  - name: PVA SDK
+    description: NVIDIA Programmable Vision Accelerator (PVA) SDK skills for environment setup, host readiness checks, operator specification, generation, build and test, and debugging.
+    catalog_dirs:
+      - pva-build-test
+      - pva-debug
+      - pva-doctor
+      - pva-opgen
+      - pva-setup
+      - pva-spec
@@ -0,0 +1,153 @@
+# Skill Benchmark: pva-build-test
+
+> ✅ **Overall verdict: PASS — Recommended for publication**
+
+## Publication Recommendation
+
+Recommended for publication based on the completed evaluation evidence in this report.
+
+## Evaluation Metadata
+
+- Skill: `pva-build-test`
+- Validation status: `passed`
+- Evaluation date: 2026-10-28
+- Evaluator version: `1.6.3`
+- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-5-5`), Codex (`openai/openai/gpt-6.1-sol`)
+- Tasks: 4 evaluation tasks (3 positive, 1 negative)
+- Dataset digest: `sha256:a4f9d1d2892bbba5832f465f6dd847921574c86e3234fd9cba37cf22369034bb` (skill-evaluator-dataset-snapshot/1)
+- Attempts per task: 1
+- Environment: `k8s-sandbox`
+- Tier 2 evidence: required for publication
+- Tier 3 evidence: required for publication
+
+Each task attempt ran in its own isolated sandbox pod.
+
+## Tier 1 LLM Provenance
+
+SkillSpector provider, model, analyzer revision, and determinism evidence retained for this scan.
+
+| Skill | Provider | Adapter | Analyzer | Model | Revision | Determinism |
+| --- | --- | --- | --- | --- | --- | --- |
+| pva-build-test | nv_inference | openai | mcp_tool_poisoning | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.1` | nondeterministic / provider_defaults |
+| pva-build-test | nv_inference | openai | semantic_developer_intent | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.1` | nondeterministic / provider_defaults |
+| pva-build-test | nv_inference | openai | semantic_quality_policy | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.1` | nondeterministic / provider_defaults |
+| pva-build-test | nv_inference | openai | semantic_security_discovery | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.1` | nondeterministic / provider_defaults |
+| pva-build-test | nv_inference | openai | meta_analyzer | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.1` | nondeterministic / provider_defaults |
+
+- Controls for `pva-build-test`: temperature=not set (source=provider_default, forwarded=no); seed=not set (source=unset, forwarded=no); reasoning_effort=not set (source=provider_default, forwarded=no)
+
+## What This Report Answers
+
+The three-tier evaluation checks whether the skill:
+
+- is safe to use;
+- produces correct answers;
+- is discovered and activated when needed;
+- helps the agent complete the user's goal and expected workflow; and
+- avoids wasted skill and tool usage.
+
+## Results at a Glance
+
+| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |
+|---|---:|---:|
+| Overall | 98.1% — baseline ran, but no comparable score was available; uplift unavailable | 95.3% — baseline ran, but no comparable score was available; uplift unavailable |
+| Security | 100.0% → 100.0% (±0.0 points) | 100.0% → 100.0% (±0.0 points) |
+| Correctness | 95.0% → 100.0% (+5.0 points) | 85.0% → 100.0% (+15.0 points) |
+| Discoverability | 93.3% — baseline ran, but no comparable score was available; uplift unavailable | 81.7% — baseline ran, but no comparable score was available; uplift unavailable |
+| Effectiveness | 100.0% → 100.0% (±0.0 points) | 100.0% → 100.0% (±0.0 points) |
+| Efficiency | 97.4% — baseline ran, but no comparable score was available; uplift unavailable | 94.7% — baseline ran, but no comparable score was available; uplift unavailable |
+
+**How to read this table:** baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points.
+
+Example: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline.
+
+A partial dimension was calculated from only the available configured signals; review the detailed report before relying on it.
+
+### Comparison details
+
+Shared-source comparisons use the same signals and weights on both conditions. They do not replace the full dimension scores above or change the publication verdict.
+
+| Agent | Dimension | Basis | Baseline → With skill (uplift) | Explanation |
+| --- | --- | --- | --- | --- |
+| claude-code | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. |
+| claude-code | Efficiency | Token-only comparison | 98.1% → 94.8% (-3.3 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. |
+| codex | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. |
+| codex | Efficiency | Token-only comparison | 92.5% → 89.3% (-3.2 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. |
+
+## Token Usage
+
+Actual Tier 3 execution usage is reported for every observed agent/case pair and both conditions.
+
+| Agent | Dataset case | With skill | Without skill | Delta | Change | Coverage |
+|---|---|---:|---:|---:|---:|---|
+| claude-code | All cases | 653,417 | 473,030 | +180,387 | +38.13% | skill 4/4; base 4/4 |
+| claude-code | build-test-emu-is-not-hardware | 208,791 | 183,553 | +25,238 | +13.75% | skill 1/1; base 1/1 |
+| claude-code | build-test-missing-inputs | 205,522 | 107,944 | +97,578 | +90.40% | skill 1/1; base 1/1 |
+| claude-code | build-test-negative | 35,473 | 35,300 | +173 | +0.49% | skill 1/1; base 1/1 |
+| claude-code | build-test-unverified-corpus | 203,631 | 146,233 | +57,398 | +39.25% | skill 1/1; base 1/1 |
+| codex | All cases | 408,644 | 278,266 | +130,378 | +46.85% | skill 4/4; base 4/4 |
+| codex | build-test-emu-is-not-hardware | 133,572 | 122,645 | +10,927 | +8.91% | skill 1/1; base 1/1 |
+| codex | build-test-missing-inputs | 131,478 | 55,392 | +76,086 | +137.36% | skill 1/1; base 1/1 |
+| codex | build-test-negative | 13,318 | 13,182 | +136 | +1.03% | skill 1/1; base 1/1 |
+| codex | build-test-unverified-corpus | 130,276 | 87,047 | +43,229 | +49.66% | skill 1/1; base 1/1 |
+| ALL AGENTS | Dataset aggregate | 1,062,061 | 751,296 | +310,765 | +41.36% | skill 8/8; base 8/8 |
+
+Prompt tokens include cached reads, so total tokens are `prompt + completion` (cached is not added twice). The Efficiency score uses `(prompt - cached) + completion`. Change is `100 * (with - without) / without`; negative means fewer tokens. N/A comparisons include their reason in the coverage column; coverage is never estimated.
+
+## Tier Status
+
+| Tier | Purpose | Status | Evidence |
+|---|---|---|---|
+| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 11 validator(s); 11 finding(s) |
+| Tier 2 | Semantic deduplication | **PASSED** | 2 validator(s); 0 finding(s) |
+| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 4 task(s) |
+
+## Findings and Observations
+
+<details>
+<summary>Show detailed findings and successful checks</summary>
+
+- **MEDIUM** QUALITY/quality_correctness: SKILL_SPEC recommended field missing: 'metadata.tags' (`skills/pva-build-test/SKILL.md`)
+- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/pva-build-test/SKILL.md`)
+- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/pva-build-test/SKILL.md`)
+- **MEDIUM** SECURITY/Sudo/Root Execution (PE2): Privilege Escalation: sudo  (`references/direct-ssh-tegra.md:117`)
+- **LOW** QUALITY/quality_correctness: No examples provided (`skills/pva-build-test/SKILL.md`)
+- 6 additional finding(s) are available in the full evaluation artifacts.
+
+</details>
+
+## Scoring Methodology
+
+<details>
+<summary>Show dimension definitions, source signals, and thresholds</summary>
+
+| Dimension | Question | Scored signals |
+|---|---|---|
+| Security | Is it safe to use? | `security` (100%) |
+| Correctness | Is the answer correct? | `accuracy` (100%) |
+| Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) |
+| Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) |
+| Efficiency | Did it avoid wasted tool calls and token usage? | `skill_efficiency` (50%) + `token_efficiency` (50%) |
+
+- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.
+- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.
+- Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate.
+- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.
+- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`).
+- Efficiency is 50% tool-call productivity (the backward-compatible `skill_efficiency` wire id) and 50% `token_efficiency`. Positive-case skill routing is scored under Discoverability, not Efficiency; a negative case without a routing target is N/A. N/A sources are omitted, remaining weights are renormalized, and the dimension is marked partial.
+
+Signals present in this run:
+
+- `security` (Security): unsafe operations, secret leakage, and unauthorized access.
+- `skill_execution` (Skill Execution): whether the expected skill was selected, decoys were avoided, and the workflow executed.
+- `skill_efficiency` (Tool Productivity): tool-call productivity (legacy wire id; routing is scored under Discoverability).
+- `accuracy` (Accuracy): final-answer correctness against the reference answer.
+- `goal_accuracy` (Goal Accuracy): whether the user's goal was achieved.
+- `behavior_check` (Behavior Check): whether the expected workflow behavior was followed.
+- `token_efficiency` (Token Efficiency): actual uncached prompt plus completion usage (50% of Efficiency).
+
+</details>
+
+## Freshness
+
+Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes.
@@ -0,0 +1,155 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+-->
+
+# PVA development skills
+
+Agent skills for creating, changing, or repairing NVIDIA PVA operators
+in your PVA SDK application or PVA Solutions source tree. They turn
+requirements into code, verify it on Tegra hardware, and repair failures within
+a debug budget.
+
+## Skills
+
+| Skill | Purpose |
+| --- | --- |
+| [Doctor](https://github.com/NVIDIA/skills/blob/main/skills/pva-doctor/SKILL.md) | Check the SDK, toolchain, project, target, and SDK references. |
+| [Setup](https://github.com/NVIDIA/skills/blob/main/skills/pva-setup/SKILL.md) | Install missing SDK, NVVPU, and L4T runtime components, then rerun Doctor. |
+| [Spec](https://github.com/NVIDIA/skills/blob/main/skills/pva-spec/SKILL.md) | Define operator behavior, correctness tests, and the OpGen handoff. |
+| [OpGen](https://github.com/NVIDIA/skills/blob/main/skills/pva-opgen/SKILL.md) | Generate and integrate the operator, then invoke Build-Test and Debug. |
+| [Build-Test](https://github.com/NVIDIA/skills/blob/main/skills/pva-build-test/SKILL.md) | Build, verify correctness on hardware, and measure requested performance cases. |
+| [Debug](https://github.com/NVIDIA/skills/blob/main/skills/pva-debug/SKILL.md) | Diagnose and repair a reproduced failure. |
+
+## Prerequisites
+
+- An authenticated coding agent with local skill support and shell/file access.
+  See [agent support](#agent-support) below.
+- A development host running regular x86-64 Ubuntu supported by the requested SDK.
+- An Arm64 Jetson board running compatible Jetson Linux (L4T), with working
+  key-based SSH and a trusted host key.
+- A matching SDK installation guide (file or URL), access to its installer,
+  and administrative permissions for required installation steps.
+- A writable project: your PVA SDK application or a PVA Solutions source package
+  or checkout. This is where OpGen adds or changes code. Install Git LFS if the
+  project uses it.
+
+When PVA Solutions is selected, Setup uses
+[NVIDIA/PVA-Solutions](https://github.com/nvidia/pva-solutions) by default and
+clones into the supplied project path, or `$WORK_DIR/source` when omitted.
+It reuses an existing source tree and accepts another repository or archive.
+Doctor checks the selected source and reports missing required files.
+
+Run the agent and build tools on the development host. `PVA_BUILD_MODE=L4T`
+cross-compiles there for execution on the board. `PVA_BUILD_MODE=EMU` builds
+for emulation on the development host.
+
+Setup installs or reuses the SDK, NVVPU, required build and reference tools,
+and board runtime. EMU is optional. Setup calls Doctor before
+making changes and again afterward; no existing Doctor report is required.
+Setup reads the installer URL from the guide. Supply a separate installer path
+or URL only when the guide does not provide one. The guide and installer must
+match the requested SDK release.
+Setup reads the board's installed BSP and checks SDK compatibility. Supply an
+expected BSP release only when you want to require a specific release.
+
+## Set up a workspace
+
+Copy all six PVA skill folders from the public
+[NVIDIA skills catalog](https://github.com/NVIDIA/skills/tree/main/skills)
+into `$HOME/pva-sdk-skills`. Keep each folder's scripts, references, and assets.
+The resulting layout is `$HOME/pva-sdk-skills/pva-setup/SKILL.md`, and the
+same layout for the other five skills.
+
+```bash
+git clone --depth 1 https://github.com/NVIDIA/skills.git "$HOME/nvidia-skills"
+mkdir -p "$HOME/pva-sdk-skills"
+for pva_skill in pva-setup pva-doctor pva-spec pva-opgen pva-build-test pva-debug; do
+  cp -R "$HOME/nvidia-skills/skills/$pva_skill" "$HOME/pva-sdk-skills/"
+done
+```
+
+Use a separate workspace for each operator task. Replace the project path below
+with your writable PVA SDK application or PVA Solutions source directory.
+
+```bash
+mkdir -p "$HOME/operator_test/my_operator"
+cd "$HOME/operator_test/my_operator"
+ln -s "/absolute/path/to/your/project" source
+
+pva_skill_dir=.claude/skills # For Codex, use .agents/skills
+mkdir -p "$pva_skill_dir"
+for pva_skill in "$HOME/pva-sdk-skills"/pva-*; do
+  ln -s "$pva_skill" "$pva_skill_dir/"
+done
+```
+
+Reload skills or restart the agent session after creating the links.
+`source/` points to the project OpGen will modify; the surrounding workspace
+holds your inputs, generated configuration, and reports. Each skill carries its
+required scripts and templates. Install all six for the full workflow;
+OpGen invokes Build-Test and Debug by their installed skill names.
+Generated SDK references belong in `references/sdk/` in this workspace.
+Doctor records their verified absolute location in its report.
+
+## Agent support
+
+- **Claude Code:** Used for the recorded operator runs and the commands below.
+- **Codex:** Supports [local skills](https://learn.chatgpt.com/docs/build-skills#where-to-save-skills)
+  in `.agents/skills`, invoked with `$pva-spec` or another skill name.
+  These skills include `agents/openai.yaml` metadata. OpGen uses the agent's
+  available skill mechanism for Build-Test and Debug. Without a callable skill
+  tool, it loads the child skill's `SKILL.md` and executes its full workflow
+  before resuming OpGen with the reports and verdict.
+
+## Run Setup → Doctor → Spec → OpGen
+
+Create `requirements.md` in the workspace: operator name/path, task mode (`new`,
+`change`, or `repair`), required behavior, supported inputs, API, and performance
+requirements. For an existing operator, include relevant source and test paths.
+
+Run these **headless Claude Code commands** one at a time from the development
+host's workspace. Replace the guide and board settings below. Setup resolves
+the SDK release, installer, package names, and installation paths from the
+guide and package metadata. Package paths are not fixed in the skill.
+You can provide an explicit SDK release or path override when needed.
+
+```bash
+export PVA_OPERATOR_MAX_DEBUG_ATTEMPTS=3
+
+# 1. Setup: check and prepare the host and board
+claude -p "Run pva-setup for spec, opgen, and build-test.
+Installation guide: http://docs.nvidia.com/pva/sdk/installation.html.
+Target: user@<board-IP>; PVA generation: <generation for your board>.
+Project: $PWD/source; work directory: $PWD."
+
+# 2. Doctor: verify readiness after Setup
+claude -p "Run pva-doctor for spec, opgen, and build-test using
+$PWD/reports/pva-setup-report.md.
+Project: $PWD/source; work directory: $PWD."
+
+# 3. Spec: run after Doctor is ready
+# Spec runs its finalizer to create and validate configure/opgen-inputs.json.
+claude -p "Invoke pva-spec through the Skill tool using
+$PWD/requirements.md and $PWD/reports/doctor-report.md.
+Project: $PWD/source; work directory: $PWD."
+
+# 4. OpGen: run after Spec returns CONFIGURED
+claude -p "Invoke pva-opgen through the Skill tool using $PWD/configure/opgen-inputs.json."
+```
+
+Doctor writes `$WORK_DIR/reports/doctor-report.md` automatically. The examples
+use `$PWD` as the work directory so Spec and OpGen can use the resulting files.
+
+OpGen runs Build-Test and invokes Debug as needed. Read the final result in
+`reports/opgen-report.md`.
+
+See [reference maintenance](https://github.com/NVIDIA/skills/blob/main/skills/pva-doctor/references/reference-maintenance.md) for SDK-reference
+regeneration and upgrades.
+
+## Maintaining the helpers
+
+Spec owns [finalize_opgen_inputs.py](https://github.com/NVIDIA/skills/blob/main/skills/pva-spec/scripts/finalize_opgen_inputs.py).
+OpGen ships [an identical copy](https://github.com/NVIDIA/skills/blob/main/skills/pva-opgen/scripts/finalize_opgen_inputs.py)
+so handoff validation works when the skills are installed separately.
+After changing the Spec helper, update the OpGen copy too.
+Generated SDK references and operator reports belong in the operator workspace.