chore(sync): bring measurement-neutral upstream aiperf changes into 1.0.x

.agents/skills/aiperf-code-review/SKILL.md

@@ -17,6 +17,7 @@ description: Review the current branch against origin/main, capture findings in
 
   Requirements:
   - Treat `artifacts/code-review.md` as a living document. Update it in place if it already exists.
+  - **CLI docs check:** If the diff adds or removes any CLI parameter (i.e., any `Annotated[..., CLIParameter(...)]` field or `--flag` name in `src/aiperf/config/flags/cli_config.py`), verify that `docs/cli-options.md` was regenerated to reflect those changes. The file is auto-generated by `make generate-cli-docs`; a missing regeneration is a blocking finding. The description field inside `Field(description=...)` is the source of truth for what appears in the docs — changes there also require a regen.
   - For each finding, record:
     - status: `Confirmed`, `Partially confirmed`, or `Not confirmed`
     - source-level evidence with exact file paths and line references

.cursor/rules/python.mdc

@@ -42,6 +42,7 @@ Python 3.11+ async AI benchmarking tool for measuring LLM inference server perfo
 - Use mermaid diagrams instead of ASCII art in markdown files.
 - Do not create markdown files to document code changes or decisions.
 - Do not over-comment code. Removing code is fine without adding comments to explain why.
+- Multi-line docstrings are permitted. Use them when the behavior is non-obvious.
 - No emojis in code or comments.
 - Hide a metric from the console table with `console_group = MetricConsoleGroup.NONE`; group it into a separate section with `MetricConsoleGroup.{USAGE,CACHE,PREDICTION,AUDIO,REASONING,SPEC_DECODE,GPU_POWER_EFFICIENCY_NVIDIA,GPU_POWER_EFFICIENCY_AMD}`. Default is `DEFAULT`. See `docs/metrics-reference.md` "Metric Console Group Reference".
 - Platform-conditional code MUST branch on `IS_WINDOWS` / `IS_MACOS` / `IS_LINUX` from `aiperf.common.constants`, never on `platform.system()` directly. The constants are evaluated once at import time, are uniformly greppable, and produce smaller diffs. See `src/aiperf/common/bootstrap.py` and `src/aiperf/config/comm/ipc.py` for canonical examples.
@@ -50,6 +51,10 @@ Python 3.11+ async AI benchmarking tool for measuring LLM inference server perfo
 
 Numeric metric values crossing a serialization boundary or feeding a numerical algorithm must be finite or explicitly `None`. Use `aiperf.common.finite` (`FiniteFloat`, `scrub_non_finite`, `nan_safe_mean`/`std`, `is_finite_value`). Mechanical CI invariants in `tests/unit/property/test_finite_invariants.py` reject new violations and ratchet existing debt to zero via baseline files. See [`docs/dev/patterns.md`](docs/dev/patterns.md) § "NaN/Inf Discipline Pattern" and [`docs/dev/global-invariants.md`](docs/dev/global-invariants.md) for the full contract.
 
+## CLI Flag Routing Under `--config`
+
+A CLI flag passed alongside `--config` MUST either change the resolved config or raise an error naming the flag — never be silently ignored, which for a benchmarking tool means publishing numbers that do not match what was asked for. `CLIConfig` is the source of truth, structurally: `resolve_config` receives a `CLIConfig` and a path and nothing else, so a flag reaches `AIPerfConfig` only by being a field on it. Every field must be classified in `src/aiperf/config/flags/_config_flag_routing.py` as `ROUTED_UNDER_CONFIG`, `UNROUTED_UNDER_CONFIG`, `EXEMPT_FROM_CONFIG_ROUTING`, or one of the conditional sets (`COMPANION_ROUTED`, `MAGIC_LIST_ONLY_UNDER_CONFIG`). Adding a flag without classifying it fails `test_every_cli_config_field_is_classified`; claiming a flag is routed when nothing routes it fails `test_routed_field_never_silently_no_ops`. Prefer routing the flag (check whether a builder already exists) over listing it as unrouted. See [`docs/dev/global-invariants.md`](docs/dev/global-invariants.md) § "CLI flag routing under `--config`".
+
 ## Build and Test Commands
 
 ```bash

.gitattributes

@@ -0,0 +1,14 @@
+# Normalize line endings to LF on checkout, including on Windows, where
+# git's default core.autocrlf=true otherwise converts text files to CRLF.
+# tests/ui/test_operator_*.py string/regex-strips import lines out of
+# src/aiperf/operator/ui/lib/*.js before eval'ing them as data: URL modules;
+# that stripping assumes LF and silently no-ops on CRLF, leaving the original
+# relative import in place and breaking module resolution on Windows CI.
+* text=auto eol=lf
+
+# Vendored third-party sources are checked out byte-for-byte on every platform.
+# Their value is being identical to a pinned upstream commit, and CI asserts a
+# sha256 over the raw bytes -- newline conversion on Windows checkouts would
+# change the digest without changing the content. This must stay last: later
+# patterns win, so the blanket rule above would otherwise re-normalize them.
+src/aiperf/dataset/loader/vendor/** -text

.github/CODEOWNERS

@@ -14,4 +14,4 @@
 # Docs: https://docs.github.com/articles/about-code-owners
 
 # Default owners for everything in the repository — the core NVIDIA AIPerf team.
-*   @ajcasagrande @debermudez @matthewkotila @ilana-n @lkomali @ganeshku1 @FrankD412
+*   @ai-dynamo/aiperf-codeowners

.github/copilot-instructions.md

@@ -37,6 +37,7 @@ Python 3.11+ async AI benchmarking tool for measuring LLM inference server perfo
 - Use mermaid diagrams instead of ASCII art in markdown files.
 - Do not create markdown files to document code changes or decisions.
 - Do not over-comment code. Removing code is fine without adding comments to explain why.
+- Multi-line docstrings are permitted. Use them when the behavior is non-obvious.
 - No emojis in code or comments.
 - Hide a metric from the console table with `console_group = MetricConsoleGroup.NONE`; group it into a separate section with `MetricConsoleGroup.{USAGE,CACHE,PREDICTION,AUDIO,REASONING,SPEC_DECODE,GPU_POWER_EFFICIENCY_NVIDIA,GPU_POWER_EFFICIENCY_AMD}`. Default is `DEFAULT`. See `docs/metrics-reference.md` "Metric Console Group Reference".
 - Platform-conditional code MUST branch on `IS_WINDOWS` / `IS_MACOS` / `IS_LINUX` from `aiperf.common.constants`, never on `platform.system()` directly. The constants are evaluated once at import time, are uniformly greppable, and produce smaller diffs. See `src/aiperf/common/bootstrap.py` and `src/aiperf/config/comm/ipc.py` for canonical examples.
@@ -45,6 +46,10 @@ Python 3.11+ async AI benchmarking tool for measuring LLM inference server perfo
 
 Numeric metric values crossing a serialization boundary or feeding a numerical algorithm must be finite or explicitly `None`. Use `aiperf.common.finite` (`FiniteFloat`, `scrub_non_finite`, `nan_safe_mean`/`std`, `is_finite_value`). Mechanical CI invariants in `tests/unit/property/test_finite_invariants.py` reject new violations and ratchet existing debt to zero via baseline files. See [`docs/dev/patterns.md`](docs/dev/patterns.md) § "NaN/Inf Discipline Pattern" and [`docs/dev/global-invariants.md`](docs/dev/global-invariants.md) for the full contract.
 
+## CLI Flag Routing Under `--config`
+
+A CLI flag passed alongside `--config` MUST either change the resolved config or raise an error naming the flag — never be silently ignored, which for a benchmarking tool means publishing numbers that do not match what was asked for. `CLIConfig` is the source of truth, structurally: `resolve_config` receives a `CLIConfig` and a path and nothing else, so a flag reaches `AIPerfConfig` only by being a field on it. Every field must be classified in `src/aiperf/config/flags/_config_flag_routing.py` as `ROUTED_UNDER_CONFIG`, `UNROUTED_UNDER_CONFIG`, `EXEMPT_FROM_CONFIG_ROUTING`, or one of the conditional sets (`COMPANION_ROUTED`, `MAGIC_LIST_ONLY_UNDER_CONFIG`). Adding a flag without classifying it fails `test_every_cli_config_field_is_classified`; claiming a flag is routed when nothing routes it fails `test_routed_field_never_silently_no_ops`. Prefer routing the flag (check whether a builder already exists) over listing it as unrouted. See [`docs/dev/global-invariants.md`](docs/dev/global-invariants.md) § "CLI flag routing under `--config`".
+
 ## Build and Test Commands
 
 ```bash

.github/workflows/nightly.yml

@@ -593,6 +593,9 @@ jobs:
       # advisory; this gate sits inside the publish path.
       - name: Run unit tests against installed nightly wheel (amd64 only)
         if: matrix.arch == 'amd64'
+        timeout-minutes: 45
+        env:
+          MALLOC_ARENA_MAX: "2"
         run: |
           set -euo pipefail
           NIGHTLY=$(find dist/ -maxdepth 1 -name "aiperf_nightly-*.whl" -type f | head -1)
@@ -620,8 +623,9 @@ jobs:
           # package is on sys.path via the venv; src/ is NOT on sys.path,
           # so `import aiperf` resolves to the installed wheel rather than
           # the source tree.
+          # -n auto on the many-core builder starved the runner of memory.
           .venv-wheel-tests/bin/pytest tests/unit \
-            -n auto \
+            -n 8 --dist=worksteal \
             -m 'not performance and not stress' \
             --tb=short
 

.github/workflows/run-fern-checks.yml

@@ -1,4 +1,4 @@
-# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 name: "Fern Docs Checks"
 
@@ -19,10 +19,10 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v6
-    - name: Set up Node.js 20
+    - name: Set up Node.js 22
       uses: actions/setup-node@v5
       with:
-        node-version: "20"
+        node-version: "22"
     - name: Install Fern CLI
       run: npm install -g fern-api@$(jq -r '.version' fern/fern.config.json)
     - name: Set up Python 3.12

.github/workflows/run-integration-tests.yml

@@ -8,6 +8,10 @@ on:
     branches:
       - main
 
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
 jobs:
   integration-tests:
     runs-on: ${{ matrix.os }}
@@ -16,6 +20,10 @@ jobs:
       matrix:
         os: ["ubuntu-latest"]
         python-version: ["3.11", "3.12", "3.13"]
+        shard: [0, 1, 2, 3]
+    env:
+      AIPERF_SHARD_INDEX: ${{ matrix.shard }}
+      AIPERF_SHARD_TOTAL: 4
 
     steps:
     - uses: actions/checkout@v6

.github/workflows/run-unit-tests.yml

@@ -15,7 +15,7 @@ jobs:
   build:
     runs-on: ${{ matrix.runner }}
     # Windows x64 runners can finish the component tier just past 30 minutes.
-    timeout-minutes: 45
+    timeout-minutes: 90
     # Force CPython UTF-8 mode so file reads default to UTF-8 instead of the
     # Windows cp1252 code page (no-op on POSIX). Without it, reading a UTF-8
     # file on Windows raises UnicodeDecodeError on bytes like 0x8f.
@@ -108,36 +108,104 @@ jobs:
         # builder pod. -n auto without this OOM-kills the runner agent.
         PYTEST_XDIST_AUTO_NUM_WORKERS: "4"
       run: |
-        make test-ci PYTHON_VERSION=${{ matrix.python-version }}
+        # Only collect coverage on the linux/3.12 leg where Codecov upload runs.
+        COV=${{ (matrix.os == 'linux' && matrix.python-version == '3.12') && '1' || '0' }}
+        make test-ci PYTHON_VERSION=${{ matrix.python-version }} COV=$COV
     # Mirrors POSIX ``make test-ci`` semantics: run BOTH tiers and
     # aggregate exit codes so unit-test failures don't mask component-
     # integration failures (or vice versa). Without the explicit
     # ``always() && runner.os == 'Windows'`` guard on the second step,
     # GitHub Actions' implicit ``success()`` short-circuits the
     # component-integration step on any unit-test failure.
-    - name: Run unit tests (Windows)
+    - name: Run unit and zmq tests (Windows)
+      # Merged into one invocation to pay collection overhead once instead of
+      # twice. tests/unit/conftest.py's no_sleep patch applies only to
+      # tests/unit/**; tests/zmq/** uses real asyncio.sleep as before.
+      # --no-looptime is omitted: looptime is opt-in per test via marker/
+      # fixture and zmq tests use neither, so they run with a real-time loop.
+      # --continue-on-collection-errors preserves the previous two-step
+      # behaviour where a unit collection error did not block the zmq suite.
       if: runner.os == 'Windows'
+      # -n logical, not -n auto: xdist's ``auto`` asks psutil for the PHYSICAL
+      # core count, and the GitHub x86 Windows VM reports 1 physical core for
+      # its 4 vCPUs, so ``auto`` ran the whole suite in a single worker (~3x
+      # slower than windows-11-arm, which happens to report 3). ``logical``
+      # uses the logical count on both and tracks the runner size if it grows.
       run: |
-        uv run pytest tests/unit -n auto --tb=short -m 'not performance and not stress and not slow'
-    - name: Run zmq real-transport tests (Windows)
-      # Real-socket libzmq tests (no looptime). ``!cancelled()`` lets them run
-      # even if the unit step failed, matching the POSIX ``make test-ci``
-      # exit-code aggregation across tiers.
-      if: ${{ !cancelled() && runner.os == 'Windows' }}
-      run: |
-        uv run pytest tests/zmq --tb=short --no-looptime -m 'not performance and not stress and not slow'
+        uv run pytest tests/unit tests/zmq -n logical --dist=worksteal --tb=short --continue-on-collection-errors -m 'not performance and not stress and not slow'
     - name: Run component integration tests (Windows)
       # ``!cancelled()`` overrides the implicit ``success()`` so component
       # integration still runs even if the unit-test step failed — matching
       # POSIX ``make test-ci`` which aggregates exit codes across both tiers.
       # Cancellation is still respected so we don't burn runner minutes
       # finishing after a manual stop.
       if: ${{ !cancelled() && runner.os == 'Windows' }}
+      # -n logical for the reason given on the unit-test step above.
       run: |
-        uv run pytest tests/component_integration -n auto --tb=short -v -m 'not performance and not stress and not slow'
+        uv run pytest tests/component_integration -n logical --dist=worksteal --tb=short -v -m 'not performance and not stress and not slow'
     - name: Upload results to Codecov
       if: matrix.os == 'linux' && matrix.python-version == '3.12'
       uses: codecov/codecov-action@v5
       with:
         token: ${{ secrets.CODECOV_TOKEN }}
         slug: ai-dynamo/aiperf
+
+  botorch-regression:
+    name: BoTorch Regression Tests (CPU)
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    permissions:
+      contents: read
+    env:
+      PYTEST_XDIST_AUTO_NUM_WORKERS: "4"
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        persist-credentials: false
+    - name: Set up Python 3.13
+      uses: actions/setup-python@v6
+      with:
+        python-version: "3.13"
+    - name: Set up uv with cache
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        cache-dependency-glob: "uv.lock"
+    - name: Install CPU-only BoTorch dependencies
+      run: |
+        make ci-install PYTHON_VERSION=3.13
+        uv pip install --torch-backend cpu -e ".[botorch]"
+    # Split across two steps rather than two lines of one ``run:``. ``run:``
+    # executes under ``bash -e``, so a failure in the first file would abort
+    # the step and the slow suite would never run. ``!cancelled()`` on the
+    # second overrides the implicit ``success()`` so both suites always
+    # report -- matching the Windows tiers above.
+    - name: Run BoTorch GP kernel regression tests
+      run: |
+        uv run pytest -n auto --dist=worksteal \
+          tests/unit/orchestrator/search_planner/test_botorch_kernel.py
+    - name: Run BoTorch GP candidates_func regression tests (slow)
+      if: ${{ !cancelled() }}
+      run: |
+        uv run pytest -n auto --dist=worksteal -m slow \
+          tests/unit/orchestrator/search_planner/test_optuna_dsp_kernel.py
+
+  # The merge ruleset requires this single check rather than each matrix leg,
+  # whose names change whenever a leg is added or renamed. always() keeps it
+  # from being skipped, because GitHub treats a skipped required check as passed.
+  unit-tests-gate:
+    name: Unit Tests Gate
+    needs: [build, botorch-regression]
+    if: always()
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    permissions: {}
+    steps:
+    - name: Require every unit test job to pass
+      env:
+        RESULTS: ${{ join(needs.*.result, ' ') }}
+      run: |
+        echo "Job results: ${RESULTS}"
+        for result in ${RESULTS}; do
+          [ "${result}" = "success" ] || exit 1
+        done

.gitignore

@@ -57,3 +57,4 @@ src/aiperf/_build_info.py
 
 # Chaos/adversarial test run artifacts
 tests/scripts/.chaos_runs/
+.worktrees/

.pre-commit-config.yaml

@@ -29,27 +29,33 @@ repos:
       - id: check-toml
       - id: check-yaml
       - id: check-shebang-scripts-are-executable
+      # The vendored SPEED-Bench resolver is byte-pinned to upstream by a
+      # sha256 CI gate. Any whitespace or newline rewrite breaks that pin.
       - id: end-of-file-fixer
         types_or: [cuda, proto, textproto, python]
+        exclude: ^src/aiperf/dataset/loader/vendor/
       - id: mixed-line-ending
+        exclude: ^src/aiperf/dataset/loader/vendor/
       - id: no-commit-to-branch
         args: [--branch, main]
       - id: requirements-txt-fixer
       - id: trailing-whitespace
+        exclude: ^src/aiperf/dataset/loader/vendor/
   - repo: https://github.com/codespell-project/codespell
     rev: v2.4.2
     hooks:
     - id: codespell
       additional_dependencies: [tomli]
       args: ["--toml", "pyproject.toml"]
-      exclude: (?x)^(.*stemmer.*|.*stop_words.*|^CHANGELOG.md$|.*shakespeare.txt|ATTRIBUTIONS-.*\.md)
+      exclude: (?x)^(.*stemmer.*|.*stop_words.*|^CHANGELOG.md$|.*shakespeare.txt|ATTRIBUTIONS-.*\.md|src/aiperf/dataset/loader/vendor/.*)
   - repo: local
     hooks:
       - id: add-license
         name: add-license
         entry: make add-copyright
         language: python
         require_serial: true
+        exclude: ^src/aiperf/dataset/loader/vendor/
       - id: generate-cli-docs
         name: generate-cli-docs
         entry: make generate-cli-docs