Skip to content

[Validation] Add reproducible CubeCL CPU and accelerator conformance runner #50

Description

@code-tc

Objective

Provide the reproducible native validation lane used to accept CubeCL HAL work on CPU and at least one designated accelerator, with exact environment identity and retained machine-readable evidence.

Scope

  • Define the initial supported validation matrix after Introduce backend abstraction and minimal CubeCL backend #34 records the CubeCL runtime decision.
  • Pin runner image/toolchain, Rust, CubeCL, OS packages and test/benchmark commands.
  • Record device vendor/model, driver, runtime/backend, memory and relevant capabilities.
  • Run shared host-versus-CubeCL correctness fixtures and collect structured results.
  • Separate cold compilation, warm execution, transfer and peak-memory measurements.
  • Retain logs, results and benchmark artifacts by commit.
  • Detect unavailable, changed or incorrectly emulated hardware and fail/skip explicitly.
  • Keep benchmark thresholds and supported-platform declarations human-owned.

Acceptance criteria

  • A clean run can reproduce the CPU lane from documented commands.
  • A designated accelerator lane proves actual device dispatch, not compilation or CPU emulation.
  • Every artifact identifies commit, toolchain, CubeCL, OS, device, driver and backend/runtime.
  • Numerical failures include operation, dtype, shape, expected/actual summary and tolerance contract without dumping sensitive data.
  • Cold/warm/transfer/memory metrics are separate and machine-readable.
  • Missing or changed hardware cannot be reported as a passing accelerator test.
  • An independent reviewer can download and inspect retained artifacts.
  • Runner credentials cannot access production or unrelated repositories/resources.

Non-goals

Choosing final performance thresholds, declaring broad hardware support, browser/WASM validation (owned by #39), or retiring OpenCL.

Tembo execution contract

automation:
  eligible: true
  executor_profile: tembo
  readiness: blocked
  primary_repository: TinyChain-Inc/ha-ndarray
  base_branch: main
  writable_scope:
    - ".github/workflows/**"
    - "scripts/validation/**"
    - "tests/conformance/**"
    - "benches/**"
    - "docs/validation/**"
  prerequisites:
    - "#34 runtime/version decision"
  concurrency_group: cubecl-validation-infrastructure
  required_commands:
    - "cargo fmt --all -- --check"
    - "cargo test --no-default-features --features cubecl"
    - "cargo test --all-features"
    - "run the documented designated-accelerator validation command introduced by this issue"
  evidence:
    cpu_lane: required
    actual_designated_accelerator: required
    exact_environment_identity: required
    structured_results: required
    retained_artifacts: required
  external_actions: designated-runner-only
  draft_pr_only: true
  completion_authority: human-infrastructure-and-HAL-review
  budget: {size: large, max_revisions: 3, max_wall_minutes: 180}

Program: #33

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions