Skip to content

Accelerate and strengthen asset tests - #7282

Open
AntoineRichard wants to merge 26 commits into
isaac-sim:developfrom
AntoineRichard:antoiner/asset-tests-redesign
Open

Accelerate and strengthen asset tests#7282
AntoineRichard wants to merge 26 commits into
isaac-sim:developfrom
AntoineRichard:antoiner/asset-tests-redesign

Conversation

@AntoineRichard

@AntoineRichard AntoineRichard commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR redesigns the asset tests around a clear ownership model:

  • shared contract tests (renamed from the old iface suite) cover only the
    public base-class API, data, selectors, writers, cache invalidation, and
    backend parity;
  • backend unit/kernel tests cover backend-specific transforms, staging,
    ordering, target modes, cache behavior, TensorAPI layouts, and unsupported
    capability declarations;
  • focused integration tests retain real-solver coverage for each supported
    asset family;
  • WrenchComposer keeps literal unit coverage plus one real delivery-parity
    test.

The articulation integration tests now use one persistent composite scene per
backend
. Each module creates isolated two-environment actor islands, resets
once, and exposes the same four core pytest nodes:

  1. initialization and partial joint state;
  2. joint and body property round-trips;
  3. drive delivery and live dynamics;
  4. floating-root state and wrench response.

Newton retains its model-change notification assertions in the property node.
PhysX adds one spatial-tendon node, while OVPhysX adds spatial-tendon and native
actuator-effort nodes, for totals of 4, 5, and 6 probes respectively. Partial
writes target environment 1 and prove environment 0 is preserved in real
backend state.

This grows real articulation coverage from 7 to 15 probes while reducing their
aggregate warmed wall time from 13.86s to 11.61s. The test functions remain
separate, but scene construction and shutdown are paid only once per module.

Newton's focused asset tests are Kit-free, use locally authored models, and do
not depend on Nucleus. The small composite Newton model runs on CPU because it
was faster locally; a duplicate CPU/GPU matrix is not required for this shared
contract. PhysX and OVPhysX retain GPU execution where it exercises meaningful
backend behavior. Cable and MPM are intentionally unchanged and excluded from
this work.

Assertions removed from the old large integration matrices have explicit new
owners: shared behavior moved to contract tests, backend branches moved to
unit/kernel tests, and real solver seams stayed in integration tests. The final
rebase also preserves upstream Newton root/body link and COM alias assertions
inside the reduced rigid-object integration probes.

The stronger tests exposed and fixed PhysX COM/inertia TensorAPI layout
handling and made OVPhysX device-manager reuse hardware-aware. Shared writer
contracts now verify selected and unselected backend state instead of merely
checking that a call does not raise.

No new required or optional dependency is introduced.

Performance

The five comparable CI-style scopes dropped from 19m57.60s to 2m14.11s
wall time: 8.93x faster overall. Before and after use the same repository
test orchestrator with one fresh subprocess per selected test file, a warmed
writable Warp cache, and the same machine/environment.

Scope Before files / cases After files / outcomes Before wall After wall Speed-up
Shared assets 8 / 4,331 6 / 1,238 pass, 103 skip 84.02 s 17.92 s 4.69x
Newton assets (no cable/MPM) 6 / 644 15 / 59 pass 608.44 s 50.57 s 12.03x
PhysX assets 7 / 486 12 / 44 pass 255.30 s 31.92 s 8.00x
OVPhysX assets 9 / 492 12 / 60 pass 220.05 s 28.80 s 7.64x
WrenchComposer 3 / 412 2 / 15 pass 29.79 s 4.90 s 6.08x
Aggregate 33 / 6,365 47 / 1,416 pass, 103 skip 1,197.60 s 134.11 s 8.93x

The final scopes take 34.06 seconds of aggregate pytest time and 134.11
seconds of subprocess wall time. Controller-owner and OV manager-lifecycle
checks are reported separately and excluded from both comparison denominators.
The committed results report contains the environment, exact commands,
per-suite pytest times, disposition map, and coverage-hole audit.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • Test-suite redesign
  • Documentation update

Testing

  • ./isaaclab.sh -f
  • shared asset scope: 1,238 passed, 103 skipped in 17.92s subprocess wall
    time
  • Newton asset scope: 59 passed in 50.57s subprocess wall time
  • PhysX asset scope: 44 passed in 31.92s subprocess wall time
  • OVPhysX asset scope: 60 passed in 28.80s subprocess wall time
  • WrenchComposer scope: 15 passed in 4.90s subprocess wall time
  • direct warmed articulation modules: 15 passed in 11.61s aggregate wall
    time
  • independent whole-branch and focused contract reviews

Screenshots

Not applicable; this change affects tests and backend correctness.

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with ./isaaclab.sh -f
  • I have made corresponding changes to the documentation
  • My changes generate no new unexpected warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package
  • My name already exists in CONTRIBUTORS.md

@AntoineRichard
AntoineRichard requested a review from a team August 21, 2026 16:46
@github-actions github-actions Bot added documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team labels Aug 21, 2026
@greptile-apps

greptile-apps Bot commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR substantially redesigns asset testing around shared contracts, focused backend units, and minimal local integration seams while reducing suite runtime. It also corrects PhysX COM/inertia TensorAPI layout handling, improves deformable classification, and enables hardware-aware OVPhysX manager reuse.

  • Replaces broad backend test matrices with shared API, data, writer, selector, and cache contracts.
  • Adds focused Newton, PhysX, and OVPhysX backend and lifecycle coverage using local fixtures.
  • Updates PhysX rigid-collection and deformable production behavior uncovered by the stronger tests.
  • Revises OVPhysX runtime and scene configuration for sequential CPU and CUDA contexts.

Confidence Score: 5/5

The PR appears safe to merge, with no concrete changed-code defect identified.

The production changes preserve documented asset buffer ordering and lifecycle boundaries, while the expanded focused tests cover the corresponding PhysX layouts, deformable classification, and OVPhysX mixed-device reuse.

Important Files Changed

Filename Overview
source/isaaclab_ov/isaaclab_ov/physics/ovphysx_manager.py Removes the process-wide device lock and configures CPU/GPU behavior through scene attributes and runtime construction, with dedicated mixed-device lifecycle coverage.
source/isaaclab_physx/isaaclab_physx/assets/rigid_object_collection/rigid_object_collection.py Converts structured COM and inertia buffers into the body-major float layouts expected by PhysX collection setters.
source/isaaclab_physx/isaaclab_physx/assets/deformable_object/deformable_object.py Centralizes surface-versus-volume classification using material schemas with topology fallback.
source/isaaclab/test/assets/contract/capabilities.py Defines explicit backend capability declarations used to retain unsupported contract cases as reasoned skips.
source/isaaclab/test/assets/contract/test_asset_contract_writes.py Consolidates shared writer-contract execution, including selected and unselected backend-state checks.
docs/superpowers/specs/2026-08-21-asset-test-suite-redesign-results.md Documents the redesigned ownership model, measured performance, test commands, coverage disposition, and known capability exclusions.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Contracts["Shared asset contracts"] --> Newton["Newton focused units and local integration"]
  Contracts --> PhysX["PhysX focused units and local integration"]
  Contracts --> OVPhysX["OVPhysX focused units and local integration"]
  PhysX --> Layout["COM and inertia TensorAPI layout fixes"]
  PhysX --> Classification["Deformable view classification"]
  OVPhysX --> Lifecycle["CPU/CUDA manager lifecycle reuse"]
  Newton --> Results["Faster asset-test gates"]
  Layout --> Results
  Classification --> Results
  Lifecycle --> Results
Loading

Reviews (1): Last reviewed commit: "Strengthen asset test contracts" | Re-trigger Greptile

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

The asset-test redesign establishes shared contracts, backend-focused units, and small real-solver seams, with accompanying PhysX and OVPhysX fixes. Before merge, remove the committed benchmark/work-report artifact and eliminate the redundant, unenforced public-member snapshots.

  • Design and architecture: The contract/unit/integration ownership split and explicit backend-capability table are coherent. However, the committed redesign-results document is a machine- and revision-specific work artifact prohibited by repository guidance, and public_surface.py contains unused duplicate member inventories that can silently diverge from the enforced mappings.
  • API: The touched production APIs retain their signatures: PhysX collection COM/inertia changes reshape only at the TensorAPI boundary, deformable type inference preserves material-before-topology precedence, and the OVPhysX device lifecycle change affects internal manager behavior. The public-surface audit is useful, but its unused duplicate snapshots should be removed or made authoritative.
  • Implementation: The contract bootstrap, backend capability declarations, manager patch scoping, writer readbacks, fused-layout checks, and backend regression seams were traced. Required cleanup is limited to removing the generated benchmark report from the repository and consolidating the duplicated public-member inventory.

Minor fixes needed. Posted 2 actionable findings inline.

Automated review; human maintainers own approval decisions.

- Times in the comparison are the repository orchestrator's aggregate pytest
and subprocess wall times, not the outer shell duration.

## Copy-ready PR performance section

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Warning · Design Architecture — Committed generated benchmark results artifact

This file is a generated work report: it contains a "Copy-ready PR performance section", a rebase-specific base SHA, a machine-specific environment (GPU, driver, wheel versions), sandbox paths such as /tmp/isaaclab-task8-warp, and a per-module disposition map. Repository guidelines prohibit committing generated plans, scratch files, or agent artifacts. It will go stale immediately after merge and carries no reader-facing documentation value. Keep these measurements in the pull-request description instead.



_PUBLIC_MEMBER_SNAPSHOT = {
"AssetBase": """

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Suggestion · Design Architecture — Unused member snapshots duplicate enforced mapping

Only the BaseRigidObjectData, BaseRigidObjectCollectionData, and BaseArticulationData entries of _PUBLIC_MEMBER_SNAPSHOT feed PUBLIC_SURFACE_CONTRACTS; the AssetBase, BaseRigidObject, BaseRigidObjectCollection, and BaseArticulation entries are never referenced and respell the same member names that the _mapping(...) calls list inline. A future API change updates only the audited copy, leaving a silently diverging second inventory. Drop the unreferenced entries or build the mappings from them.

Remove Isaac Lab's process device lock and avoid enabling the wheel's sticky CPU-only mode for ordinary CPU scenes. Explicitly author CPU and GPU scene dynamics settings, and cover lifecycle reuse in one process.
Organize the shared asset contracts by API, data, and write behavior while keeping assertion definitions reusable. Declare backend capabilities explicitly and classify the public base surface so missing coverage cannot disappear silently.
Exercise installed PhysX through kitless boundary stubs and classify each contract family with its actual backend capability. Make public-surface coverage an exact, reasoned map to concrete contract targets.
Classify rigid-object and collection cache invalidation under the writes capability. Cover all three asset families in the selector regression.
Replace randomized size matrices and duplicated PhysX scenarios with literal unit coverage and one rotated force-at-position parity case.
Cover distinct global, torque, reset, and raw-buffer paths with literal tests. Make the retained PhysX parity geometry detect rotated induced torque.
Seed an inactive composer with test-only raw data so the merge guard remains observable.
Split backend-specific logic into fast Kitless units and retain a minimal set of local real-solver seams. Move actuator runtime and termination behavior to their owning suites, and fix collection COM and inertia TensorAPI layouts exposed by the new acceptance coverage.
Add literal backend readbacks and temporal actuator contracts that catch cached-only or stale-state regressions. Replace the skipped remote surface-gripper cases with local real and fast unit seams, and isolate Kitless imports from later real imports.
Restore every target-package module imported under lightweight stubs so transitive data classes cannot retain the fake manager. Keep the local SurfaceGripper integration visible to Isaac Sim's short-CI source selector.
Move helper and kernel coverage into fast unit modules and replace remote asset matrices with focused local solver seams. Cover fused collection layouts, native actuator adaptation, deformable lifecycle, and CPU staging while retaining mixed-device behavior.
Verify inertial and friction setters against raw backend bindings. Prove implicit and native actuator commands move the real solver, and own CPU warmup and selective reset behavior with focused tests.
Scope backend manager mocks to tests and classify real solver modules.

Keep the fast Newton guard under target and document final speed-ups.
Keep internal design and execution notes out of the review while retaining the measured results artifact. Make the published gate commands self-contained with their cache and EULA prerequisites.
Make fast suites jointly collectable and marker-selectable, replace no-op writer seams with observable state checks, and declare backend limitations explicitly. Keep retained integration fixtures local and hardware-aware.
Reuse one backend scene per articulation module so broader real-solver coverage does not repeatedly rebuild simulation state. Qualify PhysX unit module names to keep mixed collection collision-free and refresh the measured performance report.
Retain the upstream link and center-of-mass alias checks in the reduced Newton integration seams. Refresh the benchmark report against the rebased develop head so the PR speed-up remains reproducible.
@AntoineRichard
AntoineRichard force-pushed the antoiner/asset-tests-redesign branch from d815b16 to a3d7711 Compare August 24, 2026 08:56
Align the real articulation probes across Newton, PhysX, and OVPhysX while preserving backend-specific coverage. Exercise partial writes against two live environments in one cached scene so multi-environment behavior is proven without restoring repeated startup cost.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant