Skip to content

Add LiDAR noise and dropout robustness benchmark (#279) - #306

Merged
ashishsinghbora merged 1 commit into
StellarResearch:mainfrom
chaewon-research:feature/lidar-robustness
Nov 10, 2026
Merged

ashishsinghbora merged 1 commit into
StellarResearch:mainfrom
chaewon-research:feature/lidar-robustness

Conversation

@chaewon-research

Copy link
Copy Markdown
Contributor

Summary

Adds a LiDAR robustness benchmark for the 3D drone navigation environment, evaluating a fixed PPO policy under five standardized sensor conditions.

The benchmark uses 30 matched evaluation seeds per condition and deterministic policy actions, without retraining between conditions.

Changes

  • Added five LiDAR conditions covering noise, dropout, combined perturbations, and severe sensor degradation.
  • Added success, collision, timeout, obstacle clearance, path length, and cumulative reward metrics.
  • Added failure classifications for obstacle and boundary collisions, freezing, and oscillatory timeouts.
  • Added CSV and JSON exports, a robustness plot, and the adaptive-rl benchmark lidar CLI command.
  • Added benchmark documentation and automated tests.

Validation

  • 80 focused and regression tests passed.
  • Ruff formatting and lint checks passed.
  • Mypy passed across 43 source files.
  • Completed a 30-episode evaluation for each of the five conditions.

Example results

Using a nominal PPO checkpoint trained for 50,000 timesteps, severe sensor degradation increased the observed collision rate from 26.7% to 46.7%. Paired episode comparisons identified seven newly occurring collisions and one avoided collision under the severe condition.

Performance was not monotonically worse across all sensor conditions. Moderate dropout and combined perturbations produced higher success rates than ideal sensing. These findings are based on one trained policy and 30 evaluation seeds per condition, and should be interpreted as descriptive rather than statistically conclusive.

The full experimental setup, results, and limitations are documented in docs/BENCHMARKS.md.

Related to #279.

@ashishsinghbora
ashishsinghbora merged commit 233b4bc into StellarResearch:main Nov 10, 2026
2 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants