Skip to content

[Feature/adaptive rl doc] created the cli for the doctor which checks the dependencies and hardware requirements for the ARL to work on the system - #302

Open
LazyKaddu wants to merge 6 commits into
StellarResearch:mainfrom
LazyKaddu:feature/adaptive-rl-doc

Conversation

@LazyKaddu

Copy link
Copy Markdown

Resolves #290

Summary of Changes

  • Core Diagnostic Engine: Created SystemDoctor in src/adaptive_rl/diagnostics/doctor.py to evaluate 10 distinct categories of local environment health.
  • Implemented 10 Diagnostic Checks: Added automated verifications for Python versions (3.10 to 3.12), hardware acceleration (CUDA/MPS), core and GUI dependencies, custom Gymnasium environment registration, model checkpoint serialization, Git repository state, I/O write permissions, YAML config structural integrity, and classical planner pathfinding execution.
  • CLI Integration: Registered the adaptive-rl doctor command in src/adaptive_rl/cli.py using Typer.
  • Rich Output & Remediation: Formatted terminal output using Rich (green checkmarks, yellow warnings, red crosses) and paired failures with exact, actionable remediation commands (e.g., specific pip commands).
  • JSON Export & Exit Codes: Added a --json flag to export a machine-readable structural diagnostic report, and wired the command to return a non-zero exit code (sys.exit(1)) on critical failures.
  • Unit Tests: Added tests/test_doctor.py providing isolated unit tests using unittest.mock to verify the diagnostic logic, JSON schema, and Typer CLI exit codes.
  • Documentation: Updated README.md and introduced docs/GETTING_STARTED.md to prominently document the system doctor as a first-step troubleshooting tool.

Verification

  • pytest tests/ passes
  • ruff check src/ tests/ passes
  • mypy src/ passes

@LazyKaddu

Copy link
Copy Markdown
Author

Hi @ashishsinghbora waiting for the review this pr

@ashishsinghbora ashishsinghbora left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LazyKaddu I went through this PR in detail. The overall idea of a "system doctor" is useful, but I don't think this is merge-ready yet. The main issue is that several diagnostics look stronger than what they actually validate.

The biggest concern is project-context detection. "check_git_health()", "check_io_permissions()", and "check_yaml_configs()" all operate relative to the current working directory. That means "adaptive-rl doctor" can produce completely different results depending on where the user invokes it from. Running the command outside the repository can report missing "configs/", create "artifacts/" and "logs/" in the user's current directory, and report the wrong Git state. A system diagnostic command should resolve the actual ARL project/package root rather than assuming "cwd == repository root".

The classical planner check is also misleading. "check_classical_planner()" implements a tiny local A* function inside the diagnostic itself. It never imports or executes ARL's actual planner implementation. Therefore, this check can pass even if the real planner in "adaptive_rl/algorithms" is completely broken. This should test the project's real planner with a minimal deterministic fixture.

There is a similar issue with Gymnasium registration. The code attempts to import "adaptive_rl.environments", but deliberately ignores an "ImportError" and continues checking the registry. That can hide the exact failure that prevented environment registration. A failed environment import should be surfaced as a diagnostic failure with the underlying exception, not silently swallowed.

The hardware check also needs clearer semantics. Treating “no CUDA/MPS” as a warning is reasonable if CPU execution is officially supported, but the doctor should determine whether the selected ARL configuration actually requires GPU acceleration. Otherwise a perfectly valid CPU-only deployment can be presented as unhealthy.

The remediation commands are another concern. Hard-coded commands such as "pip install torch" or a specific CUDA wheel can easily produce an environment that conflicts with the project's pinned dependencies. The doctor should preferably reference the project's supported installation mechanism/version constraints instead of inventing a generic pip command.

Finally, the tests are heavily mocked and mostly verify that mocked dependencies return the expected status. They don't give much confidence that the doctor works against a real ARL installation. I'd strongly recommend integration tests covering at least: running from the repository root, running from another directory, missing/broken environment registration, real planner execution, malformed YAML, CPU-only machines, and missing optional dependencies.

So my recommendation is Request changes. The architecture is a good start, but before merging, the doctor needs to prove that it is diagnosing the actual ARL system, not just successfully executing a collection of isolated sanity checks.

@LazyKaddu

Copy link
Copy Markdown
Author

Hey @ashishsinghbora you can check the changes i have created according to the suggestions you have given

@AryanXCode646

Copy link
Copy Markdown
Contributor

Hi @LazyKaddu
Thanks for addressing the earlier review feedback. The project-root handling, environment registration error reporting, and planner-check changes are steps in the right direction. However, there are still correctness, packaging, and verification gaps that need to be resolved before this PR is ready to merge.

Remaining issues

1. P1 — CI lint and formatting checks are still failing

The latest visible CI run reports that “Check code formatting with ruff” failed. A passing Python 3.10 test job does not compensate for a failing required quality check.

Please format the changed files, resolve any remaining lint/type-check issues, and rerun the complete relevant quality checks. Please share the results from the updated commit rather than relying on the verification claims in the PR description.

2. P1 — The classical planner check still needs real integration coverage

check_classical_planner() imports adaptive_rl.algorithms.planning.AStarPlanner, but test_check_classical_planner_pass() injects a mocked module into sys.modules. This allows the test to pass without proving that the import resolves to ARL's actual planner implementation.

Please verify the import path against the real repository implementation and execute the planner using a deterministic fixture without replacing the planner module with a mock. The check must fail meaningfully if the actual planner is missing or broken.

3. P1 — Dependency checks do not align with the project's declared dependencies

check_core_dependencies() treats PyTorch, Stable-Baselines3, Gymnasium, and SciPy as core requirements. However, the project metadata declares PyTorch, Stable-Baselines3, and Gymnasium under the optional rl extra, while SciPy is not listed in the shown dependency definitions. The remediation message recommends pip install -e ., which does not install the declared rl extra.

Please align the diagnostic checks with pyproject.toml. Distinguish mandatory dependencies from optional RL and GUI dependencies, ensure the dependency inventory is complete, and provide installation instructions that actually resolve the reported missing packages. Avoid treating an intentionally omitted optional extra as a broken installation.

4. P1 — Project-root resolution must support the installation modes the CLI claims to support

get_project_root() relies on a fixed number of parent-directory traversals from __file__. This works for the expected source-tree layout but does not reliably identify a repository root for a conventional non-editable package installation.

In that scenario, Git/configuration checks can inspect the wrong location, and the I/O diagnostic may attempt to create artifacts/ and logs/ inside a Python installation directory.

Please distinguish a real checkout from an installed package, avoid assuming that package installation implies the repository is present, and handle unavailable project-level checks explicitly. Add coverage for invocation from another working directory and for the supported installation layouts.

5. P1 — The I/O permission probe can delete an existing file

check_io_permissions() uses a fixed .write_test filename, touches it, and then unlinks it. If that file already exists, the diagnostic can delete the user's existing file.

Please use a uniquely created temporary file with guaranteed cleanup and ensure the probe never removes a pre-existing file. Also verify that temporary filesystem changes are handled safely when a probe fails partway through execution.

6. P2 — YAML validation and automated tests need stronger coverage

check_yaml_configs() currently verifies YAML parsing, which is useful, but parsing alone does not establish that a configuration satisfies ARL's required schema or contains valid values. The current test coverage also relies heavily on mocked dependencies.

Please make the diagnostic's scope explicit and add focused regression or integration tests for malformed YAML, valid YAML with invalid configuration structure, optional dependency combinations, actual planner execution, and execution outside the repository root. Where full schema validation already exists in ARL, reuse it rather than introducing a second implementation.

Required verification

Before requesting another review, please:

  • Run ruff format on the changed Python files and verify formatting with ruff format --check.
  • Run ruff check src/ tests/ and mypy src/.
  • Run the full relevant test suite on Python 3.10, 3.11, and 3.12.
  • Verify adaptive-rl doctor in both text and --json modes, including correct exit codes and behavior when individual diagnostic checks encounter errors.
  • Confirm that the required CI checks pass on the latest commit.

The objective is to ensure that the doctor diagnoses the actual installed ARL environment, gives accurate remediation advice, and remains safe to run on a user's machine. Please address the blockers, add regression coverage for the identified cases, and report any intentionally unsupported scenarios explicitly.

I recommend keeping this PR in Changes Requested until the outstanding correctness and validation issues are resolved and the relevant CI checks are green. Thanks for continuing to work through the feedback.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Engineering] Add a full-system health check and adaptive-rl doctor command

3 participants