Repository navigation
Conversation
|
Hi @ashishsinghbora waiting for the review for this pr |
ashishsinghbora
left a comment
There was a problem hiding this comment.
@LazyKaddu I went through this PR in detail. The integrated smoke test is a good addition, but I don't think this is merge-ready yet. The main concern is that several of the claimed “protocol invariants” are testing implementation behavior rather than proving that the research protocol is actually being executed correctly.
The biggest issue is the shock-window implementation. "shock_returns" are generated using the same environment and same policy as the pre-shift evaluation, but there is no actual shift/disturbance introduced between the baseline and shock phases. That means the test is calling this a “Shock Window” without actually testing a distribution/environment shift. For an Adaptive RL protocol, this is a critical semantic gap.
The Fixed vs Adaptive comparison is also weaker than it appears. Both arms are initialized from the same frozen weights, which is correct, but the Fixed arm is never actually evaluated as an independent experimental trajectory during the shock window. Instead, both "fixed_post_returns" and "adaptive_post_returns" are initialized with "list(shock_returns)". This makes the assertion that both arms have identical initial shock returns effectively tautological rather than an independently verified invariant.
The checksum validation is also not validating an artifact checksum. The manifest computes a SHA-256 hash over the manifest contents and then stores that hash inside the same manifest. The test only checks that the resulting value has 64 characters. It never recomputes the checksum from persisted output or verifies an actual artifact. So the test currently proves “a SHA-256-looking string was generated,” not artifact integrity.
Another concern is the policy mutation assertion:
"adaptive_l2_delta > 0.0"
This proves that some weights changed, but it does not prove that the online adaptation behaved correctly. Conversely, "fixed_l2_delta == 0.0" is a useful invariant, but it should ideally be enforced against the actual frozen policy used for evaluation, not only against the locally copied SB3 model state.
The runner also creates multiple PPO instances sharing the same "env" object. That makes the experimental lifecycle harder to reason about and can introduce state contamination between training and evaluation. For a research protocol, the environment lifecycle should be explicit: separate training/evaluation environments or controlled resets with clearly defined shift state.
There is also a significant determinism problem. The PR describes the test as deterministic, but the runner performs PPO training without explicitly controlling seeds for the environment, model, NumPy/PyTorch, and action/evaluation state. A short training run can pass today and fail later due to stochastic initialization or optimization behavior. “Fast” and “deterministic” are different properties; this test needs to guarantee the latter.
The test itself also checks several things that are mostly structural rather than semantic. For example, checking "len(fingerprint) == 64" does not establish that the fingerprint corresponds to the expected policy, and checking that recovery values are "int" does not establish that "T_H" was computed correctly.
Finally, the runner is doing a lot of protocol orchestration directly inside a new experimental module. Before merging this, I'd want the protocol phases and their invariants to be clearly separated so that the smoke test exercises the same production protocol implementation rather than becoming a second implementation of the protocol.
My recommendation is Request changes.
The highest-priority fixes are:
- Introduce an actual controlled environment shift/disturbance during the shock window.
- Execute Fixed and Adaptive arms through independently controlled evaluation trajectories.
- Make the test genuinely deterministic with explicit seeds.
- Validate persisted artifact/manifest checksums by recomputation.
- Strengthen the assertions around recovery correctness instead of only checking types/presence.
- Avoid shared environment state between training and independent evaluation phases.
The smoke test is a good direction, but right now it can pass while important parts of the actual Adaptive RL research protocol are not being tested.
|
@ashishsinghbora can you now check the revised version of the smoke test that i have send |
|
Hi @LazyKaddu Thanks for the follow-up revisions. The separate fixed/adaptive environments and explicit shift configuration are steps in the right direction. However, after reviewing the latest changes, I found several outstanding correctness issues that need to be resolved before this PR is ready to merge. Remaining issues1. P1 — Duplicate
Please remove the naming collision by either giving the smoke-test helper a distinct name or integrating the workflow into the existing production runner. Avoid maintaining two competing implementations of the same protocol. 2. P1 — Manifest checksum generation and verification are inconsistent The runner calculates the checksum from a serialized manifest containing Please establish one canonical serialization procedure and use it for both checksum generation and verification. Add a regression test that passes against the persisted manifest, including after serialization and deserialization. 3. P1 — Artifact integrity is not fully verified The current checksum covers manifest metadata, not the actual bytes of the persisted Please record the SHA-256 digest of the persisted checkpoint and independently recompute it during verification. Keep the manifest checksum and artifact checksum as clearly defined, separate checks. 4. P1 — Recovery assertions are self-referential The test invokes Please add deterministic fixtures with independently calculated expected recovery times. Cover immediate recovery, delayed recovery, failure to recover within the horizon, and persistence/window boundary cases. 5. P2 — Smoke-test return values and seed handling need clearer semantics
Please either use the production evaluation routine or explicitly document and test this limited smoke-test behavior. Use a reproducible seed schedule appropriate to the intended evaluation coverage, and assert that the nominal and shifted environments differ in the intended, effective dynamics. 6. P2 — Documentation and validation coverage do not match
Required verification before requesting another reviewPlease address the constructor/name collision first, then fix the checksum contract and strengthen the remaining assertions. Run at least:
The current Checks page reports a workflow run completed with no jobs, so it does not establish that the new smoke test has actually passed. Please ensure the applicable CI jobs execute and provide their results. The feature is a useful addition, but a smoke test must verify the intended workflow rather than merely contain assertions for it. I recommend keeping this PR in Changes Requested until the blocking issues are fixed, the regression tests pass, and the CI results are verifiable. Thanks for addressing these points. |
Summary of Changes
shift_runner.py: Added the AdaptiveShiftRunner which encapsulates the complete RL research workflow (Training, Freezing, Evaluation, Shock Window, Forking into Fixed vs. Adaptive Arms, Online Adaptation Blocks, and Recovery Analysis).
test_research_protocol_smoke.py: Created a fast, deterministic pytest that executes this runner with miniature configuration limits (e.g., training_steps=128, bounded evaluation steps) to ensure it can run rapidly in CI pipelines.
Missing Environment Fix: Changed the target environment from gridworld (which wasn't registered in the repository or Gymnasium) to the supported drone environment to fix the RegistryError.
Policy Attribute Fix: Resolved an AttributeError in shift_runner.py where the code was incorrectly trying to access algo.policy.state_dict(). We corrected it to access the underlying Stable-Baselines3 model properly (algo.model.policy.state_dict()).
Discovered and fixed a tricky Windows-specific bug in tests/test_cli.py. The tmp_path paths were generating Windows backslashes (C:\Users...), causing the YAML parser to crash by mistaking \U for a malformed Unicode escape sequence.
We fixed this by appending .as_posix() to the paths, ensuring they always use forward slashes safely across all operating systems.
Verification
pytest tests/passesruff check src/ tests/passesmypy src/passes