Skip to content

Implement SAC online adaptation under distribution shift (#273) - #307

Open
AryanXCode646 wants to merge 2 commits into
StellarResearch:mainfrom
AryanXCode646:fix/issue-273-sac-online-adaptation
Open

AryanXCode646 wants to merge 2 commits into
StellarResearch:mainfrom
AryanXCode646:fix/issue-273-sac-online-adaptation

Conversation

@AryanXCode646

Copy link
Copy Markdown
Contributor

Summary

This PR implements Soft Actor-Critic (SAC) online adaptation under distribution shift, achieving parity with the existing PPO adaptation workflow.

Closes #273

Key Changes

1. SAC Online Adaptation

  • Added sac_shift_runner.py for the SAC adaptation experiment.
  • Implemented nominal training, checkpoint freezing, and independent Fixed and Adaptive arms.
  • Preserved the existing distribution-shift experiment protocol.

2. Replay Buffer Isolation

  • Ensured adaptation uses post-shift transitions only.
  • Prevented nominal training transitions from entering the adaptation buffer.
  • Added episode-indexed transition tracking to prevent future-data leakage.

3. Adaptation Workflow

  • Implemented ten update blocks: B5–B14.
  • Added deterministic update-seed derivation.
  • Preserved the Fixed arm's zero-update invariant.
  • Integrated adaptive SAC actor and critic updates.

4. Audit and Recovery

  • Added parameter-delta tracking and SHA-256 fingerprints.
  • Added target-network and entropy-temperature audit fields.
  • Integrated existing recovery calculations.
  • Added JSON/CSV reporting and experiment manifest support.

5. Statistical Error Handling

  • Fixed statistical-summary handling for insufficient replicate counts.
  • Zero replicates: insufficient_data.
  • One replicate: not_estimable.
  • Two to seven replicates: insufficient_replicates.
  • Eight or more replicates: formal statistical analysis.
  • Preserved the preregistered MIN_VALID_N = 8 threshold.

6. Tests and Documentation

  • Added SAC adaptation and statistical edge-case tests.
  • Updated the research Treatment Card.
  • Updated supporting algorithm and experiment integrations.

Validation

Reported local validation:

  • pytest -q: 210 passed.
  • Statistical edge-case tests: 9 passed.
  • git diff --check: no reported errors.

Research Integrity

The implementation preserves the existing SAC loss formulation, research protocol constants, recovery methodology, post-shift replay isolation, and B5–B14 adaptation schedule.

No changes to the frozen minimum replicate threshold.

Scope

This PR addresses GitHub Issue #273 and adds SAC support for online adaptation under distribution shift.

Aryan Xcode and others added 2 commits October 25, 2026 01:56
… shift (Issue StellarResearch#273)

- Add sac_shift_runner.py with full protocol compliance (nominal training, checkpoint freeze, shock window, fork, B5-B14 update blocks, and recovery analysis)
- Guard optional heavy dependencies (torch, stable-baselines3) in ppo.py and sac.py with informative error messages
- Add test_sac_adaptation.py to verify protocol invariants with graceful skip when heavy dependencies are absent
- Update docs/research/TREATMENT_CARD.md with SAC online adaptation specifications, hyperparameters, and audit requirements

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Research] Implement SAC parity for online adaptation under distribution shift

1 participant