[vishwagw]

Description

Closes #XXX

Adds examples/dqn_training.ipynb. It trains a DQN agent on a BSK-RL environment in plain PyTorch through the standard Gymnasium API, with no RL library. All existing training examples use RLlib PPO, so this is the first example of a value-based, off-policy method, and the first that trains a BSK-RL env with a hand-written loop.

The key point is the time-discounted TD target for BSK-RL's semi-MDP steps:

target = rewards + self.gamma**d_ts * (1.0 - terminated) * next_q

d_ts comes from info["d_ts"] and gamma is per second. This matches the convention in TimeDiscountedGAEPPOTorchLearner and the time-discounted GAE example.

Changes

• examples/dqn_training.ipynb: new example
  • Same scanning/downlink satellite and environment as rllib_training, so the two can be compared directly
  • Shows info["d_ts"] for each action
  • Q-network, a replay buffer that stores d_ts, and a DQN agent (Huber loss, gradient clipping, Polyak target updates, 4 gradient updates per environment step)
  • A 20k-step training run sized for the docs build, a learning-curve plot, and a greedy-vs-random evaluation on held-out seeds
• examples/_default.rst: adds the example under Training
• docs/source/release_notes.rst: release note entry

No new dependencies: uses torch (in the rllib extra, which the docs CI already installs) and matplotlib (pulled in by bsk).

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

How should this pull request be reviewed?

• By commit
• All changes at once

How Has This Been Tested?

• Executed the notebook end to end with jupyter nbconvert --execute. It runs in about 10.5 minutes on a 2-vCPU Linux machine. Output: DQN greedy return +0.203 with 10% failures, vs random −0.200 with 50% failures.

• Checked robustness across seeds with the same hyperparameters (20k steps, 10 held-out evaluation episodes each):

  Seed	DQN greedy return	DQN failure rate	Random return	Random failure rate
  0	+0.203 ± 0.331	10%	−0.648 ± 0.345	90%
  1	+0.100 ± 0.046	0%	−0.521 ± 0.428	80%
  2	+0.237 ± 0.093	0%	−0.352 ± 0.560	60%

  Earlier settings were less reliable across seeds: 1 update per step, higher learning rate, larger per-second gamma, and Double DQN. That's why the example uses 4 updates per step with lr=2.5e-4.

• Caveat: my test environment couldn't download de430.bsp, so these runs used DE421 as a local stand-in for the planetary ephemeris. With the real file, the exact numbers may differ slightly.

• ruff check, black --check and isort --check pass on the notebook.

• No library code changed, so no unit or integration tests were added. The docs build executes the notebook, which covers it.

Future Work

• The notebook adds about 10 minutes