Hindsight policy gradients

This software supplements the paper "Hindsight policy gradients".

The implementation focuses on clarity and flexibility rather than computational efficiency.

Examples

Training an agent in a bit flipping environment (k = 8) using a weighted per-decision hindsight policy gradient estimator (HPG):

python3 hpg/scripts/run.py hpg/examples/flipbit8/flipbit8_bs2_hpg

Training an agent in a bit flipping environment (k = 8) using a goal-conditional policy gradient estimator (GCPG):

python3 hpg/scripts/run.py hpg/examples/flipbit8/flipbit8_bs2_gcpg

Combining the corresponding results into a single plot (see folder "results/flipbit8_bs2"):

mkdir -p results/flipbit8_bs2
cp -r hpg/examples/flipbit8/flipbit8_bs2_hpg hpg/examples/flipbit8/flipbit8_bs2_gcpg results/flipbit8_bs2
python3 hpg/scripts/analysis.py results/flipbit8_bs2

Dependencies

• matplotlib (2.1.1)
• numpy (1.17.2)
• pandas (0.23.4)
• scipy (1.3.0)
• seaborn (0.9.0)
• tensorflow (1.12.0)
• gym (0.13.1)
• atari-py (?, https://github.com/openai/atari-py)
• mujoco-py (2.0.2.6, https://github.com/openai/mujoco-py)
• ray (0.7.2)