Scaling Covariance Matrix Adaptation
MAP-Annealing to High-Dimensional
Controllers
Deep RL Workshop at NeurIPS 2022 — 9 December 2022
Bryon Tjanaka
University of Southern California
tjanaka@usc.edu
Matthew C. Fontaine
University of Southern California
mfontain@usc.edu
Aniruddha Kalkar
University of Southern California
kalkar@usc.edu
Stefanos Nikolaidis
University of Southern California
nikolaid@usc.edu
Website
scalingcmamae.github.io
Quality Diversity for Reinforcement Learning (QD-RL)
"Normal"
Front
Back
Covariance Matrix Adaptation MAP-Annealing (CMA-MAE)
Fontaine and Nikolaidis 2022, "Covariance Matrix Adaptation MAP-Annealing." Preprint.
| Low-dimensional QD | (High-dimensional) QD-RL | |
|---|---|---|
| CMA-MAE | ✓ | ⨯ |
CMA-MAE
CMA-ES
Key Insight: Scale CMA-MAE to QD-RL by replacing CMA-ES with scalable algorithms.
Scalable CMA-MAE
Scalable CMA-ES
Experiments
QD Ant
QD Half-Cheetah
QD Hopper
QD Walker
| Deep RL-based QD | Scalable CMA-MAE |
|---|---|
✓ SOTA | ✓ SOTA on 3/4 tasks |
⨯ Requires GPU | ✓ Runs on CPU |
⨯ Neural network training | ✓ No training |
⨯ Many hyperparameters | ✓ Minimal hyperparameters |
Scaling Covariance Matrix Adaptation
MAP-Annealing to High-Dimensional
Controllers
Deep RL Workshop at NeurIPS 2022 — 9 December 2022
Bryon Tjanaka
University of Southern California
tjanaka@usc.edu
Matthew C. Fontaine
University of Southern California
mfontain@usc.edu
Aniruddha Kalkar
University of Southern California
kalkar@usc.edu
Stefanos Nikolaidis
University of Southern California
nikolaid@usc.edu
Website
scalingcmamae.github.io