"Normal"

Front

Back

Quality Diversity (QD)
Differentiable Quality Diversity
(DQD)
Quality Diversity for Reinforcement Learning (QD-RL)
Approximating Gradients
for DQD in RL
Experiments
Results

Quality Diversity (QD)
Differentiable Quality Diversity
(DQD)
Quality Diversity for Reinforcement Learning (QD-RL)
Approximating Gradients
for DQD in RL
Experiments
Results

Front: 40%
Back: 50%

$\text{QD score} = \sum_{i=1}^M f(\bm{\phi}_i)$

Quality Diversity (QD)
Differentiable Quality Diversity
(DQD)
Quality Diversity for Reinforcement Learning (QD-RL)
Approximating Gradients
for DQD in RL
Experiments
Results

CMA-MEGA

Key Insight: Search by following objective and measure gradients.

Fontaine and Nikolaidis 2021, "Differentiable Quality Diversity." NeurIPS 2021 Oral.

CMA-MEGA

Quality Diversity (QD)
Differentiable Quality Diversity
(DQD)
Quality Diversity for Reinforcement Learning (QD-RL)
Approximating Gradients
for DQD in RL
Experiments
Results

Policy Gradient Assisted MAP-Elites
(PGA-MAP-Elites)

O. Nilsson and A. Cully 2021. "Policy Gradient Assisted MAP-Elites." GECCO 2021.

Quality Diversity (QD)
Differentiable Quality Diversity
(DQD)
Quality Diversity for Reinforcement Learning (QD-RL)
Approximating Gradients
for DQD in RL
Experiments
Results

Problem: Exact gradients are unavailable!

Solution: Approximate $\bm{\nabla} f$ and $\bm{\nabla m}$ .

DQD	QD-RL
Exact Gradients	Approximate Gradients
CMA-MEGA	CMA-MEGA with gradient approximations

Approximating $\bm{\nabla} f$

↑
Expected discounted return

Off-Policy Actor-Critic Method (TD3)

S. Fujimoto et al. 2018, "Addressing Function Approximation error in Actor-Critic Methods." ICML 2018.

Approximating $\bm{\nabla} \bm{m}$

↑
Black Box

	CMA-MEGA (ES)	CMA-MEGA (TD3, ES)
$\bm{\nabla} f$	ES	TD3
$\bm{\nabla} \bm{m}$	ES	ES

CMA-MEGA (ES) & CMA-MEGA (TD3, ES)

Quality Diversity (QD)
Differentiable Quality Diversity
(DQD)
Quality Diversity for Reinforcement Learning (QD-RL)
Approximating Gradients
for DQD in RL
Experiments
Results

QD Ant

QD Half-Cheetah

QD Hopper

QD Walker

Independent Variables

Algorithm:
CMA-MEGA (ES), CMA-MEGA (TD3, ES),
PGA-MAP-Elites, MAP-Elites, ME-ES
Environment:
QD Ant, QD Half-Cheetah, QD Hopper, QD Walker

Dependent Variable

QD Score

Quality Diversity (QD)
Differentiable Quality Diversity
(DQD)
Quality Diversity for Reinforcement Learning (QD-RL)
Approximating Gradients
for DQD in RL
Experiments
Results

	CMA-MEGA (ES)	CMA-MEGA (TD3, ES)
PGA-MAP-Elites	Comparable on 2/4	Comparable on 4/4
MAP-Elites	Outperforms on 4/4	Outperforms on 4/4
ME-ES	Outperforms on 3/4	Outperforms on 4/4

	PGA-MAP-Elites	CMA-MEGA (ES), CMA-MEGA (TD3, ES)
Objective Gradient Steps	5,000,000	5,000

Approximating Gradients for
Differentiable Quality Diversity in
Reinforcement Learning

GECCO 2022 — 11 July 2022

Bryon Tjanaka, Matthew C. Fontaine,
Julian Togelius, Stefanos Nikolaidis

dqd-rl.github.io

Quality Diversity (QD)
Differentiable Quality Diversity
(DQD)
Quality Diversity for Reinforcement Learning (QD-RL)
Approximating Gradients
for DQD in RL
Experiments
Results

Supplemental

Website
DQD Benchmark
DQD StyleGAN+CLIP
OpenAI-ES
TD3
Environment Details
Pseudocode
Algorithm Comparison
Final Metrics
Additional QD Archives

DQD Benchmark

$sphere(\bm{\phi}) = \sum_{i=1}^n (\bm{\phi}_i-2.048)^2$ $clip(\bm{\phi}_i) = \begin{cases} \bm{\phi}_i & \text{if } -5.12 \le \bm{\phi}_i \le 5.12 \\ 5.12/\bm{\phi_i} & \text{otherwise} \end{cases}$ $\bm{m}(\bm{\phi}) = \left(\sum_{i=1}^{\lfloor\frac{n}{2}\rfloor} clip(\bm{\phi}_i), \sum_{i=\lfloor\frac{n}{2}\rfloor+1}^n clip(\bm{\phi}_i) \right)$

Fontaine et al. 2020, "Covariance Matrix Adaptation for the Rapid Illumination of Behavior Space." GECCO 2020.

DQD StyleGAN+CLIP

Fontaine and Nikolaidis 2021, "Differentiable Quality Diversity." NeurIPS 2021 Oral.

OpenAI-ES

$\bm{\nabla} f(\bm{\phi}) \approx \frac{1}{\lambda_{es}\sigma} \sum_{i=1}^{\lambda_{es}} f(\bm{\phi} + \sigma \bm{\epsilon}_i) \bm{\epsilon}_i$

TD3 Overview

CVT Archive

Vassiliades et al. 2018. "Using Centroidal Voronoi Tessellations to Scale Up the Multidimensional Archive of Phenotypic Elites Algorithm." IEEE Transactions on Evolutionary Computation 2018.

Sliding Boundaries Archive

Fontaine et al. 2019. "Mapping Hearthstone Deck Spaces through MAP-Elites with Sliding Boundaries." GECCO 2019.

Back to Supplemental

Approximating Gradients forDifferentiable Quality Diversity inReinforcement Learning

Approximating Gradients for
Differentiable Quality Diversity in
Reinforcement Learning