"Normal"

Front

Back

Quality Diversity (QD)
Differentiable Quality Diversity
(DQD)
Quality Diversity for Reinforcement Learning (QD-RL)
Approximating Gradients
for DQD in RL
Experiments
Results

Quality Diversity (QD)
Differentiable Quality Diversity
(DQD)
Quality Diversity for Reinforcement Learning (QD-RL)
Approximating Gradients
for DQD in RL
Experiments
Results

Performance: 2,300
Front: 40%
Back: 50%

QD Objective

For every output $x$ of the measure function $\bm{m}$ , find $\bm{\phi}$ such that $\bm{m}(\bm{\phi}) = x$ , and $f(\bm{\phi})$ is maximized.

Quality Diversity (QD)
Differentiable Quality Diversity
(DQD)
Quality Diversity for Reinforcement Learning (QD-RL)
Approximating Gradients
for DQD in RL
Experiments
Results

CMA-MEGA

Key Insight: Search by following objective and measure gradients.

Fontaine and Nikolaidis 2021, "Differentiable Quality Diversity." NeurIPS 2021 Oral.

CMA-MEGA

Quality Diversity (QD)
Differentiable Quality Diversity
(DQD)
Quality Diversity for Reinforcement Learning (QD-RL)
Approximating Gradients
for DQD in RL
Experiments
Results

Policy Gradient Assisted MAP-Elites
(PGA-MAP-Elites)

O. Nilsson and A. Cully 2021. "Policy Gradient Assisted MAP-Elites." GECCO 2021.

Quality Diversity (QD)
Differentiable Quality Diversity
(DQD)
Quality Diversity for Reinforcement Learning (QD-RL)
Approximating Gradients
for DQD in RL
Experiments
Results

Hypothesis:

Since CMA-MEGA performs well in DQD domains,
it will outperform existing QD-RL algorithms
(i.e. PGA-MAP-Elites and MAP-Elites).

Problem: Environments are non-differentiable!

Solution: Approximate $\bm{\nabla} f$ and $\bm{\nabla m}$ .

DQD	QD-RL
Exact Gradients	Approximate Gradients
CMA-MEGA	CMA-MEGA with gradient approximations

Approximating $\bm{\nabla} f$

↑
Expected discounted return

Off-Policy Actor-Critic Method (TD3)

S. Fujimoto et al. 2018, "Addressing Function Approximation error in Actor-Critic Methods." ICML 2018.

Approximating $\bm{\nabla} \bm{m}$

↑
Black Box

	CMA-MEGA (ES)	CMA-MEGA (TD3, ES)
$\bm{\nabla} f$	ES	TD3
$\bm{\nabla} \bm{m}$	ES	ES

CMA-MEGA (ES) & CMA-MEGA (TD3, ES)

Quality Diversity (QD)
Differentiable Quality Diversity
(DQD)
Quality Diversity for Reinforcement Learning (QD-RL)
Approximating Gradients
for DQD in RL
Experiments
Results

QD Ant

QD Half-Cheetah

QD Hopper

QD Walker

Independent Variables

Algorithm:
CMA-MEGA (ES), CMA-MEGA (TD3, ES),
PGA-MAP-Elites, MAP-Elites, ME-ES
Environment:
QD Ant, QD Half-Cheetah, QD Hopper, QD Walker

Dependent Variable

QD Score

Quality Diversity (QD)
Differentiable Quality Diversity
(DQD)
Quality Diversity for Reinforcement Learning (QD-RL)
Approximating Gradients
for DQD in RL
Experiments
Results

	CMA-MEGA (ES)	CMA-MEGA (TD3, ES)
PGA-MAP-Elites	Comparable on 2/4	Comparable on 4/4
MAP-Elites	Outperforms on 4/4	Outperforms on 4/4
ME-ES	Outperforms on 3/4	Outperforms on 4/4

	PGA-MAP-Elites	CMA-MEGA (ES), CMA-MEGA (TD3, ES)
Objective Gradient Steps	5,000,000	5,000

Future Directions

Differentiable Quality Diversity for
Reinforcement Learning by
Approximating Gradients

ALOE 2022 Workshop, 29 April 2022

Bryon Tjanaka, Matthew C. Fontaine,
Julian Togelius, Stefanos Nikolaidis

dqd-rl.github.io

Quality Diversity (QD)
Differentiable Quality Diversity
(DQD)
Quality Diversity for Reinforcement Learning (QD-RL)
Approximating Gradients
for DQD in RL
Experiments
Results

Differentiable Quality Diversity forReinforcement Learning byApproximating Gradients

Differentiable Quality Diversity for
Reinforcement Learning by
Approximating Gradients