Learning robust behaviors?

Quality diversity optimization can help.

1. 

   Quality Diversity (QD)

2. 

   Differentiable Quality Diversity
   (DQD)

3. 

   Quality Diversity for Reinforcement Learning (QD-RL)

4. 

   Approximating Gradients
   for DQD in RL

5. 

   Experiments

6. 

   Results

1. 

   Quality Diversity (QD)

2. 

   Differentiable Quality Diversity
   (DQD)
3. 

   Quality Diversity for Reinforcement Learning (QD-RL)
4. 

   Approximating Gradients
   for DQD in RL
5. 

   Experiments
6. 

   Results

1. 

   Quality Diversity (QD)
2. 

   Differentiable Quality Diversity
   (DQD)

3. 

   Quality Diversity for Reinforcement Learning (QD-RL)
4. 

   Approximating Gradients
   for DQD in RL
5. 

   Experiments
6. 

   Results

CMA-MEGA

Key Insight: Search by following objective and measure gradients.

Fontaine and Nikolaidis 2021, "Differentiable Quality Diversity." NeurIPS 2021 Oral.

CMA-MEGA

CMA-MEGA

CMA-MEGA

1. 

   Quality Diversity (QD)
2. 

   Differentiable Quality Diversity
   (DQD)
3. 

   Quality Diversity for Reinforcement Learning (QD-RL)

4. 

   Approximating Gradients
   for DQD in RL
5. 

   Experiments
6. 

   Results

Reinforcement Learning

Policy

$$\pi_{\bm{\phi}}(a | s)$$

Expected Discounted Return

$$f(\bm{\phi}) = \mathbb{E}_{\xi\sim p_{\bm{\phi}}}\left[\sum_{t=0}^T\gamma^t r(s_t,a_t) \right]$$

1. 

   Quality Diversity (QD)
2. 

   Differentiable Quality Diversity
   (DQD)
3. 

   Quality Diversity for Reinforcement Learning (QD-RL)
4. 

   Approximating Gradients
   for DQD in RL

5. 

   Experiments
6. 

   Results

Problem: Environments are non-differentiable!

Solution: Approximate $\bm{\nabla} f$ and $\bm{\nabla m}$.

Off-Policy Actor-Critic Method (TD3)

Evolution Strategy (OpenAI-ES)

CMA-MEGA (ES) & CMA-MEGA (TD3, ES)

CMA-MEGA (ES) & CMA-MEGA (TD3, ES)

CMA-MEGA (ES) & CMA-MEGA (TD3, ES)

CMA-MEGA (ES) & CMA-MEGA (TD3, ES)

CMA-MEGA (ES) & CMA-MEGA (TD3, ES)

CMA-MEGA (ES) & CMA-MEGA (TD3, ES)

1. 

   Quality Diversity (QD)
2. 

   Differentiable Quality Diversity
   (DQD)
3. 

   Quality Diversity for Reinforcement Learning (QD-RL)
4. 

   Approximating Gradients
   for DQD in RL
5. 

   Experiments

6. 

   Results

Parameters

• 1M evaluations
• Archive of 1000 cells
• 3-layer policy

1. 

   Quality Diversity (QD)
2. 

   Differentiable Quality Diversity
   (DQD)
3. 

   Quality Diversity for Reinforcement Learning (QD-RL)
4. 

   Approximating Gradients
   for DQD in RL
5. 

   Experiments
6. 

   Results

QD Ant

QD Half-Cheetah

QD Hopper

QD Walker

Future Directions

Future Directions

Additional Projects

Lunar Lander Tutorial

Potential Future Projects

Learning collaborative strategies.

QD-RL for real-world
robots.

Enhance and publish
pyribs.

Supplemental

• Website
• DQD Benchmark
• DQD StyleGAN+CLIP
• OpenAI-ES
• Environment Details
• Pseudocode
• Algorithm Comparison
• Final Metrics
• Additional QD Archives

DQD StyleGAN+CLIP

Fontaine and Nikolaidis 2021, "Differentiable Quality Diversity." NeurIPS 2021 Oral.

OpenAI-ES

$$\bm{\nabla} f(\bm{\phi}) \approx \frac{1}{\lambda_{es}\sigma} \sum_{i=1}^{\lambda_{es}} f(\bm{\phi} + \sigma \bm{\epsilon}_i) \bm{\epsilon}_i$$