---
title: "Optimizers in Neural Networks"
author: "Sébastien De Greef"
format:
  revealjs: 
    theme: solarized
    navigation-mode: grid 
    controls-layout: bottom-right
    controls-tutorial: true
notebook-links: false
crossref:
  lof-title: "List of Figures"
number-sections: false
---

## Introduction to Optimizers

Optimizers are crucial for training neural networks by updating the network's weights based on the loss gradient. They impact the training speed, quality, and the model's final performance.

---

## Role of Optimizers

- **Function**: Minimize the loss function
- **Mechanism**: Iteratively adjust the weights
- **Impact**: Affect efficiency, accuracy, and model feasibility

---

## Gradient Descent

- **Usage**: Basic learning tasks, small datasets
- **Strengths**: Simple, easy to understand
- **Caveats**: Slow convergence, sensitive to learning rate settings

---

## Stochastic Gradient Descent (SGD)

- **Usage**: General learning tasks
- **Strengths**: Faster than batch gradient descent
- **Caveats**: Higher variance in updates

---

## Momentum

- **Usage**: Training deep networks
- **Strengths**: Accelerates SGD, dampens oscillations
- **Caveats**: Additional hyperparameter (momentum)

---

## Nesterov Accelerated Gradient (NAG)

- **Usage**: Large-scale neural networks
- **Strengths**: Faster convergence than Momentum
- **Caveats**: Can overshoot in noisy settings

---

## Adagrad

- **Usage**: Sparse data problems like NLP and image recognition
- **Strengths**: Adapts the learning rate to the parameters
- **Caveats**: Shrinking learning rate over time

---

## RMSprop

- **Usage**: Non-stationary objectives, training RNNs
- **Strengths**: Balances decreasing learning rates
- **Caveats**: Still requires learning rate setting

---

## Adam (Adaptive Moment Estimation)

- **Usage**: Broad range of deep learning tasks
- **Strengths**: Efficient, handles noisy/sparse gradients well
- **Caveats**: Complex hyperparameter tuning

---

## AdamW

- **Usage**: Regularization heavy tasks
- **Strengths**: Better generalization than Adam
- **Caveats**: Requires careful tuning of decay terms

---

## Conclusion

Choosing the right optimizer is crucial for training efficiency and model performance. 

Each optimizer has its strengths and is suited for specific types of tasks.


