Deep Learning / 07
Optimization
Explore gradient descent, adaptive optimizers, learning-rate schedules, and gradient clipping.
This is a knowledge index. Published titles open their articles; the remaining entries outline planned coverage.
Subtopics
- 07.01
Optimization in Neural Networks
Planned guide: Optimization in Neural Networks. Focus on intuition, assumptions, and a worked example.
- 07.02
Gradient Descent
Planned guide: Gradient Descent. Focus on intuition, assumptions, and a worked example.
- 07.03
Batch Gradient Descent
Planned guide: Batch Gradient Descent. Focus on intuition, assumptions, and a worked example.
- 07.04
Stochastic Gradient Descent
Planned guide: Stochastic Gradient Descent. Focus on intuition, assumptions, and a worked example.
- 07.05
Mini-Batch Gradient Descent
Planned guide: Mini-Batch Gradient Descent. Focus on intuition, assumptions, and a worked example.
- 07.06
Momentum
Planned guide: Momentum. Focus on intuition, assumptions, and a worked example.
- 07.07
Nesterov Momentum
Planned guide: Nesterov Momentum. Focus on intuition, assumptions, and a worked example.
- 07.08
AdaGrad
Planned guide: AdaGrad. Focus on intuition, assumptions, and a worked example.
- 07.09
RMSProp
Planned guide: RMSProp. Focus on intuition, assumptions, and a worked example.
- 07.10
Adam
Planned guide: Adam. Focus on intuition, assumptions, and a worked example.
- 07.11
AdamW
Planned guide: AdamW. Focus on intuition, assumptions, and a worked example.
- 07.12
Learning Rate
Planned guide: Learning Rate. Focus on intuition, assumptions, and a worked example.
- 07.13
Learning Rate Scheduling
Planned guide: Learning Rate Scheduling. Focus on intuition, assumptions, and a worked example.
- 07.14
Warmup
Planned guide: Warmup. Focus on intuition, assumptions, and a worked example.
- 07.15
Cosine Decay
Planned guide: Cosine Decay. Focus on intuition, assumptions, and a worked example.
- 07.16
Weight Decay
Planned guide: Weight Decay. Focus on intuition, assumptions, and a worked example.
- 07.17
Gradient Clipping
Planned guide: Gradient Clipping. Focus on intuition, assumptions, and a worked example.
- 07.18
Optimizer Selection
Planned guide: Optimizer Selection. Focus on intuition, assumptions, and a worked example.