Gradient descent

The Foggy Mountain Analogy

To understand how gradient descent works, imagine you are standing on a mountain range in thick, heavy fog. Your goal is to reach the lowest point of the valley, but you cannot see the bottom. Because you are blind to the landscape, you rely on your feet to feel the slope of the ground immediately around you. You identify the direction that slopes downward most steeply and take a small step in that direction. You repeat this process — feeling the slope, taking a step, and adjusting your path — until you eventually reach the bottom of the valley. The “lowest point” is the minimum error, and your “steps” are the adjustments made to the model’s parameters.

How It Works

At its core, gradient descent uses an update rule to refine the model: θ = θ − η · ∇L(θ). Here, θ represents the model parameters (the internal settings the model is trying to perfect), η (eta) is the learning rate, and ∇L(θ) is the gradient of the loss function. The gradient is a mathematical vector that points in the direction of the steepest increase; by subtracting it, the algorithm moves in the opposite direction — the direction of steepest descent.

The learning rate is a critical setting that controls step size. Too small causes slow convergence. Too large can overshoot the minimum entirely. Finding the right learning rate is one of the first challenges in training any model.

Types of Gradient Descent

Depending on how much data the algorithm examines before updating, gradient descent comes in three flavors:

• Batch Gradient Descent: Computes the gradient over the entire dataset before each update. Stable but slow for large data — you have to look at everything before taking a single step.
• Stochastic Gradient Descent (SGD): Updates parameters using one randomly selected sample at a time. Much faster per step, but the path toward the minimum zigzags noisily.
• Mini-batch Gradient Descent: Uses small batches of data (typically 32–256 samples) per update. This is the standard for training deep learning models — it balances the stability of batch processing with the speed of stochastic methods.

Modern Optimizer Variants

Basic gradient descent works, but researchers have built smarter versions that converge faster and handle tricky landscapes better:

• Momentum: Adds a fraction of the previous update to the current one, helping the algorithm “roll” through flat areas and dampen oscillations in narrow valleys.
• AdaGrad: Adapts the learning rate for each parameter individually — giving smaller updates to frequently updated parameters and larger updates to rare ones. Good for sparse data, but the learning rate can decay too aggressively.
• RMSprop: Uses an exponentially decaying average of squared gradients to keep learning rates balanced, fixing AdaGrad’s aggressive decay problem.
• Adam (Adaptive Moment Estimation): Introduced by Kingma & Ba in 2014 (arXiv:1412.6980, published at ICLR 2015). Combines Momentum (first moment) and RMSprop (second moment) with bias correction. It is the default optimizer in most deep learning frameworks.
• AdamW: As of 2024–2025, Adam with decoupled weight decay (AdamW) is the most widely used optimizer for training large neural networks and transformer-based models, often paired with learning rate warmup and cosine decay schedules.

Agent Relevance

Gradient descent is the engine behind all neural network training. Large Language Models (LLMs), computer vision systems, and reinforcement learning agents all rely on gradient descent to adjust their weights during training. When an LLM generates text, when a self-driving car identifies a pedestrian, or when an RL agent learns to play a game — gradient descent is the optimization process that made the learning possible. Without it, models have no way to improve.

Limitations and Challenges

• Local minima and saddle points: In non-convex problems (the norm in deep learning), the algorithm can get stuck in valleys that aren’t the deepest, or on flat plateaus where the gradient is near zero.
• Vanishing and exploding gradients: In deep networks, gradients can shrink to near-zero (vanishing) or grow uncontrollably (exploding) as they propagate through layers, making training unstable.
• Learning rate scheduling: Modern practice uses warmup, decay, or cosine schedules to adapt the learning rate during training rather than keeping it fixed.
• Problem-dependent choice: While Adam is the default, SGD with momentum often generalizes better for certain tasks. There is no single best optimizer.

Reader Questions

Q: Is Adam always the best optimizer?
A: No. Adam is the default in most frameworks, but SGD with momentum often generalizes better in practice for certain tasks — particularly in computer vision. The best choice is problem-dependent.

Q: What does “convergence” mean in gradient descent?
A: Convergence is the point where the model’s error stops decreasing meaningfully — the algorithm has found (or is very close to) the lowest point it can reach. Plotting the loss over iterations is the standard way to diagnose convergence, as taught in Stanford CS229.

Q: How is gradient descent different from normal equations?
A: Normal equations provide a direct, analytical solution but become computationally impossible as the number of features grows. Gradient descent is iterative and scales to the massive datasets used in modern AI.

Q: Where can I learn more about the math?
A: Sebastian Ruder’s 2016 overview remains the definitive technical guide to gradient descent variants. For the foundational course material, Stanford CS229 (main notes) is the standard reference.