Training neural networks is a tricky business. One common method is Stochastic Gradient Descent, or SGD, which updates model parameters step by step to minimize errors. But SGD has a hidden challenge: it treats all parameters the same, regardless of how often they get updated. This can cause problems when some features appear frequently in data, while others are rare but important.

Think of training a language model. Common words like “the” pop up all the time, so the parameters linked to those words get updated constantly. Rare words like “thalweg” might show up once in a blue moon. SGD uses the same learning rate for all parameters, so the common ones quickly settle into good values. Meanwhile, rare tokens barely move from their starting points, slowing down the learning of these less frequent features.

Why Does SGD Struggle with Frequency Imbalance?

SGD’s core idea is simple: calculate gradients based on random batches of data and move parameters a bit in the direction that reduces error. However, because gradients only appear when a feature is present in the batch, rare tokens see very few updates. This creates a bias. Frequent tokens dominate the learning process, while rare tokens lag behind.

This imbalance isn’t just about speed. It affects how well the model understands less common but meaningful parts of the data. Over time, the model becomes biased toward frequent patterns, potentially missing subtle but important signals.

How Momentum Smooths Out Zigzags in Gradient Descent

Another problem with basic gradient descent is zigzagging during updates. When the loss surface is uneven, with some directions steep and others flat, SGD can bounce back and forth instead of moving smoothly. This slows down convergence and wastes computing power.

Momentum helps fix this by remembering past gradients. Instead of reacting purely to the current slope, momentum accumulates a velocity vector that smooths out oscillations. This lets the algorithm take bigger, more confident steps in consistent directions while damping out back-and-forth motion in steep directions. As a result, momentum speeds up training and stabilizes it.

However, momentum alone doesn’t solve the frequency imbalance issue. It mainly addresses the shape of the loss landscape, not how often parameters get updated.

Adam: The Adaptive Optimizer That Levels the Playing Field

Adam combines the benefits of momentum with another clever trick: adaptive learning rates. It tracks the history of gradients for each parameter separately. This lets it scale updates based on how much reliable gradient information each parameter has seen.

For parameters tied to rare tokens, Adam effectively raises their learning rates. This compensates for the infrequent updates, allowing these weights to catch up faster. For frequently updated parameters, Adam reduces the learning rate to avoid overshooting.

In practice, this means Adam can learn well even when data is heavily imbalanced. Rare features no longer stay close to random initial values. Instead, they receive proportionally larger updates and converge more quickly. This adaptive behavior makes Adam a staple in training large language models and other deep learning systems.

Putting It All Together: Why Adaptive Methods Matter

SGD, momentum, and adaptive optimizers like Adam each play roles in neural network training. SGD provides a simple foundation. Momentum smooths the path through tricky optimization landscapes. Adam adds an adaptive layer that compensates for hidden biases like token frequency imbalance.

Understanding these differences helps explain why Adam often outperforms vanilla SGD in real-world scenarios. It isn’t just about moving faster or smoother — it’s about balancing updates fairly across all parameters, no matter how often they appear in the data.

For anyone working with models that handle diverse or uneven data, choosing the right optimizer can dramatically impact training speed and final performance. Adaptive methods like Adam have become essential tools for tackling the complex, noisy landscapes of modern machine learning.