The Fast Algorithm That Was 370 Times Slower

I built batch, mini-batch, and stochastic gradient descent from scratch in NumPy, on the same small linear regression problem, expecting the exercise to confirm what every course slide says: SGD is the efficient one. One sample at a time, minimal computation per step, the version that scales to datasets too big to fit in memory. I timed all three anyway, mostly as a formality. Batch gradient descent finished 30 epochs in three milliseconds. Stochastic gradient descent, doing what I’d been told was the “lightweight” version of the same job, took one point one seconds. Nearly four hundred times slower, on the exact same data, computing what is supposed to be a strictly smaller amount of arithmetic per step.

I checked my code twice, assuming I’d made an error. I hadn’t. What I’d actually done was measure two different things that “gradient descent” quietly collapses into one word: how much math an algorithm does, and how fast that math actually runs on real hardware. Those turned out to be almost unrelated questions, and the gap between them is where the rest of this experiment lives.

01 — THE RECKONING, PART ONEWhat “one sample at a time” actually costs

Batch gradient descent computes one gradient per epoch, across all 5,000 rows at once — a single matrix multiplication, handed to NumPy’s vectorized, compiled inner loop. Stochastic gradient descent computes 5,000 gradients per epoch, one per row, each one a small Python-level loop iteration wrapping a tiny NumPy operation. The “lightweight” version does less arithmetic and vastly more looping — and looping, in an interpreted language calling into a vectorized library, is exactly the operation that library was built to help you avoid.

FIG. 1 — Wall-clock time for 30 epochs on the same 5,000-row regression task. Bar length scaled to the slowest run (SGD). All three converge to a comparable final loss (0.25–0.32 MSE) — this is purely an execution-time gap.

Mini-batch gradient descent — 64 rows per step instead of 1 or 5,000 — split the difference and then some: seven times slower than batch, but 53 times faster than SGD, while landing on the lowest final loss of the three. It gets most of batch’s vectorization benefit and most of SGD’s per-step gradient noise, which turns out to help escape shallow local structure rather than hurt it. The textbook framing isn’t wrong about SGD’s role in genuinely enormous, streaming, can’t-fit-in-memory datasets. It’s incomplete for the much more common case: a dataset that fits in memory just fine, run on hardware and a language where the loop, not the arithmetic, is the expensive part.

02 — THE RECKONING, PART TWOWhere momentum earns its keep

The wall-clock surprise was about hardware. The next one was about geometry. I built a loss surface on purpose to be badly shaped — a bowl elongated 100-to-1 along one axis versus the other, the kind of ill-conditioning that shows up constantly in real models when features live on very different scales. Vanilla gradient descent needs a learning rate small enough not to blow up on the steep axis, and that same small rate starves progress on the shallow one. Momentum carries velocity from step to step, so progress the shallow axis makes early on doesn’t get thrown away every iteration — it accumulates.

At an identical, shared learning rate, vanilla gradient descent needed 206 steps to reach a target loss of 0.001. Momentum with a standard decay of 0.9 reached the same target in 70. Nothing about momentum changed the learning rate, the data, or the loss function — it changed what the optimizer remembered between one step and the next, and that memory is what let it stop re-litigating the same oscillation on every single iteration.

03 — THE RECKONING, PART THREEThe learning rate that broke one optimizer and not the other

I swept the same learning rate across vanilla gradient descent and Adam on the original regression task, looking for the exact point where one of them would fail. At a learning rate of 1.0, vanilla gradient descent’s loss diverged to infinity within the first few epochs — the update step overshot the minimum by so much that each successive gradient was larger than the last, a runaway feedback loop. Adam, at the identical learning rate of 1.0, finished with a final loss of 0.258 — indistinguishable from its performance at far more conservative rates.

The instability wasn’t the specific number 1.0 so much as what 1.0 meant differently to each optimizer. Vanilla gradient descent’s step size is the learning rate multiplied directly by the gradient — if the gradient is large, the step is large, with nothing to check it. Adam divides its step by a running estimate of that gradient’s own variance, which caps how far a single large gradient can push a parameter in one move. That’s real protection against one specific failure mode: an update so large it destabilizes the next update, and the next.

It is not protection against every failure mode, and the same sweep showed me the cost side of that trade plainly. At a cautious learning rate of 0.001, vanilla gradient descent reached a loss of 0.327. Adam, at that same cautious rate, reached 16.17 — dramatically worse. Adam’s normalized step is bounded to roughly the size of the learning rate itself, almost regardless of how large the true error is. Early in training, when the honest gradient is large and a big step is exactly what’s needed, Adam’s caution costs it speed that vanilla gradient descent, with nothing holding it back, was free to take.

04 — THE RECKONING, PART FOURWhere the gradient goes quiet

The last surface I tested wasn’t ill-conditioned, it was degenerate: a monkey saddle, f(x,y) = x³ − 3xy², with a critical point at the origin where the gradient vanishes not to a minimum or a maximum, but to a point every direction disagrees about. Starting an optimizer a hair’s width from that origin, the gradient near it is almost zero — not because the surface is flat there in any useful sense, but because the curvature itself has degenerated. Vanilla gradient descent, taking steps proportional to that near-zero gradient, needed 95 iterations just to move far enough away to register real progress. Momentum did it in 28. Adam did it in 15.

Momentum escapes because it’s still carrying velocity accumulated from whatever motion existed before the gradient went quiet — inertia doesn’t ask permission from the current gradient. Adam escapes faster still because it’s dividing by a second-moment estimate that has itself gone small near the saddle, which — perhaps counterintuitively — inflates the effective step rather than shrinking it. Two completely different mechanisms, both compensating for the same blind spot: an optimizer that only trusts the instantaneous gradient has nothing to fall back on exactly when that gradient stops being informative.

05 — THE RECURSIVE INSIGHTThe pattern under all four results

Looking at the four experiments together, I don’t see four separate lessons about four separate algorithms. I see one recurring shape: a fixed learning rate is a promise that the loss surface looks the same in every direction and at every moment, and every one of today’s failures happened at exactly the place that promise broke. SGD’s slowness was a promise about computation that broke against real hardware. The ill-conditioned bowl was a promise about direction that broke across two axes of different scale. The LR=1.0 divergence was a promise about magnitude that broke against a gradient bigger than expected. The saddle point was a promise about information that broke when the gradient had none to give.

Momentum and adaptive learning rates aren’t upgrades to gradient descent. They’re both ways of remembering something about the trajectory, because the instantaneous gradient — the only thing vanilla gradient descent ever looks at — keeps turning out not to be enough.

That reframes what “the optimizer” is actually solving. It was never just “find the direction that decreases loss.” It’s “find the direction that decreases loss, using a step size that has to be right for every axis, every gradient magnitude, and every kind of critical point you’ll cross before training ends” — and a single number, chosen once at the start, was never going to be right for all of that at once. Momentum and Adam don’t remove that tension. They give the optimizer a memory to manage it with.

06 — INTEGRATIONWhat I check before I pick an optimizer now

I stopped choosing an optimizer off a textbook’s asymptotic argument and started asking three concrete questions instead, in order. First: on my actual data, on my actual hardware, which variant is actually faster wall-clock, not FLOP-count? I benchmark it, the way this experiment forced me to. Second: is my loss surface likely to be ill-conditioned — do my features live on wildly different scales — because if so, plain gradient descent is going to spend most of its budget oscillating, and momentum is close to a free fix. Third: what learning rate am I choosing, and is it calibrated to the optimizer I’m actually using, because 1.0 was a reasonable number for one of my two optimizers today and a catastrophic one for the other.

None of that makes vanilla gradient descent obsolete — at the right learning rate, on a well-conditioned surface, it’s still the cheapest thing that works, and today’s own numbers prove it can outrun Adam when the rate is small and the gradient is honest. It just means I no longer treat “gradient descent” as one algorithm with three interchangeable flavors. It’s four different bets about how much the shape of the loss surface changes underneath you — and the only way I’ve found to know which bet is safe is to actually watch the optimizer run.

07 — NEXTWhat’s next

• [  ]RMSProp and AdaGrad — add both to the same four experiments and see where their per-parameter scaling lands relative to momentum and Adam.
• [  ]Learning rate schedules — test whether decaying the rate over training recovers Adam’s early-training speed deficit at low LR.
• [  ]Real neural network loss — repeat the saddle-point test on an actual small network’s loss surface, where high-dimensional saddle points are far more common than true local minima.
• [  ]Batch size sweep — find where mini-batch’s wall-clock advantage over SGD stops improving, and whether it ever catches batch GD outright on this hardware.

08 — PROOFRun it yourself

Every number in this post comes from one pure-NumPy script — no ML libraries, nothing invented. The code, the single dependency (NumPy), and the verbatim output of a real run are public. Clone it, run it, and diff your output against mine.

$ git clone https://github.com/rosalinatorres888/gradient-descent-experiments.git
$ pip install numpy
$ python3 gradient_descent_experiment.py

Vanilla GD:  steps to reach loss<=1e-3: 206
Momentum:    steps to reach loss<=1e-3: 70
LR 1.0  |  vanilla: DIVERGED (inf/nan)  |  Adam: 0.2581
saddle escape — vanilla: step 95   momentum: step 28   adam: step 15

FIG. 3 — Selected lines from a verbatim run. The full log ships in the repo as expected_output.txt.