The Mistake My Network Refused to Fix

The experiment was supposed to be clean. Build one two-layer network — a hidden layer, a ReLU, a softmax output over three sentiment classes — and train two copies of it, identical in every way except the loss function feeding the backward pass. One copy learns from cross-entropy. One learns from mean squared error on the same softmax outputs. Every textbook I’d read said cross-entropy would win on calibration: its confidence scores would track its actual accuracy, and MSE’s wouldn’t. I expected to confirm that and move on.

My first result said the opposite. The MSE-trained network’s calibration error came out lower than the cross-entropy network’s — 0.132 versus 0.180 — and its test accuracy was very slightly higher. I sat with that for longer than I want to admit, because it meant one of two things: either the textbook claim was wrong, or I was measuring the wrong thing. It took tracking individual predictions, not averages, to find out which.

01 — THE RECKONING, PART ONEWhat the two losses actually tell the network

Both networks share the exact same forward pass: input text, turned into TF-IDF features, multiplied through a hidden layer, passed through ReLU, multiplied through an output layer, turned into three class probabilities by softmax. They only disagree about what happens next — how the error at the output gets translated into a signal for the weights beneath it.

Cross-entropy’s gradient at the output layer reduces, after the calculus, to something almost embarrassingly simple: predicted probability minus true label. If the network is confidently wrong, that difference is close to its maximum size, and the correction is proportionally large. MSE’s gradient on the same softmax output carries an extra term from the softmax function’s own derivative — a factor that shrinks toward zero as the output saturates toward 0 or 1, regardless of whether that confident output happens to be correct. On real training data, the output-layer gradient norm for the MSE model measured consistently about three times smaller than cross-entropy’s, every epoch I checked, from epoch 1 through epoch 60. That part of the textbook claim held up exactly as advertised.

02 — THE RECKONING, PART TWOWhere the aggregate number went wrong

And yet the calibration numbers still said MSE was doing better. I extended training from 60 epochs to 300 to see whether that would change, and it clarified the picture instead of reversing it. Cross-entropy’s true training loss kept falling the entire time — 0.175 at epoch 60, down to 0.025 by epoch 300. MSE’s training loss plateaued hard: 0.289 at epoch 60, and still sitting at 0.214 at epoch 300, barely moved in the last 200 epochs. The smaller gradient wasn’t just slower. Past a certain point, it had essentially stopped.

That stall is exactly why the aggregate calibration number was misleading. A model that has quietly stopped learning also stops becoming overconfident — it never accumulates the extreme, confidently-wrong outputs that would blow up an aggregate calibration score. Its Expected Calibration Error looked good in the same way a student who never raises their hand looks like they never gave a wrong answer. The metric wasn’t lying about the numbers it measured. It was measuring something that no longer reflected the failure I actually cared about.

03 — THE RECURSIVE INSIGHTThe one test that actually showed the difference

So I stopped averaging and went looking for individual failures instead. After 8 epochs of cross-entropy training, I found 51 training examples the network was confidently — over 70% probability — wrong about. I picked ten of them and retrained both models from scratch, identical initialization, identical batch order, and tracked exactly one number per example, every epoch: the probability each model assigned to the sentence’s actual label.

FIG. 2 — Confidently-wrong examples corrected within 60 epochs, out of the same 10 tracked examples for both models. Cross-entropy corrected every one, averaging 17.2 epochs. MSE corrected one — the rest ended training with under 10% probability on the correct class, most below 4%, having barely moved since the mistake was made.

That’s the difference the aggregate metric hid. Cross-entropy found every one of its confident mistakes and, on average, un-learned it within about seventeen epochs. MSE found one. The other nine didn’t just fail to improve — most of them ended training more confidently wrong than when I started watching, final probabilities on the true class sitting between 0.2% and 9.5%, and eight of the nine below 4%. The vanishing gradient wasn’t a story about the network learning slower everywhere. It was a story about the network losing its ability to correct itself in exactly the cases where correction mattered most, while looking, on average, perfectly reasonable.

A model that stops being wrong loudly hasn’t necessarily gotten better. Sometimes it’s just stopped listening to the mistakes that would tell it so.

This is the same shape of trap I keep finding at every layer of this coursework, just wearing a new outfit: a single summary number — accuracy, calibration error, an average of anything — can be technically correct and still describe a population where the part that matters is buried by the part that doesn’t. Here, “the part that matters” was ten specific rows in a training set of nearly four thousand, and no epoch-level average was ever going to surface them. Only looking at what happened to a single wrong guess, one epoch at a time, did.

04 — INTEGRATIONWhat I check for before trusting a “well-calibrated” model now

I don’t trust an aggregate calibration or accuracy number on its own anymore, especially for a model with any mismatch between its loss and its output layer. Before I’ll believe a network is learning correctly, I now want to see what happens to its confidently-wrong predictions specifically — not just whether the loss curve looks like it’s decreasing, because a stalled, under-confident model can produce a loss curve and a calibration score that both look fine while quietly having given up on its hardest mistakes.

The practical fix here isn’t complicated — cross-entropy paired with softmax is the standard combination for exactly this reason, and nothing in this experiment argues against using it. What changed is what I’ll accept as evidence that a model is working. A shrinking loss number, on its own, told me almost nothing about whether the specific errors I most needed corrected were actually getting corrected. Tracking ten individual examples through sixty epochs told me everything the aggregate numbers had been quietly averaging away.

05 — NEXTWhat’s next

• [  ]Label smoothing — test whether softening the one-hot targets narrows the gap between the two losses, since it directly targets the overconfidence cross-entropy can produce.
• [  ]Focal loss — built specifically to up-weight hard, confidently-wrong examples; test it against both losses on the same tracked-example method used here.
• [  ]Real human-labeled data — repeat this on a dataset without VADER pseudo-labels, to rule out label noise as a confound in the calibration numbers.
• [  ]Deeper networks — check whether the same vanishing-gradient asymmetry compounds or dampens with more layers between the loss and the input.

I still owe the original question an honest answer: cross-entropy is the better loss for this task, full stop, but not for the reason my first calibration number suggested. It isn’t because MSE’s probabilities are generically worse-calibrated. It’s because MSE’s gradient goes quiet exactly when a confident mistake needs the loudest possible correction — and an aggregate score, averaged across thousands of examples that were never confidently wrong in the first place, will not tell you that. You have to go looking for the ten rows where it’s actually happening.

06 — PROOFRun it yourself

Every number in this post — the 4,952 sentences, the calibration paradox, the 10/10 versus 1/10 corrections, the 300-epoch plateau — comes from one pure-NumPy script with a seeded, deterministic pipeline. No PyTorch, nothing invented. Clone it, run it, and diff your output against mine.

$ git clone https://github.com/rosalinatorres888/cross-entropy-vs-mse.git
$ pip install numpy scikit-learn nltk
$ python3 nn_from_scratch_experiment.py

ECE — CE model: 0.1798   MSE model: 0.1317   (the misleading aggregate)
CE model:  10/10 corrected within 60 epochs (avg epoch-to-correct: 17.2)
MSE model: 1/10 corrected within 60 epochs
MSE model — true CE loss @epoch 60: 0.2893   @epoch 300: 0.2140

FIG. 3 — Selected lines from the verbatim run behind this post. The full log ships in the repo as expected_output.txt.