When Washington Stopped Being a Person

It was supposed to be easy. Build a small NLP pipeline, run it on a few sentences, confirm it does what the documentation says. I picked three deliberately different test sentences involving the word “Washington” — a government context, a historical context, a travel context — mostly to see spaCy’s named entity recognizer flex a little. It didn’t flex. It called all three “Washington” a place. Including the one about a man crossing a river in 1776.

That’s not a typo in my code. That’s the actual output of a production-grade, widely-used model, behaving exactly as trained. I kept staring at it, because I’d assumed disambiguation was mostly solved — that a model reading “Washington crossed the Delaware River” would connect “crossed” and “River” to a person doing an action, not a location doing nothing. It didn’t make that connection. And once I stopped being annoyed and started asking why, the same shape of failure showed up twice more in places I hadn’t even gone looking for it.

01 — THE RECKONING, PART ONEWhat the model actually saw

Named entity recognition doesn’t read a sentence the way a person does. It scores a token’s local context — the words immediately around it, the capitalization pattern, the part of speech — against patterns it absorbed from training data, and it outputs whichever label had the most weight of evidence during training. “Washington” appears constantly as a place name (the city, the state) and constantly less often as a reference to the man, so the model’s prior is heavily loaded toward GPE — geopolitical entity — before it even reads the rest of the sentence.

This is the part that reframed the problem for me: the model wasn’t confused. Confusion would look like uncertainty, a close call between two labels. This was confidence in the wrong answer — a fluent, decisive GPE tag delivered with no hesitation, because from the model’s point of view there was nothing to hesitate about. It had never really been asked to reason about “crossed” and “River” as verbs implying a subject. It had been asked to pattern-match a proper noun, and it did that job perfectly. I was the one who had brought a different question to a system that only ever answers the one it was trained on.

02 — THE RECKONING, PART TWOThe words a tokenizer decides don’t exist

The ambiguity problem sits on top of an earlier, quieter one: a model can’t reason about a word it never sees as a discrete unit. I ran three sentences full of contractions, hyphenation, and one stray emoji through spaCy’s tokenizer and compared the token count to a naive str.split(" "). “I can’t believe it’s already state-of-the-art.” split naively into 6 pieces. spaCy split it into 15 — “can’t” became ca + n't, “state-of-the-art.” became seven separate pieces around the hyphens and the trailing period.

Nine tokens is not a rounding error. It’s nine decisions the tokenizer made silently, about what counts as a word, that every downstream layer of the pipeline now inherits without question. A part-of-speech tagger downstream sees n't as its own negation particle rather than folded invisibly into “can’t” — which is exactly why spaCy’s tokenizer bothers to split it that way. But it means the sentence you typed and the sentence the model reasons over are already two different objects before a single layer of “real” NLP has run. Tokenization isn’t preprocessing you can skim past. It’s the first and most consequential modeling decision in the whole pipeline.

03 — THE RECKONING, PART THREEThe vector that can’t tell “not” from anything else

I wanted to see the same blindness show up somewhere further from named entities, so I ran a bag-of-words comparison on two nearly-identical sentence pairs — one plain, one negated. “I love this movie, it is fantastic.” against “I don’t love this movie, it is not fantastic.” Cosine similarity between their bag-of-words vectors: 0.87. A second pair — “great and fast” versus “not great and not fast” — came out at 0.80.

Bag-of-words treats “not” as one more token in the bag, worth exactly as much as “fantastic” or “fast” and no more. It has no concept of scope — no representation that “not” reaches forward and flips the token after it. Two sentences that mean opposite things about the same subject sit 87% of the way to identical in that vector space, because from the counter’s point of view, they mostly share the same bag of words. Anything downstream trained on these vectors — a classifier, a similarity search, a recommender — inherits that blindness for free, with no error message to tell it something went wrong.

04 — THE RECURSIVE INSIGHTThree failures, one root cause

Looking back at all three, I realized I’d been treating them as unrelated bugs to patch individually — fix the tokenizer, add negation handling, retrain the NER model on more historical text. But they’re not three problems. They’re the same problem observed at three different layers of the pipeline: every one of these systems is matching statistical surface patterns from its training distribution, and every one of them is blind to exactly the information that pattern-matching can’t encode — word order, scope, and the specific sense a rare context implies.

The tokenizer doesn’t know what a “word” means to a human; it applies rules learned from a training distribution of English text. The NER model doesn’t know who George Washington was; it applies the label distribution it saw for the string “Washington.” Bag-of-words doesn’t know what negation does to meaning; it counts. None of these are failures of effort or scale — en_core_web_sm is a real, competently trained model. They’re failures of what the representation was built to notice in the first place.

A model that’s never confused isn’t a model that’s right. It’s a model that’s never been asked the question you’re actually asking.

That reframes what “benchmark accuracy” means to me now. A tokenizer, tagger, or NER model reports strong numbers on its evaluation set because that set was drawn from the same distribution it learned from — the same bias toward “Washington-as-place” that made my third test sentence pass and my second one fail. The gap between benchmark accuracy and production performance isn’t a generalization bug to be patched later. It’s the specific, nameable set of contexts a model’s training distribution didn’t cover, and every new domain you point it at is a fresh chance to find where that boundary is.

05 — INTEGRATIONWhat I test for differently now

I stopped treating a pipeline’s first successful run as a finish line. Now, before I trust output from a tokenizer, a tagger, or an entity recognizer, I deliberately feed it the case I expect it to get wrong — a contraction, a negation, a proper noun with more than one plausible sense — the way I did here almost by accident. If it handles those cleanly, I trust it more. If it doesn’t, I now know exactly where the seam is, instead of finding out three layers downstream when a classifier makes an inexplicable mistake and I have to trace it back to a tokenization choice I never inspected.

The 40.2% out-of-vocabulary rate I measured when I ran a movie-review vocabulary against space-related newsgroup posts — against a 17.8% miss rate on held-out movie reviews from the same domain — is the same lesson in a different shape. A model’s fluency is a claim about the world it was shown, not the world in general. Mine had never been shown much about spacecraft, so a quarter of what it would need to talk fluently about that world simply wasn’t there. Not unlike walking into a conversation about a field you’ve never studied — you can follow the grammar and still miss the meaning, because the vocabulary that carries the meaning was never yours to begin with.

06 — NEXTWhat’s next

• [  ]Coreference resolution — test whether a coreference model connects “he” back to “Washington” correctly across the same three contexts, and whether that changes the entity label.
• [  ]Fine-tune on historical text — retrain NER on a corpus with more person-Washington mentions and re-run the same three test sentences to see if the prior actually shifts.
• [  ]TF-IDF negation weighting — test whether weighting “not” and its scope words differently recovers any of the lost distinction between the negation pairs.
• [  ]Multilingual tokenization — run the same edge-case audit on a non-English spaCy pipeline, where tokenization rules (and their failure modes) are different again.

None of this makes spaCy, or bag-of-words, or benchmark accuracy useless — I’d reach for all three again tomorrow. It changes what I ask of them first. Before I trust a number, I now want to know what world that number was measured in, and whether the sentence in front of me actually belongs to it.

07 — PROOFRun it yourself

Every number in this post comes from one small Python script — nothing was invented or rounded into existence. The code, pinned dependencies, and the verbatim output of the exact run behind this post are public. Clone it, run it, and diff your output against mine.

$ git clone https://github.com/rosalinatorres888/nlp-failure-modes.git
$ pip install -r requirements.txt && python3 -m spacy download en_core_web_sm
$ python3 nlp_failure_modes_experiment.py

'Washington crossed the Delaware River on Christmas night.'
  -> entities: [('Washington', 'GPE'), ('the Delaware River', 'LOC'), ...]
Cosine similarity (bag-of-words): 0.8660
Out-of-vocabulary tokens: 8373 (40.2%)

FIG. 2 — Selected lines from the verbatim run behind this post. The full log ships in the repo as expected_output.txt.