Inside the Past Tense: Raw Letters, Raw Sound

Arshavir Blackwell, PhD

Part 4 of 4. Models that read raw letters and raw sound, an attempt to force the child's U on purpose, and what the whole hunt adds up to. I'm Arshavir Blackwell and this is Inside the Black Box.

Arshavir Blackwell, PhD

TL;DR. Every result in this series was open to one objection: modern models are handed ee dee as a ready-made token before training starts, so maybe it was the tokenizer talking all along. This part takes the tokenizer away, twice.

Arshavir Blackwell, PhD

A model trained from scratch on raw characters, with no ee dee symbol to reach for, still learns the past tense backwards from a child — the regular ending first, exceptions memorized on top, no went to goed to went U. That holds across a sixteen-fold range of how often the exceptions appear.

Arshavir Blackwell, PhD

A U can be engineered, but only by breaking the training diet. The version that keeps the exceptions intact is small and needs an abrupt shock; the dramatic version is the model forgetting went, which is not what children do.

Arshavir Blackwell, PhD

The same model reading raw sound learns the genuine three-way ending rule — t, d, id — essentially perfectly on words it knows, and performs badly on words it doesn't, sitting at chance on one whole class.

Arshavir Blackwell, PhD

More training data improves performance greatly. What it doesn't improve: on roughly half of new words the model still prefers to leave the word unchanged rather than inflect it at all.

Arshavir Blackwell, PhD

Across all four parts, neither old camp wins. There is no child-like development and no discrete rule — just a distributed, frequency-weighted habit that generalizes by analogy.

Arshavir Blackwell, PhD

Our story thus far. In Part 1, language models learned the past tense backwards from a child: ee dee on everything first, exceptions like went memorized later, no trace of the famous went to goed to went U-shape. In Part 2, the wug test showed real but partial generalization, and six looks inside the machine found no rule — no switch, no readable category, no dedicated part. In Part 3, even a gradient-descent optimizer — the strongest possible search — turned up no rule, only a trick that games the metric.

Arshavir Blackwell, PhD

One objection is still standing, and it was raised by a sharp reader. Modern models don't read letters. They read chunks, and ee dee is almost certainly one of the chunks baked into their vocabulary before training even starts. If the model is handed ee dee as a ready-made brick, then maybe everything so far — the backwards learning, the leaky generalization — is just the tokenizer talking.

Arshavir Blackwell, PhD

This final part takes the brick away.

Arshavir Blackwell, PhD

A model with no ee dee

Arshavir Blackwell, PhD

I trained a model from scratch that reads one character at a time — a small transformer (about 11 million parameters, six layers), vocabulary of just 27 characters, no ee dee token, no prefab suffix. To produce the ending, it has to build it letter by letter. I trained it six separate times from different random starts and probed dozens of checkpoints across each run, exactly as in Part 1.

Arshavir Blackwell, PhD

The reversed pattern survived, cleanly. Overregularization is highest at the very start, when the model is bad at everything — goed-style forms on about half the irregulars at the first checkpoints — and falls to zero by roughly step 300 of an 8,000-step run, right as the exceptions get memorized (irregulars go from coin-flip to about 100% over the same stretch). Regulars snap in almost immediately; the exceptions are learned on top. No dip-and-recovery, no hump after competence, in any of six runs — the same "backwards from a child" signature as the big models, with the suffix now built from raw letters.

Arshavir Blackwell, PhD

Six runs agreeing that precisely is a warning, not a result. The obvious culprit was a number I had picked myself: irregular verbs showed up four times as often as regulars, and exceptions that common get memorized early — which is exactly what would kill goed on its own, whatever else was true. Rare exceptions are the condition under which a child-like U is most likely to appear; you can't over-apply a rule to a word you haven't learned yet. So I ran the whole thing across a sixteen-fold range of exception frequency, six times at each setting.

Arshavir Blackwell, PhD

The reversal didn't budge. What changed was only the timing: when the exceptions are rare, goed lingers longer before fading — because it takes the model longer to memorize words it rarely sees. When they're common, it fades fast. But across the entire range, once the model had learned the exceptions, goed never climbed back. It only kept falling — at the first probe after the exceptions were learned it was already below one percent, and zero from there on. The child's U needs that number to rise after the exceptions are known. It never rose. Not once, at any setting.

Arshavir Blackwell, PhD

So the tokenizer wasn't doing the work — or if it was, it didn't matter, since the same thing happens without it. Hand the model ee dee as a brick, or make it weld the letters together itself: either way it reaches for the common ending first and learns the exceptions after. That's a fact about how this kind of system learns, not an artifact of how we chopped up its input. (This is deliberately a small model on a controlled vocabulary, so that the single variable under test is the tokenizer and nothing else. A clean control, not a scaled-up model.)

Arshavir Blackwell, PhD

Could we force a U?

Arshavir Blackwell, PhD

If steady learning never makes a child's U, could a training schedule fake one? I tried this. I taught a fresh model the exceptions first — only irregular verbs, so it memorized went while it had barely seen an "ee dee" — and then flooded it with regular verbs, to see whether "goed" would surge after mastery, the way a child's does. Three versions:

Arshavir Blackwell, PhD

Exceptions kept, mild flood: nothing. "Goed" blipped to about 5% at the switch, then back to zero; the model learns "ee dee" for the regulars and holds onto went.

Arshavir Blackwell, PhD

Exceptions kept, hard sudden flood: a small hump — "goed" rose to about 16%, then recovered to zero — and the model never lost went (its irregulars dipped only briefly and finished about 98% correct). This is the real thing in miniature: over-apply for a moment, then rein it back. But it's small, and it takes an abrupt shock to trigger.

Arshavir Blackwell, PhD

Exceptions removed, then flood: a big rise — "goed" climbed to about 72% and never came down — but this one's a fake. The model's accuracy on went collapsed from about 100% to about 42%. It didn't over-apply a rule to words it still knew; it forgot them, because I stopped showing them.

Arshavir Blackwell, PhD

So the U isn't native to the system — you have to engineer a break in the input to get one at all. The genuine version, where the model still knows the exceptions, is small and needs a hard jolt; the dramatic version is just forgetting. Which is, almost word for word, what Pinker and Prince said in 1988 about the 1986 model: its U was a product of how the training was staged, not a stage of learning.

Arshavir Blackwell, PhD

One calibration is worth making here, because the textbook cartoon oversells the dip. When Marcus and colleagues actually counted — combing the recorded speech of 83 children for every past-tense form — overregularization turned out to be rare. The median child produced goed-type errors only a few percent of the time, and rates above ten percent were unusual (Marcus, Pinker, Ullman, Hollander, Rosen & Xu, 1992). Children in the goed phase still say went correctly the overwhelming majority of the time. The U is real, but it is a shallow scallop, not a collapse. (Maratsos, 2000, re-examined the question with denser longitudinal data and found higher rates in some children — so the exact number is still argued over. Everyone agrees it is nowhere near the dramatic dip of the diagrams.)

Arshavir Blackwell, PhD

That recalibration doesn't reopen the general question — the model still doesn't follow a child's path on its own. But it reframes my three conditions. The 16% hump, with the irregulars known throughout, has the right shape: over-apply for a while, then rein it back, without losing the stored forms. Its peak is higher than Marcus's numbers would lead you to expect, but the trajectory is the human one.

Arshavir Blackwell, PhD

The 72% version is a different animal, and the difference is worth being precise about — because on the surface it looks more like the textbook U. A child in the goed phase does say goed instead of went; that's the phenomenon. But the child keeps hearing went, and keeps producing it correctly most of the time. The error rides on top of knowledge that stays intact, which is why recovery is possible at all. My dropped-irregulars model had went taken out of its world entirely, and its accuracy fell to 42% through sheer disuse. That isn't over-applying a rule to a word you still know; it's forgetting a word you stopped hearing. Nobody stops saying went around a three-year-old.

Arshavir Blackwell, PhD

Only two of the three conditions produced a hump at all, and they invert. The small one — 16%, with went known throughout — is the one that matches what children do. The big one, far more like the textbook U, gets there by a mechanism no child undergoes.

Arshavir Blackwell, PhD

That whole conclusion has a serious objection, though, and it comes from the people who answered Pinker and Prince the first time — Plunkett and Marchman, who got a U out of a network with no abrupt break at all. Running their version properly took two more experiments and turned up something I wasn't expecting. It needs a post of its own, and it'll get one.

Arshavir Blackwell, PhD

From letters to sound

Arshavir Blackwell, PhD

One more brick to remove. The letter model answers the tokenizer objection for spelling — but the past tense is really a sound rule. Spoken English doesn't have one "ee dee"; it has three endings, chosen by the sound before them: the t in "walked," the d in "played," and the id in "wanted." A child builds the past tense out of sound. Spelling hides all of that.

Arshavir Blackwell, PhD

So I built the sound version: the same small transformer, but reading phonemes — the sounds of words, with real dictionary pronunciations — so that to produce a past tense it has to pick the correct ending sound for each stem. Same six-seed, checkpoint-by-checkpoint procedure.

Arshavir Blackwell, PhD

Two results. First, the same reversal, again: "goed"-style errors highest at the start (about half the irregulars), gone by the time the exceptions are learned, no U. That makes the backwards-from-a-child pattern three for three — big token models, raw letters, raw sound.

Arshavir Blackwell, PhD

Second, the sound model learns the real rule — on words it knows. On its familiar verbs it picks the correct ending sound — t vs d vs id — essentially 100% of the time, across all six runs. A model handed no "ee dee" of any kind, working in pure sound, builds the genuine phonological pattern.

Arshavir Blackwell, PhD

But then the wug test, in sound: 24 brand-new made-up stems it had never seen, each demanding a specific ending by the sound rule. And here the clean picture cracked. On novel words it picked the right ending only about 71% of the time. The breakdown was lopsided: perfect on the id-type (wanted-style, 100%), decent on the t-type (walked-style, 81%) — and on the d-type (played-style), 31%. With three endings to choose between, chance is 33%. On that whole class the model was guessing.

Arshavir Blackwell, PhD

A rule that's perfect on known words and no better than guessing on one whole class of new ones. Leaky, and lopsided — the same signature as everywhere else in this series.

Arshavir Blackwell, PhD

Was the rule broken, or just needing more training data?

Arshavir Blackwell, PhD

But there's an obvious objection: that model trained on just 72 verbs. That's almost nothing. Maybe the rule wasn't broken. Maybe it was just starved — it hadn't seen enough words to work the pattern out.

Arshavir Blackwell, PhD

There's an easy way to check. Feed it more. I rebuilt the experiment with 600 verbs instead of 72 — about eight times as many — and ran it again. Same model, same everything, just far more words to learn from. Then I gave it the exact same made-up words.

Arshavir Blackwell, PhD

Two clear results.

Arshavir Blackwell, PhD

The backwards learning didn't budge. Eight times more verbs, and still: ee dee first, exceptions later, no child-like dip-and-recovery. That finding has now survived every version thrown at it.

Arshavir Blackwell, PhD

The leaky rule mostly healed. With eight times more verbs, the made-up-word score climbed from about 71% to 88%. And that d ending — pure guesswork on the small model — jumped from 31% to 92%.

Arshavir Blackwell, PhD

So the rule wasn't broken. It just needed more data. Given enough examples, this little model — with no "ee dee" handed to it, working purely in sound — learns the genuine pattern, voicing and all. That's a correction worth noting as a trap: "the model can't do X" is often really "I didn't give it enough to learn X."

Arshavir Blackwell, PhD

One thing did not heal, though, and it's the interesting remainder. There are two questions you can ask about a made-up word: if the model commits to an ending, does it pick the right one? (That's what jumped to 88%.) And does it prefer the inflected form at all? On that second question, the correctly-formed past tense outscores the bare, unchanged stem only about 40% of the time — and eight times more data barely moved it. On roughly half of new words the model would sooner leave the word alone than commit to changing it. With enough examples it learns which ending belongs where almost perfectly; it still doesn't reliably decide that an ending belongs there at all. A confident rule-follower would do both.

Arshavir Blackwell, PhD

What the whole series adds up to

Arshavir Blackwell, PhD

Step back across all four parts, and the picture is consistent, and interesting, because it refuses to hand either old camp a win.

Arshavir Blackwell, PhD

The model does not learn like a child (no U; reverse order), and that isn't the tokenizer's fault: it holds in letters and in sound, at every frequency setting, and the only way I could produce a U at all was to engineer a break in the training diet. So much for the connectionist selling point of reproducing human development.

Arshavir Blackwell, PhD

The model has no clean symbolic rule — no switch, no readable category, no dedicated part, no rule direction even an optimizer can find, and no full commitment to new words even when it knows the pattern cold. So much for a separable "ee dee rule."

Arshavir Blackwell, PhD

What it has instead is a distributed, frequency-weighted "ee dee" habit — memorization-heavy, strengthened by examples, that generalizes to new words only partially, never cleanly — not in the nine-billion-parameter models of Parts 1 and 2, not at six hundred verbs.

Arshavir Blackwell, PhD

And when it does get a new word right — which is often — the engine doing it is analogy, not a rule. Thousands of stored "ee dee" examples overlap in the same connections and act like a magnet on any new stem: wug gets "wugged" because it sounds like hug and tug, not because anything checked "verb, then apply suffix." That's why the generalization is graded where a rule would be all-or-nothing, why it strengthens when you feed it more examples, whereas a rule would need none, and why no probe or optimizer can find it anywhere — it's the echo of the examples, smeared across every weight, not an operation stored at an address.

Arshavir Blackwell, PhD

The forty-year "rules versus connections" war was fought over a system nobody could open. We finally opened one. The answer it gives is neither — a third thing, closer to graceful degradation than to grammar. Reach in and jostle it, and it doesn't toggle between a rule and a lookup table. It slumps toward its most common habit and mumbles ee dee.

Arshavir Blackwell, PhD

There's something fitting about that. The past tense was supposed to be the clean case — the toy problem simple enough to reveal the architecture of language. Inside the machine, it turns out there's no architecture to reveal. Just a lot of overlapping habit, and a syllable it reaches for when it isn't sure.

Arshavir Blackwell, PhD

A personal note

Arshavir Blackwell, PhD

I was a graduate student at U.C. San Diego when a lot of this work was being done. Virginia Marchman was a postdoc of my advisor Elizabeth Bates, the psycholinguist. My other advisor, Jeff Elman, was a pioneer in neural networks and among the first to train a model by having it predict the next element in a sequence, which is, of course, exactly how modern LLMs are trained. I sat through many seminars on past-tense acquisition in children and in neural networks.

Arshavir Blackwell, PhD

Where the argument always ran aground: both sides could account for the data. Neither could look inside the thing doing the accounting, not a child's head, and not really the models either (which were too small to be truly LLMs anyway). The question wasn't unanswerable in principle. It was unanswerable with the tools in the room.

Arshavir Blackwell, PhD

You're listening to this from the other side of that. The past-tense debate can now be run — reached into, steered, switched off, retrained from scratch with the tokenizer taken away. Everything in these four posts came off a single desktop computer. Whatever stopped that roomful of very smart people, it isn't stopping you.

Arshavir Blackwell, PhD

And the chatbot you used this week descends directly from a model Jeff built to study language. Next-word prediction wasn't invented to make an assistant; it was a way of asking whether structure could be learned from sequence alone. It scaled further than anyone in those seminars imagined, but its parent question was a question about language.

Arshavir Blackwell, PhD

Jeff and Liz would be delighted and amazed. I think they'd also want the argument continued rather than settled. I'm Arshavir Blackwell and this has been Inside the Black Box.

Arshavir Blackwell, PhD

Thanks for listening to this series. Full methods, numbers, and code available on request — and if you see a hole in any of this, that's what the comments are for. Finding holes is how half these posts happened.