Can Smaller Language Models Be Smarter?

Arshavir Blackwell, PhD

I'm Arshavir Blackwell, and this is Inside the Black Box. Today we’re talking about something that may get brushed off as a simple engineering tweak: making language models smaller and more efficient. But Sean Trott’s 2025 paper, “Efficiency as a Window into Language Models,” argues that efficiency is actually a scientific lever. Not just a cost-saver. A way to learn something about how intelligence works.

Arshavir Blackwell, PhD

A quick note on the paper itself: it’s part review, part position piece. Trott pulls together findings from cognitive science, model compression, and interpretability. And here’s the line that really sets the tone. He writes: “Efficiency is not a constraint on intelligence—it may be a clue to its structure.” That’s a great framing, because it flips the usual narrative on its head.

Arshavir Blackwell, PhD

Trott starts from a familiar contrast: humans learn language and reasoning using almost no energy, while large models burn staggering amounts of compute. But instead of treating this as a failure of AI, he treats it as a signpost. If you can build a system that learns well with far less power and far less data, you might get closer to the underlying principles that make learning possible at all.

Arshavir Blackwell, PhD

And there’s a historical parallel. Early neural network and cognitive science work often used tiny, bottlenecked models, and those constraints forced the models to reveal their structure. Sometimes a small model tells you more about the mechanism than a giant model that can brute-force its way through a task. Trott’s point is: that lesson is still relevant.

Arshavir Blackwell, PhD

He also stretches the argument beyond theory. More efficient models reduce environmental cost and make AI more accessible, especially for research groups without massive compute budgets. So efficiency becomes an issue of access, fairness, and scientific clarity, not just engineering.

Arshavir Blackwell, PhD

Now, the practical side. What are the tools that get us to leaner models without wrecking the reasoning inside? Trott focuses on three: distillation, pruning, and LoRA.

Arshavir Blackwell, PhD

We can start with distillation. In the teacher–student setup, the small model imitates a larger one. Efficient, yes—but Trott calls out a real risk. If the student nails the answer but builds different internal circuits, you’ve basically created a very confident mimic. High accuracy, low understanding.

Arshavir Blackwell, PhD

That’s where mechanistic interpretability earns its keep. It’s the only way to check whether the “student” is genuinely reproducing the teacher’s reasoning, or just finding shortcuts that won’t generalize.

Arshavir Blackwell, PhD

Next is pruning. Michel et al. (2019) showed you can cut out a surprising number of attention heads without hurting performance. Sometimes the model even improves. But again, pruning works best when you know what each head actually does—syntax, long-range tracking, rare token handling. Blind pruning is just model surgery in the dark.

Arshavir Blackwell, PhD

Then there’s LoRA, which is basically the gentlest kind of fine-tuning. You leave the whole model frozen and add tiny low-rank matrices that teach it new behavior. Trott frames this as a really promising path because if interpretability tools advance, we may eventually trace those low-rank edits and understand exactly what conceptual “routes” they create.

Arshavir Blackwell, PhD

Across all of these methods, Trott returns to the same foundational issue: if you don’t know which circuits are causal, every efficiency trick carries the risk of quietly deleting something essential.

Arshavir Blackwell, PhD

And here’s a twist he emphasizes: compressing a model doesn’t necessarily make it easier to interpret. When you pack more features into fewer neurons, they blend. Interpretability can actually get harder. You get denser, more entangled representations.

Arshavir Blackwell, PhD

This is why Trott contrasts compression with sparse autoencoders, which spread out the representation instead of squeezing it. More neurons, more distinct features, more transparency. Less efficient, yes—but much easier to study.

Arshavir Blackwell, PhD

That sets up the big question the paper ends on: Can you have both? Can we build models that are both efficient per watt and interpretable per circuit? Right now, that’s still wide open. Most of the existing compression research happens on small models. When you scale to billions of parameters, all the complexity returns.

Arshavir Blackwell, PhD

But maybe the path forward isn’t just “bigger, bigger, bigger.” Maybe it’s learning how to understand and refine what’s already inside these models—making them smarter without making them enormous. Thanks for listening. I'm Arshavir Blackwell, and this had been Inside the Black Box.