The Rethought — Kimi K3 in 8GB, One Token Every 26 Seconds

The post going around says someone ran a 2.78-trillion-parameter model on a single CPU with 8 GB of RAM. No GPU, no framework, six files of portable C99, fully open source.

Every word of that is accurate. I checked.

It also needs 1.7 terabytes of free disk, and it produces one token every 26.5 seconds.

The part that got left out

The kimi-k3-in-c project is real and the engineering is genuinely good. Apache-2.0, around 9.7k stars, 8.24 GB peak memory measured.

What it doesn’t fit in the headline: a 1.56 TB checkpoint plus a 109 GB packed trunk, and performance that depends heavily on how fast you can read them. The author reports close to 5,900 MB/s on local NVMe. Network storage runs several times slower.

At 26.5 seconds per token, a 500-word answer takes about three and a half hours.

To his credit, the author says this plainly. The repo’s own framing is that it’s proof the model fits — “not a way to chat with it.” The overclaiming is entirely downstream, in the posts repeating it.

Nothing got smaller

Here’s what actually happened, and it’s more interesting than the headline.

Kimi K3 is a mixture-of-experts model, so only a slice of those 2.78T parameters is active for any given token. The implementation keeps a dense trunk resident, streams everything else from disk, and computes routed experts directly from their packed 4-bit form as they’re read rather than ever loading them whole.

So the model didn’t shrink. The working set moved down the memory hierarchy — from RAM to NVMe — and the bill arrived in latency instead of capacity.

Give it 128 GB and it runs at 5.6 seconds per token. Same output, byte for byte. More memory buys exactly one thing: less waiting on disk.

Why this is worth your attention

Because the pattern is everywhere and we keep falling for it.

You can nearly always make something fit. Cache it, compress it, page it, stream it, push it to a slower tier. “Did it fit” is almost never a meaningful question, because the answer is almost always yes if you’re willing to pay somewhere else.

The real question is what it now consumes instead. Here it’s disk bandwidth and about a thousandfold markup on latency — a completely reasonable trade for a research demonstration and a completely unreasonable one for anything a user waits on.

I’d apply the same test to the next “we made X run on Y” claim you see, including the ones in your own architecture docs. Something moved. Find out where it went.

What I’m actually thinking

The headline isn’t the achievement. Six dependency-free C files, no BLAS, deterministic enough that the test suite matches a PyTorch reference exactly — that’s the achievement, and it quietly asks how much of the framework stack we carry is essential versus incidental.

But the thing I keep coming back to is how fast “it fits in 8 GB” travelled and how completely “and 1.7 TB of NVMe, at 26.5 seconds a token” did not. Both numbers were in the same README.

So: the last time you told someone a system could handle something, which number did you lead with?

And which one did they need?