Wei-lin: Apple just published a paper showing them running a language model with nearly a trillion parameters.
Harper: Okay, sure. In a building full of servers, with a custom cooling system and its own power grid. We know.
Wei-lin: No. On four Mac Studios. The little silver boxes. Sitting on a desk, connected to each other with Thunderbolt cables.
Harper: Wait, what? A trillion-parameter model? Not some little 7-billion parameter toy?
Wei-lin: They tested a few, but the big one is called Muse Spark Open. It's 816 billion parameters. And on four Macs, they're getting over 36 tokens a second out of it.
Harper: That's faster than I can read. So you're having a real-time conversation... with a model that's usually locked away in a research lab, and it's running on hardware you could buy at the Apple Store.
Wei-lin: Exactly. This is from a post on their machine learning research blog. The whole point is showing how their framework, MLX, can do this. The problem, obviously, is that these models are just too big for one computer. The 816-billion-parameter one needs over a terabyte and a half of RAM just for the model weights.
Harper: And the top-end Mac Studio has... what, 512 gigs now? So it's not even close.
Wei-lin: Not even close. So you have to split it. And they show two ways of doing it. The first is called pipeline parallelization. You basically cut the model in half, depth-wise. The first 50 layers run on Mac number one, and it sends its output to Mac number two, which runs the next 50 layers.
Harper: Like an assembly line. Machine one does its job, passes the part to machine two.
Wei-lin: Right. It's simple, but it doesn't really speed up the generation of each new word, because that's still happening sequentially. Where it gets interesting is the second method: tensor parallelization.
Harper: Okay.
Wei-lin: Instead of splitting the model by its depth, you split it by its width. So for every single layer, part of the computation happens on machine one, part on machine two, part on three, and so on. They're all working on the same step at the same time.
Harper: So that does make it faster. But the coordination must be a nightmare. How do they talk to each other fast enough?
Wei-lin: That's the other piece. They're using a low-latency driver over Thunderbolt. It's a thing called... jaccl? J-A-C-C-L. To be honest, the specifics of how the RDMA driver works are... a bit beyond me. But the result is that the connection between the Macs is fast enough to not be the bottleneck.
Harper: So the 'data center' is just four Macs and a bunch of cables.
Wei-lin: And the power draw for one of these things, even the new M5 Ultra, is about 300 watts. The article says it's a fraction of the power of a common toaster.
Harper: So I can run a near-trillion-parameter model in my office and not blow a fuse. That's the part that's wild. It’s not just that it’s possible, it’s that it’s practical.
Wei-lin: And private. You're not sending your data to a third-party API. You can experiment on the latest, biggest models right on your desk. The paper really focuses on the new M5 Ultra chip, which has these 'GPU Neural Accelerators' that make the 'time to first token'—the initial thinking—up to 4.6 times faster than the last generation.
Harper: So it starts talking to you faster.
Wei-lin: And then it keeps talking faster because the M5 has much higher memory bandwidth. But the real story here isn't just a faster chip. It's that the line between what you can do on a desktop and what requires a massive, inaccessible server farm is blurring.
Harper: It feels like when you could suddenly edit feature films on a desktop computer. It didn't kill the big post-production houses, but it meant a whole new class of people could create.
Wei-lin: I think that's it. This isn't about replacing the cloud. It's about who gets to do research. Who gets to tinker at the frontier. And right now, the answer is anyone with a few Macs and a desk.
Harper: I can't get over the toaster thing. A supercomputer that uses less power than my breakfast.
Wei-lin: I'm Wei-Lin.
Harper: I'm Harper.