10/29/2026
There's an interesting shift in how AI inference (running a trained model, as opposed to training it) actually works under the hood. It splits into two distinct phases: prefill, where the system reads and understands your prompt, and decode, where it generates the answer one token at a time. Those two phases have different hardware needs, one is compute bound, the other is memory bandwidth bound.
That's part of why you see chipmakers starting to build specialized silicon for each phase instead of one general purpose GPU (graphics processing unit, the chip that does the parallel math for AI) trying to do both.
We've been building custom AI servers at eRacks for a while now, and this is exactly the kind of thing that shapes how we spec a box: how much of your workload is prompt processing versus generation, and what that means for the GPUs, memory, and interconnect you actually need. If you're planning an on-premise AI server, it's worth thinking about your workload mix before you buy.
https://share.google/UMh8qVcXsp9erGmtn