Running Flagship AI Models on Legacy Server Hardware

The biggest obstacle to running today's largest AI models locally isn't processing power — it's memory. While consumer GPUs continue to get faster, they still offer relatively limited VRAM, forcing enthusiasts to either buy multiple expensive graphics cards or settle for smaller models. But YouTuber MattMo wondered whether an old enterprise server packed with inexpensive RAM could handle the workload.

The system he put together is about as far from a modern AI workstation as you can get. It's a Dell PowerEdge R720, a rack-mounted server introduced around 2012, purchased used for $250. Instead of relying on GPUs, the entire system runs AI models on a pair of 10-core Intel Xeon processors with a combined total of 20 cores and 40 threads.

Without question, these CPUs are showing their age. They predate AVX2, the vector instruction set that many modern AI frameworks use to accelerate inference, leaving the processors to brute-force the workload using standard CPU instructions. That sounds like a major disadvantage, but MattMo's goal wasn't speed. It was having enough memory capacity to run flagship models locally.

Every DIMM slot in the server is populated with a 16 GB DDR3 ECC module, giving the machine 384 GB of RAM. Thanks to the abundance of surplus enterprise hardware, that mountain of memory only cost about $280. A Samsung enterprise SSD stores the operating system and the massive model files before they're loaded into memory.

The entire machine comes in at around $500 to $600, a fraction of what it would cost to build a GPU-based system capable of holding 200 GB-plus models entirely in VRAM.

To see how useful the setup was, MattMo tested several large language models using llama.cpp. Each model was asked to generate a C# sorting program. GLM 5.3 Flash produced the best result, completing the task in two minutes and 35 seconds at roughly four tokens per second. Qwen 3.8 Dense finished in six minutes and 18 seconds, while Qwen 3.8 8B Flash took seven minutes and 32 seconds.

Though the models ran successfully, performance was very slow. That led MattMo to experiment with Mixture-of-Experts (MoE) models. Rather than activating every parameter for each token like a dense model, an MoE model only uses a subset of its expert networks during inference. That dramatically reduces the amount of computation and memory traffic required, making these models particularly well-suited to older CPU-only systems.

You won't get anything like real-time interaction out of a system such as MattMo's, but it is still quite useful. The server can run enormous flagship models completely offline, without cloud subscriptions or expensive accelerator cards. Better yet, the PowerEdge still has open PCIe slots, leaving room for future GPU upgrades or partial layer offloading if more performance is needed. For more details, be sure to check out the video above.