Quick overview
CPU role in local LLM inference: preprocessing, offload layers, tokenization, and when a CPU upgrade matters.
GPUs own matrix math for on-card layers, but the CPU still matters. It handles tokenization, sampling, data movement, and any offloaded layers when VRAM runs out. A weak CPU paired with a strong GPU can still feel sluggish if your runtime spills heavily to RAM.
Quick answer
For full GPU fits, CPU needs a modern quad-core or better but is not the main spend. For CPU offload or CPU-only inference, favor more cores and RAM bandwidth. Upgrade GPU VRAM before chasing CPU clocks if OOM is the error.
CPU tasks during inference
- Tokenization and prompt assembly
- Orchestrating GPU kernels
- Running layers assigned to CPU
- Disk IO for model loading
When to invest in CPU
- Multi-user local servers
- Heavy preprocessing (RAG indexing on CPU)
- Large batch offline jobs
When GPU still dominates
Single-user chat with full GPU layer fit: CPU upgrades yield smaller gains than VRAM upgrades.
Platform notes
Match RAM channels to CPU platform guidance when building new. See choosing a processor for general desktop guidance.
How Roger N Reckon can help
Roger N Reckon balances CPU and GPU on new workstations so neither side is mismatched for your Ollama offload settings.
Practical checklist
1. Write down the model size, quant level, and target context length you need. 2. Measure current GPU VRAM, system RAM, and free NVMe space before buying parts. 3. Confirm PSU wattage against the GPU vendor's required system power. 4. Plan cooling for sustained inference, not just short benchmarks. 5. Back up data before OS or hardware changes. 6. Request a written component list, testing scope, and warranty terms for any build or upgrade. 7. Re-test with the same prompt and context after changes.
Frequently Asked Questions
Does AVX matter?
Some CPU kernels use SIMD; exact impact varies by engine build.
Are Threadripper CPUs worth it?
Only for heavy CPU offload or parallel jobs, not typical 8B GPU chat.
Will faster CPU fix OOM?
No.
Does hyperthreading help?
Some; cores matter more for parallel preprocessing.
Should I enable E-cores on Intel?
Test performance; some runtimes pin threads unexpectedly.
Final takeaway
Buy the GPU for model fit; buy CPU cores when offload or server load is real.