Appendix E: choosing hardware for AI workstations and servers

Decide in a fixed order: GPU memory size first, because a model that does not fit spills into memory about twenty times slower and every word waits for it. Then memory speed for running models or compute for training, then how GPUs connect, then power, cooling, disks and slots.

In Daveth Pro (AI Engineering)

What this page answers

Why can a slower GPU run an LLM faster?

If a card cannot fit the whole model, the remaining layers live in system memory, roughly twenty times slower, and every word reads all of them. In the example, one machine fit all 80 layers and the other only 56. Read it in “The slower card that won”

What matters most when choosing a GPU for AI?

Memory size first: will the model and its working data fit? Then speed, memory bandwidth for running models and compute for training, then how the GPUs connect, then the rest of the box. Read it in “Memory first”

The slower card that won

Two machines run the same 70-billion-parameter model, compressed to about 4 bits per weight. Machine A has two RTX 3090s from 2020, 24 GB each. Machine B has one RTX 5090 from 2025, with 32 GB and almost twice the memory speed of a 3090. Both run llama.cpp's speed test for writing text:

# machine A: 2 × RTX 3090 (48 GB in total)
| model                  |      size | ngl | test  |   t/s |
| llama 70B Q4_K_M       | 39.59 GiB |  99 | tg128 | 14.8 |
# machine B: 1 × RTX 5090 (32 GB)
| model                  |      size | ngl | test  |   t/s |
| llama 70B Q4_K_M       | 39.59 GiB |  56 | tg128 |  4.3 |

The speeds are worked out with the same rule as Down the Bits (writing speed is limited by memory speed), so they're realistic estimates rather than one measured run. The table layout is llama-bench's, shortened.

The newer, faster card is more than three times slower. Why?

  • Two GPUs add their speed together
  • The model doesn't fit in 32 GB, so part of it runs from the much slower system memory
  • llama.cpp is better tuned for older cards

Answer. Look at the ngl column: the number of the model's layers placed on the GPU. Machine A has all of them on its GPUs (99 means “all”). Machine B could only fit 56 of the 80 layers. The other 24 live in system memory, which is roughly twenty times slower than the 5090's, and every single word has to read all of them. The slow part sets the pace.

That's the first rule of AI hardware: does it fit? comes before how fast is it? A fast card that's too small loses to a slow card that's big enough, every time. Only once the model fits does speed start to matter, and then it's mostly memory speed, not the compute number on the box.

Memory first

So decide in a fixed order, each step only among the options that passed the step before. First, memory size: will the model and its working data fit in GPU memory? Second, speed: for running models, memory speed (bandwidth, in GB/s) sets how fast words come out; for training, compute speed matters more. Third, how the GPUs connect, if you need more than one. Fourth, the rest of the box: power, cooling, system memory, disks, slots. Skipping a step is how people end up with the regrets at the bottom of this page.

Drawing: 1 · Does it fit?, GPU memory (GB), weights + KV cache, or, training state, 2 · How fast?, running models: memory, speed (GB/s); training:, compute speed, 3 · How they connect, only if one GPU isn't, enough: PCIe lanes,, NVLink, 4 · The rest, power and circuit, cooling,, system RAM, disks,, slots and space, Each step only chooses among options that passed the step before., The compute numb

Why is memory a wall and the rest only trade-offs? Because a model that doesn't fit doesn't get a little slower: it either fails or spills into system memory and slows down several times over, as the hook showed. A card with less bandwidth, by contrast, just writes proportionally slower. You can live with “a bit slow”. You can't live with “doesn't fit”.

What do I need?

Here are the first two steps worked out for you. The planner adds up the memory needed (with the same rules as Down the Bits for running models and Down the Memory for training), then goes through common GPUs: how many of each you'd need, and for running models, roughly how fast words would come out for each user.

Drawing: Rough planning numbers: 8,192 tokens of context per user; KV cache at 16 bits; 1 GB working space; writing speed about 70% of memory speed divided by the bytes read per word. With several GPUs splitting the model's layers, words go through them one after another, so the speed shown is one card's; splitting each layer across cards (tensor parallelism) can do better. Training totals include

The weights stay the same, but eight people each need their own KV cache (the model's notes about each conversation), and the total jumps. “Full fine-tune” on 8B: about 140 GB, out of reach of any single card in the list except the MI300X, while QLoRA on the same model fits on a gaming card. The workload decides the hardware far more than the model's name does.

The rest of the box

Once you know which GPUs, the rest of the machine has to keep up with them. These are the parts people underestimate, each with a rule of thumb and the reason behind it. None of them makes the GPU faster, but each one can make it slower, or stop it working at all.

PartRule of thumbWhy
System memory (RAM)At least as much as all GPU memory together; twice as much is comfortable.Models are loaded through RAM, data loaders keep batches there, and anything that doesn't fit on the GPU spills into it.
CPU and PCIe lanesDesktop CPUs have about 24 usable lanes: one GPU at full width, or two at half. For 3 or more GPUs, a workstation CPU (Threadripper PRO, Xeon W) with 100+ lanes.Each GPU wants 16 lanes. Half width barely matters for running a model that fits, but slows loading and multi-GPU training.
DiskNVMe for models and datasets, separate from the system disk; 2 TB fills faster than you think.A 70B model is 40 to 140 GB per copy, and checkpoints and caches multiply it (Down the Disk).
Power supplyAbout 1.3 times the expected load, ATX 3.x for modern GPUs. See the calculator below.GPUs draw short spikes well above their rating, which trip undersized supplies.
Cooling and casePlan the airflow for the full load running for days. Several cards side by side need blower-style or server cards, or water cooling.Cards stacked together breathe each other's hot air and slow themselves down (Down the Bus).
Slots and spaceCheck slot spacing and card thickness: many current cards take 3 to 4 slots.Two thick cards often physically block each other or the second slot.
Network10 GbE or faster if you'll copy datasets around or join a second machine; 1 GbE is fine for remote access only.At 1 GbE, a 140 GB model takes about 20 minutes to copy.

The first row is the one most often skipped. A 64 GB workstation with a 96 GB GPU sounds fine until you load a 90 GB model and the loader needs room for it in RAM on the way, or a training run's data loaders start four workers each holding batches. When in doubt, buy RAM: it's the cheapest way to avoid a whole class of crashes.

Predict. Two RTX 5090s give you 64 GB of GPU memory. Is that enough for a full fine-tune of a 14B model?

Answer. Not even close. A full fine-tune with the usual optimizer needs about 16 bytes per parameter before activations: 14.77 billion × 16 is about 236 GB. You'd need around four 80 GB data-center GPUs, with sharding (FSDP or ZeRO) splitting that state between them.

The same two cards handle a LoRA or QLoRA fine-tune of a 14B model easily. That's why most people fine-tuning on their own hardware use LoRA: it turns a server problem into a workstation problem (Down the Memory).