AI Hardware Lab
The mechanics behind every number on this site — what determines whether a piece of hardware can run a given model, and why. Diagrams, not just paragraphs.
GPU memory
A GPU has its own dedicated memory, separate from your computer's system RAM. Before a GPU can run any part of a model, that model's weights have to be loaded into this memory. If the model doesn't fit, the GPU cannot run it — there is no fallback to system RAM for the parts that don't fit; the process fails or has to be restructured (e.g. split across more GPUs).
GPU memory comes in two families used across this catalog: GDDR, used on desktop and workstation cards (cheaper per gigabyte, still fast, less total bandwidth), and HBM ("High Bandwidth Memory"), stacked directly on the chip in datacenter GPUs like the H100 and H200 (higher bandwidth, higher cost).
Parameters
A model's "parameters" are the individual numeric values it learned during training — roughly, its knowledge encoded as numbers. A 70-billion-parameter model has 70 billion of these values. Each one has to be stored somewhere, and at full (unquantized) precision each parameter takes about 2 bytes, which is why parameter count translates almost directly into a memory requirement: roughly 1 GB of memory per billion parameters at the common 16-bit precision most open-weight models ship in.
Some models — DeepSeek-V3.2 and Kimi K2.6 in our model advisor, for example — use a "mixture of experts" design, where only a fraction of the total parameters activate for any given token. That reduces compute per token, but all the experts still have to be resident in memory in case they're needed, so memory requirements are still based on the total parameter count, not the active count.
Quantization
Quantization reduces the precision used to store each parameter — for example, from 16-bit down to 8-bit or 4-bit — which shrinks the memory footprint roughly in proportion (8-bit is roughly half the memory of 16-bit; 4-bit roughly a quarter). This is a real, widely used technique, and it's why you'll see the same model described with very different memory requirements in different places.
We calculate memory math on this site at 16-bit precision (1 GB per billion parameters) because it's the safest, most conservative baseline and the format most open-weight releases ship in by default. If you plan to run a quantized version of a model, its real memory requirement will be lower than what we show — often significantly. Quantization can also reduce output quality to some degree; how much depends on the model and quantization method.
Watts and power
Watts measure the rate of electricity draw, not compute capability. A higher-watt GPU doesn't hold more memory or automatically run models better — it typically means more raw compute (faster), a larger electricity bill, and more heat that has to be removed from the room. Power matters for two practical reasons: your electricity cost, and whether your facility's power and cooling infrastructure can actually support the hardware.
A single RTX 5090 draws about as much power as a couple of hair dryers running continuously. An 8-GPU server draws roughly as much as several houses. A full GB300 NVL72 rack draws roughly 100x an average home's continuous load — which is why full racks are only deployed in facilities built for that load, not offices or homes.
Bar height is not to true linear scale — a real rack draws roughly 200× a desktop card, compressed here so all three stay visible. Hover, tap, or tab to a bar for the exact real wattage.
Rack servers
Datacenter GPU servers are built to mount in a standard 19-inch rack, measured in "rack units" (U) — 1U is about 1.75 inches of vertical space. An 8-GPU HGX-based server is typically 4U–8U depending on the integrator and cooling design. A full rack is usually 42U–48U total, so a single 8U server takes up roughly a sixth of a rack; a fully populated rack like the GB300 NVL72 is designed and sold as one 48U unit, not assembled piecemeal from individual servers.
Rack-mounted systems assume rack-scale infrastructure already exists: power distribution units rated for the load, hot/cold aisle or liquid cooling, and network cabling — not just an electrical outlet.
Inference vs. training
Inference is running a model to produce output — the model's weights are fixed, and the GPU just computes a forward pass. This is comparatively cheap: it needs enough memory to hold the model plus a working buffer (what we call the 20% working room), and a single GPU or a small cluster is often enough.
Training (including fine-tuning) updates the model's weights, which means the GPU also has to hold gradients and optimizer state alongside the weights themselves — commonly 3-4x the memory of the weights alone, sometimes more. This is why training hardware recommendations lean toward multi-GPU servers with NVLink rather than single cards, even for models that would fit in a single card's memory for inference alone.
Why some models need multiple GPUs
When a model's memory requirement exceeds any single GPU's capacity, it has to be split across multiple GPUs — a technique generally called model parallelism. For that to work efficiently, the GPUs need a fast, direct connection to share data with minimal delay. NVIDIA's datacenter GPUs use NVLink for this within a server; connecting multiple whole servers into a larger cluster additionally requires InfiniBand networking between them.
Hover, tap, or tab to a server's mesh or the InfiniBand link to see what it means. The moving dots show relative traffic density, not literal packets.
This is also why we only build "clusters" out of hardware designed for it (see the
clusterable flag on each product) — plugging several desktop cards into one PC
over ordinary PCIe technically works, but is far slower for a single large model and isn't
how real multi-GPU deployments are built.