Last updated: October 2026
This explainer covers the data-center AI server: the 8-GPU systems and 72-GPU racks that train and serve models for cloud AI services. It defines the term, lists the components, explains the architecture, and gives the power, weight, and cost figures for current NVIDIA Blackwell systems, with notes on AMD, Google, and Huawei alternatives. If you want to run AI on your own hardware at home, that is a different scale and has its own guide: how to build a local AI server.
Key Takeaways
- An AI server is a computer built around accelerators (usually GPUs) rather than CPUs, with high-bandwidth memory on each accelerator and a dedicated interconnect so the accelerators work as one large pool.
- A current 8-GPU AI server draws about 2.9 kW idle and 12.5 kW at full load and holds over 1.4 TB of HBM3E, while a 72-GPU GB300 NVL72 rack draws 132 to 142 kW, weighs about 1,580 kg, and is estimated at $3.7 to $4.0 million.
- Rack power is the defining constraint: an average enterprise rack draws about 9 kW, so one AI rack needs the power and cooling of fifteen ordinary ones, which is why new AI servers are liquid-cooled and why data centers are being rebuilt around them.
What is an AI server?
An AI server is a computer designed to run the matrix math behind machine learning, so it is built around graphics processing units or other accelerators instead of general-purpose CPUs. A typical unit holds eight GPUs, two CPUs that mainly feed the GPUs, more than a terabyte of high-bandwidth memory, and network cards fast enough to join thousands of identical servers into one cluster.
The meaning of "server" is the same as in any data center: a rack-mounted machine that answers requests from other computers. What makes it an AI server is the ratio of accelerator to CPU. In a web server the CPU does the work; in an AI server the CPUs are support staff and the GPUs are the workforce. Inference, meaning running a trained model to answer prompts, and training, meaning building the model from data, both run on the same class of hardware, with training clusters simply using more of it.
How is an AI server different from a traditional server?
An AI server differs from a traditional server in four ways: it uses GPUs instead of CPUs for the main work, it uses stacked high-bandwidth memory instead of DDR5 DIMMs, it links its processors with NVLink or a similar fabric instead of relying on PCIe alone, and it draws ten to fifteen times the power of a conventional server in the same rack space.
| Attribute | Traditional 2U server | 8-GPU AI server (HGX B200 class) | 72-GPU AI rack (GB300 NVL72) |
|---|---|---|---|
| Main processor | 2 CPUs (Xeon or EPYC) | 8 NVIDIA B200 GPUs plus 2 CPUs | 72 Blackwell Ultra GPUs plus 36 Grace CPUs |
| Memory type | DDR5 DIMMs, 256 GB to 2 TB | HBM3E on each GPU, over 1.4 TB total | 288 GB HBM3E per GPU, about 20 TB pooled |
| Memory bandwidth | About 0.5 TB/s | About 8 TB/s per GPU | 8 TB/s per GPU |
| Processor interconnect | PCIe 5.0 | NVLink 5 between the 8 GPUs | NVLink 5 across all 72 GPUs, about 130 TB/s aggregate |
| Power draw (kW) | 0.5 to 1 | 2.9 idle, 12.5 at max load | 132 to 142 nominal, about 155 peak |
| Cooling | Air | Air (8U chassis) or liquid | Liquid, about 90 percent of heat to coolant |
| Rack units | 2U | 6U to 8U | Full 42U-class rack |
| Power supplies | 2 x 1 kW | 12 x 3 kW (6 active, 6 redundant) | Rack-level power shelves |
| Weight (kg) | 20 to 30 | About 140 (NVIDIA DGX B200) | About 1,580 |
| Cost (USD) | $10,000 to $40,000 | Mid six figures | $3.7 to $4.0 million (analyst estimate) |
HBM, or high-bandwidth memory, is DRAM stacked vertically on the same package as the GPU, which is how one chip reaches 8 TB/s when a desktop CPU's memory bus manages about 0.1 TB/s. That stacking is also why consumer memory got expensive in 2026: the same fabs make both, and each gigabyte of HBM displaces roughly three gigabytes of ordinary DDR5.
What components are inside an AI server?
An AI server contains six main component groups: accelerators, host CPUs, high-bandwidth memory, an accelerator interconnect, high-speed network cards, and a power and cooling system sized for ten kilowatts or more per chassis.
| Component | Role | Current example |
|---|---|---|
| Accelerators (GPUs) | Run the model's matrix multiplications | NVIDIA B200 or B300 (about 1 kW each), AMD Instinct MI350 series, Google TPU, Huawei Ascend |
| Host CPUs | Schedule work, load data, run the operating system | Intel Xeon 6, AMD EPYC, or NVIDIA Grace (Arm) in Grace Blackwell systems |
| High-bandwidth memory | Holds model weights and activations next to each GPU | HBM3E, up to 288 GB per GPU on Blackwell Ultra; HBM4 on the Rubin generation |
| Accelerator interconnect | Lets GPUs share memory and exchange results at memory-like speed | NVLink 5 and NVLink switch trays; AMD Infinity Fabric; UALink |
| Network cards | Join servers into a cluster | ConnectX-7 and ConnectX-8 at 400 to 800 Gbps, InfiniBand or Ethernet |
| Storage | Stages datasets and checkpoints | NVMe SSDs, typically 8 to 16 per chassis |
| Power supplies | Deliver and back up chassis power | Twelve 3 kW 80Plus Titanium units in one 8-GPU design |
| Cooling | Remove heat from GPUs and CPUs | Cold plates and an in-rack coolant distribution unit rated around 250 kW on NVL72 |
The accelerator interconnect is the part that separates an AI server from a workstation with several graphics cards. NVLink lets eight GPUs read each other's memory as if it were one pool, which is what allows a model larger than any single GPU's memory to run as one unit. Over PCIe alone the same job would spend most of its time waiting on data transfers.
What is AI server architecture?
AI server architecture is the layout that connects accelerators, memory, and network so a model can be split across many chips without those chips waiting on each other. It has two levels: scale-up, which joins GPUs inside one chassis or rack into a single memory domain, and scale-out, which joins racks into a cluster over InfiniBand or Ethernet.
In an 8-GPU HGX server, scale-up is the NVLink switch on the GPU baseboard. In a GB300 NVL72 rack, scale-up stretches across 18 compute trays, each carrying two Grace Blackwell superchips (one Grace CPU plus two GPUs), wired through NVLink switch trays in the middle of the rack so all 72 GPUs behave as one 20 TB memory domain. Scale-out is one 800 Gbps ConnectX-8 port per GPU that connects the rack to its neighbors.
The next step on NVIDIA's roadmap, Vera Rubin NVL144, keeps the same shape with a new Vera CPU (88 Arm cores) and Rubin GPUs on HBM4. NVIDIA's announced window for it is the second half of 2026, and its published figures are 3.6 exaflops of FP4 inference per rack, about 3.3 times the GB300 NVL72, with 260 TB/s of NVLink 6 bandwidth.
What is an AI server rack?
An AI server rack is a full cabinet delivered as one product, with compute trays, switch trays, power shelves, and liquid cooling manifolds pre-installed, so the whole rack behaves as one computer. The GB300 NVL72 is the current example: 72 GPUs, 36 CPUs, roughly 20 TB of HBM3E, 132 to 142 kW nominal draw, and about 1,580 kg.
That weight matters. A raised floor rated for ordinary racks will not carry it, so NVL72 deployments go on slab floors or reinforced zones. The rack also needs a coolant distribution unit, or CDU, which is the heat exchanger that moves heat from the cold plates to the building's chilled water loop; integrators ship in-rack CDUs rated around 250 kW to leave headroom above the rack's peak.
What is an enterprise AI server?
An enterprise AI server is an AI server sized for one company's own data center rather than a cloud provider's: usually a single 4-GPU or 8-GPU chassis, air-cooled so it fits existing racks, bought to run inference on internal data that the company does not want to send to a cloud API. Dell, HPE, Lenovo, and Supermicro sell them in the same configurations as their ordinary servers, with the GPUs and networking added.
The difference from a hyperscale deployment is in cooling and power rather than silicon. An enterprise buyer typically has 10 to 20 kW per rack available, which fits one air-cooled 8-GPU server at full load, where a cloud provider builds new halls around 140 kW liquid-cooled racks. The reasons an enterprise chooses its own AI server are the same ones that drive home users to local AI, at a larger scale: data stays on premises, there are no per-token fees, and the capacity cannot be withdrawn by a vendor policy change.
How much power do AI servers use, and how hot do they get?
A single 8-GPU AI server measured by ServeTheHome drew about 2.9 kW idle, with each GPU idling near 140 W, and just over 12.5 kW at maximum load. A 72-GPU rack draws 132 to 142 kW nominal and about 155 kW at peak.
For scale, the Uptime Institute's 2025 survey puts the average enterprise rack at about 9 kW. One NVL72 rack therefore needs the electrical feed of roughly fifteen conventional racks in the same floor space. At those densities air cooling cannot carry the heat away fast enough: a B200 or B300 GPU dissipates about 1 kW on a die a few centimeters across, so cold plates with liquid take about 90 percent of the heat straight from the chip and the room's air handles the rest. The power constraint, more than chip supply, is what limits how fast AI capacity grows, a point we cover in our AI compute wall article.
Who makes AI servers?
NVIDIA designs the reference systems (HGX baseboards, DGX branded servers, and the NVL72 rack) and manufacturers build them: Supermicro, Dell, HPE, Lenovo, ASRock Rack, Gigabyte, and the Taiwanese contract builders Foxconn, Quanta, and Wiwynn, which assemble most of the units sold to the largest cloud providers.
The alternatives split by buyer. AMD sells Instinct MI350-series accelerators through the same server vendors. Google builds TPU servers for its own cloud and does not sell them. Huawei's Ascend-based servers serve the Chinese market, where US export rules block current NVIDIA parts. Each of these uses a different interconnect, so a model tuned for one usually needs rework to run on another, which is a large part of why NVIDIA's share of the market has held.
How much does an AI server cost?
An 8-GPU Blackwell server costs in the mid six figures, and a GB300 NVL72 rack is estimated at $3.7 to $4.0 million by Loop Capital. Prices are quoted to large buyers and vary with memory capacity, networking, and support contracts, so public list prices are rare.
Cost per GPU has risen with each generation because the memory and interconnect account for a growing share of the bill of materials. The same HBM demand that pushed server prices up is the reason consumer DDR5 kits cost three to nine times their 2025 price in 2026.
Do you need an AI server to run AI at home?
No. Open-weight models from 7B to 120B parameters run on a single consumer GPU with 24 GB of VRAM or on a unified-memory mini PC with 64 to 128 GB, which is a few thousand dollars rather than a few million. The data-center hardware in this article exists to train those models and to serve millions of users at once; one household needs a fraction of one GPU.
Our local AI server build guide covers the parts, budgets, and used GPU picks for that scale, and the LLM VRAM calculator shows which models your current computer can run today.
One more monthly fee to kill
Renting your modem? That's $120+ a year.
Own it instead. It pays for itself in months. 90-day warranty and 30-day returns.
Pick your provider:
FAQ
How do AI servers work?
An AI server splits a model's weights across its GPUs, keeps those weights in each GPU's high-bandwidth memory, and uses NVLink to pass intermediate results between GPUs thousands of times per second. The CPUs load data and hand out work; the GPUs do the math. For training, thousands of servers repeat this in lockstep over a network, syncing their results after every step.
What is the difference between an AI server and a GPU server?
In practice there is none; "GPU server" is the older term. A machine with GPUs used for rendering or scientific simulation is also a GPU server, while "AI server" signals the current generation of parts: HBM memory, NVLink, and 800 Gbps networking, sized for machine learning.
How big is an AI server?
An 8-GPU AI server is 6U to 8U tall, meaning 27 to 36 cm of a standard 48 cm wide rack, and weighs about 140 kg in NVIDIA's DGX B200 form. A GB300 NVL72 is an entire rack, about 1,580 kg. For comparison a conventional two-socket server is 2U, about 9 cm tall, and 20 to 30 kg.
What operating system do AI servers run?
Linux, almost without exception. NVIDIA DGX systems ship with DGX OS, an Ubuntu derivative, and most other vendors ship Ubuntu, Red Hat Enterprise Linux, or Rocky Linux with the CUDA toolkit and container runtimes installed.
Can an AI server be used as a normal server?
Yes, but it is wasteful. The two CPUs run ordinary workloads fine, while the GPUs that account for most of the cost and power sit idle. The reverse is the common case: data centers retire conventional racks to free up power for AI racks.
Why are AI servers liquid cooled?
Because the GPUs dissipate about 1 kW each on a small die, and 72 of them in one rack produce more heat than fans can move through air in that volume. Cold plates with liquid coolant carry about 90 percent of the heat directly off the chips, which also lets the GPUs sit closer together for shorter, faster NVLink runs.
