Last updated: October 2026
This guide covers building a local AI server for a home or small office: an AI server PC that runs open-weight language models with Ollama, llama.cpp, or vLLM on your own network. It compares a do-it-yourself tower, a used enterprise server, and the prebuilt unified-memory boxes, using US street prices from October 2026. It does not rank mini PCs (see our mini PC tier guide), compare graphics cards and VRAM tiers for a buying decision (see our local AI hardware guide), or pick models per memory size (see best local AI models by VRAM).
Key Takeaways
- The single number that decides what a local AI server can run is memory the model can live in: GPU VRAM, or unified memory on Strix Halo, Apple, and DGX Spark systems. A 24 GB card runs 27B-class models at 4-bit quantization; 48 GB runs 70B.
- In October 2026 a used RTX 3090 (24 GB) sells for about $1,400 and a Tesla P40 (24 GB) for about $275, while DDR5 kits cost three to nine times what they did a year ago. Used parts are where the value is.
- Three builds cover most people: a $300 to $450 upgrade to a spare PC, a $2,000 to $2,300 single-3090 tower, and a $3,500 to $4,000 dual-3090 tower that fits a 70B model entirely in VRAM.
What is a local AI server?
A local AI server is a computer on your own network that runs AI models from its own memory, so prompts and documents never leave the building. It differs from a data-center AI server in scale only: one or two consumer GPUs instead of 72, a wall outlet instead of a 140 kW rack. The software stack is usually Linux, a model runner such as Ollama or llama.cpp, and a web front end such as Open WebUI that other devices in the house reach over the LAN.
Most home servers do inference, meaning they answer prompts with a trained model. Training or fine-tuning a model needs far more memory and is outside this guide. If you want to know what the data-center version looks like, read what is an AI server.
What hardware does a local AI server need?
A local AI server needs enough fast memory to hold the model, a power supply with headroom for the GPU, and enough airflow to run for hours. Everything else is secondary.
| Component | What matters | Target for a 24 GB build |
|---|---|---|
| GPU memory (VRAM) | The model, its context window, and the KV cache must fit. A 27B model at Q4 is about 16 to 18 GB; a 70B model at Q4 is about 40 GB | 24 GB minimum, 48 GB for 70B |
| Memory bandwidth (GB/s) | Sets tokens per second once the model fits. The RTX 3090 moves 936 GB/s; a Tesla P40 moves 346 GB/s; Strix Halo and DGX Spark move 256 to 273 GB/s | 900 GB/s or more for conversational speed on 27B models |
| System RAM | Only matters when a model spills out of VRAM (slow) or for CPU-only runs. DDR4 platforms dodge the worst of the 2026 DDR5 pricing | 32 GB |
| CPU | Almost irrelevant for GPU inference. Any 6-core desktop chip from the last six years is enough | Ryzen 5 3600 or newer, Core i5-10400 or newer |
| PCIe lanes | One x16 slot per GPU is ideal; x8 per card is fine for inference, x4 slows model loading only | x16, or x8/x8 for two cards |
| Power supply (W) | GPU board power plus 150 W for the rest, plus 20 percent. One RTX 3090 is 350 W; two are 700 W | 750 W (Corsair RM750e) for one 3090, 1,200 W (Corsair RM1200x Shift) for two |
| Storage | Model files are 4 to 45 GB each. NVMe shortens load time; capacity matters more than speed | 1 TB NVMe (Samsung 990 EVO Plus) |
| Cooling | A GPU at 350 W for an hour heats a small room. Data-center cards (P40, M40) have no fan and need a 3D-printed shroud or server-style airflow | Two 140 mm intakes, one exhaust |
VRAM is video memory on the graphics card, and it is where the model runs on a GPU build. Unified memory is a single pool shared by the CPU and GPU, which is how Strix Halo, DGX Spark, and Apple machines fit 100 GB models without a graphics card. Our LLM VRAM calculator tells you which models fit a given memory number before you buy anything.
How much does a local AI server cost?
A capable local AI server costs between $300 and $4,000 in October 2026, depending on whether you upgrade a spare PC or build a dual-GPU tower. Those numbers are higher than a year ago because memory prices rose sharply through 2026: DDR5 kits that sold for $100 to $330 in August 2025 sold for $425 to $3,400 in August 2026, and used GPU prices followed as buyers moved to the secondhand market.
| Build | Parts | Street cost (USD, Oct 2026) | Runs comfortably |
|---|---|---|---|
| Spare-PC upgrade | Existing desktop with a 500 W or larger PSU, plus a new RTX 3060 12 GB (about $300) or a used Tesla P40 24 GB (about $275 plus a fan shroud and EPS-to-PCIe power adapter) | $300 to $450 | 7B to 14B models at Q4; the P40 fits 27B at Q4 slowly |
| Single-3090 tower | Used RTX 3090 24 GB (about $1,400), used Ryzen 5 or Core i5 desktop or a new B550/B760 build with 32 GB DDR4, 750 W PSU, 1 TB NVMe, airflow case | $2,000 to $2,300 | 27B to 32B at Q4 at conversational speed; 70B with partial CPU offload at reading speed |
| Dual-3090 tower | Two used RTX 3090s, a board with two x16 slots at x8/x8, 1,200 W PSU, full tower case | $3,500 to $4,000 | 70B at Q4 entirely in 48 GB of VRAM |
| Prebuilt unified memory | GMKtec EVO-X2 128 GB (about $3,500 to $3,650) or NVIDIA DGX Spark 128 GB (ASUS Ascent GX10, $6,950) or 64 GB ($4,999) | $3,500 to $6,950 | 70B to 120B-class models at Q4 at reading speed, no GPU to buy |
| Flagship single card | New RTX 5090 32 GB ($4,999 new, $3,822 used) in the single-3090 platform above | $4,600 to $6,000 | 32B models at the fastest consumer speed available; still too small for 70B without offload |
The flagship row is there for completeness, not as a recommendation. At current prices a used 5090 costs more than two used 3090s and holds 16 GB less memory.
What is the best AI server for home use?
The best AI server for most homes in October 2026 is a tower with one used RTX 3090, about $2,000 to $2,300 complete, because it runs every 27B to 32B model at conversational speed and can take a second card later for 70B. If you need more than 48 GB in a quiet desktop and can accept about a quarter of the speed, a 128 GB Strix Halo box such as the GMKtec EVO-X2 is the best prebuilt AI server PC.
The picks by situation:
- Lowest cost, already own a desktop: add a Tesla P40 or RTX 3060 12 GB.
- Best speed per dollar: single or dual used RTX 3090.
- Most memory in a small quiet box: GMKtec EVO-X2 128 GB, or a DGX Spark if you want NVIDIA's CUDA stack and the ability to cluster up to four units.
- Cheapest 48 GB: a used two-socket rack server with two P40s, if you have a basement for it.
Which used GPU is best for a local AI server?
The used RTX 3090 is the best GPU for a local AI server in October 2026 because it is the cheapest card with 24 GB of VRAM and 900 GB/s or more of bandwidth. Nothing newer matches its memory per dollar, and nothing cheaper matches its speed.
| Card | VRAM (GB) | Bandwidth (GB/s) | Board power (W) | Price (USD, Oct 2026) | Notes |
|---|---|---|---|---|---|
| NVIDIA Tesla P40 (used) | 24 | 346 | 250 | about $275 | No fan, no display output, needs a shroud and an 8-pin EPS adapter. Slow but the cheapest 24 GB |
| NVIDIA RTX 3060 12 GB (new) | 12 | 360 | 170 | about $300 | Plug-and-play entry card; limited to 7B to 14B models |
| NVIDIA RTX 5060 Ti 16 GB (new) | 16 | 448 | 180 | about $600 | Low power, fits 14B at Q4 with room for context |
| Intel Arc Pro B60 24 GB (new) | 24 | 456 | 120 to 200 | about $650 | Needs Intel's IPEX-LLM or recent llama.cpp SYCL builds; fewer rough edges each month, still more setup than NVIDIA |
| AMD RX 7900 XTX (used) | 24 | 960 | 355 | about $950 | Fast and 24 GB, but ROCm support on consumer cards still requires more fiddling than CUDA |
| NVIDIA RTX 3090 (used) | 24 | 936 | 350 | about $1,400 | The default pick. Two cards fit a 70B model |
| NVIDIA RTX 4090 (used) | 24 | 1,008 | 450 | about $3,000 | Same memory as a 3090 for twice the price; production ended, so prices keep rising |
| NVIDIA RTX 5090 | 32 | 1,792 | 575 | $4,999 new, $3,822 used | Fastest consumer card; its 16-pin connector is best fed by a native ATX 3.1 supply rather than an adapter. The ASUS TUF version is the usual alternate when the Gigabyte card is out of stock |
When buying a used 3090, ask the seller for a screenshot of a memory stress test (OCCT or memtest_vulkan) and check that the thermal pads on the backside memory were replaced; the original pads on many 3090s let the GDDR6X run past 100 C under sustained load. Mining-era cards are fine if the fans spin without rattling.
Should you build an AI server or buy a prebuilt one?
Build a tower if you want the most memory per dollar and can live with a large, warm box; buy a prebuilt unified-memory machine if you need more than 48 GB in a quiet desktop and are willing to accept about a quarter of the speed per model size.
The trade is bandwidth. Two RTX 3090s give 48 GB at 936 GB/s per card for about $3,800 all in. An EVO-X2 gives 128 GB (96 GB assignable to the GPU) at 256 GB/s for about $3,500. The DGX Spark gives 128 GB at 273 GB/s for $6,950, plus 200 Gbps networking to cluster up to four units, which is why it costs twice as much as the AMD box for the same memory. The Mac Studio M5 Ultra with 512 GB is a different class and has its own article: M5 Ultra Mac Studio, wait or buy.
A used enterprise server is the third route: a two-socket Dell PowerEdge R730 or R740 with two Tesla P40s lands under $1,000, has the airflow those cards need, and holds 48 GB of VRAM. It also idles near 150 W, weighs about 30 kg, and sounds like a shop vacuum. Put it in a garage or basement, never a bedroom.
How do you turn a spare computer into an AI server?
Any desktop from 2018 or later with a free PCIe x16 slot, a 500 W or larger power supply, and 16 GB of RAM becomes a usable AI server by adding one GPU and installing Linux. The whole job takes an afternoon.
- Confirm the power supply has the right GPU cable: one 8-pin PCIe connector for a 3060, two for a 3090. A P40 uses a CPU-style 8-pin EPS plug, so order the adapter with the card.
- Install the card, then install Ubuntu Server 24.04 LTS from a USB stick. Choose the minimal install and enable OpenSSH so you can manage it from another computer.
- Install the NVIDIA driver with
sudo ubuntu-drivers installand reboot. Runnvidia-smi; it should list the card and its memory. - Install Ollama with its one-line script, then pull a model sized for your VRAM:
ollama pull gemma4:12b(8 GB download) for a 12 GB card, orollama pull gemma4:26b(18 GB) orollama pull qwen3.8:27bfor a 24 GB card. - Run Open WebUI in Docker for a chat interface the rest of the house can use at the server's LAN address.
Our mini PC guide walks through the same Ubuntu, Ollama, and Docker steps in more detail, including Docker sandboxing.
How much power does a local AI server use?
A single-3090 server draws about 80 W idle and 400 to 450 W while generating text, which works out to roughly 80 kWh a month if it runs two hours a day under load and idles the rest. At 17 cents per kWh that is about $14 a month.
The math: 0.45 kW for 2 hours (0.9 kWh) plus 0.08 kW for 22 hours (1.76 kWh) is 2.66 kWh a day, or about 80 kWh over 30 days. A dual-3090 box roughly doubles the load figure. Power-limit each 3090 to 280 W in nvidia-smi; it costs about 5 percent in tokens per second and cuts heat and noise noticeably. Set the server to suspend after an idle timeout if you only use it in the evening.
How do you keep a local AI server from exposing your home network?
A local AI server should never be reachable from the internet. Keep Ollama bound to the LAN or localhost, do not forward ports on your router, and reach it from outside the house through WireGuard or Tailscale, never a public port.
Ollama listens on port 11434 with no authentication. Anyone who can reach that port can run prompts, pull models, and read what is loaded. Open WebUI adds logins, but the Ollama port behind it still needs to stay off the WAN. Put the server on its own VLAN or guest network so a compromised model runner or plugin cannot see your laptops and cameras, point its DNS at Pi-hole so you can watch every outbound connection it makes, and read our MCP security guide before you give any model tools that touch files or the web. Our Pi-hole setup guide covers the DNS side.
One more monthly fee to kill
Renting your modem? That's $120+ a year.
Own it instead. It pays for itself in months. 90-day warranty and 30-day returns.
Pick your provider:
FAQ
Can you run a 70B model on a local AI server?
Yes. A 70B model at 4-bit quantization needs about 40 GB of memory, which fits on two 24 GB GPUs, on a 64 GB or 128 GB unified-memory machine, or on a 48 GB workstation card. On a single 24 GB card it runs with part of the model in system RAM at a few tokens per second. Our 70B on a mini PC article has measured figures for the unified-memory route.
Where can you buy an AI server for home use?
Prebuilt home AI servers are sold as mini PCs and small desktops: Strix Halo machines from GMKtec, Beelink, Framework, and MINISFORUM on Amazon and direct, the DGX Spark through ASUS, Dell, HP, Lenovo, MSI, Acer, and Gigabyte, and Mac Studio from Apple. There is no retail category called "AI server" below the enterprise level; the search term brings up rack systems that cost as much as a house. For a tower build, the GPU comes from eBay or a used-market seller and the rest from any PC parts retailer.
Is a used AI server worth buying?
A used two-socket rack server with Tesla P40 cards is the cheapest way to get 48 GB of VRAM, usually under $1,000 complete, but it is loud, heavy, and idles near 150 W. It makes sense in a basement or garage and for people who want a learning platform, not for a living space.
Do you need a server CPU like Xeon or EPYC for AI?
No. For GPU inference the CPU only loads the model and runs the web interface. Server CPUs help in one case: more PCIe lanes for three or four GPUs. For one or two cards a desktop Ryzen or Core chip is fine.
Can you build a local AI server without a GPU?
Yes, with unified memory or a lot of system RAM. Strix Halo mini PCs, the DGX Spark, and Apple Silicon run 70B-class models from shared memory with no graphics card. A plain desktop with 64 GB of DDR5 can also run them on the CPU, at one to three tokens per second, which is fine for batch jobs and too slow for chat.
How much VRAM do you need for local AI?
12 GB runs 7B to 14B models, 24 GB runs 27B to 32B models, 48 GB runs 70B, and 96 GB or more runs 120B-class mixture-of-experts models, all at 4-bit quantization with room for a few thousand tokens of context. Our LLM VRAM calculator checks a specific model against a specific memory size.
Is it safe to leave a local AI server running all the time?
Yes, provided it sits on its own VLAN, no router port is forwarded to it, and it reaches updates through DNS you monitor. A GPU server running 24/7 with an idle timeout costs a few dollars a month in electricity and is no more risky than a NAS, as long as the model runner is never exposed to the internet.
