Can I Run AI Locally? LLM VRAM Calculator

A grade rates how well a model runs on your hardware, never how smart it is. The score column (from the Artificial Analysis Intelligence Index) covers intelligence.

• S, runs great: a 4-bit or better file fully in GPU memory at 30 or more tokens per second.
• A, runs well: 15 or more tokens per second with a 4-bit or better file and the context you asked for.
• B, usable: readable speed, or a 3-bit file, or a long prompt that takes over a minute to read.
• C, slow or compromised: a 2-bit file, a model split layer by layer between GPU and RAM, or prompt reading over three minutes.
• D, barely runs: under 4 tokens per second, or a 1-bit file.
• F: does not fit.

For image and video models the grade comes from the time per image (S under 5 seconds, A under 15, B under 45, C under 150) or per clip of about five seconds (S under a minute, A under 3 minutes, B under 10, C under 30). Streaming weights from RAM, or 3-bit and 2-bit files, cap the grade.

Memory use is the model file plus the context cache plus a runtime buffer. File sizes come from the actual downloads on Hugging Face. The cache size comes from each model's attention layout: models that mix sliding-window, linear or recurrent layers with full attention need far less cache per token than older designs. Graphics cards lose about 0.6 GB to the driver; Macs give the GPU two thirds of memory (up to 32 GB) or three quarters (above that) unless you raise the limit; Strix Halo mini PCs give the GPU what Linux or Windows allows.

When a model does not fit on the GPU, the calculator tries keeping a mixture-of-experts model's experts in system RAM (llama.cpp's --n-cpu-moe, counted in whole layers) and then splitting a dense model's layers between GPU and RAM.

Writing a reply is limited by memory bandwidth: every token reads the active weights once. We divide the bytes read per token by each device's real-world bandwidth and add a small fixed cost per layer, calibrated against published llama.cpp results. Across 39 measured runs on NVIDIA, AMD, Intel, Apple, Strix Halo and DGX Spark hardware, the median error is 9%, and none is off by more than 30%.

Reading a prompt (the wait before the first word) is limited by compute. We start from each device's measured llama.cpp prompt speed and scale it by the model's active parameters and attention cost. When weights sit in system RAM, llama.cpp copies them to the GPU for each batch of prompt tokens, which is why our commands set -ub 2048. The median error across 56 measured runs is 9%.

Images and video are limited by compute. We take measured ComfyUI timings (for example FLUX.1 [dev] at 20 steps: 26 seconds on an RTX 3090, 11 on a 4090, 9 on a 5090) and scale them by each device's tensor throughput and platform efficiency. These are rougher, about plus or minus 50%, and models without a published timing are marked.

Model sizes, layer counts and licenses come from each model's Hugging Face repository and files. Intelligence scores come from the Artificial Analysis Intelligence Index (scores marked "est." are Artificial Analysis estimates). Speeds are calibrated on llama.cpp's public benchmark threads, the ComfyUI GPU benchmark thread, AMD and NVIDIA technical blogs and independent reviews. Every figure was checked in October 2026; the date at the top shows the last full refresh.

Nothing runs until you press it. It reads the graphics name your browser already exposes to every web page through WebGL, your processor's thread count and, in Chrome and Edge, a rounded memory figure. It shows you exactly what it read, stores nothing, sends nothing anywhere and sets no cookies. Privacy-focused browsers such as Brave, Firefox with resist-fingerprinting and Tor Browser hide these details on purpose; the calculator then asks you to pick your hardware instead. Your choices are kept only in the page address, so a copied link reproduces your view.

Small models such as Gemma 4 E4B, Qwen3.5 9B and Llama 3.1 8B fit entirely on an 8 GB card at 4-bit. With 32 GB of system RAM you can also run mixture-of-experts models such as Qwen3.6-35B-A3B and gpt-oss-20b by keeping their experts in RAM: about 40 to 50 tokens per second on an RTX 4060 by our estimate. For images, FLUX.2 [klein] 4B and Z-Image-Turbo take about 10 seconds an image by our estimate; video is possible but slow.

Llama 3.3 70B's 4-bit file is 42.5 GB, so it needs about 48 GB of GPU memory with a 16K context: two 24 GB cards, a 48 GB workstation card, or a Mac or mini PC with 64 GB or more. A 32 GB RTX 5090 can run its 3-bit file (27.5 GB). Newer models such as Qwen3.8-27B score higher than Llama 3.3 70B with about a third of the memory.

A graphics card is faster for anything that fits on it: an RTX 5090 moves 1,792 GB/s versus 614 GB/s on an M5 Max, and NVIDIA cards read prompts and generate images several times faster. A Mac (or a Strix Halo or DGX Spark mini PC) holds far bigger models, up to 512 GB of unified memory on a Mac Studio, so it runs models no single graphics card can. M5-generation Macs read prompts about three times faster than M4 Macs.

A mixture-of-experts (MoE) model has many expert blocks but uses only a few for each token. Qwen3.6-35B-A3B has 35 billion parameters but reads only about 3 billion per token. That means its experts can sit in slower system RAM while the shared parts stay on the GPU, and it still runs fast. Dense models read every parameter for every token, so they slow down sharply when they spill out of GPU memory.

They describe how the model's numbers are compressed (quantized). Q8_0 keeps about 8 bits per weight and is nearly identical to the original. Q4_K_M keeps about 4.8 bits, roughly halving the size with a small quality loss; it is the usual sweet spot. Q3 and Q2 files are smaller still but lose noticeable quality, and 1-bit files are experimental. MXFP4 and NVFP4 are newer 4-bit formats; some models, like gpt-oss, are trained in them, so they lose nothing.

Writing a reply depends on memory bandwidth, where Macs and mini PCs do well. Reading a prompt depends on raw compute, where a big graphics card is many times faster. A long document or a coding agent's context can take a minute or more before the first word on a large dense model, which is why the calculator shows a "slow prompt reading" tag and lowers the grade when an 8K-token prompt takes over a minute.

Yes. Once the model file is downloaded, Ollama, llama.cpp, LM Studio and ComfyUI run entirely on your machine, and your prompts, documents and images never leave it. If you expose a local AI server to your network, our local AI network security guide covers how to lock it down.