What hardware do you need to run an LLM locally?

If you want to run a large language model on your own hardware, there is one question to answer before you look at a single spec sheet: how much fast memory does the model need, and does your box have it? Get that right and everything else is a refinement. Get it wrong and the most expensive machine in the room will crawl — or refuse to load the model at all.

This is a plain-language guide to sizing hardware for a local LLM. No benchmarks to memorize, no jargon for its own sake. By the end you will know how to read a model's size, translate it into a memory requirement, and match it to the right class of box. If you just want the shortlist with live prices, jump to the hardware page; if you want to understand the why first, read on.

Memory is the gatekeeper, not raw speed

People shopping for AI hardware instinctively look at the processor. That's the wrong place to start. The first thing a model does is load its weights into memory, and if it doesn't fit, nothing else matters — the model simply won't run, or it will spill into slow storage and grind.

So the rule is blunt: the model has to fit in fast memory. A 7-billion-parameter model and a 70-billion-parameter model are not "a bit different"; they are an order of magnitude apart in what they demand. The processor decides how fast the answer comes once the model is loaded. Memory decides whether you can load it at all. Capacity first, speed second.

The good news is that this makes shopping simpler. Pick the model you want to run, work out its memory footprint, and that number tells you which class of box you need. The rest of this guide is how to do exactly that.

Quantization, explained simply

Here is the trick that makes local AI practical: quantization. A model's raw weights are stored at full precision — typically 16 bits per number. A 70B model at full precision is around 140 GB, which is workstation-only territory.

Quantization shrinks each number to fewer bits with very little loss of quality. The common settings are easy to reason about:

• Q8 (8-bit) roughly halves the size — a 70B drops to about 70 GB.
• Q4 (4-bit) halves it again — that same 70B lands near 40 GB, and quality stays close to the original for most business tasks.

Think of it as "halve, then halve again." Q4 is the sweet spot most local deployments use: it is the difference between needing a server and needing a box that fits under a desk. Q8 vs Q4 is the one trade-off worth knowing — Q8 preserves a little more nuance on hard reasoning, Q4 buys you roughly half the memory and more speed. For customer Q&A, drafting and search, Q4 is almost always the right call. When in doubt, start at Q4 and only step up if you can measure a difference that matters to you.

How much memory does each model size need?

Once you apply 4-bit quantization, model sizes map cleanly onto memory requirements — and onto a class of hardware. This table is the heart of the whole decision:

A few things fall out of this immediately. A 7–14B model — easily enough to answer questions from your documents, draft replies and search your own knowledge — fits in any 32 GB mini PC with room to spare. A 70B, with near-frontier answer quality, needs around 40 GB, which is exactly why the serious single-box machines ship 128 GB of unified memory: that headroom leaves room for the model, the context, and the operating system all at once. And once you are past 100B or serving many tenants, you are into high-bandwidth workstation territory.

Notice you don't need to memorize benchmarks. You pick the answer quality you want, read across, and the right box class is in the last column.

VRAM vs unified memory vs system RAM

"Memory" isn't one thing, and the differences decide how fast your model runs.

• VRAM is memory soldered onto a discrete GPU (NVIDIA, or the RTX cards in our catalog). It is the fastest memory there is for inference — the RTX 5090 SFF moves data at 1,792 GB/s — but you only get what the card ships with. A 70B at 4-bit needs ~40 GB and simply will not fit in a 32 GB GPU, no matter how fast that GPU is.
• Unified memory is a single pool the CPU and GPU share, as on Apple Silicon and AMD "Strix Halo" boxes. It is the reason a compact machine can run a 70B at all: a 128 GB unified box can hand up to 96 GB to the GPU. It is not as fast as discrete VRAM, but it is large and efficient, which is the better trade for big models.
• System RAM is ordinary computer memory. You can technically "offload" part of a model into it when the GPU runs out, but it is far slower — the model stalls waiting on data, and tokens trickle out. Offloading to system RAM is a fallback, not a plan. If you find yourself relying on it, you bought too small a box for the model you want.

The headline: a model that fits entirely in VRAM or unified memory runs well; a model that spills into system RAM does not. Size the box so it never has to spill.

Bandwidth equals speed

Once the model fits, the number that decides how fast it answers is memory bandwidth — how many gigabytes per second the chip can read. More bandwidth means more tokens per second, which you feel directly as how quickly replies stream out.

The spread is wide. The Apple Mac Studio M3 Ultra leads any single box at 819 GB/s, which is why it is the fastest pure-inference machine for models that fit its unified memory. Discrete GPUs are faster still on their own VRAM — the RTX cards hit 1,792 GB/s — but only for models small enough to fit. The AMD Strix Halo boxes run around 256 GB/s, and the NVIDIA DGX Spark at 273 GB/s, both of which are perfectly usable for a 70B in a single box.

One caveat worth repeating: the NPU — the "TOPS" figure on the spec sheet — is Copilot+ laptop marketing and does the heavy lifting for almost none of this. Local LLM inference runs on the integrated or discrete GPU, not the NPU. Don't let a big TOPS number talk you into a box with too little memory. We dig into the silicon differences in AMD vs Intel vs Apple vs NVIDIA for local LLMs.

Don't forget the context window

There is a second, quieter memory cost: the context window — how much text the model holds in mind at once. Every token of context occupies memory (the "KV cache"), and a long context for a large model can add several gigabytes on top of the weights.

In practice this means you should size for the weights plus a working margin, not the weights alone. It is the other reason the 70B-capable boxes ship 128 GB rather than a tidy 48 GB: the extra room absorbs a generous context and concurrent requests without forcing a spill into slow memory. When you read "~40 GB for a 70B," read it as the floor, not the ceiling.

What to buy

Translate the table into a purchase and it gets simple. Most small businesses want a 7–14B model, which any entry mini PC runs comfortably. If you want several knowledge bases or a 70B for sharper answers, step up to a 128 GB unified box.

The Minisforum AI X1 Pro is our default entry pick — up to 96 GB of upgradeable memory gives the most room in its class. The GEEKOM A8 is the dependable value option with a three-year warranty and EU fulfilment. When you outgrow a single knowledge base, the Minisforum MS-S1 MAX loads a 70B in 128 GB of unified memory and is ready to cluster later. For the full field with current prices and disclosed buy links, see the hardware page — and for the deeper comparisons, the best mini PCs for local AI in 2026 and can a mini PC run a 70B model?.

A word on price: AI hardware has been unusually volatile through 2026 because of the memory shortage. Entry boxes start around ~€490 and performance machines run ~€2,400 and up — both indicative only. Always confirm the live figure on the hardware page, which we re-verify monthly.

The box is only half the product

Sizing the hardware is the easy half. A mini PC running a raw model is a confidential database that will read its own secrets aloud to anyone who asks — capacity does nothing to stop that. What makes a local AI safe to put in front of customers is the governance around it: the boundary between your internal knowledge and the public agent, and the loop that improves it. That layer, not the silicon, is the actual product. Local hosting strengthens your DSGVO and EU AI Act posture by keeping data in the building, but the governance is what makes that posture real.

So size the box to the model you want — then put a governed core on it. We make the full case in the product is governance and explain the sovereignty model in sovereign AI, explained. When you're ready to choose hardware, see the full shortlist.