Which Local LLMs Fit on a 16 GB GPU?

A 16 GB graphics card can run many local large language models, but the model’s parameter count alone does not tell you whether it will fit during use. The GPU also needs memory for the data the model uses while generating text. This includes a cache that stores information about tokens, or pieces of text, the model has already processed. Longer prompts and responses need more memory for that cache.

For most users, a 7–8B model stored with 4-bit quantization is the most comfortable starting point. Quantization stores the model’s weights, the values it uses to generate text, with fewer bits, so they take up less memory. A 12–14B model at 4-bit can also fit, but it leaves less room for long prompts and runtime overhead. The best choice depends on the model, its quantization format, the context length (the amount of text handled in one request), and the software that runs the model, called the runtime.

Start with a 7–8B model at 4-bit

A 7–8B model at 4-bit generally leaves more GPU memory available for prompts and other inference needs than a larger model does. For reference, llama.cpp lists a Llama 3.1 8B model in GGUF Q4_K_M, a 4-bit model format, at 4.9 GB. That figure describes the model file, not all the GPU memory required while generating text.

The Qwen3 measurements show the difference between lower-precision and quantized weights. In Qwen’s Transformers benchmark, Qwen3-8B used 15,947 MB in BF16, a 16-bit number format, with a one-token input. It used 6,177 MB in AWQ-INT4, a 4-bit quantization format. At an input length of 30,720 tokens, the AWQ-INT4 result rose to 11,001 MB. These are benchmark results for that setup, not guarantees for every 16 GB card or inference program. See the Qwen3 memory benchmark and llama.cpp’s quantized model-size examples.

A useful starting decision is to try a reputable 4-bit 7–8B model when you want room to work with longer prompts or avoid running close to the GPU’s capacity. Examples include Qwen3-8B and Llama 3.1 8B. A model’s published context limit describes the amount of text it supports in one request. It does not show how much text a particular GPU can handle at once. Qwen3-8B lists a native context length of 32,768 tokens and supports longer contexts with YaRN; Meta lists a native context limit for Llama 3.1. Neither limit guarantees that the full context fits in 16 GB.

A 12–14B model fits with less room to spare

A 12–14B model at 4-bit is a reasonable option when you want a larger model and can accept tighter memory limits. Qwen’s benchmark provides a concrete example: Qwen3-14B in AWQ-INT4 used 9,962 MB at an input length of one token and 12,438 MB at 14,336 tokens. At 30,720 input tokens, it used 15,323 MB, close to the capacity of a nominal 16 GB GPU.

That last result makes a 14B model a poor default for long-context use on a 16 GB card. The runtime also needs working memory, and other GPU activity reduces the memory available to the model. If you want to use a 14B model, begin with a moderate context and check whether your chosen runtime completes the prompts you expect to send. Shorten the context or choose a smaller model if the runtime runs out of memory.

The same benchmark measured Qwen3-14B in BF16 at 28,402 MB with an input length of one token. That configuration does not fit on a 16 GB GPU in the measured setup. Qwen3-8B in BF16 used 15,947 MB even at that short input length, leaving almost no practical margin.

Qwen’s figures come from its Transformers benchmark, whose cited table does not identify the GPU or full hardware and software setup. Treat them as reference measurements rather than a compatibility guarantee. AWQ-INT4 and GGUF Q4_K_M both reduce weights to roughly 4-bit representations, but they use different formats and implementations, so their memory use will not match exactly.

Context length uses memory beyond the weights

The model’s weights occupy only part of the GPU memory needed during generation. The runtime also stores a key-value cache, usually called the KV cache. The KV cache stores information from processed tokens so the model can use it to generate the next tokens. As the prompt and generated text grow, the cache grows too.

The context length is the total text window the runtime supports for a request, including the prompt and generated tokens. A model card’s maximum context length tells you about the model’s supported text window; it does not establish that the full window fits on your GPU. Qwen3-14B’s rising memory use across the benchmark’s input lengths illustrates this distinction.

Quantization comes in several formats. A model file’s size helps estimate how much memory its weights use, but it does not include all the memory used by the KV cache and the runtime. Compare measurements for the model format and runtime you plan to use, and leave spare GPU memory. The model-file size alone does not show how much video memory (VRAM) the model needs.

Check whether the model fully fits

“Fits” can describe different setups. A model fits fully in GPU memory when its weights and inference allocations stay on the GPU. Some runtimes can place part of the model or its work in system RAM instead of GPU memory. This technique is called CPU offload. Offload can let a larger model run, but it does not mean the model fits fully in VRAM, and performance depends on the hardware and runtime.

To test your setup, load the model with the quantization format and runtime you plan to use. Then try a prompt close to the length you expect to use. Watch for an out-of-memory error or a runtime that moves work to system memory. If either occurs, reduce the context length. Closing other GPU workloads is another option; you can also move to a smaller or more heavily quantized model. Your results will depend on the GPU’s usable memory and the runtime’s allocations.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com