The Math Behind Local LLM VRAM Requirements

Stop Guessing, Start Calculating: The VRAM Math Every Local LLM Developer Needs

If you have spent any time in the open-source AI community recently, you have probably seen someone excitedly announce they are running a 70B parameter model locally — only to follow up an hour later asking why their system crashed with an OOM (Out of Memory) error. The enthusiasm is understandable. The math illiteracy is not.

Deploying Large Language Models locally — whether for privacy, cost savings, or offline availability — is the new frontier for developers. But unlike spinning up an AWS EC2 instance for a standard web app, running LLMs on consumer or prosumer hardware requires a precise understanding of GPU memory. Get the math wrong, and your system does not just slow down. It crashes.

Here is the definitive guide to calculating exactly how much VRAM you need before you torch your GPU.

The Fundamental Formula: Parameters × Bytes Per Parameter

The core calculation is deceptively simple. Every parameter in a neural network occupies a certain number of bytes in memory. The total VRAM consumed by just loading the model weights is:

VRAM (bytes) = Number of Parameters × Bytes Per Parameter

For a model stored in full FP32 (32-bit floating point) precision, each parameter takes 4 bytes. For FP16 or BF16 (half precision), it is 2 bytes. For INT8 quantization, it is 1 byte. And for the increasingly popular 4-bit quantization formats like GPTQ or GGUF Q4, it is 0.5 bytes per parameter.

Let us run the numbers for Meta's Llama 3 70B model:

• FP32: 70 billion × 4 bytes = 280 GB VRAM
• FP16/BF16: 70 billion × 2 bytes = 140 GB VRAM
• INT8: 70 billion × 1 byte = 70 GB VRAM
• 4-bit (Q4): 70 billion × 0.5 bytes = 35 GB VRAM

Suddenly, that 'I will just run it on my RTX 4090' plan looks a lot more complicated. Even at 4-bit quantization, 35 GB exceeds the 4090's 24 GB VRAM ceiling. And these numbers only account for the model weights themselves.

The Hidden Memory Tax: KV Cache

Model weights are just the starting point. The real VRAM killer that catches most developers off guard is the KV (Key-Value) cache — the memory required to store attention states during inference.

Every time you generate a token, the model needs to remember all previous tokens in the sequence through key and value matrices. The formula for KV cache memory is:

KV Cache (bytes) = 2 × num_layers × num_kv_heads × head_dim × sequence_length × bytes_per_value

The factor of 2 accounts for both the key and value tensors. Let us break this down for Llama 3 70B with a 4,096-token context window at FP16 precision:

• Layers: 80
• KV heads: 8 (using Grouped Query Attention)
• Head dimension: 128
• Sequence length: 4,096
• Bytes per value: 2 (FP16)

KV Cache = 2 × 80 × 8 × 128 × 4,096 × 2 = ~1.34 GB

That seems manageable — until you scale up. Extend the context to 128K tokens (as Llama 3 supports), and the KV cache balloons to roughly 41.9 GB. For models without Grouped Query Attention that use the full number of attention heads for KV, these numbers multiply dramatically.

For a batch size greater than 1 — say, you are serving multiple users simultaneously — multiply the KV cache by the batch size. Serving 8 concurrent users with 128K context? That is over 335 GB just for the KV cache alone.

The Overhead Nobody Talks About

Beyond weights and KV cache, several other memory consumers eat into your VRAM budget:

CUDA Context and Framework Overhead: Simply initializing PyTorch with CUDA reserves between 300 MB and 1 GB of VRAM depending on your GPU and driver version. Libraries like vLLM, llama.cpp, or Hugging Face Transformers each add their own baseline memory footprint.

Activation Memory: During inference, intermediate activation tensors are computed and discarded layer by layer. For single-request inference, this is usually modest — roughly 50-200 MB for most architectures. But it scales with batch size and sequence length.

Memory Fragmentation: GPU memory allocators do not pack data perfectly. Expect 5-10% overhead from fragmentation, especially during long-running sessions where memory is repeatedly allocated and freed.

A practical total VRAM formula looks like this:

Total VRAM ≈ Model Weights + KV Cache + Activation Memory + Framework Overhead + 10% Fragmentation Buffer

Real-World Examples: What Actually Fits Where

Let us map this to actual hardware developers commonly use:

NVIDIA RTX 4090 (24 GB VRAM)

• Llama 3 8B at FP16 (16 GB weights): ✅ Fits with room for ~4K context
• Llama 3 8B at 4-bit GGUF (4.5 GB weights): ✅ Fits easily, room for long context
• Llama 3 70B at 4-bit (35 GB weights): ❌ Does not fit
• Mistral 7B at FP16 (14 GB weights): ✅ Comfortable fit

Dual RTX 3090 Setup (48 GB combined)

• Llama 3 70B at 4-bit (35 GB weights): ✅ Tight but workable with short context
• Llama 3 70B at INT8 (70 GB weights): ❌ Does not fit

NVIDIA A100 80 GB

• Llama 3 70B at FP16 (140 GB weights): ❌ Needs 2× A100s
• Llama 3 70B at INT8 (70 GB weights): ✅ Fits with modest context
• Mixtral 8x7B at FP16 (~90 GB weights): ❌ Does not fit on single card

Apple M2 Ultra (192 GB unified memory)

• Llama 3 70B at FP16 (140 GB weights): ✅ Fits using MLX or llama.cpp
• Llama 3 70B at 4-bit (35 GB weights): ✅ Generous room for long context

Apple Silicon deserves special mention here. While unified memory is significantly slower than dedicated VRAM for this workload (roughly 3-5× slower token generation), the sheer capacity of M2 Ultra and M4 Max chips makes them surprisingly viable for running models that would require multi-GPU setups on NVIDIA hardware.

Quantization: The Art of Strategic Precision Loss

Quantization is the single most impactful lever for fitting models into limited VRAM. But not all quantization is equal.

The GGUF format popularized by llama.cpp offers a spectrum of options. The naming convention tells you the bit width: Q4_K_M means 4-bit quantization with a 'medium' quality setting using the K-quant method. Here is a practical quality-to-size breakdown for a 7B parameter model:

• Q2_K: ~2.7 GB — Significant quality loss, only for experimentation
• Q4_K_M: ~4.1 GB — Best balance of quality and size for most users
• Q5_K_M: ~4.8 GB — Near-FP16 quality for many tasks
• Q6_K: ~5.5 GB — Minimal quality loss
• Q8_0: ~7.2 GB — Nearly indistinguishable from FP16
• FP16: ~14 GB — Full half-precision baseline

Research from the open-source community — including extensive benchmarking by users on r/LocalLLaMA — consistently shows that Q4_K_M retains roughly 95-97% of FP16 Perplexity scores for most general-purpose tasks. Below Q4, degradation becomes noticeable, particularly for reasoning and code generation.

Newer techniques like AWQ (Activation-aware Weight Quantization) and AQLM push quality even higher at low bit widths, but they require GPU-specific kernels and are not universally supported across inference engines.

The Offloading Escape Hatch

What if your model does not fit entirely in VRAM? Tools like llama.cpp and ExLlamaV2 support partial GPU offloading, where some layers reside in system RAM while others stay on the GPU.

The math here is layer-by-layer. A 70B model with 80 layers at 4-bit quantization uses roughly 437 MB per layer. If you have 24 GB of VRAM and need ~2 GB for overhead and KV cache, you can fit approximately 50 layers on GPU and offload 30 to system RAM.

The tradeoff is speed. Layers in system RAM process at DDR5 bandwidth (~50-60 GB/s) rather than GDDR6X bandwidth (~1 TB/s on a 4090). Expect 3-10× slower generation for offloaded layers. The result is a model that runs, but at significantly reduced tokens-per-second.

A Practical Pre-Flight Checklist

Before downloading that shiny new model, run through this checklist:

1. Count parameters — Check the model card on Hugging Face
2. Choose your precision — FP16, INT8, or 4-bit quantization
3. Calculate weight memory — Parameters × bytes per parameter
4. Estimate KV cache — Use the formula above with your target context length
5. Add 2 GB overhead — For CUDA context, activations, and framework
6. Add 10% buffer — For fragmentation and safety margin
7. Compare to your GPU VRAM — If total exceeds capacity, quantize further or plan for offloading

Looking Ahead: Why This Math Is Changing

Several emerging trends are shifting the VRAM equation. Techniques like PagedAttention (used in vLLM) dramatically reduce KV cache waste by borrowing virtual memory concepts from operating systems. Speculative decoding adds a small draft model's memory footprint but can double generation speed. And new architectures like Mamba and other state-space models promise linear memory scaling with sequence length instead of the quadratic scaling inherent in traditional attention.

Meanwhile, NVIDIA's upcoming consumer GPUs are rumored to push VRAM ceilings higher, and AMD's MI300X already offers 192 GB of HBM3 for data center deployments. The hardware is catching up — but the math will always matter.

The bottom line is straightforward: local LLM deployment is not about hope and hype. It is about arithmetic. Do the math first, or your GPU will do it for you — by crashing.