FP16 Precision and VRAM: Why Half Precision Isn't Half Quality

TL;DR: FP16 stores model weights in 2 bytes instead of FP32's 4 bytes — that's half the VRAM for essentially zero quality loss on inference. All RTX 20, 30, and 40-series cards support native FP16 tensor cores. Use FP16 (or BF16 on RTX 40-series) as your default dtype and drop to quantization only when the model still doesn't fit.

What Is FP16 Precision? (The 30-Second Explanation)

FP16 (16-bit float, or "half precision") stores each model weight in 2 bytes instead of FP32's 4 bytes. Same number, half the storage, slightly less numeric precision.

The bit layout: 1 sign bit + 5 exponent bits + 10 mantissa bits. Compare that to FP32's 1 sign + 8 exponent + 23 mantissa — FP16 has a smaller numeric range (±65,504 vs ±3.4×10³⁸) and lower fractional precision.

For practical purposes: FP32 is a bank account balance stored to 10 decimal places. FP16 stores it to 4 decimal places. For a balance of $42,500.00, both round to the same answer. The precision gap only matters for very large or very small values — not the typical range of neural network weights.

Why FP16 Matters for Local LLM Builders

Every billion model parameters takes 4 GB in FP32 and 2 GB in FP16. That math compounds fast:

Q4_K_M

4.9 GB

4.4 GB

8.1 GB Llama 3.1 8B in FP32 is 32 GB — no single consumer GPU fits it. In FP16 it's 16 GB — fits on an RTX 4080. In Q4_K_M it's 4.9 GB — fits on any GPU with 6 GB+.

Anyone running FP32 for local inference is wasting 2× the VRAM for zero real-world benefit. This happens because some tools default to FP32, and users don't notice until they hit OOM on a model that "should" fit.

FP16 vs FP32 Quality — The Real Comparison

Difference

−0.1 (unmeasurable)

Statistically indistinguishable The reason the gap is essentially zero: model weights were trained in mixed precision (mostly FP16/BF16). Upcasting to FP32 adds no information — the weights never had FP32 precision to begin with.

The only scenario where FP16 genuinely hurts quality: models that were originally trained exclusively in FP32. That's rare for modern LLMs. Every major model released in the last two years was trained in FP16 or BF16.

FP16 vs FP32 Speed on RTX 30/40-Series

FP16 Advantage

4× faster

2× faster Note: RTX 4090 FP16 peaks at 330 TFLOPS with 2:4 structured sparsity enabled, but the dense (real-world) figure is 165 TFLOPS.

For inference specifically, the speedup is less than 4× because inference is memory-bandwidth-bound, not compute-bound. But FP16 still wins — loading half as much data per weight is a direct throughput improvement regardless of compute headroom.

How FP16 Compares to BF16 and When to Use Each

FP16 and BF16 are both 16-bit (2-byte) formats — same VRAM footprint, different bit layout. The VRAM math is identical; the practical difference comes down to numeric range and hardware generation support.

The key difference: FP16 has a smaller numeric range (±65,504). BF16 keeps FP32's full numeric range (±3.4×10³⁸) by using 8 exponent bits but cuts mantissa to 7 bits. BF16 trades fractional precision for range; FP16 does the opposite.

For inference only, this distinction barely matters — model weights are fixed and you're doing a single forward pass, not accumulating errors across gradient updates.

FP16 — Best for RTX 20 and 30-Series

• RTX 20-series (Turing): Native FP16 tensor cores at full speed. BF16 has no native support — falls back to FP32 internally, losing the memory bandwidth advantage.
• RTX 30-series (Ampere): Native FP16 AND BF16 tensor cores, both at full speed. FP16 has slightly better tooling compatibility in older llama.cpp builds.
• GTX 10/16-series: FP16 is supported but there are no tensor cores — runs at FP32-equivalent speed, just halves memory footprint.

Default choice for RTX 20/30: use FP16. BF16 works but FP16 has broader compatibility.

BF16 — Best for RTX 40-Series and A100/H100

• RTX 40-series (Ada Lovelace): BF16 tensor cores run at full speed, equal to FP16. More numerically stable for activation-heavy workloads.
• Model releases: Llama 3, Mistral, Qwen 2.5, Gemma 2 — all officially released in BF16. That's their native format; using BF16 for inference is technically correct.
• A100/H100: BF16 is substantially faster than FP16 due to architectural optimization. BF16 is the datacenter standard.

Default choice for RTX 40: use BF16. On RTX 30, FP16 is equally good and slightly more compatible with older inference tools.

Mixed Precision — FP16 for Activations, INT4/INT8 for Weights

Modern inference doesn't have to be all-or-nothing. A common pattern:

• Weights stored in INT4 or INT8 (quantized to save VRAM)
• Activations computed in FP16 (intermediate calculations)

This is what Q4_K_M in GGUF actually means under the hood: INT4 weights + FP16 activations. Low VRAM, fast decode, good quality. Pure FP16 (weights and activations both FP16) is only necessary when you need maximum quality and have the VRAM to spare.

"Half Precision Means Half the Quality" — Common Misread

The naming is the problem. "Half precision" sounds like you're getting half the model quality. You're not.

Correction 1: "Half precision" refers to storage format precision, not output quality. Model output quality depends on training quality, model size, and architecture — not dtype alone. A better-trained FP16 model will outperform a poorly-trained FP32 model every time.

Correction 2: The precision loss in FP16 (fewer mantissa bits) causes problems during training — specifically gradient underflow and numerical instability in certain operations. It's harmless during inference where weights are fixed and you're doing a single forward pass.

Correction 3: Meta, Mistral AI, and Google all release their flagship models in FP16 or BF16 by default. If half precision degraded quality meaningfully, they wouldn't.

Where the confusion comes from: FP16's limited numeric range can cause NaN/Inf overflow in training, which is exactly why BF16 was invented — to fix that instability. This training-specific problem gets incorrectly generalized to inference quality concerns.

FP16 in Practice — Choosing Dtype on Different GPUs

RTX 4070 (12 GB)

Llama 3.1 8B FP16 = 16 GB — 4 GB over the limit. Won't load fully.

Practical ladder for this card:

1. Try Q8_0 first (8.6 GB, near-identical quality to FP16)
2. If VRAM is tight with context, drop to Q5_K_M (5.7 GB)
3. Default workhorse: Q4_K_M (4.9 GB, minor quality tradeoff that most users can't detect)

FP16 on a 4070 is only viable for models 7B and smaller.

RTX 4080 (16 GB)

Llama 3.1 8B FP16 = 16 GB exactly — zero headroom. Any context window will push it over.

Better approach: use Q8_0 for stability. Reserve FP16 for models 7B and under, where you have headroom for context.

RTX 4090 (24 GB)

The FP16 sweet spot. Llama 3.1 8B FP16 (16 GB) + 8K context KV cache (~1.5 GB) = ~17.5 GB. Comfortable.

Alternatively: use the 24 GB for a 13B model in Q8_0 (14 GB) — you get a bigger model at near-FP16 quality vs a smaller model at FP16.

How to run FP16 in Ollama:

# Pull and run the FP16 variant directly by tag
ollama pull llama3.1:8b-instruct-fp16
ollama run llama3.1:8b-instruct-fp16

# Monitor VRAM while it loads
watch -n 1 nvidia-smi

Try both FP16 and Q8_0. Compare VRAM usage and response quality on your specific use case. For most tasks you won't notice a difference — and Q8_0 gives you more context headroom on the same card.

Related Concepts for Local AI Builders

• BF16 — FP16's cousin. Same VRAM footprint, better numeric range. Preferred on RTX 40-series and all newer GPUs.

• Quantization / bits per weight — The next step after FP16 when you need to shrink models further. Q8_0 is the closest to FP16 quality; Q4_K_M is the practical workhorse.

• VRAM — FP16 is the first lever for VRAM reduction before you touch quantization. Get this right before reaching for quantized formats.

• FP16 glossary entry — Canonical definition and technical spec reference.