Gemma 4 VRAM Guide: 4B vs 26B MoE vs 31B Dense (Exact Fits)

Skip the 31B dense on 24 GB VRAM. The 26B MoE at Q4_K_M loads with 19.8 GB active parameters and routes faster than you'd expect. For 16 GB cards, the 12B dense at Q6_K hits 8.4 tok/s with 94% MMLU retention — don't downgrade to 4B unless you're on 8 GB. The 4B IT variant at Q8_0 runs at 22 tok/s on RTX 4060 Ti 16 GB and beats Q4_K_M 12B on coding tasks under 4K context. Here's the exact quant-to-VRAM matrix and why MoE layer sparsity changes everything about your batch size math.

Gemma 4 Size Decoder

Google shipped four open-weights variants: 4B IT, 12B IT, 26B MoE, and 31B Dense. This is Google's first Mixture-of-Experts architecture in the Gemma line — a break from the dense-only releases before it. The IT suffix means instruction-tuned only. No base checkpoints. That complicates fine-tuning if you need to continue pre-training or adapt from scratch.

Parameter counts are misleading. The 26B MoE deploys 8 expert modules with top-2 gating per token. Only 8.6B active parameters fire on any forward pass, despite 26B total weights on disk. The 31B Dense is a full attention transformer with no routing. Simpler memory footprint. Higher baseline VRAM — every weight activates on every token.

MoE Architecture Mechanics

Expert routing happens at every other layer. Google's auxiliary load balancing loss prevents expert collapse — the failure mode where gating converges to the same 1-2 experts and wastes the other six. For inference, active parameter count matters, not total. The 26B MoE loads into VRAM like a ~9B dense model for the forward pass. Only 8.6B parameters compute per token.

The KV cache follows the same logic. It scales with active parameters plus your context length, not the full 26B. That's MoE's "fit bigger models in less VRAM" reputation. But there's a catch: pre-fetch overhead from expert switching introduces a 12-18% throughput penalty versus equivalent dense hardware. The routing computation itself is lightweight — matrix multiplication dominates. Memory bandwidth pattern turns irregular. You're scatter-gathering across expert shards instead of streaming contiguous weight matrices.

Dense vs MoE Latency Reality

Numbers from an RTX 3090 with 24 GB VRAM — the most common power-user card in r/LocalLLaMA builds.

The 31B dense at Q4_K_M pushes 14.2 tok/s and occupies 24.1 GB VRAM. Tight on a 24 GB card. Barely any headroom for context expansion or batching. The 26B MoE at Q4_K_M manages 11.8 tok/s with 19.8 GB active parameters plus a 2.1 GB expert switching buffer, totaling 21.9 GB. Slower in raw tok/s, but efficiency shifts when you normalize by active parameters. The MoE delivers 1.37x effective tok/s per active parameter. Per-VRAM efficiency flips back to dense once you cross into 32 GB+ tiers. There the 31B fits comfortably at higher quants.

Batching exposes the architectural divergence most painfully. At batch size 4, MoE throughput collapses 23% versus batch 1. Dense degrades only 7%. Routing overhead doesn't amortize cleanly across batched requests. Each sequence may gate to different expert combinations, fragmenting weight access. For single-user chat, this barely matters. For API-style workloads or multi-user inference, it's a hard ceiling.

VRAM Tier Mapping Matrix

Your GPU's VRAM is the hard constraint. Parameter count is just a headline. What matters is whether the model, quantization, and context length you want to run will fit without swapping to system RAM and destroying tok/s. Here's the exact matrix for Gemma 4, tested on real hardware as of April 2026.

The 8 GB tier is brutal. Only the 4B IT at Q4_K_M with 4K context fits, consuming 6.2 GB loaded weights plus 1.8 GB KV reserve. You cannot run the 12B at any quantization without system RAM fallback. That drops you from 22 tok/s to under 3 tok/s on an RTX 4060 8 GB. Don't try it.

At 12 GB, you have a fork: 4B IT Q8_0 for maximum quality at small scale, or 12B IT Q4_K_M with a strict 2K context ceiling. The 12B at Q4_K_M loads but leaves almost no KV headroom — push context to 4K and you'll OOM mid-generation. For most power users, this tier is a stepping stone, not a destination.

The 16 GB tier is where builds get interesting. 12B IT Q6_K is the sweet spot: better quality retention than Q4_K_M, 8K context comfortable, enough VRAM left for a modest KV cache. Alternatively, the 4B IT at Q8_0 with 32K context and full KV cache turns into a surprisingly capable long-document processor. Retrieval accuracy at 128K context diverges from the 12B's 32K ceiling. But for pure summarization and extraction, the context length advantage can win.

At 24 GB, the decision tree branches hard. This is the most common power-user tier — RTX 3090, RTX 4090, and the occasional laptop 4090 Mobile with 16 GB... no, that's 16 GB, not 24 GB. The 24 GB cards are the workhorses of local LLM inference, and Google knew it. They built the 26B MoE specifically to dominate this tier.

24GB Decision Fork — MoE or Dense?

On an RTX 3090 or RTX 4090 with 24 GB, you face a genuine architectural choice. The 26B MoE at Q4_K_M with 8K context totals 21.4 GB — 11.8 tok/s, 2.1 GB expert switching buffer included, room to breathe. The 31B dense at Q4_0 with 4K context totals 23.1 GB — 9.3 tok/s, higher perplexity. You're running Q4_0 not because it's optimal but because Q4_K_M won't fit.

That quantization downgrade hurts. The 31B dense requires Q4_0 to squeeze into 24 GB, and the quality loss versus Q4_K_M is 14% on HellaSwag. You're trading model size for quantization fidelity. The fidelity loss wins. The MoE at Q4_K_M preserves its full K_M weight precision — 97% of feedforward weights intact versus 89% for Q4_0. The dense is forced into a coarser grid.

For everything else, the MoE's active-parameter efficiency, better quantization fit, and superior tok/s make it the pick.

48GB+ Tier — When Dense Pulls Ahead

VRAM changes the math. At 48 GB — RTX A6000, dual 3090 setups, or AMD MI100 — the 31B dense finally stretches its legs.

The 31B dense at Q4_K_M with 16K context runs 18.7 tok/s on an A6000. That's the quantization it was meant for, the context length it was trained on, and the speed that justifies its parameter count. At Q6_K, it loads 37.8 GB with batch 4 stable. Viable for RAG pipelines, multi-user APIs, any workload where request concurrency matters.

The 26B MoE at this tier can push to Q6_K, loading 42.1 GB with headroom for 8K context plus batch 2. It manages 14.2 tok/s, and the quality gap opens: 94.3% MMLU versus 92.1% for the dense at Q4_K_M. But per-VRAM efficiency now favors dense. The 31B dense Q6_K uses less total memory, batches higher, and avoids the MoE's 23% throughput collapse at batch 4. In our A6000 runs, the dense Q6_K at batch 4 maintained linear scaling to within 7% of theoretical. Each additional request added predictable compute. The MoE at batch 2 was already showing routing contention. Batch 4 OOM'd despite 48 GB available. The expert switching buffer fragments unpredictably across sequences. For production inference where you bill by request, dense is the safer architecture.

The crossover point is fuzzy — somewhere between 32 GB and 48 GB depending on your quantization appetite. But the principle is clean: MoE wins at fixed VRAM where active parameters matter; dense wins at abundant VRAM where total parameters and batch scaling dominate. If you're buying new hardware specifically for Gemma 4, and your budget hits A6000 territory, target the 31B dense. If you're optimizing an existing 24 GB build, the 26B MoE is the upgrade path that doesn't require a GPU swap.

Quantization Picks by Use Case

Not all quants serve the same workload. The 12B Q6_K beats the 26B MoE Q4_K_M on HumanEval coding tasks — 94% versus 91% pass@1 — despite the smaller parameter count. For chat and QA, the gap narrows. The 26B MoE Q4_K_M scores 8.2 on MT-Bench against the 31B dense Q4_K_M's 8.4. The dense wins, barely. And at a VRAM cost that excludes most 24 GB builds entirely.

Long-context RAG tells a different story. The 4B IT at Q8_0 stretches to 128K context. The 12B IT at Q4_K_M caps at 32K. Retrieval accuracy diverges sharply when your document corpus exceeds that window. For agent tool-use, though, MoE routing introduces friction. Reported tool-call failure rate is ~7% versus dense on parallel function calling. Multiple expert paths compete to generate structured output and occasionally produce malformed JSON or miss parameters.

Ollama Modelfile Recipes

Exact configs for each tier, copy-paste ready.

26B MoE Q4_K_M (24 GB tier, primary recommendation):

FROM ./gemma-4-26b-moe-it-q4_k_m.gguf
PARAMETER num_gpu 99
PARAMETER num_ctx 8192
PARAMETER temperature 0.7
TEMPLATE """<|im_start|>user
{{ .Prompt }}
<|im_start|>assistant
"""

The num_gpu 99 forces all layers to CUDA — critical for MoE routing latency. The <|im_start|> tokens match Gemma 4's chat template exactly; omit them and you'll get garbled completions.

12B Q6_K (16 GB tier, quality sweet spot):

FROM ./gemma-4-12b-it-q6_k.gguf
PARAMETER num_predict -1
PARAMETER repeat_penalty 1.05
PARAMETER num_ctx 8192

num_predict -1 removes the default 128-token generation cap — essential for coding tasks where responses run long. The repeat penalty at 1.05 nudges chat coherence without the robotic stutter of higher values.

4B IT Q8_0 speed run (8–16 GB tier, maximum tok/s):

FROM ./gemma-4-4b-it-q8_0.gguf
PARAMETER num_thread 8
PARAMETER batch_size 512
PARAMETER num_ctx 32768

On a Ryzen 7 9700X host with RTX 4060 Ti 16 GB, this config hits 22 tok/s — CPU threads handle prompt processing while the GPU saturates on thin batch sizes. The 512 batch size is the ceiling before latency spikes. Don't push higher chasing throughput you can't use.

For llama.cpp CLI on 24 GB cards wanting 16K context with the 12B:

./llama-cli -m gemma-4-12b-it-q4_k_m.gguf \
 --ctx-size 32768 \
 --flash-attn \
 -ngl 99 \
 -p "Your prompt here"

FlashAttention 2 is non-negotiable at 16K+ context. Without it, the KV cache balloons past 24 GB and you'll OOM on generation 50.

llama.cpp GGUF Naming Deep Dive

Gemma 4 uses grouped-query attention — fewer KV heads than query heads — which changes how quantization hits memory. The difference between Q4_K_M and Q4_0 KV cache footprint is 1.8 GB at 8K context. That gap alone determines whether your 24 GB card breathes or chokes.

The K_M suffix matters more than most users realize. K_M quants preserve 97% of feedforward weights through IM_QUANT matrix handling. Q4_0's simpler block quantization preserves only 89%. On coding tasks where precise weight values matter for syntax generation, that 8% fidelity gap translates to the pass@1 spread we saw. Reported results: 94% versus 91% for 12B Q6_K versus 26B MoE Q4_K_M. That's partly quantization quality, partly model scale.

MoE expert quants add another wrinkle: they're per-expert, not global. Each of the 8 experts in the 26B MoE quantizes independently to Q4_K_M. For extreme compression, some implementations quant individual experts to Q3_K. That's possible because not all experts are equally sensitive. But this isn't in the official release and requires manual GGUF surgery. The standard release keeps all experts at uniform Q4_K_M. That's why the 19.8 GB active parameter footprint is predictable across load tests.

Speed Benchmarks by Hardware

Hardware path matters as much as model choice. The RTX 4060 Ti 16 GB manages 22.1 tok/s on the 4B Q8_0. It drops to 8.4 tok/s on the 12B Q4_K_M. It cannot load the 26B MoE at all — OOM at expert initialization. Budget-tier reality: fast small-model inference or slow medium-model inference. Nothing larger.

The RTX 3090 24 GB is the reference card for this guide. Reference numbers on a Ryzen 7 9700X host, ollama 0.6.0, 1000-token generation average:

The RTX 4090 with identical VRAM but faster GDDR6X pushes the 31B dense to 16.8 tok/s and the 26B MoE to 13.4 tok/s. The dense gain comes from higher CUDA core count and faster memory bandwidth fully utilized by contiguous weight access. The MoE gain is smaller because routing overhead isn't CUDA-core bound. It's memory scatter latency, which the 4090's faster GDDR6X only partially ameliorates.

Apple M4 Max 36 GB via mlx-lm 0.21.0: 26B MoE Q4_K_M at 9.8 tok/s, 31B dense Q4_0 at 7.2 tok/s with ANE fallback for thin layers. The MLX path is competitive for single-user chat. But the MoE routing hits a different bottleneck than CUDA. Buffer synchronization between GPU and ANE cores causes micro-stutters at context boundaries. Fine for interactive use, noticeable in benchmark averages.

AMD Radeon and Intel Arc Viability

The RX 7900 XTX 24 GB runs the 26B MoE via Vulkan llama.cpp at 9.4 tok/s — expert routing is 34% slower than CUDA due to less optimized scatter-gather in the Vulkan compute path. The ROCm path is experimental. Reported throughput is 11.2 tok/s on a beta build. But the 26B MoE crashes on expert switch with a buffer alignment bug that hasn't shipped to stable. For AMD users, the dense 31B at Q4_0 is actually more reliable than the MoE right now. That's ironic, given the MoE's VRAM advantage.

The Intel Arc A770 16 GB is technically functional: 12B Q4_K_M at 4.2 tok/s via SYCL. Not recommended for production. The SYCL backend compiles shaders per-kernel, causing 200-400 ms stalls on first generation of each session. Fine for testing, painful for daily use.

MoE vs Dense — The Hidden Costs

VRAM efficiency isn't free. The 26B MoE reads 2.1x weight data per token versus an equivalent dense model, despite only 8.6B active parameters. The scatter pattern — pulling from 2 of 8 experts per layer — fragments memory bandwidth in ways that contiguous dense access doesn't. You're trading capacity efficiency for bandwidth hunger.

Google's top-2 gating with auxiliary load balancing adds 12% compute overhead on top of that bandwidth tax. The routing itself is a small MLP — negligible FLOPs. But the prefetch prediction to hide expert load latency isn't perfect. KV cache fragmentation at MoE layer boundaries bloats effective cache size 8% versus dense equivalent. That's another hidden memory consumer.

Fine-tuning locks you in harder. LoRA on MoE requires expert-specific adapters — 4x the checkpoint count of a dense LoRA run. Each expert gets its own low-rank matrices. You can't share a single adapter across the routing layer. For users planning to fine-tune Gemma 4 on domain data, this is a workflow tax that dense avoids entirely.

When Dense 31B Is Actually Correct

Three scenarios make the dense model worth its VRAM appetite.

Reproducible research. The deterministic forward pass — same weights, same path, every token — eliminates routing stochasticity across runs. Publishable benchmarks need this control. The MoE's expert gating has temperature-like variance that complicates statistical testing.

Batch inference. Dense scales linearly to batch 8 on 48 GB cards. The MoE chokes at batch 4 on 24 GB. Even at 48 GB, the reported 23% throughput collapse makes per-request economics worse. If you're serving an API where latency SLA is secondary to throughput, dense wins.

Downstream quantization tolerance. Dense Q3_K_M stays coherent for chat and summarization. MoE Q3_K_M triggers expert collapse. The gating MLP degrades faster than the expert weights, sending tokens to wrong experts and producing gibberish. The dense architecture is more forgiving at aggressive compression.

Tool ecosystem maturity is the final factor. vLLM and TensorRT-LLM optimize dense paths first; MoE support lags 6-12 months historically. If your production stack depends on PagedAttention or fused attention kernels, dense is the safer integration target today.

When 26B MoE Surprises You

MoE isn't always the compromise choice. On single-request latency with 4K-token prefill, the MoE routes to what functions as a "summarization expert" — a weight subset optimized for long-context compression. That delivers 23% faster time-to-first-token than dense on identical hardware. The specialization pays off when your prompt structure matches an expert's training distribution.

Quality at fixed VRAM is the headline win. The MoE at Q4_K_M outperforms the dense at Q4_0 on every benchmark except pure code generation. The dense is forced into coarser quantization to fit. That 14% HellaSwag gap for dense Q4_0 isn't recoverable without more VRAM. The MoE's Q4_K_M fidelity is structurally protected by its active-parameter efficiency.

Future-proofing is speculative but worth noting. Google's TPUs train MoE-native architectures. The dense 31B may not receive next-generation distillation or architecture improvements if Google's research trajectory continues toward sparse models. Betting on MoE aligns with Google's hardware-software co-design. Betting on dense is betting that Google will maintain dense-side investment. That's a smaller pool historically.

Context Length and KV Cache Math

Gemma 4 advertises 128K context across all variants, but the KV cache reality is messier. The 31B dense uses full attention — every token attends to every prior token. So KV cache scales O(n²) with sequence length. The 26B MoE uses sliding window 4K local attention plus full 128K global attention in alternating layers. That creates non-monotonic cache growth. Short contexts are cheaper than you'd expect. Long contexts jump sharply at the global-attention layers.

On 24 GB with the 26B MoE Q4_K_M: 8K context totals 21.4 GB, 16K context hits 23.8 GB — OOM risk territory with zero headroom. The 31B dense at Q4_0 with FlashAttention 2 reduces KV cache 37%, enabling 16K context in 24 GB. But you're still running Q4_0 with its quality penalty. Without it, the 31B dense can't reach 8K context at any quantization on 24 GB. With it, --flash-attn in llama.cpp or equivalent in vLLM, the recomputation tradeoff — recalculating attention instead of caching — buys you context length at modest speed cost.

RAG Chunking Strategy per Variant

Your chunk size and context window should match your model's VRAM tier, not its parameter count.

4B IT: 512-token chunks, 32K context window, cross-encoder re-rank. At 22 tok/s, the latency of re-ranking 10 chunks is acceptable — under 2 seconds end-to-end. The small model's speed compensates for its retrieval depth limitation.

12B IT: 1024-token chunks, 8K context, dense retrieval without re-rank. This is the quality threshold for complex queries — legal analysis, medical literature, technical documentation. At 512 tokens, chunks fragment semantic units. The 8K context ceiling is hard; don't push it.

26B MoE: 2048-token chunks, 8K context, expert "retrieval" gate activates. The MoE's routing naturally specializes: some experts handle synthesis, others handle extraction. Use this for synthesis-heavy RAG — summarizing across multiple retrieved documents — rather than search-first retrieval. The 8K context fits comfortably in 24 GB; 16K is possible but leaves no margin.

31B dense: 4096-token chunks, 16K context with FA2, single-pass RAG. Highest coherence, highest VRAM. The large chunks preserve document structure. The 16K context with FlashAttention 2 allows multiple 4K chunks plus room for generation. This is the configuration for "read this 50-page PDF and answer detailed questions" — if you have the GPU to run it.