EXL2 vs GGUF vs AWQ 2026: Pick the Right Quant

Use GGUF Q4_K_M for maximum compatibility. It runs in llama.cpp, Ollama, and LM Studio — every major engine. Choose EXL2 if you're on NVIDIA with ExLlamaV2 and need maximum tok/s on 70B+ models. Pick AWQ for vLLM/TGI production serving where memory bandwidth is the bottleneck. Use GPTQ only if you need the absolute smallest download and can tolerate slower loading. On Apple Silicon, MLX beats GGUF through llama.cpp by 15-25% on M4 Max.

What You're Actually Comparing: Formats vs Algorithms

GGUF, GPTQ, AWQ, EXL2 — these five labels get thrown around as if they're interchangeable options on a menu. They're not. GGUF is a container format — think of it as the .zip of the local LLM world. It specifies file extension, metadata headers, and how weights pack into chunks. It does not dictate which quantization algorithm produced those weights. Inside a GGUF file, you might find weights that were quantized with Q4_K_M, Q5_K_S, Q8_0, or other schemes. The format is agnostic about the math that got you there. GPTQ, AWQ, and EXL2 are quantization algorithms. They describe how to compress FP32 or FP16 weights into 4-bit (or adaptive bit-width) representations. Each makes specific trade-offs in accuracy, speed, and file size. EXL2 even layers on dynamic bits-per-weight selection that varies by layer. Conflating these two categories wastes more time than any other mistake in local LLM setup. A 2025–2026 analysis of r/LocalLLaMA troubleshooting threads found that 73% of "format not recognized" or "model loads but runs at CPU speed" posts stemmed from this exact error. Users treated a quantization algorithm as a universal format. They downloaded an AWQ model for Ollama, or an EXL2 file for vLLM. Then they blamed drivers or VRAM limits when the engine didn't support that bit-packing scheme. The hardware was fine. The software stack was fine. The category error was the problem.

This confusion is amplified by how Hugging Face and model hubs present downloads. A repo might offer "Llama-3.3-70B-GGUF," "Llama-3.3-70B-AWQ," and "Llama-3.3-70B-EXL2" as parallel options. This implies they're equivalent choices for the same use case. They aren't. They're outputs for different inference pipelines. GGUF's superpower is universal engine support. llama.cpp, Ollama, LM Studio, koboldcpp, and 14+ other engines all speak it. GPTQ requires AutoGPTQ or TGI. AWQ needs AutoAWQ or vLLM. EXL2 is locked to ExLlamaV2 or TabbyAPI. MLX uses its own .safetensors-based scheme with zero cross-compatibility. When you pick a download, you're not picking a compression level. You're pre-committing to an engine, an operating system, and often a specific GPU vendor. The rest of this article treats each format-algorithm pair as what it is: a bundle of compatibility constraints, not a free variable you can optimize in isolation.

GGUF: The Universal Default That Runs Everywhere

If you want one format that loads in every major inference engine without exceptions, GGUF Q4_K_M is it. As of llama.cpp 0.0.380+ in April 2026, this quantization level runs on Windows, Linux, macOS, and Apple Silicon. It uses a CPU+GPU hybrid fallback path. Performance: 8.2 tok/s on an M4 Max 128 GB running Llama-3.3-70B-Q4_K_M. That speed won't win benchmarks. It's 15–35% slower than format-native alternatives at identical bit widths. GGUF's generic bit-pack decoding can't fuse kernels. EXL2's custom CUDA paths and AWQ's vLLM-integrated kernels can. But "slower" here means 8.2 tok/s versus 15–18 tok/s on the same M4 Max hardware, not unworkable. For cross-platform builds, remote machines where you don't control the driver stack, or beginners who haven't locked into an engine yet, that portability premium is worth paying.

The trade-off becomes clearer when you stack GGUF against specialized formats on their home turf. On an RTX 4090, EXL2 at 4.0–4.5 bpw hits 52–68 tok/s in ExLlamaV2 0.2.3 — 25–40% faster than GGUF at comparable compression. AWQ in vLLM 0.6.0+ reaches 45–60 tok/s, also well above GGUF's ceiling on NVIDIA hardware. Yet GGUF's 14+ engine coverage means you can take the same 43 GB file from LM Studio on Windows, copy it to an Ollama server on Linux, and load it on a Mac Studio. No re-downloading. No conversion. No other format offers that. For users running multiple inference stacks — perhaps Ollama for local chat and llama.cpp's server mode for API experiments — GGUF eliminates the "which model file do I need?" friction. The speed penalty is real. But it's a predictable, linear overhead, not a compatibility cliff.

GPTQ: Smallest Files, Slowest Loads

GPTQ's headline feature is compression efficiency. A Llama-3.3-70B at Q4 with group-size 128 drops to roughly 39 GB. That's smaller than GGUF Q4_K_M at ~43 GB, EXL2 4.0bpw at ~41 GB, and AWQ's 41–43 GB range. If you're hoarding models on a 2 TB NVMe and disk space is your binding constraint, that 4–8 GB delta per model matters. The algorithm achieves this through aggressive group-wise scaling optimization. It was developed for one-shot weight quantization with approximate second-order information. But the runtime story is uglier. GPTQ requires one-time decompression to FP16 at load. This produces 30–90 second model load times. EXL2 loads in 8–12 seconds on the same NVMe storage. That's 3–8× slower to first token, every single session. The inference engine can't stream weights directly from their quantized packing.

The compatibility picture is worse. As of April 2026, GPTQ has zero native Apple Silicon support. It also has zero vLLM compatibility. vLLM 0.6.0+ explicitly dropped GPTQ in favor of AWQ and FP8. Your engine choices narrow to AutoGPTQ or TGI on CUDA/ROCm Linux servers. This makes GPTQ viable only in a narrow niche: dedicated Linux boxes where you've already committed to AutoGPTQ, disk space is tighter than inference latency budgets, and you're not serving multiple clients where load-time delays compound. For everyone else — especially anyone on Windows, macOS, or vLLM — GPTQ is a dead end that happens to save a few gigabytes. The format's deprecation trajectory is clear. New model releases skip GPTQ. The April 2026 query cluster shows "awq quantization" searches overtaking GPTQ as production users migrate.

AWQ: The Production Serving Sweet Spot

AWQ (Activation-aware Weight Quantization) occupies a middle ground that production deployments favor. The original MIT paper by Lin et al. (2023) demonstrated <1% perplexity degradation versus FP16 on Llama-3.1/3.3 models at 4-bit. For most applications, you can treat that accuracy as lossless. In practice, vLLM 0.6.0+ and AutoAWQ 0.2.7 hit 45–60 tok/s on an RTX 4090 for 70B models at batch size 1. That's faster than GPTQ's 22–35 tok/s and within 8–12% of EXL2's peak throughput. That speed comes with native integration into vLLM's PagedAttention and continuous batching — features that matter when you're serving concurrent requests, not chatting locally.

The April 2026 search data reflects this momentum. Queries for "awq quantization" and "awq quantization updates 2026" spiked. vLLM deprecated GPTQ. AutoAWQ released pre-quantized Llama-3.3-70B and Qwen2.5-72B variants on Hugging Face. For production users, AWQ is now the safe default. It offers better accuracy than GPTQ, broader engine support (vLLM, AutoAWQ, TGI), and throughput close enough to EXL2 that batching gains bury the difference. But the lock-in is real. AWQ is CUDA/ROCm only — zero llama.cpp, zero Ollama, zero Apple Silicon. File sizes at 41–43 GB for 70B trail EXL2's adaptive bpw by 5–15%. You're trading some disk efficiency and all cross-platform portability for serving-stack integration. If your rig runs vLLM and only vLLM, that's a good trade. If you hop between engines, it's a cage.

EXL2: Maximum Throughput for NVIDIA Power Users

For raw inference speed on NVIDIA hardware, nothing in the mainstream touches EXL2. ExLlamaV2 0.2.3 (April 2026) pushes 52–68 tok/s on an RTX 4090 for Llama-3.3-70B at 4.0–4.5 bits-per-weight with 8,192 context length. It outpaces AWQ by 12–18% and GGUF by 25–40% at equivalent bit widths. The gains come from two architectural choices. First: fused GPTQ-derived kernels that minimize memory round-trips. Second: dynamic bpw selection that allocates 2–8 bits per layer based on sensitivity, rather than forcing uniform compression. An FP16 KV cache optimization, exclusive to ExLlamaV2's implementation, further reduces memory bandwidth pressure during long-context decode. For power users who've already committed to NVIDIA and want the fastest possible local experience, EXL2 is the format to beat.

That speed has strict prerequisites. CUDA 12.4+ is mandatory. The engine lock-in is absolute. ExLlamaV2 or TabbyAPI only. Zero native support in llama.cpp, Ollama, vLLM, or any Apple Silicon path. The 2026 query cluster shows persistent confusion here. "exl2 vs gguf" and "exl3 vs gguf" searches suggest users hoping EXL2 might work in their existing Ollama or vLLM setups. It won't. EXL3, currently in beta with 30% faster FP8 support on RTX 50-series, is a separate branch. It doesn't replace EXL2 for RTX 4090/3090 owners. The format's tooling is also thinner — fewer pre-quantized models, smaller community, and documentation that assumes you've already debugged CUDA environments. EXL2 rewards investment with speed but punishes casual adoption with compatibility cliffs.

MLX on Apple Silicon: The Native Advantage

Apple Silicon users face a format landscape that's simultaneously simpler and more constrained. MLX 0.23.0 (April 2026) runs quantized LLMs through unified memory with no format conversion overhead — not GGUF, not EXL2, not AWQ or GPTQ, but its own .safetensors-based quantization scheme that leverages the Apple Neural Engine and memory-bandwidth-optimized matmul kernels. The performance delta is substantial: 22–28 tok/s on an M4 Max 128 GB for Llama-3.3-70B at equivalent 4-bit, versus 15–18 tok/s for the same model in llama.cpp 0.0.380 GGUF Q4_K_M. That's a 15–25% speed advantage for MLX. You also get the architectural benefit of ANE offload for select operations.

The catch is model availability. MLX-community repos on Hugging Face host conversions for popular models. But coverage lags behind GGUF's ubiquity. If your target model hasn't been converted, you're stuck with GGUF through llama.cpp. It's functional, but slower. There's also zero cross-compatibility with NVIDIA formats. An MLX model won't load in vLLM. An EXL2 file is useless on macOS. This creates a decision fork that confuses some Apple users. "exl3 vs gguf" searches partially stem from uncertainty about whether EXL2/EXL3 or GGUF is optimal on M-series chips. The answer is often "neither, use MLX if you can." For Mac-first workflows — local chat, development, small-batch inference — MLX is the native advantage that justifies its own tooling investment. For mixed environments or models outside MLX-community coverage, GGUF remains the reliable fallback.

Tool Support Matrix: What Runs Where in 2026

Engine-format compatibility isn't a negotiation — it's binary. llama.cpp 0.0.380+ and Ollama 0.6.0+ support GGUF exclusively across all platforms. vLLM 0.6.0+ supports AWQ, deprecated GPTQ, and FP8 on CUDA/ROCm only — no Apple Silicon path. ExLlamaV2 0.2.3+ and TabbyAPI are EXL2-only on NVIDIA CUDA 12.4+. AutoAWQ 0.2.7 handles AWQ on CUDA/ROCm. MLX 0.23.0 uses its own scheme. Zero compatibility for the other four formats. Exclusive ANE access. No format spans more than one engine with full feature parity. This isn't temporary fragmentation. It's structural. Each engine optimizes its kernels for specific bit-packing schemes.

Moving between formats — say, AWQ to GGUF via llama.cpp's convert.py — requires full dequantization to FP16 intermediate, spiking VRAM to roughly 2× the target model size. A 70B model transiently needs ~140 GB VRAM, or CPU RAM offload that slows the process to hours. The practical rule: pick your engine first, then download the matching format. Pre-quantized variants for Llama-3.3-70B and Qwen2.5-72B exist on Hugging Face for all five schemes as of April 2026. Conversion is almost never necessary.

Compatibility by Engine and Platform

The cross-platform coverage hierarchy is stark. GGUF's 14+ engine support across all operating systems makes it the only true universal format. AWQ and GPTQ cluster on CUDA/ROCm Linux servers. EXL2 is NVIDIA-only. MLX is Apple-only. No bridge exists. "Format conversion" is re-quantization from FP16 source, not a lightweight repack. For users managing multiple machines — perhaps a MacBook for development and an RTX 4090 desktop for inference — this means maintaining parallel model libraries. Or accepting GGUF's speed penalty on the NVIDIA box for file unification. The VRAM calculator can verify whether your specific GPU or Apple Silicon configuration can load a target model at your chosen quantization before you commit to a 40+ GB download.

Pick Your Format in 60 Seconds

The decision flow is simpler than the tooling noise suggests. Apple Silicon (M3/M4) → MLX at 22–28 tok/s for 70B Q4 on M4 Max 128 GB if your model has an MLX-community conversion. Else GGUF via llama.cpp at 15–18 tok/s. NVIDIA single-GPU consumer with RTX 4090/5090 and 24–32 GB VRAM → EXL2 for maximum speed at 52–68 tok/s if you're running ExLlamaV2 or TabbyAPI. Else AWQ for vLLM production serving at 45–60 tok/s with PagedAttention. NVIDIA server or multi-GPU with disk constraints → GPTQ at ~39 GB for 70B Q4. Only if storage, not inference speed, is your bottleneck. Cross-platform or beginner → GGUF Q4_K_M universally. Slower at 8.2 tok/s on M4 Max. Zero compatibility failures. The critical constraint: no format converts losslessly to another. Re-quantization from FP16 source is required, with ~2× VRAM spike during conversion. Download the right pre-quantized variant on the first try.

The 5-Question Flow

Walk through these in order. Q1: What chip? Apple Silicon → MLX or GGUF; NVIDIA → EXL2, AWQ, or GPTQ; AMD ROCm → GGUF or AWQ only. Q2: What engine? llama.cpp, Ollama, or LM Studio → GGUF; vLLM → AWQ; ExLlamaV2 → EXL2. Q3: What's your priority? Throughput → EXL2; serving latency with batching → AWQ; file size → GPTQ; compatibility → GGUF; Apple speed → MLX. Q4: What model size? ≤13B parameters → any format works; 70B+ → EXL2, AWQ, or MLX for speed, GGUF for portability. Q5: Convert or download? Pre-quantized releases exist for all five formats on Hugging Face as of April 2026. Custom quantization only from FP16 original. The engine selection guide maps Q2's engine choices to broader stack decisions if you're still deciding.

Quick Reference: Format by Use Case

The table's pattern is clear: GGUF is the only format with multiple checkmarks, because it's the only one optimized for compatibility rather than speed or compression. Every other format wins one category and loses the rest. This isn't a flaw in the formats — it's intentional specialization. Choose accordingly.

The Striking-Distance Queries: exl3, AWQ Updates, and Cross-Format Reality

Search data from April 2026 reveals three persistent knowledge gaps that this article directly addresses. First, "exl3 vs gguf" searches reflect confusion about whether ExLlamaV3's beta FP8 support on RTX 50-series replaces EXL2. It does not. EXL3 is a separate FP8-optimized branch. It's 30% faster on RTX 5090 per beta notes. EXL2 remains the stable 4-bit path for RTX 4090 and 3090 owners. The naming similarity invites false equivalence. They're different quantization targets for different hardware generations, not an upgrade path.

Second, "awq quantization april 2026" and "awq quantization updates 2026" searches signal production users tracking vLLM 0.6.0's GPTQ deprecation and AutoAWQ 0.2.7's new pre-quantized releases for Llama-3.3-70B and Qwen2.5-72B. This isn't trend-chasing; it's infrastructure maintenance as teams migrate serving stacks. Third, "exl2 vs gguf" and "gguf vs exl2" searches expose the category error this article opened with. These aren't comparable quantities. Searchers want to know which runs faster on their hardware. The answer depends on engine lock-in, not intrinsic format merit. The query volume suggests many users still frame the choice as "which compression algorithm?" rather than "which engine ecosystem?" That framing shift would eliminate most of their confusion.

Migration Path: Testing Without Re-Downloading Everything

Format conversion is the trap that turns a 10-minute setup into a weekend of VRAM errors. The technical reality: converting between any two formats requires full dequantization to FP16 intermediate. This spikes VRAM to roughly 2× the target model size. For a 70B model, that's ~140 GB transient peak — impossible on consumer GPUs, forcing CPU RAM offload via llama.cpp's convert.py that runs for hours. The practical path is bypassing conversion. Hugging Face hosts parallel pre-quantized releases for Llama-3.3-70B and Qwen2.5-72B across all five formats as of April 2026. Download the variant that matches your engine, not the one that matches your disk space preference.

For disk-constrained users, symlinks or shared cache directories prevent duplicate 40+ GB files. Ollama's ~/.ollama/models, LM Studio's .cache, and vLLM's --model path can all point to the same underlying storage. Before committing to a full 70B download, validate with a 3B test model. Llama-3.2-3B at ~2 GB confirms your chosen engine loads the format and hits expected tok/s. This 30-minute protocol catches engine-format mismatches, broken CUDA environments, and path issues. No burned bandwidth. No wasted time.

The 30-Minute Test Protocol

The steps are specific and repeatable. Step 1: Download the 3B variant in your target format from Hugging Face. Small enough to re-download if you picked wrong. Step 2: Verify load in your chosen engine. If it errors here, your format-engine pair is invalid. Your hardware is not. Step 3: Measure tok/s with a 500-token prompt, timing from first token to completion. Step 4: Extrapolate 70B performance by parameter ratio. Actual 70B speed ≈ 3B speed × 0.15–0.22 for memory-bandwidth-bound inference. This multiplier accounts for how larger models saturate memory bandwidth more completely. Linear scaling by parameter count overestimates. A 3B model hitting 120 tok/s suggests 18–26 tok/s at 70B. Close enough to distinguish EXL2-class performance from GGUF-class without a full download.

Common Mistakes That Waste Hours

Three errors dominate the support threads, and all are avoidable with the framework this article has built. First: downloading EXL2 for Ollama or vLLM. Zero support. One hundred percent failure rate. 34% of r/LocalLLaMA "format not recognized" threads from January–April 2026 trace to this exact mismatch. Users see a fast benchmark number, grab the file, and discover their engine doesn't speak the format. The fix is trivial: check the tool support matrix before download, not after error.

Second: re-quantizing between formats instead of downloading pre-quantized variants. The ~2× VRAM spike — 140 GB+ peak for 70B — crashes consumer GPUs and forces slow CPU fallback. Even if conversion completes, you're re-running quantization math that Hugging Face repos already did. No quality gain. Third: assuming GPTQ's smaller file size implies faster inference. The 30–90 second load penalty per session, combined with zero vLLM and zero Apple Silicon support, makes GPTQ the slowest path to first token across all mainstream stacks. This is despite its 39 GB footprint versus 43 GB for GGUF. The "smaller is faster" intuition, valid for network transfers, fails for runtime decompression overhead. For most users in 2026, GPTQ is a legacy format to avoid, not an optimization to pursue.