What GPU do I need to run a coding LLM locally?

For 7B models like Qwen2.5-Coder, an RTX 4060 (8GB) handles it at 30-50 tok/s. For the better 32B model, you need RTX 4090 (24GB). As of July 2026, Qwen2.5-Coder 32B is the best single-GPU coding model on the market.

Can I run a coding LLM on my gaming GPU?

Yes. Any NVIDIA GPU with 8GB+ VRAM can run small coding models. RTX 3080, 4070, and 4090 are solid choices. The key is VRAM, not peak performance—coding LLMs need enough memory for context, not maximum throughput.

Is Q4 quantization good enough for coding?

Yes. Q4_K_M quantization loses only 1-3% accuracy on benchmarks and is the standard for 2026. Published community benchmarks put Qwen2.5-Coder 32B at Q4 within about a point of its full-precision HumanEval score—barely noticeable in real coding work.

Should I use Ollama or llama.cpp for coding?

Ollama is simpler and handles GPU setup automatically. llama.cpp is faster and gives you more control. For beginners: Ollama. For power users who want to squeeze extra tok/s: llama.cpp. Both work fine for coding tasks.

Best Hardware for Local Coding LLMs [2026 Guide]

Some links on this page may be affiliate links. We disclose it because you deserve to know, not because it changes anything. Every recommendation here comes from benchmarks, not budgets.

Run Qwen2.5-Coder 32B on an RTX 4090 or go smaller with 14B on an RTX 4070. Most people don't need 70B models for code—32B is the productivity sweet spot. The GPU matters less than having enough VRAM for context. This guide maps the VRAM requirements of the top open-source coding models at every quantization level, using published benchmarks, so you can pick the right rig today.

Quick Pick: Coding LLM Hardware Tiers

Tier	GPU	Price	Best Model	Performance
Budget	RTX 4060 (8GB)	$250	Qwen2.5-Coder 7B	35-45 tok/s
Mid-Range	RTX 4070 Ti (12GB)	$550	Qwen2.5-Coder 14B	50-70 tok/s
High-End	RTX 4090 (24GB)	$1,800	Qwen2.5-Coder 32B	85-110 tok/s

Prices as of July 2026. Tok/s measured at Q4_K_M quantization with 2K-token context.

Why Coding LLMs Are Different

Coding isn't like chat. You need completion speed more than response length, reasoning ability more than trivia, and context window more than raw token throughput. Qwen2.5-Coder and DeepSeek V3 specialize in code—they trained on GitHub commits, LeetCode, and algorithm datasets. Llama and Mistral are generic and weaker at programming tasks by 15-20 percentage points.

The second difference is context matters. A 2K-token context window gets you one function. A 32K window gets you a whole file plus conversation. If your GPU can only run 1K-token context before hitting VRAM limits, you lose the ability to keep code snippets in memory, and coding becomes painful.

Tip

Coding models are benchmark-happy and benchmarks actually predict real performance here. HumanEval and SWE-bench correlate directly with whether the model can fix bugs in your actual code. For coding (unlike chat), benchmarks matter more than marketing.

The Hardware Tiers: What Runs What

7B Models (4-8 GB VRAM)

Qwen2.5-Coder 7B-Instruct scores 88.4% on HumanEval and fits in 5.2 GB at Q4_K_M quantization. RTX 4060 8GB handles it comfortably with room for 4K-token context. Speed is 35-45 tokens per second, which is acceptable for single-threaded coding assistance.

Who should buy this: students, laptop users, anyone prototyping locally for the first time. It's not slow, but you feel the wait for longer completions. Good for debugging, pair programming, and learning; weak on long-form code generation.

Entry-level cost: $250-350 for a used RTX 4060 or entry RTX 3060.

14B Models (8-14 GB VRAM)

Qwen2.5-Coder 14B requires 8.7 GB at Q4_K_M, 10.7 GB at Q5_K_M, 12.8 GB at Q6_K. It scores 90.2% on HumanEval and runs at 50-70 tok/s on an RTX 4070 Ti. This is the sweetspot for most developers—good reasoning, good speed, fits on one midrange card.

The 14B outperforms the 7B by 2 percentage points on HumanEval and 4-5 points on SWE-bench (practical agent benchmarks). The difference is noticeable: it catches more bugs, writes better boilerplate, and requires fewer refinement prompts.

Entry-level cost: $550-700 for an RTX 4070 Ti or used RTX 3090.

32B Models (16-24 GB VRAM)

Qwen2.5-Coder 32B is where coding performance peaks on consumer hardware. At 92.7% HumanEval, it beats most proprietary models except Claude Opus 4.8 (56 on Artificial Analysis vs 51-52 for open-source). At Q4_K_M, it needs 20.2 GB. At Q6_K, 26 GB. The RTX 4090 handles it at full speed with 4K context.

This is the model to pick if you can afford it. It's not overkill—the 32B vs 14B jump is 2.5 points on HumanEval (doesn't sound big, but it's the difference between passing LeetCode hard problems and not). DeepSeek-Coder V2 (236B) is larger but slower and overkill for most use cases.

Speed: 85-110 tok/s at Q4_K_M on RTX 4090. 45-60 tok/s on RTX 4070 Ti if you use tensor parallelism or offload.

Entry-level cost: $1,500-2,000 for RTX 4090; $600-800 used.

70B+ Models (48+ GB VRAM Multi-GPU)

DeepSeek V4 (236B sparse) and similar large models need either dual RTX 3090s (48 GB combined VRAM) or a single H100/A100. For local development, this is overkill. Qwen2.5-Coder 32B covers 99% of real-world coding tasks. Only buy dual-GPU if you're fine-tuning or running multi-concurrent inference.

Warning

Dual-GPU setup adds complexity: you need NVLink (RTX 3090/4090 support it, but most cards don't), PCIe bandwidth matters, and power supply jumps to 1200W minimum. Cost is $2,500-4,000. The 32B model solves the same problem at 1/10th the cost and complexity.

Benchmark Comparison: Real Numbers

Figures below are compiled from published benchmark results (HumanEval, SWE-bench) and community-reported llama.cpp throughput for an RTX 4090 at Q4_K_M quantization with 2K-token context. Last compiled: July 10, 2026.

Model	Size	HumanEval	SWE-bench	Tokens/sec
Qwen2.5-Coder 32B	32B	92.7%	62.4%	105
Qwen2.5-Coder 14B	14B	90.2%	48.1%	68
DeepSeek-Coder-V2	16B (2.4B active)	81.1%	44.7%	92
Llama 3.1 8B	8B	72.6%	35.2%	95
Claude Opus 4.8	Proprietary	92.3%	65.2%	API-only

The gap between 32B and 14B is real. On SWE-bench (fixing actual bugs in code), Qwen 32B wins by 14 points. That's the difference between a tool that saves you time and one that wastes it with false suggestions.

DeepSeek V2 is interesting: it's only 16B total parameters but 2.4B active per token. It runs fast but scores lower on HumanEval and SWE-bench. Good for speed-first use cases; weak on complex reasoning.

How Much VRAM Do You Actually Need?

The VRAM requirement depends on model size, quantization, and context window. Here's the formula:

VRAM (GB) ≈ (Model Size in Billions × Bytes per Parameter) + Context Overhead

For Qwen2.5-Coder 32B at different quantizations:

Q3_K_S (aggressive): 13.5 GB (risky, some quality loss)
Q4_K_M (recommended): 20.2 GB (1-2% quality loss)
Q5_K_M (high quality): 24.8 GB (near-lossless)
Q6_K (archive): 29.1 GB (indistinguishable from FP16)
FP16 (full precision): 64 GB (overkill)

Most people pick Q4_K_M. It's the 2026 standard—losses are imperceptible in real coding work, and the speed jump vs Q5 is 15-20%.

Note

Your context window eats VRAM too. A 32K context window can add 1-2 GB on top of model size. If your GPU has exactly 24 GB and you want Qwen 32B at Q4 with 8K context, you're cutting it close. Leave 2 GB headroom to avoid OOM crashes mid-generation.

Software: Ollama vs llama.cpp vs vLLM

Three tools dominate 2026 local coding.

Ollama is the easiest. One command pulls the model, handles quantization selection, and serves an OpenAI-compatible API. No drivers to troubleshoot, no flags to memorize. Beginners should start here.

ollama run qwen2.5-coder:32b-q4_k_m

Speed hit: ~5-10% slower than llama.cpp raw because of overhead. Not noticeable in practice.

llama.cpp is faster and gives you control. You can tweak context length, batch size, and tensor split across GPUs manually. Power users and multi-GPU setups go here. Harder to set up, but worth it if you're squeezing the last 10% performance.

vLLM is overkill for single-user coding. It's built for serving 100 concurrent requests. Unless you're running a coding API server, skip it.

Recommendation: Ollama for beginners and single-GPU setups. llama.cpp for multi-GPU or if you already know what tensor parallel means.

Quantization: Q4 vs Q5 vs Q6

Quantization maps 32-bit floats to 4, 5, or 6 bits. Smaller bit depth = smaller VRAM = faster inference = tiny quality loss.

Q4_K_M (4-bit) loses 1-3% on benchmarks. Reported HumanEval scores for Qwen2.5-Coder 32B at Q4 sit within about a point of FP16—invisible in real work. VRAM: 20.2 GB on RTX 4090. Speed: 105 tok/s.

Q5_K_M (5-bit) loses <1% on benchmarks but costs 5 GB more VRAM. Only pick this if you're paranoid about quality or have the VRAM to spare. Speed: 85 tok/s (noticeably slower).

Q6_K (6-bit) is near-perfect but wasteful. 29 GB VRAM for a 32B model. Only use if you have dual-GPU and need the highest quality.

Q3_K_S (3-bit) is risky. Saves 3-4 GB but loses 5-8% on HumanEval. Can still be useful for prototyping on laptops.

Our recommendation: Q4_K_M. It's the Goldilocks zone for 2026.

Setup in 5 Minutes (Ollama)

Step 1: Download Ollama from ollama.ai.

Step 2: Pull Qwen2.5-Coder 14B (for RTX 4070) or 32B (for RTX 4090):

ollama pull qwen2.5-coder:14b-q4_k_m

Step 3: Run it:

ollama run qwen2.5-coder:14b-q4_k_m

Step 4: Open another terminal and use it as an API:

curl http://localhost:11434/api/generate -d '{
  "model": "qwen2.5-coder:14b-q4_k_m",
  "prompt": "Write a Python function that..."
}'

Step 5: Connect it to your editor. VS Code extension: Codeium. Or use LM Studio for a GUI (slower, more user-friendly).

No drivers, no CUDA confusion, no manual VRAM tuning. Ollama handles it.

For deeper dives, see our guides on quantization explained and local coding stack setup.

FAQ

Should I buy the RTX 5090 or wait for the RTX 60-series?

The RTX 5090 is excellent for coding. 32 GB of VRAM lets you run 70B models at Q4 or 32B at Q5/Q6 with massive context. The RTX 60-series isn't shipping until late 2026 at earliest. If you're building now, the 5090 won't regret. Cost: $1,500-2,000 used by Q3 2026.

Can I use an AMD GPU instead?

Yes, but setup is harder. AMD ROCm on WSL2 requires version pinning and HSA_OVERRIDE_GFX_VERSION flags. RX 9060 XT (16GB) is the budget pick at $400; RX 9070 XT (16GB) matches RTX 4070 Ti performance. If you know ROCm, go for it. If you're new, stick with NVIDIA.

What about Mac M-series chips?

Mac Mini M4 Pro (32GB unified memory) runs Qwen2.5-Coder 32B at 35-45 tok/s using MLX. Quieter and cheaper than RTX 4070, but slower. Good for coding if you already own a Mac; not worth buying one just for local coding vs an RTX GPU.

Is local coding LLM worth it vs paid APIs?

One month of Claude Pro ($20) or ChatGPT Plus ($20) costs $240/year. An RTX 4070 Ti ($550 used) costs $550 once and runs Qwen2.5-Coder 32B forever at $0/month. If you code daily, it breaks even in 3 months and saves $4,500 over 5 years. Plus: zero latency, zero API rate limits, 100% private.

Will quantization hurt my code quality?

No. On SWE-bench (real bug fixing), Q4_K_M loses <1%. On HumanEval, it's 1-2%. You won't notice. The only time quantization matters is extreme (Q2/Q3) or if you're doing math-heavy work where rounding errors accumulate.

Should I fine-tune a model for my codebase?

Not yet. Fine-tuning costs $5,000+ in compute time and only helps if your codebase has unique domain-specific patterns (aerospace code, proprietary frameworks, etc.). Qwen2.5-Coder 32B already trained on GitHub and generalizes beautifully. Use it first. Fine-tune if it underperforms on your repo's specific patterns.

Final Verdict

Pick Qwen2.5-Coder 32B on an RTX 4090 if you code daily and can afford it. The benchmark lead over the 14B is real—2.5 percentage points on HumanEval means fewer bugs slipping through, fewer refinement prompts needed, fewer hallucinated functions. At 85-110 tok/s, it's fast enough for flow state.

Pick Qwen2.5-Coder 14B on an RTX 4070 Ti if you want the midpoint. It's 90% as good for 60% of the cost and runs just as fast as many consumer workflows need.

Pick Qwen2.5-Coder 7B on an RTX 4060 if this is your first local AI rig. You'll learn what works, and if you hit walls, you'll know exactly where to upgrade next.

Don't overkill. Dual-GPU and 70B models are not worth it for coding—Qwen 32B solves the problem at 1/10th the cost and complexity. Don't underbuy on VRAM. An 8GB GPU with a 14B model at Q3 quantization is frustrating. Step up to 12GB or accept the 7B tier. Your future self will thank you.