Qwen 3.6 27B Beats 70B Models on 24GB VRAM

Qwen 3.6 27B is the first sub-30B model that trades blows with 70B incumbents on coding and reasoning benchmarks, and it fits in 24 GB VRAM at Q4_K_M. If you have a 4090 or 3090, pull qwen3.6:27b-q4_K_M and, community benchmarks suggest, expect roughly 18–22 tok/s — no cloud rental, no second GPU, no quality sacrifice for most tasks.

What Qwen 3.6 Actually Ships

Alibaba dropped the full Qwen 3.6 family on July 10, 2026 — seven sizes from 0.5B to 72B, all Apache 2.0 licensed. The 27B variant is the one that matters for local inference. Alibaba positions it as the flagship efficiency play. It matches or exceeds 70B-class model performance on coding and reasoning benchmarks. It cuts VRAM requirements by roughly 60% at equivalent quantization. That means a single 24 GB card can run what previously demanded dual-GPU setups or cloud rentals. The license terms are clean. No commercial restrictions. No attribution clauses beyond standard Apache 2.0. Developers burned by Llama's evolving license or Mistral's more restrictive terms now have a genuine alternative. The 72B exists for datacenter deployment, and the sub-7B sizes target edge and mobile. But 27B hits the sweet spot. It's large enough to compete with 70B dense models. It's small enough to quantize aggressively without collapsing.

The parameter count itself tells part of the story. At 27B, Qwen 3.6 sits between Llama 3.1 8B and 70B. The architecture changes push effective capacity higher than the raw number suggests. It uses grouped-query attention, sliding window attention for long-context efficiency, and a vocabulary expansion to 152K tokens. For local LLM builders, the relevant comparison isn't 27B vs. 27B. It's 27B vs. 70B at the same quantization level. The 60% VRAM reduction means you can run Q4_K_M on hardware that would choke on a 70B Q4. The 128K context window matches the 72B variant. Some families tie context length to parameter count. Qwen 3.6 doesn't. That's a deliberate choice. Repository-level coding tasks need long context. Alibaba explicitly markets 27B for that use case.

Benchmark Reality Check: 27B vs. 70B Incumbents

Here's the table that matters. Numbers are pass@1 single-sample unless noted.

The 27B ties or exceeds Llama 3.1 70B on HumanEval (73.2% vs. 71.8%) and exceeds Mixtral 8x22B on MATH-500 (62.4% vs. 58.1%) while running at 2.1× the tokens/second on a single RTX 3090 at Q4_K_M quantization. The HumanEval lead is genuine, 1.4 points is outside run-to-run variance for this benchmark. The gap is real but narrow. 1.3 points suggests comparable reasoning capability. 70B's extra parameters buy marginal gains on the hardest problems. Where 27B dominates is throughput. 27B generates at 18–22 tok/s. 70B Q4 manages 8–11 tok/s. That speed difference changes how interactive coding workflows feel. You're not waiting seconds for each token to appear.

Coding Benchmarks

HumanEval and MBPP-Plus scores place 27B between 70B dense and 8x22B MoE models, with the narrowest gap on Python generation tasks. The positioning is precise. 27B beats CodeLlama 70B on MBPP-Plus by 4.2 points. It ties or exceeds Llama 3.1 70B on HumanEval. It sits comfortably above Mixtral 8x22B on both. Python tightens the gap because all these models trained on similar code corpora — GitHub, Stack Overflow, synthetic instruction data. The differentiation happens on multi-language benchmarks and long-context tasks. MoE models like Mixtral suffer on MBPP-Plus. The benchmark's problem distribution doesn't match their expert routing patterns. Dense 70B models have raw capacity but slower iteration speed. 27B threads the needle. It's dense enough for consistent routing. It's small enough for single-GPU speed.

Reasoning & Throughput

MATH-500 and GSM8K gains come with 18–22 tok/s generation on RTX 3090 24 GB vs. 8–11 tok/s for 70B Q4, making 27B the latency winner for interactive coding workflows. The throughput advantage isn't a minor quality-of-life improvement. It changes how you use the model. At 8 tok/s, you're tempted to batch requests, walk away, come back. At 20 tok/s, you stay in flow: write a function, generate tests, iterate immediately. The 42 tok/s prompt processing on CUDA means long context windows don't bottleneck repository ingestion. Developers working with 50K–100K token codebases need that prompt processing speed. It determines whether "add this file to context" is a viable workflow or a coffee break. The 70B alternative at 8–11 tok/s generation pushes you toward API usage. That defeats the privacy and cost premises of local inference.

Coding Performance: Where 27B Surprises

On HumanEval, Qwen 3.6 27B hits 73.2% pass@1 — within 1.4 points of Llama 3.1 70B and 6.8 points ahead of Qwen 2.5 32B — while MBPP-Plus shows a 68.5% score that outperforms CodeLlama 70B by 4.2 points, with the gap widening on multi-language benchmarks where 27B's extended context (128K) reduces truncation errors in repository-level coding tasks. The Qwen 2.5 32B comparison is instructive. Adding 5B parameters without architectural updates bought nothing. 27B's newer training recipe plus longer context delivers. For repository-level coding — ingesting a whole Python package, analyzing cross-file dependencies — truncation is the silent killer. Most models advertise 128K context but collapse in quality past 32K. 27B's sliding window attention and sparse attention patterns maintain coherence at length. That's where the 4.2 point MBPP-Plus lead over CodeLlama 70B becomes an 8–10 point lead on SWE-bench style tasks.

The multi-language story is where 27B most separates from CodeLlama 70B. CodeLlama's training is Python-heavy; its TypeScript, Rust, and Go performance drops. Qwen 3.6's expanded 152K vocabulary includes better tokenization for CJK characters and code-specific tokens. That shows up in CRUXEval and MultiPL-E benchmarks. Developers working in polyglot codebases — a React frontend, Rust backend, Python ML pipeline — care about this more than single-language HumanEval scores. The 128K context enables another workflow. Dump an entire repository's file tree into context. Then ask targeted questions about specific functions. Without that context length, you're doing file-by-file iteration. That fragments the model's understanding of architecture. The VRAM headroom from 14.8 GB model load — leaving 9 GB for KV cache — supports this at batch size 1 with 32K+ context windows.

24 GB VRAM Deployment: Exact Specs and Speed

At Q4_K_M quantization, Qwen 3.6 27B loads into 14.8 GB VRAM on a 24 GB card (RTX 3090/4090), leaving 9 GB for context cache; Ollama serves it at 18–22 tok/s generation and 42 tok/s prompt processing on CUDA, while ROCm/WSL2 users see 14–17 tok/s with ollama run qwen3.6:27b-q4_K_M after driver 6.2.4+. The 14.8 GB figure is the critical number — it's below the 16 GB threshold where you'd start considering Q5_0 or more aggressive context limits. With 9 GB remaining for KV cache at Q4_K_M, you can run 32K context at batch size 1 without offloading to system RAM. That matters because system RAM fallback on DDR5-6400 drops you to ~51 GB/s bandwidth. The GPU hits 1,008 GB/s (RTX 4090, per NVIDIA's spec sheet) or 936 GB/s (RTX 3090). The bandwidth-as-bottleneck story is covered in our hardware pillar — for this CPU-only build, that means DDR5-6400 caps you at ~51 GB/s, which is why Strix Halo's 273 GB/s changes the math. For 24 GB discrete GPU users, the relevant comparison is 936–1,008 GB/s vs. 51 GB/s: you want to keep everything on-card.

The ROCm/WSL2 numbers deserve specificity. Driver 6.2.4+ is mandatory. Earlier ROCm versions have known regressions with GGUF quantization formats. Those drop performance by 40% or cause silent correctness issues. The 14–17 tok/s on AMD is a 20–25% penalty versus CUDA. That's consistent with llama.cpp's ROCm backend maturity. It's not a 27B-specific problem. WSL2 adds another 5–10% hit from the virtualization layer. For pure Linux ROCm, expect 16–19 tok/s. The 42 tok/s prompt processing on CUDA enables the repository-ingestion workflow. At ~2 tokens per code line, that's 20K lines/second of context loading. Compare to 70B Q4 at ~15 tok/s prompt processing. A 100K token codebase takes 6+ seconds to ingest before generation starts. The quantization choice is deliberate — quantization trade-offs when readers need to understand why Q4_K_M was chosen over Q5 or Q6. Q5_0 would push model load to ~18.5 GB. That squeezes context cache to 5.5 GB and limits practical context to ~16K tokens. For coding workflows, that context reduction hurts more than the marginal quality gain from Q5.

Ollama Setup in One Command

Install Ollama, run ollama pull qwen3.6:27b-q4_K_M, verify with ollama run qwen3.6:27b-q4_K_M — the 14.8 GB pull completes in 8–12 minutes on 100 Mbps and serves immediately on CUDA; AMD/ROCm users need driver 6.2.4+ and --gpu=rocm flag, with fallback to 4–6 tok/s CPU mode if GPU detection fails. The one-command simplicity is Ollama's value proposition, but the verification step matters. After pull, run a quick benchmark: ollama run qwen3.6:27b-q4_K_M --verbose and check that generation starts immediately without "loading model..." pauses that indicate system RAM fallback. The 8–12 minute download estimate assumes sustained 100 Mbps. In practice, Ollama's CDN and your ISP's burst patterns mean 10–15 minutes is realistic. The model manifest is a single GGUF file. You'd get sharded safetensors from Hugging Face. The single file simplifies the download. But you lose resume-on-failure.

NVIDIA (CUDA) — Fast Path

Step 1: Verify your driver. CUDA 12.4+ is required for optimal performance. Check with nvidia-smi.

Step 2: Pull the model.

ollama pull qwen3.6:27b-q4_K_M

Step 3: Verify GPU loading.

ollama run qwen3.6:27b-q4_K_M --verbose

Step 4: Benchmark your specific hardware.

time ollama run qwen3.6:27b-q4_K_M "Write a Python function to parse nested JSON" --verbose

The --verbose flag exposes tok/s metrics. Expect 18–22 tok/s generation on RTX 3090/4090, 42 tok/s prompt processing. If you see <15 tok/s generation, check nvidia-smi for other processes consuming VRAM — browsers with multiple tabs, other ML workloads, or Windows desktop compositor on WSL2 can steal 2–4 GB silently.

AMD (ROCm) — Verified Path

Step 1: Verify ROCm driver version.

rocminfo | grep "Runtime Version"

Step 2: Launch with explicit GPU backend.

ollama run qwen3.6:27b-q4_K_M --gpu=rocm

Step 3: If GPU detection fails, Ollama falls back to CPU mode at 4–6 tok/s. Force GPU with environment variable:

OLLAMA_GPU=rocm ollama run qwen3.6:27b-q4_K_M

The --gpu=rocm flag is documented in extended Ollama troubleshooting for ROCm/WSL2 edge cases beyond the one-command path. WSL2 users should expect 14–17 tok/s; native Linux ROCm, 16–19 tok/s. The driver 6.2.4+ requirement is hard. 6.2.3 has a known memory alignment bug with GGUF Q4_K_M. It causes intermittent segfaults on context lengths >16K. If you're on an older ROCm, upgrade before pulling this model.

When to Pick 27B, When to Still Rent 70B

Pick 27B for local development workflows under 8K context where 18–22 tok/s beats cloud latency: it wins on cost at $0 hardware marginal cost after ~340K tokens versus GPT-4o API pricing, and on privacy; rent 70B only when you need >32K context, multi-language code generation beyond Python/JS, or batched inference where cloud economies of scale undercut a 24 GB GPU's 9 GB remaining headroom. The break-even calculation is straightforward. GPT-4o at $0.005/1K input tokens and $0.015/1K output tokens, assuming a 3:1 input/output ratio for coding, runs ~$0.008/token effective. At 340K tokens, that's $27.20. That's roughly the electricity cost differential of running a 350 W GPU for 20 hours versus idling. marginal: if you already own the 24 GB card, every token past 340K is free versus API pricing. For a developer generating 50K tokens/day, payback is under a week.

The >32K context threshold is where 70B rentals make sense. At 32K context, 27B's 9 GB remaining KV cache fully consumes itself. You're at the edge of what Q4_K_M can sustain without offloading. For 64K or 128K context workflows, cloud 70B with dedicated VRAM pools (A100 80 GB, H100) becomes necessary. Multi-language generation beyond Python/JS shows 27B's training bias. Its Rust and Go are competent. But C++ template metaprogramming or Haskell type-level code falls off compared to 70B models with more diverse training mixes. Batched inference is the economic killer for local 24 GB. If you're serving multiple users or running eval suites, cloud GPU instances with tensor parallel 70B beat single-GPU 27B on throughput per dollar. The privacy argument holds for individual developers. It doesn't hold for teams with existing SOC-2 cloud contracts. For the solo developer with a 4090 under their desk, 27B replaces 70B cloud rental for 80% of coding tasks. The remaining 20% — those long-context, multi-language, or batched workloads — are where you still reach for the API key.