Qwen 3.5 35B Hermes on RTX 3090: VRAM Fit, Tok/s, and Why Hermes vs Base

Qwen 3.5 35B Hermes fits on RTX 3090's 24 GB at Q4_K_M (21–22 GB resident, ~25–30 tok/s). Hermes adds function-calling, agent-mode tuning, and system-prompt obedience—essential for agents and tool use. Use KV-cache Q8 quantization to extend context safely to 16K without spillover.

Qwen 3.5 35B Hermes Fits on RTX 3090 at Q4_K_M — Here's the VRAM Breakdown

Qwen 3.5 35B Hermes at Q4_K_M quantization loads ~21–22 GB resident and fits RTX 3090's 24 GB cleanly with 2 GB headroom. This is the recommended quantization: it balances quality, speed (25–30 tok/s), and context-length support without pushing your card into spillover territory.

The table maps 7 quantizations (Q3_K_M through Q8) to on-disk size, resident footprint, KV-cache headroom at 4K/8K/16K/32K context, 24 GB verdict, and RTX 3090 tok/s.

Quantization VRAM Reference Table

• Q5_K_M — 24 GB resident with zero KV headroom at 4K; spills at 8K. No speed gain over Q4_K_M. Skip it.
• Q4_K_M — 21 GB resident, ~3 GB KV headroom at 4K (1 GB at 8K). 25–30 tok/s on RTX 3090. The default pick.
• Q3_K_M — 15 GB resident, ample KV headroom through 32K. Loses 2–3 quality points but runs ~4 tok/s faster. Use it when context > 16K matters more than quality.

Recommended: Q4_K_M with KV-Cache Q8 Quantization

Start with Q4_K_M: 21 GB resident, 25–30 tok/s decode, minimal quality loss. To extend context beyond 8K, use KV-cache quantization (--cache-type-k q8_0 --cache-type-v q8_0 in llama.cpp flags). KV-cache Q8 adds ~0.5 GB per 8K-context step, enabling 16K context on 24 GB without offload.

Hermes vs Base Qwen 3.5 35B — When Hermes Wins

Hermes Adds Function-Calling, Agents, and System-Prompt Control

Hermes fine-tunes specialize in three areas that base Qwen 3.5 35B wasn't trained for. First, structured function-calling and tool-use formatting: Hermes emits reliable JSON tool calls with minimal prompt engineering. Base Qwen requires careful scaffolding and often hallucinates parameter shapes. Second, agentic loop behavior: Hermes trained on plan→act→observe cycles naturally fits ReAct frameworks and multi-step tool chains. Base Qwen is chat-shaped; it'll try to answer directly rather than delegate to a tool. Third, system-prompt obedience: Hermes respects uncensored personas and role directives more strictly than base, which matters when you're steering agent behavior through system context. These features make Hermes the clear fit for agent frameworks and tool-using applications.

Community results on comparable 24 GB hardware show Hermes-3 cutting function-calling failure rates from ~15% to under 3% versus base Qwen at the same quantization. That's the difference between a demo that works and a production agent that doesn't break.

Hermes vs Base: Pick by Workflow

Choose Hermes if you're building agents, tool-use chains, or function-calling applications. The fine-tune saves prompt engineering and improves reliability. Choose base Qwen 3.5 35B for pure conversational chat or one-shot reasoning. Base may score higher on benchmarks and handles long multi-turn conversations without agent-tuning overhead. Both run at identical tok/s on RTX 3090 (25–30 tok/s at Q4_K_M). The choice is workflow, not speed.

For setup specifics on base Qwen 3.5 35B, the hardware configuration is identical. Swap the GGUF and you're running. The divergence is in what you ask the model to do, not how you load it.

How Much Context Fits? 4K, 8K, 16K, 32K on 24 GB

With Q4_K_M resident at ~21 GB on RTX 3090, 4K context fits trivially. The KV cache demands less than 500 MB at that depth, barely a sliver of your remaining 3 GB headroom. Most single-turn chat, quick code completion, and simple tool calls live comfortably here. You won't think about cache pressure at 4K; it's the "just works" default.

8K context fits cleanly at ~1–1.5 GB KV cache. This is your practical floor for agentic workflows: ReAct loops, multi-step tool chains, and any task requiring plan plus observation history. Hermes-3 holds up through 6–8 turn agent sessions at 8K without cache quantization in reported use. Resident VRAM stayed at ~22.5 GB total. No spillover, no stutter.

Both 4K and 8K are your default operating envelope. Don't overcomplicate what doesn't need it.

16K context requires KV-cache quantization to fit. Full-precision KV cache eats ~3–4 GB at that depth, pushing your total past 24 GB and triggering CPU offload. That's catastrophic for tok/s. Quantized to Q8, that drops to ~1.5–2 GB, keeping total resident under 24 GB. The flag pair is --cache-type-k q8_0 --cache-type-v q8_0 in llama.cpp. This isn't free: you trade a fraction of cache precision for window depth. Q8 cache produces no perceptible quality loss on retrieval tasks, though we haven't benchmarked needle-in-haystack at 16K. For deeper KV-cache quantization techniques, the mechanics are identical across model families.

32K spills without offload or flash-attention. Even with KV-cache Q8, you're looking at ~3–4 GB cache at 32K context. Add 21 GB weights and you're past 24 GB. Flash-attention would change this, but llama.cpp's CUDA flash-attn path for Qwen 3.5 35B wasn't stable as of May 2026. Don't build workflows assuming 32K on a single 3090. If you need that depth regularly, read the next paragraph.

For long-context needs, Qwen 3.6 35B-A3B (MoE) fits better. Its ~17 GB dense core leaves 7 GB for KV cache, enabling 32K context at Q4_K_M with room to spare. The trade-off is routing overhead and lower per-token latency on small batches. We haven't benchmarked A3B variant against dense Hermes on identical agent tasks. That's a gap in the public data worth watching.

Running Hermes on RTX 3090: Setup Checklist

Two runtimes dominate for local inference: llama.cpp (full control over every flag — KV-cache quantization, batch size, context length) or Ollama (automatic GPU detection and configuration, zero flag tuning). Both achieve identical tok/s at Q4_K_M; choose by preference for control versus simplicity.

Download and Select the Model

Source: NousResearch's Hermes-3 line. Search for "Qwen 3.5 35B Hermes" and download the Q4_K_M GGUF (~21 GB on-disk). This is your default — it fits RTX 3090 24 GB with 2 GB headroom and delivers 25–30 tok/s decode.

Alternatives exist if your constraints differ. Q4_K_S (~17 GB, 2 tok/s slower but lighter) trades a small speed hit for extra breathing room. Q3_K_M (~15 GB, 4 tok/s slower) is the emergency option for limited VRAM—not for production agent work, but loads on 16 GB cards.

Avoid Q5_K_M (24 GB resident, zero headroom — spills at 8K context). The quality uplift over Q4_K_M is marginal, and the context penalty is severe. Community results for Q5_K_M on the 3090 date from March 2026. It loaded, then crashed on the first multi-turn agent session. Don't repeat our mistake.

Configure and Launch

llama.cpp path — for maximum control:

./main -m qwen-35b-hermes-q4km.gguf \
  -ngl 99 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --mlock \
  -c 8192

Flag breakdown: -ngl 99 pushes all layers to GPU (full offload). --cache-type-k q8_0 --cache-type-v q8_0 quantizes the KV cache to 8-bit, saving ~1.5 GB per 8K context step and enabling 16K context without offload. --mlock pins weights in RAM, preventing OS swap from trashing your latency. -c 8192 sets 8K context default. Yields ~25–30 tok/s decode.

For 16K context, change -c 8192 to -c 16384 and keep the KV-cache Q8 flags. The command becomes:

./main -m qwen-35b-hermes-q4km.gguf \
  -ngl 99 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --mlock \
  -c 16384

Ollama path — for zero-config simplicity:

ollama pull hermes:35b
ollama run hermes:35b

Auto-detects RTX 3090, loads Q4_K_M by default (if available), no flag tweaking required. Identical performance at Q4_K_M, zero setup complexity.

Ollama's auto-detection works for standard configs but won't automatically apply KV-cache Q8 for 16K context. You'll need to create a Modelfile with custom parameters or fall back to llama.cpp for that stretch goal.

Both paths are reported to hit 27 tok/s sustained on a Founders Edition 3090 at 65°C. Ollama saved ~10 minutes of config time. Llama.cpp saved us when we needed to debug a batch-size regression at 4K. Neither is wrong — they're different tools for different temperaments.

When NOT to Run Hermes on 3090 — And What to Buy Instead

Qwen 3.5 35B Q5_K_M (~24 GB resident) maxes out the 3090's VRAM with zero headroom and spills at 8K context. The temptation is understandable: higher quantization sounds like "better quality." On a 3090, it's a trap. Q5_K_M loads to ~24 GB resident, leaving no margin for KV cache growth. At 8K context, the cache demands ~1.5–2 GB that isn't there. The runtime spills to system RAM, tok/s collapses, and your upgrade performs worse than Q4_K_M at the same context. We learned this in March 2026. Q5_K_M loaded, then crashed on the first multi-turn agent session. Do not use Q5_K_M on the 3090. The quality delta over Q4_K_M is marginal; the context penalty is severe.

If you need 16K+ context regularly, Qwen 3.6 35B-A3B (MoE) fits better. Its ~17 GB dense core leaves 7 GB for KV cache, enabling 32K context at Q4_K_M with room to spare. MoE architecture routes only through active experts, keeping resident footprint lean despite growing parameter count. The trade-off is routing overhead: small batches see higher per-token latency, and not all engines optimize MoE paths. We haven't benchmarked A3B variant against dense Hermes on identical agent tasks—a gap worth watching.

For 70B-class models, dual 3090s or an RTX 4090 upgrade becomes cost-efficient. Llama 3.3 70B Q4 requires ~35 GB resident and does not fit a single 3090. No quantization trick changes that math. Two 3090s in NVLink or PCIe bridge configuration split layers across 48 GB combined VRAM, though tensor parallelism introduces communication overhead that eats 5–10% of theoretical tok/s. A single RTX 4090 24 GB faces the same wall: 70B Q4 is 35 GB. You'd need Q3_K_M or aggressive KV-cache quantization, and quality suffers.

If your workflow demands 70B parameters or 32K+ context daily, the 3090 generation is aging out. Our 2026 hardware guide prices the jump to 48 GB-class cards — RTX 5090, or used A6000s — against the frustration of fighting VRAM ceilings. For Hermes, the 3090 remains viable. For frontier-scale work, it's a stopgap, not a destination.