If you are deploying large language models locally on consumer silicon—operating an NVIDIA GeForce RTX 4090 with its 24GB of high-speed GDDR6X video memory (VRAM)—you quickly confront the harsh memory wall of modern artificial intelligence. A state-of-the-art 70-billion-parameter open-weights model (such as Llama 3.3 70B or Qwen 2.5 72B) requires approximately 140 gigabytes of memory just to load its weights in uncompressed 16-bit floating-point precision (FP16). Even a more modest 32-billion-parameter model consumes over 64GB in native precision, making unquantized local inference on a single graphics card mathematically impossible. To compress multi-billion-parameter weights down into single-GPU memory envelopes, the open-source AI community engineered revolutionary post-training quantization (PTQ) formats. However, developers and homelab architects frequently face severe confusion when selecting an inference engine and compression format: GGUF (llama.cpp) versus AWQ (Activation-Aware Weight Quantization) versus EXL2 (ExLlamaV2). Does GGUF’s CPU-offloading flexibility beat EXL2’s raw token-per-second generation speed, and where does AWQ fit for high-concurrency vLLM serving? Here is Kieran Mercer’s forensic memory and compute benchmarking analysis: perplexity degradation curves, kernel memory bandwidth saturation, KV-cache quantization, and optimal RTX 4090 local inference configurations.

Kieran Mercer’s Hardware Architecture Verdict: Do not standardize on a single quantization format across all local AI workflows. If your goal is single-user interactive chat, coding completion, or local agent execution on an RTX 4090 where the entire model fits inside 24GB VRAM, EXL2 (ExLlamaV2) is the undisputed speed champion. EXL2’s custom CUDA kernels and variable bit-rate quantization (e.g. 4.25 bpw to 5.0 bpw) saturate the RTX 4090’s 1,008 GB/s memory bus, generating an astounding 85 to 110 tokens/second on 14B models and 35+ t/s on 32B models—beating GGUF by 30% to 50%. However, if you need hybrid CPU/GPU RAM offloading for models too large for 24GB (like Llama 70B running 48 layers on GPU and 32 layers on system DDR5), GGUF (llama.cpp / Ollama) is mandatory. For multi-user API serving with continuous batching in vLLM or Aphrodite, AWQ (4-bit activation-aware) delivers the lowest perplexity loss and highest batched throughput.

What Is the Difference Between GGUF, AWQ, and EXL2 Quantization?

Direct Answer: GGUF is a universal container format designed for llama.cpp that supports partial layer offloading across both CPU system RAM and GPU VRAM. EXL2 is an ultra-fast, GPU-only format optimized exclusively for modern NVIDIA CUDA hardware using variable fractional bit-rates. AWQ is an activation-aware 4-bit format designed for high-concurrency serving engines like vLLM that preserves critical outlier weights for near-zero perplexity loss.

To understand why these formats produce divergent token generation speeds and text quality, we must analyze how LLM autoregressive token generation operates on GPU silicon. In the token generation (decode) phase of an LLM, compute is strictly memory-bandwidth bound, not compute-bound. To predict a single next token, the GPU must stream every single weight of the entire neural network from VRAM across the memory bus into the streaming multiprocessors (SMs).

On an RTX 4090 featuring a 384-bit memory bus running at 21 Gbps, total theoretical memory bandwidth is 1,008 GB/s. If a 14B model is quantized down to 4 bits per weight, the total model footprint in memory is roughly 8.5 gigabytes. At 1,008 GB/s, the theoretical maximum speed at which that 8.5GB model can be read is approximately 1008 / 8.5 ≈ 118 tokens per second. How close a quantization engine gets to this theoretical limit depends entirely on the efficiency of its custom CUDA dequantization kernels.

EXL2: Fractional Bit-Rates and Memory Bus Saturation

ExLlamaV2 (EXL2), developed by turboderp, was built from scratch in pure C++ and custom CUDA to extract the absolute maximum performance from modern NVIDIA architecture (Ampere, Ada Lovelace, Hopper). Unlike traditional quantization formats that restrict models to rigid integer bit-widths (e.g. exactly 4-bit or 8-bit), EXL2 introduces variable fractional bits per weight (bpw).

Neural networks do not distribute information uniformly; early attention projection layers and downstream feed-forward network (FFN) gate matrices are far more sensitive to quantization noise than deep intermediate layers. EXL2 quantizes sensitive layers at 5.5 or 6.0 bpw while aggressively compressing non-critical layers down to 3.2 bpw. This allows developers to dial in an exact model size—such as 4.25 bpw—to squeeze a 32B model into precisely 21.5GB of VRAM, leaving 2.5GB for a massive 16k context window KV cache. On an RTX 4090, EXL2 routinely achieves 92% to 96% memory bus saturation, delivering the fastest single-user inference latency in the AI industry.

GGUF: The Universal Offloading Swiss Army Knife

GGUF (Georgi Gerganov Universal Format) is the binary file format powering the legendary llama.cpp project and its popular wrapper, Ollama. GGUF’s core architectural strength is hardware universality and memory flexibility. While EXL2 requires an NVIDIA GPU, GGUF runs seamlessly across NVIDIA CUDA, AMD ROCm, Apple Silicon Metal, Intel oneAPI, and pure CPU x86 AVX-512 instructions.

More importantly, GGUF allows partial layer offloading. If you want to run Llama 3.3 70B (which requires ~42GB at Q4_K_M), you cannot fit it onto a 24GB RTX 4090. EXL2 will simply throw a CUDA out-of-memory error and crash. GGUF, however, allows you to offload 42 layers into your 24GB VRAM and offload the remaining 38 layers into system DDR5 memory. While running split layers across the PCIe bus drops token generation speed from 35 t/s down to 8 to 12 t/s, it allows homelab developers to run massive frontier models that would otherwise require multiple enterprise GPUs.

Inference Metric (Qwen 2.5 32B on RTX 4090) EXL2 (ExLlamaV2 4.25 bpw) GGUF (llama.cpp Q4_K_M) AWQ (vLLM 4-bit GEMM)
Single-Stream Decode Speed 44.2 tokens / sec (Fastest) 31.8 tokens / sec 36.5 tokens / sec
Batched Multi-User Concurrency Moderate (Optimized for single stream) Moderate (Server mode supports slots) Exceptional (PagedAttention in vLLM)
VRAM Footprint (32B Model) 18.2 GB (Fractional tuning) 19.8 GB 19.4 GB
CPU RAM Hybrid Offloading No (Requires 100% VRAM) Yes (Seamless partial offloading) No (GPU VRAM only)
Perplexity Degradation vs FP16 Minimal (+0.12 PPL at 4.5 bpw) Low (+0.18 PPL on K-quant) Near-Zero (+0.08 PPL via outlier protection)

The 4:00 PM Actionable Implementation Playbook

To configure your local RTX 4090 homelab inference workstation for maximum speed and intelligence, implement Kieran Mercer’s hardware tuning protocol:

  1. Deploy TabbyAPI or ExLlamaV2 for Personal Coding & Chat: For personal developer agents (paired with Cline, Continue.dev, or Open WebUI), run models in EXL2 format via TabbyAPI. Download 4.0 to 5.0 bpw quants from Hugging Face (such as turboderp/Qwen2.5-32B-Instruct-exl2). This delivers instantaneous 45+ tokens/second with sub-second time-to-first-token (TTFT).
  2. Enable 8-Bit or 4-Bit KV-Cache Quantization: By default, inference engines store the attention Key-Value (KV) cache in uncompressed FP16. At 32,768 context lengths, the KV cache alone consumes 8GB of VRAM! Enable 8-bit (FP8) or 4-bit (Q4) KV-cache quantization inside ExLlamaV2 (--cache_mode fp8) or llama.cpp (-ctk q8_0 -ctv q8_0). This cuts KV cache memory consumption in half with zero perceptible loss in reasoning accuracy, freeing up space for larger models.
  3. Deploy vLLM with AWQ for Multi-Container Homelab Serving: If you are serving multiple internal microservices, LangChain agents, and family members simultaneously from your server, deploy vLLM with 4-bit AWQ models (such as casperhansen/llama-3.3-70b-instruct-awq on dual GPUs or 14B/32B on single GPU). vLLM’s PagedAttention dynamically manages KV memory, handling 10+ concurrent requests without out-of-memory crashes.
  4. Use GGUF Exclusively When Models Exceed 24GB VRAM: When you want to experiment with 70B+ frontier models on a single RTX 4090, use llama.cpp / Ollama with a GGUF Q4_K_M quant. Offload 45 layers to the GPU: 
    ollama run llama3.3:70b-instruct-q4_K_M --gpu-layers 45
    Your RTX 4090 handles the heavy matrix multiplications, while 64GB of host DDR5 memory supplies the remaining capacity, delivering reliable 10 t/s generation.

Frequently Asked Questions

Can I run GGUF models inside vLLM?

Modern vLLM versions (v0.6+) have added experimental GGUF support for certain architectures, but AWQ, GPTQ, and FP8 remain vLLM’s native, fastest formats. For pure GGUF serving, llama.cpp’s native server or Ollama is vastly more stable.

What is the difference between Q4_K_M and Q4_K_S in GGUF?

In GGUF K-quants, ‘M’ stands for Medium and ‘S’ stands for Small. Q4_K_M uses 4-bit quantization for feed-forward networks but preserves critical attention weights at higher 5-bit or 6-bit precision, offering noticeably superior reasoning quality with only a negligible 200MB size penalty over Q4_K_S.

Does quantizing a model to 4-bit make it dumber?

Modern post-training quantization methods (AWQ and EXL2) result in less than a 1% to 2% difference on standardized MMLU and HumanEval benchmarks compared to uncompressed FP16. In real-world chat and coding workflows, the quality difference is virtually imperceptible, while generation speed is tripled.