Adaptive LLM quantization: 41 points of HumanEval on the same weights#

A uniform latency-optimized 4-bit LLM quantization scored 39.0% on HumanEval. An adaptive 4-bit-average configuration on the same base weights scored 79.9% on the same benchmark. The two artifacts have the same average bit width and produce models of roughly the same on-disk size. They differ by 41 absolute percentage points on coding because they compress different layers differently.

That gap is the practical case for caring which quantization recipe a model card actually used. "4-bit" is a label. The mechanism underneath is what determines whether a 70B model on your Mac writes working Python or hallucinates broken syntax. The wider 2026 picture of Apple Silicon for local LLMs, including which engine emits which quantization family, sits in the wider 2026 picture of Apple Silicon for local LLMs.

The principle: not all layers tolerate compression equally#

Transformer layers do not all play the same role. Two coarse classes matter for quantization:

• Attention layers encode mathematical coherence, spatial relationships, and structural output quality. They route information between tokens and decide which prior context is salient for the next prediction. Quantization noise in attention weights propagates through every downstream layer and degrades coherent reasoning quickly. Attention is sensitive.
• MLP layers (the feed-forward blocks between attention) contain the bulk of a model's parameter count. They act primarily as a dense knowledge store, with each layer applying a per-token nonlinear transformation. MLP layers tolerate aggressive compression because their role is more distributed and individual weights matter less.

A uniform quantization scheme treats both classes the same. A 4-bit weight in attention and a 4-bit weight in MLP get the same group size, the same scale representation, the same error budget. That is operationally simple and quality-blind.

An adaptive scheme allocates precision where it matters. Attention layers stay at 4 to 8 bit. MLP layers compress to 2 to 6 bit. The average bit count can be held at roughly 4, but the load-bearing layers keep their fidelity. The result, on coding and reasoning benchmarks where attention quality is decisive, can be a difference measured in tens of percentage points rather than tenths.

The 41-point number#

The published benchmark this post takes its title from compared two configurations of one base model:

• A uniform 4-bit quantization optimized for inference latency. HumanEval: 39.0%.
• An adaptive 4-bit-average configuration that protected attention and compressed MLP harder. HumanEval: 79.9%.

The magnitude of the gap is workload-dependent. HumanEval rewards precise code generation, where one missing token destroys a function. Tasks with more error tolerance (summarization, casual chat, classification) show smaller gaps. The directional finding is what is robust: across multiple analyses, uniform aggressive quantization at sub-Q4 effective precision severely degrades coding and multi-step reasoning, and adaptive schemes recover most of that loss at the same average bit width.

The corollary for buyers running these models on Mac Studios or other local hardware: the right question is not "did this fit in 4 bits" but "which 4 bits did it use." For the underlying perplexity curve and where Q4_K_M lands relative to Q5 and Q6 on the same Mac, see the GGUF perplexity curve and where to land.

What this looks like in real GGUF quantizations#

Three concrete examples of adaptive quantization in current llama.cpp and Unsloth artifacts:

GGUF Q4_K_M. The K-quant family was designed around this principle. Q4_K_M uses Q6_K (a 6-bit scheme with super-block scale and minimum) for half of the attention.wv (value projection) and feed_forward.w2 (output projection) tensors, and Q4_K for the rest. Those two tensors are empirically the most sensitive to quantization error in transformer architectures. The other 50% of attention.wv and feed_forward.w2 weights, plus the rest of the model, stay at the smaller Q4_K. The overall artifact is "4-bit" by label but ~5.0 to 5.3 bpw in reality, with the sensitive tensors getting the extra precision they need.

Q4_K_M with imatrix. An importance matrix (imatrix) is computed by running a calibration dataset through the FP16 model and measuring per-channel activation magnitudes. The quantizer then biases error away from high-activation channels during weight assignment. The mechanism is orthogonal to the per-tensor precision split: it operates within each quantization block to better match the empirical activation distribution. Q4_K_M with imatrix typically reduces perplexity by 10 to 30% versus naive Q4_K_M at the same bit width.

Unsloth Dynamic Q4_K_XL. The Unsloth team analyzes per-layer KL divergence between the quantized and original outputs, then upcasts the layers that lose the most signal. In Dynamic Q4_K_XL the attention layers and MoE routing layers (for MoE models) are held at 8 bit while the rest of the model stays at 4 bit. This is the same idea as the published 79.9% adaptive scheme, applied as a shipping artifact rather than a research configuration. Unsloth Dynamic quants are typically several percentage points better than uniform 4-bit on coding and reasoning benchmarks at a small size penalty.

What MLX uniform 4-bit looks like#

MLX's native quantization is uniform affine group quantization with default group size 64. Every linear layer in the model gets the same treatment: weights are grouped into blocks of 64, a scale and zero-point are stored per group, and weights are rounded to 4 bits within each group. The implementation is simple and fast on Metal, and it produces a clean, predictable artifact that runs efficiently.

It also has no per-tensor sensitivity awareness. The attention layers get the same compression as the MLP layers. The attention.wv tensors that GGUF Q4_K_M held back at Q6 are quantized to the same 4 bits as everything else in MLX. For chat and summarization workloads, this is usually fine. For coding and multi-step reasoning, the same model in MLX 4-bit will score measurably worse than the GGUF Q4_K_M version on the same benchmark, despite the same nominal bit width.

This is one reason cross-engine benchmark claims need to specify the recipe. "4-bit MLX vs 4-bit GGUF" is comparing two structurally different artifacts and the speed-vs-quality tradeoff is real. The structural comparison is documented in the same-bits-different-artifact split between K-quants and MLX 4-bit.

MLX does have its own learned-quantization scheme (DWQ, distillation-based) that improves quality at 2 to 4 bits. DWQ underperforms at Q6 and Q8 because the underlying loss at those precisions is already too small for distillation to meaningfully improve. For the MLX-on-Mac case, the practical recommendation is to use DWQ or Unsloth Dynamic at 4 bit on coding workloads, and uniform 4-bit only when speed dominates and quality tolerance is high.

Practical recipe for picking a quant#

A working heuristic for local LLM users running 7B to 70B models on Apple Silicon or comparable hardware:

• Chat, summarization, classification: uniform Q4 (MLX 4-bit, GGUF Q4_K_S) is usually fine. The quality loss is small and the speed advantage is real.
• Code, math, agent loops, multi-step reasoning: prefer an adaptive scheme. GGUF Q4_K_M with imatrix, Unsloth Dynamic Q4_K_XL, or another adaptive recipe. The 41-point absolute gap on HumanEval is the worst case, not the typical case, but the directional improvement is consistent.
• Long context (16K+): the quant interacts with the KV cache budget. An adaptive 4-bit may add 5 to 10% to the on-disk footprint versus uniform 4-bit, which can matter when total unified memory is tight. Run the working-set math at your target context length before committing.
• Above Q6: the recipe matters less. The underlying loss at Q6 and Q8 is small enough that adaptive schemes give diminishing returns. Pick the artifact that loads fastest and has the broadest tool support.

What this means for the buyer#

The 41-point HumanEval gap is the load-bearing reason quantization recipe deserves first-class attention when picking a model artifact for local inference, not a footnote. Two "4-bit" models can produce dramatically different outputs on identical prompts. The label is a starting point; the recipe is the answer.

For Mac Studio buyers, the implication is that the choice of inference engine (MLX vs llama.cpp) is partly a choice of quantization family. MLX's uniform 4 bit is faster on small dense models. GGUF Q4_K_M with imatrix on the same model can be measurably more accurate. On coding and reasoning workloads, that accuracy difference is what actually matters, and the speed tradeoff is rarely worth a 30 to 40 point swing on HumanEval-class benchmarks.

The cleanest workflow is to pick the recipe before the engine: choose an adaptive 4-bit quant (Q4_K_M with imatrix, Unsloth Dynamic Q4_K_XL, or MLX DWQ) for code and reasoning work, then pick whichever engine runs that artifact fastest on your hardware. The bit label is not the contract. The recipe is. For the engine-side decision tree this slots into, see the engine-by-engine comparison of MLX and llama.cpp on Apple Silicon.