Llama vs Qwen vs Gemma on a CPU: 3.5 GiB of KV Cache for a 0.6B Model
DOI: 10.5281/zenodo.24177567[1] · View on Zenodo (CERN)
| Badge | Metric | Value | Status | Description |
|---|---|---|---|---|
| [s] | Reviewed Sources | 12% | ○ | ≥80% from editorially reviewed sources |
| [t] | Trusted | 88% | ✓ | ≥80% from verified, high-quality sources |
| [a] | DOI | 69% | ○ | ≥80% have a Digital Object Identifier |
| [b] | CrossRef | 15% | ○ | ≥80% indexed in CrossRef |
| [i] | Indexed | 65% | ○ | ≥80% have metadata indexed |
| [l] | Academic | 73% | ○ | ≥80% from journals/conferences/preprints |
| [f] | Free Access | 100% | ✓ | ≥80% are freely accessible |
| [r] | References | 26 refs | ✓ | Minimum 10 references required |
| [w] | Words [REQ] | 3,768 | ✓ | Minimum 1,000 words of article body. Current: 3,768 |
| [u] | Original [REQ] | 100% | ✓ | ≥85% unique sentences (no padding or repeated blocks) and no leaked drafting notes. Current: 100% unique |
| [e] | Evidence [REQ] | 21 | ✓ | Cites at least 1 external source(s): links, DOIs or a reference list. Current: 21 |
| [d] | DOI [REQ] | ✓ | ✓ | Zenodo DOI registered for persistent citation. DOI: 10.5281/zenodo.24177567 |
| [o] | ORCID | ✓ | ✓ | Author ORCID verified for academic identity |
| [p] | Peer Reviewed | — | ○ | Peer reviewed by an assigned reviewer |
| [h] | Freshness | 73% | ✓ | ≥60% of references from 2025–2026. Current: 73% |
| [c] | Data Charts | 4 | ✓ | Original data charts from reproducible analysis (min 2). Current: 4 |
| [g] | Code | ✓ | ✓ | Source code available on GitHub |
| [m] | Diagrams | 3 | ✓ | Mermaid architecture/flow diagrams. Current: 3 |
| [x] | Cited by | 0 | ○ | Referenced by 0 other hub article(s) |
Abstract #
Engineers who search “llama vs qwen” or “gemma vs llama” usually want to know what they actually get when one of these models runs on hardware they already own. We ran the current ~1B-class model from each family side by side on an ordinary 6-vCPU x86 server with no GPU: Llama-3.2-1B-Instruct, Gemma-3-1B-it, Qwen3-0.6B and Qwen2.5-0.5B-Instruct, all in bfloat16 through PyTorch. For each model we measured zero-shot accuracy on 30 items each of ARC-Easy, ARC-Challenge and HellaSwag, prefill and decode speed at 2 and 6 threads, and peak resident memory. We also computed the key-value (KV) cache per token from each configuration and checked it against the cache the model actually allocated. Llama-3.2-1B scored highest on pooled accuracy, at 52.2% (95% interval 42.0-62.2%) against 42.2% for Qwen2.5-0.5B, but on 90 items per model all four intervals overlap. Llama-3.2-1B also decoded at 13.653 tokens/s on 6 threads. That is faster than the half-sized Qwen3-0.6B, which reached 8.294 tokens/s. The largest gap is in memory. Qwen3-0.6B stores 112 KiB of KV cache per token, 3.5 times as much as Llama and 9.3 times as much as Qwen2.5. At 32k tokens that is 3,584 MiB, or 2.5 times its own weight file, while Gemma’s sliding-window design needs 139 MiB. Mistral’s smallest current open model has 3.85 billion parameters, so it is not in this race.
1. Introduction #
The architecture side of this comparison is covered in our earlier explainer on how Llama, Mistral, Gemma and Qwen differ in attention layout and memory behaviour [1][2]. This article measures. Instead of reading configuration tables, we loaded the smallest current model from each family on a plain CPU server and asked what an engineer receives for each gigabyte of RAM and each second of compute.
The question matters because the 1B class is where these families meet on cheap hardware. Meta positions Llama 3.2 1B for “highly constrained environments, such as mobile devices” [2][3]. Google’s Gemma 3 family starts at 1B parameters [3][4]. Alibaba’s Qwen3 dense line starts at 0.6B [4][5]. Mistral’s newest compact line starts at 3B [5][6]. Vendor tables report accuracy, but they rarely report decode speed on a CPU, resident memory, or how fast the KV cache grows. Those numbers decide whether a model fits next to an application on an existing server.
RQ1: Llama vs Qwen vs Gemma at ~1B parameters: how far apart is their zero-shot accuracy on standard multiple-choice benchmarks, and is the gap larger than the measurement noise? RQ2: On an ordinary CPU, how fast does each model prefill and decode at 2 and 6 threads, and how much memory does it hold while doing so? RQ3: How much KV cache does each model need per token and at 4k and 32k context, and how does that change which model is cheapest to run?
2. Existing Approaches (2025-2026 state of the art) #
2.1 Small-model surveys and leaderboards #
A 2025 survey of about 160 papers argues that models in the 1-8B range can match or beat much larger ones on many tasks [6][7]. SLM-Bench, published in Findings of EMNLP 2025, evaluates 15 small language models on 9 tasks and 23 datasets across 4 hardware configurations. It reports 11 metrics covering correctness, computation and consumption, and finds that some models lead on accuracy while others lead on energy efficiency [7][8]. Studies like these are broad, but they rarely show the four numbers an operator needs for a single CPU box side by side: accuracy, tokens per second, resident memory and KV-cache growth.
2.2 Edge and CPU deployment studies #
Hardware-centred studies ask a different question: which backend runs a small model best? A 2025 comparison of CPUs, GPUs and NPUs found that neural processing units won by a wide margin. It also found that results must be normalised by memory bandwidth before architectures can be compared fairly [8][9]. On a Raspberry Pi 5 and two Jetson boards, a 2025 energy study found that Llama 3.2 gave the best balance of accuracy and power efficiency among the models it tested [9][10]. These studies fix the model and vary the hardware. We do the opposite: one ordinary server, with one model from each family.
2.3 The model reports themselves #
Each family documents its own design. The Llama 3 paper describes the dense transformer and the grouped-query attention (GQA) used across the herd [10][11], and the 3.2 card lists the 1B model at 1.23B parameters, with shared embeddings, a 128k context and up to 9 trillion training tokens [2][3]. Gemma 3 alternates five local sliding-window layers with one global layer, “to reduce the KV-cache memory that tends to explode with long context” [3][4]. Qwen3-0.6B has 28 layers, 16 query and 8 KV heads, tied embeddings and a 32K context [4][5]. Its model card puts the non-embedding parameter count at 0.44B [11][12]. Its predecessor, Qwen2.5, was pre-trained on 18 trillion tokens; Qwen3 doubled that to about 36 trillion [12][13] [4][5]. Gemma 3 1B was trained on 2 trillion tokens and accepts 32K tokens of context [13][14].
2.4 Attention layout and the KV cache #
The KV cache stores keys and values for every past token, so that each new token does not recompute them. Its size per token is fixed by the configuration: layers × KV heads × head dimension × 2 (keys and values) × bytes per value. An EMNLP 2025 study showed that common GQA configurations are “highly suboptimal for long-context scenarios” and that better choices can cut both memory and FLOPs by more than 50% compared with Llama-3’s GQA [14][15]. A 2026 survey of serving systems treats the KV cache as the central memory problem of LLM inference [15][16]. For a 1B model on a CPU, the cache can outgrow the weights, as Section 4.3 shows.
2.5 Quantised runtimes #
Most local deployments use llama.cpp with 3-8-bit GGUF files. A 2026 study measured those formats on Llama-3.1-8B for accuracy, perplexity and CPU prefill and decode throughput [16][17]. Another 2026 study predicted llama.cpp decode throughput from file metadata. A model that counted only the parameters active for each token reached 13.1% mean absolute percentage error on one held-out set, against 49.4% when it counted all parameters [17][18]. We stay in bfloat16, the format the weights ship in, so the four families are compared on equal terms. Quantisation shrinks every bar in our charts, but it does not change the per-token KV formula.
flowchart TD
Q[Which 1B-class open model for a CPU box?] --> A[Small-model surveys and leaderboards]
Q --> B[Edge and CPU hardware studies]
Q --> C[Vendor technical reports and cards]
Q --> D[Attention layout and KV cache research]
A --> A1[Accuracy across many tasks, little on memory]
B --> B1[Fix the model, vary the hardware]
C --> C1[Self-reported scores, different shot counts]
D --> D1[Formulas and serving systems, no small-model table]
A1 --> G[Gap: one CPU, one model per family, accuracy plus speed plus RAM plus KV cache]
B1 --> G
C1 --> G
D1 --> G
3. Method and Data #
3.1 Models #
| Family | Model | Parameters | Embedding share | Weights on disk, bf16 | Licence |
|---|---|---|---|---|---|
| Llama | Llama-3.2-1B-Instruct | 1,235,814,400 | 21.25% | 2.472 GB | Llama 3.2 Community License |
| Gemma | Gemma-3-1B-it | 999,885,952 | 30.2% | 2.0 GB | Gemma Terms of Use |
| Qwen | Qwen3-0.6B | 596,049,920 | 26.1% | 1.503 GB | Apache-2.0 |
| Qwen | Qwen2.5-0.5B-Instruct | 494,032,768 | 27.56% | 0.988 GB | Apache-2.0 |
Source: our parameter counts from the loaded models; file sizes from the Hugging Face Hub. Licences from each model card.
The official Meta and Google repositories require a licence click-through, so we used ungated mirrors whose weight files are identical in size to the official ones. Each file was checked against the Hub’s SHA-256 hash while it loaded. We included two Qwen generations because Qwen’s ~1B-class slot changed shape between them, and that change turns out to be the biggest memory finding in this article. The next size up, Qwen2.5-1.5B, did not fit our disk budget.
3.2 Accuracy #
We scored the first 30 items of the ARC-Easy test set, the ARC-Challenge test set [18][19] and the HellaSwag validation set [19][20], giving 90 four-way questions per model. Scoring follows the zero-shot log-likelihood method of the Language Model Evaluation Harness: the model reads the question, we sum the log-probability of each answer’s tokens, and we pick the answer with the highest sum divided by its length in bytes (“acc_norm”). No chat template is applied. The harness authors warn that results are sensitive to exactly these choices [20][21], so our numbers are comparable with each other, not with vendor tables. We report 95% Wilson intervals. A NeurIPS 2025 analysis showed that small evaluations make noisy decisions [21][22], and 30 items per task is small. We use the accuracy test as a sanity check, not a leaderboard.
To save compute, each question is run once and its KV cache is reused for all four answers. On two items per model, we compared this against a full forward pass per answer. The largest log-likelihood difference per model ranged from 0.29 to 0.43 nats, which is consistent with bfloat16 rounding, and the chosen answer was the same in every check.
3.3 Speed, memory and KV cache #
Speed uses a fixed 512-token prompt. We time the prefill (one forward pass over the prompt) and then 128 greedy decode steps, first with PyTorch limited to 6 threads and then to 2. Memory is the process’s peak resident set size (VmHWM) after the accuracy and speed runs. The KV cache per token comes from each configuration. For Gemma, the 22 sliding-window layers are capped at their 512-token window and only the 4 global layers grow with context. We checked the formula against the cache tensors actually allocated after the 512-token prefill. The two matched exactly for Llama and both Qwen models. Gemma’s allocated cache was 0.9983 of the formula, because its sliding layers kept one token fewer than the window.
The machine is a 6-vCPU AMD EPYC virtual server at 2.0 GHz with AVX2 but no AVX-512 or AMX, so bfloat16 matrix maths has no native hardware path. Other jobs shared the host, so treat the speed numbers as typical of a busy server rather than a quiet laboratory. Runs used torch 2.10.0 and transformers 5.2. The four benchmark runs took 1,073.4 seconds of wall time in total.
flowchart LR
H[Hugging Face Hub, pinned revisions] --> S[Stream weights into RAM, SHA-256 check]
S --> A[Zero-shot scoring: ARC-Easy, ARC-Challenge, HellaSwag]
S --> P[Speed: 512-token prefill, 128-token decode, 6 and 2 threads]
S --> K[KV cache: config formula vs allocated tensors]
A --> R[results.json]
P --> R
K --> R
R --> C[Charts and derived metrics per GiB and per second]
Code, data subsets, raw per-model outputs and charts: stabilarity/hub/research/llama-vs-qwen-vs-gemma-cpu-benchmark.
4. Results #
4.1 RQ1, accuracy: Llama vs Qwen vs Gemma on three benchmarks #
Pooled over the 90 questions, Llama-3.2-1B answered 47 correctly (52.2%), Gemma-3-1B 43 (47.8%), Qwen3-0.6B 39 (43.3%) and Qwen2.5-0.5B 38 (42.2%). Every model is clearly above the 25% chance level. The ranking follows model size, but the evidence for it is thin. Llama’s interval runs from 42.0% to 62.2% and Qwen2.5’s from 32.5% to 52.5%, so even the widest gap, 10.0 points, sits inside overlapping intervals.

Per task, ARC-Easy put Llama and Qwen2.5 level at 60.0%, with Gemma and Qwen3 at 53.3%. ARC-Challenge was the hardest set for every model: Gemma reached 40.0%, Llama and Qwen3 36.7%, and Qwen2.5 30.0%. Its interval reaches down to 16.7%, so it cannot be told apart from chance. HellaSwag, the sentence-completion test, separated the models most: 60.0% for Llama, 50.0% for Gemma, 40.0% for Qwen3 and 36.7% for Qwen2.5. Without length normalisation, Llama still scores highest on each task, at 73.3% on ARC-Easy, 43.3% on ARC-Challenge and 50.0% on HellaSwag.
Llama vs Qwen. Llama-3.2-1B leads both Qwen models on the pooled score and on HellaSwag. It also has about twice the parameters of Qwen3-0.6B and 2.5 times those of Qwen2.5-0.5B, so the comparison is not like for like in size. Per billion parameters, the Qwen models give more accuracy above chance: 34.86 points for Qwen2.5 and 30.76 for Qwen3, against 22.03 for Llama.
Vendor numbers are not comparable. Meta reports 59.4 on ARC-Challenge and 41.2 on HellaSwag for the 1B instruct model, both zero-shot [2][3]. Google reports 38.4 on ARC-Challenge (25-shot) and 62.3 on HellaSwag (10-shot) for the pretrained Gemma 3 1B [13][14]. Different prompts, shot counts and scoring rules move these numbers by tens of points. That is why we put every model through one identical procedure.
4.2 RQ2, speed and RAM: qwen vs gemma vs llama on six CPU cores #

Prefill tracks model size. At 6 threads, Qwen2.5-0.5B processed the 512-token prompt at 96.14 tokens/s, Qwen3-0.6B at 83.14, Gemma-3-1B at 55.15 and Llama-3.2-1B at 45.76. Going from 2 to 6 threads made prefill 1.946 to 2.227 times faster.
Decode, which produces the words a user waits for, does not follow size. At 6 threads, Qwen2.5-0.5B decoded at 15.267 tokens/s and Llama-3.2-1B at 13.653. Gemma-3-1B reached 9.981 and Qwen3-0.6B 8.294. Qwen3-0.6B has 0.482 times Llama’s parameters but decodes at only 0.607 times Llama’s speed. It also hardly benefits from extra threads: 7.843 tokens/s at 2 threads and 8.294 at 6, a 1.058-fold gain, against 1.745 for Llama. Our reading is that one decode step in a small, deep model is dominated by fixed per-layer costs, not by arithmetic. Qwen3-0.6B runs 28 thin layers per token, Llama 16 wider ones. This fits a wider observation from llama.cpp measurements: headline parameter count alone is a poor predictor of decode throughput [17][18].
The thread result has a practical consequence. Three 2-thread processes would each run at the 2-thread rate. Multiplying that rate by three gives an upper-bound aggregate decode of 23.48 tokens/s for Llama, 23.07 for Gemma, 23.53 for Qwen3 and 35.44 for Qwen2.5. That is 1.72 to 2.837 times the single 6-thread process. These are our estimates, not measurements: the three processes would compete for memory bandwidth, and each would need its own copy of the weights in RAM. For a service that handles many short requests, several small workers still beat one wide one.
| Model | Pooled acc_norm (95% CI) | Prefill 6t (tok/s) | Decode 6t (tok/s) | Decode 2t (tok/s) | Peak RSS (GiB) | KV per token (KiB) | KV at 32k (MiB) |
|---|---|---|---|---|---|---|---|
| Llama-3.2-1B-Instruct | 52.2% (42.0-62.2) | 45.76 | 13.653 | 7.826 | 3.152 | 32 | 1,024 |
| Gemma-3-1B-it | 47.8% (37.8-58.0) | 55.15 | 9.981 | 7.689 | 2.93 | 26 (full) | 139 |
| Qwen3-0.6B | 43.3% (33.6-53.6) | 83.14 | 8.294 | 7.843 | 2.171 | 112 | 3,584 |
| Qwen2.5-0.5B-Instruct | 42.2% (32.5-52.5) | 96.14 | 15.267 | 11.813 | 1.783 | 12 | 384 |
Source: our measurements on a 6-vCPU AVX2 server, bfloat16, November 2026; results.json in the linked repository. “6t” and “2t” are PyTorch thread counts. KV per token for Gemma is before the sliding window applies.
Peak resident memory ranged from 1.783 GiB for Qwen2.5 to 3.152 GiB for Llama. That includes the Python runtime, PyTorch and a short context, not only the weights. Measured as points above chance per GiB of peak memory, the four models are close: 9.66 for Qwen2.5, 8.64 for Llama, 8.45 for Qwen3 and 7.77 for Gemma (Chart 4).

Accuracy and speed can also be combined into one figure: pooled accuracy multiplied by decode tokens per second, which is roughly correct answers delivered per second of generation. On that measure Llama scores 7.13, Qwen2.5 6.446, Gemma 4.769 and Qwen3 3.594.
4.3 RQ3, memory: gemma vs llama vs qwen on the KV cache #

This is where the families differ most. Qwen3-0.6B has the fewest parameters of the three current models and the largest KV cache: 28 layers × 8 KV heads × 128 dimensions × 2 × 2 bytes = 112 KiB per token. Llama-3.2-1B stores 32 KiB (16 layers × 8 heads × 64 dimensions) and Qwen2.5-0.5B 12 KiB (24 layers × 2 heads × 64 dimensions). Without its window, Gemma-3-1B would store 26 KiB (26 layers × 1 head × 256 dimensions). Qwen3’s cache per token is 3.5 times Llama’s and 9.333 times that of its own predecessor.
The context length decides how much that costs. At 4k tokens, Qwen3 needs 448 MiB of cache, Llama 128, Qwen2.5 48 and Gemma 27. At 32k, Qwen3’s full advertised context, it needs 3,584 MiB, 2.5 times its 1.503 GB weight file and more than its whole 2.171 GiB peak in our runs. Llama needs 1,024 MiB at 32k, 7.367 times Gemma’s 139 MiB. Gemma’s advantage comes from the design described in its report, where only 1 layer in 6 attends to the full context [3][4]. In the 1B configuration we ran, that leaves 4 global layers and a 512-token window on the other 22 (the report describes a 1,024-token span for the family). At 32k, Qwen3’s cache is 25.784 times Gemma’s.
Two practical consequences follow. First, on a machine with a few spare gigabytes, Qwen3-0.6B’s long-context limit is set by RAM, not by its 32K window. Two concurrent 32k-token sessions would need about 7 GiB of cache on top of the weights. Second, the GQA settings of these small models look tuned for short-context quality more than for memory. That is consistent with the EMNLP 2025 result that common GQA choices waste memory at long context [14][15].
flowchart TD
S[Need a 1B-class model on a CPU] --> C{Typical context per request?}
C -->|Long, 8k to 32k| G[Gemma-3-1B: 139 MiB KV at 32k]
C -->|Short, under 4k| Q{Priority?}
Q -->|Best accuracy| L[Llama-3.2-1B: 52.2 percent pooled, 13.653 tok/s]
Q -->|Lowest RAM and fastest| W[Qwen2.5-0.5B: 1.783 GiB, 15.267 tok/s]
Q -->|Apache-2.0 and newest Qwen| T[Qwen3-0.6B: watch the 112 KiB per token cache]
G --> V[Validate on your own task before committing]
L --> V
W --> V
T --> V
4.4 Why Mistral is not in the small-model race #
Mistral’s newest compact family, Ministral 3, comes in 3B, 8B and 14B sizes, each with base, instruct and reasoning variants, image understanding and an Apache 2.0 licence [5][6]. Its smallest instruct checkpoint has 3,849,090,048 parameters according to the Hub’s safetensors metadata, including the vision encoder [22][23]. In bfloat16 that is about 7.7 GB (3.85 billion × 2 bytes, our arithmetic), more than three times Llama-3.2-1B and well beyond the memory our test machine had free. We found no general-purpose open Mistral text model below 3B on the Hub as of November 2026. A fair “qwen vs llama vs gemma vs mistral” test at ~1B parameters therefore cannot be run, so Mistral enters this comparison as an architecture, not a measurement. Our cross-architecture article covers that side [1][2].
5. Discussion #
What practitioners should take from this. At 1B parameters on a CPU, the family label matters less than three numbers: decode speed on your thread budget, resident memory, and KV cache per token at the context you actually use. Llama-3.2-1B had the best accuracy and decode throughput of the 1B pair. Qwen2.5-0.5B was the leanest and fastest overall, and its accuracy was statistically indistinguishable from the others on our sample. Gemma-3-1B is the only one of the four whose long-context memory stays small. Qwen3-0.6B, the newest and smallest current model, was the slowest to decode and the most expensive to hold at long context. For a cost-sensitive deployment, that is the opposite of what its parameter count suggests. Our earlier analysis of local deployment costs explains how these memory figures translate into hardware budgets [23][24].
These results agree with the wider literature in two places. Llama 3.2 came out best on balance in the edge energy study as well [9][10]. Model rankings also shift with the efficiency metric chosen, as SLM-Bench found at larger scale [7][8]. Our thread results add a cheap operational lever: on CPUs, several narrow workers serve more tokens than one wide one, most of all for small, deep models.
Limitations and threats to validity.
- Sample size. 30 items per task and 90 per model give intervals about 20 points wide. The accuracy ordering is a point estimate, and none of the pairwise gaps is established at the 95% level. Larger subsets would be the first thing to add [21][22].
- Scoring setup. Zero-shot log-likelihood without chat templates favours models trained on raw text. Instruction-tuned use with generated answers could rank the models differently [20][21]. Qwen3’s thinking mode plays no part in log-likelihood scoring.
- Runtime. We ran eager PyTorch in bfloat16 on a CPU without native bfloat16 instructions. llama.cpp with 4-8-bit weights would be faster and smaller for every model [16][17], and the decode ordering could change under a runtime with lower per-layer overhead.
- Shared host. Other jobs ran on the same server. Absolute speeds can vary between runs, and each speed figure comes from a single run.
- Peak memory. Peak resident memory includes the Python runtime, PyTorch and our scoring buffers, so it overstates what a lean server process needs. The KV-cache figures are exact for the configurations as published.
- Mirrors. The Llama and Gemma weights came from ungated mirrors. The file sizes match the official repositories, but we did not compare hashes against the gated originals.
What would change the conclusion. Three findings would change it. Larger evaluation sets that separate Llama from Qwen2.5 beyond the intervals would turn a hint into a ranking. A quantised runtime in which Qwen3-0.6B decodes faster than Llama-3.2-1B would remove its speed penalty. KV-cache quantisation or a sliding-window variant of Qwen3 would shrink its 112 KiB per token. None of these would change the arithmetic of the cache formula.
6. Conclusion #
RQ1 Finding: Llama-3.2-1B leads Gemma-3-1B, Qwen3-0.6B and Qwen2.5-0.5B on zero-shot accuracy, but the lead is within the noise of a small test. Measured by pooled length-normalised accuracy over 90 questions = 52.2% (95% CI 42.0-62.2%) for Llama against 47.8%, 43.3% and 42.2% for the others, all above 25% chance, with overlapping intervals. This matters for the series because a cost decision at the 1B scale should rest on speed and memory as well as on a few points of benchmark accuracy.
RQ2 Finding: On an ordinary CPU, decode speed does not follow parameter count. Measured by greedy decode at 6 threads = 15.267 tokens/s for Qwen2.5-0.5B, 13.653 for Llama-3.2-1B, 9.981 for Gemma-3-1B and 8.294 for Qwen3-0.6B, with peak memory from 1.783 to 3.152 GiB. Qwen3-0.6B gains only 1.058 times from tripling its threads. This matters for the series because splitting a CPU into several 2-thread workers may serve 1.72 to 2.837 times more tokens than one 6-thread process (our estimate).
RQ3 Finding: The smallest current-generation model has the largest memory appetite. Measured by KV cache per token = 112 KiB for Qwen3-0.6B against 32 KiB for Llama-3.2-1B, 26 KiB for Gemma-3-1B before its window and 12 KiB for Qwen2.5-0.5B. At 32k tokens that is 3,584 MiB for Qwen3, 2.5 times its weights, against 139 MiB for Gemma. This matters for the series because at long context the KV cache, not the weights, decides what fits on a cheap server.
The next question is whether 4-bit runtimes and KV-cache quantisation keep these rankings or reshuffle them on the same hardware.
References (24) #
- Stabilarity Research Hub. (2026). Llama vs Qwen vs Gemma on a CPU: 3.5 GiB of KV Cache for a 0.6B Model. doi.org. dtl
- Stabilarity Research Hub. (2026). Cross-Architecture Memory Comparison — Llama vs Mistral vs Gemma vs Qwen. tib
- meta-llama. (2024). meta-llama/llama-models (GitHub repository). github.com. tr
- (2025). [2503.19786] Gemma 3 Technical Report. doi.org. dtil
- (2025). [2505.09388] Qwen3 Technical Report. doi.org. dtil
- Liu, Alexander H., Khandelwal, Kartik, Subramanian, Sandeep, Jouault, Victor, et al.. (2026). Ministral 3. doi.org. dtil
- Gupta, Akanksha, Thomas, Bijo, Asnani, Harshita, Madduru, Phanindra Reddy, et al.. (2025). Small Language Models (SLMs) Can Still Pack a Punch: A survey (updated 2026). doi.org. dtil
- Nghiem Thanh Pham, Tung Kieu, Duc Manh Nguyen, Son Ha Xuan, et al.. (2025). SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts. doi.org. dcrtil
- Prieto, Pablo, Abad, Pablo. (2025). Edge Deployment of Small Language Models, a comprehensive comparison of CPU, GPU and NPU backends. doi.org. dtil
- Islam, Md Romyull, Deng, Bobin, Dhar, Nobel, Nguyen, Tu N., et al.. (2025). Characterizing and Understanding Energy Footprint and Efficiency of Small Language Model on Edges. doi.org. dtil
- Grattafiori, Aaron, Dubey, Abhimanyu, Jauhri, Abhinav, Pandey, Abhinav, et al.. (2024). The Llama 3 Herd of Models. doi.org. dtil
- (2025). huggingface.co. r
- (2024). [2412.15115] Qwen2.5 Technical Report. doi.org. dtil
- (2025). ai.google.dev. v
- Yingfa Chen, Yutong Wu, Chenyang Song, Zhen Leng Thai, et al.. (2025). Cost-Optimal Grouped-Query Attention for Long-Context Modeling. doi.org. dcrtil
- Jiang, Jiantong, Yang, Peiyu, Zhang, Rui, Liu, Feng. (2026). Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization. doi.org. dtil
- Kurt, Uygar. (2026). Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct. doi.org. dtil
- Qiu, Xinyu, Xu, Chuhong, Su, Bo, Chen, Ziyao, et al.. (2026). GGUF-Metadata Prediction of Single-Sequence llama.cpp Throughput Across Three Systems. doi.org. dtil
- Clark, Peter, Cowhey, Isaac, Etzioni, Oren, Khot, Tushar, et al.. (2018). Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. doi.org. dtil
- Zellers, Rowan; Holtzman, Ari; Bisk, Yonatan; Farhadi, Ali; Choi, Yejin. (2019). HellaSwag: Can a Machine Really Finish Your Sentence?. doi.org. dcta
- Biderman, Stella, Schoelkopf, Hailey, Sutawika, Lintang, Gao, Leo, et al.. (2024). Lessons from the Trenches on Reproducible Evaluation of Language Models. doi.org. dtil
- David Heineman, Valentin Hofmann, Ian Magnusson, Yuling Gu, et al.. (2025). Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation. doi.org. dcrtil
- (2025). huggingface.co. r
- Stabilarity Research Hub. (2026). Local LLM Deployment — Hardware Requirements and True Costs. tb