Compare

Loading community benchmarks…

Community benchmarks

Compare

Put two chips, runtimes, or runtime versions side by side. Everything else is held fixed or called out, so a ratio is worth exactly what the matched runs say.

Add your benchmark

Runs compared: 46
Like-for-like pair: 1
Runtimes available: 2

Compare

Side A

Side B

Model

Chip

Quantisation A

Quantisation B

Backend

Prefill size

Decode size

Cooldown

45 runs vs 1 run · 1 like-for-like pair

Held fixedChip: Apple M4 MaxPrefill: PP512Decode: TG128

llama.cpp has only 1 matching run. Treat its figures as indicative.
These figures mix different models. Pick a model for a like-for-like comparison.
The sides differ in conditioning, decode workload, harness schema. Ratios may reflect benchmark protocol rather than the runtime.

Runtime comparison

BaseRT vs llama.cpp

Ratios read side A relative to side B. Geometric mean over 1 like-for-like pair.

Decode: 1.38×+37.8%
Prefill: 0.91×-8.7%

Side A

BaseRT

45 runs

2 contributors

Latest 20 Sep 2026

Decode · TG128: 139.7tok/s median
best 583.3 · worst 22.8
Prefill · PP512: 1,715tok/s median
best 9,998 · worst 221

Models: gemma-3-1b-it-Q4, gemma-3-1b-it-Q8, gemma-4-26B-A4B-it-Q4, gemma-4-26B-A4B-it-Q8, gemma-4-E2B-it-Q4, gemma-4-E2B-it-Q8, gemma-4-E4B-it-Q4, gemma-4-E4B-it-Q8, gpt-oss-20b-MXFP4, gpt-oss-20b-Q4, gpt-oss-20b-Q8, Llama-3.1-8B-Instruct-Q4, Llama-3.1-8B-Instruct-Q8, Llama-3.2-1B-Instruct-Q4, Llama-3.2-1B-Instruct-Q8, Llama-3.2-3B-Instruct-Q4, Llama-3.2-3B-Instruct-Q8, Mistral-7B-Instruct-v0.3-Q4, Mistral-7B-Instruct-v0.3-Q8, muse-glimmer-30B-kquant-17gb, muse-glimmer-30B-kquant-dynamic, NVIDIA-Nemotron-3-Nano-30B-A3B-Q4, Qwen3-0.6B-Q4, Qwen3-0.6B-Q8, Qwen3-1.7B-Q4, Qwen3-1.7B-Q8, Qwen3-30B-A3B-Instruct-2507-Q4, Qwen3-30B-A3B-Thinking-2507-Q4, Qwen3-4B-Instruct-2507-Q4, Qwen3-4B-Instruct-2507-Q8, Qwen3-4B-Q4, Qwen3-4B-Q8, Qwen3-4B-Thinking-2507-Q4, Qwen3-4B-Thinking-2507-Q8, Qwen3-8B-Q4, Qwen3-8B-Q8, Qwen3.5-2B-Base-Q4, Qwen3.5-2B-Base-Q8, Qwen3.5-2B-Q4, Qwen3.5-2B-Q8, Qwen3.5-35B-A3B-Q4, Qwen3.6-27B-Q4, Qwen3.6-35B-A3B-Q4, Qwen3.8-27B-Q4, Qwen3.8-27B-Q4-mtp
Chips: Apple M4 Max
Versions: BaseRT 0.2.4, BaseRT 0.2.6
Quantisations: mxfp4, passthrough_gguf, Q4, Q8
Backends: Metal
Cooldown: off
Conditioning: warmup_only
Decode workload: TG128, TG128 @ 1 ctx
Harness schema: basert-benchmark-harness/1

Side B

llama.cpp

1 run

1 contributor

Latest 11 Sep 2026

Decode · TG128: 22.9tok/s median
best 22.9 · worst 22.9
Prefill · PP512: 243tok/s median
best 243 · worst 243

Models: Qwen3.8-27B
Chips: Apple M4 Max
Versions: llama.cpp b10809 (5266f24da)
Quantisations: Q4_K_M
Backends: BLAS + Metal
Cooldown: off
Conditioning: runtime_native_warmup
Decode workload: TG128
Harness schema: computearena-measurements/1

What differs besides the runtime

Facet	BaseRT	llama.cpp
Model	gemma-3-1b-it-Q4, gemma-3-1b-it-Q8, gemma-4-26B-A4B-it-Q4, gemma-4-26B-A4B-it-Q8, gemma-4-E2B-it-Q4, gemma-4-E2B-it-Q8, gemma-4-E4B-it-Q4, gemma-4-E4B-it-Q8, gpt-oss-20b-MXFP4, gpt-oss-20b-Q4, gpt-oss-20b-Q8, Llama-3.1-8B-Instruct-Q4, Llama-3.1-8B-Instruct-Q8, Llama-3.2-1B-Instruct-Q4, Llama-3.2-1B-Instruct-Q8, Llama-3.2-3B-Instruct-Q4, Llama-3.2-3B-Instruct-Q8, Mistral-7B-Instruct-v0.3-Q4, Mistral-7B-Instruct-v0.3-Q8, muse-glimmer-30B-kquant-17gb, muse-glimmer-30B-kquant-dynamic, NVIDIA-Nemotron-3-Nano-30B-A3B-Q4, Qwen3-0.6B-Q4, Qwen3-0.6B-Q8, Qwen3-1.7B-Q4, Qwen3-1.7B-Q8, Qwen3-30B-A3B-Instruct-2507-Q4, Qwen3-30B-A3B-Thinking-2507-Q4, Qwen3-4B-Instruct-2507-Q4, Qwen3-4B-Instruct-2507-Q8, Qwen3-4B-Q4, Qwen3-4B-Q8, Qwen3-4B-Thinking-2507-Q4, Qwen3-4B-Thinking-2507-Q8, Qwen3-8B-Q4, Qwen3-8B-Q8, Qwen3.5-2B-Base-Q4, Qwen3.5-2B-Base-Q8, Qwen3.5-2B-Q4, Qwen3.5-2B-Q8, Qwen3.5-35B-A3B-Q4, Qwen3.6-27B-Q4, Qwen3.6-35B-A3B-Q4, Qwen3.8-27B-Q4, Qwen3.8-27B-Q4-mtp	Qwen3.8-27B
Backend	Metal	BLAS + Metal
Conditioning	warmup_only	runtime_native_warmup
Decode workload	TG128, TG128 @ 1 ctx	TG128
Harness schema	basert-benchmark-harness/1	computearena-measurements/1

Like-for-like pairs

Each row is a configuration present on both sides. Values are per-cell medians; ratios read BaseRT relative to llama.cpp.

Configuration	Decode A	Decode B	Ratio	Prefill A	Prefill B	Ratio	Runs A / B
Qwen3.8-27B Apple M4 Max	31.6	22.9	1.38×	222	243	0.91×	1 / 1

Side A · BaseRT

Top 25 of 45 runs by decode throughput.

Model / format	Chip / backend	Decode	Prefill	Contributor	Date	Report
Qwen3-0.6B-Q4 BaseRT 0.2.6Q4	Apple M4 Max Metal	583.3 TG128 @ 1 ctx	9,998 PP512	basecompute	19 Sep 2026	View Benchmark
Qwen3-0.6B-Q8 BaseRT 0.2.6Q8	Apple M4 Max Metal	455.8 TG128 @ 1 ctx	9,749 PP512	basecompute	19 Sep 2026	View Benchmark
Llama-3.2-1B-Instruct-Q4 BaseRT 0.2.6Q4	Apple M4 Max Metal	371.7 TG128 @ 1 ctx	5,963 PP512	basecompute	19 Sep 2026	View Benchmark
gemma-3-1b-it-Q4 BaseRT 0.2.6Q4	Apple M4 Max Metal	361.2 TG128 @ 1 ctx	7,448 PP512	basecompute	19 Sep 2026	View Benchmark
Llama-3.2-1B-Instruct-Q8 BaseRT 0.2.6Q4	Apple M4 Max Metal	320.5 TG128 @ 1 ctx	6,058 PP512	basecompute	19 Sep 2026	View Benchmark
Qwen3-1.7B-Q4 BaseRT 0.2.6Q4	Apple M4 Max Metal	318.4 TG128 @ 1 ctx	4,146 PP512	basecompute	19 Sep 2026	View Benchmark
Qwen3.5-2B-Q4 BaseRT 0.2.6Q4	Apple M4 Max Metal	301.7 TG128 @ 1 ctx	1,718 PP512	basecompute	19 Sep 2026	View Benchmark
Qwen3.5-2B-Base-Q4 BaseRT 0.2.6Q4	Apple M4 Max Metal	295.1 TG128 @ 1 ctx	1,713 PP512	basecompute	19 Sep 2026	View Benchmark
gemma-3-1b-it-Q8 BaseRT 0.2.6Q8	Apple M4 Max Metal	281.0 TG128 @ 1 ctx	7,527 PP512	basecompute	19 Sep 2026	View Benchmark
Qwen3-1.7B-Q8 BaseRT 0.2.6Q8	Apple M4 Max Metal	224.1 TG128 @ 1 ctx	4,025 PP512	basecompute	19 Sep 2026	View Benchmark
Qwen3.5-2B-Base-Q8 BaseRT 0.2.6Q8	Apple M4 Max Metal	208.0 TG128 @ 1 ctx	1,727 PP512	basecompute	19 Sep 2026	View Benchmark
Qwen3.5-2B-Q8 BaseRT 0.2.6Q8	Apple M4 Max Metal	205.4 TG128 @ 1 ctx	1,721 PP512	basecompute	19 Sep 2026	View Benchmark
Llama-3.2-3B-Instruct-Q4 BaseRT 0.2.6Q4	Apple M4 Max Metal	178.8 TG128 @ 1 ctx	2,202 PP512	basecompute	19 Sep 2026	View Benchmark
gemma-4-E2B-it-Q4 BaseRT 0.2.6Q4	Apple M4 Max Metal	175.3 TG128 @ 1 ctx	7,670 PP512	basecompute	19 Sep 2026	View Benchmark
NVIDIA-Nemotron-3-Nano-30B-A3B-Q4 BaseRT 0.2.6Q4	Apple M4 Max Metal	169.1 TG128 @ 1 ctx	1,632 PP512	basecompute	20 Sep 2026	View Benchmark
gpt-oss-20b-Q4 BaseRT 0.2.6Q4	Apple M4 Max Metal	160.5 TG128 @ 1 ctx	1,709 PP512	basecompute	20 Sep 2026	View Benchmark
Qwen3-4B-Q4 BaseRT 0.2.6Q4	Apple M4 Max Metal	157.5 TG128 @ 1 ctx	1,725 PP512	basecompute	19 Sep 2026	View Benchmark
gpt-oss-20b-Q8 BaseRT 0.2.6Q8	Apple M4 Max Metal	146.9 TG128 @ 1 ctx	1,700 PP512	basecompute	20 Sep 2026	View Benchmark
gpt-oss-20b-MXFP4 BaseRT 0.2.6mxfp4	Apple M4 Max Metal	146.7 TG128 @ 1 ctx	1,715 PP512	basecompute	20 Sep 2026	View Benchmark
Qwen3-30B-A3B-Thinking-2507-Q4 BaseRT 0.2.6Q4	Apple M4 Max Metal	140.1 TG128 @ 1 ctx	1,797 PP512	basecompute	20 Sep 2026	View Benchmark
Qwen3.5-35B-A3B-Q4 BaseRT 0.2.6Q4	Apple M4 Max Metal	140.1 TG128 @ 1 ctx	895 PP512	basecompute	20 Sep 2026	View Benchmark
Qwen3-30B-A3B-Instruct-2507-Q4 BaseRT 0.2.6Q4	Apple M4 Max Metal	139.8 TG128 @ 1 ctx	1,789 PP512	basecompute	20 Sep 2026	View Benchmark
Qwen3.6-35B-A3B-Q4 BaseRT 0.2.6Q4	Apple M4 Max Metal	139.7 TG128 @ 1 ctx	886 PP512	basecompute	20 Sep 2026	View Benchmark
Llama-3.2-3B-Instruct-Q8 BaseRT 0.2.6Q4	Apple M4 Max Metal	133.1 TG128 @ 1 ctx	2,218 PP512	basecompute	19 Sep 2026	View Benchmark
gemma-4-E2B-it-Q8 BaseRT 0.2.6Q8	Apple M4 Max Metal	132.2 TG128 @ 1 ctx	7,448 PP512	basecompute	19 Sep 2026	View Benchmark

Side B · llama.cpp

Top 1 of 1 run by decode throughput.

Model / format	Chip / backend	Decode	Prefill	Contributor	Date	Report
Qwen3.8-27B llama.cpp b10809 (5266f24da)Q4_K_M	Apple M4 Max BLAS + Metal	22.9 TG128	243 PP512	fabian	11 Sep 2026	View Benchmark