Compare

Loading community benchmarks…

Community benchmarks

Compare

Put two chips, runtimes, or runtime versions side by side. Everything else is held fixed or called out, so a ratio is worth exactly what the matched runs say.

Add your benchmark

Runs compared: 129
Like-for-like pairs: 12
Runtimes available: 2

Compare

Side A

Side B

Model

Chip

Quantisation A

Quantisation B

Backend

Prefill size

Decode size

Cooldown

80 runs vs 49 runs · 12 like-for-like pairs

Held fixedChip: Apple M5 ProPrefill: PP512Decode: TG128

These figures mix different models. Pick a model for a like-for-like comparison.
The sides differ in cooldown, conditioning, decode workload, harness schema. Ratios may reflect benchmark protocol rather than the runtime.

Runtime comparison

BaseRT vs llama.cpp

Ratios read side A relative to side B. Geometric mean over 12 like-for-like pairs.

Decode: 1.09×+9.1%
Prefill: 1.23×+23.4%

Side A

BaseRT

80 runs

2 contributors

Latest 18 Sep 2026

Decode · TG128: 89.9tok/s median
best 2,068.0 · worst 10.6
Prefill · PP512: 3,318tok/s median
best 190,107 · worst 327

Models: gemma-3-1b-it, gemma-4-26B-A4B-it, gemma-4-E2B-it, gemma-4-E4B-it, gpt-oss-20b, Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct, Mistral-7B-Instruct-v0.3, Muse-Glimmer-30B, NVIDIA-Nemotron-3-Nano-30B-A3B, Qwen3-0.6B, Qwen3-1.7B, Qwen3-30B-A3B-Instruct-2507, Qwen3-30B-A3B-Thinking-2507, Qwen3-4B, Qwen3-4B-Instruct-2507, Qwen3-4B-Thinking-2507, Qwen3-8B, Qwen3-8B-base-q2, Qwen3-8B-base-q3, Qwen3-8B-base-q5, Qwen3-8B-base-q6, Qwen3.5-2B, Qwen3.5-2B-Base, Qwen3.5-35B-A3B, Qwen3.6-27B, Qwen3.6-35B-A3B, Qwen3.8-27B, tinystories-lay8-hs512-hd8-33M
Chips: Apple M5 Pro
Versions: BaseRT 0.2.4, BaseRT 0.2.5
Quantisations: mxfp4, passthrough_gguf, Q2, Q3, Q4, Q5, Q6, Q8
Backends: Metal
Cooldown: off, on
Conditioning: idle_reset_then_warm, warmup_only
Decode workload: TG128, TG128 @ 1 ctx
Harness schema: basert-benchmark-harness/1

Side B

llama.cpp

49 runs

2 contributors

Latest 17 Sep 2026

Decode · TG128: 61.5tok/s median
best 1,443.3 · worst 11.9
Prefill · PP512: 1,644tok/s median
best 124,181 · worst 373

Models: Gemma-4-E4B-It, Gpt-Oss-20B, Llama 3.2 3B Instruct, Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, Qwen3-0.6B, Qwen3-30B-A3B-Instruct-2507, Qwen3-4B, Qwen3-4B-Instruct-2507, Qwen3-8B, Qwen3.6-27B, Qwen3.6-35B-A3B, Tinystories Lay8 Hs512 Hd8 33M
Chips: Apple M5 Pro
Versions: llama.cpp b10809 (5266f24da), llama.cpp b9960 (a935fbffe)
Quantisations: BF16, F16, Q2_K, Q3_K_M, Q4_0, Q4_1, Q4_K_M, Q5_K_M, Q6_K, Q8_0
Backends: BLAS + Metal
Cooldown: off
Conditioning: runtime_native_warmup
Decode workload: TG128
Harness schema: computearena-measurements/1

What differs besides the runtime

Facet	BaseRT	llama.cpp
Model	gemma-3-1b-it, gemma-4-26B-A4B-it, gemma-4-E2B-it, gemma-4-E4B-it, gpt-oss-20b, Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct, Mistral-7B-Instruct-v0.3, Muse-Glimmer-30B, NVIDIA-Nemotron-3-Nano-30B-A3B, Qwen3-0.6B, Qwen3-1.7B, Qwen3-30B-A3B-Instruct-2507, Qwen3-30B-A3B-Thinking-2507, Qwen3-4B, Qwen3-4B-Instruct-2507, Qwen3-4B-Thinking-2507, Qwen3-8B, Qwen3-8B-base-q2, Qwen3-8B-base-q3, Qwen3-8B-base-q5, Qwen3-8B-base-q6, Qwen3.5-2B, Qwen3.5-2B-Base, Qwen3.5-35B-A3B, Qwen3.6-27B, Qwen3.6-35B-A3B, Qwen3.8-27B, tinystories-lay8-hs512-hd8-33M	Gemma-4-E4B-It, Gpt-Oss-20B, Llama 3.2 3B Instruct, Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, Qwen3-0.6B, Qwen3-30B-A3B-Instruct-2507, Qwen3-4B, Qwen3-4B-Instruct-2507, Qwen3-8B, Qwen3.6-27B, Qwen3.6-35B-A3B, Tinystories Lay8 Hs512 Hd8 33M
Backend	Metal	BLAS + Metal
Cooldown	off, on	off
Conditioning	idle_reset_then_warm, warmup_only	runtime_native_warmup
Decode workload	TG128, TG128 @ 1 ctx	TG128
Harness schema	basert-benchmark-harness/1	computearena-measurements/1

Like-for-like pairs

Each row is a configuration present on both sides. Values are per-cell medians; ratios read BaseRT relative to llama.cpp.

Configuration	Decode A	Decode B	Ratio	Prefill A	Prefill B	Ratio	Runs A / B
Qwen3-8B Apple M5 Pro	59.7	49.9	1.20×	1,755	1,416	1.24×	13 / 14
Qwen3-0.6B Apple M5 Pro	415.8	357.2	1.16×	20,451	14,530	1.41×	6 / 4
Llama-3.1-8B-Instruct Apple M5 Pro	48.8	55.5	0.88×	1,784	1,424	1.25×	4 / 5
Llama-3.2-3B-Instruct Apple M5 Pro	94.6	115.2	0.82×	4,297	3,318	1.30×	5 / 3
Qwen3-30B-A3B-Instruct-2507 Apple M5 Pro	98.1	95.1	1.03×	3,493	1,769	1.97×	2 / 6
gemma-4-E4B-it Apple M5 Pro	66.6	69.4	0.96×	4,563	2,263	2.02×	4 / 3
Qwen3-4B-Instruct-2507 Apple M5 Pro	76.9	92.6	0.83×	3,379	2,568	1.32×	4 / 3
gpt-oss-20b Apple M5 Pro	106.9	81.5	1.31×	1,110	1,851	0.60×	4 / 2
Qwen3.6-27B Apple M5 Pro	13.3	14.9	0.90×	360	375	0.96×	2 / 4
Qwen3-4B Apple M5 Pro	103.9	93.6	1.11×	3,384	2,705	1.25×	3 / 1
Qwen3.6-35B-A3B Apple M5 Pro	113.2	61.5	1.84×	1,275	1,673	0.76×	1 / 3
tinystories-lay8-hs512-hd8-33M Apple M5 Pro	2,068.0	1,443.3	1.43×	190,107	124,181	1.53×	1 / 1

Side A · BaseRT

Top 25 of 80 runs by decode throughput.

Model / format	Chip / backend	Decode	Prefill	Contributor	Date	Report
ivnle/tinystories-lay8-hs512-hd8-33M BaseRT 0.2.4Q4	Apple M5 Pro Metal	2,068.0 TG128	190,107 PP512	arki05	17 Sep 2026	View Benchmark
basecompute/Qwen3-0.6B BaseRT 0.2.4Q4	Apple M5 Pro Metal	516.8 TG128	20,778 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/Qwen3-0.6B BaseRT 0.2.4Q4	Apple M5 Pro Metal	512.0 TG128	20,603 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/Qwen3-0.6B BaseRT 0.2.5Q4	Apple M5 Pro Metal	481.5 TG128 @ 1 ctx	19,595 PP512	isu	18 Sep 2026	View Benchmark
basecompute/Qwen3-0.6B BaseRT 0.2.4Q8	Apple M5 Pro Metal	350.2 TG128	20,537 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/Qwen3-0.6B BaseRT 0.2.4Q8	Apple M5 Pro Metal	349.8 TG128	20,366 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/Qwen3-0.6B BaseRT 0.2.4Q8	Apple M5 Pro Metal	311.0 TG128	18,474 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/gemma-3-1b-it BaseRT 0.2.4Q4	Apple M5 Pro Metal	293.1 TG128	15,066 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/gemma-3-1b-it BaseRT 0.2.4Q4	Apple M5 Pro Metal	270.4 TG128	13,717 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/Qwen3-1.7B BaseRT 0.2.4Q4	Apple M5 Pro Metal	235.4 TG128	8,040 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/Llama-3.2-1B-Instruct BaseRT 0.2.4Q4	Apple M5 Pro Metal	229.1 TG128	11,673 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/Qwen3.5-2B-Base BaseRT 0.2.4Q4	Apple M5 Pro Metal	218.2 TG128	2,645 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/Qwen3.5-2B BaseRT 0.2.4Q4	Apple M5 Pro Metal	217.7 TG128	2,641 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/Qwen3-1.7B BaseRT 0.2.4Q4	Apple M5 Pro Metal	210.3 TG128	7,268 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/gemma-3-1b-it BaseRT 0.2.4Q8	Apple M5 Pro Metal	210.3 TG128	15,061 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/Llama-3.2-1B-Instruct BaseRT 0.2.4Q4	Apple M5 Pro Metal	206.3 TG128	10,528 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/Llama-3.2-1B-Instruct BaseRT 0.2.4Q4	Apple M5 Pro Metal	205.3 TG128	12,067 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/gemma-4-E2B-it BaseRT 0.2.4Q4	Apple M5 Pro Metal	144.6 TG128	13,240 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/Qwen3-1.7B BaseRT 0.2.4Q8	Apple M5 Pro Metal	139.7 TG128	8,043 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/Qwen3.5-2B-Base BaseRT 0.2.4Q8	Apple M5 Pro Metal	131.4 TG128	2,642 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/Qwen3.5-2B BaseRT 0.2.4Q8	Apple M5 Pro Metal	130.7 TG128	2,638 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/Qwen3-1.7B BaseRT 0.2.4Q8	Apple M5 Pro Metal	125.1 TG128	7,267 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/Qwen3.5-35B-A3B BaseRT 0.2.4Q4	Apple M5 Pro Metal	113.4 TG128	1,304 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/Qwen3.6-35B-A3B BaseRT 0.2.4Q4	Apple M5 Pro Metal	113.2 TG128	1,275 PP512	arki05	11 Sep 2026	View Benchmark
basecompute/gpt-oss-20b BaseRT 0.2.4Q4	Apple M5 Pro Metal	110.9 TG128	1,155 PP512	arki05	11 Sep 2026	View Benchmark

Side B · llama.cpp

Top 25 of 49 runs by decode throughput.

Model / format	Chip / backend	Decode	Prefill	Contributor	Date	Report
Tinystories Lay8 Hs512 Hd8 33M llama.cpp b10809 (5266f24da)Q4_0	Apple M5 Pro BLAS + Metal	1,443.3 TG128	124,181 PP512	arki05	17 Sep 2026	View Benchmark
Qwen3-0.6B llama.cpp b10809 (5266f24da)Q4_K_M	Apple M5 Pro BLAS + Metal	358.3 TG128	14,509 PP512	arki05	11 Sep 2026	View Benchmark
Qwen3-0.6B llama.cpp b10809 (5266f24da)Q4_K_M	Apple M5 Pro BLAS + Metal	358.1 TG128	14,344 PP512	arki05	11 Sep 2026	View Benchmark
Qwen3-0.6B llama.cpp b10809 (5266f24da)Q4_K_M	Apple M5 Pro BLAS + Metal	356.2 TG128	14,552 PP512	arki05	11 Sep 2026	View Benchmark
Qwen3-0.6B llama.cpp b10809 (5266f24da)Q8_0	Apple M5 Pro BLAS + Metal	283.0 TG128	14,942 PP512	arki05	11 Sep 2026	View Benchmark
Llama-3.2-3B-Instruct llama.cpp b10809 (5266f24da)Q4_K_M	Apple M5 Pro BLAS + Metal	117.7 TG128	3,318 PP512	arki05	11 Sep 2026	View Benchmark
Llama-3.2-3B-Instruct llama.cpp b10809 (5266f24da)Q4_K_M	Apple M5 Pro BLAS + Metal	115.2 TG128	3,315 PP512	arki05	11 Sep 2026	View Benchmark
Qwen3-30B-A3B-Instruct-2507 llama.cpp b10809 (5266f24da)Q2_K	Apple M5 Pro BLAS + Metal	102.7 TG128	1,893 PP512	arki05	11 Sep 2026	View Benchmark
Qwen3-30B-A3B-Instruct-2507 llama.cpp b10809 (5266f24da)Q3_K_M	Apple M5 Pro BLAS + Metal	97.3 TG128	1,708 PP512	arki05	11 Sep 2026	View Benchmark
Gpt-Oss-20B llama.cpp b10809 (5266f24da)Q4_K_M	Apple M5 Pro BLAS + Metal	96.3 TG128	1,849 PP512	arki05	11 Sep 2026	View Benchmark
Qwen3-30B-A3B-Instruct-2507 llama.cpp b10809 (5266f24da)Q4_K_M	Apple M5 Pro BLAS + Metal	95.2 TG128	1,845 PP512	arki05	11 Sep 2026	View Benchmark
Qwen3-30B-A3B-Instruct-2507 llama.cpp b10809 (5266f24da)Q4_K_M	Apple M5 Pro BLAS + Metal	95.0 TG128	1,831 PP512	arki05	11 Sep 2026	View Benchmark
Qwen3-4B-Instruct-2507 llama.cpp b10809 (5266f24da)Q4_K_M	Apple M5 Pro BLAS + Metal	94.4 TG128	2,568 PP512	arki05	11 Sep 2026	View Benchmark
Qwen3-4B llama.cpp b9960 (a935fbffe)Q4_0	Apple M5 Pro BLAS + Metal	93.6 TG128	2,705 PP512	isu	10 Sep 2026	View Benchmark
Qwen3-4B-Instruct-2507 llama.cpp b10809 (5266f24da)Q4_K_M	Apple M5 Pro BLAS + Metal	92.6 TG128	2,566 PP512	arki05	11 Sep 2026	View Benchmark
Qwen3-30B-A3B-Instruct-2507 llama.cpp b10809 (5266f24da)Q5_K_M	Apple M5 Pro BLAS + Metal	84.7 TG128	1,644 PP512	arki05	11 Sep 2026	View Benchmark
Qwen3-30B-A3B-Instruct-2507 llama.cpp b10809 (5266f24da)Q6_K	Apple M5 Pro BLAS + Metal	78.9 TG128	1,646 PP512	arki05	11 Sep 2026	View Benchmark
Llama 3.2 3B Instruct llama.cpp b10809 (5266f24da)Q8_0	Apple M5 Pro BLAS + Metal	76.6 TG128	3,516 PP512	arki05	11 Sep 2026	View Benchmark
Qwen3-8B llama.cpp b10809 (5266f24da)Q2_K	Apple M5 Pro BLAS + Metal	72.8 TG128	1,477 PP512	arki05	11 Sep 2026	View Benchmark
Gemma-4-E4B-It llama.cpp b10809 (5266f24da)Q4_K_M	Apple M5 Pro BLAS + Metal	72.1 TG128	2,263 PP512	arki05	11 Sep 2026	View Benchmark
Qwen3-8B llama.cpp b10809 (5266f24da)Q2_K	Apple M5 Pro BLAS + Metal	69.6 TG128	1,465 PP512	arki05	11 Sep 2026	View Benchmark
Gemma-4-E4B-It llama.cpp b10809 (5266f24da)Q4_K_M	Apple M5 Pro BLAS + Metal	69.4 TG128	2,235 PP512	arki05	11 Sep 2026	View Benchmark
Qwen3.6-35B-A3B llama.cpp b10809 (5266f24da)Q3_K_M	Apple M5 Pro BLAS + Metal	67.5 TG128	1,737 PP512	arki05	11 Sep 2026	View Benchmark
Gpt-Oss-20B llama.cpp b10809 (5266f24da)F16	Apple M5 Pro BLAS + Metal	66.6 TG128	1,852 PP512	arki05	11 Sep 2026	View Benchmark
Qwen3.6-35B-A3B llama.cpp b10809 (5266f24da)Q5_K_M	Apple M5 Pro BLAS + Metal	61.5 TG128	1,533 PP512	arki05	11 Sep 2026	View Benchmark