Compare

Loading community benchmarks…

Community benchmarks

Compare

Put two chips, runtimes, or runtime versions side by side. Everything else is held fixed or called out, so a ratio is worth exactly what the matched runs say.

Add your benchmark

Runs compared: 13
Like-for-like pair: 1
Runtimes available: 2

Compare

Side A

Side B

Model

Chip

Quantisation A

Quantisation B

Backend

Prefill size

Decode size

Cooldown

12 runs vs 1 run · 1 like-for-like pair

Held fixedModel: Qwen3 4BPrefill: PP512Decode: TG128

llama.cpp has only 1 matching run. Treat its figures as indicative.
The sides differ in cooldown, conditioning, decode workload, harness schema. Ratios may reflect benchmark protocol rather than the runtime.

Runtime comparison

BaseRT vs llama.cpp

Ratios read side A relative to side B. Geometric mean over 1 like-for-like pair.

Decode: 1.11×+11.0%
Prefill: 1.25×+25.1%

Side A

BaseRT

12 runs

2 contributors

Latest 20 Sep 2026

Decode · TG128: 102.9tok/s median
best 165.7 · worst 56.3
Prefill · PP512: 2,083tok/s median
best 3,412 · worst 726

Models: Qwen3-4B, Qwen3-4B-Q4, Qwen3-4B-Q8
Chips: Apple M1 Max, Apple M3 Ultra, Apple M4 Max, Apple M4 Pro, Apple M5 Pro, NVIDIA GB10
Versions: BaseRT 0.2.4, BaseRT 0.2.6
Quantisations: Q4, Q8
Backends: CUDA, Metal
Cooldown: off, on
Conditioning: idle_reset_then_warm, warmup_only
Decode workload: TG128, TG128 @ 1 ctx
Harness schema: basert-benchmark-harness/1

Side B

llama.cpp

1 run

1 contributor

Latest 10 Sep 2026

Decode · TG128: 93.6tok/s median
best 93.6 · worst 93.6
Prefill · PP512: 2,705tok/s median
best 2,705 · worst 2,705

Models: Qwen3-4B
Chips: Apple M5 Pro
Versions: llama.cpp b9960 (a935fbffe)
Quantisations: Q4_0
Backends: BLAS + Metal
Cooldown: off
Conditioning: runtime_native_warmup
Decode workload: TG128
Harness schema: computearena-measurements/1

What differs besides the runtime

Facet	BaseRT	llama.cpp
Model	Qwen3-4B, Qwen3-4B-Q4, Qwen3-4B-Q8	Qwen3-4B
Chip	Apple M1 Max, Apple M3 Ultra, Apple M4 Max, Apple M4 Pro, Apple M5 Pro, NVIDIA GB10	Apple M5 Pro
Backend	CUDA, Metal	BLAS + Metal
Cooldown	off, on	off
Conditioning	idle_reset_then_warm, warmup_only	runtime_native_warmup
Decode workload	TG128, TG128 @ 1 ctx	TG128
Harness schema	basert-benchmark-harness/1	computearena-measurements/1

Like-for-like pairs

Each row is a configuration present on both sides. Values are per-cell medians; ratios read BaseRT relative to llama.cpp.

Configuration	Decode A	Decode B	Ratio	Prefill A	Prefill B	Ratio	Runs A / B
Qwen3-4B Apple M5 Pro	103.9	93.6	1.11×	3,384	2,705	1.25×	3 / 1

Side A · BaseRT

Top 12 of 12 runs by decode throughput.

Model / format	Chip / backend	Decode	Prefill	Contributor	Date	Report
Qwen3-4B-Q4 BaseRT 0.2.6Q4	Apple M3 Ultra Metal	165.7 TG128 @ 1 ctx	2,461 PP512	basecompute	20 Sep 2026	View Benchmark
Qwen3-4B-Q4 BaseRT 0.2.6Q4	Apple M4 Max Metal	157.5 TG128 @ 1 ctx	1,725 PP512	basecompute	19 Sep 2026	View Benchmark
Qwen3-4B-Q8 BaseRT 0.2.6Q8	Apple M3 Ultra Metal	116.8 TG128 @ 1 ctx	2,441 PP512	basecompute	20 Sep 2026	View Benchmark
basecompute/Qwen3-4B BaseRT 0.2.4Q4	Apple M5 Pro Metal	107.3 TG128	3,384 PP512	arki05	10 Sep 2026	View Benchmark
Qwen3-4B-Q8 BaseRT 0.2.6Q8	Apple M4 Max Metal	105.4 TG128 @ 1 ctx	1,712 PP512	basecompute	19 Sep 2026	View Benchmark
basecompute/Qwen3-4B BaseRT 0.2.4Q4	Apple M5 Pro Metal	103.9 TG128	3,372 PP512	arki05	10 Sep 2026	View Benchmark
Qwen3-4B-Q4 BaseRT 0.2.6Q4	Apple M1 Max Metal	101.9 TG128 @ 1 ctx	927 PP512	basecompute	20 Sep 2026	View Benchmark
Qwen3-4B-Q4 BaseRT 0.2.6Q4	NVIDIA GB10 CUDA	91.3 TG128 @ 1 ctx	2,822 PP512	basecompute	19 Sep 2026	View Benchmark
Qwen3-4B-Q4 BaseRT 0.2.6Q4	Apple M4 Pro Metal	90.3 TG128 @ 1 ctx	734 PP512	basecompute	20 Sep 2026	View Benchmark
Qwen3-4B-Q8 BaseRT 0.2.6Q8	Apple M1 Max Metal	71.3 TG128 @ 1 ctx	920 PP512	basecompute	20 Sep 2026	View Benchmark
basecompute/Qwen3-4B BaseRT 0.2.4Q8	Apple M5 Pro Metal	64.7 TG128	3,412 PP512	arki05	11 Sep 2026	View Benchmark
Qwen3-4B-Q8 BaseRT 0.2.6Q8	Apple M4 Pro Metal	56.3 TG128 @ 1 ctx	726 PP512	basecompute	20 Sep 2026	View Benchmark

Side B · llama.cpp

Top 1 of 1 run by decode throughput.

Model / format	Chip / backend	Decode	Prefill	Contributor	Date	Report
Qwen3-4B llama.cpp b9960 (a935fbffe)Q4_0	Apple M5 Pro BLAS + Metal	93.6 TG128	2,705 PP512	isu	10 Sep 2026	View Benchmark