TL;DR — We quantized Z.ai's GLM-5.3-Flash (multimodal MoE, 1M-token context) to 4-bit weights while keeping its BF16 MTP draft head and vision tower. The result: 177.7 GiB vs ≈599 GiB BF16 (−70%), full 1M-token context served on 2× DGX Spark (where the NVFP4 comparison checkpoint OOM'd 9/9 boots at any context), beats NVFP4 at every tested concurrency on 4× H100, and scores 85.0% on AIME 2026. It's public, MIT: canada-quant/GLM-5.3-Flash-W4A16-MTP.
Why
GLM-5.3-Flash is one of the most capable open-weight multimodal models you can run today: vision + text, a 1M-token context window, MTP speculative decoding, and an MIT license. But in BF16 it needs eight 80 GB GPUs per replica. That footprint is the thing standing between "great open model" and "great open model we can actually afford to serve."
The goal of this project was narrow and practical: shrink the serving footprint without giving up quality, and without giving up speed. Not a new model — every bit of capability here comes from the base GLM-5.3-Flash. Just a cheaper way to run it.
What we did
The artifact is a W4A16 quantization: only the 36,288 routed-expert GEMMs are INT4 (symmetric, group-128, GPTQ). Attention, router, shared experts, embeddings, the BF16 vision tower, and the BF16 MTP draft head ride unquantized, with canonical FP32 parameters preserved verbatim. That drops the checkpoint from ≈599 GiB to 177.7 GiB — ≈3.4× compression. Quantization ran on our 8× B300 node. vLLM selects the INT4×FP16 Marlin kernel at load, so it's ready to serve as-is.
The piece we were careful to keep is the MTP (multi-token-prediction) layer — preserved at BF16 post-quantization. This is not optional decoration: MTP is GLM-5.3-Flash's built-in draft head for speculative decoding, worth +40–50% single-stream. It proposes future tokens, the full model verifies them in one pass, and accepted tokens are exactly what the model would have produced anyway. It's a lossless speedup — it changes latency, not answers. On SM120 hardware, MTP adds +69.9% at c1, +37.6% at c8, +47.8% at c32 — keep it on everywhere.
Hardware matrix
This checkpoint serves on four hardware configurations. One row is fit-estimated (labeled); the rest are bench-validated.
| Config | GPUs | Context | Notes |
|---|---|---|---|
| H100/H200 (SM90), TP=4 | 4× 80 GB | 262K, bf16 KV | Bench-validated recipe |
| H200 (SM90), TP=2 | 2× 141 GB | ≈64–128K, fp8 KV | Fit-estimated (not rig-measured) |
| RTX PRO 6000 (SM120), TP=4 | 4× 96 GB | 262K, fp8 KV | Bench-validated |
| DGX Spark GB10 (SM121), TP=2 | 2× 128 GB UMA | 1M, fp8 KV | DFlash2 block-diffusion spec-decode stack |
Throughput: head-to-head vs NVFP4
All numbers are output tok/s, thinking ON, identical flags/prompts/hardware per row. The primary comparison is against the strongest community NVFP4 quant (LibertAIDAI/GLM-5.3-Flash-NVFP4).
2× DGX Spark GB10 (SM121), TP=2, DFlash2
8K prompts, 256-token completions, aggregate tok/s:
| Config | seq1 | seq2 | seq4 | seq6 |
|---|---|---|---|---|
| W4A16+DFlash2 @262K (this) | 33.29 | 42.01 | 80.98 | 121.81 |
| NVFP4-MTP4 reference | 21.8 | 41 | 66 | 60.5 |
| Δ | +53% | +2.4% | +23% | +101% |
| W4A16+DFlash2 @1M serve | 32.99 | 35.61 | 59.61 | 67.05 |
A 256K-token single prompt on the 1M serve decodes at 30.85 tok/s (TTFT 5.8 s). The comparison is also structural: the NVFP4 checkpoint (90.67 GiB/node weights vs ≈30 for W4A16) OOM'd all 9 boot attempts across 1M/300K/262K on the 121 GiB unified-memory nodes (dmesg-verified) — it cannot serve at any usable context on 2× Spark.
4× H100/H200 (SM90), TP=4, MTP both sides, isl/osl 8192/1024
| Concurrency | W4A16 MTP N=2 (this) | NVFP4 MTP N=2 | NVFP4 MTP N=5 |
|---|---|---|---|
| c=1 | 183.89 | 180.96 | 175.44 |
| c=8 | 249.94 | 251.14 | 220.05 |
| c=32 † | 1161.13 | 770.66 | 1018.13 |
† c=32 with --gpu-memory-utilization 0.92 (KV-pool capacity): 1161.13 tok/s. The earlier banked 689.46 was a confounded, capacity-throttled outlier — the c=32 lever is KV-pool size, not a kernel gap.
Fixed-drafter 2×2 re-confirm (2026-09-03), both arms on the corrected vLLM #54374-port drafter overlay: matched fixed-stack c1/c8 are within noise (W4A16 180.22/254.39 vs NVFP4 182.13/252.89, −1.05%/+0.59%) and the saturated c32 win holds (+18.1%, 804.02 vs 680.99). MTP-at-batch isolation: MTP adds +41.1%/+14.3%/−4.0% (W4A16) at c1/c8/c32; with MTP off on both, W4A16 leads +2.7%/+10.5%/+22.4% — the c32 advantage is the marlin INT4 MoE GEMM at batch, not an MTP artifact.
4× RTX PRO 6000 (SM120, Blackwell), TP=4, matched MTP N=2
| Concurrency | W4A16 (this) | NVFP4 | Δ |
|---|---|---|---|
| c=1 | 109.90 | 109.34 | +0.5% |
| c=8 | 318.54 | 319.46 | −0.3% |
| c=32 | 534.37 | 530.75 | +0.7% |
Parity within noise. MTP-at-batch on SM120 (2026-09-03): the drafter pays at every concurrency — +69.9% c1 (106.97 vs 62.96), +37.6% c8 (320.51 vs 232.89), +47.8% c32 (539.63 vs 365.02) — a sign flip vs SM90, where the drafter costs ≈4% at saturated c32. Keep MTP on at every concurrency on SM120. KV-dtype is only a c1 ≈5% lever (fp8 vs bf16 ≈ free at c1/c8, ≈4.7% cost at c32).
Throughput vs EXL3
A second community baseline — MiaAI-Lab's EXL3 2×-DGX-Spark recipe — was brought up to full 1M-context serving on the same 2× DGX Spark hardware on 2026-09-03 (KV pool 1,200,348 tokens = 1.20× at 1M). The matched-protocol head-to-head is FINAL (decode ladders, vendor-protocol rows, prefill ladder, and a clean c32 refire all banked and sha256-verified).
At like-for-like decode rate (excluding TTFT, 8K prompts, 256-token completions): this checkpoint leads single-stream (33.0 vs 29.9 agg tok/s, +10.4% c1) and the long-prefill regime by +31–39% (≈1.2K tok/s vs 864–914 @300K-class), with the bigger KV pool (1,360,420 = 1.30× at 1M vs EXL3's 1.20×). EXL3 leads at concurrency (c2 +67%, c4 +89%, c6 ≈+81% after correcting for its max-num-seqs=4 cap; clean c32 refire 881–898 aggregate tok/s). The EXL3 vendor README overstated measured decode ≈2.2×. The matched AIME 2026 quality arm is NOT RUN (owner pause, 2026-09-03) — the quality verdict is open.
Quality
AIME 2026 (30 problems, max thinking, 131,072-token budget, DFlash2 stack on 2× Spark): 102/120 = 85.0% — the n=120 campaign is final. Four completed epochs: 28/30, 25/30, 26/30, 23/30. Miss decomposition: 16/18 are budget-exhaustion truncations at the 131,072-token cap, 2/18 genuine wrong answers.
GSM8K: 0.97 — parity with the FP8 baseline (quant-insensitive). GPQA-Diamond n=198 @131k: within noise of NVFP4 on both SM90 and SM120.
AIME 2025 — closed on H100, resolved on RTX PRO 6000
4 epochs × 30, max thinking, 131,072 budget. On H100 (SM90): parity within noise — W4A16 0.8833 (106/120) vs NVFP4 0.9000, 0.42σ. The earlier depressed SM90 read (0.7833) was caused by a serving-stack drafter corruption bug (upstream vLLM issue #54374: wrong FlashAttention scheduling path for the windowed drafter wrote out of bounds and poisoned the target model), since fixed.
On RTX PRO 6000 (SM120): a real ≈2σ raw deficit (0.8083 vs 0.9000) was mechanism-decomposed to ≈63% budget-wall mechanical losses + ≈37% SM120 marlin kernel numerics. The fix stack (zero-cost answer-commit hook + thinking-budget force-commit) is VALIDATED: deficit closed within noise (0.8833 two-way / 0.8917 hook-credited vs NVFP4 0.9000).
Long-context decode at batch
SM120, 524K ctx, 2026-09-04 legs. Single-stream decode is KV-length-insensitive to at least 486K (143.94 → 177.15 → 357.34 tok/s at 138.6K/277K/486K on bf16 KV). The at-batch per-stream collapse saturates with KV length (≈2.2–3.7 tok/s/stream at c4–c32 from 138.6K through 486K). MTP acceptance holds at extreme KV length and concurrency (60% c1 @486K; 85.7% c1 → 73.9% c8 → 71.6% c32 (n=4,714) @138.6K) — the collapse is target-model forward compute at batch × long-KV, NOT drafter acceptance. Long prefills serialize at batch (≈6.1–7.6K tok/s aggregate, essentially flat 138.6K→486K) — admission pacing/prefill scheduling is the lever for long-context batch UX.
Limitations (honestly)
- SM120 kernel-numerics residual: the AIME 2025 RTX PRO 6000 deficit was dominated by budget-wall mechanical losses (now fixed via the commit-hook stack), but ≈37% traces to SM120 marlin W4A16 kernel numerics producing heavier-tailed thinking chains. No upstream alternative SM120 W4A16 MoE kernel path exists yet — a custom loop-salvage arm is the open fix-design work.
- EXL3 concurrency tradeoff: the EXL3 recipe leads at concurrency (c2 +67%, c4 +89%) on 2× Spark. This checkpoint leads single-stream (+10.4% c1), long-prefill (+31–39%), and KV capacity (1.30× vs 1.20× at 1M). Different serving stacks, different strengths.
- EXL3 quality arm not yet run: the matched AIME 2026 quality comparison against EXL3 is paused (owner-side, 2026-09-03). The quality verdict is open — it resumes on the owner's word.
- Fit-estimated row: the 2× H200 TP=2 configuration is a fit estimate (not rig-measured). We label it as such rather than overclaiming.
What's next
- EXL3 quality arm: queued on the deployment lane — identical temp-0.0 / 131,072-token / n=120 config, awaiting the owner's green light.
- 1M-budget AIME 2026 rescue: the 16 budget-exhaustion truncations are queued for a 1M-budget re-run now that the EXL3 endpoint is up at full context.
- SM120 kernel numerics: the flip-rate at near-tie positions is the target metric for the kernel-numerics work. A custom stagnation-based loop-salvage/answer-commit arm is the open design.