Hy3 at 4 bits with its MTP head intact: a W4A16 quantization

TL;DR — We quantized Tencent Hy3 (295B-parameter MoE, 21B active) to 4-bit weights while keeping its multi-token-prediction (MTP) draft layer in BF16. The result holds FP8-level quality at 57% of the FP8 footprint (≈598 GB BF16 → 172 GB), preserves MTP speculative decoding (worth +40% single-stream vs the same-scheme quant that shipped without it), and takes the high-concurrency throughput crown among 4-bit quants. It's public, Apache-2.0: canada-quant/hy3-w4a16-mtp.

Why

Hy3 is one of the strongest open-weight MoE models you can run today, but it's big: 295B parameters — ≈598 GB of weights in BF16, and still ≈300 GB in Tencent's official FP8 release. That footprint is the thing standing between "great open model" and "great open model we can actually afford to serve."

The goal of this project was narrow and practical: shrink the serving footprint without giving up quality, and without giving up speed. Not a new model — every bit of capability here comes from the base Hy3 (80 layers: 79 MoE + 1 dense, 192 experts per MoE layer with top-8 routing, GQA attention). Just a cheaper way to run it.

What we did

The artifact is a W4A16 quantization: the routable MoE expert weights — 45,504 matrices, ≈97% of parameters — compressed to INT4 (group-size 128, symmetric, GPTQ via llm-compressor), with attention, dense layer 0, the shared expert, the router, embeddings, the LM head, and the NextN/MTP head all preserved in BF16. Calibration ran on an agentic + code + math + chat blend (512 samples × 2048 tokens). That drops the checkpoint from ≈598 GB to ≈172 GB — ≈3.4× compression, and 57% of the ≈300 GB FP8 release's footprint. vLLM selects the INT4×FP16 Marlin/Machete kernel at load, so it's ready to serve as-is.

The piece we were careful to keep is the MTP (multi-token-prediction) layer — model.layers.80, including its own 192 experts — re-injected at BF16 post-quantization. This is not optional decoration: transformers silently drops the NextN layer otherwise, which is exactly how Tencent's own AngelSlim GPTQ artifact shipped MTP-less — it cannot speculate at all. MTP is Hy3's built-in draft head for speculative decoding: it proposes future tokens, the full model verifies them in one pass, and accepted tokens are exactly what the model would have produced anyway. It's a lossless speedup — it changes latency, not answers. Preserved here, MTP speculates at 62.1% aggregate greedy acceptance (76.8% at position 0) ≈ 2.24 effective tokens per decode step with num_speculative_tokens=2.

Does it hold up?

We measured quality against the official tencent/Hy3-FP8 release on the same 8×H100 box (lm-eval 0.4.12 through vLLM OpenAI endpoints, greedy, spec-decode off, non-thinking):

Every task within ≈0.7 pp of FP8 — inside eval noise.

We also ran the reasoning-mode field (thinking=high, 0-shot, native 256K context, a uniform 131,072-token generation budget for every model) against the other public 4-bit Hy3 quants:

Every quant is statistically indistinguishable from FP8 and from each other (AIME n=30 is ±7–9 pp per cell). 4-bit expert quantization, done carefully, is quality-neutral — so the differentiators between these artifacts are serving traits: footprint, MTP, and throughput.

How fast is it?

All throughput numbers are output tokens/s on the same box, harness, and settings: 8×H100-SXM, vLLM 0.25.1.

• Single-stream (best spec config): 144 tok/s with MTP k=2 (TPOT 6.7 ms) — ties FP8 (143). The AngelSlim GPTQ artifact is capped at 103 because it has no MTP head — that's +40% for ours from the same quantization scheme. Credit where due: cyankiwi's AWQ posts 152 tok/s single-stream.
• Concurrency 8: k=1 tops the whole 12-config matrix at 822 tok/s.
• Concurrency 32: 2,146 tok/s — +21% over FP8 (1,854), ahead of AngelSlim (2,135) and AWQ (2,028). The high-concurrency throughput crown.
• k=3+ regresses; we publish the honest k=1/2/3/5 numbers in the model card rather than cherry-picking.

The story is clear and, we think, useful: quality is a wash across the 4-bit field, so you choose on serving traits. We give AWQ the single-stream edge and AngelSlim 6 GB of disk; we take MTP speculative decoding and the concurrency crown, at 57% of the FP8 footprint.

Running it

• Stock vLLM ≥ 0.25 — no custom patches or forks (hy_v3 support landed in v0.20.0).
• --enable-expert-parallel at TP=8 (the g128 constraint). Runs on 8×H100-80GB with headroom, or 4×H100 at TP=4; reference hardware is 8×H200 SXM5 or 8×RTX PRO 6000.
• Native 256K context, validated at max_model_len 262144.
• One-shot script and Docker image in the companion repo (scripts/quickstart_hopper.sh, docker/, docs/deploy-hopper.md); the full methodology and audit trail live in its docs/RUNLOG.md.

Limitations (honestly)

• AWQ edges us single-stream (152 vs 144 tok/s); our single-stream win over the same-scheme GPTQ artifact comes entirely from MTP, which it lacks.
• k=3+ speculative configs regress — k=1/k=2 are the sweet spots, and we publish the full matrix.
• AngelSlim's MTP-less GPTQ artifact is 6 GB smaller on disk.
• The asymmetric INT4 weights require --enable-expert-parallel to serve correctly at TP=8.