Summary
When serving a GSQ INT3 checkpoint (e.g. ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ) via vLLM on an SM80 (Ampere) GPU — I tested on an NVIDIA CMP 170HX (64GB, ~1591 GB/s measured HBM bandwidth) — single-request decode throughput is noticeably lower than expected for the hardware's memory bandwidth (~60 tok/s), while the same GPU serving the equivalent FP8 checkpoint + speculative decoding reaches 120+ tok/s with >90% HBM bandwidth utilization.
This isn't a GSQ quantization-quality issue — the model output quality is excellent. It traces to the Humming GEMM kernel (the backend vLLM uses for low-bit weight-only inference), which currently gets only ~34-43% effective HBM bandwidth at M=1 (single-token decode) on SM80, confirmed both by my own kernel-level profiling and by inclusionAI's own published A800 benchmark data (humming_w3a16_int3_float16.json in their repo shows the same ~34% figure on A800).
I filed the detailed technical report upstream: inclusionAI/humming#72
Why I'm posting here too
Anyone deploying GSQ INT3/2bit checkpoints on an Ampere-class card (A100, A800, RTX 30-series, CMP 170HX, etc.) via vLLM will likely hit the same decode-speed ceiling regardless of GSQ calibration quality. Newer architectures (SM90/SM120/SM121) have received dedicated small-batch kernel tuning in Humming recently; SM80 hasn't (as of humming-kernels==0.1.10, the SM80 heuristics file's most recent change is an interface-only signature update).
Posting this here mainly so future SM80 users searching for "GSQ slow decode" or "vLLM GSQ low tok/s" can find the root cause and the upstream issue, rather than assuming it's a quantization problem with the GSQ checkpoint itself. Happy to share my full benchmark script/data if useful for calibration-vs-kernel disambiguation on your side.
Summary
When serving a GSQ INT3 checkpoint (e.g.
ISTA-DASLab/Qwen3.8-27B-3Bit-GSQ) via vLLM on an SM80 (Ampere) GPU — I tested on an NVIDIA CMP 170HX (64GB, ~1591 GB/s measured HBM bandwidth) — single-request decode throughput is noticeably lower than expected for the hardware's memory bandwidth (~60 tok/s), while the same GPU serving the equivalent FP8 checkpoint + speculative decoding reaches 120+ tok/s with >90% HBM bandwidth utilization.This isn't a GSQ quantization-quality issue — the model output quality is excellent. It traces to the Humming GEMM kernel (the backend vLLM uses for low-bit weight-only inference), which currently gets only ~34-43% effective HBM bandwidth at M=1 (single-token decode) on SM80, confirmed both by my own kernel-level profiling and by inclusionAI's own published A800 benchmark data (
humming_w3a16_int3_float16.jsonin their repo shows the same ~34% figure on A800).I filed the detailed technical report upstream: inclusionAI/humming#72
Why I'm posting here too
Anyone deploying GSQ INT3/2bit checkpoints on an Ampere-class card (A100, A800, RTX 30-series, CMP 170HX, etc.) via vLLM will likely hit the same decode-speed ceiling regardless of GSQ calibration quality. Newer architectures (SM90/SM120/SM121) have received dedicated small-batch kernel tuning in Humming recently; SM80 hasn't (as of
humming-kernels==0.1.10, the SM80 heuristics file's most recent change is an interface-only signature update).Posting this here mainly so future SM80 users searching for "GSQ slow decode" or "vLLM GSQ low tok/s" can find the root cause and the upstream issue, rather than assuming it's a quantization problem with the GSQ checkpoint itself. Happy to share my full benchmark script/data if useful for calibration-vs-kernel disambiguation on your side.