· ~8 min read
Serving Frontier MoE Models at Scale: FP8 KV Caches, INT4 Weights, and KV Cache Integrity
How to squeeze frontier MoE models like Kimi K2.6 and GLM 5.2 onto GPU memory: quantised KV caches double context, INT4 weights cut checkpoint size 40%, and page-level integrity checks protect shared caches. With real SGLang configs and benchmarks.
sglangfp8int4kv-cache+6 more