Serving Frontier MoE Models at Scale: FP8 KV Caches, INT4 Weights, and KV Cache Integrity

Frontier mixture-of-experts models are wonderful to serve and very hard to fit into GPU memory. In August 2026, Cloudflare published the operating details of how its Workers AI fleet serves two of the most demanding open models — Moonshot AI's Kimi K2.6 and Z.ai's GLM 5.2 — using three techniques that layer on top of each other: quantising the KV cache, compressing the model weights, and adding page-level integrity checks to protect the cache that hundreds of requests now share. The numbers are the interesting part. FP8 KV caches double the context a GPU can hold resident. INT4 weights shrink GLM 5.2's checkpoint from 705 GB to 421 GB. And the safety check that keeps concurrent requests from reading each other's memory costs under 1% on throughput.

All of Cloudflare's experiments run on SGLang, the open-source serving framework, with patches upstreamed as they go. That means every technique below is reproducible on hardware you control.

Why the KV Cache Is the Real Bottleneck

For a long-context MoE model, the KV cache — the attention keys and values the model stores for every processed token — grows faster than the weights and fills GPU memory first. By default it is stored in 16-bit (BF16). Cloudflare stores it in FP8 (e4m3), which halves its size. On Kimi K2.6 that raises the amount of context one GPU can hold from roughly 686,000 tokens to about 1.37 million.

The benefit is not raw speed — quantising adds work per token, because the FP8 attention kernel must convert values as it reads them. The benefit is residency: how many concurrent requests can stay in memory at once. From Cloudflare's measurements on a disaggregated H200 deployment, decoding Kimi K2.6:

Concurrent requests	BF16 KV cache (tok/s)	FP8 KV cache (tok/s)
1	137	125
8	731	689
16	1,106	1,028
32	1,558	1,489
64	Out of memory	2,192

At any single concurrency level BF16 is a few percent faster per token. But it runs out of cache at 32 requests and cannot admit a 33rd, while FP8 keeps going to 64 and reaches 2,192 tokens per second — about 41% higher than BF16's peak, for roughly 30% less cost per token. Aggregated throughput, not per-request latency, is where the win lives.

Cloudflare verified that FP8 does not change model answers across its evaluation suite:

Benchmark	BF16 KV	FP8 KV
GSM8K	94.24	94.09
ARC-Easy	89.06	89.14
ARC-Challenge	66.72	67.49
MMLU	89.11	89.04
MMLU-Pro	80.29	79.29
Tool-call validity	92.2%	92.6%

Enabling It in SGLang

SGLang exposes quantised KV caches directly. FP8 needs scaling factors for quantisation and dequantisation — currently only per-tensor (scalar) scaling factors are supported, loaded either from the checkpoint (k_scale and v_scale parameters) or from a JSON file:

# FP8 e4m3 — the format Cloudflare uses for Kimi K2.6
python3 -m sglang.launch_server \
    --model-path moonshotai/Kimi-K2.6 \
    --kv-cache-dtype fp8_e4m3

# FP8 e5m2 — slightly lower accuracy than e4m3
python3 -m sglang.launch_server \
    --model-path moonshotai/Kimi-K2.6 \
    --kv-cache-dtype fp8_e5m2

# NVFP4 on Blackwell hardware (SM100+)
python3 -m sglang.launch_server \
    --model-path nvidia/Kimi-K2.6-NVFP4 \
    --kv-cache-dtype nvfp4

If scaling factors are missing, SGLang defaults them to 1.0, which causes accuracy issues. Supply them via --quantization-param-path when they are not in the checkpoint:

{
  "kv_cache": {
    "dtype": "float8_e4m3fn",
    "scaling_factor": {
      "0": { "0": 1.0, "1": 1.0 }
    }
  }
}

The outer keys are tensor-parallel ranks; the inner keys are layer indices.

Compressing the Weights: INT4 for Decode, FP8 for Prefill

The weights are the other demand on GPU memory. For GLM 5.2, Cloudflare compresses from FP8 to INT4 with no measurable accuracy loss. The checkpoint shrinks from 705 GB to 421 GB — about 40% — and per-GPU memory across an 8-way tensor-parallel deployment drops from roughly 88 GB to 52 GB, leaving room for around 1.18 million tokens of KV cache on the same hardware.

Smaller weights make decode faster for a fundamental reason: generating each token means streaming the model's weights out of GPU memory, so decode speed is bounded by memory bandwidth. Move less data and every token arrives sooner. The effect is largest at low concurrency, where per-request latency matters most:

GLM FP8 (tok/s)	GLM INT4 (tok/s)	INT4 gain
60	92	+55%
425	513	+21%
683	825	—
994	1,267	+27%
1,672	1,933	+16%

Prefill behaves differently. It is compute-bound, and INT4 weights must be expanded back out before the model can multiply with them, so that extra step makes prefill slower. GLM sustains about 10,160 tokens per second of prefill in FP8 versus 8,660 in INT4.

This is where disaggregated prefill and decode stop being a nice-to-have and become a requirement. If you run prefill and decode on the same pool, you have to pick one dtype for both and eat the trade-off. Split them, and each pool gets the format that wins: INT4 for decode, FP8 for prefill. Cloudflare reports model accuracy stays within 0.8 points of the FP8 model across every benchmark — indistinguishable in practice.

SGLang Weight Quantisation Options

SGLang offers both online quantisation (compute scaling factors at runtime) and offline (pre-quantise a checkpoint). For GLM-style INT4 decode you want the offline path or a pre-quantised checkpoint:

# ModelOpt FP4 for decode (NVFP4 path, Blackwell + flashinfer_trtllm backend)
python3 -m sglang.launch_server \
    --model-path zai-org/GLM-5.2 \
    --quantization modelopt_fp4

# Explicit GEMM backend selection
python3 -m sglang.launch_server \
    --model-path zai-org/GLM-5.2 \
    --quantization modelopt_fp4 \
    --fp4-gemm-backend flashinfer_trtllm_routed

The --fp4-gemm-backend flag matters: NVFP4 GEMM relies on FlashInfer/TRT-LLM kernels, and the wrong backend silently leaves performance on the table. On non-Blackwell hardware, MXFP4 block-size-16 (--kv-cache-dtype fp4_mx_block16) is the fallback.

The Scaling Factors Caveat

One trap worth calling out: FP8 KV caches carry hidden overhead. Quantisation requires extra memory for block-based scaling factors, which eats into the raw bit-width saving. SGLang's own documentation reports that FP4 with block size 16 supports approximately 1.78× more tokens than FP8 and roughly 3.56× more than BF16 — which means FP8's real-world capacity gain over BF16 is closer to 2× on paper but the ratio shrinks once scaling factors are accounted for. Measure on your own workload before you commit.

Protecting a Shared KV Cache

Both techniques above have the same side effect: many more requests share one GPU's memory at once. That efficiency is the point, but it also means hundreds of requests are reading and writing pages of the same physical KV cache. Paged attention, continuous batching and cache reuse all rely on getting bookkeeping exactly right — and at Cloudflare's request volume, even a one-in-a-billion mistake would show up regularly.

Cloudflare's answer is a page-level integrity check. Every physical cache page gets a tag that changes whenever the page is reallocated. The server records which pages and tags each request expects to use, and before supported decode operations read from the cache, those mappings are checked. If anything does not match, the affected request is aborted rather than allowed to return data from the wrong page.

The cost decides whether a safety check ships. Measured on a mid-sized production model in a two-prefill, two-decode configuration, with 8,192-token inputs and 1,000-token outputs:

Concurrency	Throughput change	p95 latency change
1	−0.53%	+0.42%
2	−0.38%	+0.54%
4	−0.79%	+0.63%
8	−0.43%	+0.80%

Under 1% on both throughput and tail latency, even at the upper bound of the 95% confidence interval. Two implementation details make that possible: the validation runs as a separate batch check rather than being fused into the attention kernel (which would introduce a race between GPU thread groups), and it is enabled per deployment — the default path uses a no-op tracker with no measurable overhead, so deployments that do not need it pay nothing.

Putting It Together

The three techniques compose into a coherent serving strategy for a long-context MoE model:

1. Prefill pool — BF16/FP8 KV cache (compute-bound, keep the slightly higher precision and throughput).
2. Decode pool — FP8 KV cache for residency, INT4 weights for memory-bandwidth-bound token generation.
3. Shared cache — page-tag integrity checking on any pool where multiple requests share cache pages.

Cloudflare's roadmap is extending FP8 KV caches across more of the fleet, validating NVFP4 weights on Blackwell, and pushing integrity checks toward default-on at negligible cost. All of it is upstreamed into SGLang, so the same flags and formats are available to any self-hosted deployment.

The practical takeaway for a team running its own inference: measure your workload's concurrency distribution before choosing KV-cache dtypes, put prefill and decode on separate pools if you can, and do not ship quantised weights without an evaluation suite that confirms they are indistinguishable on your tasks. The cheapest optimisation in LLM serving is the one that doubles your memory-resident context for free — and FP8 KV caches are exactly that.