Kimi K3: Moonshot AI's Open 2.8 Trillion Parameter Model

On 16 July 2026, Moonshot AI released Kimi K3 — a 2.8-trillion-parameter Mixture-of-Experts model that is the first open-weight model to cross the 3T-class threshold. With a 1-million-token context window, native vision capabilities, and two novel architectural innovations, K3 represents a step change in what enterprises can deploy on their own infrastructure.

Architecture

Kimi K3 introduces two architectural advances that together yield approximately 2.5× improvement in scaling efficiency over its predecessor, Kimi K2.

Kimi Delta Attention (KDA) reworks how information flows across long sequences. Standard multi-head attention computes pairwise interactions between all tokens, which scales quadratically with context length. KDA introduces a delta-based gating mechanism that allows the model to selectively attend to information changes rather than reprocessing the full context at every layer. This is particularly important for the 1M-token window, where naive attention would be computationally prohibitive.

Attention Residuals (AttnRes) address information flow across model depth. Deep transformers — K3 has over 100 layers — suffer from gradient vanishing and representational collapse in early layers. AttnRes creates residual connections specifically for the attention mechanism, allowing each layer to build on the attention patterns of preceding layers rather than recomputing from scratch.

The model uses a Stable LatentMoE framework with 896 experts, activating 16 per token. At this sparsity level, routing becomes a first-order engineering challenge. K3 introduces Quantile Balancing, which derives expert allocation directly from router-score quantiles, eliminating heuristic updates and the sensitive balancing hyperparameter that plagues traditional MoE training.

Component	Specification
Total parameters	2.8T
Active parameters per token	~50B (16/896 experts)
Context window	1M tokens
Architecture	KDA + AttnRes + Stable LatentMoE
Attention	Kimi Delta Attention with GQA
Training optimiser	Per-Head Muon
Precision	MXFP4 weights, MXFP8 activations (QAT)
Vision	Native, via cross-attention

Benchmark Performance

Kimi K3 trades blows with the best closed models across coding, agentic, and knowledge work benchmarks. The table below shows select comparisons:

Benchmark	Kimi K3 (max)	Claude Fable 5 (max)	GPT 5.6 Sol (max)
DeepSWE	67.5	70.0	73.0
Terminal-Bench 2.1	88.3	84.6	88.8
SWE Marathon	42.0	35.0	39.0
BrowseComp	91.2	88.0	90.4
Toolathlon-Verified	73.2	77.9	74.9
GPQA Diamond	93.5	92.6	—
MMMU-Pro	81.6	81.2	83.0

K3 leads on Terminal-Bench 2.1 (88.3), BrowseComp (91.2), and SWE Marathon (42.0) — benchmarks that reward sustained agentic reasoning rather than single-shot accuracy.

Enterprise Deployment

K3's size presents real infrastructure requirements. At MXFP4 quantisation, the weights occupy approximately 1.4 TB. Full BF16 inference requires over 5 TB of VRAM, necessitating multi-node deployments.

Moonshot AI has contributed KDA-compatible prefix caching to the vLLM community, enabling competitive token pricing despite the model's scale. The recommended deployment configuration uses supernodes with 64 or more accelerators connected via high-bandwidth interconnects.

Strategic Implications

K3 validates three trends that matter for enterprise AI strategy:

1. The open-weight frontier is expanding. A 2.8T open model that competes with proprietary frontier systems means enterprises can evaluate capability parity without vendor lock-in.

2. Architecture innovation is accelerating. KDA and AttnRes are not incremental improvements — they change the cost curve for long-context and deep-stack inference.

3. MoE at scale is a solved engineering problem. Stable LatentMoE with Quantile Balancing demonstrates that expert routing at 896-way sparsity can be stable and efficient in production.

For enterprise teams evaluating next-generation LLM infrastructure, Kimi K3 is not just a model — it is a reference implementation of what open-weight, long-context, multi-modal AI looks like at the 2026 frontier.