LLM Inference Kernels: The Open-Source Arms Race

The cost of running a large language model at scale is not the model — it is the inference. A 70-billion-parameter model in BF16 occupies 140 GB of weights; serving it at meaningful concurrency means keeping those weights resident, moving them through tensor cores, and managing KV cache across thousands of concurrent requests. Every microsecond of kernel time and every byte of memory traffic compounds into real money. In mid-October 2026, four open-source releases — vLLM Humming, DeepSeek DSpark, DeepSeek DeepInfer, and llm-d v1.0 — each attack a different layer of that cost. Together, they represent the most significant shift in open-source inference infrastructure since vLLM itself.

The Three Fronts

Inference cost decomposes into three bottlenecks, and each product targets a different one:

Bottleneck	Product	Approach	Key Metric
Weight memory bandwidth	vLLM Humming	JIT-compiled quantized GEMM kernels	2.5–4.5× over DeepGEMM; 14–43% end-to-end throughput
Token generation latency	DeepSeek DSpark	Confidence-scheduled speculative decoding	60–85% per-user speedup
Serving infrastructure	DeepSeek DeepInfer / llm-d v1.0	Disaggregated prefill/decode, distributed scheduling	155T tokens/day (DeepInfer); multi-tenant QoS (llm-d)

These are complementary, not competitive. A production deployment might use Humming kernels inside a vLLM instance orchestrated by llm-d, with DSpark accelerating token generation. The open-source ecosystem is building a stack, not a single winner.

vLLM Humming: One Kernel Library for Every Quantization Format

Humming is the most immediately useful release. Developed by Ant Group and the vLLM community, it is a JIT-compiled GEMM kernel library for quantized inference that supports every NVIDIA GPU from Turing (SM75) onward and a wide range of quantization formats — WNA16, W8A8, W4A8, W4A4, with INT and FP weight types from 1 to 8 bits.

The architectural insight is that quantization kernels are not one-size-fits-all. Different models use different weight precisions (GPTQ INT4, AWQ, FP8, MXFP4, NVFP4), different GPUs have different tensor core capabilities (Hopper FP8, Blackwell MXFP4/MXFP8), and different batch sizes impose different memory-bandwidth constraints. Hand-writing a kernel for each combination is intractable. Humming instead JIT-compiles the right kernel on demand from CUDA components, tuned for the specific GPU, quantization format, and workload.

The performance numbers are striking. On Hopper H200, Humming's W4A8-MXFP4 path reaches approximately 4.5× the throughput of DeepGEMM's W8A8 baseline at small and medium batch sizes. On H20 GPUs running DeepSeek-V4.1-Flash with tensor parallelism of 8, switching the MoE backend from Marlin W4A16 to Humming W4A8 improves end-to-end throughput by 14–43% and reduces mean time-to-first-token by 25–32%, with the advantage growing at higher concurrency. At concurrency 64, output throughput rises from 442 to 634 tokens per second.

Humming is integrated into vLLM starting from version 0.29.0. Enabling it is a single flag:

vllm serve deepseek-ai/DeepSeek-V4.1-Flash-1017 \
    --tensor-parallel-size 8 \
    --linear-backend humming --moe-backend humming

The kernel library itself is remarkably lightweight — the PyPI wheel is approximately 500 KB, with only PyTorch and NVCC as dependencies. On-demand compilation avoids bundling a large collection of precompiled kernels. Any environment that can run PyTorch 2.7 can run Humming.

The significance extends beyond raw speed. Humming unifies the quantization landscape: instead of choosing between Marlin, FlashInfer, and DeepGEMM — each supporting a subset of formats — a single backend handles them all. For teams deploying multiple models with different quantization schemes, this eliminates a class of integration work.

DeepSeek DSpark: Speculative Decoding That Schedules Its Own Verification

DSpark, released by DeepSeek in June 2026, attacks a different bottleneck: the sequential nature of autoregressive decoding. Each token depends on the previous one, so generation is inherently serial. Speculative decoding breaks this serialisation by having a small draft model propose multiple tokens, which the target model verifies in parallel. The speedup depends on how many draft tokens the target accepts.

DSpark's contribution is confidence-scheduled verification. Rather than always verifying the same number of draft tokens, DSpark estimates which prefix of the draft is likely to survive and adjusts verification length based on both model confidence and current serving load. This is combined with semi-autoregressive generation: a parallel backbone drafts most tokens quickly, and a lightweight sequential head refines later tokens to maintain coherence.

The results, reported by DeepSeek on DeepSeek-V4.1-Flash and V4-Pro, are significant. At matched system capacity, per-user generation speed improves by 60–85% for V4-Flash and 57–78% for V4-Pro over the previous MTP-1 baseline. Aggregate throughput improvements are even larger — 661% for V4-Flash at a 120-token-per-second-per-user service target — because DSpark avoids the operational cliff where the old system could maintain only a small number of concurrent requests at that responsiveness level.

Independent developer benchmarks corroborate the gains. Rafael Caricio reported a five-run mean of 60.31 tokens per second for DeepSeek-V4.1-Flash with DSpark, compared to 39.88 with MTP-1 and 26.33 without speculative decoding — a 1.51× improvement over MTP-1 and 2.29× over no-spec.

DSpark is MIT-licensed and integrated into DeepInfer, DeepSeek's production inference framework.

DeepSeek DeepInfer: The Production Stack

DeepSeek open-sourced DeepInfer on 20 October — the inference framework that powers their production services, where a single cluster serves 155 trillion tokens per day at a revenue of approximately $38 per GPU-hour at off-peak pricing. This is not a research prototype; it is the engine behind a service serving millions of users.

DeepInfer's architecture is built around five system-level ideas:

1. Optimised kernels — FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels
2. Parallelism — attention data parallelism, balanced context parallelism, MoE expert parallelism with expert-aware load balancing (EPLB)
3. DSpark and confidence-scheduled verification — speculative decoding integrated into the serving path
4. SWA bounded replay — reduces decoder prefill work by replaying only a bounded suffix (the last 128 tokens) through the decoder layers, nearly halving prefill computation
5. Asynchronous scheduling — overlaps CPU batch scheduling with device execution, enabled by default

The production numbers are self-reported but instructive. With 64 GPUs in the decode pool (EP64), DeepInfer achieves above 25,000 output tokens per second per GPU at 256 requests per rank, with near 100 tokens per second per user. At 8 requests per rank, per-user speed passes 300 tokens per second. The reported input cache-hit rate is 96%, which explains the gap between the 1.5 million cache-inclusive input tokens per second per GPU and the 60,000 uncached — reused prefixes replace repeated prefill with a cache read.

DeepInfer currently supports only DeepSeek-V4.1-Flash on NVIDIA SM100 (Blackwell) and Huawei Ascend 950 NPUs. The KV cache management, prefill/decode disaggregation, and load-balancing code are not included — DeepSeek says these are tightly coupled to internal infrastructure. The smallest efficient deployment described uses 64 GPUs for prefill and 64 for decoding.

llm-d v1.0: Distributed Inference as Kubernetes Native

While DeepInfer optimises a single cluster, llm-d v1.0 — released on 20 October by a community of over 500 contributors from 25 organisations, co-founded by Google with Red Hat and IBM — optimises the orchestration layer. llm-d is a Kubernetes-native, CNCF Sandbox project for distributed LLM inference that runs across NVIDIA, AMD, and Intel GPUs and Google TPUs, behind both vLLM and SGLang model servers.

The v1.0 release focuses on Day 2 operations: serving multiple tenants, balancing competing priorities, upgrading without downtime, and supporting agentic workloads. The core capabilities include:

• Prefix-cache-aware routing — directs requests to infrastructure that already holds the shared context, reducing redundant prefill. On an agentic coding workload on Google TPU v7 (Ironwood), this cut median time-to-first-token by more than 85% and raised total throughput 2.4× versus a plain Kubernetes Service.
• KV cache offloading — moves KV cache blocks between GPU HBM, host memory, and prefill/decode pools via NIXL, extending the useful life of cached context.
• Prefill/decode disaggregation — separates compute-heavy prefill from memory-bound decode into independent pools, managed as a single logical unit via DisaggregatedSet. Rolling updates move both phases together on matching versions while each scales independently.
• Wide expert parallelism — distributes large mixture-of-experts models across nodes using DeepEP for expert communication, with topology-aware scheduling that places tightly coupled parallelism within the same NVLink domain.
• Multi-tenant quality of service — priority and fairness-aware routing that helps latency-sensitive requests and bursty agent workloads share a fleet without starving one another.

Performance highlights from production deployments include 3× higher output throughput and 2× faster TTFT with prefix-cache-aware routing versus round-robin (Llama 3.1 70B on 4× AMD MI300X), 40% reduction in TTFT and ITL with predicted-latency scheduling, and up to 70% higher tokens per second with prefill/decode disaggregation versus standard vLLM on NVIDIA B200.

The architectural bet of llm-d is composability. Teams keep their existing engine (vLLM or SGLang), model, and cluster, and adopt only the serving components they need. The router, scheduler, and disaggregation layer are co-designed but independently deployable.

What the Numbers Mean in Practice

The headline numbers — 4.5× kernel speedup, 85% speculative decoding speedup, 155 trillion tokens per day — are impressive but require context. Kernel speedups are microbenchmark results that translate to 14–43% end-to-end improvement, because kernels are one component of a serving stack that also includes scheduling, memory management, and network overhead. Speculative decoding speedups depend on draft acceptance rate, which degrades with growing context in multi-turn coding sessions. Production throughput numbers assume high cache-hit rates that require specific workload patterns (multi-turn conversations with shared prefixes).

The real story is not any single number but the compounding effect. Humming kernels reduce per-token cost. DSpark reduces the number of forward passes per token. DeepInfer optimises the scheduling and caching that determine utilisation. llm-d orchestrates the fleet so that prefill and decode are placed optimally across accelerators. A deployment that combines all four attacks every layer of the inference cost stack.

The Open-Source Advantage

All four releases are open-source — Humming under Apache 2.0, DSpark under MIT, DeepInfer under Apache 2.0, and llm-d under Apache 2.0. This matters because inference infrastructure is not a place where vendor lock-in is tolerable. Models change, GPU generations change, and the optimal serving configuration for a given workload changes with them. An open stack that can be inspected, modified, and reconfigured is a strategic asset in a way that a proprietary endpoint is not.

The competitive landscape is also shifting. When DeepSeek open-sources the framework behind 155 trillion tokens per day, and when vLLM and Ant Group jointly release kernels that make every NVIDIA GPU since Turing faster, the marginal cost of running open models approaches the cost of the hardware itself. The proprietary inference advantage — optimised kernels, speculative decoding, distributed scheduling — is being systematically eroded.

Recommendations

For teams running inference today, the practical takeaways are:

1. Upgrade to vLLM 0.29.0+ and enable Humming. The --linear-backend humming --moe-backend humming flags are a single-line change with double-digit throughput improvements on Hopper and Blackwell.
2. Evaluate DSpark for latency-sensitive workloads. If your serving pattern involves multi-turn conversations where the draft model can predict the next few tokens accurately, the 60–85% per-user speedup is substantial.
3. Watch DeepInfer for production patterns. Even if you cannot deploy it directly (SM100 and Ascend 950 only), the SWA bounded replay and asynchronous scheduling techniques are applicable to any inference stack.
4. Adopt llm-d for multi-tenant serving. If you are running vLLM or SGLang on Kubernetes, llm-d's prefix-cache routing and disaggregation are production-ready and deliver measurable improvements.

The inference arms race is not over. But the open-source ecosystem now has kernels, speculative decoding, production frameworks, and orchestration layers that match or exceed what proprietary platforms offer. The question is no longer whether open-source inference is competitive — it is which combination of these tools fits your workload.