DeepSeek V4 Flash 0731: Initial Experiences on 2x DGX Spark

Overview

DeepSeek AI released DeepSeek-V4-Flash-0731, an updated checkpoint of their mixture-of-experts (MoE) model that power their flaghship DeepSeek-V4 reasoning model. This model retains the same DeepseekV4ForCausalLM architecture and DSpark speculative-decoding structure as the preview release, but delivers improved performance, stability, and compatibility. We deployed this new checkpoint on our 2x NVIDIA DGX Spark cluster with vLLM, DSpark speculative decoding, and the Anemll serving framework, and put it through rigorous benchmarking and validation. Here's what we found.

This evaluation expands on our prior Atlas multi-model orchestration setup, focusing specifically on the performance characteristics, serving constraints, and production readiness of the new 0731 checkpoint.

---

Deployment Configuration

Our testbed is a 2x DGX Spark cluster connected via InfiniBand with the following serving configuration:

Parameter	Value	Notes
Model	deepseek-ai/DeepSeek-V4-Flash-0731	Revision 9e165c30e2704aec5d9d593cce3eebd58bbef1cb
Context Length	1,048,576 tokens	1M context ceiling
Tensor Parallelism	2	ai1 (head) + ai2 (worker)
Speculative Decoding	DSpark, MTP-5	5-token probabilistic sampling
KV Cache Dtype	nvfp4_ds_mla	NVFP4 for DeepSeek MLA
max_num_seqs	6	Maximum concurrent sequences
max_model_len	1,048,576	Matches context length
GPU Memory Utilization	0.835	Anemll optimized
Block Size	256	Token chunk size
Available KV Cache	14.58 GiB	~1,894,867 tokens
Max Concurrency at 1M	1.81x	Can handle ~1.8 concurrent 1M-context sessions

Serving Stack:

• Runtime: Anemll ghcr.io/anemll/dspark-vllm-gx10:0.1.1
• Backend: vLLM with DSpark patches
• Moe Backend: flashinfer_b12x
• Async Scheduling: Enabled
• Chunked Prefill: Enabled (threshold=1024)
• Prefix Caching: Enabled
• CUDA Graphs: Disabled (enforce-eager: true) — required for GB10 unified memory

---

Benchmark Methodology

We conducted two distinct benchmark suites:

1. Screenshot Matrix (Anemll Default)

Unique cold prefixes, thinking=false, max_tokens=min_tokens=128, ignore_eos. This measures pure decode performance after prefill, which is the typical steady-state for most agent applications where prompts are reused or cached.

2. Natural Completion Sweep (PR #14 Standard)

Natural ~512-token completions with DEFAULT_THINKING=max. This represents real-world usage where the model spends time in <think> blocks unless capped. This matches our earlier historien DeepSeek benchmarking approach.

All benchmarks used distinct first cache blocks to prevent prefix caching from artificially inflating subsequent runs — a critical consideration when evaluating true cold-start performance.

---

Performance Results

Screenshot Matrix (Decode-Focused)

Prompt Length	Concurrency 1	Concurrency 2	Concurrency 4	Concurrency 6	Aggregate (c=6)
256 tokens	82.6 tok/s	56.9 tok/s	54.3 tok/s	35.9 tok/s	162 tok/s
2,048 tokens	64.6 tok/s	50.1 tok/s	29.6 tok/s	20.8 tok/s	78 tok/s
8,192 tokens	65.9 tok/s	44.2 tok/s	16.0 tok/s	10.2 tok/s	29 tok/s
32,768 tokens	62.0 tok/s	39.7 tok/s	8.7 tok/s	8.2 tok/s	8 tok/s
131,072 tokens	74.9 tok/s	36.0 tok/s	8.4 tok/s	8.6 tok/s	2 tok/s

Key Observations:

• Single-stream (c=1) is remarkably flat at ~62-83 tok/s across all prompt lengths from 256 to 128K. This means solo decode performance is consistent regardless of context size.
• c=6 with short prompts (256 tokens) delivers 162 aggregate tok/s — excellent for multi-user scenarios with typical agent turns.
• c=4/6 with long prompts (32K+) drops to ~8 tok/s per stream due to serialized prefill steps (issue #27). This is a fundamental limitation: long cold prefills must run sequentially.
• 128K × c=6 median TTFT: 282 seconds (4.7 minutes) — this is the cost of six simultaneous cold 128K prefills.

Natural Completion Sweep (Real-World Usage)

Prompt Length	Concurrency	TTFT (s)	Prefill tok/s	Decode tok/s	Aggregate tok/s
256	1	0.63	447	75.4	69.1
256	2	0.81	357	58.3	104.9
256	4	1.26	222	46.8	164.5
256	6	1.42	197	36.9	191.2
2,048	1	0.81	2,563	68.8	62.0
2,048	2	1.11	1,911	57.0	97.6
2,048	4	1.38	1,505	44.0	154.7
2,048	6	6.06	342	34.7	143.7
8,192	1	4.80	1,713	73.9	43.7
8,192	2	7.51	1,176	49.8	56.2
8,192	4	14.50	578	37.4	72.3
8,192	6	18.38	454	23.6	73.1
32,768	1	22.96	1,428	64.0	16.6
32,768	2	26.82	1,287	41.5	24.8
32,768	4	44.85	756	17.4	26.7
32,768	6	60.75	550	10.8	27.9
131,072	1	78.75	1,665	65.2	5.9
131,072	2	111.17	1,306	30.9	6.6

Key Takeaways:

• c=1 decode remains strong at 64-76 tok/s across all context lengths — matching our prior DeepSeek V4 Flash findings.
• Aggregate throughput peaks at c=6, 256 tokens: 191.2 tok/s — this is the sweet spot for multi-user agent workloads.
• c=6, 2048 tokens: 143.7 aggregate tok/s — still excellent, with per-stream decode at 34.7 tok/s.
• Prefill performance degrades with concurrent long contexts: Six 32K prefills take ~60 seconds with prefill tok/s dropping to 550 (from ~342 at c=6, 256 tokens).

---

Serving Audit Results

We ran the full serving audit suite against the 0731 checkpoint on our 2x DGX Spark cluster running Anemll 0.1.1:

Phase	Tool	Result	Expected
Throughput	bench-miaai.py	C1: 74-76 tok/s, C6: 168-180 tok/s	C1: 73-76, C6: 168-180
Spec-Decode Health	spec-acceptance.py	~72% acceptance	~68-75%
Long-Context Quality	ruler-lite.py	8/8 at 8k/32k	8/8
Tool Calling	tool-battery.py	7/7 tests passed	7/7
Deep-Context Tools	deepctx-tool-battery.py	8/8 tests passed	8/8
Issue #55 Truncation	n/a	finish=length (correct)	Always finish=length
Garble Sweep	context-garble-sweep.py	CLEAN at all lengths through ~900k	CLEAN

Acceptance Curve (Per-Position):

• Position 0: ~0.93 (near-perfect)
• Position 1: ~0.85
• Position 2: ~0.65
• Position 3: ~0.45
• Position 4: ~0.33

This is the normal DSpark curve — acceptance drops as draft tokens progress, which is expected behavior for speculative decoding with MoE models.

---

Production Observations

What Works Well

1. Excellent Single-User Performance: At c=1, decode speeds of 64-76 tok/s are outstanding for a 1M-context MoE model. This is comparable to much smaller dense models.

2. Strong Multi-User Throughput: With c=6 and short prompts (256 tokens), we achieved 191.2 aggregate tok/s. For typical agent interactions (1-4K context), c=6 delivers 143-164 aggregate tok/s.

3. Rock-Solid Stability: The 0731 checkpoint passed all audit tests — throughput, acceptance, retrieval, tool calling, and context integrity. No regressions from the preview checkpoint.

4. Efficient KV Cache: NVFP4 with MLA quantization keeps KV cache memory at 14.58 GiB for ~1.89M tokens, enabling 1.81x concurrency at 1M context on our 2x Spark setup.

5. Proper Context Limits: The server correctly rejects prompts exceeding 1M tokens — this is not a bug but a protective measure.

Constraints and Trade-offs

1. Long Context Prefill is Expensive: Six concurrent 128K prefills take 282 seconds (4.7 minutes). This is due to serialized prefill execution (issue #27) where long prefills must run one at a time.

2. CUDA Graphs Must Be Disabled: The enforce-eager: true requirement for GB10 unified memory means no CUDA graph capture, which costs ~28% in decode performance (74.6 tok/s with breakable graphs vs 95.9 tok/s without). This is a GB10-specific constraint.

3. Acceptance Overhead: DSpark speculative decoding adds ~4.5x undercounting in SSE-based measurements. Always use usage.completion_tokens for accurate counting.

4. Prefix Caching is Essential: Without unique first cache blocks, later requests can unfairly benefit from earlier prefill work, skewing benchmark results.

5. Cold Start Cost: First request after restart incurs 6-10s cold autotune overhead. Warm-up requests are essential for accurate timing.

Configuration Recommendations

Based on our testing, here are our production serving recommendations:

# Optimal 2x DGX Spark configuration for DeepSeek-V4-Flash-0731
model: deepseek-ai/DeepSeek-V4-Flash-0731
revision: 9e165c30e2704aec5d9d593cce3eebd58bbef1cb

# Serving parameters
tensor_parallel_size: 2
max_num_seqs: 6
max_model_len: 1048576
gpu_memory_utilization: 0.835

# Performance-critical settings
kv_cache_dtype: nvfp4_ds_mla
speculative: dspark,5,probabilistic
enforce_eager: true
moe_backend: flashinfer_b12x
async_scheduling: true
chunked_prefill: true
prefix_caching: true

# Tokenizer/encoder
trust_remote_code: true
# Note: Anemll auto-injects encoding_dsv4.py

Do NOT change:

• max_num_seqs > 6 — will exceed KV cache capacity
• gpu_memory_utilization > 0.835 — risk of OOM
• max_model_len > 1M — model hard limit

---

Comparison: 0731 vs Preview Checkpoint

Metric	Preview	0731	Change
C1 Decode (tok/s)	~73	74-76	+1-3%
C6 Aggregate (256t)	~182	191.2	+5%
Acceptance Rate	~70%	~72%	+2%
Stability	Good	Excellent	Improved
encoding_dsv4.py	Manual	Auto-injected	✅
Issue #21 Hotfix	Required	Included	✅

The 0731 checkpoint is a straight upgrade — better performance, improved stability, and built-in fixes for known issues.

---

Lessons Learned

Benchmarking Pitfalls

1. Token Counting: SSE-based token counting undercounts by ~4.5x. Always use API-reported usage.completion_tokens.

2. Prompt Length Estimation: Word-count filler estimates are unreliable (~1.56 tok/word for our test data, not 1.3). Always verify via /tokenize endpoint or the 1M ceiling may be mis-scored.

3. Pathological Prompts: Repeated-word prompts collapse the DSpark drafter and can fake a ~40% regression. Use real task prompts for accurate testing.

4. Warm-Up Required: First request after restart includes 6-10s cold autotune. Run ≥2 warm-up requests per prompt length before timing.

Serving Best Practices

1. Enable Prefix Caching: Critical for multi-turn conversations and repeated prompts. Reduces prefill overhead significantly.

2. Use NVFP4 MLA: The nvfp4_ds_mla KV cache dtype provides the best memory efficiency vs performance trade-off for DeepSeek.

3. Keep enforce_eager: true on GB10: CUDA graphs cause issues with GB10 unified memory. The ~28% performance cost is necessary for stability.

4. Set Proper Context Limits: The 1M ceiling is a server protection, not a bug. Over-length prompts are correctly rejected.

5. Monitor Acceptance Rates: Per-position acceptance curves are normal. A drop from ~0.93 (pos0) to ~0.33 (pos4) is expected DSpark behavior.

---

Use Cases and Fit

Ideal For

• Multi-agent systems with typical 1-4K context turns (c=6 delivers 143-164 aggregate tok/s)
• Single-user deep work with 1M context (64-76 tok/s decode)
• Production agent deployments where stability and correctness matter more than raw speed
• Long-context retrieval (RULER-lite: 8/8 at 8k/32k)
• Tool calling workflows (7/7 basic, 8/8 deep-context)

Less Suitable For

• High-throughput short-prompt batch processing (other architectures may be better)
• Real-time applications requiring <50ms latency (consider smaller, faster models)
• Massive concurrent users (>6 simultaneous) — scale horizontally with more DGX Spark nodes

---

Next Steps

We're continuing to:

1. Test Vision Coexistence: Early results show approximately ~1.95-2.04M KV tokens with vision, providing ~1.81-1.95× concurrency at 1M text context. See vl-nvfp4-coexist-2026-08-11.md for details.

2. Explore Stage-C Image: The Stage-C Docker image with VLLM_DSPARK_GPU_REJECTED_CONTEXT_MASK may offer different performance characteristics.

3. Evaluate 200K/16 Keys Concurrency: The Keys Concurrency Patch enables 315.1 aggregate tok/s at c=16 with 200K context — interesting for high-concurrency scenarios.

4. Compare with Dense Models: For use cases where MoE overhead isn't beneficial, we're benchmarking against dense alternatives.

---

Conclusion

DeepSeek-V4-Flash-0731 is a production-ready, high-performance MoE model that delivers excellent results on 2x DGX Spark infrastructure. With 74-76 tok/s single-stream decode and 191.2 tok/s aggregate at c=6, it's well-suited for both single-user deep work and multi-agent deployments.

The model passes our full serving audit suite with flying colors — 7/7 tool calling, 8/8 long-context retrieval, clean garble sweep through ~900k tokens, and proper context limit enforcement. Combined with Anemll's vLLM+DSpark serving framework, this is one of the most capable open-source model deployments available today.

Bottom Line: If you're serving DeepSeek V4 Flash on DGX Spark hardware, upgrade to 0731. It's faster, more stable, and includes all the fixes you need for production deployment.

---

Resources

• dspark-0731 Repository — Our deployment scripts, benchmarks, and configuration
• Anemll vLLM DSpark — The serving framework we use
• DeepSeek-V4-Flash-0731 Model Card — Official model repository
• HBKU DGX Spark — Hardware platform details
• Graphwiz AI Atlas Setup — Our multi-model orchestration article
• Keys Concurrency Patch — For high-concurrency scenarios

---

Benchmarks conducted on 2x NVIDIA DGX Spark systems with InfiniBand interconnect, running Anemll dspark-vllm-gx10:0.1.1. Results may vary based on hardware configuration, network topology, and model revision.