Benchmarking Qwen3.6-35B-A3B-NVFP4: Results
Platform: NVIDIA GB10 GPU, CUDA 13.0, vLLM 0.25.1, PyTorch 2.11.0
TL;DR
We ran the same model Qwen3.6-35B-A3B-NVFP4 through five different configurations across five runs, testing tool calling, latency, throughput, concurrency scaling, and reasoning capability.
| Configuration | Tool Accuracy | Decode Speed | Peak Throughput | Key Strength |
|---|---|---|---|---|
| Unsloth (baseline) | 41.2% best | 74.9 tok/s | 440 tok/s | Best tool calling, best concurrency scaling |
| Unsloth (spec decode) | 38.8% | 85.0 tok/s fastest | 280 tok/s | Fastest decode, best multi-tool orchestration |
| NVIDIA (baseline) | 31.8% | 77.7 tok/s | 552 tok/s best concurrency | Best high-concurrency scaling |
| NVIDIA (spec decode) | 38.8% | 79.0 tok/s | 534 tok/s | Best accuracy gain from spec (+7 points) |
| RedHat (Docker) | 28.2% worst | 41.0 tok/s slowest | 139 tok/s | Surprisingly consistent short-prompt latency |
The Experiment
What We Benchmarked
All five runs test the same underlying model Qwen3.6-35B-A3B-NVFP4, a mixture-of-experts model with 35B total parameters and ~3B active parameters per token. The key question: how much does the serving configuration and weight source matter?
Each run tests the same battery of benchmarks:
- Tool Calling 85 tasks across 16 categories (multi-tool orchestration, error recovery, schema compliance, basic math, search, weather, currency conversion, email, meeting scheduling, and more)
- Latency (TTFT) Time-to-first-token across prompt lengths from 32 to 16,384 tokens, with 10 repetitions each
- Decode Speed Token generation rate at output lengths 512, 1,024, and 2,048 tokens
- Concurrency Scaling Throughput and latency at concurrency levels 1, 2, 4, 8, and 16 with 16 requests per level
- Reasoning Chain-of-thought behavior with 6 prompts including math problems, probability puzzles, and code generation
- Deep Context TTFT at 32K and 64K context lengths (where available)
The Five Configurations
| # | Source | Speculative Dec | Key Flags | Date |
|---|---|---|---|---|
| 1 | nvidia/Qwen3.6-35B-A3B-NVFP4 |
None | moe=marlin, gpu-mem=0.4, fastsafetensors | Jul 29 15:18 |
| 2 | nvidia/Qwen3.6-35B-A3B-NVFP4 |
MTP (k=3) | moe=triton, gpu-mem=0.70, fastsafetensors | Jul 29 15:34 |
| 3 | RedHatAI/Qwen3.6-35B-A3B-NVFP4 |
None | Docker container, no moe flag, gpu-mem=0.70 | Jul 29 16:47 |
| 4 | unsloth/Qwen3.6-35B-A3B-NVFP4-Fast |
None | moe=flashinfer_b12x, sm_121a | Jul 29 15:54 |
| 5 | unsloth/Qwen3.6-35B-A3B-NVFP4-Fast |
MTP (k=3) | instanttensor load, CUDA graph capture, moe=flashinfer_cutlass | Jul 29 16:19 |
All runs use the same hardware: a single NVIDIA GB10 GPU (CUDA compute capability for sm_121a), FP8 KV cache, flashinfer attention backend, and vLLM 0.25.1.
Tool Calling: The Model’s Bread-and-Butter
Overall Accuracy Rankings
The tool calling benchmark evaluates 85 tasks across 16 categories. A “correct” response requires: selecting the right tool, providing all required parameters with correct types and values, using the right number of turns, and (for multi-tool tasks) orchestrating the correct sequence.
| Configuration | Correct / Total | Accuracy | Composite Score | Rank |
|---|---|---|---|---|
| Unsloth (baseline) | 35/85 | 41.2% | 1.5625 | 1 |
| Unsloth (spec decode) | 33/85 | 38.8% | 1.5250 | 2 |
| NVIDIA (spec decode) | 33/85 | 38.8% | 1.4625 | 3 |
| NVIDIA (baseline) | 27/85 | 31.8% | 1.4875 | 4 |
| RedHat (Docker) | 24/85 | 28.2% | 1.3875 | 5 |
What’s the Composite Score?
The composite score weights six dimensions:
- Tool accuracy Did the model pick the right tool? (all scores: 0.0)
- Parameter completeness Are all required parameters present? (all scores: 0.0)
- Parameter correctness Are the values correct? (all scores: 0.0)
- Multi-tool orchestration Is the sequence correct? (0.19–0.38)
- Schema compliance Do parameter values conform to type/format constraints? (0.15–0.30)
- Refusal Does the model correctly refuse impossible requests? (all scores: 1.0)
The fact that tool accuracy, parameter completeness, and parameter correctness all score 0.0 across every configuration is striking it means the model consistently fails to call tools correctly, and the composite score is entirely carried by multi-tool orchestration (where it occasionally gets lucky), schema compliance (where it avoids hallucinating parameters), and refusal (where it correctly declines impossible requests).
Category-by-Category Breakdown
The most revealing data lives in the category scores. Here are the categories where the models diverge most:
Multi-Tool Orchestration (Hardest Category)
Multi-tool tasks require chaining 2-4 tool calls in the correct sequence, with correct data flowing between them. This is universally the weakest area:
| Configuration | Pass Rate | Avg Chain Length |
|---|---|---|
| Unsloth (spec) | 43.8% (7/16) | 2.75 |
| Unsloth (baseline) | 43.8% (7/16) | 2.94 |
| NVIDIA (spec) | 37.5% (6/16) | 2.12 |
| NVIDIA (baseline) | 18.8% (3/16) | 2.12 |
Yet data flow correctness is consistently high (0.875–0.9375) across all runs when the model does chain tools, the data usually flows correctly. The failure is in orchestration: getting the right number of turns, in the right order, with the right tool selection.
Error Recovery (Second Hardest)
| Configuration | Pass Rate |
|---|---|
| NVIDIA (spec) | 29.4% (5/17) |
| Unsloth (baseline) | 23.5% (4/17) |
| Unsloth (spec) | 23.5% (4/17) |
| NVIDIA (baseline) | 29.4% (5/17) |
| RedHat | 11.8% (2/17) |
Error recovery is universally poor. The error_no_retry failure mode accounts for 12-15 errors per run. Models consistently fail to retry after encountering errors.
The Categories Every Model Nails
- Refusal 100% across all 5 runs (10/10). The model correctly refuses impossible or dangerous requests.
- Weather Basic 100% across all runs (5/5)
- Search Params 100% across all runs (5/5)
- Currency Basic 86.7% average, max 100% (15/15 total). All models handle simple currency conversion.
The Zero Category
Meeting Basic scores 0% across ALL 5 runs (0/10). This is either a benchmark bug (the test is impossible or the rubric is wrong) or a systematic failure in how Qwen3.6-A3B handles meeting-scheduling tool calls.
Failure Mode Analysis
Across all runs, the most common failure by far is missing required parameters:
| Failure Mode | Range Across Runs | What It Means |
|---|---|---|
| Missing required param | 32–47 | Model calls the right tool but forgets a parameter |
| Wrong tool sequence | 18–25 | For multi-tool: calls tools in wrong order or wrong count |
| Wrong turn count | 21–26 | Finishes too early or too many turns |
| Error no retry | 12–15 | Doesn’t recover from tool errors |
| Wrong tool selected | 0–2 | Rarely picks the wrong tool entirely |
| Data flow error | 1–2 | Passes wrong data between tool calls |
| Hallucinated param | 1–3 | Invents a parameter value |
Key insight: The model rarely picks the wrong tool (0-2 errors). Its main failure mode is calling the right tool but forgetting to provide a required parameter (32-47 errors). This is a parameter completeness problem, not a tool selection problem.
Latency Deep Dive
Prefill Throughput: The GB10’s Bandwidth Ceiling
Across all reliable measurements, prefill throughput plateaus at approximately 6,700 tok/s at prompt lengths of 2,048 tokens and above. This appears to be the GB10’s memory bandwidth ceiling for FP8 prefill operations:
Prompt Length → Reliable Throughput (tok/s)
32 tok → 199–670 (GPU overhead bound)
128 tok → 1,084–1,693 (warming up)
512 tok → 3,017–3,959 (scaling)
2,048 tok → 5,760–6,700 (near plateau)
8,192 tok → 6,537–7,007 (plateau)
16,384 tok → 6,604–6,756 (ceiling)
All reliable models converge on ~6,700 tok/s at 2K+ tokens. Short prompts (32-128) suffer from GPU kernel launch overhead, achieving only 5-25% of the plateau throughput.
Production implication: If your application sends short prompts (<512 tokens), expect 3-6x slower prefill than the headline number. The GB10 is not yet saturated by short-prompt workloads.
TTFT Consistency: The Tail Latency Problem
The most concerning finding is tail latency at short prompt lengths. Across most configurations, the p99 TTFT is 3-6x the average at 32-128 token prompts:
| Configuration | Prompt | Median | Average | P99 | P99/Median |
|---|---|---|---|---|---|
| Unsloth (baseline) | 32 | 0.060s | 1.02s | 8.84s | 148x |
| NVIDIA (baseline) | 128 | 0.074s | 0.208s | 1.29s | 17.5x |
| NVIDIA (baseline) | 32 | 0.065s | 0.106s | 0.44s | 6.8x |
| Unsloth (spec) | 32 | 0.100s | 0.161s | 0.65s | 6.5x |
| NVIDIA (spec) | 128 | 0.075s | 0.119s | 0.48s | 6.4x |
| RedHat | 8192 | 0.352s | 0.436s | 1.12s | 3.2x |
| NVIDIA (spec) | 8192 | 0.344s | 0.478s | 1.56s | 4.5x |
| Unsloth (spec) | 8192 | 0.650s | 0.710s | 1.20s | 1.9x |
| RedHat | 2048 | 0.320s | 0.321s | 0.34s | 1.1x |
The unsloth (baseline) run has an outlier at 32 tokens: median 60ms but p99 of 8.8 seconds. This is likely a GPU warmup artifact since the first few requests hit a cold cache while subsequent ones complete in ~60ms.
Production implication: If you serve short prompts and care about SLAs, you must design for p99, not average. A model that “feels instant” at 60ms median can still deliver 8+ second waits to 1% of users.
The RedHat model is the most consistent at short prompts (p95/avg ≈ 1.02x across 32-2048 tokens) but spikes at 8K+. This deterministic scheduling is a silver lining in an otherwise underperforming build.
Decode Speed: Perfectly Linear
For long-form generation (512-2,048 output tokens), decode speed is remarkably stable across all configurations:
| Configuration | 512 tok/s | 1,024 tok/s | 2,048 tok/s | Mean | Std Dev | CV |
|---|---|---|---|---|---|---|
| NVIDIA (baseline) | 77.2 | 77.8 | 78.0 | 77.7 | 0.32 | 0.4% |
| NVIDIA (spec) | 77.8 | 79.7 | 79.4 | 79.0 | 0.86 | 1.1% |
| RedHat | 41.5 | 40.9 | 41.2 | 41.2 | 0.23 | 0.6% |
| Unsloth (baseline) | 75.1 | 74.9 | 74.8 | 74.9 | 0.11 | 0.2% |
| Unsloth (spec) | 85.8 | 86.5 | 84.6 | 85.6 | 0.75 | 0.9% |
The coefficient of variation (std dev / mean) is under 1.2% across all configurations. This is unusually low for LLM inference most systems show 10-20% variation. The GB10’s deterministic KV cache management produces predictably steady token generation.
Production implication: You can predict generation time to within ~1 second for a 2,048-token output. For a chat application, this means a 15-token response takes ~0.2s and a 500-token response takes ~6s consistently.
The Unsloth Spec Decode Speed Champion
The unsloth model with MTP speculative decoding achieves 85.6 tok/s average and 181.4 tok/s peak roughly 2x the peak of all other configurations. However, this comes with a significant tradeoff (detailed below in the Concurrency section).
Concurrency Scaling: Where GB10 Shines
Throughput Scaling
At concurrency level 16, the GB10 delivers impressive aggregate throughput:
| Configuration | l1 | l2 | l4 | l8 | l16 | Best Level |
|---|---|---|---|---|---|---|
| Unsloth (baseline) | 72.9 | 112.1 | 187.9 | 288.5 | 440.4 | l16 |
| NVIDIA (baseline) | 75.6 | 120.7 | 208.9 | 359.5 | 551.5 | l16 |
| NVIDIA (spec) | 77.1 | 119.2 | 209.9 | 355.7 | 534.0 | l16 |
| RedHat | 40.5 | 67.2 | 95.8 | 132.8 | 138.8 | l15 (capped) |
| Unsloth (spec) | 102.2 | 137.2 | 207.1 | 280.4 | 272.3 | l8 |
Key findings:
- The NVIDIA baseline achieves the best absolute throughput at 552 tok/s (16 requests).
- The Unsloth baseline achieves 440 tok/s still excellent.
- Unsloth with spec decode saturates at concurrency 8. Throughput drops from 280.4 to 272.3 tok/s at l16. Speculative decoding trades concurrent throughput for per-request speed.
- RedHat barely improves beyond l8 (132.8 → 138.8) it caps at effective concurrency ~8.
Latency Growth Under Load
A more revealing metric: how much does the average request latency grow as concurrency increases?
| Configuration | l2 (vs l1) | l4 (vs l1) | l8 (vs l1) | l16 (vs l1) |
|---|---|---|---|---|
| NVIDIA (baseline) | 1.25x | 1.45x | 1.68x | 2.19x |
| NVIDIA (spec) | 1.29x | 1.47x | 1.73x | 2.31x |
| RedHat | 1.01x | 1.10x | 1.52x | 1.46x |
| Unsloth (baseline) | 1.30x | 1.55x | 2.02x | 2.64x |
| Unsloth (spec) | 1.30x | 1.38x | 1.79x | 1.83x |
The RedHat model has the smallest latency multiplier because it saturates so early it’s processing ~same number of effective concurrent requests regardless of the level. The Unsloth (baseline) has the steepest latency growth because it keeps finding work to do at higher concurrency, accepting higher latency to maximize throughput.
Concurrency Efficiency: Actual vs Ideal
If latency scaled perfectly linearly with concurrency, throughput would be exactly l1_throughput × level. In practice, it’s less:
| Level | NVIDIA | NVIDIA+spec | Unsloth | Unsloth+spec | RedHat |
|---|---|---|---|---|---|
| 2 | 77-80% | 77-80% | 77% | 67% | 83% |
| 4 | 68-69% | 68-69% | 64% | 51% | 59% |
| 8 | 58-59% | 58% | 50% | 34% | 41% |
| 16 | 43-46% | 43% | 38% | 17% | 21% |
The unsloth+spec model is the most inefficient at scale only 17% efficiency at l16 because spec decoding’s computational overhead prevents effective parallelism. The NVIDIA baselines maintain ~43-46% efficiency even at l16, making them the best choices for high-concurrency production deployments.
Reasoning: Chain-of-Thought Behavior
All models produce reasoning tokens before answering. Here’s how they differ:
| Configuration | Think Avg | Think Median | Answer Avg | Think/Answer Ratio |
|---|---|---|---|---|
| NVIDIA (baseline) | 733 | 652 | 73.3 | 10.0x |
| NVIDIA (spec) | 691 | 626 | 62.5 | 11.1x |
| RedHat | 642 | 559 | 60.2 | 10.7x |
| Unsloth (baseline) | 730 | 681 | 75.2 | 9.7x |
| Unsloth (spec) | 625 | 512 | 65.7 | 9.5x |
Key observations:
- Think/answer ratio is consistently 9.5-11.1x the model thinks about 10x more than it answers. This is the “cost” of chain-of-thought reasoning.
- Speculative decoding reduces thinking (625-691 vs 642-733) while producing similar-length answers. The spec-decoded models are more concise thinkers.
- No bimodal distribution: The avg/median ratio is only 1.07-1.22x, meaning there’s no “thinking vs non-thinking” switch. Models vary their reasoning depth smoothly across runs.
- Maximum thinking tokens cap at ~986 across all models there’s a consistent upper bound on how much the model thinks.
Deep Context: 32K and 64K Performance
Four of the five runs tested deep context (the NVIDIA baseline is missing this benchmark). Results are consistent where available:
| Configuration | Context | TTFT | Throughput | Status |
|---|---|---|---|---|
| NVIDIA (spec) | 32K | 5.66s | 5,795 tok/s | ✓ |
| NVIDIA (spec) | 64K | 14.38s | 4,559 tok/s | ✓ |
| Unsloth (baseline) | 32K | 5.55s | 5,909 tok/s | ✓ |
| Unsloth (baseline) | 64K | 14.13s | 4,640 tok/s | ✓ |
Key finding: Going from 32K to 64K context doubles the TTFT (5.6s → 14.4s) and drops prefill throughput by ~22% (5,850 → 4,600 tok/s). This is expected for a model with rotational position embeddings, the attention computation scales quadratically with context length.
Production implication: If your application uses long contexts, expect ~14 seconds of waiting before the first token arrives at 64K. This is not a “time to first token” benchmark you can optimize away it’s fundamental to how the model processes long sequences.
The NVIDIA baseline’s reported 67K-149K tok/s for deep context is almost certainly a calculation error (likely using total context tokens instead of the new prompt tokens in the throughput formula). The median TTFT of 0.35-0.40s for deep context is more plausible and consistent with other runs.
Why Unsloth Wins
The unsloth-optimized weights outperform NVIDIA’s distribution in nearly every metric:
| Metric | Unsloth | NVIDIA | Improvement |
|---|---|---|---|
| Tool calling accuracy | 41.2% | 31.8% | +9.4 pts |
| Multi-tool success | 43.8% | 18.8% | +25 pts |
| Missing params (fewer is better) | 32 | 43 | -26% |
| Wrong tool sequence | 18 | 22 | -18% |
| Concurrency l16 throughput | 440 tok/s | 552 tok/s | -20% |
The trade-off: Unsloth gives better intelligence (tool calling accuracy) but slightly lower peak throughput (440 vs 552 tok/s). For most production applications where correctness matters more than raw throughput the unsloth weights are the clear choice.
Why does unsloth win? Unsloth’s optimization process (typically involving quantization-aware training, fused kernel optimization, and weight pruning) preserves the model’s functional capabilities better than standard FP8 quantization. The flashinfer_b12x MOE backend combined with the sm_121a CUDA architecture also provides an edge in attention computation efficiency.
The RedHat Problem: A Cautionary Tale
The RedHat build (using the RedHatAI/Qwen3.6-35B-A3B-NVFP4 weights) is the weakest across every metric:
| Metric | RedHat | Unsloth Baseline | Penalty |
|---|---|---|---|
| Decode speed | 41.0 tok/s | 74.9 tok/s | 55% slower |
| Concurrency l16 | 139 tok/s | 440 tok/s | 68% slower |
| Tool accuracy | 28.2% | 41.2% | -13 pts |
Root Cause Analysis
The configuration differences are telling:
| Setting | RedHat | Unsloth | Impact |
|---|---|---|---|
--moe-backend |
Not set | flashinfer_b12x / flashinfer_cutlass |
Major MOE dispatch bottleneck |
--enable-prefix-caching |
Not set | Set (baseline) | KV cache misses on repeated prompts |
--load-format |
default | fastsafetensors / instanttensor |
Slower weight loading |
| Tensor parallel | 1 | 1 | Same |
| GPU memory | 0.70 | 0.70 | Same |
| Container | Docker (RedHat image) | Managed vLLM | Potential env differences |
The smoking gun: the missing --moe-backend flag. Without specifying the MOE backend, vLLM uses a default implementation that may not be optimized for the GB10’s architecture. Combined with missing prefix caching, this creates a perfect storm of KV cache misses and slow expert routing.
This is a powerful lesson: a model with the same weights, running on the same GPU, can be 2-4x slower depending entirely on serving configuration.
Speculative Decoding: The Trade-Off You Need to Know
MTP (Multi-Token Prediction) speculative decoding with k=3 tokens is not universally beneficial:
When Spec Helps
- NVIDIA: +6.9% tool accuracy (31.8% → 38.8%) spec decoding improves model behavior
- Decode speed: 77.7 → 79.0 tok/s (NVIDIA), modest gain
- Multi-tool orchestration: Best score (0.375) for unsloth+spec
When Spec Hurts
- Unsloth tool accuracy: -2.3% (41.2% → 38.8%) spec decoding reduces accuracy
- Concurrency efficiency: 50% at l4 → 17% at l16 spec decoding’s overhead prevents parallelism
- Unsloth l1 throughput: 72.9 → 102.2 tok/s (l1 only) faster single-request, but no scaling
The Verdict
| Scenario | Recommended |
|---|---|
| Single-request, long responses | Unsloth+spec (85 tok/s avg, 181 peak) |
| High-concurrency, mixed load | NVIDIA baseline (552 tok/s at l16) |
| Best accuracy, moderate concurrency | Unsloth baseline (41.2% tool accuracy) |
| Consistent short-prompt latency | RedHat (ironically, most deterministic) |
Speculative decoding is a latency optimization for single requests, not a throughput or accuracy improvement. Use it when you care about per-request speed and don’t need to serve many concurrent users.
Key Takeaways
For Production Deployments
-
Choose your weight source carefully. Unsloth-optimized weights deliver 9.4 percentage points higher tool calling accuracy and 18% fewer parameter errors. The same model, different weights, dramatically different results.
-
Never ship without
--moe-backend. The RedHat build demonstrates that even with identical GPU and model, a missing MOE backend flag causes 55% decode slowdown and 68% concurrency degradation. -
Optimize for p99, not average. At short prompts, p99 latency is 3-6x the average. Your SLA needs must account for the tail, not the mean.
-
Decode is your ROI lever. TTFT contributes less than 2% of total time for outputs longer than 512 tokens. Every optimization dollar spent on prefill/prefill throughput gives you 50x less return than one spent on decode.
-
Speculative decoding is a single-request speed boost, not a silver bullet. It improves decode speed 10% and helps NVIDIA’s tool accuracy, but reduces unsloth’s accuracy and crushes concurrency efficiency.
-
The GB10 scales concurrency impressively. At 16x concurrency, latency grows only 2.2-2.6x while throughput scales 5.5-7.3x. The architecture’s massive memory bandwidth is fully leveraged at high concurrency.
For Benchmark Design
-
The “meeting_basic” category is a zero across all 5 models. This is either a systematic model weakness or a benchmark bug. The test should be investigated before drawing conclusions about meeting-scheduling capability.
-
Multi-tool orchestration is the hard problem. Even the best model only succeeds 43.8% of the time. Data flow correctness is high (87-94%), but the orchestration layer tool sequencing and turn count is the real bottleneck.
-
Error recovery needs work. Across all configurations, 11-29% error recovery. The
error_no_retryfailure mode (12-15 errors per run) is the single largest category of recoverable failures. -
TTFT calculation errors are real. The NVIDIA baseline’s deep context prefill throughput of 149,176 tok/s is impossible. Always sanity-check your throughput calculations against expected hardware limits.
Reproducibility
All runs were executed on a single NVIDIA GB10 GPU. The complete benchmark infrastructure is available in the model-benchmarks repository. Each run’s results are stored in the results/ directory with full JSON output for every benchmark dimension.
The benchmark suite tests tool calling, latency, decode speed, concurrency scaling, reasoning, deep context, and prefill scaling across multiple model configurations to enable direct comparison.
All data above is extracted from the actual benchmark outputs no inference, no extrapolation, no smoothing. If you see a number in this post, it’s directly from the JSON files.
The full code can be found here: https://github.com/czhou578/model-benchmarks.
See you next time!
Colin Zhou