380

Qwen 3.6 on Dual RTX 5090 GPUs: Local LLM Deployment and Benchmarks

Deploying Qwen 3.6 on Two RTX 5090 GPUs: Benchmarks, Model Selection, and Lessons Learned

This article documents my experience deploying the Qwen 3.6 model family on two NVIDIA RTX 5090 GPUs to build a local inference infrastructure for multi-agent AI systems.

It covers the complete model selection process, including real-world benchmarks, deployment failures, performance optimizations, and lessons learned along the way.

If you are wondering which large language models can realistically run on consumer GPUs, how much context they can support, or how to balance inference quality against hardware limitations, this guide may help you avoid some expensive mistakes.

微信图片_20261110100534_41_92.jpg

 

1. Background: Exploring Local LLM Inference on Consumer GPUs

As large language models continue to evolve, one question has become particularly interesting to me: how much inference performance can we actually extract from consumer-grade GPUs?

Cloud-based APIs have made powerful AI models widely accessible, but understanding how these models behave at the infrastructure level is equally fascinating. Model architecture, GPU memory allocation, inference optimization, and concurrency management all play important roles in determining real-world performance.

This curiosity led me to experiment with deploying the Qwen 3.6 model family locally, using two NVIDIA RTX 5090 GPUs as the foundation for a multi-agent inference environment.

Why Focus on Multi-Agent Workloads?

My primary interest lies in multi-agent AI systems, where multiple AI agents collaborate to complete complex tasks. Different agents handle planning, information retrieval, execution, and verification, communicating through function calling and structured tool interactions.

Compared with conventional chatbot applications, these workloads place greater demands on the underlying inference infrastructure:

  • Reliable function calling: Models must consistently select the correct tools and generate valid JSON arguments.

  • High concurrency: Multiple agents need to execute requests simultaneously without significant performance degradation.

  • Large context capacity: Each agent maintains its own conversation history, instructions, tool definitions, and retrieved information.

  • Efficient inference: The system must maintain reasonable throughput and latency as the number of active agents increases.

These requirements make multi-agent workloads an interesting environment for evaluating the practical limitations of local LLM inference.

Rather than trying to replace cloud APIs, the purpose of this experiment was to better understand how different model architectures utilize GPU resources and how inference optimizations affect actual performance.

The Hardware and Model Selection Challenge

My test environment consisted of two NVIDIA RTX 5090 GPUs, each equipped with 32GB of GDDR7 memory, providing a combined 64GB of VRAM.

The central question was straightforward:

Which Qwen 3.6 model could deliver the best balance of function calling accuracy, context capacity, and inference throughput on this hardware?

At the time of testing, I focused on two available configurations:

  • Qwen3.6-27B: A dense architecture with 27 billion parameters activated during inference.

  • Qwen3.6-35B-A3B: A Mixture-of-Experts (MoE) architecture with 35 billion total parameters, but only approximately 3 billion activated per token.

The larger total parameter count of the MoE model did not necessarily mean higher GPU memory efficiency or better inference performance. Similarly, the dense model's conventional architecture did not automatically make it the safer choice.

The real challenge was finding the configuration that made the most effective use of the available hardware.

To answer that question, I compared KV cache capacity, function calling accuracy, inference throughput, and concurrency stability through a series of practical benchmarks.

The results were not always what I expected.

微信图片_20261110100537_42_92.jpg

 

2. Understanding KV Cache: The Biggest Constraint in Local Inference

Before comparing models, we need to understand one of the most important factors in local LLM deployment: KV cache.

What Is KV Cache?

Think of KV cache as the model's short-term working memory.

A language model stores learned knowledge in its weights. These weights represent what the model learned during training.

However, during a conversation, the model also needs to process the current context, including previous messages, instructions, constraints, and generated responses.

KV cache stores the attention keys and values associated with previously processed tokens, allowing the model to reuse them instead of repeatedly recalculating everything from scratch.

The important point is that KV cache consumes additional GPU memory as the context grows.

Every additional token requires more memory for its cached attention information.

This creates a fundamental hardware trade-off:

Available context capacity depends on how much VRAM remains after loading the model weights and allocating runtime overhead.

The more memory consumed by model weights, the less memory remains available for KV cache.

And when multiple agents run concurrently, they must share that available cache capacity.

The VRAM Budget

Consider the Qwen3.6-27B dense model using AWQ 4-bit quantization.

AWQ (Activation-aware Weight Quantization) compresses model weights to lower precision, reducing their memory footprint.

In this configuration, the approximate memory allocation per GPU was:

VRAM Allocation Memory per GPU
Model weights (AWQ 4-bit) ~9.6 GB
KV cache ~17 GB
CUDA graphs and other overhead ~5 GB
Total ~32 GB

With FP8 KV cache enabled, the inference engine reported a total KV cache capacity of 263,408 tokens across the deployment.

That sounds like a lot.

However, this capacity is shared across concurrent requests.

With 10 agents running simultaneously, the average available context capacity becomes:

263,408 ÷ 10 ≈ 26,000 tokens per agent.

For a simple chatbot, 26K tokens may be sufficient.

For a multi-agent system, however, a single complex task can easily consume 20K–30K tokens when it includes conversation history, tool definitions, retrieved documents, and intermediate reasoning results.

This exposes the central trade-off:

Larger model weights leave less memory for KV cache, which limits concurrent agents, context length, or both.

Once this constraint is understood, comparing model architectures becomes much more meaningful.

3. Comparing Three Model Candidates

I evaluated three possible configurations to determine which model would perform best under the hardware constraints.

Candidate 1: Qwen3.6-27B Dense — The Starting Point

The 27B dense model was my initial choice.

It offered strong model capabilities, a conventional architecture, and relatively straightforward deployment.

I deployed it with:

  • AWQ 4-bit weight quantization

  • FP8 KV cache

  • vLLM as the inference engine

  • Tensor parallelism across two RTX 5090 GPUs

FP8 KV cache reduces the memory required to store attention keys and values compared with FP16 or BF16, allowing a larger number of tokens to remain cached.

Benchmark Results

Metric Qwen3.6-27B AWQ
Total KV cache capacity 263,408 tokens
Context per agent at 10 concurrent requests ~26K tokens
Function calling accuracy (BFCL) 94.3%
Throughput at 10 concurrent requests 965 tokens/s

The function calling results were particularly encouraging.

For evaluation, I used BFCL, the Berkeley Function Calling Leaderboard, a benchmark designed to measure how accurately language models select tools and generate function arguments.

I sampled 50 examples from each of six benchmark categories, creating a 300-example evaluation set.

The 27B model achieved 94.3% accuracy, demonstrating strong performance in structured tool calling.

The main limitation was not accuracy.

It was context capacity.

With only approximately 26K tokens available per agent under a 10-agent workload, the system had limited headroom for complex tasks.

Lesson 1: A Small Benchmark Almost Led Me to the Wrong Conclusion

During the initial evaluation, I encountered an important testing mistake.

AWQ 4-bit quantization is a form of lossy compression. It reduces the precision of model weights, potentially affecting output quality.

There is a common concern that aggressive quantization can degrade function calling and structured output generation.

I wanted to verify whether this was happening.

Although BFCL was available, integrating it with my custom deployment required modifying parts of the evaluation pipeline. To save time, I initially created a smaller benchmark.

The test contained:

  • 5 function definitions

  • 24 Chinese prompts

  • Simple function calls

  • Complex nested JSON structures

  • Multiple-tool selection

  • Rejection scenarios

The first results were alarming.

In the complex nested-structure category, the AWQ model achieved only 33% accuracy, compared with 67% for the FP8 version.

That appeared to be a dramatic degradation.

I immediately concluded that AWQ was severely damaging the model's function calling capabilities and documented the finding.

Later, I completed the BFCL integration and repeated the evaluation using 300 standardized examples.

The results told a completely different story.

Configuration BFCL Accuracy
Qwen3.6-27B AWQ 4-bit 94.3%
Qwen3.6-27B FP8 93.7%
Difference 0.6 percentage points

The two models performed almost identically, with the small difference falling within the expected statistical variation of the test.

What went wrong with my original benchmark?

The complex nested-structure category contained only four examples.

With such a small sample, a single additional error could change the reported accuracy by 25 percentage points.

A few isolated failures were being magnified into what appeared to be a systematic problem.

The benchmark itself was not necessarily poorly designed. It simply lacked enough examples to support a reliable conclusion.

Key takeaway: Small-scale tests are useful for identifying possible issues, but they should not be used as definitive evidence of major performance regressions.

A small custom benchmark can help determine whether a problem deserves further investigation.

Final conclusions should be supported by a sufficiently large, preferably standardized evaluation.

Candidate 2: Qwen3.6-35B-A3B MoE — The Unexpected Winner

The second candidate was Qwen3.6-35B-A3B, based on a Mixture-of-Experts architecture.

Unlike a traditional dense model, an MoE model contains multiple specialized expert networks and activates only a subset during inference.

In this configuration, the model contains 256 experts and activates 8 of them during each inference step.

Although the model has approximately 35 billion parameters in total, only around 3 billion parameters are activated per token.

A simple analogy is to imagine a team of specialists.

Instead of asking every specialist to work on every problem, the system selects the most relevant experts for each task.

This allows the model to maintain a large total parameter count while reducing the computation required for each token.

The architecture seemed promising, but I initially had two concerns.

Concern 1: Actual KV Cache Capacity Might Be Lower Than Expected

At first, the 35B total parameter count made me worry that the MoE model would consume more VRAM than the 27B dense model.

After all, even though only a subset of experts is activated for each token, the complete set of model weights still needs to be stored or otherwise made available during inference.

The relevant question was therefore not simply how many parameters were active.

It was how much GPU memory the loaded model actually consumed and how much remained available for KV cache.

Architectural estimates can provide a starting point, but they do not capture every runtime allocation.

The only reliable answer would come from deploying the model and examining the inference engine's memory statistics.

As it turned out, my initial theoretical estimate was indeed too optimistic.

Concern 2: Would Multi-Token Prediction Help an MoE Model?

I also wanted to experiment with Multi-Token Prediction (MTP).

MTP is an inference acceleration technique in which the model predicts multiple future tokens and uses a verification process to accelerate generation when those predictions are accepted.

However, I repeatedly encountered the claim that:

MTP does not help MoE models because MoE inference is memory-bandwidth-bound.

The argument is straightforward.

MoE inference can require moving substantial amounts of expert weights from GPU memory to the compute units.

If transferring those weights is already the main bottleneck, speculative prediction and verification may introduce additional memory traffic without providing meaningful speed improvements.

I initially accepted this explanation and did not test MTP.

Then a colleague questioned the assumption.

The RTX 5090 offers substantially higher memory bandwidth than previous-generation consumer GPUs. Would the same limitation still apply?

That question prompted me to examine the hardware specifications.

The RTX 5090 Bandwidth Calculation

The RTX 5090 provides approximately 1,792 GB/s of theoretical memory bandwidth.

For comparison, the RTX 4090 provides approximately 1,008 GB/s.

That is an increase of roughly 78%.

For Qwen3.6-35B-A3B, I estimated that generating one token required reading approximately 6GB of active expert weights.

Using peak theoretical bandwidth:

6 GB ÷ 1,792 GB/s ≈ 0.00335 seconds

That corresponds to approximately 3.35 milliseconds of idealized weight-transfer time.

This calculation does not account for every memory access, computational overhead, or real-world bandwidth limitation. However, it provided a useful reason to question whether the conventional assumption applied to this hardware.

Instead of relying on the general claim, I decided to benchmark MTP directly.

MTP Benchmark Results

Configuration Throughput Latency
35B-A3B without MTP 1,123 tokens/s 9.5 s
35B-A3B with MTP 1,348 tokens/s 7.8 s
Improvement +20% −17%

The results were clear.

Enabling MTP increased throughput by approximately 20% while reducing latency by approximately 17%.

For my dual RTX 5090 configuration, MTP provided a meaningful performance improvement.

This does not prove that MoE models are never bandwidth-limited, or that MTP will improve performance on every GPU.

It demonstrates that a general architectural assumption should not replace hardware-specific testing.

Key takeaway: Performance conclusions are conditional on the hardware, model implementation, inference engine, and workload being tested.

What may be true for one GPU generation is not necessarily true for another.

Complete Benchmark Comparison: 27B Dense vs. 35B-A3B MoE

After tuning the MoE deployment, I compared it against the original 27B configuration.

Metric 27B Dense 35B-A3B MoE + MTP Advantage
KV cache capacity 263K 717K 2.7×
Context per agent at 10 concurrent requests ~26K ~72K 2.7×
Function calling accuracy (BFCL) 94.3% 93.3% Comparable
Throughput at 10 concurrent requests 965 tok/s 1,348 tok/s +40%
Concurrency stability 100% 100% Equal

The results were better than I expected.

The 35B-A3B configuration offered approximately 2.7 times the KV cache capacity of the 27B dense model.

This increased the average available context from around 26K to 72K tokens per agent under a 10-agent workload.

Function calling accuracy remained comparable, with only a one-percentage-point difference in the benchmark.

Meanwhile, total throughput increased by approximately 40%.

For my workload, the MoE model delivered a much better combination of context capacity, inference speed, and tool calling quality.

The main disadvantage was deployment complexity.

Getting the model running required resolving several configuration problems, which I describe in Appendix B.

However, those were one-time setup issues rather than ongoing performance limitations.

Candidate 3: Qwen3.5-9B Dense — Efficient but Not Capable Enough

I also considered the smaller Qwen3.5-9B dense model.

Its main advantage was lower memory consumption.

With fewer model weights occupying VRAM, more memory could theoretically be allocated to KV cache.

This could have provided substantially larger context capacity.

However, the model failed an even more important requirement: function calling quality.

According to the published benchmark figures I reviewed, Qwen3.5-9B achieved a BFCL score of approximately 66, compared with scores around 93–94 for the larger candidates.

That gap was too significant for my use case.

A multi-agent system depends on agents correctly selecting tools, filling structured parameters, and producing valid machine-readable outputs.

A failure in any of these operations can disrupt an entire task.

Even an extremely large context window is of limited value if the model cannot reliably execute the required tool calls.

Therefore, I eliminated the 9B model from consideration.

This reinforced an important principle:

Model capability should be the first selection criterion. Memory efficiency and throughput only become meaningful after a model meets the application's minimum accuracy requirements.

4. The Best Qwen 3.6 Configuration for Two RTX 5090 GPUs

Based on my benchmarks, the best configuration for this specific hardware and workload was:

Qwen3.6-35B-A3B + MTP + AWQ 4-bit Quantization + FP8 KV Cache

The reasoning comes down to three factors.

1. Much Larger Context Capacity

The deployment provided approximately 717K tokens of aggregate KV cache capacity, around 2.7 times that of the 27B dense model.

With 10 agents running concurrently, this translated to roughly 72K tokens per agent under an even allocation.

That provided considerably more flexibility for complex agent workflows involving long histories, retrieved documents, and tool outputs.

2. Comparable Function Calling Accuracy

The model achieved 93.3% accuracy on my BFCL evaluation set, compared with 94.3% for the 27B dense model.

For this workload, the one-percentage-point difference was small enough that the two models offered broadly comparable function calling capabilities.

The MoE architecture did not introduce the major degradation in structured output reliability that I had initially feared.

3. Higher Inference Throughput

With MTP enabled, the MoE configuration reached 1,348 tokens per second at 10 concurrent requests.

That represented approximately 40% higher throughput than the 27B dense model.

The relatively small number of activated parameters helped deliver efficient token generation despite the model's larger total parameter count.

I also retained the original 27B deployment configuration as a fallback, allowing me to switch back if necessary.

Final Recommendation

For anyone deploying large language models on two high-end consumer GPUs, I would strongly recommend evaluating three dimensions together:

  • KV cache capacity and effective context length

  • Function calling and structured output reliability

  • Throughput under realistic concurrency

Parameter count alone does not tell you which model will perform best.

Real-world benchmarks are essential.

5. Four Lessons From the Deployment Process

The most valuable outcome of this project was not simply discovering which model performed best.

It was identifying several mistakes in my evaluation methodology.

Here are the four lessons I would carry into future deployments.

Lesson 1: Never Draw Dramatic Conclusions From Tiny Benchmarks

My initial 24-prompt benchmark suggested that AWQ quantization severely degraded complex function calling.

The apparent difference was 33% versus 67%.

However, the relevant category contained only four examples, making the results highly sensitive to individual failures.

When I repeated the evaluation using 300 standardized BFCL examples, the difference between AWQ and FP8 was only 0.6 percentage points.

A small sample can be useful for quickly detecting potential problems.

It cannot reliably establish the magnitude or statistical significance of a performance regression.

For important deployment decisions, use larger evaluation sets and standardized benchmarks whenever possible.

Lesson 2: Do Not Blindly Apply General Performance Assumptions

I initially believed that MTP would provide little benefit for MoE models because of memory bandwidth limitations.

That assumption caused me to overlook a potentially useful optimization.

After examining the RTX 5090's bandwidth and testing the configuration, I discovered that MTP improved throughput by approximately 20%.

The broader lesson is that hardware characteristics matter.

Before accepting a general claim about inference performance, check whether its underlying assumptions apply to your specific GPU, model, and workload.

A simple theoretical calculation can help identify which optimizations are worth benchmarking.

Lesson 3: Validate Theoretical Memory Estimates Against Actual Runtime Data

Before deploying Qwen3.6-35B-A3B, I estimated its potential KV cache capacity using architectural parameters such as layer count, KV attention heads, and head dimensions.

My initial estimate suggested a capacity of approximately 2 million tokens, roughly 7.6 times that of the 27B model.

However, after deployment, the vLLM startup logs reported only about 790K tokens of KV cache capacity.

My theoretical estimate had overestimated the practical capacity by approximately 2.5 times.

The calculation had not adequately accounted for several sources of memory overhead, including:

  • MoE memory alignment requirements

  • CUDA runtime allocations and caching

  • Vision encoder components

  • Other model-specific and inference-engine overhead

The lesson is simple.

Architectural calculations are useful for planning, but the inference engine's actual runtime statistics should determine deployment decisions.

Lesson 4: The Best Model Is the One That Fits the Workload

Model selection is not a contest to find the highest parameter count.

The 27B dense model offered strong accuracy but relatively limited KV cache capacity.

The 9B model offered better theoretical memory efficiency but insufficient function calling capabilities.

The 35B-A3B MoE configuration delivered the best balance for my use case.

It maintained strong tool calling accuracy while significantly increasing both effective context capacity and throughput.

The correct selection process should begin by defining your requirements.

How much context does each agent need?

What level of function calling accuracy is acceptable?

How many requests must run concurrently?

What throughput and latency targets must the system meet?

Once these requirements are clear, benchmark candidate models against those constraints.

The best model is not necessarily the largest or smallest.

It is the one that provides the strongest results within your actual hardware and application requirements.


Appendix A: How to Calculate KV Cache Capacity

KV cache memory requirements can be estimated from a model's attention architecture.

The basic formula is:

KV cache bytes per token = 2 × Number of layers × Number of KV heads × Head dimension × Bytes per element

The components are:

  • 2: One allocation for keys and another for values.

  • Number of layers: The number of attention layers in the model.

  • Number of KV heads: The number of key-value attention heads.

  • Head dimension: The dimension of each attention head.

  • Bytes per element: The storage precision used for KV cache.

Typical precision values include:

Data Type Bytes per Element
FP8 1
FP16 2
BF16 2

Example: Qwen3.6-27B

For the 27B model, the architectural parameters used in my calculation were:

  • 64 layers

  • 4 KV heads

  • 256 dimensions per head

  • FP8 KV cache

Substituting these values:

2 × 64 × 4 × 256 × 1 = 131,072 bytes per token

The total theoretical KV cache capacity can then be calculated as:

KV cache capacity = Available KV cache memory ÷ KV cache bytes per token

For a tensor-parallel deployment, the calculation must account for the total allocated cache memory and how attention heads are distributed across GPUs.

In my deployment, the calculated capacity matched the 263,408-token figure reported in the vLLM startup logs.

This confirmed that the formula accurately described the 27B configuration under the tested settings.

However, as demonstrated by the MoE deployment, architectural calculations do not always capture every source of runtime memory overhead.

Always compare estimates with actual inference-engine logs.


Appendix B: Three Deployment Issues With Qwen3.6-35B-A3B

Switching from the 27B dense model to the 35B-A3B MoE model introduced several configuration issues.

Three of them caused container startup failures or repeated crashes.

Issue 1: Incorrect Quantization Format

The first problem was a mismatch between the specified quantization backend and the model's declared format.

The model expected AWQ, rather than compressed-tensors.

The correct argument was:

--quantization awq

Using the wrong quantization format prevented the model from loading correctly.

Issue 2: Unsupported Data Type

The second problem involved the model's computation data type.

For this AWQ deployment, the compatible setting was:

--dtype float16

The configuration did not support bfloat16 in the way I initially attempted to use it.

Changing the argument to float16 resolved this issue.

Issue 3: Mamba Block Alignment Assertion Failure

The third issue was the most difficult to diagnose.

The Qwen 3.5 hybrid architecture incorporates Mamba-like linear attention components in some layers.

In my deployment, the relevant implementation had a block alignment requirement involving a block_size of 2096.

However, the vLLM configuration used a default max_num_batched_tokens value of 2048.

This was 48 tokens below the required threshold, triggering an assertion failure.

The resulting behavior was confusing.

The inference container would start, attempt to load the model, crash, and restart repeatedly.

GPU monitoring showed memory being allocated and released over and over.

The logs provided little guidance beyond an easily overlooked AssertionError.

The solution was to explicitly increase the batching limit:

--max-num-batched-tokens 8192

After applying this setting, the deployment was able to proceed.

If you encounter repeated container crashes while deploying a Qwen 3.5 or 3.6 MoE model with a similar hybrid architecture, this parameter is worth investigating.

These settings reflect the specific model files, vLLM environment, and implementation used in my testing. Other model variants or inference engine versions may have different compatibility requirements.

Thoughts: Local Deployment vs. Cloud APIs

This deployment experiment was primarily driven by the author's personal interest in LLM architectures, GPU performance, and inference optimization. While running models locally provides valuable insights and greater control over the infrastructure, it also requires substantial hardware investment, careful configuration, and ongoing maintenance.

For developers and businesses running high-concurrency AI applications or multi-agent systems, managing local GPU infrastructure may not always be the most practical or cost-effective solution.

If your priority is reliable, large-scale access to leading AI models rather than managing inference infrastructure, CostRouter offers a simpler alternative. With significantly discounted token pricing, access to original model capabilities without performance degradation, and support for high-concurrency workloads, CostRouter helps teams focus on building AI applications instead of maintaining GPU servers.


Conclusion

Deploying a large language model on consumer GPUs is ultimately an exercise in balancing competing constraints.

Model accuracy matters, but so do memory allocation, concurrency, context capacity, and inference throughput.

In my dual RTX 5090 setup, Qwen3.6-35B-A3B with MTP, AWQ 4-bit quantization, and FP8 KV cache provided the strongest overall results.

Compared with the 27B dense configuration, it delivered approximately 2.7 times the KV cache capacity and 40% higher throughput while maintaining comparable function calling accuracy.

More importantly, the project reinforced the importance of reliable evaluation.

Small benchmarks can be misleading. General performance assumptions do not always transfer across hardware generations. Theoretical memory calculations must be validated against actual runtime behavior.

For local multi-agent AI systems, the best approach is to define your requirements, benchmark realistic workloads, and select the model that makes the most effective use of your available hardware.

 

Test date: June 19, 2026

Hardware: 2 × NVIDIA RTX 5090 GPUs (32GB GDDR7 per GPU)

Inference engine: vLLM

All benchmark results reflect the specific hardware, software configuration, and test conditions described in this article.