Qwen 3.6 on Dual RTX 5090 GPUs: Local LLM Deployment and Benchmarks
Deploying Qwen 3.6 on Two RTX 5090 GPUs: Benchmarks, Model Selection, and Lessons Learned
This article documents my experience deploying the Qwen 3.6 model family on two NVIDIA RTX 5090 GPUs to build a local inference infrastructure for multi-agent AI systems.
It covers the complete model selection process, including real-world benchmarks, deployment failures, performance optimizations, and lessons learned along the way.
If you are wondering which large language models can realistically run on consumer GPUs, how much context they can support, or how to balance inference quality against hardware limitations, this guide may help you avoid some expensive mistakes.
1. Background: Exploring Local LLM Inference on Consumer GPUs
As large language models continue to evolve, one question has become particularly interesting to me: how much inference performance can we actually extract from consumer-grade GPUs?
Cloud-based APIs have made powerful AI models widely accessible, but understanding how these models behave at the infrastructure level is equally fascinating. Model architecture, GPU memory allocation, inference optimization, and concurrency management all play important roles in determining real-world performance.
This curiosity led me to experiment with deploying the Qwen 3.6 model family locally, using two NVIDIA RTX 5090 GPUs as the foundation for a multi-agent inference environment.
Why Focus on Multi-Agent Workloads?
My primary interest lies in multi-agent AI systems, where multiple AI agents collaborate to complete complex tasks. Different agents handle planning, information retrieval, execution, and verification, communicating through function calling and structured tool interactions.
Compared with conventional chatbot applications, these workloads place greater demands on the underlying inference infrastructure:
-
Reliable function calling: Models must consistently select the correct tools and generate valid JSON arguments.
-
High concurrency: Multiple agents need to execute requests simultaneously without significant performance degradation.
-
Large context capacity: Each agent maintains its own conversation history, instructions, tool definitions, and retrieved information.
-
Efficient inference: The system must maintain reasonable throughput and latency as the number of active agents increases.
These requirements make multi-agent workloads an interesting environment for evaluating the practical limitations of local LLM inference.
Rather than trying to replace cloud APIs, the purpose of this experiment was to better understand how different model architectures utilize GPU resources and how inference optimizations affect actual performance.
The Hardware and Model Selection Challenge
My test environment consisted of two NVIDIA RTX 5090 GPUs, each equipped with 32GB of GDDR7 memory, providing a combined 64GB of VRAM.
The central question was straightforward:
Which Qwen 3.6 model could deliver the best balance of function calling accuracy, context capacity, and inference throughput on this hardware?
At the time of testing, I focused on two available configurations:
-
Qwen3.6-27B: A dense architecture with 27 billion parameters activated during inference.
-
Qwen3.6-35B-A3B: A Mixture-of-Experts (MoE) architecture with 35 billion total parameters, but only approximately 3 billion activated per token.
The larger total parameter count of the MoE model did not necessarily mean higher GPU memory efficiency or better inference performance. Similarly, the dense model's conventional architecture did not automatically make it the safer choice.
The real challenge was finding the configuration that made the most effective use of the available hardware.
To answer that question, I compared KV cache capacity, function calling accuracy, inference throughput, and concurrency stability through a series of practical benchmarks.
The results were not always what I expected.