The shift to high-concurrency LLM inference engines

Ollama handles single-user workloads.

You know that scale changes everything. Many engineers start with Ollama for fast local iteration on a MacBook before migrating to vLLM on Kubernetes clusters to handle the massive throughput required for high-scale, multi-user, enterprise-grade, production-ready, real-world, live APIs. Ollama has over 100,000 GitHub stars and 52 million monthly downloads, making it the most adopted local runtime. However, Ollama flatlines at 22 requests per second because it does not batch requests.

TGI is in maintenance mode.

vLLM is the standard choice for production and scales well. vLLM handles 3.23x more requests than Ollama at 128 concurrent requests. LinkedIn uses vLLM to support 50 generative AI use cases, improving time per output token by 7%. Amazon uses vLLM’s continuous batching to manage token volume for its Rufus shopping assistant. SGLang provides 29% higher throughput than vLLM when requests share context like chatbots or RAG through RadixAttention.

The deployment landscape divides into specific workload classes. Single-user local environments favor Ollama or LM Studio for ease of use. Performance-local setups use ExLlamaV3 to maximize quality on NVIDIA hardware. Multi-user serving environments require vLLM or SGLang for continuous batching. Production batched tiers rely on TensorRT-LLM and Triton.

Memory management and speed

Memory management determines the efficiency of every token. During generation, the model stores intermediate calculations called key-value pairs for every token. A single sequence in Llama 70B consumes 1.7GB. Multiply this by 10 concurrent users and you use more VRAM on cache than on model weights. vLLM solves this with PagedAttention, which breaks memory into small reusable pages. This approach reduces memory fragmentation. Memory waste drops to under 4% using this method. SGLang uses RadixAttention to cache shared computation so the system does not redo work when prompts share prefixes. This makes SGLang the choice for multi-turn applications and agent pipelines.

The formula for KV cache per request is 2 times the number of layers, KV heads, head dimension, sequence length, and bytes per element. For a Llama 2 70B model with 80 layers, 8 KV heads, and 128 head dimension, an FP16 KV cache at 4K context requires 0.4 GB per request. At 200 concurrent requests, this requires 80 GB of cache alone.

Quantization also changes the memory footprint. GGUF, the successor to GGML, packages weights and metadata into a single binary with support for levels from Q2 to Q8. Aggressive compression introduces reconstruction error and accuracy tradeoffs. Traditional methods like GPTQ attempt to minimize error layer by layer, but they assume a fixed bit depth. Advanced methods like picoLLM learn optimal bit distribution across model components. This prevents the significant accuracy loss seen when traditional methods treat all weights the same. For example, GPTQ-quantized models lose more than half of their intelligence at 2-bit and 3-bit precision. SpinQuant uses learned rotations to eliminate outliers and achieves a 4.4-point accuracy gap compared to full-precision LLaMA-3-8B on the MMLU benchmark.

Hardware and specialized engines

Hardware choice determines the final performance. H200 SXM provides 141 GB of VRAM, which helps when pushing context past 4K or concurrency past 100 users. This extra capacity often beats the H100 in bandwidth-sensitive workloads. H200 provides 4.8 TB/s of memory bandwidth compared to 3.35 TB/s for the H100. NVIDIA H200 delivers up to 1.9x faster Llama 2 70B inference versus H100 when using TensorRT-LLM with FP8 and a batch size of 64. TensorRT-LLM extracts maximum performance from NVIDIA hardware through tensor optimization and memory layout tuning. TensorRT-LLM demands 1-2 weeks of setup time and locks you into a single vendor. Most production teams use Triton to route traffic to TensorRT-LLM backends on H100 or H200 nodes.

The choice depends on the workload. High-concurrency chat apps and LLM APIs on mixed hardware require vLLM. Standardized NVIDIA clusters where every millisecond matters require TensorRT-LLM. Teams using the Hugging Face ecosystem favor TGI. Structured generation and agent pipelines favor SGLang.

Which engine suits your specific budget?