The rise of Ollama 0.4 and the performance divide in local LLM deployment

Ollama dominates the developer prototyping workflow

Isaac Johnson recently tested Ollama with his pybsposter application. While testing gemma3n:e4b and gemma4:4b on a Ryzen 9 mini PC, he encountered timeouts. He also used Python and pytest to test models. Using Gemini 2.5 Pro via Vertex AI, he built system messages. Ollama simplifies model management through a Go-based server that wraps a llama.cpp inference backend. This architecture allows a single pull command to download quantized models via the Ollama Hub. He updated his 14b and 7b models using the ollama list command to verify his local versions. Recent updates to Ollama added GLM-OCR and web search to the API. The tool also provides structured outputs constrained to JSON schema. I find Ollama is clearly the best choice for developers who need to experiment with various parameter sizes and quantization levels without complex configurations. You might already know that Ollama handles chat request templating and model loading automatically. Users require 8 GB of RAM for 7B models, 16 GB for 13B, and 32 GB for 33B.

Throughput benchmarks reveal the vLLM production advantage

Teams moving from prototyping to production face a stark performance gap between Ollama and vLLM. In a benchmark using a single NVIDIA RTX 4090 with 24GB VRAM, an AMD Ryzen 9 7950X, 64GB of DDR5 RAM, and CUDA 12.6, Ollama (Q4_K_M) delivered 62 tokens per second for Llama 3.1 8B during single-user testing, while vLLM (FP16) reached 71 tokens per second in the same scenario. The difference scales as concurrency increases. At 10 concurrent users, vLLM sustained 485 total tokens per second for Llama 3.1 8B (FP16), whereas Ollama reached 148 total tokens per second. At 50 users, vLLM maintained 920 tokens per second, but Ollama plateaued at 155. For DeepSeek-R1-Distill-Llama-8B, vLLM sustained 840 tokens per second at 50 users, while Ollama hit 142.

vLLM uses PagedAttention to manage GPU memory dynamically, but Ollama allocates memory statically per model load. This distinction defines performance under load. I find vLLM is unequivocally the superior choice for production deployment because its continuous batching engine aggregates requests into unified GPU operations. vLLM delivers higher throughput and lower latency across all concurrency levels from 1 to 256 concurrent users. In single-user scenarios, Ollama’s lighter server overhead makes it competitive, delivering 58 tokens per second for DeepSeek-R1 against vLLM’s 67 tokens per second. However, at high concurrency levels above 16, vLLM’s inter-token latency increases. Ollama’s latency becomes erratic with massive spikes. vLLM delivers a peak of 793 tokens per second compared to Ollama’s 41 tokens per second. At peak throughput, vLLM shows a P99 latency of 80 ms, while Ollama hits 673 ms. Even when I set OLLAMA_NUM_PARALLEL to 32 to tune for parallelism, Ollama’s time to first token increased sharply as it struggled to juggle more requests.

Hardware scaling and the move toward local inference

The demand for local inference grows as enterprises manage data residency and avoid rising cloud costs. AMD targets this market with the Instinct MI350P PCIe accelerator. This dual-slot, air-cooled card provides 144GB of HBM3E memory and reaches 4,600 TFLOPS at lower precision. The card supports up to eight accelerators per air-cooled system and provides 4TB/s of bandwidth. It supports modern formats like MXFP4, MXFP6, FP8, and INT8. This hardware helps with inference workloads, including agentic AI and RAG pipelines. The MI350P allows deployment in standard servers without changes to power or cooling configurations. AMD integrates the MI350P into an open ecosystem that includes Kubernetes GPU Operator support and AMD Inference Microservices. AMD’s software stack also includes compatibility with frameworks like PyTorch. AMD reported a 57% year-over-year increase in Data Center revenue, with total revenue reaching $10.3 billion in the first quarter. While vLLM provides high-throughput production, hardware like the MI350P allows companies to scale AI workloads within existing CPU-based systems. This approach avoids the capital requirements of full GPU platform deployment. I wonder if the growing reliance on specialized PCIe accelerators will eventually force a redesign of standard server racks.