Follow us
Breaking
Product Reviews

The economics of vLLM and proprietary APIs

Self-hosting open-weight models via vLLM becomes cost-effective at high, predictable volumes, such as when an NVIDIA H100 on RunPod reaches a break-even point of 12.1 billion tokens per month compared to Together AI.

Share

A single NVIDIA H100 GPU costs roughly $2,900 per month on RunPod based on November 2026 pricing, while a billion tokens of a small hosted model like Claude Haiku 5.5 costs about $180. This gap defines the financial divide between self-hosting an open-weight model like vLLM or Llama 4 and renting access to frontier models from OpenAI or Anthropic. Self-hosting makes financial sense when inference volume remains high, stable, and predictable. Most organizations find that API costs scale linearly with usage, whereas self-hosting costs front-load into hardware and engineering talent before flattening near zero.

Infrastructure and human capital costs

Self-hosting requires significant upfront capital for hardware or continuous rental fees for cloud GPU instances. A single NVIDIA H100 left running for a month costs between $2,900 and $6,500 in cloud rent at public on-demand prices. Organizations must also budget for power, cooling, monitoring, and security patching. Engineering time represents a massive hidden expense. A specialist might cost between £950 and £1,500 per day, meaning two days of maintenance per month adds $1,900 to $3,000 to the operational bill.

The specialized talent required to deploy open-source models often exceeds hardware expenses. MLOps engineers, DevOps specialists, and security engineers command high salaries, with senior ML engineers earning $150,000 to $250,000 annually. Deployment is not a one-time project because teams must continuously manage model drift and driver updates.

Factor Hosted API Self-hosted GPU
Billing Per token used Per GPU-hour of uptime
Scales to zero Yes No
Operational burden None High
Best fit Spiky or low volume High, steady volume

A minimal internal deployment for a small team runs between $125,000 and $190,000 annually. This assumes cloud GPU instances, a single GPU setup, and part-time engineering support. For enterprise-scale implementations serving thousands of users, costs can exceed $8,000,000 to $12,000,000 annually. At this scale, engineering headcount becomes the dominant cost driver, easily surpassing infrastructure expenses.

API pricing and utilization thresholds

Proprietary APIs provide a pay-as-you-go model that suits low or spiky workloads. OpenAI’s GPT-5.2 Pro costs $21.00 per million input tokens and $168.00 per million output tokens. In contrast, DeepSeek’s V3.2-Exp thinking models cost $0.28 per million input tokens and $0.42 per million output tokens. Using a single NVIDIA H100 on RunPod to run gpt-oss-120b costs twelve times more than using Together AI to serve the same model for a billion tokens.

The break-even point depends entirely on utilization. A single RunPod H100 breaks even with Together AI’s gpt-oss-120B at approximately 12.1 billion tokens per month. If you rent the H100 from AWS London instead, the break-even point rises to 27.2 billion tokens per month. If your GPU sits idle during non-business hours, the real cost per token increases by more than three times.

For regulated industries like healthcare or defense, the data privacy advantage of self-hosting outweighs the cost benefits of APIs. Sending sensitive data to third-party servers poses a risk of HIPAA or GDPR fines. Local inference keeps all data on the machine. A healthcare startup using a local Llama 3 model for patient FAQs cut AI costs by 95% while keeping all medical data on-premises. However, frontier APIs from Anthropic and OpenAI still lead in hard reasoning and complex tool use.

Model Tier Provider Input Price (per 1M tokens) Output Price (per 1M tokens)
GPT-5.2 Pro OpenAI $21.00 $168.00
GPT-5.2 Mini OpenAI $0.25 $2.00
V3.2-Exp DeepSeek $0.28 $0.42
Grok 4 xAI $3.00 $15.00
Grok 4 Fast xAI $0.20 $0.50

Scaling strategies and hybrid models

Successful enterprises rarely choose one single path. Most adopt a hybrid multi-model strategy where they deploy two or three models simultaneously. Companies use an API gateway to route simple classification tasks to cheap open-source models and escalate complex reasoning tasks to expensive Claude or GPT models. This routing can cut inference costs by up to sixty percent.

Managed open-model endpoints provide a middle ground. Services like Together AI or Amazon Bedrock allow companies to use open-weight models without managing hardware. These providers charge per token, which preserves the ability to scale to zero when usage drops. If volume grows and becomes highly predictable, a company can move that same model onto dedicated GPUs to capture better economics.

The decision to move from an API to self-hosting involves a simple calculation: divide the monthly self-hosted cost by the API cost per token. If the resulting number is higher than your monthly token volume, stay with the API. Does the added control of a private cluster justify the engineering burden for a team with no dedicated MLOps staff?

The most effective way to reduce costs is to stop sending every prompt to the largest model. Use smaller models for extraction, routing, and summarization. Implement prompt caching to reuse stable system instructions and tool definitions. If a workload requires massive, continuous throughput, self-hosting is the only way to avoid a ballooning API bill.

Share

Technewsdaily

Senior tech writer covering AI, gadgets and cybersecurity. Breaking down the news that matters, every day.