When Is Self-Hosting LLM Inference on Your Own GPUs Cheaper Than Paying for an API?

If your service uses a large language model to generate text, you can run it on GPUs you own or rent, or send requests to a provider’s API, an application programming interface. When the model generates a response, that work is called inference. An API charges for use, while self-hosting means you pay for the hardware and the people who keep it running.

Self-hosting costs less when your GPUs handle enough requests to spread their fixed costs across many responses, and when the model you host meets the same quality, response-time, and reliability requirements as the API model you would otherwise use. Low or irregular traffic, expensive staffing, or a need for extra capacity during peaks can make the API less costly overall. Compare your actual workload and full costs to decide; no single token-volume cutoff applies to every service.

Compare equivalent service and full costs

Start by comparing your costs with the API model that meets your quality needs. A small, quantized model running on your GPU may cost less per token than a more capable API model, but the savings are meaningful only if the smaller model performs well enough for the same tasks. Quantization reduces the memory or computing resources a model needs by representing its internal values with less precision; it can also affect output quality.

To estimate your monthly API bill, count the input and output tokens your service actually uses. A token is a piece of text that a model processes or generates, and providers often charge different rates for input and output tokens. Cached inputs, tool calls, and other services can have separate charges. For example, OpenAI’s API pricing lists rates by model and distinguishes input, output, and certain tool charges. Check the relevant provider’s current pricing before making a decision.

A basic estimate is:

Monthly API cost = (input tokens × input price per token + output tokens × output price per token) + other charges

For self-hosting, count more than electricity. Total cost of ownership (TCO) is the full cost of buying and running the service over a chosen period. Include the GPU and host, plus electricity and cooling or facility costs. Account for engineering work and spare capacity for outages or traffic peaks. Divide those costs by the tokens your service delivers while meeting its response-time and quality requirements.

If you already have a GPU, calculate costs in two ways. The incremental-cost view counts the extra electricity and support needed to serve the workload on spare capacity. The full-cost view also accounts for the hardware’s purchase, replacement, and other uses. Use the first calculation to decide how to use spare capacity. Use the second to decide whether to buy GPUs instead of using an API.

A simple break-even estimate is:

Break-even token volume = monthly fixed self-hosting cost / (API cost per token − self-hosted variable cost per token)

This estimate works only when self-hosting’s variable cost per token is lower than the API price. Costs such as electricity rise with use; hardware purchases and some operating costs remain fixed or increase in steps. A detailed comparison should include both.

Traffic patterns change the cost per token

A GPU’s cost per hour does not reveal what each response costs. If you keep a GPU ready for requests but use it only occasionally, fewer tokens share its cost. Steady traffic or planned batch jobs spread that cost more widely. Utilization here means how much of the GPU’s capacity goes to useful work while it handles the traffic and meets your service requirements.

Throughput also depends on the workload. Long prompts require more computation and memory than short ones. When a service must answer quickly, it has less time to group requests together. Batching groups requests so a GPU can process them more efficiently. Users may wait longer because the service needs time to gather requests into each group. Count only the rate at which your service processes requests while meeting its response-time target.

A January 2026 study measured electricity use for selected models and configurations on consumer NVIDIA Blackwell GPUs. At a stated electricity rate of $0.12 per kilowatt-hour, it reported these costs for one Qwen3-8B quantization:

RAG means retrieval-augmented generation: the service finds relevant text and adds it to the model’s prompt. In this study, the RAG workload with 32K context used about 5.5 times as much electricity per token as the workload with 8K context. The study’s benchmark results show why a short-prompt test may not predict costs for long-context production requests.

Those figures cover GPU electricity for the tested setup, not full service costs. The study’s authors say their measurements exclude CPU, memory, and cooling overhead, and their synthetic workloads do not represent all the variation in production traffic. Treat the results as evidence that workload shape matters, not as a general cost-per-token promise.

Treat break-even examples as scenarios

The same study estimates that a $2,000 RTX 5090 would take about 292 days to pay back at 30 million tokens per day when compared with an API priced at about $0.23 per million tokens. Against a $15-per-million-token API comparator, it estimates about four days. At one million tokens per day against the lower-priced comparator, its estimate is 8,772 days.

These calculations describe specific scenarios; they do not set a threshold that applies to every service. The study compares prices using January 2026 assumptions. It calculates one API rate by giving input and output tokens equal weight. Its low-cost and frontier API comparators also represent different capability levels. The GPU’s short time to recover its cost against the more expensive API shows how much the result depends on which model you replace.

API pricing itself changes. Stanford’s 2025 AI Index reported that prices for models at a selected GPT-3.5-equivalent MMLU score declined over time. That historical trend does not predict future rates, but it supports testing a self-hosting plan against lower future API prices. The AI Index report provides the figures and their context.

Decide with your workload, not a benchmark headline

Self-hosting deserves a closer cost analysis when traffic is large and steady, batch jobs can use otherwise idle capacity, or your organization already has GPUs and the expertise to operate them. It also makes sense when keeping data in a controlled environment has value beyond reducing the bill. That value should be stated separately from the cost comparison.

An API is more likely to cost less when traffic is sparse or unpredictable, an inexpensive API model already meets the quality target, or the self-hosted service would need extra GPUs for peak demand and redundancy. Long prompts and strict response-time targets can also raise the hardware required. A 2025 analysis of inference economics explains why GPU price and theoretical computing capacity alone do not determine how quickly a GPU processes requests. Memory bandwidth, model size, batch size, and response-time limits also matter.

To compare the options, measure input and output tokens separately and record context lengths and traffic peaks. Then test the exact model and hardware with a realistic number of simultaneous requests. Check that the self-hosted model meets the same task-quality and latency requirements as the API. Finally, calculate both incremental and full ownership costs, including spare capacity and operating work. The ACL study on inference cost estimation also cautions that average-request estimates leave out factors such as peak demand, redundancy, and latency variation.

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com