On-Premises LLMs: When Do They Pay Off?

On-premises large language models (LLMs) start to pay off when your organization has sustained, high-volume inference demand, strict data-residency or privacy requirements, and the engineering capacity to manage the infrastructure. For most teams, the break-even point is somewhere between 1 million and 10 million tokens per day, depending on model size, hardware costs, and utilization. This guide explains the cost and capacity math, the hidden trade-offs, and how to decide if self-hosting is right for you.

Formula

Daily GPU cost = GPUs required × GPU cost per hour × hours per day

Tokens per GPU per day = (Daily tokens × 1,000,000) ÷ GPUs required

Cost per million tokens = Daily GPU cost ÷ Daily tokens (in millions)

Where:

• Daily tokens – total tokens processed per day (input + output), in millions.
• GPU cost per hour – fully loaded cost of one GPU (hardware amortization, power, cooling, space, maintenance) per hour.
• GPUs required – number of GPUs needed to serve the model at your target latency and throughput.
• GPU utilization – percentage of time GPUs are actively processing requests (not idle).
• Hours per day – how many hours the system runs (usually 24).

Worked Example

Using the default inputs: 5 million tokens per day, $2.50 per GPU-hour, 4 GPUs, 70% utilization, 24 hours.

1. Daily GPU cost = 4 × $2.50 × 24 = $240
2. Tokens per GPU per day = 5,000,000 ÷ 4 = 1,250,000 tokens
3. Cost per million tokens = $240 ÷ 5 = $48

So with these assumptions, you’d pay $48 per million tokens. Compare that to a typical API price of $3–$15 per million tokens for a similar-sized model, and on-prem doesn’t look cost-effective at this scale. But if you double the daily tokens to 10 million, the cost per million drops to $24, and at 50 million it’s under $5.

When On-Prem LLMs Make Financial Sense

The core economic advantage of on-prem is that you pay fixed costs (hardware, electricity, staff) regardless of usage, while APIs charge per token. That means on-prem becomes cheaper once your volume is high enough to amortize the fixed costs. The break-even point depends on:

• Model size – a 7B-parameter model can run on a single consumer GPU; a 70B model needs 4–8 high-end GPUs. Larger models have higher fixed costs but also higher API prices, so they can break even at lower volumes.
• Hardware cost – a high-end GPU (e.g., A100, H100) costs $10,000–$30,000 to purchase, plus power and cooling. Renting dedicated servers changes the math.
• Utilization – if your usage is bursty (e.g., 9-to-5 business hours), you’ll waste capacity at night. APIs let you pay only for what you use.
• API pricing – commercial APIs charge anywhere from $0.15 to $60+ per million tokens depending on model and provider. Higher API prices make on-prem more attractive.

A rough rule of thumb: if you’re processing more than 10 million tokens per day consistently, on-prem is often worth investigating. Below that, APIs are usually cheaper unless you have regulatory or security reasons.

Capacity Planning: How Many GPUs Do You Need?

GPU requirements depend on model size, quantization, batch size, and latency targets. A 7B model in 4-bit quantization can run on a single 24GB GPU and serve maybe 500–2,000 tokens per second with batching. A 70B model needs 2–4 GPUs just to fit in memory, and can serve 1,000–5,000 tokens per second with optimized inference engines.

To estimate capacity:

1. Determine your peak tokens per second (TPS) requirement. For example, if you expect 100 concurrent users each generating 10 tokens per second, that’s 1,000 TPS.
2. Find the throughput per GPU for your chosen model (often available from vendor benchmarks or your own tests).
3. Divide required TPS by per-GPU TPS, then round up for headroom.

Remember that throughput drops with longer prompts and larger batch sizes, and that you need redundancy for failover. A common mistake is to size for average load instead of peak, leading to slow responses during spikes.

Hidden Costs Beyond the GPUs

The hardware is only part of the total cost of ownership. On-prem also includes:

• Electricity – a single A100 can draw 400W under load; a rack of 8 draws 3.2kW just for GPUs, plus cooling. At $0.15/kWh, that’s about $3,500/year per GPU.
• Storage and networking – model weights can be 50–150GB; you need fast NVMe storage and high-bandwidth interconnects (e.g., InfiniBand) for multi-GPU inference.
• Engineering time – someone must set up the inference server, monitor it, update models, and fix issues. This is often 0.5–2 full-time engineers.
• Downtime and maintenance – hardware fails, software bugs happen, and you need to handle them yourself.
• License and support – if you use commercial models or enterprise support, those add costs.

These hidden costs can easily double the apparent hardware cost. A realistic fully-loaded cost per GPU-hour is $2–$5 for high-end GPUs, not just the purchase price divided by hours.

When On-Prem Is the Only Option

Cost isn’t the only factor. Some organizations cannot use cloud APIs due to:

• Data privacy regulations – healthcare, finance, and government data may not be allowed to leave the premises.
• Security policies – even if legal, sending proprietary code or customer data to a third party may be unacceptable.
• Air-gapped environments – some systems have no internet connection at all.
• Low-latency requirements – if you need sub-10ms responses, the network round-trip to a cloud API may be too slow.
• Customization and control – fine-tuning a model on your own data may require access to weights, which some APIs don’t allow.

In these cases, on-prem is not a cost decision – it’s a compliance or technical necessity. The cost guide still helps you budget, but the choice is already made.

Alternatives to Full On-Prem

If you’re not ready to commit to buying hardware, consider hybrid options:

• Dedicated cloud instances – rent a single-tenant GPU server from a cloud provider. You get isolation and control without buying hardware, and you can shut it down when not needed.
• Serverless inference – some providers offer per-second billing for GPU instances, which can be cheaper than per-token APIs for moderate usage.
• Smaller models – a 7B model might meet your needs at a fraction of the cost of a 70B. Distilled models can be surprisingly capable.
• Hybrid routing – use on-prem for high-volume, privacy-sensitive workloads, and route overflow to APIs during spikes.

Start with APIs to validate your use case, then measure actual token volume and latency needs. Only then run the cost model with real numbers.

Common Mistakes and How to Avoid Them

• Ignoring utilization – if your GPUs sit idle 60% of the time, your effective cost per token doubles. Consider sharing the infrastructure across teams or using spot instances for non-critical jobs.
• Underestimating engineering effort – deploying an LLM is not like installing a database. You need to handle model serving, batching, caching, and monitoring. Budget for at least one dedicated engineer.
• Forgetting about model updates – you’ll want to upgrade to newer models, which means re-benchmarking and possibly buying more hardware.
• Only counting GPU cost – as shown above, power, cooling, storage, and staff can double your costs.
• Not testing with your own workloads – vendor benchmarks are often optimistic. Run your own prompts and measure throughput and latency.

Frequently Asked Questions

How many tokens per day do I need to justify on-prem?

There’s no universal number, but a common rule of thumb is 10 million tokens per day or more. At that volume, a 70B model on 4 GPUs might cost $10–$20 per million tokens on-prem versus $30–$60 via API. Below 1 million tokens per day, APIs are almost always cheaper.

What’s the cheapest way to start with on-prem LLMs?

Start with a small model (7B) on a single consumer GPU (e.g., RTX 4090) using quantized weights. This can cost under $2,000 in hardware and serve hundreds of thousands of tokens per day. It’s a good way to learn and measure your actual needs before scaling up.

Can I use on-prem LLMs for real-time chat?

Yes, but you need to plan for latency. A 7B model on a good GPU can generate 50–100 tokens per second, which is fine for chat. Larger models may need multiple GPUs and optimized inference (e.g., vLLM, TensorRT-LLM) to keep latency under 1 second per response.

How do I estimate the number of GPUs for my workload?

First, measure your peak tokens per second. Then look up the throughput of your chosen model on a single GPU (from benchmarks or your own tests). Divide the required TPS by per-GPU TPS and round up. Add 20–30% headroom for spikes and failover.

Is it cheaper to rent dedicated servers than to buy?

For short-term projects, renting is cheaper because you avoid capital expenditure. For long-term, high-utilization workloads, buying can be cheaper after 1–2 years. Use the calculator above to compare your specific scenario.

What about fine-tuning? Does that change the cost?

Fine-tuning adds a one-time cost for training, which requires more GPUs and time. However, a fine-tuned smaller model can sometimes replace a larger base model, reducing inference costs. Factor in the training cost but spread it over your expected inference volume.

Related

• Game Dev Crunch vs Sustainable Production: What Studios Can Change
• AI Agents Build Safer Cyber Defenses with Guardrails?
• Micro Vacuums: What to Check Before Choosing a Model

in t | Daily tokens (millions) | - | 5 | 0
in d | GPU cost per hour (USD) | - | 2.5 | 0
in g | GPUs required | - | 4 | 1
in u | GPU utilization (%) | - | 70 | 0
in h | Hours per day | - | 24 | 1
out c | Daily GPU cost | USD | g * d * h | 2
out t | Tokens per GPU per day | millions | (t * 1000000) / g | 2
out p | Cost per million tokens | USD | c / t | 2