Three months earlier, a fintech company launched a small AI assistant for relationship managers. The pilot was successful; 50 internal users asked questions about policies, product eligibility, and customer documentation. The first month's cloud services bill was reasonable, the feedback was positive, and leadership asked: "Can we roll this out to the whole business?"
Three months later, the assistant is no longer a pilot. It now supports thousands of employees, summarizes long documents, drafts client emails, and answers operational questions through a retrieval-augmented generation system. Usage rises every week, and the cloud bill grows faster than expected.
The CFO asks the obvious question about cost, and someone in architecture says:
"Shouldn't we just run this on-prem? Wouldn't that be cheaper?"
It is a fair question, but the real question is not whether on-premises infrastructure is cheaper. The real question is what kind of AI workload you are running, at what utilization, with what quality requirements, and under what operating constraints.
That distinction matters because LLM economics are very different from traditional ones.
Enterprises have three options, each with a different cost model:
-
Managed API (e.g., Amazon Bedrock, Azure OpenAI, Vertex AI)
You call a hosted model and pay per token(input and output). You get fast access to frontier models like Claude; you get managed scaling, enterprise security, and minimal operational overhead. Also, this is the only way to use proprietary models. -
Self-hosted on cloud GPUs
You run open-weight models like Llama, Mistral, Qwen, and Gemma on rented GPUs on AWS, Azure, GCP or other cloud providers. You pay for GPU hours, storage, and networking, plus the engineering effort to deploy, monitor, scale, and optimize the serving stack. -
On-premises
The most capital-intensive option. You buy the GPU servers, operate the data center, manage the hardware lifecycle, and run the model-serving infrastructure yourself. You can run open-weight models only. Proprietary models like Claude are not available to run on premises.
Before comparing costs, one rule shapes the entire conversation. You cannot run Claude or GPT frontier models on-premise. These models are only accessible through their vendors' APIs and licensed cloud partners like Amazon Bedrock, Vertex AI, and Azure AI Foundry. Frontier proprietary models still hold a measurable lead over open-weight alternatives on reasoning, instruction-following, and agentic tasks, which is why they remain the default choice for high-stakes workloads.
This means "Managed LLM API vs. on-prem" is a false comparison. The real choice is between managed proprietary models (Claude, GPT, Gemini) and self-hosted open-weight models (Llama, Mistral, Qwen).
Our Fintech's bill shock isn't unique. AI cost overruns are an industry-wide pattern. IDC predicts Global 1,000 enterprises will underestimate AI infrastructure costs by 30% through 2027. Gartner projects global AI spending will reach $2.5 trillion in 2026, with $401 billion of that going to infrastructure.
The cost driver is not the platform. It is the volume of tokens flowing through your system that is driven by prompt size, retrieved context, agent loops, and evaluations. These grow no matter where the model runs.
Let's anchor on 100 million tokens per month, a realistic mid-market workload, like a customer-service copilot handling 100,000 conversations or an internal AI assistant serving a few hundred employees.
To match Claude's quality with an open model, you need something in the Llama 70B class, which in turn needs a lot of compute and GPU power.
| Option | Monthly Cost |
|---|---|
| Amazon Bedrock Claude Sonnet 4.5 | ~$660 (pay-per-use, ~70% input / 30% output tokens) |
| AWS p5.48xlarge (1-year Savings Plan) | ~$22,600 (running 24/7) |
| Dell PowerEdge XE9680 (on-prem) | ~$22,000 (monthly equivalent) |
The on-prem figure is a monthly run rate, not a purchase price. A configured XE9680 server costs roughly $350K–$500K upfront; spread over a 3-year lifecycle and combined with power, cooling, colocation, networking, and the engineer who keeps it running, the realistic monthly cost is $22K–$30K per box.
At this volume, Amazon Bedrock is roughly 30× cheaper than either hardware option.
The reason: GPU rentals and on-prem servers bill for capacity, not usage.
| Monthly Volume | Amazon Bedrock | AWS p5.48xlarge | On-prem |
|---|---|---|---|
| 1B tokens | $6,600 | $22,600 | $22,000+ |
| 3B tokens | $19,800 | $22,600 | $22,000+ |
| 3.4B tokens | $22,440 ← break-even | $22,600 | $22,000+ |
| 5B tokens | $33,000 | $22,600 | $22,000+ |
The break-even with hardware sits around 3.4 billion tokens per month.
The Verdict:
- • Below 1B tokens/month → Amazon Bedrock wins by 5 – 30x.
- • 1B to 3B tokens/month → Amazon Bedrock still wins. Optimize (caching, routing, batching) before switching.
- • 3B to 10B tokens/month → Cloud GPUs become viable if traffic is steady.
- • Above 10B tokens/month → Self-hosting wins on cost through cloud GPUs for flexibility, or on-prem when sovereignty, latency, or long-term ownership matter more.
We use Amazon Bedrock and AWS hardware in this comparison, but the structural economics are the same across the major cloud providers (Azure, GCP etc).
Most enterprises run purely on Managed LLM APIs (e.g., Amazon Bedrock, Azure OpenAI, and Google Vertex AI) in production. If a single managed API meets your quality, latency, and cost requirements, adding self-hosted GPUs only adds operational surface area. Managed LLM APIs like Amazon Bedrock, Azure OpenAI, and Vertex AI are the right architecture choices for the vast majority of AI workloads.
For most teams, the bigger wins come from optimizing within Managed LLM APIs:
- • Prompt caching discounts cached input tokens by up to 90%, the single largest lever and the most underused.
- • Semantic caching sits one layer above, matching by meaning, typically eliminating approx. 30% of repeated queries.
- • Model routing sends each request to a cheap model first (Haiku, Nova Lite) and escalates only when needed.
- • Provisioned Throughput locks in predictable capacity for steady, high-volume workloads.
Applied together, these typically cut costs by 50–70% without changing platforms, which is the comparison most "should we leave Cloud Managed LLM APIs?" debates skip entirely.
Before choosing where to run, answer three questions:
- • Does your use case need Claude or GPT-class reasoning, or will a smaller open model do?
- • Is your traffic steady and high-volume, or bursty and unpredictable?
- • Do regulations, sovereignty, or air-gap requirements force your data to stay on-premises?
Your answers map to one of three paths.
Stay on a Managed LLM API like Amazon Bedrock, Azure OpenAI, or Vertex AI if your workload needs frontier reasoning, has bursty or unpredictable traffic, or sits below a few billion tokens a month. This covers roughly 80% of enterprise AI today. Cost optimization comes from optimization (caching, routing, batching), not platform changes.
Self-host open-weight models on cloud GPUs like Llama, Mistral, and Qwen when a specific task is high-volume, steady, and an open model handles it well. The break-even with managed APIs sits around 3 billion tokens per month, and only when GPU utilization stays high. Cloud GPUs are usually the right form of self-hosting if you do not want to procure hardware and put upfront capex.
Move to on-premises when sovereignty, air-gap, or long-term ownership preferences are required or when sustained inference volume genuinely justifies owning the hardware. The cost argument is rarely the deciding factor. The decision is strategic, not financial.
Our fintech didn't need to leave Cloud Managed LLM APIs. After enabling prompt caching, routing simpler requests to smaller models like Claude Haiku, and moving document summarization to batch inference, the team brought the bill down by more than 60% in the next quarter without changing platforms.
The hardest part of getting AI economics right isn't choosing the platform. It's the discipline behind it, knowing where every token is spent, designing the routing layer that protects your bill, tuning prompt and semantic caching, building RAG pipelines that retrieve concise and relevant results, and putting observability in place so cost stays predictable as usage scales.
This is where Zero&One comes in as an AWS Premier Tier Services Partner. We help enterprises design, deploy, and operate AI architectures that are fast to ship, cost-efficient to run, and ready for production scale. Our work covers the parts that matter most, including choosing the right model for each workload, deciding where to run it, optimizing cost and performance, and operating it day-to-day so your team can stay focused on the business outcomes that brought you to AI in the first place.
