- How inference cost appears on your invoices
- What drives inference cost
- Inference cost vs. training cost vs. infrastructure cost
- What inference cost means for your business budget
- How to reduce inference cost
- What's your company's inference cost?
Inference cost is the cost of running an AI model to generate a response. Every time your product, tool, or agent sends a request to an AI model and receives an output, inference is happening—and that inference has a cost.
For your business, inference cost shows up on invoices from AI providers like Anthropic, OpenAI, Google (Gemini), and managed cloud AI services like AWS Bedrock, Azure OpenAI, and Vertex AI. It's billed in tokens—the units AI models use to measure text. A single AI response, like drafting a short email or answering a support question, costs a fraction of a cent.
At scale, across hundreds of employees or thousands of automated tasks per day, those fractions add up fast. Across companies using AI in production, the average cost in April 2026 was $0.77 per million tokens, though how much AI tokens cost varies by more than 20× depending on which models a company uses.
Inference cost is distinct from training cost (paid once by the model provider) and cloud infrastructure cost (servers, storage, networking). It's the operating cost of AI—what you pay every time the model answers a question or completes a task.
How inference cost appears on your invoices
AI providers don't prominently use the word "inference" in their billing dashboards—they bill by token usage, model, and time period. But inference is what's happening behind every token charge.
Anthropic invoices: Charges broken out by model (Claude Haiku, Claude Sonnet, Claude Opus) and by input vs. output tokens.
Total invoice = (input tokens × input rate) + (output tokens × output rate) for each model used. Claude Opus—Anthropic's most capable model at the time—cost approximately $1.00 per million cached tokens in April 2026.
OpenAI invoices: Same structure—model, input tokens, output tokens. Different pricing tiers for GPT-5.4, GPT-5.4 mini, GPT-5.4 nano, and GPT-5.6. In April 2026, GPT-5-nano ran around $0.07 per million tokens, while GPT-5.5 was closer to $1.57 per million.
AWS Bedrock invoices: Model inference charges appear as line items within your AWS bill, under the Bedrock service. Listed by model (Claude, Llama, Titan) and by token type.
Azure OpenAI and Vertex AI: Inference charges appear within your existing cloud bill, filterable by AI service.
If you see charges from any of these sources, you're paying for inference. The charges are per-request (aggregated into monthly token totals), usage-based, and variable month-to-month.
What drives inference cost
Model tier—the biggest driver. Lightweight models (gpt-5.4-nano, Claude Haiku 4.5, GPT-5.6 Luna) cost $0.20–$1 per million input tokens (July 2026, per Anthropic and OpenAI).
- Standard models (Claude Sonnet 5, GPT-5.6 Terra) cost around $2 per million input tokens (July 2026).
- Premium flagship models (Claude Opus 4.8, GPT-5.6 Sol) cost around $4-$5 per million input tokens (July 2026)—output tokens run 5–6× higher across all three tiers.
The same workload costs 5–25× more. Separately, Ramp's own April 2026 usage data showed premium models represented 45.8% of tokens consumed by businesses—but 55.9% of total cost.
Output tokens vs. input tokens. Output tokens cost 5–6× more than input tokens (public list pricing, July 2026, per Anthropic and OpenAI). A model generates output by producing tokens one at a time, which is more computationally expensive than processing input. Most business applications generate significantly more output than input, so the effective cost per request skews toward output pricing.
Volume. Inference cost scales linearly with usage—twice as many requests means twice the cost. Unlike seat-based software where adding users doesn't change your base bill, every AI interaction generates inference cost.
Context window size. Long-context requests—large documents, extended conversation history, large codebases—consume proportionally more tokens. A request that includes a 50-page document in context may cost 50–100× a simple one-line query.
Agentic loops. AI agents that take multi-step actions generate inference cost at every step. A task that requires 20 model calls to complete costs 20× a task that requires one.
This is why agent costs can scale unexpectedly. The top 5% of AI-spending companies spent roughly 80× the median company's inference cost, which is why AI agent spending controls matter as agent adoption grows.
Caching. Prompt caching reduces input token costs for applications with repeated context. Cache hit rates vary dramatically by workload—from below 20% for tasks that process unique documents to above 80% for tasks that reuse the same instructions or reference material.
The same model can cost 5× more or less depending on how requests are structured, per Ramp's own April 2026 data. Caching is an engineering choice, not a model property—one of several factors behind what drives AI token cost increases at scale.
Inference cost vs. training cost vs. infrastructure cost
Three costs are often confused:
| Cost type | What it is | Who pays it | When it occurs |
|---|---|---|---|
| Training cost | Cost to build and train the AI model | The model provider (Anthropic, OpenAI, Google) | Once, when the model is created |
| Inference cost | Cost to run the model to generate responses | Your business, via API or managed service | Every time the model processes a request |
| Infrastructure cost | Servers, storage, networking that run your application | Your business, via cloud provider | Ongoing, regardless of AI usage |
When Anthropic charges you for application programming interface (API) usage, that's inference cost. When AWS charges you for cloud servers running your application, that's infrastructure cost. When OpenAI spent hundreds of millions training a new model, that's training cost—already embedded in the per-token pricing you pay.
For budgeting purposes:
Inference cost is the one you control directly. You choose the model, the volume of requests, and the context window size.
Training cost is sunk (paid by the provider). Infrastructure cost is managed separately, much like COGS vs. operating expenses get tracked differently on your income statement.
What inference cost means for your business budget
If your AI use is internal (employee tools): Inference cost is operating expenses (OpEx)—a productivity investment that scales with headcount and usage patterns. The main budgeting challenge is visibility: Inference costs are distributed across multiple vendor invoices and can swing significantly month-to-month.
If your AI use is in the product: Inference cost is cost of goods sold (COGS)—it scales with customer usage and directly affects gross margins. A product where inference COGS exceeds 25–30% of revenue is under margin pressure. Model selection and prompt optimization are the primary levers.
If you're running AI agents: Inference cost has the highest variance and the highest risk. Agents run autonomously, make variable numbers of model calls per task, and can generate unexpected cost spikes. Spend limits and monitoring are essential.
How to reduce inference cost
Switch to lighter models where quality allows. This is the highest-ROI action most companies can take. Test whether a lightweight model (5–25× cheaper, per Anthropic and OpenAI) delivers acceptable quality for your specific use case.
For many tasks—summarization, classification, simple Q&A—it does. In April 2026, the cost difference was material: $1,000 bought approximately 14,144 million GPT-5-nano tokens, vs. approximately 639 million GPT-5.5 tokens—roughly a 22× difference.
Implement prompt caching. Anthropic and OpenAI offer discounts on repeated context. Cache hit rates above 80% are achievable for tasks that reuse the same instructions or reference material repeatedly. For tasks that process unique documents or images, rates below 20% are typical—reviewing how your AI requests are structured will tell you whether caching is worth pursuing.
Optimize prompt length. Shorter, more targeted requests reduce input tokens. Every unnecessary sentence sent to the model is a cost.
Set per-system spending limits. Prevents runaway cost from agent loops or unexpected volume spikes. Set the limit at 3× expected monthly cost to give headroom without being a permissive ceiling—one of several ways to reduce AI token costs without sacrificing quality.
What's your company's inference cost?
Knowing what inference is doesn't tell you what you're paying for it.
Most finance teams can't answer that question without digging through invoices from Anthropic, OpenAI, AWS, and Azure separately—none of which are labeled in a way that makes the totals obvious. Ramp’s AI Token Spend Management pulls all of it into one view: every inference charge, by provider, by model, by team, and by month. If you've been seeing AI charges you can't explain, this is where you start.
This data comes from aggregated and anonymized insights from Ramp’s Token Spend Management product, which gives companies daily visibility into AI usage and costs across providers and models.
FAQs
In most cases, yes. If you're accessing models through OpenAI's API, Anthropic's API, or a managed cloud AI service (Bedrock, Vertex, Azure OpenAI), your API bill is your inference cost.
Three reasons drive month-to-month swings: volume changes—more users, more usage, more requests. Context window changes—features that include more context in each request cost proportionally more. Model changes—if engineering switches to a more capable model, costs can change significantly overnight.
Switching to a different top model by cost is common and drives outsized swings for the companies that do it.
Token prices have dropped significantly since 2022—OpenAI's GPT-3 Davinci, the leading model at the time, cost around $20 per million tokens as a blended rate; today's lightweight models cost a small fraction of that. However, product teams typically adopt more capable (and more expensive) models as they mature, and agentic use cases consume tokens at much higher rates. Per-token costs fall—total inference spend tends to grow.
Compute cost is infrastructure—you're renting cloud servers (think AWS, Google Cloud, Azure). Inference cost is usage of a model the provider runs.
You don't manage the hardware. You pay per token consumed. Infrastructure is provisioned and relatively predictable, and inference is on-demand and directly tied to usage volume.
“I assumed I would have to choose between speed and control. What I found is that you can have both. A well-designed system takes friction out, for the finance function and for everyone else.”
Justin Webster
CFO, Denver Broncos
“A well-run district should not have to choose between getting work done at the school site and keeping control of the dollars behind it. We're not hiring more people to do more jobs, so we have to be smarter about the process. With Ramp, the purchase, the receipt, and the record stay together from the start. ”
Nick Brizeno
Director of Purchasing, San Marcos Unified School District
“In senior living, scale only works if the communities still feel personal. We needed the back office to carry more of the complexity, not the people serving residents. Ramp helped us build that infrastructure, so the experience in the community could stay human.”
Ryan Cole
CFO, Agemark Senior Living
“AI is moving faster than the finance context around it. Prices change, models change, and the value is not always obvious from an invoice. We needed enough detail to know which bets deserved more investment — and which ones did not.”
Greg Cooley
Controller, AngelList
“Invoices, cards, tokens. The categories change but the principle doesn't: know where the money is going, remove the work around it, and make sure the spend is worth it.”
Maciej Mylik. Finance
ElevenLabs
“We weren’t trying to retrofit an old finance system. We had a blank canvas, and Ramp gave us the foundation to build a global finance function of the future.”
Justin Dourado
Director of Finance, Othership
“There's just no surprises anymore. No more waiting two months to find out how a job did. We know how it's doing as it's happening.”
Erich Kuss
Financial Systems Manager, Infinity Home Services
“More token spend isn’t proof that AI is working. Less isn’t proof that it isn’t. What matters is whether we’re buying the right level of intelligence for the work. Ramp lets us make that judgment in the same place we manage every other type of spend.”
Cody Nutt
Senior Director of Business Systems, Daxko
