- How inference cost appears on your invoices
- What drives inference cost
- Inference cost vs. training cost vs. infrastructure cost
- What inference cost means for your business budget
- How to reduce inference cost
- What's your company's inference cost?
Inference cost is the cost of running an AI model to generate a response. Every time your product, tool, or agent sends a request to an AI model and receives an output, inference is happening—and that inference has a cost.
For your business, inference cost shows up on invoices from AI providers like Anthropic, OpenAI, Google (Gemini), and managed cloud AI services like AWS Bedrock, Azure OpenAI, and Vertex AI. It's billed in tokens—the units AI models use to measure text. A single AI response, like drafting a short email or answering a support question, costs a fraction of a cent.
At scale, across hundreds of employees or thousands of automated tasks per day, those fractions add up fast. Across companies using AI in production, the average cost in April 2026 was $0.77 per million tokens, though how much AI tokens cost varies by more than 20× depending on which models a company uses.
Inference cost is distinct from training cost (paid once by the model provider) and cloud infrastructure cost (servers, storage, networking). It's the operating cost of AI—what you pay every time the model answers a question or completes a task.
How inference cost appears on your invoices
AI providers don't prominently use the word "inference" in their billing dashboards—they bill by token usage, model, and time period. But inference is what's happening behind every token charge.
Anthropic invoices: Charges broken out by model (Claude Haiku, Claude Sonnet, Claude Opus) and by input vs. output tokens.
Total invoice = (input tokens × input rate) + (output tokens × output rate) for each model used. Claude Opus—Anthropic's most capable model at the time—cost approximately $1.00 per million cached tokens in April 2026.
OpenAI invoices: Same structure—model, input tokens, output tokens. Different pricing tiers for GPT-5.4, GPT-5.4 mini, GPT-5.4 nano, and GPT-5.6. In April 2026, GPT-5-nano ran around $0.07 per million tokens, while GPT-5.5 was closer to $1.57 per million.
AWS Bedrock invoices: Model inference charges appear as line items within your AWS bill, under the Bedrock service. Listed by model (Claude, Llama, Titan) and by token type.
Azure OpenAI and Vertex AI: Inference charges appear within your existing cloud bill, filterable by AI service.
If you see charges from any of these sources, you're paying for inference. The charges are per-request (aggregated into monthly token totals), usage-based, and variable month-to-month.
What drives inference cost
Model tier—the biggest driver. Lightweight models (gpt-5.4-nano, Claude Haiku 4.5, GPT-5.6 Luna) cost $0.20–$1 per million input tokens (July 2026, per Anthropic and OpenAI).
- Standard models (Claude Sonnet 5, GPT-5.6 Terra) cost $2–$3 per million input tokens (July 2026).
- Premium flagship models (Claude Opus 4.8, GPT-5.6 Sol) cost around $5 per million input tokens (July 2026)—output tokens run 5–6× higher across all three tiers.
The same workload costs 5–25× more... Separately, Ramp's own April 2026 usage data showed premium models represented 45.8% of tokens consumed by businesses—but 55.9% of total cost.
Output tokens vs. input tokens. Output tokens cost 5–6× more than input tokens (public list pricing, July 2026, per Anthropic and OpenAI). A model generates output by producing tokens one at a time, which is more computationally expensive than processing input. Most business applications generate significantly more output than input, so the effective cost per request skews toward output pricing.
Volume. Inference cost scales linearly with usage—twice as many requests means twice the cost. Unlike seat-based software where adding users doesn't change your base bill, every AI interaction generates inference cost.
Context window size. Long-context requests—large documents, extended conversation history, large codebases—consume proportionally more tokens. A request that includes a 50-page document in context may cost 50–100× a simple one-line query.
Agentic loops. AI agents that take multi-step actions generate inference cost at every step. A task that requires 20 model calls to complete costs 20× a task that requires one.
This is why agent costs can scale unexpectedly. The top 5% of AI-spending companies spent roughly 80× the median company's inference cost, which is why AI agent spending controls matter as agent adoption grows.
Caching. Prompt caching reduces input token costs for applications with repeated context. Cache hit rates vary dramatically by workload—from below 20% for tasks that process unique documents to above 80% for tasks that reuse the same instructions or reference material.
The same model can cost 5× more or less depending on how requests are structured, per Ramp's own April 2026 data. Caching is an engineering choice, not a model property—one of several factors behind what drives AI token cost increases at scale.
Inference cost vs. training cost vs. infrastructure cost
Three costs are often confused:
| Cost type | What it is | Who pays it | When it occurs |
|---|---|---|---|
| Training cost | Cost to build and train the AI model | The model provider (Anthropic, OpenAI, Google) | Once, when the model is created |
| Inference cost | Cost to run the model to generate responses | Your business, via API or managed service | Every time the model processes a request |
| Infrastructure cost | Servers, storage, networking that run your application | Your business, via cloud provider | Ongoing, regardless of AI usage |
When Anthropic charges you for application programming interface (API) usage, that's inference cost. When AWS charges you for cloud servers running your application, that's infrastructure cost. When OpenAI spent hundreds of millions training a new model, that's training cost—already embedded in the per-token pricing you pay.
For budgeting purposes:
Inference cost is the one you control directly. You choose the model, the volume of requests, and the context window size.
Training cost is sunk (paid by the provider). Infrastructure cost is managed separately, much like COGS vs. operating expenses get tracked differently on your income statement.
What inference cost means for your business budget
If your AI use is internal (employee tools): Inference cost is operating expenses (OpEx)—a productivity investment that scales with headcount and usage patterns. The main budgeting challenge is visibility: Inference costs are distributed across multiple vendor invoices and can swing significantly month-to-month.
If your AI use is in the product: Inference cost is cost of goods sold (COGS)—it scales with customer usage and directly affects gross margins. A product where inference COGS exceeds 25–30% of revenue is under margin pressure. Model selection and prompt optimization are the primary levers.
If you're running AI agents: Inference cost has the highest variance and the highest risk. Agents run autonomously, make variable numbers of model calls per task, and can generate unexpected cost spikes. Spend limits and monitoring are essential.
How to reduce inference cost
Switch to lighter models where quality allows. This is the highest-ROI action most companies can take. Test whether a lightweight model (5–25× cheaper, per Anthropic and OpenAI) delivers acceptable quality for your specific use case.
For many tasks—summarization, classification, simple Q&A—it does. In April 2026, the cost difference was material: $1,000 bought approximately 14,144 million GPT-5-nano tokens, vs. approximately 639 million GPT-5.5 tokens—roughly a 22× difference.
Implement prompt caching. Anthropic and OpenAI offer discounts on repeated context. Cache hit rates above 80% are achievable for tasks that reuse the same instructions or reference material repeatedly. For tasks that process unique documents or images, rates below 20% are typical—reviewing how your AI requests are structured will tell you whether caching is worth pursuing.
Optimize prompt length. Shorter, more targeted requests reduce input tokens. Every unnecessary sentence sent to the model is a cost.
Set per-system spending limits. Prevents runaway cost from agent loops or unexpected volume spikes. Set the limit at 3× expected monthly cost to give headroom without being a permissive ceiling—one of several ways to reduce AI token costs without sacrificing quality.
What's your company's inference cost?
Knowing what inference is doesn't tell you what you're paying for it.
Most finance teams can't answer that question without digging through invoices from Anthropic, OpenAI, AWS, and Azure separately—none of which are labeled in a way that makes the totals obvious. Ramp’s AI Token Spend Management pulls all of it into one view: every inference charge, by provider, by model, by team, and by month. If you've been seeing AI charges you can't explain, this is where you start.
This data comes from aggregated and anonymized insights from Ramp’s Token Spend Management product, which gives companies daily visibility into AI usage and costs across providers and models.
FAQs
In most cases, yes. If you're accessing models through OpenAI's API, Anthropic's API, or a managed cloud AI service (Bedrock, Vertex, Azure OpenAI), your API bill is your inference cost.
Three reasons drive month-to-month swings: volume changes—more users, more usage, more requests. Context window changes—features that include more context in each request cost proportionally more. Model changes—if engineering switches to a more capable model, costs can change significantly overnight.
Switching to a different top model by cost is common and drives outsized swings for the companies that do it.
Token prices have dropped significantly since 2022—OpenAI's GPT-3 Davinci, the leading model at the time, cost around $20 per million tokens as a blended rate; today's lightweight models cost a small fraction of that. However, product teams typically adopt more capable (and more expensive) models as they mature, and agentic use cases consume tokens at much higher rates. Per-token costs fall—total inference spend tends to grow.
Compute cost is infrastructure—you're renting cloud servers (think AWS, Google Cloud, Azure). Inference cost is usage of a model the provider runs.
You don't manage the hardware. You pay per token consumed. Infrastructure is provisioned and relatively predictable, and inference is on-demand and directly tied to usage volume.
“Most banks treat the back office as a cost to keep down. We treat ours as a return to compound, which is why we run it on Ramp. Now we put our clients on Ramp, too.”
Patrick Gaughen
President & COO, Hingham Institution for Savings
“Browserbase builds infrastructure so AI agents can do real work. Ramp is doing the same for finance. It’s not another tool. It’s a system purpose-built for AI-driven finance, and that’s why we chose Ramp as our financial operating system from day one.”
Paul Klein IV
Founder & CEO, Browserbase
“We used to pay up to $20k a year for our AP platform. With Ramp, we’re earning back well over that amount. That's money that belongs to the mission now, not to the back-office software.”
Heidi Coffer
Chief Financial Officer, Boys & Girls Clubs of San Francisco
“The tricky thing about corporate travel policy is timing. We didn't need a stricter policy. We needed the policy to show up earlier. With Ramp Travel, it finally does.”
Keith Frantz
Director of Enterprise Risk Management, Prosper
“We're accountable to our funders, our partners, and the families we serve. That accountability starts with how we manage every dollar. Ramp makes it easy for our team to spend wisely, track in real time, and keep overhead low so more resources reach the families navigating infertility.”
Rachel Fruchtman
CFO, Jewish Fertility Foundation
“Each member of our team has an outsized impact due to our focus on using high-leverage tools like Ramp.”
Lauren Feeney
Controller, Perplexity
“With Ramp, we haven’t had to add accounting headcount to keep up with growth. The biggest takeaway is that instead of hiring our way through it, we fixed the workflow so we can keep supporting the organization as we scale.”
Melissa M.
VP of Accounting at Brandt Information Services
“In the public sector, every hour and every dollar belongs to the taxpayer. We can't afford to waste either. Ramp ensures we don't.”
Carly Ching
Finance Specialist, City of Ketchum
