Generative AI FinOps on Azure: understanding, measuring and holding token costs

1. Why generative AI breaks classic FinOps

Cloud FinOps was built on resources: a running virtual machine, an allocated disk, a sized cluster. You see them in the portal, you tag them, you turn them off. Generative AI bills something else: tokens, that is fragments of text going into and out of a model, at a price per million that depends on the model, the deployment type, the direction (input or output) and, increasingly, on cache usage. An Azure OpenAI resource with zero calls costs zero; the same resource can cost ten thousand euros the following month without any resource alert firing.

Three properties make the topic treacherous. Cost is driven by behaviour, not infrastructure: a 3,000-token system prompt repeated on every message costs more than the team that wrote it imagines. It is asymmetric: an output token often costs four times an input token, and reasoning models produce billed "invisible" tokens. And it is shared: without a gateway, every team draws on the same quota and the same bill, and nobody is accountable.

Rule #1: you cannot govern what you do not measure per team. Before any optimisation lever, set up token measurement per team and per application. Everything else follows.

2. The cost model: what is billed, and by whom

On Microsoft Foundry (which carries Azure OpenAI and the other models sold by Azure), the bill breaks down into five items. Exact prices change often; remember the structure rather than the figures.

The deployment type is the structuring pricing decision. Standard (pay as you go) comes in three scopes — global, data zone (Europe or United States) and regional — with prices and quotas rising as the scope narrows; Batch processes asynchronous jobs within 24 hours at about half the global standard price; Provisioned reserves capacity in PTUs at an hourly cost, reduced through a monthly or yearly reservation. The same model can therefore cost one to three times as much depending on how it is deployed.

3. Measure: tokens per team, independently of the code

Reliable measurement does not come from applications — each team would count its own way, or not at all — but from the gateway every call goes through. Azure API Management, as the AI gateway in front of Foundry, provides that accounting.

1. One API Management product per team and one subscription per application. This split becomes the allocation key: it has nothing to do with security (authentication is the Entra token), it is for metering.
2. The llm-emit-token-metric policy sends input, output, total and cached tokens of every call to Application Insights, with free dimensions: team, application, model, deployment type, user. This is the source of truth for showback.
3. On the Azure Cost Management side, tag Foundry, API Management, Redis and AI Search resources with the same "team" key when they are dedicated; for shared resources, allocate pro rata to the tokens measured by the gateway.
4. Build a dashboard that joins the two: tokens per team (gateway) and euros per team (bill, after allocation). The gap between them reveals fixed costs (under-used PTUs, Redis, observability) to be discussed separately.

<!-- Token metrics per team (product) and per application (subscription) -->
<llm-emit-token-metric namespace="ai-finops">
  <dimension name="Team"        value="@(context.Product.Name)" />
  <dimension name="Application" value="@(context.Subscription.Name)" />
  <dimension name="Model"       value="@(context.Request.MatchedParameters["deployment-id"])" />
  <dimension name="Deployment"  value="@(context.Request.Headers.GetValueOrDefault("x-deployment-type","standard"))" />
</llm-emit-token-metric>

For agents, add OpenTelemetry tracing: the GenAI semantic conventions (gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.request.model) let you drill down to a conversation or a tool call, and spot the looping agent or the system prompt that has grown obese.

4. The eight technical levers, from cheapest to most committing

Once measurement is in place, savings come from eight levers. The first ones are implemented in a day; the last ones commit the company for a year.

1. Shrink prompts: a system prompt cut from 3,000 to 800 tokens on an assistant handling 50,000 messages a day saves 110 million input tokens a month. Measure average prompt length per application; it is often the first item.
2. Pick the right model per task: "mini" or "nano" models cost 10 to 30 times less than frontier models and are enough for classification, extraction and summarisation. Route by task type, not by habit; API Management's unified model API lets you switch models without touching applications.
3. Use prompt caching: put stable content (instructions, tool definitions, reference documents) at the start of the prompt; cached tokens are billed at a fraction of the input price, and deducted 100% from utilisation on provisioned. Beware: recent models may charge cache writes depending on the retention chosen — structure prompts to favour reads.
4. Enable semantic caching at the gateway: the llm-semantic-cache-lookup and -store policies serve an already produced answer when a prompt is semantically close to a previous one. On an internal FAQ, 20 to 40% of calls no longer reach the model.
5. Move non-interactive work to Batch: catalogue enrichment, document summaries, bulk classification — anything that tolerates a 24-hour delay costs half price in global or data zone Batch, with a separate quota that does not disturb interactive traffic.
6. Choose the deployment scope: global standard is the cheapest; the Europe data zone satisfies most residency requirements for a moderate premium; regional is only justified by an explicit regulatory constraint.
7. Go provisioned when load is predictable: above a certain regular volume, reserved capacity (PTU) is cheaper than pay as you go and guarantees latency. Size with the Foundry calculator from actually measured tokens, deploy first, then reserve.
8. Reserve: the Azure reservation on PTUs (1 month or 1 year, scoped to a subscription, management group or billing account) applies a significant discount on the hourly rate. It does not reserve capacity, it guarantees a price: create deployments before buying the reservation to avoid over-buying.

Recommended cost architecture for a mature platform: a base of reserved PTUs sized on baseline traffic, overflow to global or data zone standard through an API Management backend pool with circuit breaker, and Batch for everything non-interactive. That is the profile that minimises cost per token without degrading latency.

5. Hold: enforceable quotas and caps per team

Measuring and optimising are not enough if one team can, through a bug or enthusiasm, consume the others' budget. Caps must be technical, not contractual.

• API Management's llm-token-limit policy sets, per product (team), a tokens-per-minute rate and a quota over a period (day, week, month). When the quota is reached, calls return 429: the team comes to talk, the budget is intact.
• Foundry quotas (tokens per minute per model, region and subscription) remain the limit of last resort; distribute them deliberately across deployments rather than letting first come take all.
• Azure Cost Management budgets, per subscription and resource group, with alerts at 50, 80 and 100% sent to the team lead and FinOps, close the loop on the bill side.
• For agents, also limit the number of steps and tool calls per session: a looping agent burns tokens on every iteration, and no budget alert catches it in real time.

<!-- Per-team cap: 50,000 tokens/min and 20 million tokens per month -->
<llm-token-limit counter-key="@(context.Product.Id)"
                 tokens-per-minute="50000"
                 token-quota="20000000"
                 token-quota-period="Monthly"
                 estimate-prompt-tokens="true"
                 remaining-tokens-variable-name="remainingTokens"
                 remaining-tokens-header-name="x-tokens-remaining" />

6. Allocate and forecast: showback, chargeback and the forecasting model

Showback — showing each team what it consumes — is often enough to cut the bill by 20 to 30% in a quarter, without any technical optimisation: teams discover their system prompt, their oversized model, their forgotten night job. Chargeback — actually billing back — comes later, once measurement is stable and accepted.

• Monthly report per team: input, output and cached tokens, estimated cost at list price, share of fixed platform costs, three-month trend, top 3 applications.
• Business unit cost: bring tokens down to the unit the business understands — cost per conversation, per processed document, per resolved ticket, per active user. That figure decides whether a use case is profitable, not the token total.
• Forecasting model: for each use case, estimate average tokens per interaction × interactions per user × users × model price. A spreadsheet is enough; update it with real measurements after the pilot. Most gaps come from input tokens (RAG context longer than planned) and conversation retries.
• Provisioned threshold: compute the point where the hourly cost of reserved PTUs falls below the pay-as-you-go cost of your baseline traffic. Below it, stay standard; above it, reserve the base and overflow.

7. The monthly FinOps loop in five steps

AI FinOps is not a project but a routine. One hour per month per team, three hours for the platform.

1. Read: the showback report per team and the platform dashboard (tokens, costs, cache hit rate, PTU utilisation, 429 errors).
2. Explain: every variation above 15% has a cause — new use case, modified prompt, model change, traffic. The team explains it in one sentence in the report.
3. Decide: adjust quotas, move a job to Batch, change a model, resize or renew a reservation, retire an unused agent.
4. Apply: decisions become infrastructure-as-code changes (API Management policies, Foundry deployments, budgets), reviewed and deployed like any other change.
5. Verify: the following month, the effect of each decision is measured. What produced no effect is reverted.

8. The five mistakes I see most often

• Buying PTUs before measuring. The reservation guarantees a price, not usage; without a known baseline, you pay for empty capacity. Deploy standard, measure three months, then reserve.
• One global quota for everyone. The first batch job blocks interactive applications, and nobody owns the bill. One product per team, one quota per product.
• Ignoring output and reasoning tokens. Dashboards that only count prompts underestimate cost by a factor of two to four on reasoning models.
• Optimising the model before the prompt. Changing model saves 30%; cutting a redundant system prompt and enabling caching saves 60% without changing quality.
• Confusing data residency with region. Many "data in Europe" requirements are met by the Europe data zone, markedly cheaper and better provisioned with quota than regional deployment.

9. Where to start

With measurement, in two weeks: an API Management gateway in front of Foundry if there is none, one product per team, the token metrics policy, a dashboard. At the first showback report, three quarters of the savings appear by themselves: prompts to cut, models to downgrade, jobs to move to Batch. Quotas follow the next month, once teams know their normal consumption. Provisioned and reservations are decided in month three, on real figures.

If you want an order of magnitude of what your platform could save, that is the kind of calculation done in thirty minutes from a bill export.