Taking an AI agent platform to production on Azure with Microsoft Foundry and API Management

1. Why a platform, not a collection of projects

The first wave of GenAI projects was built team by team: each product created its own Azure OpenAI resource, generated an API key, stored it in an environment variable and wired up a prompt. Six months later the IT department finds twenty resources across twelve subscriptions, keys floating around Git repositories, no idea of the cost per team, and no way to cut off a runaway usage. Then the CISO asks the killer question: "Who can call which model with which data?" Nobody can answer.

An AI platform answers that question by construction. It centralises five things every project was redoing badly: identity (who calls), the gateway (where calls go through), models and agents (what is called), tools (what agents are allowed to do) and observability (what happened and what it cost). Each team keeps the freedom of its use cases; the platform sets the rules once.

Design rule #1: no API key, end to end. If a secret exists anywhere, it will end up in a repository, a ticket or a log file. Microsoft Entra identity replaces the key at every link: user, agent, gateway, model, tool.

2. The five-layer reference architecture

The architecture fits in five layers, each carried by an identified Azure service. Arrows always point the same way: a call enters through identity, crosses the gateway, reaches a Foundry model or agent, which only accesses tools through the gateway, and everything is traced.

The following sections detail each layer with the decisions to make and, where useful, the exact configuration.

3. Layer 1 — Key-less identity

Everything starts with disabling local (key-based) authentication on Foundry and Azure OpenAI resources. As long as a key works, someone will use it. Once the key is disabled, only Entra ID tokens get through, and every call is attributable to an identity.

• Users: OIDC authentication through Entra ID, with Conditional Access (MFA, compliant device) like any enterprise application. The agent receives a token on behalf of the user and can propagate it to tools (on-behalf-of) when the tool must enforce the person's rights.
• Agents and services: one managed identity per agent or workload. On AKS, Workload Identity federates the Kubernetes service account with an Entra identity: the pod gets a token without any secret mounted in the container.
• Gateway: API Management authenticates to Foundry with its own managed identity. Consumers authenticate to API Management with an Entra token validated by the validate-azure-ad-token policy; the APIM subscription key is for metering, not security.
• Foundry roles: assign the dedicated roles with least privilege. Foundry Agent Consumer for an application that calls an agent, Foundry User for a team that builds, Project Manager for a project owner. At account, project or agent scope depending on the perimeter.

Validation test: delete every key from your vault and redeploy. If anything breaks, you have found a debt to clear before going to production.

4. Layer 2 — API Management as the AI gateway

Azure API Management has become a real AI gateway: it imports a Microsoft Foundry resource directly (Azure OpenAI models or other providers), connects to it with a managed identity and applies policies specific to large language models. This is where quotas, caching, content filtering and load balancing are decided, for all teams at once.

Four policies structure the layer. They apply at the level of an API, a product (a team, a department) or an operation.

1. llm-token-limit: caps tokens per minute or per period (hour, day, month) on any counter key — subscription, product, calling identity. This is the FinOps tool: a team has its quota and cannot consume the others'. Prompt token estimation on the gateway side rejects oversized requests before they even reach the model.
2. llm-emit-token-metric: sends prompt, completion and total tokens to Application Insights, with custom dimensions (team, application, model, user). Without this policy you will never know who spent what.
3. llm-semantic-cache-lookup and llm-semantic-cache-store: cache responses by semantic similarity of the prompt (embeddings + Azure Managed Redis). On the repetitive questions of an internal assistant the cache hit rate often exceeds 20%, that many tokens not billed and answers served instantly.
4. llm-content-safety: forwards the prompt to Azure AI Content Safety before sending it to the model, and blocks the categories you define (hate, violence, self-harm, sexual content, jailbreak via Prompt Shields). The filter runs before the model, whatever the provider.

<policies>
  <inbound>
    <base />
    <!-- The consumer authenticates with an Entra ID token, not a key -->
    <validate-azure-ad-token tenant-id="{{tenant-id}}">
      <audiences><audience>api://ai-platform</audience></audiences>
    </validate-azure-ad-token>

    <!-- Token quota per APIM product (= per team): 50,000 TPM, 20M / month -->
    <llm-token-limit counter-key="@(context.Product.Id)"
                     tokens-per-minute="50000"
                     token-quota="20000000" token-quota-period="Monthly"
                     estimate-prompt-tokens="true"
                     remaining-tokens-variable-name="remainingTokens" />

    <!-- Content filtering before the model -->
    <llm-content-safety backend-id="content-safety" shield-prompt="true">
      <categories output-type="EightSeverityLevels">
        <category name="Hate" threshold="4" />
        <category name="Violence" threshold="4" />
      </categories>
    </llm-content-safety>

    <!-- Semantic cache: similar answers served without calling the model -->
    <llm-semantic-cache-lookup score-threshold="0.05"
                               embeddings-backend-id="embeddings"
                               embeddings-backend-auth="system-assigned" />

    <!-- Token metrics to Application Insights, per team and application -->
    <llm-emit-token-metric namespace="ai-platform">
      <dimension name="Team" value="@(context.Product.Name)" />
      <dimension name="Application" value="@(context.Subscription.Name)" />
      <dimension name="Model" value="@(context.Request.MatchedParameters["deployment-id"])" />
    </llm-emit-token-metric>

    <!-- Gateway managed identity to Foundry: no key -->
    <authentication-managed-identity resource="https://cognitiveservices.azure.com" />
    <set-backend-service backend-id="foundry-pool" />
  </inbound>
  <outbound>
    <base />
    <llm-semantic-cache-store duration="300" />
  </outbound>
</policies>

Two additions are worth it. Backend pools with circuit breakers spread calls across several deployments — a region with reserved capacity (PTU) first, pay-as-you-go as overflow — and fail over automatically when a deployment returns 429 errors. The unified model API (preview) exposes several providers behind a single entry point and translates formats, so you can change model without touching applications.

Since late 2025 the AI gateway also integrates directly into the Microsoft Foundry portal (preview): per-deployment quotas, agent and MCP tool registration are done from Foundry, with the API Management instance still available for advanced policies. That is where Microsoft is heading: Foundry for the control plane, API Management for the data plane.

5. Layer 3 — Microsoft Foundry: models, agents and guardrails

Microsoft Foundry (formerly Azure AI Foundry) is the control plane of the models-and-agents layer. Three decisions are made there once and for all.

• The model catalogue open to each project. Foundry offers Azure OpenAI, Anthropic, Mistral, Meta, Cohere and others; the deployment type (standard, provisioned, batch, managed compute) sets the cost / latency / capacity-guarantee trade-off. Decide per project which models are allowed, and block the rest with RBAC rather than convention.
• Agent hosting in Foundry Agent Service: container or source code (Agent Framework, LangGraph, Semantic Kernel or custom code), a dedicated Entra identity per agent, its own endpoint, a per-session isolated sandbox, immutable versions and traces in Application Insights. The agent becomes a governed object, with an owner, a version and a log.
• Guardrails attached to deployments: input and output content filters, Prompt Shields against injections, protected material and personal data detection, and custom blocklists using exact match or regular expressions (up to 10,000 terms per list). A regex blocklist on card number, IBAN or internal employee ID formats costs an hour and prevents a leak.

The rule is to double the guardrails: Content Safety at the gateway for everything coming in, Foundry filters on each deployment for what goes out. The two layers do not see the same things and do not fail at the same time.

Finally, the Foundry control plane provides the inventory: the list of agents and models across the company, their consumption, their owner. That is the document the CISO was asking for, updated in real time.

6. Layer 4 — Governed MCP tools, not rogue connectors

A useful agent acts: it reads a Jira ticket, queries a database, creates a SharePoint page. The Model Context Protocol (MCP) standardises this access to tools, but an MCP server wired directly to an agent amounts to giving an LLM unfiltered network access. The solution is the same as for models: everything goes through the gateway.

• Expose your existing REST APIs as MCP servers from API Management, without rewriting code: the gateway translates OpenAPI operations into MCP tools, with the usual policies (authentication, throttling, IP filtering, logging).
• Route existing MCP servers (Jira, Azure DevOps, SharePoint, your own) through API Management: the endpoint the agent sees is the gateway's, never the server's. Authentication uses OAuth 2.0 or user identity passthrough, depending on whether the tool must act on behalf of the platform or the person.
• Authorise per tool, not per server: a support agent may read tickets, not delete them. The policy is written once in the gateway and applies to every agent using the tool.
• Register tools in the Foundry inventory so they are discoverable by teams and governed centrally, instead of being recreated in every project.

On a Kubernetes platform the same role can be played by a dedicated agent gateway (agentgateway, the CNCF project backed by Solo.io, with kagent to declare agents and kmcp for MCP servers). That is the architecture I implemented on AKS for a platform serving about 3,000 employees: the user's JWT is validated at the gateway, access is authorised tool by tool, and the agent's pod carries a Workload Identity with no secret at all. API Management and agentgateway cover the same need; the choice depends on where your agents live — Foundry Agent Service or your own clusters.

7. Layer 5 — Observability and token FinOps

A platform you do not measure is a platform the CFO will eventually shut down. Three data flows are enough, provided you set them up from the first agent.

1. Traces: agents emit OpenTelemetry spans following the GenAI semantic conventions (gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, tools called, durations). Foundry Agent Service produces them natively; your own agents use the OpenTelemetry SDK. Destination: Application Insights, or Azure Monitor managed Prometheus and Grafana on AKS.
2. Token metrics: the gateway's llm-emit-token-metric policy provides the accounting truth, independent of agent code, with team, application and model dimensions. Showback and alerts are built on it.
3. Costs: Azure budgets per subscription and resource group, alerts at 80% and 100%, and a dashboard that joins tokens consumed and the actual bill per team. With reserved capacity (PTU), the dashboard also tracks utilisation to size the reservation.

AI FinOps fits in a monthly loop: cost review per team, quota adjustment in the gateway, reserved vs pay-as-you-go decision, and retirement of unused agents. Layer 2 quotas make the decision enforceable without a meeting: when a team hits its cap, its calls return 429 and it comes to talk.

8. The go-live checklist

Twelve boxes to tick before opening the platform beyond the pilot team. If one is missing, you will pay for it as an incident.

1. Local authentication disabled on every Foundry and Azure OpenAI resource; no key in circulation.
2. Private network access: Private Endpoints on Foundry, API Management, Redis and storage; no public endpoint on models.
3. Managed identity or Workload Identity for every agent and for the gateway; Foundry roles at least privilege.
4. Token quotas per team in the gateway, with a monthly quota on top of the per-minute rate.
5. Token metrics with team / application / model dimensions, and a showback dashboard shared with budget owners.
6. Content filters at the gateway and on every deployment, Prompt Shields on, blocklists on your sensitive formats.
7. MCP tools reachable only through the gateway, per-tool authorisation, tool call logging.
8. End-to-end OpenTelemetry tracing with user → agent → model → tool correlation.
9. Agent evaluation before production (test sets, quality and safety metrics) and re-evaluation on every model or prompt change.
10. Resilience strategy: multi-region backend pool with circuit breaker, fallback from reserved to pay-as-you-go.
11. Infrastructure as code for everything (Terraform or Bicep, APIM policies included); no setting clicked by hand in a portal.
12. Documented onboarding process for a new agent: owner, allowed models, tools, quota, evaluation, security review.

9. The five mistakes I see most often

• Adding the gateway afterwards. Teams have already coded the model's direct URL and key; migrating costs ten times more than starting with the gateway on a single pilot project.
• Confusing the APIM subscription key with security. The subscription key identifies an application for metering; authentication is the Entra token. Without validate-azure-ad-token, a stolen key gives access to everything.
• A single global quota. If all teams share one quota, the first one to launch a batch job blocks the others. One quota per APIM product is the difference between an incident and a plain 429 for the offending team.
• Agents talking directly to tools. Every hard-wired MCP connector is an unreviewed network authorisation and one more secret. A single gateway is the only way to answer "which agent accessed what".
• No continuous evaluation. A model version change alters answers; without a test set replayed automatically, the regression is discovered by users.

10. What it changes in practice: field experience

On the agentic platform I architected for a large French media group (about 3,000 employees served), the approach described here had three measurable effects. First, onboarding a new agent went from a project to a form: owner, models, tools, quota, and the agent is published in the catalogue with its Workload Identity, its endpoint behind the gateway and its dashboards already fed. Second, cost per team became a figure managers read every month, not a quarterly surprise. Third, an agent's security review is based on facts — the list of tools it can call, the active filters, its traces — not on a team's promise.

The platform was built on AKS with Microsoft Foundry for models, an agent gateway for per-tool authorisation and OpenTelemetry for observability. The same architecture deploys with API Management as the gateway and Foundry Agent Service for hosting when the company does not want to run a Kubernetes cluster: the five layers are what matters, not the product chosen for each.

11. Where to start

Not with the agents. Start with layer 1 and layer 2 on a single low-risk use case: disable keys, put API Management in front of Foundry, set a quota and token metrics. In four weeks you will have a tiny but governed platform on which every subsequent agent costs less than the previous one. Layers 3 to 5 come next, in order of need: guardrails as soon as an agent faces users, MCP tools as soon as an agent acts, full FinOps as soon as a second team joins.

If you want an outside view of your starting point, that is exactly what the first 30-minute call is for.