Running Agentic AI at Enterprise Scale

• 40%+

  Agentic AI projects Gartner predicts get cancelled by 2027

• 1 gateway

  Every agent call, one identity layer

• 1 pilot → N teams

  Built to be re-used, not rebuilt

Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027 — not because the models can't do the task, but because of escalating costs, unclear business value, and inadequate risk controls. That's the number every enterprise AI conversation should start from, because it's the honest base rate for what's actually happening at companies your size right now.

We recently led a build that beat that number: a large enterprise client's first agentic AI platform on Microsoft Foundry, which started as a single team's pilot and was deliberately architected from day one to become the company-wide standard other teams would build their agents on top of — with governance, cost control, and identity handled once, centrally, instead of reinvented by every team that showed up after the pilot worked.

The Metric Everyone Gets Wrong First

The instinct on a first agent project is to treat token consumption as the thing to minimize — smaller prompts, fewer calls, cheaper models everywhere. That's optimizing the wrong variable. A cheap agent that gives a wrong answer, or one that takes five retries to get a right one, costs more than an expensive agent that gets it right once.

The metric that matters is value delivered per token spent, not tokens avoided. That reframing drives almost every decision below — you're not trying to spend less, you're trying to make every token spent count toward a correct, auditable, on-policy outcome.

Governance as a Pipeline Gate, Not a Meeting

The platforms that survive contact with a second team treat governance as code. Microsoft Foundry's own governance model for AI agents is built around exactly this: continuous evaluation of a sample of production traffic, scheduled evaluation against held-out test datasets to catch drift before it becomes a problem, and scheduled red-teaming for adversarial testing — all running against the agent, not against a person's judgment call in a meeting.

We wired that into the deployment pipeline itself. Every agent update went through CI/CD for hosted agents — GitHub Actions and the Azure Developer CLI building a new hosted agent version on every push to main, deployed as a canary release, with automated rollback if quality regressed. That sounds obvious once you say it out loud; the failure mode we see most often is a team that builds this discipline for their pilot and quietly drops it the moment a second team wants to move fast. The gate has to live in the pipeline, not in a person's head — because a person's head doesn't scale past the first team.

Publishing Agents as Versioned Artifacts

That CI/CD pipeline is what let us treat an agent the way you'd treat any other production service: versioned, tested, and published through a repeatable process instead of edited live. Combined with GitHub Copilot in the inner dev loop and GitHub Enterprise for the outer loop — branch policy, required reviews, the evaluation gate above — an agent update looked like any other software change: proposed, tested, evaluated, promoted. That's the difference between "someone's prompt in a notebook" and a system another team can safely depend on.

Model Routing: Paying for Capability You Actually Use

Not every step in an agentic workflow needs the same model. Microsoft Foundry's model router is a trained routing model, not a rules engine — it matches each incoming prompt to the best-suited model under a Quality, Cost, or Balanced mode, so classification and extraction steps land on smaller, cheaper models while multi-step reasoning gets routed to a frontier model. Getting routing right is what makes the "value per token" framing above real rather than aspirational — it's the mechanism, not just the goal.

Zero Trust: Every Agent Call Through One Door

No agent talked to a model endpoint directly. Every call, from every team, was routed through Azure API Management's AI gateway capabilities, with identity enforced there instead of reimplemented per team — the same principle behind Microsoft Entra's Zero Trust framework for AI agents. That gave us one place to:

• Enforce managed identity and scoped permissions per agent instead of shared API keys
• Apply consistent rate limits and cost controls across every team's traffic
• Log and audit every call centrally, which is what actually makes a governance story credible to a security or risk team
• Swap or version a backing model as a policy change, without touching agent code

For an organization where identity and audit requirements are non-negotiable, gateway-as-choke-point is what makes "zero trust" more than a slide — it's the one architectural decision that makes every other governance claim actually verifiable. Microsoft's own Cloud Adoption Framework guidance on operating AI agents lands on the same architecture: "use an AI gateway" — specifically Foundry's AI Gateway, powered by Azure API Management — "to create a unified control point for policy enforcement" across the entire agent estate, not per team.

From One Team's Pilot to the Company's Platform

The build didn't stop at the first working agent. The whole point was for this to become the organization's standard hosted agent platform, alongside its existing data platform investment — which meant the pipeline, evaluation gates, and gateway policies had to be reusable by construction, not bespoke to the first team. Every subsequent team's first agent went through the same CI/CD pipeline and the same evaluation gate as the pilot did, with ongoing testing and iteration support as the platform — and the teams building on it — matured toward production.

That reusability is the actual hard part of this kind of engagement. Building one good agent is a project. Building the platform, governance, and pipeline that let the fifth team ship their agent in a week instead of a quarter is what turns it into infrastructure — and it's the difference between being part of Gartner's 40% and not.

What We'd Tell Any Enterprise Starting This Today

• Decide your evaluation gates before you write the first agent, not after the first incident
• Route every call through one identity and gateway layer from day one — retrofitting it in front of agents that already have direct model access is much harder than starting there
• Treat cost as "value per token," and route models accordingly, or you'll either overpay or under-deliver
• Design the first pilot's pipeline assuming a second and tenth team will build on it
• Schedule quarterly reviews to retire agents nobody's using — a dormant agent is quota consumed and attack surface open for no business value, not a harmless leftover

If you're weighing this kind of platform build for your own organization, we help enterprise and mid-market teams design and stand up exactly this pattern — from the Azure AI landing zone through governance, routing, and rollout. See our Azure Cloud Advisory work, or read more on the Microsoft agent framework landscape and Foundry Agent Service if you're still comparing options.

Have questions about what a governed agent platform would take for your organization? Book a discovery call or join our Discord community.

Frequently Asked Questions

Why do most agentic AI projects get cancelled?

Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls. In our experience, the pattern behind that number is almost always the same: a team ships a working pilot with no governance gate, no cost control, and no path for a second team to build on it — so it either breaks something or stalls the moment it needs to scale.

What does "governance" actually mean for an AI agent platform?

In practice it means an automated gate between an agent and production: every new hosted agent version runs through evaluation and quality checks — Microsoft Foundry supports this natively as continuous evaluation of production traffic, scheduled evaluation against test datasets to catch drift, and scheduled red-teaming — before a canary release, with automated rollback if quality regresses. Governance that lives in a review meeting instead of a pipeline doesn't survive contact with a second team.

What is model routing and why does it matter for cost?

Model routing sends each request to the smallest model capable of handling it well instead of sending every call to the most capable, most expensive model by default. Microsoft Foundry's own model router is built for exactly this — it's a trained routing model that matches each prompt to the best-suited model under a Quality, Cost, or Balanced mode. The savings come from not overpaying for the calls that never needed the expensive model in the first place.

Why route agent traffic through Azure API Management instead of calling models directly?

Azure API Management's AI gateway capabilities give you one place to enforce identity, rate limits, and policy for every agent and every team, instead of each team wiring its own auth and hoping it's consistent. Backends can authenticate with managed identity instead of stored API keys, and a model or provider swap becomes a policy change at one layer instead of a code change in every agent.

How do you roll a platform like this out beyond the first team that builds on it?

Treat the first deployment as a reference implementation, not a one-off: the CI/CD pipeline, evaluation gates, and gateway policies get built to be reused, not rebuilt. Each new team's first agent moves through the same pipeline as the pilot, with ongoing testing and iteration as it moves toward production — that's what turns a project into a platform other teams can safely build on.