The Hardest Problems in Shipping AI Agents Are Operational, Not Architectural

The practical lessons from hands-on work with Microsoft Foundry point to a recurring conclusion: the hardest problems in shipping AI agents are operational, not architectural. Model selection—the question most teams start with—was rarely the primary challenge. Grounding quality, tool reliability, observability, and security controls accounted for most production issues.

Production readiness is about reliability, not intelligence. A customer-support agent can pass testing and still fail in production when the finance API times out, the knowledge base is stale, or tool calls break under peak traffic. The failure is not the model; it is the system around it.

The proposed remedy is a set of layers in addition to the language model itself: a knowledge layer (Azure AI Search), tool integration, monitoring (Azure Monitor), and security via managed identity and Key Vault. Grounding is singled out as the highest-leverage improvement. Sample code shows an agent configured with Azure AI Search as a knowledge source, and reliable retrieval "is usually more valuable than sophisticated prompting." That is a claim, not a benchmark—no controlled comparison is provided.

The operational advice is concrete. The recommendation is to design for tool failure before designing for capability. A typical scenario: an agent calls the Finance API, the API times out, and a fallback response is returned. Retry and fallback logic, with cached responses as a safety net, are the suggested pattern.

Observability is the second pillar. Azure Monitor tracing tracks latency, tool failure, token usage, and grounding success. A sample dashboard shows 5,120 requests per day, 3.2-second average latency, and 97% grounding success—illustrative numbers, not measurements from a deployed system. The underlying point holds: what cannot be observed cannot be operated.

Cost is treated as an ongoing concern rather than an afterthought. Token consumption rises unexpectedly in production; prompt compression, response caching, RAG filtering, and context trimming are the listed mitigation techniques. The advice is to plan cost optimization from day one.

Security is framed as a design requirement, not a deployment task. The agent should never access data beyond a user's permissions, and controls such as managed identity, Key Vault, RBAC, and audit logs should be in place from the start.

On evaluation, success should not be measured by final response quality alone. An agent can produce a correct answer through an incorrect process—retrieving irrelevant documents, making unnecessary tool calls, or skipping citations. Reviewing retrieval quality, tool execution, citation accuracy, and the reasoning path matters as much as the output.

No claim is made that Foundry is unique in offering grounding, tracing, and security tooling; these capabilities exist in competing platforms. What it does offer is a ten-point checklist: business goals defined, grounding implemented, security enabled, evaluation pipeline established, monitoring configured, failure handling tested, cost optimization reviewed, governance defined, feedback loop available, and rollback strategy prepared. Whether that checklist is sufficient for production readiness remains unresolved. The contrast—an enterprise support agent versus a financial compliance agent using the same model but requiring different architectures—suggests the real work begins with the business problem, not the model. The suggested next step is to build a first agent, configure grounding, enable tracing, and only then evaluate quality. That sequencing is the takeaway.