Microsoft's Foundry Agent Optimizer reads failure traces to tune agent configuration

Microsoft has added a reflection-based optimization feature to its Foundry Agent Service. Called Agent Optimizer, it proposes edits to an agent's natural-language configuration — instructions, skills, and tool descriptions — by reading the traces of failed runs rather than relying on scores alone.

Most agent teams tune configuration by hand: write instructions, run cases, read failures, reword, repeat. It is a manual trial-and-error loop with known weaknesses. A quality failure lacks a stack trace; the cause can sit in the instruction, a tool description, or the model itself. Fixing one case by editing the instruction often breaks another that previously passed.

Agent Optimizer takes a different route. Given an agent, existing evaluators, and a representative dataset, it searches for a better configuration. Each round it proposes a candidate change and tests it on a small batch of cases first; only promising candidates proceed to full evaluation. It maintains a Pareto frontier of candidates that win on different cases rather than a single best, and draws the next proposal from that pool. A human reviews the leaders and their scores, then approves one, which becomes the agent's next version with the same model and tools.

The approach builds on reflection-based optimization methods such as GEPA, which can match or beat reinforcement-learning tuning with far fewer trials. The reasoning: every evaluation run leaves a natural-language trace of the agent's reasoning and tool calls, and a stronger model can read that trace to identify why a run scored poorly and propose a targeted fix.

Microsoft ran internal benchmarks across six prompt agents using gpt-4o-mini as the agent model and a stronger model as the reflector. Improvements ranged from +6.7 percentage points on a sentiment classification agent to +17.5 points on a tax-assistant agent with a rubric-based judge. The changes are readable: the optimizer adds concrete rules targeting failure patterns rather than rewording prompts at random. One agent in the set, retail-tools, calls tools during evaluation, and the optimizer improved it while it kept using those tools.

The company is careful with its claims. These are single-seed runs on small datasets, so the magnitudes are directional, though gains were consistent across all six agents. Agent Optimizer is most useful when the agent already runs on real evaluators and production-representative data, when fixes tend to trade one failure for another, and when maintaining several agents. It helps less when the root cause is infrastructure — a tool returning bad data, for instance — rather than agent configuration.

Two caveats stand out. The feature is in preview, and the documented plans are work in progress, not commitments to deliver. Microsoft notes its own results will vary with agent, data, and evaluators. The first round benchmarks cover prompt agents tuned on system instructions only; a planned follow-up covers hosted agents, larger evaluation sets, and more optimization targets such as skills, tool descriptions, and model selection. That next round should clarify how far the approach extends beyond instruction tuning.