Microsoft Foundry Documents Production Agent Tuning Pipeline Using Agent Optimizer
Microsoft Foundry has documented a staged tuning pipeline that treats a production agent’s configuration—instruction files, skill definitions, tool descriptions, and model choice—as the object under test. The workflow requires triage, human-reviewed traces, a sandboxed test harness, a release policy script, and staged canary promotion before an agent change reaches traffic, with Agent Optimizer still in preview.

Overview
Microsoft Foundry has documented a pipeline for tuning a production agent without hand-editing prompts or swapping models in place. The workflow, built around the Agent Optimizer SDK and a hosted agent, treats the agent’s configuration as the object under test: instruction files, skill definitions, tool descriptions, and model choice. A scheduled GitHub Actions workflow or Azure Monitor alert starts each cycle, with the objective set as a quality gain or cost reduction, subject to a mandatory quality floor, an allowed regression margin, and latency and cost limits.
Pipeline stages
- Continuous evaluation scores production traces on groundedness, task adherence, tool-call accuracy, and safety checks.
- Before launching an optimizer job, triage is required: backend outages, traffic-mix shifts, and evaluator or rubric updates can move scores without any change to the agent.
- If a configuration change is the right lever, the workflow samples recent traces into a versioned dataset, uses MinHash sampling to filter repetitive messages, redacts private content, and pins behavior to a known agent version.
- Humans review each trace and add ground-truth outcomes, paired with a sandboxed test backend so evaluation never touches live payment APIs or customer databases.
Search space and candidate generation
Agent Optimizer searches over the editable configuration: instructions.md, domain SKILL.md files, tools.json descriptions, and a model search space defined in eval.yaml. Fixed business rules live outside the editable scope. Custom code evaluators, running in isolation without network access, inspect tool-call sequences and sandboxed ledger state — for example, blocking a second submit_refund call after a timeout, or a status claim that contradicts the last get_refund_status result. The optimizer proposes up to max_candidates revisions, ranked on a validation dataset.
Promotion and rollback
Promotion is staged. The candidate is deployed but receives no traffic; baseline and candidate are both graded on a held-out regression suite the optimizer cannot access. A release policy script compares the two and exits non-zero on a regression or critical failure. Passing that suite only clears the gate for canary verification: synthetic sessions with version_indicator set to v2 test container startup and tool connections first, then 5% and 25% slices of new sessions run against v1 control traffic. The endpoint selector keeps 100% of default traffic on v1 until promotion.
Rollback has two modes:
- During canary, clearing the routing indicator sends new sessions back to v1.
- After promotion, the endpoint selector must be switched back explicitly, and in-flight v2 sessions have to be stopped or drained.
Idempotency keys and compensating reversals handle transactions that landed during canary.
Operational requirements
The application team must supply the sandboxed test harness, reviewed outcome labels, the release policy script, and session routing; Agent Optimizer itself is in preview. The documentation also warns that fixing a single failure mode does not satisfy every release requirement, and that the held-out suite needs periodic fresh traces to avoid overfitting a fixed test set. The automation opens a pull request for engineer review, so a human still reads the diff before the agent’s behavior changes.
Read the original at techcommunity.microsoft.com →