Salesforce Tightens AI Agent Testing Before Errors Reach Customers

Salesforce is adding new testing, spending, and governance tools to Agent Fabric as companies put more AI agents to work across their businesses. The goal is simple: catch bad answers, broken connections, and rising costs before they affect customers.

The October 27 announcement adds Agent Evals, API Evals, enhanced cost controls, a Custom Agent Scanner, and Open Policies for Federated API Gateways. Salesforce says Agent Fabric already processes more than 1.2 billion requests each day, making oversight a central issue as companies connect agents to more systems.

Testing agents before they cause trouble

Salesforce describes Agent Fabric as an AI agent control plane that oversees and governs agents across different systems. That matters because many organisations now have agents running in separate environments, using different technologies and relying on APIs that can change over time.

Andrew Comstock, Salesforce MuleSoft and Agent Fabric senior vice president and general manager, described the problem this way: “I have too many of them. The technologies are totally different. Nothing talks to each other. Everything runs in a silo.” He added that the challenge is not specific to Salesforce. “It’s across everything,” he said.

Salesforce says fewer than half of organisations with AI agents evaluate their performance on real-world traffic. That leaves failures hidden in production until they lead to customer complaints or budget overruns. As companies add more agents, unnoticed errors and rising expenses can spread across more workflows.

Agent Evals is designed to address that gap. The capability automatically creates and runs tests before and during deployment, checking agents, large language models, and Model Context Protocol servers against set quality benchmarks. It can also evaluate agents in production and identify behavior that falls outside an organisation’s thresholds.

When an agent fails to meet a benchmark, Agent Evals can trigger an intervention under company policy. That response might include approval steps, remediation, or an Agent Kill Switch. Salesforce’s October announcement schedules a November beta and a January 2027 general-availability target for Agent Evals.

MuleSoft first introduced the concept at Dreamforce in September. The earlier description included generating tests, measuring results against success criteria, and connecting failures to corrective action. The new rollout extends that work into production checks and adds specific timing for availability.

Salesforce uses a retail example to show how the process might work. If moving an agent to a cheaper model produces worse answers to order-status questions, evaluations can detect the decline, quarantine the agent, obtain approval where required, and revise and retest it. This is a hypothetical workflow, not a reported customer result.

APIs, budgets, and a broader control layer

API Evals focuses on the software connections agents depend on. The forthcoming tool generates functional and performance tests from API specifications, helping identify breaking changes before they enter production. Both Agent Evals and API Evals are scheduled to enter beta in November and become generally available in January 2027.

Salesforce is also adding a Triggered-Agent Kill Switch to the January roadmap. The basic kill switch became generally available in September. It uses gateway controls to block governed traffic, with an intervention that can apply to a request, session, agent, or tenant before access is restored.

Cost management is another major part of the update. The FinOps Dashboard and Model Wallet provide a consolidated view of AI spending, while budgets and policies can enforce usage limits by provider or model. Model Wallet can set dollar or token budgets with daily, weekly, or monthly limits.

Salesforce says techniques such as semantic caching, payload compression, and response trimming reduced token consumption by up to 75% in testing. That figure is a vendor test result, not a promise about a customer’s total AI bill.

Discovery is expanding as well. The Custom Agent Scanner, which enters beta with general availability planned for January, is intended to find and catalogue agents, models, and APIs across internal systems, environments behind firewalls, and additional platforms. Agent scanners previously described by MuleSoft in April focused on identifying agents across supported environments.

The registry does not require companies to move their agents to Salesforce. “We’re not moving the agent; the agent’s still running there,” Comstock said. “We’re just pulling the pointer so that you can get a universal registry across all of your agents.” Customers also do not need to use Agentforce to use Agent Fabric, he said; the need depends more on the diversity of an organisation’s deployments than its size.

Open Policies for Federated API Gateways will apply governance across multiple API management systems, including Azure, Apigee, and Kong. Salesforce says the updates support continuous evaluation and governance, giving companies more control as they scale their AI deployments.

“Enterprises can only scale agents as fast as they can keep them accurate and on budget,” Comstock said. “Agent Fabric unifies them, bringing evaluations, cost controls, and intervention together so teams can contain issues instantly and scale agents with confidence.”

Agent Fabric sits in a growing market that also includes Microsoft Agent 365, ServiceNow’s AI Control Tower, and OpenAI Frontier. Salesforce agreed earlier this month to acquire US cloud software company Braze in a deal valued at approximately $5.66bn, adding another major business move to the company’s AI and enterprise technology push.

Stay connected via Google News

Follow us for the latest AI News