Why AI Agent Architecture Matters More Than the Model

Most teams begin an AI project with one question: which model should we use? That choice matters, but the system around the model can matter just as much. A change to the harness in LangChain’s Terminal-Bench experiment moved a coding agent from roughly 30th place into the top five, without changing the agent’s core model.

That result points to a practical lesson for developers and enterprise leaders. An AI agent is not only a language model. It is a loop, a set of tools, a runtime, a data pipeline, and a control system that must work together under real operating limits.

The harness can change what an agent can do

The basic system is simple to describe. A headless harness runs an agent loop with no interface of its own. The large language model picks an action, a tool executes that action, and the resulting observation feeds back into the loop. The process repeats until the task is complete.

That harness controls how the agent interacts with tools and how it handles the results. In the Terminal-Bench experiment, changing only the harness moved the coding agent from approximately 30th place into the top five. The model stayed the same, but the surrounding machinery changed its performance.

Decode’s Python agent, also called Decode, shows why deployment choices cannot be separated from the agent’s job. It divides execution into three modes, each with a different latency profile and a different need for inference providers.

• Interactive, online: A terminal user interface connects to a live session, so steering depends on low latency.
• Remote, offline: The agent runs on a server using Kitaru, ZenML’s agent runtime, deployed to GCP, with agents executing on Modal.
• Async, online: A live session works with a job queue while background workflows fan out language-model calls and post results later.

The cost model follows those latency needs. Interactive work pays per token because users need responses during a live session. Offline and asynchronous work can pay per GPU-hour, since the system can batch jobs and wait for results.

Latency changes the economics

Decode’s default test model, Qwen3.6 35B, runs on a single H200 at about $4.54 per hour. Modal’s published price for an H200 SXM is $0.001261 per second, which works out to about $4.54 per hour.

The difference between token pricing and GPU pricing becomes clear with document processing. At frontier API rates of $3 per million input tokens and $15 per million output tokens, processing 1,000 documents with 30,000 input tokens and 500 output tokens per document costs near $97.

The same work, batched on a serverless GPU at around 3,000 tokens per second, takes under three hours of GPU time and costs roughly $13. That is not a universal answer for every workload, but it shows why an agent’s timing requirements should shape its provider and infrastructure choices.

Reservations charge the peak rate for the full contract. Serverless capacity follows the demand curve, which can make it cheaper when peak-to-average ratios are high. Serverless discounts typically range from 2–5× when peak-to-average ratios reach 5–10×.

Enterprise systems are outgrowing pilot-era stacks

Agentic AI has moved past pilot chatbots. Enterprises now use autonomous systems that trigger workflows, access sensitive data, and make operational decisions. That shift makes piecemeal stacks more dangerous because quick-win systems can create brittle integrations, runaway costs, stale or untrusted data, and gaps in governance.

The Discover the Enterprise Agentic AI Technology Stack blueprint maps six layers: application; data and AI lifecycle management tools; foundational models; agentic execution and orchestration engines; data platforms; and infrastructure. Looking at all six layers helps organizations see where an agent depends on other systems, rather than treating the model as the entire product.

Enterprise adoption already reflects that layered approach. Eighty-five percent of enterprises use two or more orchestration tools, 64% use three, and only 15% run a single platform. Microsoft AI Foundry and Copilot Studio appear in 70% of stacks, OpenAI’s Agents SDK in 68%, and Anthropic’s Claude Platform in 47%.

That mix is not staying fixed. More than half of respondents, 53%, expect their primary control plane to be hybrid by the end of 2026. Microsoft leads current primary usage, while Anthropic leads future consideration among the vendors listed.

Enterprises rate overall platform satisfaction at 4.17 out of 5, but ease of implementation scores 3.91 and value for money scores 3.63. The gap suggests that organizations like what these platforms can do, while still struggling with the work and cost required to run them.

Control matters as much as capability

Purchase decisions focus on flexibility at 29%, security and permissions at 17%, production reliability at 15%, and control over agent execution at 15%. Builders put the most investment into agent monitoring and debugging at 31%, followed by security and permissions enforcement at 30%.

The concerns match those priorities. Security and permissioning limitations lead at 37%, followed by vendor lock-in at 23%, limited visibility and observability at 22%, and inflexibility around models and tools at 16%. One in five enterprises cannot stop a runaway AI agent’s spending in real time. Among enterprises with 10,000 or more employees, 18% exercise only reactive control over spending, compared with 23% of smaller enterprises.

Agent maturity also remains uneven. Two percent of respondents say 76% to 100% of their systems are advanced and largely autonomous. Fourteen percent place 51% to 75% of their systems in the complex, multi-agent pipeline category, while 47% report that 26% to 50% of their systems are true orchestration. Another 35% report only 1% to 25% true orchestration, and 3% are still deploying chatbots.

Seventy-one percent say a quarter or fewer of their deployed agents can complete multi-step work autonomously. Bill Wong captures the challenge: “Agent demonstrations look alike, but operational realities do not.”

RAG works best when models handle less

Cost control does not always require a cheaper model. Routing only the genuinely ambiguous 10% to 15% of cases to a language model cut inference cost by roughly six times compared with an all-model baseline.

In high-stakes classification, a three-stage cascade starts with deterministic resolution, retrieves relevant evidence for unclear cases, and escalates only the unresolved cases to the language model. The deterministic stage handles most cases without a model call and produces decisions that are fully explainable.

Retrieval then supplies evidence for ambiguous cases, making retrieval quality more important than generation quality in that part of the process. The model should handle only the cases that remain unresolved, which reduces inference costs and limits the model’s role in sensitive decisions.

An asymmetric risk prompt tells the model to treat uncertainty as a reason to escalate instead of forcing a binary classification. Evaluation should measure retrieval quality and final decision accuracy separately, with deliberate oversampling of ambiguous cases. Human reviewer feedback can also become part of the retrieval data, improving future performance.

Cascade architecture is a mature approach for high-stakes AI because it makes clear which parts of a decision should not be handled by models. As Andrew Kum-Seun puts it, “The most critical architectural decision an IT leader can make is building a technology stack designed not for today’s answers, but for tomorrow’s unknowns.”

Stay connected via Google News

Follow us for the latest AI News