As enterprise leaders start deploying agentic workflows, they must establish the infrastructure to build and run them, one capable of fluidly routing a diverse set of workloads across the most efficient compute resources.
This requires the ability to manage heterogeneous infrastructure, utilizing high-performance accelerators for large-scale training and inference, and utilizing CPUs for the critical orchestration layer of agentic AI. As autonomous agents become more prevalent, CPUs are ideally suited for managing agent state, semantic routing, tool selection, and spinning up secure, isolated sandboxes to safely execute untrusted generated code.
The Google Axion advantage
Google Cloud, with its workload-optimized Compute Engine portfolio, which includes general-purpose and specialized offerings, shines in addressing this need.
Google Axion processors within this portfolio comprise a family of custom Arm processors engineered for performance, efficiency, and versatility, with a feature set that supports general-purpose workloads, CPU-based AI workloads, and other specialized tasks requiring Arm-native compatibility and direct hardware access.
Axion is Google’s first custom Arm-based server CPU, introduced in April 2024. It is designed specifically for hyperscale cloud and AI-era data center workloads.
Axion also leverages more than a decade of Google’s custom silicon innovation. This enables Google to more readily incorporate customer feedback into chip designs and address the more general, though complex, needs of CPUs.
Matching workload type to the processor
Bhumik Patel, Director of Software Ecosystem Development at Arm, says the key to all of this is to match the workload type as closely as possible to computing capacity. CPU-powered cloud instances are a practical option for certain AI workloads, particularly those with smaller datasets or less complex models.
“Agentic tasks such as orchestrating, talking to APIs, and memory management are all ones CPUs are good at, so it’s a distributed and concurrent AI workload,” Patel tells The New Stack. Intelligent workload-processing apportionment makes agentic AI more cost-effective and efficient than running all workloads on a single compute type.
This efficiency is quantifiable. The Google Kubernetes Engine Agent Sandbox running on Google Axion N4A provides up to 30% better price performance than the next hyperscale cloud provider, says Google’s Mo Farhat, Axion Group Product Manager. The GKE Sandbox is an open-source Kubernetes-native primitive designed to execute untrusted AI-generated code safely.
“Agentic tasks such as orchestrating, talking to APIs, and memory management are all ones CPUs are good at, so it’s a distributed and concurrent AI workload.”
Intelligent workload decoupling makes agentic AI significantly more cost-effective. Google Cloud’s fluid computing foundation enables engineering teams to reserve specialized accelerators strictly for heavy reasoning and generative workloads, while leveraging Axion CPUs for high-concurrency orchestration and context management.
Secure execution with the GKE Agent Sandbox
As agents begin to generate and execute dynamic code autonomously, security is non-negotiable. Running AI-generated code directly in a standard cluster poses severe security risks, as untrusted code could potentially access other apps or the underlying cluster node.
The Google Kubernetes Engine (GKE) Agent Sandbox resolves this by providing an isolated environment for safely executing untrusted code. Running on Axion-powered N4A instances, the sandbox provides up to 30% better price performance than comparable workloads on other hyperscalers.
The vertical stack isolates sensitive tasks at the kernel level with sub-second latency.
The vertical stack isolates sensitive tasks at the kernel level with sub-second latency. GKE Agent Sandbox natively supports gVisor (an open-source application kernel developed by Google that acts as a secure sandbox for containers) and default-deny Kubernetes network policy. Agent Sandbox provides pluggable interfaces for open-source sandboxes, such as Kata Containers, enabling users to customize their kernel isolation.
Powered by gVisor technologies with software support from Arm’s architecture, the sandboxes intercept and validate system calls before they reach the host kernel. These isolated execution environments enable deployment of autonomous systems at scale without sacrificing performance or operational agility.
To manage resources efficiently when agents sit idle, GKE Pod snapshots allow users to save and restore the exact process state of sandboxed environments. This functionality provides four major architectural benefits:
Fast startup: Reduces sandbox startup time by restoring from a pre-warmed snapshot rather than initializing from scratch.
Long-running agents: Pauses sandboxes that take a long time to run and resumes them later—or moves them across nodes—without losing progress.
Stateful workloads: Persist an agent’s context, such as conversation history or intermediate calculations.
Reproducibility: Captures a specific state to use as a baseline for spinning up multiple new sandboxes.
Getting started
As token generation, autonomous workflows, and continuous agent interactions grow exponentially, relying exclusively on accelerator-backed stacks for every task will become financially and architecturally unsustainable.
The combination of CPU and accelerator execution accounts for bursts in agent activity and unpredictable demand spikes by eliminating the inference tax. Google Cloud’s full-stack advantage enables organizations to deploy the right machine for the job.
By using Google Axion and GKE Agent Sandbox, builders can optimize total cost of ownership and security while maintaining the performance required for AI agents.
Google made GKE Agent Sandbox generally available in May 2026 and, in the same post, introduced a second project called Agent Substrate. These two announcements concede a point that Kubernetes veterans have been reluctant to call out.
That indirect admission is that the platform that won the container decade is not the right control plane for AI agents. Agent Sandbox provides agents with a secure environment to run untrusted code. Agent Substrate adds a scheduling layer that routes around the Kubernetes control plane because the API server was never designed for how agents behave.
Compare agents to processes in an operating system rather than services in a data center, and the mismatch is obvious. A modern OS runs thousands of processes that spend most of their life asleep. It wakes each one on an event, hands it a slice of a CPU, then pages its idle memory out to disk to make room for the next. Agents behave almost exactly like those processes. Kubernetes was originally created to manage a fixed set of long-running, replicated services. This fundamental design explains why much of the agent infrastructure now runs on Kubernetes rather than being integrated into it as one of the workloads, such as a Deployment or a StatefulSet.
What an agent actually is as a workload
An agent is a long-running, stateful session that stays idle for most of its life, wakes to execute a burst of code, then goes quiet again. The code it runs is generated by a model at runtime. The host has to treat it as untrusted by default. Each session needs a stable identity, the ability to pause and resume without losing memory, and hard isolation from its neighbors.
Think of the agent as a process in a time-sharing OS. Just as the scheduler suspends a sleeping process and restores it the moment a keystroke arrives, an agent runtime must hibernate an idle session and restore it with its working memory intact. The wake path is where the user is waiting, so every millisecond on it is felt.
Consider a coding agent that a developer leaves open across an afternoon. It runs for ten seconds when a prompt lands, then waits twenty minutes for the next one. Multiply that by every developer on a team, and you have thousands of sessions that are alive on paper and asleep in practice.
The hyperscalers have already moved in this direction. The session-aware, isolated runtime for agents has now become the fourth compute offering in addition to virtual machines, containers, and serverless.
Sessions that sleep for hours
Agent sessions are bursty in a way web services never are. Holding a full Pod for each idle session wastes the memory and CPU that the Pod reserves, which is why the emerging runtimes snapshot idle sessions out of compute entirely.
Code the platform did not write
Because a model writes the code an agent executes, the runtime cannot assume the workload is well-behaved. It must be able to run a process capable of performing any action, which moves isolation responsibilities from the container boundary to the kernel boundary.
State that has to survive a nap
An agent that loses its context each time it suspends becomes unusable. So, the runtime must save its volatile RAM and filesystem state during hibernation and restore them upon resuming.
Why the Kubernetes control plane sits in the wrong place
Kubernetes schedules work through a central API server and a scheduler designed for a modest number of long-lived Pods. That design assumes placement decisions are rare and durable. Agents violate the assumption by generating a constant stream of fine-grained scheduling events, making the control plane the bottleneck rather than the referee.
The scheduling policies are the first to be strained. Researchers studying agent scheduling have documented that the round-robin and random placement strategies common in Kubernetes clusters work well when requests are short and arrival rates are high, because a bad decision is amortized quickly. Agent requests run longer and arrive less often, so a poor routing choice lingers, amplifying tail latency for the user stuck behind it.
The second pressure point is the API server itself. Storing every agent, active or idle, as a Kubernetes object would mean millions of resources in a system never sized for that many. Agent Substrate’s own architecture notes are blunt about this, acknowledging that there is no clever way to make the standard control plane hold that many objects, so the runtime keeps most agents out of it. Routing takes a similar detour, with a dedicated networking layer that sends each request straight to the correct session and wakes it if it is asleep.
Kubernetes is a fine data center scheduler, and it stays useful for provisioning the machines underneath. It is the wrong scheduler for a workload that looks like a swarm of sleeping processes.
Agent Sandbox, a secure box for untrusted code
Agent Sandbox is the layer that answers the isolation problem. It is an open-source execution environment built on Kubernetes that gives each agent a hardened place to run model-generated code, and Google moved it to general availability after roughly 16x growth in GKE sandboxes in under five months.
The mental model is a jail rather than a container. A normal container shares the host kernel and trusts the workload to stay in its lane, whereas a sandbox assumes the workload is hostile and puts a real boundary around it. Agent Sandbox reaches that boundary through gVisor by default, adds a default-deny network policy, and exposes a pluggable interface so teams can swap in Kata Containers for full kernel isolation.
Customers such as LangChain and Lovable are already running millions of agents on it, which is what forced the performance work. The result is a runtime that treats security and speed as the same problem rather than opposing ones.
Warm pools for the cold-start problem
Spinning up a fresh sandbox per request would add seconds of latency, so Agent Sandbox keeps a warm pool of pre-provisioned replicas. Google reports the API can allocate 300 sandboxes per second per cluster, with 90 percent of allocations finishing in 200 milliseconds.
Pod snapshots for idle sessions
Idle agents are suspended via Pod snapshots and resumed on demand in seconds, freeing the underlying compute rather than paying to keep a sleeping session resident.
Kernel isolation as the default
The isolation is not an add-on for the paranoid. gVisor and network lockdown ship as the baseline, on the assumption that any agent might run something it should not.
Agent Substrate, a runtime for millions of mostly-idle agents
If Agent Sandbox is the secure box, Agent Substrate is the runtime that decides which agent runs where. It reuses the secure runtime and snapshotting from Agent Sandbox and pairs them with a small, focused control plane that sits alongside a Kubernetes cluster, taking the standard control plane off the critical path.
The trick is virtual memory overcommit applied to compute. An OS lets programs address far more memory than the machine physically holds by paging cold pages to disk. Agent Substrate does the same with sessions, multiplexing a large registry of stateful actors onto a much smaller pool of pre-warmed worker Pods and snapshotting the idle ones out to storage. The project reports 30x or more oversubscription with sub-second activation, because the worker Pods are already running when an event arrives and never wait on the Kubernetes scheduler.
The developer-facing shape consists of two custom resources, a WorkerPool that defines the ready compute, and an ActorTemplate that defines the agents. Substrate is framework-agnostic, running ADK, LangChain, Claude Code, or any OCI container as an actor, which is what lets it host full agent harnesses rather than single agents.
A control plane beside Kubernetes, not inside it
Substrate does not replace Kubernetes. It uses Pods and autoscaling for provisioning and layers its own scheduler on top for the agent-specific decisions Kubernetes handles poorly.
The path to production through kagent
Solo.io has already wired Substrate into kagent, its Kubernetes-native agent platform, exposing Agent Substrate as a selectable runtime so an OpenClaw-style harness can be scheduled as an actor onto a worker pool from a single UI.
Choosing where your agents run
The two projects are not competitors, and neither replaces the cluster underneath. The decision is about which layer owns which job, and most real deployments will use all three together.
Requirement
Recommended layer
Rationale
Running untrusted model-generated code safely
Agent Sandbox
Kernel isolation via gVisor, though it adds overhead
Millions of idle sessions on limited hardware
Agent Substrate
Actor multiplexing trades some cold-path latency for density
Provisioning machines and long-lived services
Kubernetes
Proven at scale, but not tuned for bursty agents
A production agent platform with a UI
kagent on Substrate
Packages the runtime, still early and evolving
In practice, most teams will not pick just one of these. A team running coding agents at scale is likely to sandbox the code, schedule it through Substrate, and let Kubernetes provision the nodes beneath both.
What this means to the Cloud Native ecosystem
Anyone who has managed a busy cluster will easily recognize this pattern. The agent is a process, the worker pool is a set of CPU cores, snapshotting an idle session is paging memory to disk, and oversubscription is the same bet virtual memory has always made. Kubernetes remains the underlying machine, and the layer that schedules agents is being rebuilt on top of it to better match how agents actually run.
The open question is who owns that layer. Agent Substrate is a first look rather than a finished product; kagent is early, and rival runtimes will arrive as the cost of idle agents becomes a line item that platform teams can no longer ignore. The next thing worth watching is whether this agent control plane consolidates around a single open project, the way container orchestration once settled on Kubernetes. The alternative is that every hyperscaler ships its own, and the fragmentation that agents were meant to escape returns one layer up.
Microsoft recently took the wraps off Brain, the internal AI system that continuously monitors Azure’s health and, increasingly, acts on what it finds — declaring outages, pausing harmful rollouts, and notifying affected customers.
To dive deeper, The New Stack sat down with Russinovich, who is also Azure’s deputy CISO and a technical fellow, to talk about how the Brain project came to be and how it evolved over time.
Brain, as Microsoft describes it, is Azure’s centralized AIOps system for cloud health. It operates as an intelligent layer on top of Azure Resource Graph (ARG), and together, the company says, the two form a real-time digital twin of Azure’s health.
A real-time digital twin
While Brain today uses many AI tools, the project is actually much older than the generative AI boom, and to get started, the team had to build a solid foundation first. “At the heart of this system is Azure Resource Graph, which started as ‘let’s create a digital twin of Azure, so we can understand the relationship between the different resources in Azure,’” Russinovich tells The New Stack.
That internal digital twin became a public service at the urging of what Russinovich calls whale customers, those “that have huge estates across many different tenants and subscriptions that wanted to do easy queries across the whole thing, like, ‘What Linux VMs do I have, and what versions of Linux are they on?'”
It was actually the root cause analysis on top of that graph that Brain really began with. “Many times you can just trace dependencies and say, well, these services all depend on this other service that has gone unhealthy, and so that I think was the genesis of having Brain go and start to have a lot of ML-driven algorithms to identify root cause on top of the graph,” Russinovich explains.
Around the same time, Microsoft kept encountering a measurement gap: A service team’s own health metrics indicated everything was fine, but customers saw failures. Russinovich says that could happen because the Azure team wasn’t “measuring what customers are experiencing, or because they’re aggregating at scopes that hide customer-specific problems.”
So Microsoft decided to standardize. “We decided, let’s go standardize on the way that we measure health,” Russinovich says, “and we came up with service level indicators, SLIs.”
Getting services across Azure to actually emit them through shared libraries that conformed to the schema was a complex task that took several years.
“It’s kind of a whole bunch of different things that happened in parallel that all have come together,” he says. “There’s just a tremendous amount of data engineering that goes into this, and trying to keep it as automated as possible.”
Three signals feed Brain
In his blog post, Russinovich writes that Azure’s reliability challenge isn’t a lack of tooling but a “comprehension problem,” with a hyperscale cloud now producing more signal than humans can read.
Azure runs hundreds of services across more than 80 regions, 500+ data centers, and 800,000+ kilometers of fiber and subsea cable. And yet, he writes, Microsoft still sometimes learns about a quietly degrading service from a customer before its own systems detect it.
Today, Brain pulls from three classes of signals. The standardized SLIs come first. Service teams also build and register their own domain-specific monitors, which run alongside telemetry-like deployments, support volume, and cross-service dependency signals. Third-party indicators make up the rest.
Brain produces the same four outputs for any subject, whether that’s a service, a region, a deployment unit, or a customer’s resources.
Based on this, Brain produces the same four outputs for any subject, whether that’s a service, a region, a deployment unit, or a customer’s resources. It reports the health state, how severe the issue is, who is impacted, and — crucially — why it reached that conclusion.
Those conclusions then drive alerts and remediations. Brain declares outages based on blast radius, Russinovich notes, and scopes customer notifications to the impacted subscriptions and regions. The system automatically routes incidents to the appropriate service team and sends deployment-gate signals to pause rollouts causing the issues.
Russinovich says the system is “very pluggable in terms of what signals go into it, and includes even things like customer support tickets that have been opened, and social media posts that mention Azure.” He says Brain is “primarily monitoring,” but “it also can take automated repair actions, too. So for some incidents, teams can specify if, when this happens, go try these things, and Brain kicks those off as well.”
He says the SLIs are “emitted at the scale unit level, so that we can do aggregations for overall health. We can pinpoint specific customers that are being impacted, because we know what customers map to which scale units, and that’s the way that the auto notification triggers off that.”
Why ML sets the thresholds
Microsoft’s original plan for turning SLIs into health determinations was the textbook one, asking every service team to define its own SLOs. It didn’t work.
Russinovich says the schema work itself was hard enough, but “even more challenging is coming up with an SLO that is actually a good SLO.” Teams sandbagged their thresholds, he says.
“[Everyone] wants to be very conservative because they don’t want to get paged or have customers told that things are unhealthy when they’re not, so they’re like, ‘You know what, my SLO is 5% of API queries can fail, and then let’s call it unhealthy,’ when actually that’s not a good way to determine health or regressions as rollouts happen, so we decided, ‘Let’s just stop asking them to define their SLOs.'”
“Everyone wants to be very conservative because they don’t want to get paged… so we decided, ‘Let’s just stop asking them to define their SLOs.'”
Instead, ML models now derive the thresholds from each service’s own behavior, per scale unit and per region.
“There’s a baseline for behavior of the service in this region versus that region,” Russinovich says, “and then we can see when there’s regressions.” The resulting SLOs are dynamically adjustable and automated, he explains.
Tying a regression back to the change that caused it is harder still, he says. Brain tracks rollouts of service updates, and “we’ve got ML algorithms too that can identify with confidence this rollout is causing a regression.” But, he says, the rollout that just reached a scale unit isn’t necessarily the culprit.
“A change doesn’t necessarily show up as a regression immediately. It can have latency; it can take hours to show up, or in some cases even days… there can be many, many deployments that have happened over the last day, and you’re like, which one was it?” That’s why, he says, “there’s a lot of ML going into symptom versus change mapping and automated detection.”
Agents that fix outages
The published post keeps its results vague. Detection precision “has improved significantly”; a “substantial majority” of Brain-integrated outages were auto-communicated to customers in the past year; and time-to-notification improved “materially” over manual notifications. In the interview, Russinovich puts numbers on some of it.
He says, “The thing that frustrates customers the most is when they’ve got to call us and tell us there’s an issue, because then they’re like, you guys don’t even know that there’s a problem. I have to tell you there’s a problem. If we can tell them, hey, there’s a problem, we know about it.” Auto-notification, he says, has driven “this reduction of like four to 6x in terms of customer support tickets open, because Brain is automatically notifying them, and they know that we’re on it.”
“The second you put a human in that loop, you can blow right past the 15 minutes.”
“Our time-to-mitigate goal is 15 minutes, from some problem to actually being resolved within 15 minutes,” Russinovich says. “The second you put a human in that loop, you can blow right past the 15 minutes.”
According to Russinovich, the company hits this 15-minute notification window for 80 to 90 percent of the services on Brain. Often, it’s also much shorter and closer to five minutes.
One caveat here is that not everything runs through Brain yet. Microsoft prioritized what it calls its critical services, the foundation the rest of Azure depends on, and Russinovich puts their coverage at “like 70 or 80% of them, and then the tail’s being worked on.”
He notes the rest aren’t flying blind. “It’s not like the services that aren’t on Brain don’t have health systems and alerting and everything. Brain improves things, even for those services.”
For the engineers who do get paged, Brain assembles the picture they used to piece together by hand.
“The incident gets populated initially with an automated collection of information that will say, here’s the graphs of availability on this SLI over these scale units over the last 24 hours, here’s the list of impacted customers, this is the scale unit, here’s the other information supporting this, and so already there you’re saving the engineer huge amounts of time just in going and information gathering and just presenting it right in front of them.”
Agents on top
In the Brain announcement, Russinovich writes that “agents need something to be agentic about.” A triage agent that doesn’t know the dependency graph can’t triage anything, he argues, and the health model is “the prerequisite, not the consequence, of agentic operations at this scale.”
At this point, Microsoft has started running agents on top of Brain. A system called Triangle, which Microsoft Research also described in a 2025 paper, gives each service team an LLM-based agent trained on its historical incidents and troubleshooting guides, with an orchestrator that routes ambiguous incidents among them.
“This Triangle system has agentic representatives for the services, where the Triangle orchestrator then fans it out and says, ‘Here’s the incident; raise your hand if you think it’s yours,’” Russinovich says. Without it, tickets would bounce from team to team — something Microsoft calls handoffs — which increases response times.
“We don’t have to write down every single rule prescriptively… let the agent do things based on its own judgment.”
It’s still early days for Triangle, though. “We’re still relatively early, so there’s only a small number of services onboarded to it, but already for them the handoffs are much faster and more direct than pre-Brain,” Russinovich says.
In the long term, he wants agents to replace the deterministic remediation rules that teams write today.
“We don’t have to write down every single rule prescriptively,” he says. “And have this tree of decision making, but rather let the agent do things based on its own judgment, which has a whole bunch of benefits, like the system keeps up to date automatically. Then it can also find paths to resolution that we might miss in the deterministic rules that we’ve got.” On agents that actually fix things, he says, “We still consider ourselves at the beginning of that.”
Most engineering teams I talk to can ship an AI demo. The prototype works, stakeholders are impressed, and everyone agrees the use case has potential. Then the project hits a wall.
The reasons for this can vary, but new research shows that difficulties in collecting and parsing real-time data from multiple sources are often the problem. And it’s compounded by a growing skills shortage.
“Only 32% of organizations report having agentic AI running in production.”
According to Confluent’s 2026 Data Streaming Report, only 32% of organizations report having agentic AI running in production. At the same time, two-thirds of respondents cited data infrastructure and data quality as barriers to the success of agentic AI. The models work in controlled conditions, but production is a different story.
Why the demo-to-production gap is so wide
Demos tend to work because everything around them is controlled. The data is static and curated carefully to support exactly what the model will be asked to do. Production environments don’t always offer those luxuries.
In production, AI systems have to query data that lives across dozens of sources, including databases, event streams, application logs, and third-party feeds. Much of that data is poorly governed, and little of it is designed to be consumed by an AI agent in real time. Models that looked impressive in pilots return unreliable results because they’re working with stale, incomplete, or uncontextualized data.
The instinct is to tune the model, but the problem is more likely to be the data feeding it.
In the report, 72% of IT leaders cited insufficient infrastructure for real-time data processing as a barrier to scaling AI, up from 61% the year before. That increase suggests the problem isn’t going away; it’s getting more visible as teams move projects into production.
“The instinct is to tune the model, but the problem is more likely to be the data feeding it.”
AI systems need data that’s trustworthy, contextualized, and current, and those properties are hard to guarantee when data is sitting in siloes that weren’t built for continuous consumption. Batch pipelines almost always introduce latency, lack formal data contracts, and obfuscate lineage. The AI system ends up working with an inconsistent, partial snapshot of the business instead of what’s actually happening now.
The skills problem makes this harder
The report reveals another challenge: 71% of IT leaders cited a shortage of relevant expertise and skills as a barrier to AI adoption.
The work of application development has shifted from encoding business logic to creating an information environment where automated systems can learn and generalize. Building reliable AI applications requires developers to be stronger data engineers. They need to understand distributed systems, streaming architectures, data quality controls, and how to build pipelines that hold up under real-world conditions. They need to reason about data lineage, schema evolution, and what happens when an upstream source changes. And the QA patterns that work for deterministic software — where the same input yields the same output — don’t transfer to probabilistic systems.
Most developers haven’t had to think this way before. The discipline of getting the right data to the right system at the right time, in a governed and reusable way, has gone from a specialist concern to a requirement for anyone building production AI.
This affects how organizations should think about closing the demo-to-production gap. The investment in data engineering skills needs to keep pace with the investment in AI itself.
What production-ready AI actually requires
Organizations that make it out of the pilot stage treat data infrastructure as a first-class concern from the start. That means building real-time pipelines rather than batch processes. It means applying schema definitions, ownership metadata, and quality checks at the point of data production rather than in the data lake. And it means structuring data as reusable products that different teams and applications can build on, so the engineering work supporting one AI application can accelerate the next one, rather than starting from scratch.
The 2026 report found that 88% of IT leaders said data streaming platforms help address data infrastructure and quality issues for agentic AI. That’s because they address the specific reasons AI projects stall — real-time data delivery, upstream governance, and making data trustworthy enough to use at inference time.
The shift is already happening
For the first time, the report found that investments in data streaming outranked those in AI and machine learning, by 88% to 82%. Organizations that have tried to ship production AI are increasingly recognizing that the model isn’t the hardest part.
“For the first time, the report found that investments in data streaming outranked those in AI and machine learning, by 88% to 82%.”
So if you’re stuck at the pilot stage, resist the urge to keep optimizing the model. A better question is whether the data feeding the model is fresh, accurate, and well-governed, and whether your pipelines were actually built for production AI or a demo that only had to work once.
Agentic systems usually have two jobs: Build context, then use that context to produce an answer or action.
Many failures that look like LLM problems start in the context-building step. The answer the LLM gives is limited by the context it was given, or it finds through tool calls. If the agent model cannot find the right sources, then improving the generation model will not improve the overall system.
“Many failures that look like LLM problems start in the context-building step.”
A client, Specstory, wanted to give users the ability to ask questions from the agent’s history. For example, why a team chose Authlib for authentication and what alternatives they considered. The chatbot needs the right prior conversations, decisions, and tradeoffs from a large corpus of coding sessions. The model and system prompt help only after those chat turns have been retrieved and are in context.
If retrieval ranks implementation snippets above the discussion where the team weighed alternatives, the agent can still produce a confident answer. It may find code that imports Authlib and a few inline comments, then describe the decision based on implementation evidence rather than the actual trade-off discussion.
The same pattern showed up in an AnkiHub operator review in our private community. A request for help studying based on lecture slides only works if the agent’s tool calls retrieve the right flashcards. The hard part is not finding any related cards. A lecture on the function of the heart may match hundreds of cards. Ranking decides whether the core cards make it into context or whether the system has to raise top_k and flood the prompt.
The exact setup changes by product. The context-building step might use local search, semantic search, web or API calls, or database queries. It might be handled by an agent, a fixed workflow, or application code. The process stays the same: gather the right context, then generate from it.
For example, a coding agent runs rg, opens files, reads logs, and inspects tests before writing a patch. A research agent searches the web and internal notes before writing an answer. A study assistant searches deck facts and user context before suggesting what to learn next.
When context building fails, the symptoms look like generation failures.
Retrieval failures mimic generation bugs
Symptom
Retrieval cause
Hallucination
The answer source never made it into context.
Context rot
Low recall forces a high top_k, so noisy results fill the context window.
Latency
Weak retrieval leads to more tool calls, larger candidate sets, and larger context windows.
A better model helps with reasoning and writing, but it cannot give a better answer without the right context.
“A better model helps with reasoning and writing, but it cannot give a better answer without the right context.”
The Mixedbread OfficeQA-Pro Eval shows the same pattern at the benchmark scale. OfficeQA-Pro uses 89,000 pages of financial documents, dense tables, scanned PDFs, and questions that require reasoning across documents. Giving Codex better search tools reduced tool calls and improved answer quality.
Plain-text tools like grep and rg work (ish) on flat code files. They do not work well when context lives in PDFs, tables, chat histories, multi-modal inputs, web results, and permissioned data. In those cases, the agent needs a retrieval that can combine exact terms, meaning, metadata, permissions, and ranking quality.
Retrieval needs traces and evals
Once retrieval enters the architecture, the next question is whether it finds the right information.
For that, you need traces and evals. For each retrieval step, the minimum trace is the input, the outputs, and a way to label whether each output was relevant.
For a coding agent using rg the input is the command, the output is the returned snippets, and the label says which snippets helped, which were noise, and which relevant files were missing.
For product retrieval, the step might be BM25, semantic search, hybrid search with reranking, a generated SQL query, or something else. Capture the query or arguments, the returned documents or chunks, and whether those results were helpful.
Trace each retrieval step by itself, then evaluate the full context-building pass. The local trace answers “Did this query return useful material?” The full trace answers “Did the system collect everything the model needed before generation?” If it did, failures are a generation problem. If not, it’s a retrieval problem.
You cannot know where the failure started or what to fix without traces.
Different failures need different fixes
“Improve retrieval” is too broad to be useful, as different problems require different solutions. If a relevant document is missing, the trace should show where it disappeared: query building, retrieval, filtering, ranking, or final context assembly.
The failed step, plus what the trace shows, tells you what change to make.
Failed step
What the trace shows
Change to make
rg / grep
A conceptual query returns literal matches while missing relevant files.
Add semantic search over files or chunks, or generate better keyword queries before calling rg.
BM25
The query uses the right concept but different words from the source material.
Add semantic search, synonyms, or query expansion.
Semantic search
Exact names, error strings, document IDs, or domain terms are missing from the results.
Add a keyword or BM25 path, or boost exact term matches.
Hybrid retrieval
The relevant passage is ranked 7th, but the context only takes the top 5.
Add or tune a reranker, or raise candidate top_k before reranking.
The right fix depends on what the system was trying to retrieve. A decision-history question requires the decision, the alternatives, and the chats in which the team worked through them. A study question depends on the lecture material, deck metadata, semantic matches, and the user’s study context.
The architecture
Once you trace individual retrieval calls, the full architecture has a simple shape: fan out to context-building tools, then fan in to generate the final output.
The retrieval layer might be a search engine, a vector database, an SQL query, a local file tool, a web search API, or a custom service. The pattern stays the same: build candidate context, narrow it, rank it, assemble it, then generate from it.
Give agents human search controls
Semantic search compares embeddings (numerical representations of meaning). It helps when wording differs, but most retrieval intents also depend on structured constraints. A meeting search box can use semantic search over transcripts and notes, but a useful interface also lets someone filter by person, date, project, and source.
A finance search may need the latest filing, a specific quarter, or an official source in addition to the closest semantic match. In e-commerce, the best semantic match for “32×30 cargo pants” may be an out-of-stock product. The system still has to decide whether to hide it, return it with a backorder note, or show it so the user can check later. That product decision is a retrieval decision because it changes which candidates reach the agent.
In a chat interface, those controls are in the tool schema, query planner, or app logic. If an agent runs the search, it needs arguments for the same constraints a human would set with filters, sliders, tabs, and sort menus.
A retrieval system usually needs several controls working together:
Find EADDRINUSE, Authlib, or a specific SEC accession number.
Semantic match
Finds related content when the wording differs.
Find the meeting where the team discussed authentication tradeoffs.
Hard filters
Removes invalid results before ranking.
Limit by tenant, permissions, person, date range, size, or stock status.
Sorts
Orders candidates by a structured field.
Prefer the newest, latest filing, lowest price, highest rating, or recency.
Ranking
Scores candidates based on their likely usefulness for this request.
Combine semantic match, exact match, freshness, source quality, and use.
Reranking
Uses a slower model or scorer on a smaller candidate set.
Compare the query against the top 100 candidates before returning 10.
Here, a chunk means a small piece of source content, and a candidate is a chunk returned by the first search step. Ranking is the scoring step that orders those candidates. Context assembly then selects which chunks and structured fields to include in the model prompt.
Better ranking improves precision, which means a larger share of the returned chunks is useful. If the relevant chunks are near the top, the system can pass fewer chunks to the model, use fewer tokens, reduce latency, and expose the model to less noise. If the right chunk is ranked 40th and the context only includes the top 10, the system behaves as if the retrieval missed it.
People and agents use the same basic search path: ask for results, inspect what comes back, and decide what to use. A person can skim ten search results, compare titles, snippets, dates, domains, and URLs, and decide whether the result set looks right. They can open the third result, ignore the rest, and search again with a better query. An agent usually receives a bounded set of returned documents and reasons from the context. If the right source falls below the cutoff, the agent may answer from partial context. To avoid this, the system has to retrieve more candidates, run more searches, or pass more evidence into the model.
A missed document can change what the agent searches for next. Suppose someone asks why the team chose Authlib. If the first search misses the transcript where the team compared Authlib with alternatives, the agent may search the codebase instead. It finds imports, callback handlers, tests, and maybe a comment. Then it asks follow-up questions about OAuth configuration. The context starts to look complete, but it supports the wrong answer. It explains how Authlib was used and why it makes sense in the codebase, not why the team chose it.
“The context starts to look complete, but it supports the wrong answer.”
But ranking cannot repair every search problem. If the agent failed to request the latest filing, a reranker may faithfully select an older document with a closer wording match. If the tool has no date_range, person, source_type, size, or in_stock argument, the model has to impose hard constraints in the search text and hope that retrieval infers them. A hard filter gives the model less to infer, making semantic search more reliable.
Scale changes the retrieval problem
Search systems already have tools for this: indexes, filters, facets, sorts, caching, bounded reranking, and freshness jobs. Agent systems need the same discipline.
A human might search, adjust a date filter, scan the first page, then search again. One agent request can do that many times in seconds: rewrite the query, run keyword and semantic search, inspect thin results, issue follow-up searches, fetch sources for citations, and ask for more context before answering. With many concurrent users or agents, the retrieval layer can become a bottleneck.
Humans often wait through a slow search if the result is good. Agent systems often turn slow and uncertain search into more work. When ranking is weak, teams compensate by raising top_k, running keyword and semantic searches in parallel, adding reranking, fetching more source documents, and passing larger evidence bundles to the model. That can improve answers, but it moves the cost into tokens, latency, and retrieval load. A better ranking lets the system return fewer, better candidates, rather than making every request carry a larger pile of possible evidence.
With a small corpus, you can still search comprehensively quickly and cheaply, even with fully agentic approaches. That’s what I recommend when you’re starting and don’t have much data. Don’t add complexity until you need it. But with millions or billions of chunks, every extra retrieval call, candidate, ranking pass, and returned token adds up quickly.
Multi-stage retrieval is the production shape
Most production systems should split retrieval into stages, even when the UI is a chat box.
Stage
What happens
Trace question
Search argument construction
The app or agent turns the request and state into a query, filters, and sort.
Did it ask for the right content with the right constraints?
Candidate generation
The system finds plausible chunks from text, vectors, or structured data.
Did the right source enter the candidate set?
Filtering
Permissions and product constraints narrow what can be returned.
Was the source correctly excluded or wrongly lost?
Sorting
Structured fields order results when order matters.
Was the latest, cheapest, highest-rated, or current item surfaced?
Ranking
The system scores the candidates based on their usefulness for this request.
Was the source present but ranked too low?
Summary return
The system returns only the fields the agent needs.
Did the app receive usable evidence and provenance?
Context assembly
The app selects, formats, and budgets evidence for the model.
Did useful evidence get dropped before generation?
Evaluation
Humans or automated checks label whether the retrieval path worked.
Can the team turn the failure into a specific fix?
Each stage leaves a different repair path. If the agent chose the wrong filters, changing the embedding model will not help. If the latest document was available but the tool never sorted by date, the fix belongs in the search arguments or retrieval API. If the right source was present but below the cutoff, the fix belongs in the ranking. If the right source came back but was dropped before generation, the bug is in context assembly.
As retrieval becomes a core part of agent architecture, teams increasingly need infrastructure that can combine semantic search, exact matching, filtering, ranking, and large-scale retrieval in a single system. Depending on requirements, this may involve search and retrieval platforms such as Vespa, Elastic, or Coveo, each of which supports different approaches to ranking, retrieval, and operational scale.
The important point is not the specific technology choice, but recognizing that retrieval quality has become a first-class engineering concern. As agent workloads grow, retrieval systems are increasingly determining the accuracy, cost, latency, and reliability of the overall application.
OpenAI announced ChatGPT Work on July 9 and began rolling it out to Pro, Enterprise, and Edu users. It runs on the new GPT-5.6, opens a user’s local files, edits Google Workspace and Microsoft 365 documents, and carries a multi-step task through to a finished deliverable.
Reuters placed it directly against Anthropic’s Claude Cowork, and both target the same person — a non-coder who wants the power of a coding agent without the terminal. Counting Anthropic, Microsoft, Perplexity, and Amazon, five leading labs have now released an agent of this kind. The batch shows that the newest agents are organized more by their intended users than by their functions.
Based on the target user persona, four archetypes appear: the knowledge worker, the power user who self-hosts, the developer, and the enterprise. The personas often overlap since one individual can embody all three roles. Additionally, a product like Claude Code caters to both solo developers and platform teams.
Therefore, consider these deployment archetypes categorized by the main buyer, rather than strict separations. The archetype only represents what marketing promotes. Behind the scenes, each lab has almost consistently decided who owns the runtime, persists memory, manages credentials, and enforces policy.
Four archetypes, based on the user persona
Let’s analyze each of the four archetypes individually, as each differs in the level of control available to users.
The first archetype serves the knowledge worker. A vendor operates the runtime and sells the agent as a delegation to someone who lives in documents rather than code.
ChatGPT Work is the newest, alongside Claude Cowork, which now runs cloud sessions on web and mobile while keeping local-file access on the desktop; Microsoft’s Copilot Cowork, a cloud-hosted agent that executes long-running tasks inside the Microsoft 365 trust boundary; Perplexity Computer, which works across local files and Microsoft apps; and Amazon Quick, the successor to Q Business as that product closes to new customers at the end of July. The user grants access and supervises the result. In most cases, the vendor manages the runtime and persisted state, except for Perplexity’s local option.
A second archetype belongs to the power user who self-hosts. The provider controls the persistent agent process and chooses where to store the state and credentials, often on a Mac mini that has become a piece of personal infrastructure in its own right.
OpenClaw and Hermes are the reference examples, open source, and run on the operator’s own machine. Self-hosting involves managing the control plane rather than full local custody, since both options still allow access to a hosted model and the storage of credentials for external services.
Related reads:
“Microsoft has proved it can survive major changes in the tides of technology… Today, it faces another evolution in one of its core cash cows, as late-stage unicorns and AI labs alike push deeper into Office territory.”
The developer gets the third archetype, whose runtime spans the IDE, the terminal, the repository, and a cloud sandbox. Claude Code, OpenAI Codex, GitHub Copilot in agent mode, and the open-source OpenCode all live here, and Amazon’s developer agent is folding into its Kiro tool. Coding-agent execution is extending from the local IDE to vendor-managed sandboxes and asynchronous cloud workers, making this the most challenging archetype to categorize clearly.
The fourth archetype is built for enterprise workflows and integration with business processes. They run an open agent framework, such as LangGraph or CrewAI, on a managed, governed runtime. ADK on the Gemini Enterprise Agent Platform, Strands on the Bedrock AgentCore, Microsoft Agent Framework on Foundry Agent Service, and Claude Managed Agents belong to this category. OpenAI’s Agents SDK can be hosted on some of these runtimes, including AgentCore, which AWS lists as one of its supported frameworks. The vendor operates the infrastructure, and the customer configures identity, policy, and retention on top of it.
Customer defines identity, policy, and retention; platform brokers
Programmable platform
The line below the personas
The persona-based approach abstracts four things: where execution runs, where state is persisted, how authority is delegated, and where policy is enforced.
Anthropic describes its design as decoupling the brain from the hands. The harness that calls Claude runs separately from the sandbox where code executes, and a session, an append-only log of every model call, tool call, and result, connects the two. Because the sandbox is kept separate from the brain, the agent can start reasoning before any container exists, and the code it runs remains far from the developer’s credentials.
The same four planes show up at the other vendors. AgentCore Runtime gives each session a dedicated microVM with an isolated CPU, memory, and filesystem, and meters compute usage. Google can route governed traffic through its Agent Gateway, where Model Armor policies inspect configured ingress and egress flows, while Agent Identity and an Agent Registry track the fleet. Microsoft assigns each hosted agent a dedicated Entra Agent ID and runs it in a per-session sandbox whose filesystem survives idle periods.
Deploying agents on a managed runtime is more like leasing a workshop than purchasing a tool… a long-term tenant installs their own locks, maintains their records, and takes their tools when the lease ends.
Deploying agents on a managed runtime is more like leasing a workshop than purchasing a tool. The landlord manages the building and supplies the power, but a long-term tenant installs their own locks, maintains their records, and takes their tools when the lease ends. The personas often conceal this difference. Currently, the vendor typically operates the runtime, so the key question is how much control the customer can still exert and what they can take away across the four planes.
Where the line falls
What differentiates each offering is not the compute operator, since vendors handle nearly all of it. It is how much of those four planes a product leaves the customer to configure and export. A packaged experience hands nearly all four to the vendor and returns supervision and a finished outcome. A programmable platform operates the infrastructure but lets the customer define identity, policy, and retention and move the code elsewhere.
Copilot Cowork shows that the two axes are separate. It is a packaged knowledge-worker experience, yet it runs on a governed enterprise platform and inherits Microsoft’s identity, compliance, and audit controls.
The persona sells the product, but the four planes decide the lock-in.
A product can be packaged on the surface and programmable underneath, which is why the personas and the control planes have to be read as different questions. ChatGPT Work makes the same point from the developer side, since OpenAI’s new desktop app folds Chat, Work, and Codex into a single surface, though OpenAI has not detailed how far the runtime or credential store are shared beneath it. The persona sells the product, but the four planes decide the lock-in.
Usecase
Agent Type
Tradeoff
Delegate a knowledge task to an agent you supervise
A knowledge-worker agent such as ChatGPT Work, Copilot Cowork, or Amazon Quick
The vendor typically operates the runtime and persists state, and you configure little below the surface beyond access and approval
Keep state and credentials on hardware you control
A self-hosted agent such as OpenClaw or Hermes, in a local configuration
You control the persistent process, though model inference and some tools may still be remote
Ship code changes across the IDE, repo, and CI
A developer coding agent such as Claude Code, Codex, or Copilot
Execution spans your tools and a cloud sandbox, so ownership is split and worth mapping before you commit
Run many governed agents with audit and identity
An enterprise runtime platform such as AgentCore, Agent Platform, Foundry, or Managed Agents
The vendor operates the infrastructure while you define identity, policy, and retention, in exchange for coupling workflow logic to one cloud
Real deployments combine the rows rather than picking one. Teams on Foundry Agent Service commonly run open-source orchestration, such as LangGraph, for agent logic while leaning on the platform for governed execution, and Microsoft’s own hosted runtime now supports long-running personal agents like OpenClaw and Hermes with durable state. The boundary between experience and platform is not a thick, well-defined boundary, but a thin line a single system can cross, bridging rival camps.
The agent market is not being decided by open against closed. Open frameworks like Strands, ADK, and Microsoft Agent Framework are precisely what the governed runtimes are built to host. Vendors now manage the runtime across nearly all archetypes, shifting the competition to who can most effectively configure and export state, identity, and underlying policy.
If coding agents enter managed sandboxes alongside knowledge-worker agents, the developer archetype will be established on the platform side. The map will then transform into what the vendors are already outlining. When enterprise teams assess agents, their most important question should be how much of execution, state, identity, and policy they can configure and extract from the product. They should not focus on which persona it presents or whether its framework is open, as the framework no longer determines the product’s value or locking-in capabilities.
Thomas Dohmke, who stepped down as GitHub’s CEO last year to become a founder again, is opening a preview of a distributed Git network on Wednesday that is designed to keep fleets of AI coding agents from overwhelming a single central server — and one that may soon compete directly with GitHub’s core service.
Entire, Dohmke’s post-GitHub startup, is launching a preview of this on Wednesday (but for now, it is behind a waitlist). With this, developers can mirror an existing GitHub repository onto Entire’s own infrastructure in one step.
“In the era of agents, centralized Git hosting has become a fundamental constraint, as the strain of billions of agents and developers hammering a central server shows up in the form of rate limits, high latency, or even outages,” says Dohmke in today’s announcement. “Today, we begin to return Git to its original promise, with a distributed, and soon fully decentralized and open-source network of interconnected nodes around the world. By doing so, we enable any developer or agent to host their code in-region, pushing, pulling, and cloning close to where they operate, fast and without bottlenecks, while still part of a global, collaborative network.”
The key here is that the code stays on GitHub, as Entire stresses, but coding agents can work with the Entire mirror and, as the company notes, “build without rate limits.”
Entire’s mirror is meant to absorb the constant flow of traffic that a fleet of agents can generate. That traffic, after all, is part of the reason GitHub is often buckling under pressure these days and startups like Entire have an opening.
Centralized Git hosting, Dohmke says in an interview with The New Stack, has become “a fundamental constraint” now that billions of agent and developer operations land on the same servers, showing up as rate limits, latency, and outages.
Given GitHub’s recent availability issues, it’s no surprise that startups are trying to get into this space. Entire is one — and it has the pedigree — but in June, Cursor also announced Origin, its own Git forge rebuilt for swarms of agents that are cloning and committing against a single repository in parallel.
Entire is starting with active regions in the United States, the European Union, and Australia, but the team says that now that is has spun up its first few regions, it will add more soon.
‘Git as a database’
To build its network, Entire rewrote the server part of git. GitHub, GitLab, and Bitbucket all wrap the server-side of the Git binary and build their infrastructure around it. Entire started from scratch.
“We see Git really as a database,” Dohmke says. The open source Git project has two halves, he explains: the client that an agent uses to talk to a repository, and the server a host runs to manage storage. Rather than build on that stock server, like most companies would do, “we made the decision of not going that route, and instead implemented our own Git backend.”
That only makes sense if Entire’s version has significantly better performance than the stock Git server, of course. Entire says its benchmarks have pushed the network to a sustained rate of 570,000 clones per hour, 586 pushes per second, and roughly 470 combined clone-and-push operations per second.
Pushing to a native Entire branch can run up to 25 times faster than pushing through to GitHub, Dohmke says.
Entire it will open-source both the git backend and the benchmark suite.
The foundation layer, now real
When Dohmke first described Entire’s plans to The New Stackin February, he described a three-layer platform that included a Git-compatible database at the bottom, a semantic reasoning layer in the middle, and an interface on top. Even then, he said that the database, unlike a centralized Git host, could be a globally distributed network of nodes.
But in February, Dohmke also said Entire wouldn’t necessarily end up competing with GitHub, and that code repositories would stay central to the pitch.
Pressed on whether that still holds now that Entire hosts its own copy of the GitHub repo, he calls the mirror complementary, in part because Entire can offer enterprises the ability to keep their code in a local region to fulfill local regulations. He also notes that GitHub has a huge ecosystem and an extended feature set.
“I think the question for the buyer really is, is it not better for me from an availability and reliability perspective, that I have both of these products, so if one of them is down — there’s always going to be single points of failure and human errors — then I have my mirror on the other side,” he says. “But we certainly will, in deals, compete for the dollar spent at a much smaller scale compared to the multi-billion-dollar business that is GitHub today.”
Credit: Entire
For now, that keeps the two complementary. Dohmke argues that GitHub remains the “source of truth,” or “cold storage,” while the working copy lives on Entire. But he also says that Entire will launch native repositories in the coming months, and those wouldn’t need GitHub underneath at all. All of this will be open-sourced as well.
Entire raised its $60 million seed round in February, when it had 15 employees. Felicis led it, with Microsoft’s venture arm among the backers. The company is now past 40 people and aiming for 60 by the end of the year.
Entire beyond Git: the semantic memory layer
Entire is building its middle layer — the semantic reasoning layer — in parallel with the Git platform.
The semantic layer now integrates with every major coding agent, including Claude Code, Codex, Cursor, Factory AI, and GitHub Copilot, and records each session, prompt, and tool call in the repository alongside the code.
Having this data is useful for agents, and it was the first core service the company launched. Now, it is also building more services on top of that history.
The company is adding Entire Blame, for example, which shows not just who last touched a line but the agent session and prompt behind it. There is also Entire Review, which fans out several agents for an intent-aware review, and the company is adding a code and semantic search feature that lets agents (and developers) search across code changes and the reasoning that produced them.
“Session logs are now the second most important artifact in software development, and they belong in the repository alongside the code,” Dohmke says.
Industrial machinery generates more alarms than technicians can triage. For each important alarm requiring follow-up, the technician pulls historical context,...
Industrial machinery generates more alarms than technicians can triage. For each important alarm requiring follow-up, the technician pulls historical context, determines the correct procedure, checks whether a specialist signal confirms the failure mode, and writes up a recommendation. This process remains consistent, and is well-suited for an AI agent. This post discusses a per-alarm…
AI agents are changing how you interact with your PC. Creators, developers, and AI enthusiasts are already using these agents extensively to assist with...
AI agents are changing how you interact with your PC. Creators, developers, and AI enthusiasts are already using these agents extensively to assist with day-to-day tasks such as coding, video editing, and content management. NVIDIA and Microsoft are teaming up to enable the next generation of developers to build on-device agents on the Windows platform, with easier setup, native security…
Remember Rosie the Robot from The Jetsons? She wasn’t just a housekeeper. She was a glimpse into a future where technology could hold a conversation, help with everyday tasks, and feel almost part of the family.
Rosie could handle her routines with ease, but the real promise was what she represented: a world where machines could engage in authentic dialogue and adapt both the conversation and the tasks they perform to meet the moment.
In many ways, customer service AI has been chasing that vision. But most solutions are still stuck in the early stages.
Login
Industrial machinery generates more alarms than technicians can triage. For each important alarm requiring follow-up, the technician pulls historical context,...
AI agents are changing how you interact with your PC. Creators, developers, and AI enthusiasts are already using these agents extensively to assist with...
