Finding the most capable model at the lowest cost has always been the goal. But as agentic AI evolves, a new problem is frustrating engineers: token consumption is becoming too high across AI systems. Every agent operation consumes tokens, but in very different ways. For instance, a moderately complex agent request can consume 20,000 to 60,000 tokens across its reasoning chain, but a nontrivial engineering task can burn 150,000 to 200,000 tokens per problem.
Every agent operation consumes tokens, but in very different ways.
Developers are realizing that selecting the right model is important, but the bigger issue is limiting unnecessary token movement throughout an agent’s workflow. That is why teams are starting to consider how to accomplish the same tasks with fewer tokens.
Compounding costs across agents
The costs add up quickly. A task that takes about 50,000 tokens with one agent can easily consume several hundred thousand with multiple specialized agents together. That’s because each one needs enough context to do its job. It’s not unusual for an agent to process 30,000 tokens of context just to return a 500-token response, and those exchanges add up over the course of a workflow. Each handoff effectively pays a tax in input tokens that compounds with every loop iteration.
Each handoff effectively pays a tax in input tokens that compounds with every loop iteration.
This is especially noticeable among multi-agent architectures. When one agent delegates to another, it must encode its current state and task instructions into the downstream agent’s context window. The receiving agent processes all of that, produces a result, and passes it back. Then the orchestrating agent reingests it alongside everything else it’s tracking. Every exchange adds another layer of overhead.
Building more token-efficient architectures
A growing collection of strategies addresses this problem. Three practical solutions stand out:
Compress context, preserve reasoning
The most direct solution is to reduce the amount of context an agent carries from step to step. Rather than accumulating an ever-growing interaction history and replaying it with every task, systems can summarize earlier portions of a conversation or working memory before passing them forward.
One way to do that is by narrowing the agent’s field of view. Instead of handing it an entire codebase or document collection, the system surfaces only what’s relevant to the task at hand. Go too far, though, and the agent can lose important context that it will need later.
To make this work, the system needs a compact memory layer of key facts and decisions alongside the compressed context. The agent needs to recall the reasoning chain without having to reread it each time.
Route tasks to cheaper models
Hierarchical routing lets engineers parse a JSON response, format a log entry, or check whether a file exists without using the same model used to architect a system design. It assigns each subtask to the smallest model that can reliably do the job. A lightweight model, for instance, handles routine classification, extraction, and formatting steps, while a more suitable model handles decisions that truly require deeper reasoning.
So if 60 to 70 percent of an agent’s steps are routine operations, then a smaller model can handle the work at a fraction of the cost, substantially reducing the overall token spend for the workflow.
Cache reasoning, skip redundancy
That’s where semantic caching comes in. Instead of solving the same problem twice, it compares the meaning of a new request using embeddings. If the match is close enough, the agent can reuse earlier work instead of generating a new reasoning chain.
The savings make a difference, especially in scenarios involving customer support systems that answer similar questions all day, or even document-processing pipelines that handle thousands of nearly identical files. In such scenarios, reusing prior reasoning can significantly reduce the number of tokens an organization consumes.
Measuring what matters
But these workflows only deliver value if teams see their impact, and many organizations are still figuring out how to measure them effectively. Poorly designed agent loops or inefficient multi-agent handoffs can dominate costs in ways that are invisible when you’re only looking at per-request pricing.
It’s also easy to focus too much on tokens. Running an agentic application also means paying for GPUs, memory, vector databases, and the tooling needed to monitor everything in production. Saving tokens helps, but it won’t solve the whole problem if the rest of the stack is still expensive.
The next phase of AI infrastructure
The next generation is increasingly focused on building systems based on better architectural decisions. That means context management, model invocation, task decomposition, and intermediate work reuse are becoming just as important as inference pricing or even benchmark scores.
Model capabilities will continue to improve, and inference costs will likely fall. But if autonomous agents become the dominant way organizations build AI applications, the systems that scale most effectively may not be the ones with the cheapest models, but those that waste the fewest tokens.
The systems that scale most effectively may not be the ones with the cheapest models, but those that waste the fewest tokens.
Cyberattacks used to be the biggest security issue surrounding AI infrastructure, but that could be changing. A recent cargo theft outside Chicago suggests another vulnerability — and it’s one that has nothing to do with malware or prompt injection.
Just last week, the Cook County Sheriff’s Office recovered two stolen trailers containing roughly $1.3 million in data center equipment and copper wiring, taken from separate shipments originating hundreds of miles away. One trailer held about $300,000 worth of copper wire — reported stolen in Pine Hill, Alabama — destined for data center construction. The other carried roughly $1 million in data center infrastructure equipment, stolen out of Jacksonville, Florida. Both ended up at the same truck yard in Elk Grove Township, outside Chicago.
Viewed in the context of the AI boom, it highlights that the physical supply chain itself is becoming a new target for bad actors.
Viewed in the context of the AI boom, it highlights that the physical supply chain itself is becoming a new target for bad actors.
A new high-value cargo
We’re all familiar with typical bottlenecks like GPU shortages, power constraints and cooling capacity, which have plagued the AI era since its inception. But we forget that building an AI data center requires an enormous volume of specialized hardware moving through freight networks. These include servers, networking gear, fiber, switchgear, cooling systems, power distribution equipment and thousands of pounds of copper. Each represents capital investment and potential deployment delays.
As hyperscalers accelerate the construction of data centers, the exposure of these items between the factory and data center creates a risk category that the industry as largely ignored.
When one delay cascades
Large GPU clusters depend on the synchronized delivery of dozens of interconnected systems. A training cluster is a tightly coupled system of servers, switches, optics, power distribution, and cooling that must be installed together. Missing networking hardware can idle racks, delayed power equipment can postpone an entire deployment and stolen copper can stall electrical work. So when one component category disappears, the delay cascades across everything.
So when one component category disappears, the delay cascades across everything.
Cargo theft by the numbers
Infrastructure resilience increasingly depends on whether critical hardware arrives at the construction site at all — and on schedule. Verisk CargoNet reported that U.S. and Canadian cargo theft losses jumped roughly 60% in 2025 to nearly $725 million, even as the total number of incidents held essentially flat — a sign that thieves are becoming more selective about high-value freight. Metal theft rose 77%, driven largely by demand for copper, while organized groups shifted toward enterprise computing hardware. CargoNet expects that focus on high-value technology — RAM modules, storage drives and enterprise computing equipment — to carry into 2026. For broader context, the Department of Homeland Security has estimated that cargo theft overall costs as much as $35 billion a year.
The Chicago incident fits squarely inside that trend.
Beyond firewalls and malware
Obviously, cargo theft isn’t an engineer’s problem. But organizations building AI infrastructure may need to broaden their thinking about deploying AI capacity on aggressive timelines.
Cloud providers, colocation operators and hardware vendors have already invested heavily in defending infrastructure from digital threats. As AI infrastructure becomes more valuable, protecting the physical systems behind it may deserve similar attention.
The next supply-chain conversation
The AI boom has already forced the industry to rethink electricity, cooling, networking and semiconductor manufacturing. Physical logistics may be next.
It starts long before the equipment reaches the data center.
If the value of AI infrastructure continues to climb into the billions of dollars, the industry’s definition of “infrastructure security” is likely to expand beyond firewalls and identity management. It starts long before the equipment reaches the data center.
Microsoft’s latest AI services announcement suggests the era of standardizing on a single model may be ending. This week, the company launched a $2.5 billion AI adoption business designed to help enterprises customize AI deployments and use multiple models rather than lock themselves into a single provider — part of a larger shift toward systems that route each request to the model best suited to the task.
Simply put, the company that arguably has the deepest single-model partnership in the industry is now selling model swappability as the product.
Betting $2.5 billion on flexibility
Microsoft said Thursday it is creating a new operating entity, Microsoft Frontier Company, to help corporate customers select AI technologies that actually work for their businesses and produce a return on investment, Reuters reported. The unit launches with $2.5 billion in funding from Microsoft and will work with customers including Unilever and Novo Nordisk.
The new firm will help customers choose and integrate AI tools — from Microsoft and external providers — with each customer’s internal data. Customers will own the results of that work rather than handing it back to Microsoft. The move puts Microsoft alongside Palantir, which is doing similar work with large customers using Nvidia’s open-source models, and Amazon Web Services, which recently launched a $1 billion embedded-engineering unit of its own.
What’s most telling is the reasoning. Judson Althoff, CEO of Microsoft Commercial Business, told Reuters the new firm grew partly out of Microsoft’s own experience watching models like DeepSeek and Google’s Gemini catch up to OpenAI. Referring to the original Copilot, he said, “we made a mistake by binding it to OpenAI models only.” Customers, Althoff said, care more about the combination of their data and the models than about any particular model — and they need the ability to swap models quickly as the state of the art shifts.
“We made a mistake by binding it to OpenAI models only.”
One model no longer fits
Consider a typical customer service application that might need to summarize a support ticket, analyze a 300-page contract, generate an email, transcribe a meeting and review source code. Those aren’t necessarily the same problem. A model like Google’s Gemini, with a context window of a million tokens or more, may be the right choice for the contract. A small, fast model like OpenAI’s GPT-5.4 mini or Anthropic’s Claude Haiku may handle ticket summaries at a fraction of the cost. The transcription may go to a purpose-built model like Whisper. And if regulators require customer data to stay on-premises, an open-weight model like Meta’s Llama or Mistral is often the preferred choice.
Instead of choosing a single foundation model, developers increasingly choose several — and the application decides which one handles each request.
AI gateways become core infrastructure
The model is just one component of the stack, so the decision to route the request has to live somewhere. That’s why developers are forgoing hard-coding an application to a single model and building systems that can choose among several. The routing logic might prioritize cost for one request, speed for another, or keep sensitive workloads on a local model. That way, if one provider experiences an outage, traffic can be routed elsewhere without changing the application itself.
The company that arguably has the deepest single-model partnership in the industry is now selling model swappability as the product.
That changes what developers build
Once companies stop relying on a single model, the challenge shifts to building the systems that decide which model to use for each request.
That means developers need tools to route requests, compare model performance, monitor reliability, control costs, enforce security policies and switch to another model if one goes down, which is a very different engineering problem, especially since deciding which model should respond to a request occurs every time someone uses your application. At enterprise scale, those decisions happen millions of times a day, so they have to be fast, reliable and easy to manage.
The ecosystem is already responding
Open-source proxies like LiteLLM and gateways like Portkey normalize APIs across providers. Orchestration frameworks such as LangChain and LangGraph assume the presence of multiple models from the start. The Model Context Protocol (MCP) is making tool integrations portable across models rather than bound to one vendor. And the cloud providers themselves — Amazon Bedrock, Azure AI Foundry, Google Vertex AI — now expose many models behind a single API.
Orchestration is the new moat
Core models will keep improving. But as performance converges for many business tasks, orchestration becomes the challenge. Microsoft’s announcement is one indication that the largest vendors believe enterprises are heading in that direction and are willing to spend billions to be the ones holding the routing layer.
Instead of treating the model as the platform, enterprises are consistently treating it as a replaceable component behind an orchestration layer.
The cloud era taught developers not to tie applications too tightly to one server; containerization made infrastructure portable. Now the same philosophy is being applied to AI. Instead of treating the model as the platform, enterprises are consistently treating it as a replaceable component behind an orchestration layer.
When Subquadratic launched earlier this year, it could build a sparse-attention model that could handle a 12-million token context window and be significantly faster than today’s large language models. But it didn’t launch the model widely and it didn’t publish benchmarks.
Given the company’s large claims, that created quite a bit of skepticism. In June, Subquadratic published its first model card and benchmarks for its small model, SubQ 1.1, supplied third-party verification from data firm Appen, and started talking about its first design partners who now have access to its model.
So far, however, few people have actually used its model. To talk about the company, why its model isn’t widely available yet, and what it has in store for the near future, we met up with Subquadratic co-founder and CTO Alex Whedon.
“We’re not a sparse attention company either.” — Alex Whedon, Subquadratic.
One thing Whedon definitely wanted to clear up is that the company’s current model may be based on sparse attention, but that isn’t its full mission.
“We’re not a sparse attention company either,” Whedon tells The New Stack. “We’ve been working on non-attention architectures for quite a while as well. We think that we will be the first people to leapfrog ourselves in terms of the next model architecture.”
We’ll get back to that.
What the model card shows
It’s the company’s SubQ 1.1 Small model that people are talking about now. This model is built on Subquadratic Sparse Attention (SSA), an attention mechanism the company says scales close to linearly with context length instead of quadratically.
“In the case of Subquadratic Sparse Attention specifically, which is one of a couple model architectures we worked with, the idea is that not all of the token relationships matter,” Whedon explains. “Token relationship compute is why you see this quadratic scaling law.” This means there are almost a million possible two-token relationships in a 1,000-token input in a full attention matrix.
For SubQ 1.1 Small, the strongest results are in long-context retrieval, which makes sense, given that this is where the architecture should have its biggest edge.
Credit: Subquadratic.
On the needle-in-a-haystack test, SubQ 1.1 Small scores near-perfect from 1 million tokens out to 12 million, even though it was trained mostly at 1 million. It hits 99.12 percent on Nvidia’s harder RULER test, which asks the model to trace and aggregate facts across a 128,000-token context rather than just find one.
On general capability, it lands just below the mid-tier frontier models, at 85.4 on GPQA Diamond against 87.5 for Sonnet 4.6. On the LiveCodeBench coding benchmark, it scores 89.7, below Opus 4.8 and GPT-5.5, but slightly better than Sonnet 4.6.
Efficiency is where the model shines, though. The company says that at 1 million tokens, SubQ uses 64.5x less compute than dense attention and runs 56x faster than FlashAttention-2 on a single attention layer. At the full 12-million-token window, it puts the attention compute reduction at close to 1,000x.
Credit: Subquadratic.
“Even in full dense attention, the relative importance of over 99 percent of tokens is very low, attention scores are below 0.1,” Whedon says. “We actually show this in our model card. So clearly we’re just wasting compute most of the time, and in fact we’re maybe making the modeling task harder, because we’re introducing noise.”
“Transformers are a brute-force approach to the problem of text modeling,” he says. “You could say, ‘I’m going to compare every single individual token to every other possible individual token.’ That’s what transformers do. Very brute force, very naive. It just assumes that the first needs to look at the second, the third, the 50th, and the 5,000th. That’s not how humans read text.”
SSA also differs from retrieval-augmented generation, which drops chunks of text before the model sees them. “Every token of the text is being seen by the model,” he says. “It’s just not being redundantly compared to every other token of the text.”
On capability, SubQ 1.1 Small lands roughly in Sonnet 4.6 territory, sometimes a bit above, sometimes below. But its edge, the company says, is size and cost.
“What we posted publicly was fewer than 100 billion parameters,” Whedon says about the size of the model. “I would venture to say that our model is smaller than any of the models offered by OpenAI or Anthropic. But our next model will not be.”
Smaller, cheaper, built for enterprises
Subquadratic is also making the pitch that its model’s capabilities will be especially interesting for enterprises.
“We think that’s a pretty interesting enterprise offering,” he says. “We’ve seen a lot of people in the enterprise space talking about using the mid-tier models as opposed to the frontier for large data-processing tasks, which is exactly where we’re trying to plug in.”
Given that a lot of enterprise problems start with searching through large heaps of data, this makes sense. You can pack a lot of documents into a 12-million token context window, after all. Most of today’s models break down well before the user fills their million-token windows, but with its near-perfect retrieval scores, SubQ may be a good answer for these problems.
As Whedon noted, the model’s first users are design partners, not the public. “We’re giving access to the model to design partners now, and these are mostly enterprises, largely with eight- to nine-figure spend,” Whedon says. “This is a core market that we really care about. It has been since day one.” A limited individual-access release will follow before any general availability.
The launch led with claims instead of benchmarks by choice.
“We were announcing mostly research,” he says. “We could have maybe messaged the launch a little bit differently. There was some debate about how we were going to message it.”
Built on an existing model
One question from May hasn’t gone away, though. The model card states that Subquadratic “started with an existing open-weight frontier model by replacing its dense attention with Subquadratic Sparse Attention (SSA),” and then ran roughly one trillion tokens of long-context continued pretraining on books, documents, and repository-scale code.
That confirms what some of the skeptics suspected at launch, when OpenAI researcher Will Depue wrote that SubQ was “almost surely a sparse attention finetune of Kimi or DeepSeek.” What’s new here then is the SSA mechanism and the long-context training recipe, not a model trained from scratch. The company has not said which open-weight model it started from.
The biggest lever on long-context retrieval was pretraining on very long sequences, Whedon says, something SSA’s efficiency made cheap enough to run as routine.
“Nobody’s talking about multimillion-token pretraining,” he says.
Credit: Subquadratic.
Why hybrids don’t go far enough
There have, of course, been attempts to improve on quadratic scaling, but Whedon thinks most of those attempts only go — almost literally — halfway. Hybrid models such as Nvidia’s Mamba-based Nemotrons, Qwen’s Gated DeltaNet layers, and the various linear-retention designs swap out some of the attention layers, but they don’t go all the way.
“If 80 percent of the layers are not quadratically scaling, then your maximum payoff is like a 5x increase as you scale toward infinity,” he says. “We see a 60x increase at 1 million tokens, almost 1,000x at 12 million. That is the type of payout that you only get if you actually change the scaling law, as opposed to a scalar win.”
He actually credits DeepSeek’s own sparse attention mechanism with making his company’s pitch easier.
Credit: Subquadratic.
“They showed that you could dynamically select relationships without a significant quality trade-off,” Whedon says. “However, they did so by redundantly using a smaller but still full-attention model that ends up using the vast majority of the compute at scale.”
Subquadratic ran its own benchmark against GLM 5.2. “At 1 million tokens, 58 percent of the prefill latency comes from that selection mechanism,” Whedon says. “So that selection mechanism, which is supposed to be seen as cheap, actually dominates the compute, because it’s a quadratically scaling component.”
Beyond sparse attention
It’s also why Whedon pushes back on the “sparse attention company” label. Subquadratic has been working on what he calls “zero attention,” architectures that drop the attention mechanism altogether.
“Attention is kind of similar to RAG in that you have queries, keys, and values that represent information about the tokens that you’re processing,” Whedon says. “There’s this discreteness of representation, where everything is represented within these nice little boxes. That’s super convenient. It’s easy to build a brute-force solution around it. But it also means your ability to compress information is limited. If you had a more continuous, abstract way of representing the information, then you could compress it further, which means you can make smaller models, or you could just scale things up again to create another leap in intelligence.”
He traces the idea to world models and to Yann LeCun’s work. “The stuff we’re doing takes a lot of inspiration from world models, not the video modality in this case, but some of the things LeCun is talking about,” he says. “Rethinking how to represent long-range dependencies, how to keep a long-range state, how to rethink the objective function.” He stops there. “That’s probably all I could say for now.”
Subquadratic has also marketed only one of the three kinds of efficiency it says it is chasing. “We care about compute, sample, and memory efficiency,” Whedon says. “We’ve done a lot of work on all three, but have only really talked about the compute efficiency publicly.”
The near-term plan
The near term plan for Subquadratic, however, is more modest. “Over time, yes,” Whedon says, when asked whether Subquadratic could rival OpenAI and Anthropic on raw quality in the long run. “In the shorter term, we have to be strategic. If we try to boil the ocean on much less capital, it’s not going to go well for us.”
The next model, he says, will likely be a mid-tier size rather than a frontier-class one (think SubQ 1.2 Medium), that he expects to outperform most of the competition in its tier.
How the team will bring the model to market, though, remains to be seen. I wouldn’t be surprised if the team launched its model on one of the hyperscaler’s large model platforms, but Whedon remained tight-lipped about the company’s plans.
The fact that we met with the Miami-based Whedon in San Francisco, though, gives you a bit of a hint of what the team is currently up to.
OpenClaw finally dropped its iOS and Android apps this week, meaning you can now ditch the Telegram and WhatsApp methods to talk directly to your personal AI agent. But what’s arguably more exciting is that the app isn’t actually running the AI on your phone. It’s just hooking up to an agent you’ve already got running somewhere else. Your phone now acts as a window into that agent, complete with voice, notifications, and camera access.
It’s a nice design choice, and exactly where personal AI agents are headed.
Phones become authenticated endpoints
The phone is basically becoming a really smart remote control for OpenClaw. Instead of cramming an increasingly powerful agent onto a phone with battery and memory constraints, developers are treating the phone as one more screen for an agent that lives elsewhere. The agent keeps working whether your phone is in your hand or charging in the other room.
Within this model, the phone approves actions, pings you with notifications, lets you talk to the agent, and shares your camera when the agent needs eyes on something.
Persistent runtimes replace mobile constraints
But OpenClaw isn’t the first to do this. Anthropic’s Claude Cowork with Dispatch follows a remarkably similar pattern. Users assign work from their phones, but execution occurs on a persistent desktop runtime. The mobile app acts as a companion for starting tasks, monitoring progress, and receiving results rather than becoming the agent itself.
OpenAI is moving in a similar direction as well. With Codex, developers increasingly interact with long-running coding agents that continue working independently and can be checked on from multiple clients, instead of treating the phone as the place where the agent runs.
Different companies, different products, but a similar architectural bet to keep the agent running in a persistent runtime and give people lightweight clients to interact with it.
When multiple teams independently converge on the same architectural pattern, it’s often an early signal that the industry has found a model that solves a real engineering problem.
The engineering problems are totally different now
This shift changes what developers spend their time thinking about. Building mobile apps used to mean worrying about battery life, memory limits, offline mode, and squeezing the best performance out of a phone. If the agent is running somewhere else, most of those concerns fade into the background.
Now, a new set of questions comes to mind, such as how a phone securely connects to a long-running agent? How do you manage permissions across multiple devices? What happens if every client disconnects but the agent keeps working?
Agent identity beyond login screens
There’s a downstream effect here. Once the phone is just one of several trusted endpoints talking to your agent, you need a much more robust approach to identity. You’re not logging a user into an app anymore. You’re authenticating devices into an ongoing relationship with a persistent agent.
As that agent gains the ability to read your files, send emails, call APIs, and control external tools, authentication becomes load-bearing infrastructure.
Distributed agents reshape developer tooling
Zooming out a bit and looking at the bigger picture highlights how personal AI agents increasingly resemble distributed systems rather than mobile apps. The intelligence lives in a persistent runtime while the phone is one authenticated endpoint among several.
For developers, the mobile app is only part of the job. They also have to build the components that keep an agent running, connect it to a user’s devices, and ensure those connections remain secure.
The agent keeps running independently, while the phone is simply another place to check in, approve actions, or start a conversation.
Looking at OpenClaw alongside Anthropic and OpenAI, it’s hard not to notice the same pattern. The agent keeps running independently, while the phone is simply another place to check in, approve actions or start a conversation. That architecture solves many practical problems, which may explain why several companies are heading in the same direction.
AI has changed the web right before our eyes. With Google’s AI Overviews doing the heavy lifting, publications that once owned the first page of search results are being replaced by summaries. Readers get their answer without ever clicking through. Much of the traffic has simply stopped.
Cloudflare on Wednesday announced a slew of updates for publishers who are facing this new reality. From new crawler classifications and analytics dashboards to Answer Engine Optimization tools and an expansion of its Pay Per Crawl program, it’s clear the company is trying to become the economic pipes of the AI web.
The shift from ‘keep out’ to ‘let’s make a deal’
A year ago, Cloudflare’s pitch was practically defensive, asserting that website owners should be able to block AI crawlers. And while that still holds true, the company has pivoted to discussing building “rails” for an “agentic economy.” And it makes sense. If AI agents are already browsing the web, collecting content, and in some cases buying things, someone needs to handle the business end of how the sites they visit are compensated. Cloudflare thinks that someone should be Cloudflare.
Paying for value, not visits
Roughly a year ago, Cloudflare launched Pay Per Crawl, which let publishers set a price for AI companies to pay when they fetched a page. Now the company is pushing toward Pay Per Use, which means publishers get paid when their content actually appears in an AI-generated answer.
To backtrack, under the old model, an AI crawler pays to visit your site, whether or not it does anything useful with what it finds. Cloudflare says it’s already testing this with Ceramic.ai and You.com, each running slightly different versions of the concept.
Instead of charging for access — which is basically a toll booth — publishers are charging for value.
The economics here flip. Instead of charging for access — which is basically a toll booth — publishers are charging for value. That’s closer to how affiliate marketing or licensing deals work, and it’s a much harder problem to solve. It requires knowing which content contributed to which answer, which means an attribution infrastructure that doesn’t really exist at scale yet.
Credit: Cloudflare.
Crawlers need clearer labels
But here’s where things get more technical — and a bit political.
Cloudflare wants AI companies to stop lumping all their crawlers together. Right now, a single bot from a major AI company might be fetching pages for search indexing, model training, and agent tasks all at once. That makes it impossible for site owners to say yes to one use and no to another.
Starting September 15, Cloudflare plans to change the defaults for new and free-tier sites. AI search crawling stays on, but training and agent access get blocked on ad-supported pages unless the site owner opts in. Mixed-use crawlers that refuse to separate their traffic get blocked entirely.
The company is clearly taking a shot at Google here, noting that Google’s bundled approach gives it access to roughly twice as much content as AI-native competitors. Separating crawlers’ intent has become a prerequisite for any kind of functioning market between publishers and AI companies. Because if site owners can’t distinguish between “index my page for search” and “train your model on my writing,” they’ll increasingly just block everything.
Separating crawlers’ intent has become a prerequisite for any kind of functioning market between publishers and AI companies.
Optimizing for AI answers
Cloudflare is also rolling out a dashboard designed for business teams, called Attribution Business Insights. Think of it as the AI equivalent of knowing your Google Search Console numbers, except the “search engine” is now ChatGPT or Perplexity or whatever agent your reader happened to ask.
On top of that, Cloudflare is introducing Answer Engine Optimization (AEO). The idea is that ranking in Google is no longer enough; for publishers to succeed, they also need to understand how and where their content gets cited in AI-generated responses. That’s a different optimization problem than SEO, and right now almost nobody has good tooling for it.
Infrastructure as competitive advantage
Cloudflare already sits between websites and the internet. It sees the traffic and knows what changed on a page and what didn’t. It can tell a crawler to come back later because nothing’s new, which, by the way, it says would eliminate over 50% of current AI crawl traffic.
That puts Cloudflare in a unique position. It’s already part of the path between AI companies and the web. Now it wants to become the layer that manages access, attribution, and eventually payments between them. If AI agents become the primary way people find information online, Cloudflare is betting the next battle won’t be over who builds the smartest model but instead over who builds the infrastructure everyone else relies on.
If AI agents become the primary way people find information online, Cloudflare is betting the next battle won’t be over who builds the smartest model — it’ll be over who builds the infrastructure everyone else relies on.
For two years, the AI infrastructure race has been dominated by one question: Who has the fastest GPU? Jim Keller thinks that’s becoming the wrong question.
In a recent interview with EE Times, the Tenstorrent CEO argues that the riskiest move an organization can make right now is optimizing its AI infrastructure for the models it’s running today. It’s not because those models are bad — but because they won’t be the models it’s running in 18 months. Keller invoked Rent’s Rule and Amdahl’s Law to argue that memory, networking, and system-level balance now matter more than peak floating-point performance.
Not because those models are bad — but because they won’t be the models it’s running in 18 months.
It sounds like the start of Keller’s product pitch, but there’s real weight behind it, because AI has evolved faster than the infrastructure underneath it. And the companies that spent hundreds of millions building around one generation of models are now staring down the cost of doing it all over again.
That fear is called lock-in, and it’s reshaping how the biggest players in AI think about hardware.
Workloads outgrew the GPU
In 2023 and 2024, AI infrastructure was a relatively simple procurement problem: Train large language models, serve them to users, and buy as many GPUs as Nvidia can ship. The workloads were predictable and GPUs handled them well.
Then AI outgrew the infrastructure it had been built for.
Reasoning models spend more time working through problems instead of jumping straight to an answer. Agents bounce between APIs, databases and code before completing a task. Multimodal models mix text with images, audio and video. None of those workloads stress hardware in quite the same way — and that’s forcing infrastructure teams to rethink assumptions that made perfect sense just two years ago.
AI outgrew the infrastructure it had been built for.
No single chip architecture handles all of that equally well. And the organizations building AI infrastructure are starting to realize that the question isn’t just which accelerator is fastest — it’s how do we build systems that won’t need to be torn apart every time AI takes another leap?
Nvidia is already selling one answer
Look at what Jensen Huang has been talking about, and it’s not GPUs anymore.
At GTC 2026, Nvidia unveiled the Vera Rubin platform — seven chips designed to operate as a single system: the Rubin GPU, Vera CPU, NVLink 6 switch, ConnectX-9 networking, BlueField-4 DPU, and more. The Vera CPU exists
for the CPU-intensive work of agentic AI — tool calls, code execution, orchestration. Nvidia calls these deployments “AI factories,” and the language is deliberate. They’re selling complete infrastructure, not individual accelerators.
When the company with 70% market share stops leading with GPU benchmarks and starts talking about system-level co-design, it tells you where the center of gravity in this market is moving.
That reframing matters. When the company with 70% market share stops leading with GPU benchmarks and starts talking about system-level co-design, it tells you where the center of gravity in this market is moving. Compute still matters. But Nvidia is conceding — through its product architecture if not its marketing — that raw accelerator performance alone won’t be enough for what’s coming.
Hyperscalers design their own silicon
AMD sees the same problem, even if it’s taking a different route. Helios brings together CPUs, GPUs and networking into one rack-scale platform, reflecting a broader shift away from treating the GPU as the center of the universe. The pitch isn’t “our accelerator is faster.” It’s that the infrastructure surrounding the chip increasingly matters just as much as the chip itself.
The hyperscalers have been making this argument with their wallets for even longer. Google has spent a decade co-designing its TPU silicon, interconnects and software framework — its seventh-generation Ironwood chip is now generally available — giving it unusual control over the full stack. Amazon went the opposite direction, building separate chips for separate jobs: Trainium for training, Inferentia for inference, with Trainium3 now in production and serving customers like Anthropic. Microsoft’s Maia 200 targets inference costs, while the company simultaneously deploys Nvidia’s Vera Rubin NVL72 for training and experimentation — arguably the most pragmatic dual-track strategy in the market.
Behind all of them sits Broadcom, whose AI semiconductor revenue recently crossed $10 billion in a single quarter, driven by staggering demand for custom accelerators and data center switches. Broadcom designs custom accelerators for Google, Meta and others while supplying the Tomahawk and Jericho switch silicon that connects those accelerators at data center scale. Custom ASIC shipments are projected to grow roughly 45% year over year in 2026 — triple the growth rate of merchant GPUs.
Then there are the companies that decided building a better GPU wasn’t the answer.
Cerebras questioned the need for thousands of interconnected chips, opting instead for a wafer-scale processor that keeps far more of the workload on a single piece of silicon. Groq took the opposite approach, optimizing almost entirely for inference. SambaNova focused on enterprise AI, building systems where efficiently serving multiple models matters more than posting the fastest benchmark.
Adaptability beats raw speed
The first wave of generative AI rewarded whoever could buy the most compute. That made sense when most organizations were solving the same problem. Today, AI workloads are changing so quickly that infrastructure teams are starting to optimize for something different: adaptability.
Keller’s example illustrates the point. Tenstorrent’s BlackHole architecture uses standard Ethernet instead of proprietary interconnects, allowing its hardware to slot alongside existing GPU deployments rather than replacing them. Keller told EE Times that one customer used Tenstorrent’s Galaxy servers to increase token throughput on GPUs they already owned instead of rebuilding their infrastructure from scratch.
Whether Tenstorrent’s approach becomes the industry standard is almost beside the point.
The bigger idea is already spreading. Across the industry, companies are spending less time asking how to build the fastest AI hardware and more time asking how to build hardware that won’t have to be replaced every time AI takes another leap forward.
The question that matters now
No one knows what AI workloads will look like three or five years from now. That’s the problem.
Infrastructure refresh cycles are measured in years. AI models seem to reinvent themselves every few months. Building around today’s workloads is starting to look like a risky bet when tomorrow’s could demand something different.
Infrastructure refresh cycles are measured in years. AI models seem to reinvent themselves every few months.
Every company is responding in its own way. They have different strategies but are still asking the same question: How do you build infrastructure that outlasts the AI running on it?
That may prove to be a more important engineering challenge than building the next record-breaking accelerator. And it’s the challenge driving companies from Nvidia and AMD to Google, Amazon, Broadcom and Tenstorrent.
Amazon EKS runs hundreds of thousands of Kubernetes clusters across more than thirty AWS regions. Operating at that scale has taught us something that has shaped how we build the service and that we think is useful to anyone running Kubernetes at scale: most availability problems do not stem from a component failing. They come from a component reacting to a problem in a way that makes it worse. A cache that goes stale and serves wrong answers. A health check that restarts the very process keeping a cluster alive.
What separates a resilient control plane from a fragile one is not the number of faults. It is whether a fault stays a fault or becomes an outage. This post is the story of how we keep the EKS-managed Kubernetes control plane on the right side of that line at ever-growing scale: the foundational changes we made and why, and what operating at fleet scale taught us about building systems that tolerate faults rather than spreading them. These are the reasons our most demanding customers confidently run their mission-critical workloads on EKS.
How AI and analytics workloads reshaped what “scale” means
Kubernetes was built for a particular rhythm of work. Pods came and went at predictable rates, and controllers had seconds or minutes to reconcile. The system’s design reflected that pace: strong data consistency, ordered watches, and consensus-replicated storage that puts correctness first. It worked beautifully for what it was designed to do, and it still does.
“What separates a resilient control plane from a fragile one is not the number of faults. It is whether a fault stays a fault or becomes an outage.”
But the workloads evolved faster than anyone anticipated. Foundation model training runs scale-up training jobs on thousands of GPU nodes in minutes. Real-time inference services scale from a warm baseline to thousands of replicas, then drop back within the hour. Apache Spark analytics pipelines burst from zero to tens of thousands of executor pods, chew through a dataset, and vanish.
Emerging agentic AI workloads add yet another dimension: autonomous agents that spin up, fan out, execute tasks, and tear down in seconds or less. These workloads share a trait that distinguishes them from traditional microservices: they generate enormous volumes of state transitions within compressed time windows and are deeply intolerant of delays. This velocity of state change pushed us to reinvent some of the mechanics to support a scale that was previously impossible, and to contribute what we could upstream.
How EKS reimagined Kubernetes storage foundation
Every Kubernetes cluster depends on etcd as its source of truth. Every application, every service endpoint, every scheduling decision is stored there. If etcd loses data, the cluster forgets everything it knows. Protecting that state is the most important job for a managed Kubernetes service.
Operating etcd for one cluster is well understood. Operating it for a fleet of millions is a different problem entirely. Hardware fails, networks blip, and disks degrade, so something has to handle those events without a human in the loop. And the operations etcd needs most, like replacing a failed member or recovering after a zonal event, are exactly the ones where a person acting under pressure can make a mistake that causes permanent data loss.
From the beginning, we built an operator agent that runs alongside every etcd instance and automates its entire lifecycle. The agent has two jobs. First, backup and recovery: it takes point-in-time snapshots and stores them durably outside the cluster. If too many instances are lost at once and the survivors cannot form a majority, the agent automatically detects the condition and rebuilds from the latest snapshot. Second, membership management: when an instance fails, the agent removes the terminated member and adds its replacement in an order that protects quorum and prevents split-brain.
A recent, more fundamental change was replacing etcd’s consensus mechanism, Raft, with a purpose-built journal that provides durable, ordered storage independently of etcd. In traditional etcd, a majority of members must agree on every write before it is committed. If two of three are unhealthy, the cluster becomes unavailable.
By offloading durability to the journal, etcd peers no longer negotiate quorum among themselves. Writes commit as soon as the journal acknowledges persistence, and that entire class of etcd quorum-loss failures disappeared. Since the journal handles persistence, etcd no longer needs to fsync writes to local disk, so its data store has moved to an in-memory filesystem. What was a disk-bound system became a compute-bound one, and storage latency was removed entirely from the critical path. For a deeper look at this architecture, read “Under the hood: Amazon EKS ultra scale clusters.”
“What was a disk-bound system became a compute-bound one, and storage latency was removed entirely from the critical path.”
For ultra-scale clusters, we went further and partitioned etcd into resource-specific shards. Each partition operates independently with its own storage budget and throughput capacity. The primary value is failure isolation. In a monolithic deployment, if the events keyspace exceeds its quota because a misbehaving controller creates objects faster than garbage collection can remove them, it blocks writes to everything, including node leases.
Suddenly, healthy nodes appear unhealthy because their lease renewals are being rejected. With partitioned etcd, the events partition hits its quota, but the leases partition continues operating normally. Nodes remain healthy. The scheduler keeps running.
What replacing etcd’s consensus mechanism unlocked
Removing the quorum requirement allowed us to make a change we had wanted for a long time: running etcd on the same host as the API server. In the traditional layout, every read and write crosses the network between separate machines. Each trip is fast on its own, but at thousands per second, the travel time adds up. With collocation, the API server talks to its local etcd over a loopback interface, and pod scheduling and controller reconciliation get measurably faster. For workloads where job controller queue depth is the binding constraint, shaving milliseconds off each API call means many more jobs are processed per second before the queue starts growing.
This is where the operator agent paid off. When etcd runs on the same host, an etcd member comes and goes whenever a control-plane host is replaced, which happens routinely. That only works if membership management is completely safe and automatic, which is exactly what the agent was already doing. We did not have to build a colocation from scratch; we built it on top of infrastructure that had been managing etcd membership safely since day one.
Collocation also taught us a lesson worth passing on: the convenient path needs a failover in case it breaks. The local etcd is the fast path, but if it becomes impaired, the API server fails over to another etcd member that is actively serving other API servers from the same journal. When you optimize for the common case, design just as deliberately for the moment that optimization is not available.
Fixing bottlenecks across the stack
At extreme scale, you have to address bottlenecks across the entire Kubernetes stack, and most of them are not bugs in the traditional sense. They are design choices that were correct at the scale Kubernetes originally targeted and break down only when the numbers get large. Rather than working around them internally, we fix them upstream so the entire community benefits.
One example involved the watch cache, the in-memory layer that distributes state changes from etcd to every controller watching for updates. When a controller starts, it requests a full snapshot of the current state via a mechanism called WatchList, and the existing implementation holds a shared read lock for the duration of the response build.
At hundreds of thousands of objects, that work runs long enough to starve the writer that needs exclusive access, so the cache’s resource version cannot advance. Consistent reads see a stale cache and fail over to etcd, while the response building churns through hundreds of thousands of allocations under the lock. We identified this as a limitation in the watch-cache’s locking model and are working with the community to refactor the underlying data structures and interfaces to eliminate the contention.
The same shape appears elsewhere. In the Horizontal Pod Autoscaler, a single mutex protecting the scaling state becomes a serialization point at high HPA counts, where workers spend nearly all their time blocked rather than doing useful work. A redesigned data store (PR #139142) restores parallelism and raises reconciliation throughput by orders of magnitude. In the scheduler, we identified a bottleneck (issue #138426): every scheduling cycle rebuilds a set of in-use persistent volumes by scanning every node in the cluster, even for pods that do not use storage at all. The fix computes that information lazily, and only for pods that actually need it, restoring throughput at scale.
Each of these started from a real production workload hitting a cliff, and we are working on the fixes upstream so the improvements reach every Kubernetes user.
From engineering to guarantees: EKS Provisioned Control Plane
The engineering described above made the EKS control plane more resilient and performant. But customers had a different problem: they could observe that the control plane kept up today, but they could not reserve its capacity the way they reserve compute or GPU capacity.
A team planning a thousand-node training run could secure the instances weeks in advance, yet had no equivalent mechanism for the orchestration layer that would coordinate them. EKS Provisioned Control Plane fills that gap. It exposes the control plane’s performance as dimensions you size explicitly, backed by the same kind of commitment you expect from the rest of your infrastructure.
You choose a scaling tier that maps to concrete, measurable capabilities: API request concurrency, pod scheduling rate, and cluster database size. The tiers range from XL through 8XL. At the top end, 8XL on Kubernetes 1.34 provides 16,000 concurrent API request seats, 400 pods-per-second scheduling rate, and 16 GB of cluster database storage, all backed by a 99.99% availability SLA measured in one-minute intervals.
Tiers are not static. You step up before a GPU training run or a large sales event, step back down during quiet periods, or grow permanently as your platform matures. Configuration happens through the console, CLI, eksctl, CloudFormation, or Terraform on any cluster, without recreation or downtime.
For AI workloads, orchestration capacity is planned alongside GPU capacity, available when the compute comes online. For analytics platforms submitting hundreds of jobs per minute, the control plane is ready for the burst before it arrives. And for organizations that need environmental consistency across staging, production, and disaster recovery, the same tier guarantees consistent performance characteristics everywhere.
Taking the same foundation to the edge
Some workloads cannot move to the cloud, whether due to data sovereignty requirements, latency constraints, or unreliable connectivity to the Region. Running Kubernetes in these disconnected environments introduces unique challenges: etcd must remain durable on hardware with only a few machines, the cluster must self-heal without reaching the cloud, and observability must survive network partitions that last days.
With the updated architecture for EKS local clusters on instance store Outposts, we brought edge clusters onto the same management plane and software stack as EKS clusters in the cloud.
The control plane lives in an EKS-managed account on the Outpost rather than in the customer’s account, so customers never manage control plane instances, etcd backups, or logging agents themselves, and they cannot accidentally break the thing keeping their cluster alive. The same machine images, container images, and operator agent run in both places, with edge-specific behaviors selected by configuration.
Because it is the same stack, new Kubernetes and EKS platform versions arrive in lockstep with their cloud release, and features like EKS add-ons, Pod Identity, and access entries work the same way they do in a Region.
The hardest part was keeping etcd healthy on hardware with only a few machines that may be cut off from the cloud for days at a time. We solved it by extending the same agent. It keeps a spare copy of the data continuously up to date and promotes it the instant a machine fails, so the cluster heals itself with no human involvement and no connection to the cloud.
Observability survives the disconnect, too: the metrics agent continues collecting and writing to local disk, shedding the least critical data first when space runs short, so the signals that matter most are the last to go. When the link returns, the buffered data is flushed back with its original timestamps.
All of this only works because the system was designed from the start to operate without anyone logged in. That same design is what makes it possible to deploy changes safely across the entire fleet.
Operating safely at fleet scale
Every one of these changes was deployed to a running fleet of hundreds of thousands of clusters. The journal migration and collocation required transitioning each cluster individually. Every migration follows a strict sequence: validate pre-conditions, create a point-in-time snapshot, perform the switchover, validate post-conditions. If any step fails, the system rolls back automatically.
Rollouts proceed cell by cell, zone by zone, region by region, with automated monitoring comparing latency, error rates, and throughput between updated and non-updated clusters. Any statistically significant deviation triggers an automatic halt.
What made all of this possible is that EKS is built to operate without human intervention at the individual cluster level. Through Zero Operator Access, the architecture prevents AWS personnel from having technical pathways to access customer content in the managed control plane. A system designed to work without human access must be observable, recoverable, and automatable from the start, and that same discipline is what enables operating at extreme scale.
Three operational lessons shaped how we approach this work.
The first is that a healthy leader is not the same as a working one. The control plane’s controllers run in an active-passive configuration, and early on, we treated an unhealthy standby as if cluster operations had halted. They had not; what matters is whether a leader exists. But the harder lesson: a leader can quietly stop making progress while still renewing its lease and passing every health check. The signal that caught this was watching the controller’s work queue depth. If the queue fills while the leader looks healthy, the system is falling behind in ways no liveness probe will catch.
“A leader can quietly stop making progress while still renewing its lease and passing every health check. The signal that caught this was watching the controller’s work queue depth.”
The second is that maintenance ordering matters as much as the maintenance itself. etcd defragmentation is blocking, and the pause grows with database size. When it hit the leader, every write stalled. We taught the agent to move leadership to a healthy node before defragmenting, so the disruptive work always lands on a follower while writes keep flowing.
The third is that liveness is not readiness. A process can be alive but not ready while it warms caches, and routing based solely on liveness sends requests to an instance that cannot handle them. Equally, readiness flapping during graceful draining should never trigger a restart. We keep the two signals strictly separate: one decides recovery; the other decides routing.
None of this work is visible from the outside, and that is the point. The largest clusters taught us lessons that made every cluster faster. The riskiest migrations produced safety machinery that protects every upgrade. The upstream fixes we contributed for workloads at the edge of what Kubernetes can handle flow back to every user of the project.
“None of this work is visible from the outside, and that is the point.”
When you deploy on EKS and your pods come up in seconds, even during a burst, even when something behind the scenes goes wrong, that speed is not accidental. It is the accumulated result of years of operating at scales where small problems can become big ones fast, and engineering the system to contain them before they do.
Building with AI has, until recently, meant calling up someone else’s model. But wiring an app to an API from OpenAI, Anthropic, or Google is a poor fit for government and operators of critical infrastructure. When data legally or operationally cannot leave a secured network, a hosted endpoint in someone else’s cloud is a non-starter, regardless of whether the workload is intelligence analysis, grid operations, or patient records.
“The most revealing aspect here is that Palantir didn’t ship a model, but the apparatus for deploying and owning one.”
That’s the lens through which to read Palantir’s announcement on Monday. The company introduced an “intelligent engine,” built on Nvidia AI and Nemotron open models, for running, customizing, and continuously improving AI inside air-gapped and other sovereign environments while keeping data and model weights in the customer’s hands. The most revealing aspect here is that Palantir didn’t ship a model, but the apparatus for deploying and owning one.
From calling AI to operating it
Most organizations will run a hybrid of hosted and self-hosted models for years, but a second pattern is hardening alongside this workflow. Rather than treating the model as an external service, the application communicates with an internal AI platform that routes requests to one or more models running on the organization’s GPUs. Data never leaves the perimeter, which hands security and compliance teams direct control over governance, auditing, and retention.
So we’re seeing the question shift from which model we should call to which models we should own and operate ourselves. And, just as importantly, what does it cost us to run them?
What’s actually in the box
This is where the announcement gets more concrete than the press release lets on, and where it’s worth knowing what Nemotron is.
Nemotron is Nvidia’s family of open-weight models, released in 2026 in three sizes — Nano (~31.6B parameters), Super (120B), and Ultra (550B). All three use a hybrid Mamba-Transformer mixture-of-experts design that activates only about a tenth of their parameters per token, so they run far cheaper than their headline sizes suggest, with context windows up to a million tokens.
Nvidia publishes the weights, training data, and recipes under a permissive license, and the models are deployable via open runtimes such as vLLM, SGLang, and llama.cpp, Ollama — or as Nvidia NIM microservices, the containerized, TensorRT-LLM-optimized path that ships inside the Nvidia AI Enterprise suite. Palantir, notably, was already on Nvidia’s published list of early Nemotron adopters, so this is less a cold start than a formalization.
An important note for builders weighing the move: Nemotron doesn’t top the raw-capability leaderboards; open families like DeepSeek, Qwen, and Kimi K2 generally score higher on absolute benchmarks. Nemotron’s pitch is efficiency on Nvidia silicon and genuine openness, which is exactly the axis that matters when the deployment target is your own hardware behind an air gap rather than a hosted endpoint.
Palantir’s contribution is the layer that turns “download the weights” into “run this in a classified environment and keep improving it.” The company describes three engineering surfaces: deployment (getting base and customized models into air-gapped and classified networks), context (prompts, workflow structure, and model behavior in production), and model (changing the weights themselves on proprietary data and mission outcomes). Underneath sits Palantir’s existing stack — AIP, Ontology, Foundry, and Apollo — handling data authorization, enforced isolation, and auditability.
The “self-improving” claim is a telemetry loop: The engine captures usage and trace data, then uses it to post-train and align the model toward the tasks where it adds value. This is the part that demands the most operational discipline, because a feedback loop with no evaluation harness is a recipe to drift.
The catch: Owning the stack means owning the stack
The sovereignty pitch is genuinely attractive, and it has a price tag the announcement doesn’t dwell on. Owning your models means owning everything around them. That’s GPU capital expenditure and the power and cooling to match; an inference stack you keep patched and performant; a model lifecycle — fine-tuning, evaluation, rollback — that you staff and run yourself; and a security burden that doesn’t go away just because the box is disconnected.
“Owning your models means owning everything around them.”
Ultra-class models, in particular, are not modest: running a 550B model, even at ~10% activation, requires multi-GPU server nodes, not a spare rack.
For an agency that legally cannot use a hosted API, that cost is simply the cost of doing the work, and the calculus is easy. For a commercial enterprise that could use a hosted endpoint, the math is a real trade — control and data residency on one side, capex and operational headcount on the other. The right answer is workload-specific, and anyone selling it as obvious is selling something.
Read the announcement for what it is
It’s also worth being clear about what this news is not. It’s a packaging-and-positioning announcement between two companies with aligned incentives: NVIDIA sells more GPUs and more AI Enterprise licenses, Palantir sells more platform, and “sovereign AI” is the banner both have been marching under all year. There’s no named agency customer, no contract value, and no benchmarks. Palantir CEO Alex Karp says many U.S. clients already use these models and frames the appeal as avoiding proprietary insight being baked into the weights of closed models; NVIDIA’s Jensen Huang calls open source foundational to national security. Both quotes are doing strategic work. The substance here is the integration and the go-to-market, not a technical breakthrough.
Why it still matters
Strip away the government framing, and the same pattern is spreading through finance, healthcare, manufacturing — anywhere data residency and compliance are design constraints rather than afterthoughts. The reasons it’s newly practical are concrete: open models good enough for production, MoE architectures that cut the compute bill, and an inference tooling layer (NIM, TensorRT-LLM, vLLM) mature enough that air-gapped serving is an engineering project rather than a research one.
“The model becomes a component. The engine becomes the product.”
The takeaway here for developers is that as open models converge on “good enough,” differentiation shifts to everything around them — deployment, routing, governance, evaluation, security, and lifecycle management. The model becomes a component. The engine becomes the product.
Most organizations will keep calling hosted APIs for plenty of work. But for a growing set of them, the interesting question is how much of the stack they’re prepared to own — and whether they’ve counted the cost of owning it.
Workday, the payroll and HR data platform, has been pursuing AI and agents for a while, but while other businesses may allow a little room for error, getting a payroll run in Workday 99% right is not exactly good enough.
Gabe Monroy, Workday’s chief technology officer, tells The New Stack that enterprise AI has to clear this bar before anyone will let it near their HR and finance data.
“There aren’t many systems that are more critical — or less forgiving — than ones that are dealing with people and money,” he says. There’s no tolerance for “well, it works most of the time,” Monroy says in an interview.
At its DevCon developer conference in early June, Workday laid out its plans to clear this correctness bar. The company introduced Agent-Ready Tools, a set of connectors that let agents act across the platform over the Model Context Protocol (MCP), a Developer Agent that lets people build apps and agents on Workday in plain language, and Agent Passport, which tests and verifies agents before they go into production and keeps monitoring them after, with Cisco as the first attestation partner.
Guardrails belong in the inference engine
Monroy spent most of his career in infrastructure and the developer space: at Deis, Microsoft, DigitalOcean, and, most recently, Google. At Google, he focused on building infrastructure for large AI labs to run inference at scale. For someone so focused on infrastructure, coming to Workday might seem a bit like an odd move, but Monroy argues that, at this point, LLM safety is — or at least should be — part of the core infrastructure for enterprises.
“The stakes are higher … in the world of people and money”
“The stakes are higher in the world of Workday and in the world of people and money, and that’s something that I was really excited about tackling at Workday specifically — and I do look at it as a core infrastructure,” he says. “A lot of what I’ve been doing in my recent past has been building infrastructure for large AI labs to do inferencing at scale, and what you pick up pretty quickly is that inferencing is probabilistic.
“[Inferencing] involves prefill and decode, and a whole bunch of really technical machinery in place to stream tokens out to end users, but what is nowhere in that stack today is the concept of native LLM-level enforced guardrails — guardrails that are part of the core inference.”
In his view, making it safe for enterprises to operate inference at scale has to be done at the inference engine layer. Agent gateways and similar add-ons that wrap the model from the outside sit at the wrong layer, he argues.
Workday CTO Gabe Monroy
For a system of record like Workday, the system must strictly enforce guardrails on who the user is, what their budget authority is, and where they sit in the org chart. Those constraints, Monroy says, can be wired deep into the inference rather than checked after the fact. “These are things that we have the opportunity to wire deeply into the inferences at a very, very low level, in a way that produces much safer outcomes,” he says.
Workday’s recent acquisition of Pipedream partially plays into this. With Pipedream, an agent can reach out to third-party systems outside Workday to, for example, pull a policy document from Google Drive, and the platform can then verify that this specific agent has all necessary access rights to do so.
“If you’re trying to run an agent interaction that is integrating with people and money, that orchestration loop should probably happen closer to Workday, ideally on the Workday,” he says. “I do think there is something to be said for inference engines having proximity to the underlying system, because when it comes to a low-level inference engine and runtime perspective, there are some differentiated things you can do at the inference engine level that are only possible due to proximity to.”
“I do think there is something to be said for inference engines having proximity to the underlying system…”
He likened it to car repair. If somebody hands you a toolbox, you may or may not be able to fix your car’s problem. “Maybe you’ll get the outcome, but I tell you what, if you really want your car repair done right, bring it to our shop. We got the hydraulic lift, we got the trained engineers who know how to do this stuff, and we got the tools — but our tools are the best in the industry.”
For the part of an agent’s orchestration loop that has to run next to HR and finance data, “you really should be running that in our shop.”
For more general-purpose workflows, though, he believes there is room for general-purpose platforms. Not everything needs to run on Workday, after all, and Workday does offer MCP servers that third-party tools can tap into to access its data and tools as well.
To some degree, every system-of-record vendor is making some version of the proximity-and-context case right now. Owning the context for AI agents to work with is a major moat for these companies, after all, even as APIs and MCP servers allow agents to pull in data across vendors.
Meanwhile, there is also a group of other companies positioning themselves as neutral parties benefiting from the fact that these other vendors are now opening their platforms to third-party agents. It seems unlikely that, in the long run, enterprises will want to manage multiple agent platforms, but at this point, it is still far from clear in which direction this pendulum will swing.
Happy to lose the tooling fight
If proximity is where Workday thinks it can win, there is one area where the company doesn’t want to compete: developer tools.
“We’re not going to differentiate Workday on building better tools for developers,” Monroy says. “We’re going to differentiate on safety, on trust, on the inference engine.” It’s telling that the demos at DevCon ran on Claude Code, Cursor, OpenAI’s Codex, and Google’s Antigravity rather than on anything Workday-branded. “On the tooling front, I’m happy to have Claude Code and Codex and others,” he says.
It’s no secret that developer loyalty to AI tools doesn’t really exist at this point. Developers definitely don’t want a one-off tool from a single vendor, and Monroy notes that he isn’t going to ask them to adopt one. Instead, Workday wants to own skills, which he calls “the underlying lingua franca that we’re using to converge across the system.”
Workday is happy to be the system of record for its vertical and to let others fight over the tools.
Okta has made its AI agent governance platform generally available for FedRAMP- and HIPAA-regulated environments, becoming what it claims is the first independent identity platform to extend AI agent lifecycle management inside the compliance boundaries federal agencies and healthcare organizations already trust.
The product, Okta for AI Agents – Core, elevates AI agents to first-class identities managed alongside human and machine workforces. This is a shift from the practice of treating agents as static service accounts or hardcoded API keys. The launch comes as federal agencies face mounting pressure from the recent executive order on AI innovation and security, which directs agencies to deploy AI agents and mandates that they secure them.
“The message to agencies is clear: Adopt AI aggressively, but secure it as you go,” writes Amy Johanek, Okta’s VP of Federal, in a blog post. “That puts identity at the center of the mission.”
“The fastest-growing class of NHI yet, and the hardest to see.”
Johanek also writes that AI agents are “the fastest-growing class of NHI [non-human identity] yet, and the hardest to see.” Anyone can spin one up, agents can spawn additional agents, and each connects across apps, APIs, SaaS tools, MCP servers, and data systems with little visibility, she says.
For organizations under mandates to harden systems and defend against AI-enabled criminal access, an unmanaged agent is not just an operational gap; it is more like an unguarded door, the company says.
“An unmanaged agent is not just an operational gap; it is more like an unguarded door.”
Johanek laid out four specific risks facing agencies running ungoverned agents: compliance violations when agents touch data outside authorized boundaries; compounding breach risk, where a single compromised credential doesn’t grant access to one system but to everything an agent can reach before a human can intervene; failed audits when agents run as orphaned accounts with no owner or evidence trail; and stalled AI adoption when delay becomes the only compliant option.
Moreover, the platform is organized around three governance questions: Where agents operate, what resources they can access, and what actions they’re authorized to take. Agents are registered in Okta’s Universal Directory inside an organization’s regulated cell, each assigned a unique identity and a named human owner, Johanek says. Every agent becomes a known, owned, first-class identity inside the environment, whether it came from a third-party platform or the organization’s own developers.
The platform replaces static credentials with scoped, short-lived tokens enforced at runtime. Least privilege is applied across authorization servers, third-party applications, and MCP servers. The governance layer mirrors existing federal workforce identity controls: access certifications, entitlement reviews, time-bound permissions, and a full audit logging stream that can be streamed to SIEM platforms for U.S. Government Accountability Office reporting requirements, Johanek says.
The offering also provides a kill switch
The offering also provides a kill switch. When an agent deviates from its intended mission or unexpectedly accesses sensitive data, security teams have a real-time mechanism to contain the risk before it escalates into a larger incident.
Johanek says she sees the offering as continuity rather than new infrastructure. Agencies already trust Okta to manage human identities. Okta Identity Governance achieved FedRAMP High authorization earlier this year; bringing agents into that same identity fabric, she writes, is the natural next step, not a parallel system to build and defend.
However, there is one caveat: Okta for AI Agents – Core is not authorized in Okta for US Military cells.
Over the past few months, four AI giants quietly rebuilt the same thing at once: AWS, Microsoft, Google, and Anthropic each shipped agent runtime updates that point to the same architectural shift.
Microsoft rebuilt its Foundry-hosted agents in April around per-session isolation. Anthropic’s Managed Agents virtualize the agent into a session, a harness, and a sandbox. AWS routes each session to its own microVM, and Google isolates agent code execution in a dedicated sandbox. The common pattern is not another model feature or developer tool. It is a move from request-level load balancing to session-aware execution.
This matters because enterprise agents are not ordinary API calls. They are long-running, stateful, tool-using processes that often run code influenced by user input. At its core, the agent runtime is becoming a control plane for state, identity, isolation, and lifecycle, and the unit it schedules has moved from the individual request to the agent session. The convergence across four platforms suggests session-aware execution is becoming a baseline requirement for production-grade agents.
The traditional cloud scaling model
To appreciate why this matters, it helps to start with what the traditional model got right. NGINX and HAProxy tiers typically sit in front of a pool of workers and route each incoming request to the next available backend. The state is deliberately kept elsewhere, in Redis or a database, so any worker can serve any request. That externalization of state is what enables elastic scaling, fault tolerance, and infrastructure replacement without disrupting the application. Sticky sessions were available, but enterprises treated them as an exception for stateful applications rather than the default.
The model rests on two assumptions: that requests do not depend on one another and that any backend can serve any request. For web APIs and microservices, those assumptions held for nearly two decades and enabled the scaling of the largest systems ever built. Agents challenge both at once, which is why conventional load balancing alone is no longer sufficient.
Why agents break the model
Agents pose two distinct challenges: preserving conversational state across turns and enforcing a security boundary strong enough to withstand untrusted code. Only the first can be addressed by routing alone.
Why can the agent state not be pooled?
Imagine an enterprise support agent processing a refund. It reads the order, calls a tool, and then waits for the model. The next turn asks a clarifying question. If that turn is routed to a different replica without access to the prior context, the agent loses the state it needs to finish the workflow.
The load balancer starts to behave less like a traffic distributor and more like a session router.
The industry already has a name for the workaround, session affinity, and it already concedes the cost. Load-balancing approaches increasingly rely on consistent hashing, using a conversation identifier to keep related steps in a warm state. In plain English, the load balancer starts to behave less like a traffic distributor and more like a session router.
Why the trust boundary forces isolation
The second challenge is harder because it turns on isolation rather than on the state alone. An agent can execute model-generated code shaped by user input, so the backend must be treated as a security boundary rather than a generic compute target. A shared kernel does not give untrusted, agent-generated code the tenant isolation that enterprise security teams require. Session affinity can preserve routing continuity, but it does not by itself provide per-tenant isolation for untrusted execution. For platform and security teams, that is the line where a tuned load balancer is no longer enough.
This is not a theoretical concern. When Asana disclosed a flaw in its MCP server in June 2025, the server had been live since May 1, a window of about five weeks. A tenant-isolation check failed, and around 1,000 organizations could see project data belonging to other customers. Based on the reported details, security researchers traced the issue to a server that validated the user but did not consistently enforce the agent and tenant context behind cached responses. No external attacker was involved, and data still crossed organizational boundaries. The incident shows why user identity, agent context, and session state must be consistently bound together.
The session as a unit of compute
Once that binding becomes a correctness and security requirement, the session becomes the unit that the platform schedules. The clearest evidence is in the lifecycle. A session now has a state it never had before. AWS documents that an AgentCore session remains Active while it processes work and sits Idle when it is provisioned but waiting. It reaches Terminated after a 15-minute idle timeout or an 8-hour maximum lifetime. That is the lifecycle of a long-running execution environment, not of a short-lived HTTP request.
The economics follow the same logic. When billing is tied to active sessions, concurrency, idle time, and agent sizing become the cost drivers rather than request volume. That puts a new line item in front of platform teams and finance. The better mental model is no longer a traditional load balancer. It is closer to a virtual actor runtime, where an addressable identity is instantiated on demand, kept active while it is needed, and deactivated when idle, with one live instance per key.
Four approaches to the same shift
All four platforms are moving away from treating stateful, untrusted agent work as ordinary load-balanced traffic. The key difference is the compute primitive each provider chooses for the isolated execution environment.
AWS AgentCore is the most opinionated of the four. Each session gets a dedicated Firecracker microVM with isolated compute, memory, and filesystem, and a request carrying the same runtimeSessionId is routed back to that microVM through a session header. When the session ends, the microVM is terminated and its memory sanitized.
The agents hosted on Microsoft Foundry implement a similar model through a different set of primitives. The platform creates a per-session VM-isolated sandbox on demand, runs it, and tears it down at session end, with no replica count and no warm pool to size. Each agent gets a dedicated Microsoft Entra identity, and sessions run a 15-minute idle timeout against a 30-day maximum lifetime.
Google Agent Engine is the most instructive hybrid. Its runtime keeps request scaling within the reasoning loop, a managed environment with configurable minimum and maximum instance counts, and a container_concurrency that defaults to 9. Even where Google retains request scaling for the loop, it separates untrusted code execution into an isolated Code Execution sandbox and externalizes conversation state to Sessions and Memory Bank. The provider that kept the load balancer in the loop still would not point it at stateful untrusted work.
Anthropic provides the clearest decomposition of the architecture. Managed Agents virtualize the agent into three components: a session that logs everything that happens, a harness that runs the loop and routes tool calls, and a sandbox where code runs. The harness becomes a near-stateless control plane, while the sandbox becomes a callable, rebuildable resource. The Cloudflare integration shows that the substrate can be decoupled. The agent loop runs on Anthropic while each tool call runs in a Cloudflare sandbox, which can be a full microVM or a lighter V8 isolate.
The four platforms converge on the routing and lifecycle model and diverge on the execution substrate, as the table shows.
Platform
Compute primitive per session
What it means for an enterprise buyer
AWS AgentCore
Dedicated Firecracker microVM, routed by session ID, 8-hour ceiling
The most opinionated answer, with isolation and routing fused at the microVM
Azure Foundry hosted agents
Per-session VM-isolated sandbox, dedicated Entra identity, 30-day max
Identity-led isolation suited to longer multi-day workloads
Google Agent Engine
Request-scaled loop plus a separate isolated code-execution sandbox
A hybrid that load-balances the reasoning and isolates the execution
Anthropic Managed Agents
Decoupled harness and sandbox, microVM or V8 isolate behind the loop
The most portable, with the substrate treated as a swappable layer
No single architecture is the right answer for every enterprise workload. A multi-day research agent fits Azure’s longer-lived session, a code-heavy agent needs microVM-grade isolation, and high-volume automation favors a lighter isolate. Many enterprise platforms will end up combining more than one of these patterns.
The binding the application still owns
The platform can isolate and route a session, but it does not know which human owns it. AgentCore explicitly states that it does not enforce session-to-user mappings, so the application backend must maintain the relationship between users and their session IDs and set per-user session limits. This is why the Asana incident is relevant to enterprise architecture.
Isolation was maintained while the user-to-session binding broke.
In that case, isolation was maintained while the user-to-session binding broke. The platform solves isolation and lifecycle, and it hands back identity mapping, authorization, and tenant context to the application. For an enterprise buyer, the key question is who owns that binding and how it is tested under concurrent, multi-tenant load.
How this differs from sticky sessions
A reasonable objection is that this appears to be sticky sessions combined with autoscaling on a microVM substrate. The distinction is worth drawing out. Traditional sticky sessions are usually performance optimizations, whereas in an agent runtime, session binding becomes a correctness and security requirement. Traditional load balancers route traffic to existing backends and do not own the lifecycle of the execution environment.
The new control plane provisions an environment when it first sees a session key, routes work to it, and tears it down after idle or lifetime limits. The primitive underneath is genuinely old, and its age is what makes the move legible. AWS open-sourced Firecracker at re:Invent in 2018 to pair VM-grade isolation with container speed. I covered it for The New Stack then, and ran more than a hundred microVMs on a single laptop to see that trade firsthand. The same microVM now powers Lambda and Fargate across trillions of executions a month. This is how cloud architecture tends to evolve, with existing primitives recombined around a new unit of abstraction.
The agent session is becoming the operational unit of enterprise AI infrastructure.
In summary, the important shift is not that cloud providers have found a better load balancer for agents. It is that the agent session is becoming the operational unit of enterprise AI infrastructure. Traditional load balancing routed independent requests to interchangeable workers. A session-aware runtime routes work to an isolated environment, preserves state across turns, and owns the lifecycle of the execution context. The dispatch layer did not disappear when agents arrived. It became session-aware, stateful, and lifecycle-aware, making it a control plane rather than a conventional load balancer.
Three practical questions will shape what comes next, and enterprises will need to evaluate each. The economics of per-session billing will decide how costs scale with concurrency. The open-source effort to rebuild this routing model on Kubernetes will determine its portability. The argument between microVMs and lighter isolates will decide the trade-off between isolation and density. For enterprises building agents, a runtime that owns isolation, routing, and lifecycle turns background plumbing into a deliberate architectural choice. It gives platform, security, and operations teams a foundation they can standardize on rather than rebuild for every workload.
Many agentic projects can sail through development just fine. Then they hit security review — and that’s where things can grind to a halt. Unclear identity models and overly broad permissions quickly become blockers.
You’ve probably seen this play out: A customer support agent is working well; it triages tickets and processes refunds, handles the whole workflow without a hitch. Then security asks a simple question: Under whose identity is this running? The answer stops the process cold: It’s a shared account with broad permission, no clear ownership, no audit trail, and no least-privilege controls in sight.
The root issue isn’t complicated. It’s undefined identity and poorly scoped permissions. And that challenge is accelerating fast. Research from the 2026 Tech Leader Study, conducted with Oxford Economics and IBM, shows surveyed enterprises expect to deploy an average of 1,661 AI agents, a 38% increase from today. Each new agent introduces another identity to secure, and without clear boundaries, the problem compounds quickly.
As a result, many agentic systems focus on what agents can do without defining what they should do, or under whose authority. Agents also don’t hold a fixed set of permissions. They request access, call new tools, and assume roles as they work, so access paths compound in ways no one explicitly granted or reviewed. Without a verifiable identity, there’s no accountability, making least-privilege enforcement, traceability and incident response difficult.
“Each new agent introduces another identity to secure, and without clear boundaries, the problem compounds quickly.”
To address these gaps, this guide is written for developers, architects and DevOps engineers building agentic systems — and for the IT leaders responsible for approving them.
The four identity decisions every agentic system must make
Identity decisions can’t be treated as an afterthought. Identity shapes how agents authenticate, what they can access, and how their actions are controlled and audited over time. Get it wrong early, and you’re building on a shaky foundation.
Here are the four decisions that matter most:
Workload identity vs. shared service accounts
Shared service accounts are easy, and that’s exactly what makes them dangerous. When multiple agents act under a single identity, it becomes hard to tell what happened or what went wrong after the fact. If an account is leaked or misused, everything it touched is exposed.
“Shared service accounts are easy, and that’s exactly what makes them dangerous.”
Workload identity assigns each agent its own identity. Permissions stay scoped, and actions are attributable. It requires more setup but creates isolation and auditability.
Static API keys vs. short-lived credentials
Static API keys tend to stick around forever. They get hardcoded into apps, passed around between systems, and rarely rotated — which makes them a persistent vulnerability waiting to be exploited.
Short-lived credentials work differently. They’re issued on demand, scoped to a specific task, and expire automatically. In practice, this often relies on identity federation (for example, using OIDC tokens) combined with systems that can issue dynamic credentials at runtime, rather than storing long-lived secrets in code or configuration
Direct credential handoff vs. brokered session access
Handing credentials directly to an agent is simple. It’s also opaque. You don’t have a natural point to evaluate policy or understand what’s happening in real time.
Brokered access introduces a control point into the flow. Requests go through a broker, policies are evaluated in real time, and temporary credentials are issued per session. It adds infrastructure, but restores visibility and policy enforcement.
Fragmented logging vs. full identity lineage
Most systems log what happened. Far fewer capture who initiated it or how an action propagated through a chain of agents and services.
Full identity lineage connects every step. You can trace an operation from triggers to outcomes, which can make debugging faster and enable more credible incident response. The catch is that this requires consistent identity propagation and structured logging from the beginning—it’s hard to retrofit.
When these tradeoffs become real risks
These aren’t abstract architectural preferences. They show up as concrete vulnerabilities.
Nightfall AI reports that organizations expose nearly 350 secrets per 100 employees each year, with 35% of exposed API keys still active. Combine that with persistent credentials and shared identities, and the potential blast radius grows fast.
The pattern is consistent: shared accounts and long-lived keys are faster to build but harder to secure. Workload identity and short-lived credentials require more upfront investment but can deliver more security over time.
Debugging breaches by feel
Think about what happens when an agent running on a shared account with a long-lived key suddenly spikes its data access. Was it a bug? A breach? Routine behavior? Hard to say. Revoking the key might stop the issue, but it could also break a half-dozen unrelated workflows in the process. You’re now debugging by feel.
Shortcuts reduce friction at the start and accumulate risk over time.
Standardize identity at the platform layer
The answer isn’t to rebuild authentication, authorization and auditing from scratch for every agent you ship. That’s not scalable.
Instead, standardize identity at the platform layer—centralized identity providers, policy engines and a credential broker to enforce secure defaults and make compliance straightforward rather than a constant negotiation.
“Shortcuts reduce friction at the start and accumulate risk over time.”
Agentic AI works in production when identity is designed up front and enforced at runtime, rather than assumed from a prior login. When projects are treated as an afterthought, they stall. When it’s built in deliberately, agents can operate with the control that production environments demand.
IBM supports this through an integrated identity-first approach that spans secrets management, secured access, and identity governance—helping organizations scale agentic systems securely without adding operational complexity.
We’re more than 20 years past the launch of AWS, the starter gun for the shift of compute and storage from on-prem racks to the cloud.
The rapid growth of AWS and competing services like Azure and Google Cloud underscores how many companies have made the jump from controlling their own infrastructure to renting capacity from hyperscale public clouds.
For the major providers, the public cloud has proved an incredible business. Amazon’s cloud service generated nearly 60% of its first-quarter operating profit, for example. For cloud customers, however, the tides may be turning.
Think back to the early days of the public cloud. Azure and AWS scrapped for market share, offering price cuts to entice workloads to their centralized silicon. The situation has evolved over the ensuing decades. Today, cloud costs are material and rising, prompting some companies to question whether being cloud-first is the best path forward.
Cloud bills are expanding due to increased usage of hyperscaler infrastructure, yes, but also because many customers today use the cloud for everything, rather than for what it is best suited for.
n the latest episode of The New Stack podcast, Byron Dill, Director of Solutions Engineering at Summit, tells us that shared compute and storage have their place in the modern IT mix, but that many companies would do well to segment their workloads and move some of that work back on-prem. (Think lower costs and simpler management of high-risk data.)
The argument echoes what we’ve seen recently in the AI realm. Many companies quickly adopted AI technology, only to be surprised later by the bills they incurred. The public cloud is a similar frog-boiler, albeit on a slightly longer timeframe.
In both cases — AI and the public cloud — companies have learned that a product once pitched as a way to reduce spend can evolve into the opposite without careful management. Summit, which offers managed private clouds to enterprise customers, thinks that some corporate workloads should be removed from the cloud and moved in-house.
What will that cost? How long does it take to move? And which industries are most primed to benefit from their own private cloud? We get into it all in this episode.
Modern businesses are in a constant, uphill battle against what to do with unstructured data: PDFs, contracts, scanned images, customer call recordings, meeting videos, and more. Traditional document automation workflows that rely heavily on template-based extraction or rigid rules used to make sense. But document formats have changed; they’re diverse and don’t fit standard formats, making costly, brittle, traditional systems a relic of the past.
“Modern businesses are in a constant, uphill battle against what to do with unstructured data.”
Enterprises demand faster, more accurate processing, which raises the question: How can we reliably turn messy, multimodal content into structured, actionable insights without a mountain of manual effort?
That’s where Amazon Bedrock Data Automation (BDA) comes in.
What is Amazon Bedrock Data Automation (BDA)?
Amazon Bedrock Data Automation (BDA) is a generative AI-powered, fully managed service on Amazon Web Services for end-to-end document and media automation. It enables users to automate the extraction, classification, and transformation of unstructured content across modalities such as documents, images, audio, and video.
“At its core are Foundation models which enable intelligent extraction and understanding of content.”
At its core are Foundation models (FMs) which enable intelligent extraction and understanding of content. It allows users to configure standard output for common use cases, or even define custom extraction logic using blueprints tailored to your business. BDA is designed for scalability, accuracy, and auditability, making it ideal for enterprise workflows.
Walk-through: creating a project, standard output & custom output using blueprints
1. Create a project via console
In the Amazon Bedrock Console, navigate to Data Automation → Create Project.
Enter the name of the project:
2. Standard output:
Standard output gives you the model’s default, unstructured response (text, image, audio, or video) directly from the Data Automation pipeline.
In standard output, each modality has its own options for what is needed as an output.
Document:
Image & Video:
Audio:
Now let’s test Document Modality for Standard Output:
First, click on “Test” in the upper right corner.
Next, select the document from the system, sample, or S3 and choose the modality from the dropdown menu.
Click on the “Generate results” button:
After processing, it will show the summary and content of the document:
Custom output (blueprints):
Custom output lets you define a structured, predictable format using blueprints, which ensures the output matches your exact schema, fields, and business rules.
Let’s test custom output using blueprints for the same document:
Navigate to “Custom output” and click on “Add Blueprint”:
From here, two options will appear. You can either use LLM power to generate the blueprint (where it inspects the document), or you can choose to enter field names, instructions, and other information manually.
Below is a blueprint generated by LLM which has pulled all possible fields and tables from the document:
It has extracted the information using the blueprint as demonstrated below, including the Field name, Instruction, and Results:
It also provides the Extraction type (which can be Explicit or Inferred), Confidence percentage, and other relevant information.
Additionally, it can extract information in the form of a table, such as an account summary or transaction information:
Code examples
Amazon Bedrock Data Automation (BDA) Utility Module
Description:
Helper functions to create BDA projects, blueprints, invoke jobs,
monitor job status, and fetch results.
import boto3
import time
import json
import botocore
Example: From a bank statement → Date, Description, Amount, Balance.
3. Transformation: Modify or restructure data.
Example: Convert Home Address into separate fields -> street, city, ZIP code, etc.
4. Normalization: Standardize data values.
Example: Convert multiple date formats (MM/DD/YYYY → YYYY-MM-DD).
5. Validation: Validate extracted fields against rules.
Example: Amount must be numeric; dates must match the format; balances must reconcile.
Use cases that illustrate business value
Real-world scenarios where BDA provides significant ROI include:
Financial Services: Automate processing of bank statements, invoices, and loan applications, reducing manual labor and speeding up reconciliation or underwriting.
Insurance: Ingest and extract data from claims forms, medical reports, and damaged-asset photos.
HR / Legal: Process resumes, contracts, and offer letters; extract structured data, including skills, clauses, salaries, and parties.
Customer Support: Transcribe and summarize calls, extract intent and sentiment, and feed those insights into CRM or case systems.
Security & Compliance: Analyze CCTV footage or meeting recordings to detect key actions, summarize context, and flag compliance issues.
BDA proves itself flexible and powerful, as it supports both standard outputs for basic workflows and fine-tuned custom schemas via blueprints. It is scalable and robust, with projects that enable batch processing and versions (development vs. live) for safe testing. It’s also audit-friendly, providing structured fields with types, normalization rules, and validation logic.
“Compared with rule-based systems, foundation models achieve better semantic extraction across the board.”
A true key benefit is that BDA is multimodal across formats. Users can use the BDA framework to process documents, images, audio, and video. And, best of all, it’s highly accurate. Compared with rule-based systems, foundation models achieve better semantic extraction across the board.
Amazon Bedrock Data Automation empowers businesses to transform unstructured, multimodal content into structured, trustworthy, and actionable data. With minimal setup, highly customizable blueprints, and a scalable project-based architecture, BDA helps organizations reduce manual workload and unlock insights faster.
OpenAI on Wednesday announced Jalapeño, its first custom inference accelerator, co-developed with Broadcom and supported by Canadian electronics manufacturer Celestica, and the first step in its multi-generation compute platform.
The AI company says Jalapeño was designed to work with all large language models (LLMs) and will help make AI faster, better, and cheaper. Behind that rosy mission, OpenAI isn’t shy about its desire to own the full AI stack, something more AI giants are already leaning into.
“Those serious about platforms should be serious about silicon.”
As Ben Bajarin, CEO and principal analyst at consumer technology research firm Creative Strategies, posted on X: “Those serious about platforms should be serious about silicon.”
Remember my mantra, yes this is an intentional play on Alan Kay's quote.
But with few technical details released, developers are left wondering if OpenAI’s widening footprint will be empowering or restrictive.
Get in, Big Tech. We’re all building in-house chips now.
OpenAI isn’t the only Big Tech name to mint its own AI chips.
Way back in 2016, Google designed and built its own custom hardware for TensorFlow, its machine learning software, the Tensor Processing Unit (TPU). A couple of years later, Amazon debuted AWS Inferentia, its first purpose-built chip for AI and ML. Trainium then hit the scene in 2022, shortly followed by Microsoft’s Azure Maia AI Accelerator in 2023. And nothing is certain yet, but in April, Reuters reported Anthropic is contemplating designing its own chips, though the AI company remains noncommittal for now, at least publicly.
Why is everyone jumping on the custom-chip bandwagon?
Blame the compute gold rush, as AI companies increasingly clamor for compute power — ”a compute-powered economy,” as Greg Brockman, president, chairman, and co-founder, OpenAI, puts it. And the numbers surely back it; Stanford’s 2025 AI Index Report says, “training compute doubles every five months.”
While building custom AI chips in-house doesn’t completely alleviate compute pressures, it is one way for OpenAI and its Big Tech brethren to expand compute capacity while potentially lowering costs and reducing reliance on third-party suppliers.
Exciting claims, no proof
In its announcement blog post, OpenAI describes its new chip as “designed to be the best inference platform for LLMs.”
Specifically, Richard Ho, head of hardware at OpenAI, states:
“We optimized the architecture around the kernels, memory movement, networking, and serving patterns that matter most for frontier AI models. Based on early testing, Jalapeño will efficiently execute our most important workloads close to the hardware’s theoretical limits.”
But the AI company remains tight-lipped on any real technical details.
While it claims current tests put Jalapeño’s performance “substantially better than current state-of-the-art,” it doesn’t provide benchmarks to back that up. Instead, it tells developers to expect a detailed technical report “in the coming months.”
What OpenAI does divulge is that engineering samples of the chip are currently running on ML workloads in its lab, including GPT-5.3-Codex-Spark.
Will Jalapeño serve developers, or is OpenAI’s desire to own the AI stack?
OpenAI makes no qualms about its quest for full-stack control. In doing so, the AI company claims it will make its models “faster, more reliable, and more affordable for users.”
Its logic goes a little something like this: Better infrastructure means more efficient compute, which means better training, which means better models, which means better products, which means more revenue. Then, it explains, it can reinvest that revenue in its infrastructure to make intelligence better for everyone.
But given how little the AI company has revealed about the chip’s specs, it seems developers will have to sit back and watch where the chips fall.
Jalapeño, then, is simply the next move in OpenAI’s quest to control the whole AI chessboard, moving beyond models and products to the underlying infrastructure itself.
For developers, OpenAI seems adamant on insisting its full-stack strategy will lead to better performance and pricing for everyone and ultimately empower “anyone trying to learn, create, or solve hard problems.” Still, it’s worth considering: As OpenAI’s grip tightens, will developers become beholden to its ecosystem?
Several times in its announcement, OpenAI reiterates that it designed Jalapeño for current and future LLMs — all of them. But given how little the AI company has revealed about the chip’s specs, it seems developers will have to sit back and watch where the chips fall.
Built fast with a long roadmap ahead
The few behind-the-scenes details OpenAI does choose to share boast about its development speed, stating it brought Jalapeño from design to manufacturing tape-out in nine months — “what we believe to be the fastest ASIC development cycle ever achieved in high-performance advanced semiconductors.”
The AI company chalks up that fast timeline, in part, to its own models accelerating parts of the design and optimization processes.
Looking ahead, Jalapeño is slated for deployment at a gigawatt scale in Microsoft’s and other partners’ data centers by the end of the year.
That’s just the beginning. OpenAI hints at an upcoming multi-generation roadmap, posing the question: What will it seek to control next?
Organizations are racing to apply AI across the enterprise, and infrastructure is one of the most compelling targets: automated provisioning, self-healing networks, and agents that deploy and manage servers without human intervention. The promise is real, but so is the risk.
No matter the domain, AI agents are only as good as the data they’re given. Agents without a complete and accurate picture of the network and associated infrastructure will make confident mistakes. In infrastructure, those mistakes have brand and revenue-related consequences: exposed databases with PII, failed deployments, and outages that take the entire business offline.
“Agents without a complete and accurate picture of the network and associated infrastructure will make confident mistakes.”
Most enterprise infrastructure is managed through a patchwork of siloed, fragmented tools: separate systems for IP address management, data center inventory, and device configuration. The list goes on.
Before you can trust an AI agent with your infrastructure, you need to give it something to trust: a single, unified model of what’s on your network, how it’s configured, and how it’s supposed to behave. According to NetBox Labs CEO and cofounder Kris Beevers, that’s an Infrastructure Intelligence platform.
What is infrastructure intelligence?
Whether run by AI or human agents, infrastructure is impossible to manage when critical systems contain unknowns. Infrastructure intelligence is the foundational blueprint of your infrastructure: a unified, continuously updated model that captures not just what exists, but what is intended, what has changed, and what needs attention. It is the prerequisite for automation at any scale.
“AI is raising the stakes for infrastructure management, and the challenge is no longer just documenting infrastructure; it’s also understanding it…”
“AI is raising the stakes for infrastructure management, and the challenge is no longer just documenting infrastructure; it’s also understanding it,” says Beevers. “A source of truth was enough for the last decade. But today, teams need context – a trusted, continuously updated understanding of infrastructure that helps them (and their AI agents) model, see, act, and govern with confidence. AI doesn’t eliminate the need for infrastructure data. It makes it more important than ever.”
It starts with a system of record. More than just an inventory list: it is a living representation of the intended state (what everything is supposed to look like) and the operational state (what it actually looks like right now) of your network. The gap between these two states is drift, and that is where risk lives. Without a system that tracks both states simultaneously, your team is always reacting, chasing down misconfigurations, and manually reconciling tool outputs (hoping nothing critical slips through).
Full infrastructure context connects the intent and design to the operational state, providing drift detection, observability, and lifecycle management tools in a single continuous data thread. Instead of switching between five different tools to answer a single question about a specific device, your team and your agents have all the information they need in one place. What is the device supposed to be doing? What is it actually doing? When did it change, and who changed it? Full context means that these questions have immediate answers.
Guardrails close the loop. Both humans and AI agents can make well-intentioned errors, and in infrastructure, the blast radius of these errors can be severe. For this reason, your infrastructure must have well-defined audit trails, branching workflows, change management processes, and operational validation from the beginning, not as an afterthought after something goes wrong.
Teams move from handholding every agent action to trusting the system to catch bad outcomes. Cautious early adoption quickly grows into confident, autonomous, scaled deployments.
The foundation for any automation journey
Agentic automation/Agentic NetOps is coming to infrastructure teams, whether they are ready or not.
No matter where a company is in its automation journey, Infrastructure Intelligence provides a strong foundation for everything else. Organizations that are early in their automation strategy have manual processes they want to automate — they need a clear picture of the environment to do this safely. Teams that are running agentic workflows across complex, multi-site networks share the same requirement: Infrastructure intelligence.
NetBox Labs, the commercial steward of the open-source NetBox, recently expanded its platform to ensure that every infrastructure management workflow can be addressed by agents. The announcements make infrastructure AI Agent-Native: Extending the NetBox MCP server across the entire NetBox Labs Platform and releasing an array of pre-built agent skills.
These agentic tools are designed to leverage the existing infrastructure intelligence from NetBox Labs’ systems, ensuring that all agentic network provisioning capabilities are combined with the required guardrails, validations, and protections to keep the network running smoothly.
Adding agentic features across the entire NetBox Labs infrastructure intelligence platform gives agents unprecedented knowledge, skills, and power. Agents can access NetBox Data Exchange — the world’s largest database of infrastructure metadata. NetBox Assurance and Discovery helps teams and their agents identify and mitigate drift.
“Giving AI agents access to production infrastructure without guardrails is a recipe for outages.”
According to NetBox CEO and cofounder Kris Beevers, “The future isn’t just autonomous infrastructure. It’s a trustworthy infrastructure. We know that trust and governance are the foundation of AI-driven operations. Giving AI agents access to production infrastructure without guardrails is a recipe for outages. That’s why we’ve paired these AI agent native updates with new validation tools so teams can ask, “‘Is this change safe to deploy?’ and ‘What breaks if this fails?’”
The new validation tools help agents self-correct, validate changes, and meet compliance requirements, ensuring continuous compliance and pre-change safety within the System of Record.
AIOps teams that establish a foundation of infrastructure intelligence gain more than efficiency, visibility, and control. They gain the confidence to automate services in production without losing sleep over it. Agents stop guessing and operate from verified real-time data. Teams stop reacting and focus on building. And the organization does not see AI as a liability, but as a capability that can be expanded.
NetBox Lab’s new infrastructure intelligence platform is designed for both humans and agents, making it easier to manage infrastructure across every lifecycle stage — from design through end-of-life.
Whether you’re a NetBox open source user or NetBox Labs customer, you can celebrate NetBox turning 10 at its inaugural conference, NetBox Evolve, which will be in Florida at the Kennedy Space Center on October 13, 2026.
This week, Sakana AI released Fugu, a multi-agent orchestration system designed to deliver frontier-model performance all while reducing the risks of relying on a single provider.
The Japanese AI R&D company says Fugu performs as well as Anthropic’s Fable 5 and Mythos Preview on engineering, scientific, and reasoning benchmarks by breaking up tasks into subtasks and strategically routing them across a swappable pool of expert agents. But early reactions are mixed.
While Sakana positions Fugu’s “collective intelligence” as the blueprint for AI sovereignty, not all users report frontier-model-level performance. Others note fast burn rates and unnecessarily high prices.
Many agree that, though interesting, Fugu likely won’t be the hero to AI sovereignty it hopes to be.
Is this just another router? Not really.
Sakana says Fugu’s internal routing logic is founded on its own research in learned model orchestration, specifically noting two papers, Trinity and the Conductor.
Unlike multi-model routers, such as OpenRouter Fusion, that send a prompt to multiple models and then compare or combine the results, Fugu breaks down user prompts into subtasks and determines which subtask to send to which model. In this way, Fugu “dynamically orchestrates the world’s best models to tackle complex, multi-step tasks,” so Sakana says.
From the outside, you just see what looks like one model, accessible via a single OpenAI-compatible API.
But what the AI company doesn’t tell you is how it decides which tasks get routed where; that information is proprietary. From the outside, you just see what looks like one model, accessible via a single OpenAI-compatible API.
“relying on a single company’s model for national infrastructure is a massive risk. As recent export controls have shown, access to top models can disappear overnight.”
Fugu doesn’t have to farm out every task, though. It’s a language model itself, specialized for model selection, delegation, verification, and synthesis internally, so it can also solve requests directly when its own response is sufficient.
A hero for AI sovereignty, it appears not
In an X post, Sakana CEO and co-founder David Ha writes, “relying on a single company’s model for national infrastructure is a massive risk. As recent export controls have shown, access to top models can disappear overnight.”
Human intelligence is fundamentally a collective intelligence. We solve complex problems by participating in a vast cultural network that builds upon ideas across generations.
I believe the strongest AI systems will become a collective intelligence, too.
That “massive risk” comment is likely a jab at what happened to Anthropic, when an export control directive forced the AI company to pull Fable 5 and Mythos 5 just three days after launch.
Following this news, Sakana positions Fugu as the antidote to single-provider reliance. Because it relies on a pool of “entirely swappable agents,” the idea is that Fugu is less likely to leave users in a bind if one provider suddenly restricts access. It can simply route work to other models.
The AI company considers this capability enough license to claim it’s “delivering the realistic, resilient blueprint required for AI sovereignty.” But some initial reactions call that hyperbolic:
“This is just a highly advanced router/wrapper, not a fundamental leap like Mythos/Fable was,” argues one Redditor.
Though it’s likely not fair to call Fugu a simple multi-model router, its ultimate reliance on other models means it’s not the hero for AI sovereignty it aspires to be. After all, if more than one model provider restricts access at the same time, Fugu’s capabilities also take a hit.
As another user writes on HackerNews: “As a developer outside the US I think it’s vital to have alternatives to OpenAI and Anthropic, but sadly this is not it,” calling out what they describe as the tool’s unfortunate price-to-burn-rate ratio, an “extremely slow” API, and poor quality in comparison to Fable:
“It’s nowhere remotely near usable as a day-to-day workhorse.”
Not all user reviews back up the benchmarks
Sakana points to coding, reasoning, science, and agent benchmarks to prove Fugu’s value, stating its tool consistently beats Gemini 3.1, Opus 4.8, and GPT 5.5.
Source: Sakana AI
It also highlights what it says is the success of its beta program, where almost 500 early users tested Fugu on lengthy, multi-step computational workflows.
In particular, it claims that one cybersecurity engineer confirmed Fugu successfully operated within parameters and avoided destructive actions, while other teams praised Fugu Ultra for besting GPT 5.5 in code review and maintaining an “unusually strong persona stability across long sessions.”
But moving from benchmarks and PR-ready examples to early community sentiment adds more color to the story.
One user on HackerNews calls Fugu “quite strong” for a few agentic coding tasks, but notes they weren’t able to do many deep reviews before their quota ran out, adding: “For implementation I found it weaker, it made a few mistakes that I haven’t seen frontier models make in a long time.”
A Redditor had a different experience. They, too, bemoan burn rate issues, but note: “It caught things Opus 4.8 ultra and codex 5.5xhigh clearly missed in a fairly large data ingestion / processing project.”
Some users question the price tag
Furu is generally available today in most regions (save the EU) in two tiers: a low-latency model that integrates with chatbots and tools like Codex for daily tasks and Fugu Ultra, the heavy-hitter that coordinates a deeper pool of experts for more complex, high-stakes tasks. (This is the one that’s supposed to rival Fable 5 and Mythos Preview.)
Subscription plans are available at $20, $100, and $200 monthly rates for both Fugu and Fugu Ultra. Pay-as-you-go pricing is also available, with Fugu billed at standard rates per underlying model, and Fugu Ultra running at $5 per million input tokens and $30 per million output tokens, with higher rates when context exceeds 272k.
Several early users on Reddit and HackerNews deem these price tags too high, especially when they’re experiencing what now feels like the soundtrack of new agent tools: burn rates that get away from you too fast.
As one HackerNews user jabs: “I love when they put a black box in front of the other black boxes so I can get a questionably better black box for slower service and more money!”
Is collective intelligence the future?
On X, HA posits that large-scale, monolithic models have had their time in the sun and that solving more complex real-world challenges will require a different beast: collective intelligence.
Moving forward, Sakana plans to incorporate new models in its agent pool, which could shore up that resilience Sakana is aiming for. But so far, users seem to question whether paying another company to sit between them and frontier models is really worth the spend.
Kubernetes teams automate deployments without thinking about it. CI/CD pipelines fire dozens of times a day, autoscaling adjusts replicas in the background, rollback is muscle memory. But there is one category of automation where that confidence vanishes: letting a system change CPU and memory requests on a running workload without a human reviewing it first.
And as AI inference lands on Kubernetes at scale, that hesitation is becoming hard to ignore, and increasingly expensive.
Why teams trust automation for change but not for constraint
We surveyed 321 Kubernetes practitioners at enterprise organizations earlier this year. The headline finding is one most practitioners will recognize immediately: 82% report high or complete trust in automated delivery controls. But 71% still require human review before applying resource optimization recommendations. Only 27% allow CPU and memory changes to be auto-applied, even within guardrails.
“Deploying code feels additive… rightsizing feels subtractive because you are removing safety margin from a running service, and the failure mode is fundamentally different.”
Those numbers describe a specific asymmetry. The same engineers who deploy to production dozens of times a day without hesitation slow down the moment automation wants to adjust resource allocation. And the survey data make it clear why. Deploying code feels additive. You are shipping new value, the rollback path is well understood, and if something breaks you usually see it right away. Meanwhile, rightsizing feels subtractive because you are removing safety margin from a running service, and the failure mode is fundamentally different.
As one practitioner in the survey put it: “Automated right-sizing carries a unique risk because it directly impacts the underlying stability of the application runtime. Unlike a code deployment that follows a tested path, resource changes alter the invisible contract between the workload and the scheduler.”
When you change resource requests, you change how Kubernetes schedules, prioritizes, and allocates resources. Those effects are not visible the way a code change is. You can’t trace them through a deployment pipeline. And you might not discover that something went wrong until two weeks later, when a traffic spike hits a threshold that didn’t exist at the old values. By that point, three other things have changed too, and proving causation is nearly impossible. The people responsible for those workloads are the same people who get paged at 2 a.m., and they know this.
Why AI workloads raise the stakes
That trust gap existed before inference workloads showed up. What’s changed is the cost of not closing it.
For a long time, teams could absorb the cost of manual oversight. They knew their workloads, had intuition for where the safe boundaries were, and the inefficiency of over-provisioning was a price worth paying for stability. GPU-accelerated inference workloads change that math. GPU compute is significantly more expensive per hour than CPU. The cost of over-provisioning is no longer a rounding error you can absorb quietly. And the workload behavior is less familiar, as inference jobs are bursty in ways teams haven’t built intuition for, traffic patterns shift as models are updated and usage changes, and the resource dimensions involved differ from what teams have spent years learning to tune.
That unfamiliarity compounds with scale. Rightsizing isn’t a one-lever problem the way horizontal scaling is. It involves, at minimum, CPU and memory requests and potentially limits for both, with four dimensions per workload, multiplied across hundreds or thousands of workloads per cluster. The survey data indicates that manual optimization breaks down at around 250 changes a day. Inference workloads push teams past that threshold faster than anything they’ve managed before, because the resource decisions are more frequent and the cost of getting them wrong is higher.
The economic case for automated rightsizing has never been stronger. The organization’s willingness to delegate hasn’t caught up because teams are being asked to trust automation with workloads they don’t yet have a track record with.
What the survey says about closing the gap
When we asked practitioners what would actually increase their trust in optimization automation, 48% said visibility and transparency into how decisions are made, 25% wanted proven guardrails, and 23% needed instant rollback.
Nobody asked for full manual control and very few asked for blind autonomy. What they described is automation that earns trust in stages, and that’s consistent with how the teams furthest along in their automation journey actually got there. They didn’t start with production. They started with a single namespace in a dev environment, observed the system’s behavior, compared recommendations with outcomes, and gradually expanded the scope. Different environments remained at different levels of automation maturity simultaneously, and that was intentional. Production carried more scrutiny than dev.
CI/CD followed the same curve, and the timeline is easy to forget. Most organizations took years to get from running their first automated pipeline to trusting it with production deploys without manual approval on every commit. Kubernetes resource automation is earlier in that same process, and AI workloads are extending the timeline because teams are building trust from scratch with a workload category that doesn’t yet have a track record.
Why automation design matters as much as capability
Some automation architectures deliver meaningful value only with full delegation. The system needs complete control to function the way it was designed to. That’s a form of forced autonomy, and it creates an adoption problem because it asks for exactly the level of trust that most organizations haven’t built yet. Force generally doesn’t work. Teams that feel pushed into a level of delegation they aren’t comfortable with tend to pull back entirely after the first incident.
The alternative is what I’d describe as adaptive autonomy: designing the system to work at every stage of the trust curve. A team still evaluating gets useful recommendations in read-only mode. A team ready to act but wanting boundaries can run guardrailed execution within limits they define. As confidence grows, the system handles more decisions autonomously while humans manage exceptions. And for environments where the track record supports it, closed-loop optimization runs in the background and becomes boring, which is the goal. Each stage is a legitimate operating mode, not a stepping stone you have to rush through.
That design distinction matters more with AI workloads than it ever did with traditional services, precisely because the trust-building process is starting from zero on workloads where the cost of getting it wrong is highest.
“Trust takes a long time to build and a single production incident to undermine.”
The other piece that makes this sustainable is rollout safety. Trust takes a long time to build and a single production incident to undermine. Start with the workloads showing the most headroom between requests and actual usage. Make changes incrementally, small enough that a bad outcome stays contained. Rollback needs to be fast and tied to the health signals the team already monitors. And start with opt-in, not opt-out. Let the teams willing to go first build a track record that others can look at.
The broader pattern
The 71% figure is sometimes read as resistance to automation. I think it’s a more accurate picture of how operational trust actually forms: conditional, earned over time, and moving at different speeds depending on what’s at stake. AI workloads are raising those stakes significantly, which means the path to trusted automation matters more now than it did when the cost of caution was just some unused CPU.
“Most of what gets written about Kubernetes optimization focuses on tooling capability, and the tooling is capable. The harder problem is the human one.”
Most of what gets written about Kubernetes optimization focuses on tooling capability, and the tooling is capable. The harder problem is the human one. If your team is managing AI inference workloads on Kubernetes and your optimization tooling is sitting in read-only mode, the question worth asking isn’t whether to trust the system. It’s whether the system is designed to let you build that trust gradually, starting where the stakes are low and expanding as the evidence supports it, on workloads where getting it wrong costs more than it ever has before.
For the past two years, the conversation about AI-assisted software development has been dominated by speed. A new GitLab survey of more than 1,500 developers and technology leaders found that 60% say AI coding ROI has already exceeded expectations, and 78% report their teams are writing and committing code faster since adopting AI tools.
But speed without control is a liability.
Most organizations have pursued agentic engineering by adding AI coding tools on top of their existing infrastructure. Coding agents are delivering speed, but that speed isn’t showing up across the full software lifecycle: Only 21% of respondents report productivity gains beyond code generation itself.
“Speed without control is a liability.”
The infrastructure problem runs deeper. Git backends, toolchains, and governance frameworks were built for human-scale concurrency. Agents operate at machine scale, and that mismatch shows up fast. Platform reliability breaks down with millions of agent sessions hitting the same backend, security exposure widens as agents touch dependencies at scale, and cost overruns mount as agents consume tokens inefficiently on infrastructure that wasn’t built for them.
Agentic adoption outpaced governance
The adoption curve for AI coding tools outpaced the development of required guardrails, with 80% of organizations saying they adopted AI tools faster than they developed policies to govern them, and 82% reporting that AI-generated code risks creating a new form of technical debt that their organizations are not prepared to manage.
In practice, that means platform reliability challenges under agent load, security and compliance exposure that widens as agents touch dependencies at volume, and agents operating with artificial confidence because they lack full context. Only 28% of organizations say their software development lifecycle tools are fully integrated with shared data and workflows, which means most teams are trying to govern agent actions across a toolchain that was never designed for them.
Agentic engineering needs agentic infrastructure
Agentic engineering requires two things: agentic coding and agentic infrastructure. Most organizations have the first but lack the second.
Agentic infrastructure spans four areas: the execution layer, the context layer, the governance layer, and the orchestration layer working together.
The first is machine-scale execution. Git backends, CI/CD pipelines, and deployment systems were designed for human-paced development. In the agentic era, they need to handle millions of agent sessions without breaking. When a production incident occurs, the path from symptom back to origin should take minutes, not days.
The second is context that travels with code. As Bastian Stahmer, Business Owner of Vehicle Software Development Platform at Mercedes-Benz, put it on a panel recently, “An agent can only be as good as the context and semantics fed to it.” A context graph connecting code, work items, pipelines, security findings, and production signals is what makes agents genuinely useful at scale and keeps artificial confidence in check.
“An agent can only be as good as the context and semantics fed to it.”
The third is governance built into the flow. Agent actions need to be tied to an identity, logged against a policy, and provable to a reviewer. Low-risk changes move fast, while higher-risk changes trigger review. For Mercedes, operating under automotive regulatory standards that require full traceability and human accountability, GitLab is the control plane where that accountability lives.
The fourth is orchestration. Execution, context, and governance are only as effective as the system coordinating them. The orchestration layer coordinates agent actions across the full software lifecycle according to the policies teams define, determining which agents run, in what order, and how failures and handoffs are managed. Without it, agentic infrastructure is a set of independent capabilities rather than a working system.
What’s next
The next phase of AI in software will focus less on generating code and more on governing it, according to 85% of respondents. That shift reflects how enterprises are maturing their thinking about AI, from a productivity tool to a foundational capability that needs to be trusted, traced, and maintained at scale.
When governance is built into the platform, speed and control are no longer in tension. Traceability becomes a competitive advantage. Context becomes institutional memory. And the codebase, rather than accumulating invisible risk, becomes an asset that grows more reliable over time.
Login
