AI coding company Replit is throwing its weight behind the model-routing trend by making its “intelligent model routing” system the default across every account.

The system automatically selects the underlying model to handle a task as it evolves, with Replit weighing quality, speed, and cost in its routing decisions.

The company says the feature, dubbed Auto mode, will become the default option for all users, though Core and Pro subscribers can still override it and manually select models when they want more control.

Model-routing momentum

The announcement comes hot on the heels of a flurry of activity in the model-routing realm. Earlier in August, Stripe agreed to acquire model gateway platform OpenRouter for a reported $8 billion. On the very same day, Ramp launched Router.com, which routes requests to the lowest-cost model that meets a specified performance bar.

Before all that, in July, SpaceX-owned Cursor launched its own router, which automatically selects models for coding requests and claims to deliver comparable performance at a substantially lower cost. Meanwhile, Meta is reportedly developing an internal router called “Switchboard” that scores coding tasks by difficulty and sends simpler jobs to cheaper models.

“Across one model family, per-token rates can span orders of magnitude. At the same time, the intelligence of cheaper, smaller models is now much closer to their larger frontier counterparts, providing us a lot of room for cost optimizations.”

Michele Catasta, president and head of AI at Replit, says that one reason for the wider push into routing is simple economics — the growing gap between what models cost and the level of capability developers actually need for a given task.

“Across one model family, per-token rates can span orders of magnitude,” Catasta tells The New Stack. “At the same time, the intelligence of cheaper, smaller models is now much closer to their larger frontier counterparts, providing us a lot of room for cost optimizations.”

Replit, for its part, has been moving in this direction for some time. Catasta says that the company has spent recent months experimenting with early versions of Auto mode, subagent routing, and multiple iterations.

“Like any pivotal launch, we thoroughly tested Intelligent Model Routing in beta for a long period of time before we decided to release it in public,” he says. “The most important learning is understanding from first principles the failure modes of every experiment, so we could keep hill climbing on the final system that we just shipped.”

Enter Auto mode

The foundation of that work surfaced last week when Replit introduced Free Mode, a lower-cost Agent mode that doesn’t consume usage credits and uses Auto to choose the model on theuser’ss behalf, subject to usage limits.

Now, that same Auto routing approach is being pushed more broadly across Replit. The company says intelligent model routing will become the default across every account, with all users starting in Free Mode and Replit deciding which model is best suited to the task.

Free Mode, it’s worth noting, isn’t “free” in the sense of unlimited usage. When it launched, Replit made it available to Core and Pro subscribers without consuming their usage credits, but imposed limits that reset every five hours, with higher allowances for Pro users. In Free Mode, users cannot manually select a model.

Core and Pro subscribers can, however, switch to Replit’s Power or Max modes, where they can turn off Auto and choose a model themselves. Replit may also suggest moving a task into one of those higher-powered modes when it determines that more capability is required, though those modes can incur usage costs.

For Enterprise customers, meanwhile, administrators can restrict Auto to an approved set of models for each workspace, allowing Replit to continue routing tasks automatically while keeping model choice within company policy.

The agent advantage

Even before SpaceX agreed to pay a cool $60 billion to acquire Cursor, the AI coding startup had long been investing in its own coding models, including its Composer family. More recently, under the auspices of SpaceX, Cursor has been developing more cutting-edge models, too.

Replit, by contrast, isn’t making ownership of the underlying model layer central to its pitch. Instead, it’s betting that controlling the agent and the systems around it gives Replit enough insight to make better model-selection decisions on the fly.

“Replit has owned, from the start, both the agent harness and the infrastructure surrounding models which in turn allows us to train sophisticated model routers.”

“Replit has owned, from the start, both the agent harness and the infrastructure surrounding models which in turn allows us to train sophisticated model routers,” Catasta said. “Only in this way can we always offer useful intelligence to our users at the most competitive price point.”

That becomes particularly relevant as an Agent task unfolds, with Replit noting that its system can change which model it uses as the task develops, seeking a better trade-off between capability and cost at different points in the process. But for Catasta, that kind of dynamic routing is still only one part of a much broader research problem around how agents should use models.

“Model routing is still in its early development phase, and we expect further research will move the needle on serving the best intelligence when customers most need it,” Catasta explains. “Routing is but one piece of the puzzle that is tightly integrated to many other aspects of our harness research.”

“No third-party router company could reproduce the same results for our own agent.”

Replit also argues that seeing how people use its own Agent gives it an advantage that a standalone routing provider would struggle to reproduce. Catasta says a router has to infer the nature, difficulty, scope, and intent of a request, with Replit able to train against proprietary usage data and observe those signals across its user base.

“No third-party router company could reproduce the same results for our own agent,” he says.

The post Replit’s new default: Auto mode picks the best model for each task appeared first on The New Stack.

OpenAI’s unreleased foundation model, codenamed Astra, is already working directly inside the company’s internal codebase, taking on experimental work that previously required as much as a week from a human researcher.

As part of a series of interviews with Time, OpenAI chief scientist Jakub Pachocki said Astra can take an idea for an experiment, turn it into code, run it, and return the results. A step beyond asking AI to fix a bug or write a function, OpenAI is essentially testing what happens when you hand the agent the experiment itself and let it figure out the steps in between.

For developers, Astra offers a glimpse into the future of coding agents. But giving an agent that much freedom creates another problem. Astra may already be powerful enough to trigger OpenAI’s highest-level cybersecurity safeguards.

OpenAI is essentially testing what happens when you hand the agent the experiment itself and let it figure out the steps in between.

Persistent agents change everything

OpenAI CEO Sam Altman described what the company is building as “persistent agents,” or systems that can keep working without needing a person to prompt them through every step.

Coding agents can already dig through a repository, change files, run tests, and try again when something breaks. Persistent agents are meant to keep going without a developer guiding them through each step.

That also changes what developers need from the tools around the agent. An IDE gives it somewhere to work, but a long-running agent needs infrastructure that can keep it running safely without constant oversight.

Multi-agent coordination at scale

In one demonstration witnessed by Time, 16 Astra agents worked together on a research-level math problem, splitting it into smaller pieces and then bringing their work together into a proposed solution.

For developers, it’s not hard to imagine that same setup applied to a large software project, with different agents working on different pieces at the same time. OpenAI is already experimenting with that kind of coordination, along with agents that can stay on a job for much longer.

Running several agents at once also complicates the infrastructure behind them. Developers need a way to keep the whole operation under control. Giving agents that much freedom, though, has created another problem for OpenAI: keeping them under control.

When agents escape containment

OpenAI said this month that preliminary evaluations indicate Astra may have reached the “Critical” cybersecurity capability threshold in the company’s Preparedness Framework, a finding the company disclosed alongside a pause on some frontier workloads.

Under the company’s framework, hitting that threshold brings stricter safeguards for how the model can be used.

OpenAI had already seen what could go wrong with agents that have access to tools. During a cybersecurity test, one of its internal AI agents escaped its sandbox and accessed Hugging Face systems without authorization. Astra was not the model involved, but the incident led OpenAI to pause some frontier-model research workloads while it tightened the infrastructure used to run them.

And OpenAI isn’t the only company running into this problem. Google’s AI coding agent recently broke out of the boundaries of its IDE. The circumstances were different, and the incident was less serious, but both point to the same problem that happens when giving an agent more freedom to act; you also have to make sure it stays where you put it.

Astra is now running under OpenAI’s strictest security controls. Some training and evaluation workloads have resumed, but OpenAI says a “significant number” are still paused while it upgrades the infrastructure behind them.

Astra is now running under OpenAI’s strictest security controls.

Monitoring costs real compute

The company also says it is monitoring Astra more closely when it uses tools, watching for behavior that could signal it is going beyond what it’s allowed to do. OpenAI estimates that monitoring adds about 20% to the inference compute for those workloads.

Time reports that OpenAI still plans to release Astra, although there’s no launch date yet. Until then, OpenAI is working through the same challenge developers could eventually face as agents work for longer stretches without human supervision and begin coordinating with other agents.

As agents take on more work, the IDE is only part of the picture. Developers also need a way to see what those agents are doing and keep them from going where they shouldn’t. OpenAI is already seeing the cost, with monitoring alone adding about 20% to Astra’s inference compute.

That kind of overhead could become another cost developers have to account for, along with the governance around autonomous agents that sets boundaries on what they can do.

OpenAI is already seeing what that costs, with monitoring alone adding about 20% to Astra’s inference compute.

The post OpenAI’s Astra can do a researcher’s week of work. That’s the problem. appeared first on The New Stack.

Perplexity shipped Portable Computer this week, the local-first version of its Computer agent running on an Nvidia DGX Spark workstation, and the bill is steep: A DGX Spark starts at $4,700, and even an aging 24GB RTX 3090 sells well above $1,500.

The architectural choice beneath the surface deserves as much attention as the price tag: Most platforms building agents address reliability with greater intelligence. That typically includes a larger orchestrator model, a planner model, or a critic model reviewing the work.

Probabilistic reasoning proposes the next action, and deterministic software decides whether to execute it.

Perplexity separated the two jobs instead of stacking them. Probabilistic reasoning proposes the next action, and deterministic software decides whether to execute it.

The loop controller is code, and the decisions are still a model

Perplexity uses the term “orchestrator” to refer to the runtime controller rather than the planning model. That controller assembles context, enforces policy, and executes approved tool calls inside an OS-level sandbox. The company says it is deterministic code, not yet another model. The local model proposes the next action, including which tool to call and when to ask a cloud advisor for help. The split resembles a control plane architecture, where reasoning suggests, and inspectable software retains authority. Nate Kupp, Perplexity’s vice president of Computer Enterprise and Infrastructure, tells The New Stack that the harness accounted for most of the engineering work.

Same weights, better scores

Perplexity kept the base model and the silicon constant, putting Qwen3.8-27B on the same DGX Spark across three agent stacks. On its internal Local Knowledge Work Bench, a held-out set of 53 tasks, the company reported 82.6% for Computer. Pi scored 77.6% and Hermes 74%.

On ParseBench-100, a subset covering charts, layouts, tables, and formatting, the gap widened considerably. Perplexity reported 65.1% for Computer, while Hermes reported 34.6% and Pi reported 13.9%.

Because the base weights were constant, the gap measures the system around the model rather than a better model. That does not mean weights stopped mattering. Perplexity post-trained Qwen into PPLX 27B and reported 85.4%, above its own base-model number. Harness engineering and post-training are significant factors of this approach.

The word harness also covers a lot of ground, including prompts, tool schemas, context management, verification hooks, and document processing. Orchestration code is one part of that surface.

The security boundary lives outside the model

The sandbox is the boundary, not the determinism. Perplexity says the sandbox restricts processes, filesystem paths, and network access. If the sandbox is unavailable, the harness disables itself before making any tool call. Deterministic code is valuable here because it makes policy easy to inspect and helps the system fail safely. Deterministic code can still ship a vulnerability or faithfully execute a permitted mistake.

The key distinction is not between model orchestration and code orchestration. It is whether permission is enforced by something other than asking a model in plain English to comply.

The key distinction is not between model orchestration and code orchestration. It is whether permission is enforced by something other than asking a model in plain English to comply.

The same discipline shapes how context gets spent. Perplexity reports that Qwen3.8-27B advertises a 260,000-token context window but begins to struggle beyond 100,000 tokens. The harness therefore keeps the core prompt and toolset small and loads skills on demand. Commonly used connectors became command-line tools rather than full Model Context Protocol definitions sitting permanently in context.

Deterministic execution cannot rescue reasoning that exceeds the local model. On Terminal Bench 2.1, Perplexity reported 59.6% running locally. Letting the local agent consult Claude Opus 5 raised it to 73.0%, compared with 82.4% when Opus 5 worked alone. All of this remains vendor-reported evidence on a bench that the company has yet to open-source.

For enterprises evaluating local agents, the component to scrutinize is the layer that grants and denies authority, because that is where the platform’s engineering is most evident.

The post Perplexity just separated reasoning from authority. Here’s why it matters for enterprises. appeared first on The New Stack.

Agentic reinforcement learning has been suffering from a disconnect, an uncoupling, and a misarticulation. The polarity arises from how a training engine handles resource management, compared with how a post-training live production harness does.

Microsoft wants to ensure the production harness is engaged from the start to oversee infrastructure services and agent interactions, during both initial training and subsequent reinforcement learning actions.

Redmond’s Microsoft Research division first introduced the Agent Lightning framework in August 2025 as an infrastructure concept for agent optimization to address structural challenges in post-training LLM-based agents as they enter reinforcement learning processes. Microsoft subsequently launched the Agent Lightning v1.0 release with a commit tagged on GitHub on August 16. 

Using Agent Lightning v1.0 on what Microsoft has called “modest compute” 6K training examples, reinforcement learning improves Qwen3.5-9B on OpenAI’s SWE-bench Verified benchmark from 41.8% to 56.4%, an absolute 14.6-point gain.

“Using Agent Lightning v1.0 on ‘modest compute’ 6K training examples, reinforcement learning improves Qwen3.5-9B on OpenAI’s SWE-bench Verified benchmark from 41.8% to 56.4%, an absolute 14.6-point gain.”

Who owns the interaction loop?

In traditional agentic reinforcement learning, the training engine owns the interaction loop, i.e., steps including observing the environment, selecting an action based on the policy, executing the action, receiving a numerical reward, storing and updating the policy… and so on. 

In harnessed agentic reinforcement learning, the harness owns context construction, tool execution, and the agent–environment loop. The training engine observes only a sequence of LLM request–response pairs across a service boundary. Therefore, developers do not need to reimplement the agent loop within the training environment.

According to a collected group of software engineers at Microsoft, because an AI model harness owns the loop that governs infrastructure access and operations procedures during harnessed agentic reinforcement learning, it “introduces challenges” including retokenization (re-segmenting text into new tokens during active training), sample merging, advantage calculation, loss normalization, and training backend scheduling.

All of which, if not addressed, can result in ineffective or unstable training.

How Agent Lightning v1.0 turns the tables

“[In Agent Lightning v1.0] the harness, rather than the trainer, owns context construction, tool execution, and the agent–environment interaction loop, while the training system observes and optimizes the resulting model calls across a service boundary. This formulation preserves the harness’s deployment-time context policy, tool protocols, and execution semantics without requiring its agent loop to be reimplemented inside the RL framework,” explained the Redmond team.

For coding agents, the team has said that it finds existing reinforcement learning frameworks provide limited support, including a lack of data and complete training scripts, and a reliance on large-scale computational resources. To address this gap, with Agent Lightning v1.0, Microsoft provides a “complete data-cleaning pipeline” and reproducible training scripts built on open-source datasets and models.

Is this the end of the training time liability?

For users inside Microsoft environments, this might feel like good news. Machine learning specialists and platform engineers may have a good harness or a good set of reinforcement learning tools; they would rarely have both.

Rather than having to hard-code retokenization, advantage calculation, and reward shaping every time they wanted to train an agent to call external services in reinforcement learning procedures, they can keep their existing agent architecture as an asset, rather than treating it as a training-time liability.

Nebraska-based software engineering researcher Md Rashedul “Rashed” Hasan tells The New Stack that training on the exact production harness matters, but not only for its efficiency and benchmark gains. 

Training through the real harness keeps semantics intact

“It reduces train–serve mismatch,” Hasan says. “If you train inside a simplified trainer loop and deploy inside a different harness, tool protocols, context policy, and recovery behavior can all drift. Training through the real harness keeps those semantics intact, so gains are more likely to transfer to production behavior, not only to a lab environment.”

Hasan thinks that Microsoft has clearly named the paradigm, kept the core framework small, and shipped what he defines as a “concrete coding-agent pipeline” with open data and scripts. 

“In terms of who this will appeal to, it’s application and platform engineers who already have a production agent harness (such as a coding assistant or support-triage agent), and want to improve the underlying model with reinforcement learning without rewriting deployment logic to fit a training framework. Reinforcement learning and machine learning platform teams would also use it when they need a thin, reproducible testbed for harnessed agentic reinforcement learning,” Hasan clarifies.

He agrees that the SWE-bench Verified lift with only 6K examples is “useful proof” that harnessed reinforcement learning can move a hard-coding benchmark without forcing teams to reimplement their agent stack within the trainer.

“Environment setup for coding agents, reward design, evaluation fidelity, and the retokenization, sample-merging, advantage, and loss-normalization details are still easy to get wrong. Adoption will also depend on whether teams can integrate this proxy pattern into existing orchestration, observability, and safety controls,” cautions Hasan.

“Environment setup for coding agents, reward design, evaluation fidelity, and the retokenization, sample-merging, advantage, loss-normalization details are still easy to get wrong. Adoption will also depend on whether teams can integrate this proxy pattern into existing orchestration, observability, and safety controls.”

Killing train-serve skew, the oldest & most expensive bug in machine learning

Colorado-based data science professional Priyank Jain tells The New Stack that training through the production harness to strip the framing away is really all about “killing train-serve skew”, which is the oldest and most expensive bug in applied machine learning.

“Models rarely blow up in production because the math was wrong,” Jain says. “They blow up because the training setup quietly lied to them about what production actually looks like. Training through the same harness that serves the agent is the right instinct, and honestly it’s overdue. The accuracy bump is nice, but the real win is that the thing you optimized is finally the thing you shipped.”

In terms of what Microsoft is getting right here, Jain says it shows Redmond wants to “meet developers where they already are,” i.e., allowing users to improve an existing agent with reinforcement learning without rewriting their deployment just to please a training framework.

“That’s a genuinely good call, and it opens this up to folks who aren’t reinforcement learning specialists. What I’d worry about is that it also lowers the bar for people to run reinforcement learning they don’t fully understand. The hard part was never wiring up the loop; it’s designing a reward that survives contact with a model actively looking for the path of least resistance. Make that part easy, and you’ll get more people optimizing the wrong thing faster,” Jain advises.

“3500 lines of code is small enough that an infrastructure engineer can actually read it before trusting it… and that’s a good thing.”

Just 3,500 lines of core Python code

Built with “simplicity as its first principle”, the entire framework consists of around 3,500 lines of core Python code.

Infrastructure engineering developer and founder-developer of Tooldex, a platform that autodiscovers MCP servers across LLM agents, Ria Banerjee, tells The New Stack that the simplicity element here is a positive, i.e. 3500 lines of code is small enough that an “infrastructure engineer can actually read it before trusting it,” and that’s a good thing.

“But in terms of who would actually use this tool, the honest answer is fewer teams than the framing suggests,” Banerjee says. “Realistically, you would need a GPU cluster and a Kubernetes cluster under your control. An app developer who built a support-triage agent on LangChain isn’t running that. So the real audience is platform teams that already employ infrastructure engineers.”

“Because the harness is where the behavior actually lives, but you’re now baking your harness’s quirks into the model weights… so you change your retry logic next quarter, you’ve silently shifted what the model was trained on. So now, you have to version your harness,” adds Albuquerque-based Banerjee.

Microsoft has released the complete workflow and training scripts to facilitate reproducible, harnessed agentic RL in Agent Lightning v1.0 on GitHub under the MIT license, including data cleaning and reward-hacking prevention.

The post Microsoft just released Agent Lightning v1.0. Here’s why it matters for platform engineers. appeared first on The New Stack.

What do you do when a technology you’ve become dependent on belongs to someone else? You buy the startup behind it, of course. And that’s exactly what data warehouse company MotherDuck has done with Tower, a data infrastructure startup whose technology was already powering MotherDuck’s AI-built data pipelines.

The deal, announced on Tuesday, is MotherDuck’s first acquisition in its four-year history, bringing both Tower’s technology and team in-house as MotherDuck pushes further into AI agents that can build and operate data pipelines.

Tower takes flight

Tower was founded out of Germany in late 2024 by ex-Snowflake engineers Serhii Sokolenko (CEO) and Brad Heller (CTO). Their pitch: once a developer, or an AI assistant, has written the code for a data pipeline, someone still has to package it, deploy it to the right infrastructure, wire up credentials, and maintain it — the unglamorous work, Sokolenko told The New Stack back in March, that amounts to data engineering’s “last mile.”

Tower, essentially, is a managed runtime for Python pipelines — it packages the code, deploys it, and keeps it running in production. It also offers tools built on top of that runtime, like the browser-based AI agent Tower Control that allows users to describe the pipeline they want in plain language.

Control can then generate the code, deploy it as a Tower app, and run it — effectively taking the process from prompt to production without the developer having to set up the underlying runtime themselves.

MotherDuck, for its part, is a serverless data warehouse built on the open-source database DuckDB, founded in 2022 by Jordan Tigani, a former engineering lead at Google focused on BigQuery. The company has raised some $100 million since its inception.

MotherDuck’s original pitch leaned on speed and local compute: queries could run on a laptop via DuckDB, in MotherDuck’s cloud, or across both — a departure from the likes of Snowflake, Databricks and BigQuery’s cloud-first model. More recently, MotherDuck has extended that approach to AI agents, using MCP to let agents interact directly with data. And things took a more operational turn in June with the launch of Flights, a feature that exposes a general-purpose Python runtime through the same MCP server, letting agents create, run and schedule data pipelines.

And Tower, as it turns out, was the key infrastructure underpinning Flights.

‘We became their largest customer almost overnight’

Tower’s involvement with MotherDuck actually predates Flights. Tigani says MotherDuck had initially been looking for a third-party tool it could recommend to customers as an easier way to get data into its warehouses. But then advances in AI changed the nature of the problem the company thought it needed to solve.

“When AI suddenly started to be able to solve data problems, we realized we were thinking about the problem wrong.”

“When AI suddenly started to be able to solve data problems, we realized we were thinking about the problem wrong,” Tigani tells The New Stack. “Claude can solve that problem we were trying to address by writing the connectors to help people move their data, but what it can’t do is the sandboxing and scheduling.”

That left MotherDuck needing somewhere to safely execute the code those agents generated, manage credentials and run jobs on a schedule. And as luck would have it, Tower already provided those capabilities.

“It perfectly solved our problem and let us ship Flights in only a matter of weeks,” Tigani adds.

For MotherDuck, Tower had supplied the missing execution layer; for Tower, that realization translated into a significant customer relationship. “We became their largest customer almost overnight, and our teams have been shipping together ever since,” Tigani says.

Having Tower on its radar so early also gave MotherDuck a chance to test the technology — and the team behind it — before deciding whether to build something similar itself. Tigani says the calculation ultimately came down to how quickly MotherDuck could get the capabilities it wanted into customers’ hands.

“It’s always tempting to build yourself, but after trying out Tower, we realized pretty quickly that there were a bunch of problems we were going to have to solve to make our underlying infrastructure actually work well, and Tower was pretty much exactly what we needed,” he says.

Ultimately, the more central Tower became to what MotherDuck wanted to build, the stronger the case became for owning the technology outright. Once Tower was executing jobs created and scheduled inside MotherDuck, Tigani argues, customers would inevitably hold MotherDuck accountable for the security, reliability and behavior of that runtime.

“There’s a rule I’ve relearned at every infrastructure company I’ve worked at: you can rent a feature, but you can’t rent a foundation,” Tigani says. “When an agent inside MotherDuck builds a job and schedules it, the thing executing that job is our product — whatever logo is on it.”

“There’s a rule I’ve relearned at every infrastructure company I’ve worked at: you can rent a feature, but you can’t rent a foundation.”

One example of what MotherDuck now hopes to do with that technology involves bringing Flights together with Dives, an AI-generated data visualization feature it debuted in February. Tower can generate stable URLs for jobs running through Flights, effectively allowing those jobs to act as data APIs that a Dive — or another frontend — can call.

Tigani gives the example of an application displaying user recommendations. A Dive could generate the interface for viewing those recommendations, while a Flight could handle requests to create or modify them. Rather than giving the frontend broad write access to the underlying data, the Flight can constrain and validate what changes users are allowed to make.

“When you put them together, you can build rich applications,” Tigani says.

Tower’s next chapter

All of this raises an obvious question for Tower customers. Part of the startup’s pitch was that developers could use its runtime without tying themselves to a particular data platform, and now Tower itself belongs to one.

Tower co-founder and CEO Serhii Sokolenko argues that MotherDuck represents a different kind of home to that of the industry’s larger cloud data platforms. His case is that Tower can become more deeply integrated without being bent around an architecture established long before AI agents entered the picture.

“Joining a hyperscaler usually means adapting to its legacy architecture,” Sokolenko tells The New Stack. “Joining MotherDuck lets us help shape where data and AI infrastructure are heading.”

“Joining a hyperscaler usually means adapting to its legacy architecture.”

There is still a trade-off, though. Tower is giving up some of the breadth that came with being database-agnostic in exchange for building much more specifically around one platform — a bet that tighter integration will ultimately produce a better experience than supporting many systems at arm’s length.

“By focusing Tower’s Pythonic compute specifically on MotherDuck, we’re trading broad, basic connectivity for deep, native execution,” he says.

Sokolenko’s argument is that the “lock-in” question then shifts down a layer. Tower may now be tied much more closely to MotherDuck, but because MotherDuck itself is built on DuckDB, he says the underlying data remains open and portable. The aim is to bring the runtime, agents and warehouse closer together without trapping the data itself inside a proprietary system.

That also helps explain why MotherDuck’s hybrid execution model appealed to Tower. DuckDB lets work move between local and cloud compute, which Sokolenko sees as closely aligned with Tower’s own direction.

“This directly matches Tower’s vision — allowing business users and agents to move seamlessly from local data exploration to cloud production execution,” he says.

For Tower’s existing customers, the immediate future means a move toward MotherDuck. Sokolenko says Tower customers are already in discussions with MotherDuck about migration paths, while people who have used Tower previously will be invited to try MotherDuck and its broader agentic data capabilities.

Tigani confirms that MotherDuck is working to move existing Tower customers onto Flights, though he concedes that the two products aren’t entirely identical. “There are a couple of differences, and we’re working on closing the gaps to make the transition more seamless,” he says.

Tower’s technology will meanwhile be folded more deeply into MotherDuck itself. Today, MotherDuck has two separate sandboxed, on-demand runtimes: Flights, which is backed by Tower, and Ducklings, its serverless DuckDB instances. Tigani says the plan is eventually to merge the two, combining the near-instant startup of Ducklings with the more robust sandboxing provided by Tower jobs.

Surfing the agent wave

MotherDuck is hardly alone in pushing data agents beyond answering questions. Databricks’ Genie Code can generate and run code, build pipelines and debug failures inside Databricks. Snowflake, meanwhile, has been moving in a similar direction with the likes of CoCo, its AI coding agent, while newer CoCo Automations can schedule unattended agent runs inside Snowflake-managed sandboxes.

The details differ, but both point toward a broader shift in the data industry: giving AI agents the infrastructure to act on data and operate the systems around it, rather than simply query what is already there.

“AI makes it possible to build useful features that we could never fathom five to 10 years ago. The platform is the most complex part of the data estate, and so warehouse vendors are well positioned to be at the center of whatever new patterns emerge.”

Tigani has been predicting something close to this for some time. Earlier this year, he outlined a future in which data engineering increasingly becomes an agent-supervision problem, with agents handling tasks such as building and repairing pipelines and responding to changes in schemas and data quality, while humans oversee their work. He also previously likened the advance of LLMs to a wave that data companies will have to learn to ride.

“The way I like to think about it is this — data platform vendors are reacting to new opportunities to make lives better for their customers,” Tigani says. “AI makes it possible to build useful features that we could never fathom five to 10 years ago. The platform is the most complex part of the data estate, and so warehouse vendors are well positioned to be at the center of whatever new patterns emerge. The Tower acquisition gives us a platform for deploying, tracking, and scheduling data agents, which should set us up well to surf that wave.”

The post “You can rent a feature, but you can’t rent a foundation”: why MotherDuck bought the startup already powering its data pipelines appeared first on The New Stack.

Put several AI bots to work, and a mistake by one may not stay within its assigned task. For example, that error could reach another bot’s files and login credentials, or even the computer running them all. Two releases this month offered companies very different ways of containing that risk.

On August 17, Nous Research announced that its Bot Mode would ship bundled and enabled by default in Hermes Agent v0.20.3, turning agent profiles into a roster of named bots that hand off work to one another. About a week earlier, SpaceXAI launched Grok Bot with almost the same interface: a sidebar of named teammates who sign in to your tools and keep working long after you close your laptop.

The interface converged within a week, but the answer to the question every platform team has to ask did not: When one bot goes wrong, what can it reach?

Four projects have now come to their own answer, and no two of them agree.

1. Grok Bot draws the line around the user account.
2. Hermes draws it around the profile.
3. OpenClaw draws it around an optional runtime sandbox.
4. ClawFleet draws it around a container.

Taken together, the documentation shows an industry converging on the persistent coworker interface far faster than it is converging on what constitutes an identity or a security boundary for it.

Four projects, four written answers

Every one of these products now offers the same surface. You create several named agents, assign them different jobs, and have them pass work among themselves. The naming convention alone suggests separation, since a bot called Expense Manager and a bot called Talent Scout sound like they occupy different rooms in a shared office.

The documentation says otherwise, and it says something different in each case. The unit of isolation is the account in one product, the profile directory in another, an opt-in container in a third, and the deployment topology in the fourth. Those four units are not interchangeable, and an operator who assumes the roster itself is the boundary will be right in exactly one of the four cases.

Is Grok Bot confused about what it wants to be?

SpaceXAI’s launch post leads with the promise that bots have their own computer. The documentation, last updated the same day, describes a single persistent cloud computer assigned to the user account rather than to any individual bot. Browser cookies and signed-in sessions are shared across the roster, files are visible to every bot, and command-line credentials are shared. One bot can pick up work that another bot saved.

Each bot gets its own screen on that machine, which allows several of them to run browser and desktop tools in parallel. SpaceXAI is direct about what those screens are not. The documentation calls them “separate work surfaces, not separate security boundaries.” It then instructs operators to keep a credential or file off the machine entirely if another bot on the account cannot use it.

The consequences run further than credentials. Signing in for one bot makes that session available to the others because the browser is shared. Installed connectors are account-wide, and their availability is not isolated to a single bot. The shared workspace sits at /workspace and is designed to survive computer updates and recovery, so the durable state is shared across the whole roster.

None of this is an implementation accident. It is what makes handoffs between bots cheap, and cheap handoffs are the product. But an operator reading only the launch page would build a mental model that the documentation contradicts, and that gap is where the risk sits.

Hermes gives each bot its own profile

Nous took the opposite architectural position. In Hermes, a bot is a profile, and each profile has its own configuration, memory, skills, credentials, and chat history stored in its own directory on disk. Handoffs between bots run as real invocations against the named profile, rather than as a shared context blob passed around within a single process.

Nous shipped the teammate protocol as part of v0.20.3, alongside the MCP 2.x SDK migration and a set of runtime hardening changes. The company archived the standalone plugin repository once the merge was completed. Bot Mode is on by default, so a Hermes user who updates gets the roster without opting in.

Two qualifications matter before anyone reads that as containment. A separate credential store does not guarantee different credentials, since what ends up in a new profile depends on how the operator created it and what they edited afterward. And every profile still shares the host machine, its operating system user, and its filesystem permissions. What Hermes documents is workstation-level separation of agent state, a meaningfully stronger default than a shared cloud account, but not the same as isolation.

OpenClaw’s sandbox is off by default

OpenClaw documents the most complete boundary of the four. When the sandbox is enabled with the Docker backend, agent tool execution runs inside isolated containers. At the same time, the gateway remains on the host, and the scope can be selected per session, per agent, or shared across agents. Each scope gets its own workspace. Auth material lives per agent under an agent-scoped auth profiles file. Operators can configure network isolation, resource limits, and allow-or-deny tool policies on top of it.

The documented default for that sandbox mode is off. That is a defensible choice for a project most people run on a laptop, where the container overhead buys little against a single-user threat model. The underlying setup behavior deserves more attention. If sandbox prerequisites fail during setup, the script resets sandbox mode to off rather than refusing to start, so an operator who intended isolation and encountered a Docker socket issue ends up running without sandbox isolation. The documentation also warns against mounting the host Docker socket into agent sandbox containers and flags the CLI container’s shared network namespace with the gateway as a trust boundary in its own right.

ClawFleet answers the same question by moving it into the deployment topology. The project documents a wrapper that puts each OpenClaw or Hermes agent in its own Docker container with an isolated filesystem and network. It lists roughly 500 MB of memory per OpenClaw instance and 150 MB per Hermes instance. That cost is why the other three projects make the boundary optional or skip it, and naming the number makes the trade-off legible.

How to choose which bot is right for you

Each project gives operators different advice

The operational guidance diverges as sharply as the architecture. OpenClaw’s docs read like infrastructure documentation, naming specific hazards such as the Docker socket and the shared network namespace, and telling the operator what not to do. Hermes documents the profile layout and the protocol, then leaves policy to the operator. Grok Bot’s guidance is largely the warning itself, an instruction to treat the account as the boundary and to keep sensitive credentials off the shared machine entirely.

Grok Bot carries a second disclaimer worth reading alongside the first. Sensitive actions route through an approval mechanism. SpaceXAI documents the categories that trigger it, including sending messages, publishing content, purchases and transfers, deleting data, and touching production. Enforcement runs through an LLM classifier. Cursor‘s documentation for that same engine states plainly that the classifier is not a security boundary and can make mistakes. A buyer evaluating the product therefore finds the phrase twice, attached to the two mechanisms they would most reasonably assume protect them.

AI agents still lack identities of their own

Enterprises can adopt any of these four products today and get real work done, and the honest reading is that all four are engineering their boundaries in good faith against different threat models. What none of them provides is an identity for the agent. In every case, the bot borrows the operator’s credentials, whether from a shared cloud browser, a profile directory, or a container volume, and the entire security conversation boils down to how far those borrowed credentials travel.

“There is no primitive to standardize on, so each project has invented a boundary at whatever layer it already controlled, the account, the profile, the runtime, or the container.”

That is why the four answers differ so much. There is no primitive to standardize on, so each project has invented a boundary at whatever layer it already controlled: the account, the profile, the runtime, or the container. Expect to see that gap close on the identity side rather than the agent side, through scoped delegation and per-agent credentials issued by the identity provider, rather than being copied from the human. Until then, the useful move for platform teams is unglamorous and specific. Read the security page before the launch page, because for this class of product, they describe different things.

The post Grok Bot vs. Hermes: Where each draws the security boundary appeared first on The New Stack.

On July 16, the team at Hugging Face noticed something weird moving through their production systems: An intruder that was cloning datasets, harvesting credentials, and hopping between internal clusters faster than a human could type. Five days into the investigation, the authorities were notified, and several days later, OpenAI traced the intruder back to one of its models.

“Two frontier models escaped their test environments this summer; only one of them had a funny reason.”

That prompted Anthropic to look through its own test logs, and it found that similar events had occurred three times. One Claude model probed nine thousand hosts in a single run without setting off any alarms. Another slipped a poisoned package into the public Python registry, compromising fifteen machines before it was caught. The third incident dated back to April and went undetected for three months until a rival’s error prompted the audit. Nobody had noticed. Two of the three companies involved didn’t know anything was amiss until they got a call.

Containment by instruction

A sandbox, in the context of AI testing, means a model gets a task, a set of tools, and an instruction: no access to the outside world.

OpenAI’s model found an exploit in a tool it was allowed to use, and leveraged it to access the open Internet. Anthropic’s models didn’t even need that. The models were told a path didn’t exist, so when their tasks led them down a path that wasn’t supposed to be there, they treated it as part of the exercise.

“In both instances, the only constraint was an instruction. There was no mechanism outside the model itself to ensure those instructions wouldn’t be treated as optional.”

In both instances, the only constraint was an instruction. There was no mechanism outside the model itself to ensure those instructions wouldn’t be treated as optional.

So how did they escape?

OpenAI’s agent was being evaluated on a cybersecurity benchmark called CyberGym. During the test, it chained a code-execution flaw in a dataset loader to a second flaw in the platform dataset configuration file handler. It used the combined exploit to run commands on a live processing worker. From there it grabbed a write-scoped access token and cloned a private dataset as a staging area, which became a dead drop for its own tools. By the time anyone noticed, the intrusion had been running for three days, touched several internal clusters, and compromised accounts at four other companies, according to Hugging Face’s own review.

What it wanted, in the end, was comical: the answer key to the test it was taking.

Anthropic’s three incidents were nowhere near as convoluted. Its models were being run through security exercises against third-party evaluation partners, and the only thing keeping them from accessing the Internet was an instruction not to. One incident involved an SQL injection targeting a resource the model wasn’t supposed to be able to access. Another, during an exercise where the usual safeguards were deliberately switched off, ended with a Claude model publishing a malicious package to the Python Package Index, and fifteen machines picking it up before it was pulled down. No zero-day exploit. No dramatic escape. Just locked doors that turned out to be mirages.

The structural weakness

Strip away the specifics, and both incidents share the same root cause: there was nothing outside the model’s own judgment that could enforce the limitations placed on it. A credential that shouldn’t have been able to access the open Internet did so anyway. An instruction that should’ve stopped a plan cold got treated as just another input to reason around.

“If a fence doesn’t hold during testing, it won’t hold in production.”

It feels like we’ve spent the last year reliving the same problem in new ways. For example, in deleted databases, in poisoned supply chains, in every credential an agent inherited because provisioning it correctly took one extra step nobody got around to. The only delta is where the boundary was supposed to be: around the dev experiment, meant to contain dangerous capabilities, or standing between it and every production system it could reach. If a fence doesn’t hold during testing, it won’t hold in production.

How do we break out of the loop?

Unfortunately, the answer isn’t self-improvement. A more capable model wouldn’t have caught any of this. The only way to fix it is with Infrastructure that verifies where a model actually is, rather than trusting its reports.

Egress controls that are closed by default. A test environment that can reach the open Internet is probably not a great idea. Outbound traffic should be locked to an explicit allowlist. Never assume a model will simply decline to look for a way out. “No internet access” needs to be a network-enforced fact, not just an instruction.

Narrowly scoped credentials with short lifespans. Both incidents ran, in part, on tokens that extended beyond what the tasks required. A credential built for one evaluation run, scoped to nothing beyond it, and expiring on a timer, can’t become the write-scoped key that turns into a lateral-movement problem across multiple clusters. Yes, it will be a chore; so gird your patience by imagining having to explain to company leadership why taking steps to avert a catastrophe was too tedious to bother with.

Policy gates that sit outside the agent’s reasoning. An agent that can decide for itself whether an action is in scope is grading its own test. A plan should have clear rules written in code, evaluated by something other than the agent, before anything executes. Open Policy Agent has become a de facto default here for exactly this reason. It doesn’t care how confident the plan sounds; nothing runs without proper authorization from an external source.

An audit trail that is entirely independent of the agent. Both incidents were discovered by rereading logs after the fact, not by asking the models what happened. That’s how oversight should be done. As the proverb goes: Quis custodiet ipsos custodes? Who watches the watchers? A system of record that captures what actually occurred, regardless of the agent’s own account of itself, is the only version of events worth trusting.

This is a sign

The industry spent a decade learning that the CI/CD pipeline is an attack vector that requires real fortification, not just bolted-on convenience. Test environments for frontier models are following the same arc at a faster pace. The next time one of these agents escapes containment, it’ll probably be one built around finding unlocked doors, which will make it substantially more dangerous than a coding agent that deletes a few databases.

Test rigs must be treated as if they hold something real, because, as far as the credentials are concerned, they do. A sign on a door is never going to be enough to keep everyone out; there has to be a lock whose robustness correlates to the value of what it guards.

“A sign on a door is never going to be enough to keep everyone out; there has to be a lock whose robustness correlates to the value of what it guards.”

Whether by a state-sponsored crew probing a water management system in the middle of the night, or a company’s own model trying to shave a few points off a benchmark, boundaries will always be tested. Two labs found out this summer, and the story needs to be taken seriously. The vulnerabilities are real, the transparency from the labs is welcome, and the containment failures are a cause for concern.

Catching a model that tried the handle is the easy part; both labs proved that. The more challenging, and therefore critical, part is making sure the next containment environment actually has doors that are firmly locked.

The post Securing sandboxes: What happens when AI agents escape containment? appeared first on The New Stack.

The artificial intelligence landscape has reached a pivotal inflection point. Over the past several years, the paradigm has shifted from passive, conversational Large Language Models (LLMs) to autonomous AI agents, digital software entities capable of reasoning, invoking tools, executing multi-step workflows, and making real-time decisions across enterprise systems without constant human intervention.

As organizations accelerate the production deployment of autonomous agents, modern security frameworks must evolve to keep pace. Traditional Identity and Access Management (IAM) systems were primarily designed around two distinct operational models:

• Human users: Authenticated via Multi-Factor Authentication (MFA), Single Sign-On (SSO), and interactive sessions.
• Service accounts and workloads: Authenticated via static API keys, fixed service tokens, or IP whitelisting.

Autonomous AI agents blur the line between these two models. An agent acts with the non-deterministic reasoning and delegated agency of a human, but operates at the scale, parallel velocity, and automation speed of a machine service.

To safely harness the power of autonomous workflows, enterprise security architecture must move toward continuous, agent-aware Zero Trust governance. Below are six foundational identity capabilities that organizations should adopt to secure AI agents in production environments effectively.

“Autonomous AI agents blur the line between these two models. An agent acts with the non-deterministic reasoning and delegated agency of a human, but operates at the scale, parallel velocity, and automation speed of a machine service.”

“When it comes to agentic AI identity, most organizations are woefully unprepared for inherent security risks and operational challenges of managing those identities.” – Ken Buckler, Research Director, EMA – Agentic AI Identities – Is Your Organization Prepared?

1. Verifiable agent identities & “Know Your Agent” (KYA)

Autonomous entities require verifiable digital identity frameworks that establish clear, cryptographically bound accountability for every machine action.

• Cryptographic attestation: Every agent instance should possess a unique, cryptographically signed identity bound to its underlying model version, execution environment, and deployment origin.
• Delegation chains: When a human user delegates a task to an agent (or when a primary agent spawns sub-agents), the identity system must construct an immutable, traceable chain of delegation. This ensures the infrastructure can continuously verify who authorized the initial action and what specific scope was granted.

2. Ephemeral credentials & just-in-time (JIT) tokenization

Static API keys and persistent service tokens represent a significant surface area of exposure when integrated into dynamic agentic workflows. Replacing long-lived credentials with short-lived tokens dramatically reduces the potential window of risk.

• Just-in-time (JIT) minting: AI agents should operate with ephemeral credentials generated on demand, strictly limited to the API calls required for a single operational step, and configured to expire within seconds or minutes.
• Bound OAuth flows & PKCE: Enforcing Proof Key for Code Exchange (PKCE) and strict token-binding protocols ensures that credentials cannot be reused or replayed outside of their intended runtime context.

“Replacing long-lived credentials with short-lived tokens dramatically reduces the potential window of risk.”

3. Relationship-based access control (ReBAC) & intent binding

Coarse-grained permissions, such as those in traditional Role-Based Access Control (RBAC), are often too broad for non-deterministic tool usage. Access governance should be based on fine-grained relationship models and task intent.

• Intent-bound authorization: Authorization systems should evaluate not only whether an agent has general permission to access a resource, but whether that request directly aligns with the explicitly authorized sub-task.
• Fine-grained contextual policies: Implementing relationship-based access control (ReBAC) or Attribute-Based Access Control (ABAC) allows teams to define precise conditions (e.g., “Agent X may read Document Y only if human user Z is the document owner and the active workflow is ‘Data Summarization'”).

4. Machine-speed containment & automated anomaly detection

Because AI agents operate at speeds far exceeding those of manual monitoring, security containment mechanisms must be automated, agent-aware, and built into the control plane.

• Behavioral rate & scope limits: Security controls should establish baselines for expected agent behavior to detect anomalies, such as rapid parallel tool invocations, repetitive execution loops, or unusual queries to non-standard endpoints.
• Automated circuit breakers: If an agent’s execution pattern or request velocity exceeds defined behavioral bounds, identity proxies can automatically revoke ephemeral tokens and safely isolate the workload in real time.

5. In-the-loop runtime enforcement & human approvals

Security governance cannot rely solely on static pre-authorization; policies must be evaluated continuously at runtime before individual actions execute.

• Action-level policy interception: Enforce real-time policy checks at the agent harness layer—evaluating shell commands, database queries, file operations, and outbound API calls against governance rules before execution.
• Configurable approval workflows: Establish flexible escalation paths that permit low-risk read operations automatically while requiring explicit human-in-the-loop validation for high-impact actions, such as code deployments or financial transactions.

6. Web-scale identity architecture built for machine workloads

Autonomous workflows generate significant operational volume. Identity systems must be architected to handle machine-scale throughput without performance degradation or store bloat.

• Machine-speed throughput: Multi-step workflows and parallel worker agents demand identity control planes that can handle high-volume token validation and policy evaluation with minimal latency.
• Lifecycle governance for sub-agents: Dynamically spawned sub-agents require rapid provisioning and immediate teardown upon task completion, thereby preventing the accumulation of orphaned credentials and ensuring clean session termination.
• Inline cryptographic safeguards: Prioritizing inline policy enforcement over post-mortem log reviews allows organizations to intercept unauthorized state changes before they occur, maintaining operational integrity across multi-cloud environments.

Conclusion: securing the future of enterprise automation

As AI models evolve from passive assistance tools to active operational participants, identity becomes the primary boundary for enterprise governance. By bridging the machine identity gap with verifiable agent identities, short-lived JIT credentials, fine-grained relationship authorization, and automated runtime enforcement, security leaders can confidently deploy autonomous AI agents to drive productivity while maintaining complete operational control.

The post Six identity capabilities for securing autonomous AI agents appeared first on The New Stack.

Anthropic launched a new Browser Use tool that gives Claude a structured view of a web page in addition to what is visually rendered. Announced Thursday, the tool uses the page’s accessibility tree to help Claude find and interact with specific elements directly rather than having to work out where they are on the screen.

Browser Use is part of a broader Anthropic release that also brings Computer Use, the Skills API and Files API into general availability. Developers can access the browser tool through the Claude API using browser_toolset_20260801.

Browser Use is part of a broader Anthropic release that also brings Computer Use, the Skills API and Files API into general availability.

The change gives Claude a more direct way to interact with a web page. Instead of working out a button’s position from a viewport image and targeting coordinates such as x: 640, y: 320, Claude can receive a reference such as ref_3 tied to that element and use it when it wants to act.

Page references replace coordinates

Computer Use can operate across an entire desktop by looking at screenshots and sending mouse coordinates and keyboard commands. Browser Use works within the browser itself, where it can use page structure that would be difficult to recover reliably from pixels alone.

When Claude calls read_page, the developer’s executor returns a text representation of the accessibility tree, in which elements such as links, buttons, and text boxes can be tagged with references. If Claude later wants to click a button represented by ref_3, it can send that reference along with the requested operation rather than trying to calculate where the button is on the screen.

That said, if the tab navigates to a new page or the page changes enough, a reference that pointed to a button a moment ago may no longer work. The API will not catch that on its own, so the executor has to recognize when the reference no longer matches the underlying element, reject the action and have Claude read the page again before continuing.

Batching cuts model calls

Playwright, for example, can represent a page as an ARIA snapshot and locate elements by role rather than coordinates. At the same time, Microsoft’s Playwright MCP server already exposes structured accessibility snapshots with references a model can use to identify elements. The concepts line up closely with Browser Use, but the protocols do not: Playwright MCP speaks MCP, while Anthropic’s tool uses its own client-toolset protocol, so developers would still need an adapter that translates Claude’s requests into Playwright actions and returns the results in the format Claude expects.

Puppeteer offers many of the same building blocks, exposing the browser’s accessibility tree via Accessibility.snapshot() and providing APIs for controlling Chrome and Firefox. A developer could use those APIs for navigation or page reads, then maintain Anthropic’s reference mappings on top.

A developer could use those APIs for navigation or page reads, then maintain Anthropic’s reference mappings on top.

Slightly confusing, an unrelated open-source project also called Browser Use runs AI browser agents against Chromium through the Chrome DevTools Protocol. Despite the shared name, it has no connection to Anthropic’s tool and comes with its own agent loop and browser abstractions, so connecting the two would still require integration work.

Several browser actions can happen in one turn

Anthropic is also reducing the back-and-forth between Claude and the browser by allowing multiple actions to be requested in a single model turn. Now actions can arrive together as several tool_use blocks. The application executes them in order and sends the results back together, avoiding another model call between every click and keystroke. Anthropic says that can lower latency and costs, particularly as workflows scale from a handful of interactions to dozens or hundreds.

That matters more as browser tasks get longer. Cheaper models alone will not solve the token cost problem in agentic workflows, so cutting unnecessary model calls is another way to reduce costs.

If Claude has to return to the model after every click or keystroke, a long browser task can quickly rack up model calls. Batching cuts out some of that back-and-forth by letting Claude request several actions at once, but the browser still has to carry them out in order because each one depends on what happened before it. If Claude asks to click a button, fill in a field, and submit a form, for example, the executor cannot simply move on to the next step if that first click fails, because everything that follows is now based on a page state Claude never reached.

Batching cuts out some of that back-and-forth by letting Claude request several actions at once, but the browser still has to carry them out in order because each one depends on what happened before it.

Developers host the browser

Browser Use is currently limited to the Claude API and is not available inside Claude Managed Agents. Adding it to a Messages API request exposes 27 browser operations by default. Claude can decide which of those operations it wants to use, but Anthropic does not execute them. The application has to translate each request into an action inside its own browser environment, preserve the session between turns and return enough information for Claude to understand what happened.

Loading all of those operations has a token cost. Anthropic’s pricing documentation says the default Browser Use toolset adds roughly 6,600 input tokens to a request, before counting screenshots, accessibility trees and other results sent back to Claude. Developers can turn off operations they do not need to reduce that overhead.

It also creates a different hosting split from some of the other tools Anthropic announced Thursday. Skills uploaded through the Skills API can run inside Anthropic’s code execution sandbox, while the Files API stores documents that can be reused by ID. Browser sessions, along with their downloads and uploaded files, stay in the developer’s environment.

Approval gates need rethinking

Claude can still encounter a prompt injection in web content or be redirected to an unexpected location, which is why Anthropic recommends running the browser in an isolated container or virtual machine with minimal access. JavaScript and file uploads should remain disabled unless needed, since code generated by Claude runs with the page’s privileges and can reach data or make requests available to that page.

Batching makes approval a little trickier because several actions can arrive at once, and a routine click at the beginning of a sequence could eventually lead to something that requires the user’s permission. That means the executor has to check actions as they happen and stop for approval when needed.

The post Anthropic’s new browser tool doesn’t actually run a browser appeared first on The New Stack.

On Tuesday, Warp introduced Warp Factories, open infrastructure for building cloud software factories, agentic systems that automate work across the software development lifecycle, which have been popping up in different forms from companies like Augment Code and Chainguard. 

Warp, an agent development platform, calls Warp Factories “the building blocks” for developers to create their own scalable factories. It’s pitching the infrastructure as the solution for two problems founder and CEO Zach Lloyd says are frequent engineering complaints: 1) measuring and improving coding agent ROI; 2) governance and control.

The aim is to tackle both problems by making sure “the annoying bits [are] taken care of” so developers can focus purely on optimizing factories for specific products. 

As Lloyd writes in a blog post, he “predicts software factories will be as ubiquitous as CI/CD in the next few years.” Experts tell The New Stack they see software factories gaining traction, but they’re more cautious about the timeline.

“I think the software factory is inevitable,” Lee Faus, founder and CEO, Atomic Software and former global field CTO, GitLab, tells The New Stack. “But before software factories become as foundational as CI/CD, the industry needs to solve a deeper infrastructure problem.”

Specifically, he calls out the importance of tracing agent work: “We’re spending a lot of time talking about how to build the software factory,” Faus continues. “I think we’re going to spend much more time asking what becomes the system of record for the factory.”

Build the factory without building all the infrastructure 

Lloyd acknowledges that many organizations already have engineering teams at work building cloud software factories — but he argues that’s too big to be an inside job. 

Warp Factories, thus, emerges as the infrastructure on which developers can build their own factories, providing the core components to speed development without making organizations sacrifice flexibility, programmability, customization, or ownership.

When asked about Lloyd’s take on building infrastructure, Erik Gfesser, long-time engineer, tells The New Stack he agrees it doesn’t make sense for most organizations to tackle it in house.

As Lloyd writes, Warp’s new infrastructure is “built to increase coding agent ROI over time” with evals and benchmarks to measure effectiveness and built-in self-improvement and memory. Developers get queryable metrics on agent throughput, cost, quality, and ROI, visible via the Factory control room, API, and Factory MCP. Scorers evaluate how work items move through the factory with an eye on things like token spend, code quality, and whether or not the work introduced defects. 

From there, those scores power self-improvement loops and benchmarks. “Observer” agents score select agent runs and then search for ways to make improvements by adjusting variables like the harness, model, or context before making PRs to improve underlying factory functionality. Benchmarks, meanwhile, let developers score tasks across different models and harness configurations to compare performance.

Governance gets easier, but there’s more to solve

Per Warp, the infrastructure includes features to address governance and control, alongside factory definitions as version-controlled code, definitions for distinct agents, plus skills, MCPs, and permissions. 

Looking more broadly, Faus tells The New Stack software factory governance will require more than just controlling how agents operate, though:

“A software factory without a record of change risks becoming a very efficient way to manufacture code that nobody can fully explain.”

“Shared infrastructure can make permissions, model access, tool use, MCP connections, policies, cost, and execution environments easier to manage centrally. That’s valuable,” he says. “But governance isn’t just being able to control what an agent is allowed to do. It is being able to prove what it actually did.”

As software factories help speed up code generation, he says the harder problem becomes understanding the scores of interconnected decisions both humans and agents make across the development cycle. 

For example, if one agent triages an issue, another researches it, a third implements it, and still others review and verify it, how can an engineer reconstruct why that change was made six months later? “That record has to remain connected to the change itself,” says Faus. “[Otherwise,] a software factory without a record of change risks becoming a very efficient way to manufacture code that nobody can fully explain.”

Software factories are probably the future, but it will be a slow roll-out

Though Warp’s founder is gung-ho about the rapid rise of software factories, other experts are less certain. Like Faus, Gfesser expects software factory adoption to take time: 

“My expectation is that software factory adoption will likely be fragmented across multiple vendors similarly to the early stages of CI/CD.”

“As an early adopter of CI/CD myself, I know that CI/CD didn’t catch on the way it did until quality open source products were made available for widespread usage.”

He points out that while the Warp client is open source, the server, the Warp Drive backend, and OZ (Warp’s agent orchestration layer) are proprietary. Also worth noting: OpenAI is named as the founding sponsor of Warp’s open source repository. 

“As such, my expectation is that software factory adoption will likely be fragmented across multiple vendors similarly to the early stages of CI/CD,” he says. 

The post Warp wants to make it easier to build your software factory appeared first on The New Stack.

Every engineering team deploying AI agents eventually discovers an uncomfortable truth: the model isn’t the biggest expense. The hidden cost is everything around it: repeated retrievals, duplicate prompts, unnecessary tool calls, oversized context windows, multiple agents reasoning over the same information. Individually, these architectural decisions seem harmless. At production scale, they become a severe tax on latency, infrastructure, and cloud spend.

A proof-of-concept agent that answers 50 questions a day can tolerate inefficiencies. An enterprise platform coordinating thousands of requests per minute cannot.

This article explores practical techniques for engineering token-efficient AI systems without sacrificing output quality. Rather than focusing solely on prompt compression, we will optimize the entire workflow from routing and retrieval to caching and model selection.

Why token optimization is a systems problem

Most discussions around token optimization begin and end with prompt engineering. In practice, architecture drives token consumption.

Consider a typical multi-agent workflow:

User 
  ↓
Intent Agent
  ↓
Retriever
  ↓
Research Agent
  ↓
Planning Agent
  ↓
Writer Agent
  ↓
Reviewer Agent
  ↓
Final Response

At each stage, the system might retrieve the same documents, repeat identical instructions, call the same model, and resend the entire conversation history. By the time a response reaches the user, the architecture has processed tens of thousands of unnecessary tokens.

“Improving efficiency requires redesigning the workflow, not just shortening the prompts.”

Improving efficiency requires redesigning the workflow, not just shortening the prompts.

Architecture overview

A production-ready, token-efficient architecture introduces optimization before every expensive model invocation.

User Request
       │ 
       ▼
Intent Router 
       │ 
       ▼
Semantic Cache ───────► Cached Response
       │ 
       ▼
Context Budget Manager
       │ 
       ▼
Adaptive Retriever
       │ 
       ▼
Model Router
       │ 
       ▼
LLM
       │ 
       ▼
Validated Response

“The large language model is no longer the first component. It is the final, most expensive operation.”

Notice the critical shift: the large language model is no longer the first component. It is the final, most expensive operation.

Step 1: Install modern dependencies

Use the latest package structure to avoid deprecated imports and align with the current LangChain ecosystem.

Bash
pip install \
   langchain \
   langchain-core \
   langchain-openai \
   langchain-community \
   fastapi \
   faiss-cpu \
   tiktoken \
   rank-bm25 \
   pydantic \
   python-dotenv

Step 2: Configure the model

Production systems must configure retries, timeouts, and credentials through the environment.

Python
import os
from langchain_openai import ChatOpenAI 

api_key = os.getenv("OPENAI_API_KEY") 
if not api_key: 
    raise ValueError("OPENAI_API_KEY must be configured.")

llm = ChatOpenAI( 
    model="gpt-4o-mini", 
    temperature=0, 
    api_key=api_key, 
    timeout=30.0, 
    max_retries=2, 
)

Setting a low temperature improves consistency, while explicit timeouts and retry limits help the system recover gracefully from transient API failures.

Step 3: Route before you generate

Not every request requires a large language model. Deterministic logic can often answer simple questions. Routing inexpensive requests away from the LLM yields the most significant cost reduction in production systems.

Python
def classify_request(question: str) -> str:
    q = question.lower()

    if "status" in q:
        return "metrics"

    if "runbook" in q:
        return "retrieval"
   
    return "generation"

Step 4: Add a semantic cache

One of the simplest and most effective optimizations is an exact-match cache, which returns a previously generated response when the same question is asked against the same retrieved documents, avoiding unnecessary model calls.

Python
import hashlib

# Using an exact-match (lexical) cache
exact_match_cache = {}

def cache_key(question: str, sources: list[str]) -> str:
    """
    Generate a deterministic cache key from the user question
    and the retrieved document identifiers.
    """
    fingerprint = question + "|" + "|".join(sorted(sources))
    return hashlib.sha256(fingerprint.encode()).hexdigest()

# Example usage in the pipeline:
# key = cache_key(question, source_ids)
# if key in semantic_cache:
#     return semantic_cache[key]

Step 5: Budget your context

Most retrieval pipelines return far more text than the model actually needs. Instead of stuffing the context window with every retrieved document, establish a strict context budget.

Python
import tiktoken

encoder = tiktoken.encoding_for_model("gpt-4o-mini")
MAX_CONTEXT_TOKENS = 2500

def build_context(chunks):
    context = []
    used = 0

    for chunk in chunks:
        tokens = len(encoder.encode(chunk.page_content, disallowed_special=()))

        if used + tokens > MAX_CONTEXT_TOKENS:
            break 

        context.append(chunk.page_content)
        used += tokens

    return "\n\n".join(context)

Step 6: Retrieve once

Repeated retrieval is a surprisingly common flaw in multi-agent systems. The rule is simple: retrieve once, reuse everywhere.

Python
from langchain_core.documents import Document
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings

documents = [
    Document(
        page_content="Database latency often follows connection pool exhaustion.",
        metadata={"source": "db_runbook"},
    ),
    Document(
        page_content="Node pressure can increase API response times.",
        metadata={"source": "cluster_runbook"},
    ),
]

embeddings = OpenAIEmbeddings(api_key=api_key)
index = FAISS.from_documents(documents, embeddings)

retrieved_docs = index.similarity_search(question, k=4)
shared_context = build_context(retrieved_docs)

Now, every downstream agent consumes the same optimized context instead of launching its own redundant retrieval pipeline.

Step 7: Route models intelligently

Large models should solve complex problems. Everything else belongs to a smaller, faster model.

Python
from langchain_openai import ChatOpenAI

small_model = ChatOpenAI(model="gpt-4o-mini", temperature=0, api_key=api_key)
large_model = ChatOpenAI(model="gpt-4.1", temperature=0, api_key=api_key)

def choose_model(question: str):
    """Route requests to the most appropriate model based on complexity."""
    if len(question) < 200:
        return small_model
    return large_model

This strategy drastically reduces operational costs without noticeably affecting response quality.

Step 8: Estimate tokens before sending

Without token telemetry, optimization is just guesswork. Monitoring usage makes efficiency measurable and helps engineers detect cost regressions.

Python
import tiktoken

encoder = tiktoken.encoding_for_model("gpt-4o-mini")

def estimate_tokens(messages):
    """
    Estimate input tokens for an OpenAI-style chat payload.
    Note: This is an estimate, not an exact billing calculation.
    """
    tokens_per_message = 3
    tokens_per_name = 1
    total = 0

    for message in messages:
        total += tokens_per_message
        for key, value in message.items():
            if isinstance(value, str):
                total += len(encoder.encode(value))
            if key == "name":
                total += tokens_per_name

    # Every reply is primed with additional assistant tokens.
    total += 3
    return total

Step 9: Validate responses

Production systems must return structured outputs to ensure downstream systems receive predictable, well-formed data.

Python
from pydantic import BaseModel

class AgentResponse(BaseModel):
    answer: str
    sources: list[str]

def validate_response(answer: str, sources: list[str]):
    """Validate and serialize the agent response using a structured schema."""
    response = AgentResponse(
        answer=answer,
        sources=sources,
    )
    return response.model_dump()

Step 10: Build the optimized pipeline

Finally, assemble the architectural components into a single workflow. Notice how failures degrade gracefully instead of crashing the service.

Python
import logging

from langchain_core.prompts import ChatPromptTemplate

logger = logging.getLogger(__name__)

def run_pipeline(question: str):
    """Execute the token-efficient AI workflow with graceful degradation."""
    try:
        route = classify_request(question)

        # Route deterministic requests away from the LLM.
        if route == "metrics":
            return {
                "answer": "Retrieve metrics directly from the monitoring system.",
                "sources": [],
            }

        # Retrieve context once.
        docs = index.similarity_search(question, k=4)

        context = build_context(docs)

        source_ids = [
            doc.metadata.get("source")
            for doc in docs
            if doc.metadata.get("source")
        ]

        # Check exact-match cache.
        key = cache_key(question, source_ids)

        if key in exact_match_cache:
            return exact_match_cache[key]

        # Select the most appropriate model.
        model = choose_model(question)

        # Keep trusted instructions separate from untrusted user input.
        prompt_template = ChatPromptTemplate.from_messages(
            [
                (
                    "system",
                    (
                        "Answer the user's question using ONLY the provided context. "
                        "If the answer cannot be determined from the context, say so."
                        "\n\nContext:\n{context}"
                    ),
                ),
                ("user", "{question}"),
            ]
        )

        chain = prompt_template | model

        result = chain.invoke(
            {
                "context": context,
                "question": question,
            }
        )

        payload = validate_response(
            answer=result.content,
            sources=source_ids,
        )

        # Cache validated response.
        exact_match_cache[key] = payload

        return payload

    except Exception:
        logger.exception("Token-efficient pipeline failed.")

        # Gracefully degrade instead of crashing.
        return {
            "answer": (
                "The AI pipeline encountered an error. "
                "Please continue using the standard operational workflow."
            ),
            "sources": [],
        }

What actually reduced token usage?

When teams instrument architectures like this, the largest savings rarely come from editing prompts. They come from eliminating unnecessary work.

The biggest improvements typically stem from:

• Retrieving documents once instead of multiple times.
• Caching semantically identical requests.
• Routing simple requests away from the LLM.
• Limiting context with explicit token budgets.
• Selecting the smallest suitable model.

These architectural shifts reduce cost and latency while making system behavior significantly easier to reason about.

Lessons learned

Several core principles consistently emerge when optimizing AI systems for production:

• Treat tokens like infrastructure: Tokens are a finite resource, just like CPU cycles or memory. Monitor them, budget them, and optimize them.
• Retrieval is usually the largest source of waste: Repeated retrieval often contributes more unnecessary tokens than verbose prompts. Share context whenever possible.
• Bigger models are not always better: Smaller, faster models effectively handle many operational tasks. Reserve larger models for genuinely complex reasoning.
• Caching is an engineering feature: A semantic cache is more than a performance optimization—it is a core architectural component that reduces cost, latency, and provider dependence.
• Measure before you optimize: Instrumentation must accompany every production deployment.

As AI systems mature, success will increasingly depend on engineering efficiency rather than raw model size. The hidden tax of AI agents is rarely a single expensive prompt; it is the accumulation of redundant retrievals, oversized contexts, unnecessary model calls, and repeated reasoning across distributed workflows.

“The most effective production AI systems are not the ones that generate the most tokens. They are the ones that generate only the tokens they truly need.”

By treating token consumption as a systems engineering problem, organizations can build AI platforms that are faster, less expensive, and highly scalable. Routing requests intelligently, budgeting context, sharing retrieval results, validating structured outputs, and introducing semantic caching are practical techniques that guarantee efficiency without compromising quality.

The most effective production AI systems are not the ones that generate the most tokens. They are the ones that generate only the tokens they truly need.

The post Stop the token bleed: building token-efficient multi-agent systems appeared first on The New Stack.

AI infrastructure platform company TrueFoundry has launched its open-source agent harness, TrueForge. The technology, announced Wednesday, is directly billed as an alternative to Claude Managed Agents, Anthropic’s hosted infrastructure service that runs, sandboxes, and orchestrates autonomous Claude agents.

TrueForge promises to enable software engineers to build, deploy, debug, and govern production AI agents on any model (and the company means any model) or MCP server, while reducing total agent operating costs by an estimated 50%.

While open models such as GLM-5.2 from Chinese frontier model maverick Z.ai are challenging proprietary frontier models at lower costs, most managed agent platforms still lock enterprises into a single vendor’s models, infrastructure, and pricing. 

Challenging the pervading narrative of managed agent platform lock-in

Ex-machine learning tech lead at Meta and now co-founder and CEO of TrueFoundry, Nikunj Bajaj, tells The New Stack that this pervading managed agent platform lock-in is precisely the logic behind his firm’s neutral approach to model vendor choice.

“A provider selling you a million tokens for $50 has zero incentive to tell you the same task could be done using a model that charges 50 cents for a million tokens,” Bajaj says. “Traditionally, one vendor provides the models, builds your agents and decides your token usage, in what order, and with what tools and under governance that the managed agent provider stipulates – and they’re selling the exact same setup to your competitor.”

Fundamentally, he insists, this means “the incentives are misaligned” here and so the “players in this game don’t get a voice to talk to the referee” in managed agent deployment scenarios where there’s always a tradeoff. 

“Why should building powerful agents mean giving up control of your AI stack? We give developers the managed-agent experience without forcing them into one vendor forever,” adds Bajaj.

“A provider selling you a million tokens for $50 has zero incentive to tell you the same task could be done using a model that charges 50 cents for a million tokens. The incentives are misaligned, so the players in this game don’t get a voice to talk to the referee.”

The harness underneath becomes the strategic control point

Although Claude Managed Agents only arrived as a beta release in April of this year, Bajaj and team think they can track an evolutionary curve being etched out here. This arc sees the first wave of AI agents existing on developers’ laptops, inside coding tools and prototypes. But the next wave is moving into customer-facing products typified by hosted infrastructure services with the ability to use shared workflows.

Crucially, that’s a shift that turns the harness underneath those products into a strategic control point, working as an execution layer in an operational loop between the user, the model and the systems it interacts with.

“This is indeed the reality: the harness is the critical layer between the user, LLM, and everything else,” clarifies Bajaj. “Say a developer is building an agent. They bring their own models, but the harness still decides when to call an MCP server, when to use an agent someone has already built, what context to keep, and which model handles which part of the plan.”

This means there are security implications, too. Bajaj specifies that “some actions” still need to run in a completely isolated sandbox, and some data should never be sent to a closed-source model. 

“All of that logic sits in the harness. If software engineers don’t use one, then the developer team has to build all of that logic from scratch,” he adds.

Enterprises will want to own key agent layers

In an open vendor-neutral approach to managed agent platform provision, organizations must manage persistent sessions, tool credentials, execution sandboxes, context, human approvals, debugging, access policies, and spending across every agent they operate. TrueFoundry is betting enterprises will want to own that layer rather than inherit it from a single model provider, but with enterprise governance built in at lower cost.

TrueForge routes every model call and MCP interaction through TrueFoundry’s AI Gateway, so budget enforcement, rate limits, and guardrails can be applied to deliver a governed and secure managed agent experience for enterprises.

Headless chickens, when foo and bar are behind the wheel

When organizations don’t have the same hold on the steering wheel, Bajaj says that he has personally witnessed operations where “foo” and “bar” (standard placeholder names used in computer programming for as yet-unnamed known metasyntactic variable values, rather like John Doe) end up becoming the doers of everything. 

“Every action in the system came from a generic shared account, not a person you could actually identify. So when something changed or broke, you had no idea who to talk to. Once, when we were halfway through a migration from shared access to individual access, some keys were rotated. Half the company was still on the old account, and the system broke for half the company,” he explains.

Teams can run TrueForge on their own infrastructure, bring their own models, MCP servers, and API keys, and route each task to whichever model fits the cost, latency, or quality needs of that job. But does that mean workloads might become too fragmented that way?

“On the contrary, workloads become more uniform,” enthuses Bajaj. “Most teams already bring their own models by default. What changes is that organizations get to define what it takes for a model, agent, or MCP to belong in their registry. I call it the agent development life cycle, or ADLC. Once you own that, you can enforce the same operating principles across everything.”

In practice, the TrueFoundry team confirms it has seen most AI-centric software engineering operations converge on “roughly a dozen models” for typical tasks, plus a few specialized models for niche work. 

Is Anthropic doing something wrong?

TrueForge ships with support for OpenAI, Anthropic, and 20+ additional models, along with 40+ built-in tools, sandboxed execution, human-approval workflows, large-context handling, generative UI, and web search powered by Tavily. But despite offering a Claude Managed Agents alternative, Bajaj goes to pains to point out he doesn’t hold Anthropic up as some kind of pariah. 

“This isn’t about Anthropic doing something wrong,” confirms Bajaj. “It’s that it doesn’t own every model in the world. Claude Managed Agents can only choose from the finite set of models Anthropic offers. There are open models that are terrific at certain tasks at a fraction of the cost, or simply more capable for that particular job. An open harness has a much wider set of choices.”

When you own the harness, you can get rid of the parts that don’t apply to you

He underlines his point by pointing out that Anthropic also has to build one harness for a very broad set of customers; a truth that means its system prompt has to account for all kinds of instructions, guardrails, and corner cases. 

“Many of those elements may have nothing to do with a developer’s own use case, but they still go into every call and add cost and latency. When you own the harness, you can get rid of the parts that don’t apply to you and make it extremely specialized,” he adds.

“To be clear, we support Anthropic as a first-class provider because its models are great. There will be many cases where our users want to use them. The point is not to limit that choice to Anthropic alone.”

To validate its statements here, TrueFoundry has tested the above claim on a total of 14 level-one and level-two tasks from DevRev’s public Enterprise-Bench. The company says TrueForge “came in 50% cheaper at similar accuracy”, so the savings came from using fewer tokens and having access to models outside Anthropic’s set that were better suited to specific tasks.

“To be clear, we support Anthropic as a first-class provider because its models are great. There will be many cases where our users want to use them. The point is not to limit that choice to Anthropic alone,” Bajaj concludes.

TrueFoundry is also launching a hosted, pay-per-usage version of TrueForge for teams that want the same experience without managing the infrastructure themselves.

The post An open source rival to Claude Managed Agents just launched appeared first on The New Stack.

Half of enterprise AI deployments are missing their own latency targets at peak load. This is the headline finding of Akamai’s State of AI Inference 2026 report, which surveyed 200 AI practitioners and found that 82% of organizations say their most critical use cases require end-to-end response times of 500 milliseconds or less. A total of 64% of organizations now require end-to-end response times of less than 250 milliseconds for their most important use cases, yet 50% of deployments are failing to meet these latency demands at peak load.

My colleague Ari Weil, who leads product marketing for our cloud computing business and ran point on that research, summarizes the findings well: “The enterprise AI honeymoon phase is over… they are hitting the latency wall.” 

Agentic workflows aren’t a “single round trip”

The latency issue stems from the way agents work. It’s an iterative process, somewhat like a king sending out knights, emissaries, and messengers to conduct the business of the kingdom. There are many comings and goings, not just one person sent on a single round trip. 

For instance, when an agent built on a framework like LangChain, CrewAI, or Pydantic AI received a user request, it can fan out into dozens of sequential operations such as a reasoning call, a tool invocation, an API lookup, or a context retrieval. Then an agent may execute another reasoning call to decide what to do with what just came back. Every one of these operations or “hops” that must cross a wide-area network to reach a centralized data center adds latency, and a chain of 50 hops can multiply that transport time into seconds on its own, regardless of how fast the model generates tokens.

In fact, in a paper posted to arXiv in November 2025, researchers found that CPU-side processing can account for up to 90.6% of total latency in agentic workloads. In other words, your GPU might finish a reasoning step in a few hundred milliseconds, but then it might have to wait on additional tool call runs to CPUs in distant data centers. This is what causes spikes in GPU idle time. 

“More GPU capacity does nothing for this. You can’t brute-force your way out of a wait state.”

More GPU capacity does nothing for this. You can’t brute-force your way out of a wait state. This is the part of the conversation the industry keeps skipping, mostly because “buy more GPUs” is a much quicker fix to suggest than “figure out where your CPU-bound work is actually executing and why it’s so far from the data it needs.”

We need new benchmarks to fix the latency issue

One reason the looming latency wall sneaks up on teams is that they are not looking at the right benchmarks for agentic workloads. Most LLM-serving benchmarks measure tokens per second and GPU utilization on a single box. That’s great if the workload is indeed on a single box (i.e., one model answering one prompt), but that’s not the case with agentic workloads. Those benchmarks don’t address an agentic response that, say, makes a 50-hop chain cross a WAN 4 times to reach 4 separate services. 

“Staging may pass the benchmarks because it tests the model, but production tests the whole chain, including every hop your serving engine was never designed to see.”

That’s where the gap lies: Staging may pass the benchmarks because it tests the model, but production tests the whole chain, including every hop your serving engine was never designed to see.

The 500ms wall is not a soft target

This is showing up at scale because agents are moving into production faster than most teams’ architecture is evolving to support them. LangChain’s State of Agent Engineering 2026 survey of more than 1,300 professionals found that 57.3% of organizations now have agents running in production, up from 51% a year earlier. Among those builders, latency has become the second-most-cited barrier to production, behind only output quality. 

This is a serious issue for application teams. The 500ms threshold in Akamai’s survey isn’t a performance goal teams can afford to miss. For a live customer interaction or a real-time compliance check, that 500ms determines whether the application works or it doesn’t. 

We’ve solved this problem before

There’s a reason this feels familiar to anyone who was building for the web in 1999. Akamai exists because of a nearly identical problem. MIT researchers Tom Leighton and Danny Lewin founded the company to answer a challenge posed by Tim Berners-Lee: fix what the press had started calling the “World Wide Wait,” the crushing latency of pulling every request back to a small number of centralized servers. When the trailer for The Phantom Menace crashed sites across the internet in 1999, the culprit was distance: millions of browsers all reaching for the same far-away origin server at the same moment. The fix moved content to thousands of points closer to the people requesting it, instead of trying to build a faster origin.

Agentic AI is running into the same wall, just in a different vehicle. AI works just fine on centralized inference if you’re talking about running batch jobs overnight. But today’s applications built on agentic AI are real-time loops sitting inside live transactions, and the fix for agentic lag is distribution. Instead of expanding racks of CPUs and GPUs at the center, we need to move agentic execution to where the model’s tools, context data, and users actually live.

Agentic AI needs a tiered architecture, not a bigger data center

In practice, agentic AI requires a tiered architecture, one that includes a centralized core, regional GPU clusters, and CPUs at the Edge. 

• Centralized core—perfect for heavy reasoning over large context windows, where the round trip to a large model matters less than the model’s raw capability.
• Regional GPU clusters, increasingly built on hardware like NVIDIA’s Blackwell platform—ideal for localized inference, so the heaviest compute sits closer to where demand actually concentrates.
• Edge CPUs—the essential component for speed. This is the nexus for tool execution, orchestration, and context retrieval, since these are the steps that happen most often in a chain and benefit most from sitting next to the data and APIs they call.

We’ve built Akamai Inference Cloud around this tiered framework. It’s the same distribution logic behind our AI Grid Orchestrator. We route CPU-bound orchestration and tool calling to the edge, and keep GPU-bound reasoning where it makes sense, regionally or centrally. 

What to demand before you commit

The good news is you don’t need to distribute every workload to the edge on day one. But before you commit to a production architecture, you should know which of your agent’s dozens of hops are latency-sensitive and which aren’t. Then build a defined performance budget for each one.

“The teams that treat it as a GPU-shopping decision will be back here in six months, staring at the same four-second response time, wondering why more compute didn’t help.”

My advice is this: Before signing off on a large-scale inference deployment, ask your infrastructure for four things: 

1. Portability across regions and providers
2. Elasticity to absorb peak load without falling over
3. Data locality so tool calls aren’t crossing oceans to reach the context they need
4. A performance budget you’ve actually tested against production traffic, not staging traffic.

The teams that address this infrastructure decision now will be the ones whose agents still work when the benchmark environment transitions to real users. The teams that treat it as a GPU-shopping decision will be back here in six months, staring at the same four-second response time, wondering why more compute didn’t help.

The post Agentic AI has a latency problem that more compute won’t solve appeared first on The New Stack.

Multi-tenancy has moved in one direction for 60 years: the tenant keeps getting smaller. Mainframe time-sharing carved a single machine into slices so an organization’s departments could share it, and the tenant was the org. Virtualization gave each team its own fleet of virtual machines, and the tenant became the team. Containers and Kubernetes namespaces shrank it again, until a platform team could hand every developer an isolated environment on a shared cluster.

That last step, an environment per developer, became the target state of platform engineering in the 2020s. A namespace per developer, capacity planned by seat, golden paths sized to headcount. Underneath all of it sits one assumption: a person produces one stream of work at a time, so isolating people isolates work.

Coding agents broke that assumption. A developer running five agent sessions has five changes in flight at once, each needing its own working version of the system. Anthropic’s engineers, building a C compiler with a fleet of parallel agents, ran nearly 2,000 Claude Code sessions across two weeks. Cursor’s documentation tells developers to run as many agents as you want in parallel. None of those concurrent workstreams is a person.

“The tenant has shrunk one more time. It is no longer the developer. It is the change.”

The tenant has shrunk one more time. It is no longer the developer (or even the agent). It is the change.

Tenancy demand scales with changes in flight, not headcount

Capacity planning by seat worked because changes arrived at human pace, roughly one per developer at a time. That denominator is gone. A Microsoft study of command-line coding agent adoption found that developers merged roughly 24% more pull requests over four months, and merged pull requests understate the pressure. Every change that reaches merge is preceded by iterations and abandoned attempts, and each of those also needed somewhere to run.

Run the seat math against the change math. A 50-developer organization where each engineer supervises a few agent sessions has hundreds of changes in some stage of validation on a busy day. Each one wants data it can migrate and write to without asking permission, its own view of shared message topics, and a running version of the services it touched. That is the demand of a 300-person (or more) engineering org on a 50-tenant platform. 

Every layer built on the person-tenant assumption misprices this. A per-developer namespace hands one tenant slot to what is now five concurrent workstreams. Shared staging serializes all of them into a single queue. Seat-based capacity plans budget for the number of employees while the bill tracks the number of changes in flight.

The new tenant is the change, not the agent

The tempting candidate for the new tenant is the agent, and it is the wrong one. Agents are interchangeable workers. Two agents can collaborate on one change, one agent can rotate through five changes, and a crashed agent gets replaced mid-task without anything downstream noticing. Give each agent its own environment, and you have repeated the old mistake at a new scale: isolating workers when the thing that must not leak is work.

“Give each agent its own environment, and you have repeated the old mistake at a new scale: isolating workers when the thing that must not leak is work.”

The durable unit is the change. It comes into existence when work on it starts. It accumulates state that no other tenant should see: a schema migration, test writes, new versions of one or two services, the messages it produced during validation. It needs to observe a version of the system that includes its own edits and nobody else’s. And it is torn down when it merges or is abandoned, taking all of that state with it.

Naming the change as the tenant turns a vague scaling problem into a design target, because change-level tenancy has three requirements that person-level tenancy never had to meet:

• Creating a tenant must be near free.
• Isolation must cover only what changed.
• The tenant’s lifecycle must be bound to the change itself, not to a ticket or a timer.

Platform teams already run this playbook in production

The discipline these requirements call for is not new. Anyone operating a multi-tenant production service already knows the rules: tenants share the substrate, each tenant privately owns only what makes it distinct, creating a tenant is self-service and cheap, and a tenant’s resources are reclaimed the moment it leaves. Nobody stands up a private copy of the product per customer, and nobody files a ticket to onboard one.

Those same organizations run pre-production on the opposite rules. Environments are provisioned by ticket or by seat, capacity is planned per person, and isolation is achieved by duplicating the stack when it is achieved at all. The multi-tenancy playbook that runs the product has never been applied to the platform that builds the product.

Change-level tenancy is that playbook, applied. Treat every change as a tenant of the development platform, and the three requirements stop being novel. They are the standard properties of any competently run multi-tenant system.

The tenant owns what changed and shares everything else

A SaaS tenant owns its data and configuration, never a copy of the application. A change tenant is sized the same way. It owns the one or two services it modified and an isolated database branch it can migrate and write against, and nothing else. Everything the change did not touch resolves against one shared stable environment, continuously deployed from main, so every tenant validates against real, current dependencies without owning a copy of them.

A footprint that small makes tenant creation nearly free, and creation cost is what decides whether the model scales to agent demand. Isolated data no longer requires copying a database: Neon and Xata create copy-on-write branches in seconds regardless of dataset size, consuming storage only for the data that diverges. The runtime side costs one deployment, because starting the modified services is all that is left to do. A tenant that costs one deployment can be created hundreds of times a day.

Tenants onboard and offboard themselves

Multi-tenant platforms scale because nobody provisions tenants by hand. Signup creates the tenant, cancellation removes it, and no operator sits in the loop. Change tenants need the same contract. The tenant comes into existence when work on the change starts and disappears when the change merges or is abandoned, with no ticket at the front and no cleanup script at the back.

Offboarding is the half that platform teams underestimate. At person scale, an orphaned environment was a minor waste found in a quarterly cleanup. At change scale, orphans accumulate as fast as agents abandon experiments, and the leak outgrows the cleanup.

Automatic offboarding also keeps the accounting accurate. When tenants are created and destroyed by the change’s own lifecycle events, the number of live tenants equals the number of changes in flight, and platform capacity becomes a quantity you can measure and plan against instead of a pile of environments nobody is sure anyone still uses.

Re-measure the platform in changes, not seats

The practical shift for platform teams starts with measurement. Count changes in flight at peak, not seats: open pull requests with activity in the last day is a fine proxy, and for most teams the number is already several times headcount. Then price the marginal tenant: What does one more concurrent change cost in dollars and in minutes of setup? If the answer is a full environment and tens of minutes, the platform is still doing person-level tenancy.

Those two numbers expose where the old assumptions live. Namespace quotas sized per developer, staging booked by team calendar, database seeds refreshed nightly for everybody at once: each is a seat-denominated policy waiting to fail under change-denominated load. The fix in every case is the same three requirements: near-free creation, isolation sized to the change, lifecycle bound to the change, applied to whichever part of the platform still assumes the tenant is a person.

Change-level tenancy is the prerequisite for an agent-native SDLC

Every previous definition of the tenant named a person or a group of people, and that held because only people produced changes. A platform could equate one seat with one workstream, plan capacity from the hiring plan, and keep a human in the provisioning loop. Coding agents break all three of those properties at once: one person now operates several concurrent workstreams, those workstreams are created and abandoned at machine pace, and no human is positioned to provision or clean up each one.

“The change is the only unit of isolation that stays stable when the workers become software.”

That is why the software development lifecycle (SDLC) needs its tenant redefined around the change rather than around whoever, or whatever, wrote the code: the change is the only unit of isolation that stays stable when the workers become software. 

Organizations that keep person-sized tenancy will watch agent-generated changes queue behind infrastructure built for a fraction of the load. The ones that re-platform around the change will convert agent throughput into merged work. If you’re exploring the second path, that is exactly what we built Signadot to support.

The post Per-developer environments were the goal. Agents moved the goalposts. appeared first on The New Stack.

During AI DevCon in London this summer, Lamis Mukta, member of technical staff at Anthropic, hosted a stage presentation session entitled ‘Learning while you sleep, beyond memory to dreaming’.  

Mukta set out to examine where state-of-the-art memory management sits today in a world where (as she put it) “context is often orthogonal to the model intelligence” at hand.

“The newest model we’ve just released isn’t going to go out of the box and know exactly what it takes to succeed in your organization and what tasks you want it to do,” said Mukta. “It’s like agents [initially] not knowing their way around a codebase or knowing enough about your own user preferences.”

To steer agentic services the right way, systems obviously need access to memory to create a context window.

A brief history of Anthropic memory management 

Providing a brief history of Anthropic memory management, Mukta said that traditional approaches made use of CLAUDE.md, a file that Claude reads at the start of every conversation (that includes Bash commands, code style, and workflow rules) to give Claude persistent context that it can’t infer from code alone.

Effective to a degree, this technique becomes hard to manage over time, especially when a file with very important preferences gets very, very long. 

“So a second avenue that we investigated was memory tools, and this is interesting because it leans into the idea of what happens if we let agents autonomously manage their own memory systems? We let them decide when they read, when they write, and when they update memories,” explained Mukta.

This process happens in-band i.e. within the context of a session. When dovetailed with so-called progressive disclosure, the agent only looks at the light metadata at Layer 1, before reaching for full content and original source files in Layers 2 and 3, respectively, so that the system doesn’t overload the model’s context. 

“The way I like to think about it is as if I’d had a bookshelf in my room, and every time someone talks to me, I can kind of scan and look at my list of books and see if any of the titles might be relevant to the conversation, and then pick that off the shelf and read it when I need to,” explained Mukta.

But the bottleneck here is that we’re still driven by humans and agents working together i.e. we’re still being quite opinionated about what things need skills. The additional problem here is that memories can go stale and become irrelevant to an organization’s needs. Add the fact that a memory file may be written incorrectly or even maliciously injected and you can see why a lot of guardrails need to be in place.

“We introduced the concept of dreaming, which is a process that runs asynchronously in batch with its own allocated resources, to ensure that memories themselves are effective, up to date, and [so we can] help the agents learn over time.” 

Dreaming consolidates memory & cuts irrelevance

“So we introduced the concept of dreaming, which is a process that runs asynchronously in batch with its own allocated resources, to ensure that memories themselves are effective, up to date, and [so we can] help the agents learn over time,” explained Mukta. “[This process allows us] to consolidate memory and cut things that are no longer relevant, add things that agents are missing, and clean up and organize memory systems.”

In Anthropic’s world of slumber, dreaming is an out-of-band asynchronous process which the organization says solves the in-band limitation, where agents must split effort between completing and executing tasks, while also concurrently curating memory for their future selves. Dreaming spots recurring failure patterns where agents are consistently failing (wrong units, missing topics, broken tool configs, stylistic tics like overused em dashes), and proposes memory-store updates, again for human review, but hopefully at a more effecient level. 

This architecture underpins Anthropic’s Managed Agents memory and API approach at this level, so has the frontier model company won over developers?

Bad memories can outlive sessions

Staff software engineer, cloud architect and independent researcher in AI agent systems, Jayakumar Ramalingam, tells The New Stack that “dreaming is useful, but it also creates a dangerous promotion path” i.e. one that leads from repeated mistakes to persistent policy. 

“A bad answer normally dies with the session; a bad memory can influence thousands of future sessions. Human review sounds reassuring, but at fleet scale it can easily become a rubber stamp for recommendations nobody has time to reconstruct,” Ramalingam says. 

“The industry has spent too much time treating memory as a context window problem when it is really a state management problem.”

He insists that every proposed memory should “carry provenance, evidence and an expiration condition”, and not just exist as a pattern that recurred often enough to look real. Otherwise, he thinks that dreaming may help agents remember more while making organizations forget why the memory was trusted.

“Anthropic is getting one important thing right: its agent memory should look more like versioned infrastructure than artificial cognition. The industry has spent too much time treating memory as a context window problem when it is really a state management problem,” underlines Ramalingam. 

His point is – if an agent cannot show who changed a memory, why it changed and how to roll it back, it does not have production memory, so it becomes an unaudited configuration file with an AI attached.

Dreaming is the right instinct aimed at the wrong evidence

Enterprise AI architect and founder of Besk Tech, Vladimir Beskorovainyi, tells The New Stack that “dreaming is the right instinct aimed at the wrong evidence”, because the failures it catches (wrong units, broken tool configs, too many em dashes etc) are all visible on the surface of a transcript.

“The failure that actually costs you is an agent reaching for the wrong tool for a reason that looked perfectly defensible at the time,” Beskorovainyi says. “In the systems I run in production, the log records the decision rather than the API call, and that is the only reason a review pass like this finds anything worth finding.”

“When the ‘lately’ factor quietly becomes true. That leaves us at a point where versioning tells us what changed and when, not what is correct.”

He points to what he calls “a worse problem underneath the agent’s decision” i.e. if updates are proposed from recent batches, the memory store drifts towards whatever the agent fleet happened to do lately, and so the “lately” factor quietly becomes true. That leaves us at a point where versioning tells us what changed and when, not what is correct.

“The industry spent two years insisting that memory meant embeddings, and Anthropic solved it with a filesystem and grep [a Linux command that searches for patterns in files] and that is the most interesting decision in this whole discussion,” insists Beskorovainyi.

He says the reason it matters is legibility. A memory store a developer can open and read is a memory store an engineer can audit, and (he insists) “no vector database has ever offered that”, while everything else in the architecture (the versioning, the hashes, the tiered permissions), is ordinary distributed systems engineering we have known how to do for decades.

Dreaming is the clever (but worring) part

Founder of autonomous AI penetration testing company Penetrify, Viktor Bulanek, tells The New Stack that when the industry spent two years convinced that agent memory was a vector database problem, and Anthropic shipped grep, that was a useful thing.

“In terms of what Anthropic is getting right… a memory store you can cat, diff and code review is one you can actually operate, whereas nobody has ever successfully debugged an embedding that quietly ranked the wrong chunk third,” Bulanek says.

“Anthropic’s approach to dreaming is the clever part and also the part that worries me most, because it points an automated writer at session transcripts, and transcripts are full of content the agent did not author.” 

He thinks that the versioning matters here far more than the auditability framing suggests and reminds us that “rollback is not a compliance feature”; it is the undo button for a poisoned memory a software engineer discovers three weeks after it was written, which is the incident every serious agent deployment is going to have eventually.

“But to add balance here, Anthropic’s approach to dreaming is the clever part and also the part that worries me most, because it points an automated writer at session transcripts, and transcripts are full of content the agent did not author,” Bulanek cautions. 

“Anthropic is right that human review is the answer, but bulk review of proposed diffs is exactly the control that decays fastest once the suggestions are mostly good. The other gap is that nothing in this architecture says when a stored fact stops being true. Versioning tells you what changed, it does not tell you what rotted, and a confident note about a system that was refactored last month is worse than no memory at all,” he advises.

Bulanek’s work sees him run autonomous agents in production that perform penetration testing and run for hours unsupervised with real credentials against live systems, so memory for his team is both an operational cost and a security boundary at the same time.

The Anthropic way of doing things has an endearing lack of flair to it

Co-founder and CTO of Noah Labs, Berk Yilmaz, tells The New Stack that the Anthropic way of doing things has “an endearing lack of flair to it” in his view. 

“Everyone wants memory to feel like the newest incarnation of machine intelligence, and their pitch goes something like: just give it a filesystem, versioning, searchability, and don’t let a thousand processes stamp all over each other,” Yilmaz says. “This is closer to how production AI should be done. While we have spent a long time improving models, the supporting infrastructure has not kept up, failing in incredibly prosaic engineering ways.”

Yilmaz is behind a company that develops an AI-native IDE for government and regulated systems, built for air-gapped environments and legacy codebases. He reminds us that once a memory decision is made on which past behavior should become future behavior, memory itself ceases to be inert. 

“A hallucination that dies after a single session is a pain in the neck, but a hallucination that outlives a thousand sessions is infrastructure. The same thing applies to security; if an attack succeeds in writing to memory, it has become persistent. Provenance becomes absolutely critical here, how was the system taught this, where did it learn it from, who certified it, and can I undo it? In enterprise AI, sometimes forgetting is a safety measure,” adds Yilmaz.

A pragmatist would remember that Anthropic gets paid for usage, not efficiency

AI, product & data science leader and former Meta employee, Kerstin Frailey, tells The New Stack that at face value, dreaming (for her money) “certainly sounds like it has the potential to blow up AI bills” right now.

“A cynic would say this is designed to fill the revenue hole left by tokenmaxxing before Anthropic’s IPO,” Frailey says. “An optimist would hope for a beautifully thrifty design. A pragmatist would remember that Anthropic gets paid for usage, not efficiency. A skilled practitioner would run incremental pilots, aggressively monitor costs, and routinely test for measurable improvements.”

“As a nice bonus, dreaming offers potential system improvement, too. But its familiar predecessors – garbage collection and storage compaction – are comparatively deterministic and controlled.”

She continues and notes that dreaming offers cleanup and consolidation, which she defines as a “reasonable development” for any system that constantly generates new files. 

“As a nice bonus, it offers potential system improvement, too. But its familiar predecessors – garbage collection and storage compaction – are comparatively deterministic and controlled. Unlike its namesake or those analogues, dreaming appears neither cheap nor efficient: pay an AI to do the work once, then pay AIs to regularly review, revise, and restructure it,” she adds.

Dreaming as part of Anthropic’s Managed Agents memory and API approach isn’t alone. The notion of AI model dreaming (or automatic out-of-band background memory consolidation if we’re being formal about things) is also being popularised by OpenAI for ChatGPT, in stateful agent coding platform Letta and elsewhere. 

The bottom line here may be a realization that, in AI modeling terms at least, memory is actually maintenance.

The post Anthropic gave agents the ability to dream. Then developers woke up. appeared first on The New Stack.

Databricks on Tuesday announced that it’s acquiring Electric, the startup behind the WASM-based Postgres project PGlite and the Electric sync engine, as agentic applications change how developers use databases.

The Electric team will join Neon, the serverless Postgres company Databricks acquired for about $1 billion last year and the foundation of its Lakebase database service.

The companies didn’t disclose the terms of the deal.

What Databricks bought

PGlite is a complete Postgres database in WebAssembly (WASM). It runs in the browser, a Node.js process, or inside the kind of sandboxes agents use to execute code. It supports dynamic extension loading, including pgvector, the preferred Postgres vector extension.

According to the companies, PGlite has grown from 1 million to 13 million weekly downloads over the last year.

The sync engine at the core of Electric

It’s the Electric sync engine that is core to Databrick’s interest in Electric, though. This engine keeps a central Postgres database that can then be synced in near real-time with browser tabs, mobile apps, or agents. As Databricks notes, this is the multiplayer model of Figma or Google Docs, but applied to Postgres and the agents that use it.

The Neon team, in its own announcement, notes that “complex problems like conflict resolution, partial replication, and reconnection logic make real-time sync difficult to build from scratch.” Hence why Databricks likely acquired Electric instead of trying to build this from scratch itself.

As for the future of Electric, the company’s founders James Arthur and Valter Balegas write that “everything we’ve previously open sourced stays open source.” This covers the sync engine, PGlite, Durable Streams, and TanStack DB.

What doesn’t survive the deal, however, is Electric’s hosted service. “Electric Cloud is winding down,” the founders. “Cloud users will need to self-host or move to another provider.”

The deal also extends a string of database acquisitions for Databricks that includes Neon itself and, more recently, the transactional processing startup Mooncake.

A database that lives for 10 seconds

As the Databricks team argues, traditional non-agentic applications share one database among many clients, and that database is the most permanent piece of the stack. But agent workloads change this.

In a recent post on how agentic development changes databases, Databricks’ Ippokratis Pandis, Nikita Shamgunov, and Reynold Xin write that agents now create roughly four times more databases than human users do on Lakebase. They also stress that the average project now carries about 10 database branches, and that some projects run more than 500 branch iterations deep.

For some types of applications on Lakebase, the average database compute is now alive for under 10 seconds.

Agents, as it turns out, like to branch databases the way they branch code, a pattern Neon built its architecture around.

In practice, a coding agent spins up a sandbox, instantiates PGlite inside it, builds and tests against the database, and then either throws the whole thing away or syncs the result with — in the Databricks context — a Lakebase branch. Because Lakebase separates storage from compute and keeps its data in Postgres page formats on object storage, creating that branch is a relatively cheap copy-on-write metadata operation.

“As coding agents drive the cost of creation to zero,” the Neon team writes, “the number of applications explodes, and most of them are small.” A database server, even a serverless one that scales to zero, imposes a floor on what the smallest viable app costs to run. “You can’t have an age of abundance if every app requires a fixed minimum of compute,” the post argues.

‘Two halves of the same idea’

It’s worth noting that PGlite didn’t start at Electric. Instead, it began as an experiment by Neon co-founder Stas Kelvich, who compiled Postgres to WASM to see whether it could run client-side. Electric picked the work up and turned it into a production project. “That repo became the basis of PGlite,” Arthur and Balegas write.

As Databricks’ announcement notes, this now “reunites two halves of the same idea.”

The post Databricks acquires Electric to give every AI agent its own Postgres database appeared first on The New Stack.

Cloudflare launched its CloudflareOS open-source AI workspace platform this week, promising every employee a secure workspace equipped with AI tools and access to internal company systems.

Positioned significantly beyond the notion of legacy virtual desktop infrastructure (VDI) services, which delivered the same fixed applications through a remote screen — and even past the dynamic application delivery, app masking and streaming of modern VDI iterations — this is an essentially more dynamic way of working with internal company tools, documents and systems. 

Cloudflare’s CloudflareOS makes its apps and services accessible through secure connection points that verify every user and every agentic request or connection point before access is granted. 

In AI, every new work session starts from zero

The technology proposition here is built on the fundamental truth that the typical enterprise AI tool knows a great deal about the world, but almost nothing about how a specific company operates, the shape of its internal systems, approval processes, or the ways teams actually get work done.

That means every new work session starts from zero, with employees re-explaining context the AI should already know. But how can new business context-aware agentic access freedoms be granted securely?

Rita Koslov, VP for developers & AI at Cloudflare, tells The New Stack that powering up modern agent use cases means “data is often leaving controlled systems en masse” for the first time.

“It used to be the case that, for example, people asked analytics questions in the data warehouse where the organization had control,” Koslov says. “Now, employees are asking for API keys for their own tools, agents, etc. This creates a new class of security problems that Cloudflare OS helps to solve.”

Capability-based access beats handing agents raw API keys 

Cloudflare has built what we can call capability-based access, which the company promises beats handing agents raw API keys outright.

“API keys give agents broad access to systems; a capability-access-based approach lets us grant one specific resource, then record exactly what the agent observed, and verify that anyone who sees its work is also allowed to access the source,” underlines Koslov.

Cloudflare OS enables an agent to create documents, slides, spreadsheets, workflows, other agents – or entirely new full-stack applications – all tailored to an employee’s work. What it creates can remain connected to live data sources, be modified and shared safely, and be used directly by both people and agents.

“API keys give agents broad access to systems; a capability-access-based approach lets us grant one specific resource, record exactly what the agent observed, and verify that anyone who sees its work is also allowed to access the source.”

In terms of how developers and systems operations professionals should react to this offering, Koslov suggests that “the difficult problem is not generating an app” today. Instead, the real challenge is safely running thousands (or millions) of dynamically generated apps, each with persisted state and controlled access. 

“Cloudflare OS uses Dynamic Workers, which provide lightweight isolated runtimes to load each app’s code on demand, and Durable Objects Facets to give it isolated SQLite storage under the platform’s supervision. Outbound networking is disabled by default, and Gatekeepers expose only the resources explicitly granted by the users,” Koslov says. “Dynamic Workers and Durable Objects Facets were invented because doing this was previously not possible.”

For completeness here – and once again a Cloudflare original technology service – a Gatekeeper is a service-specific Worker that sits between Cloudflare OS and an external service to interpret and understand the service’s API, its resources, and the operations that can be performed on them. 

What happens when it all goes wrong

Koslov confirms that she knows how badly things can skew out of control in unmanaged environments. 

“We know this from our own experience talking to other companies on all accounts. They’ve shared instances of internal data copied into AI tools that IT did not know were in use, AI keys embedded into agent-built applications, and even data being shared internally to people who ordinarily wouldn’t have access (or even publicly),” she adds.

Building a tailored alternative is no small project; a platform with proper security and real integration into internal systems can take years to develop and cost millions to maintain. In the meantime, employees find workarounds, IT loses track of which AI tools are running and who is using them, and costs pile up, often with little to show for it. 

CloudflareOS starts from a different premise: a company captures its knowledge, processes, and ways of working once in a form AI can actually execute, and that knowledge travels with every employee’s workspace from day one.

How do we measure business ‘context’?

“Captured business ‘context’ in this case can include company terminology, policies, operating procedures, product documentation, technical standards, sales processes, templates, and established ways of performing recurring work,” confirms Koslov.

CloudflareOS started as the platform Cloudflare built to run its own workforce. Thousands of Cloudflare employees across every team use it daily to perform research, create documents connected to live data, automate repetitive tasks, and build working apps for their day-to-day jobs. 

That same platform is now available to any organization as open-source software. Because it’s open source and runs in a company’s own Cloudflare account, organizations own what they build on it. 

The platform itself works on any AI model and controls cost. Through Cloudflare AI Gateway, organizations can use any AI model provider, so they’re not locked into one vendor. Administrators see exactly what’s being spent, broken down by person, team, or app. They can set spending budgets, rate limits, or route routine tasks to smaller, more affordable models where a top-tier model isn’t needed.

Pricing platforms by the token is the wrong meter entirely

Cautiously upbeat about the wider story playing out here, enterprise AI architect and founder of Besk Tech, Vladimir Beskorovainyi, tells The New Stack that, traditionally, the industry is pricing these platforms by the token, “and that is the wrong meter entirely” in his view.

“In this example with Cloudflare OS, what a company actually buys here is the obligation to write down how an AI-powered business process really works, and then keep that description true as the business shifts underneath it,” Beskorovainyi says. “The model is the commodity part. What costs real money is the curated context, and nobody budgets for the fact that it starts decaying the day it is written, which is exactly what decides whether any of this survives contact with production.”

“Cost broken down by person, team and app is the first time I have seen a vendor treat spend as an engineering signal rather than an invoice, and sending routine work to a smaller model is the obvious next step that most enterprises still fail to take.

Beskorovainyi insists that the organizations that win in this game will “not necessarily be the ones running the best model”; they will be the ones that could “already answer in writing what their own approval process is”, way before an agent ever asked.

“Cost broken down by person, team and app is the first time I have seen a vendor treat spend as an engineering signal rather than an invoice, and sending routine work to a smaller model is the obvious next step that most enterprises still fail to take,” advises Beskorovainyi.

Owning your own context is not the same as your context being any good

He clarifies his point and explains that the qualification here is that “owning your own context is not the same thing as your context being any good”, and so open source tooling and community connections plus an organization’s own account settle who holds the context file.

“Neither tells us whether what is recorded and logged in the context file is still true this quarter. That work stays with the customer permanently, and it is where I expect most of these deployments to come apart, not in anything Cloudflare has built,” Beskorovainyi adds.

Matthew Prince, co-founder and CEO of Cloudflare has said that his team built Cloudflare OS, “because nothing else did what we needed”, and so now, any company can start from where it took the organization’s internal software engineering function years to get to.

The apparent appeal here must come down to the dynamic nature of Cloudflare OS and its ability to work with and apply AI tools at a custom-engineered business context-aware level with zero trust by default. The platform can turn any output into a working app with its own isolated database, real-time capabilities, and access controls – once agan, that’s not legacy virtual desktop is it? 

No developer required (yet)

The bottom line from Cloudflare is that employees can use any app on Cloudflare OS  directly, or adapt it for their own needs so that it’s a case of “no developer required”, or at least until the next integration task needs to be shouldered, or the big thing comes along, or both.

The post Why AI tools know nothing about your company — until now appeared first on The New Stack.

A developer supervising four coding agents has four changes in flight at once, each in its own git worktree. That isn’t an exotic setup anymore: Anthropic’s documentation now treats a worktree per session as the default way to run agents in parallel, and what was an expert workflow two years ago is the recommended starting point today.

The branches themselves aren’t new. Git made them cheap 20 years ago so developers could isolate changes and work on several things at once, but in practice a developer switched between branches and shipped one change at a time. That kept everything below the code layer singular: one continuous integration (CI) queue, one staging environment, one database everyone tested against. The number of changes contending for those shared resources was capped by headcount, and before agents, only larger teams ever hit the cap.

“Coding agents removed the cap. The branch can no longer stop at the code layer.”

Coding agents removed the cap. Those four branches are no longer something one developer rotates through. They are four active changes moving toward merge in parallel. The gap becomes unworkable: branching is free at the code layer and missing everywhere below it. Each change needs to exist all the way down the stack, not as a diff in a directory but as a running, testable version of the system. The branch can no longer stop at the code layer.

Parallel until the first shared resource

Code branches in milliseconds. A worktree gives each agent a private copy of the repository for the cost of a checkout, and 10 agents can work side by side without seeing each other’s edits.

The output shows up downstream. Telemetry from Faros AI across more than 10,000 developers found that teams with high AI adoption merge 98% more pull requests while review time grows 91%. Nothing downstream of code generation was sized for that arrival rate.

Then each change needs to run. There is one staging cluster, one seeded database, one message queue, one set of dependent services, and every branch that reaches this floor stops being parallel. Four agents produce four candidate changes in an afternoon, and all four line up behind the same shared environment to find out whether they work.

The queue is more expensive than it looks, because agents don’t wait well. An agent blocked on an environment either sits idle holding a stale view of the system or plows ahead validating against mocks, and the developer supervising it context-switches away. By the time the shared environment frees up, the cheap part of the work has to be partially redone.

The bottleneck isn’t code generation, and it isn’t review capacity alone. It’s the first shared resource a change touches, because a branch that can’t run is a branch that can’t be trusted.

“The bottleneck isn’t code generation, and it isn’t review capacity alone. It’s the first shared resource a change touches.”

A branch is a delta, not a copy

The way out is to stop treating branching as something git does and start treating it as something every layer does. Branch-based development names the pattern: each layer of the stack offers a cheap, instant, disposable branch primitive, so a change can exist end to end without duplicating anything it didn’t touch.

The mechanic is the one git established, and everyone has been living on for two decades: branches are cheap because they share everything unchanged and carry only the delta. The rest of the stack has been relearning that idea layer by layer ever since — share by default, isolate what changed.

Naming the pattern matters because each layer discovered it separately and called it something different. Worktrees, pipeline caching, preview deploys, database branching, and environment sandboxing sound like five unrelated features. They’re the same idea applied at five layers, and seeing that changes what you ask of the layers that lack it.

The upper layers learned this years ago

CI absorbed the lesson a decade ago. Every branch gets its own pipeline run on a shared runner pool, with build caches doing the copy-on-write work of reusing unchanged artifacts. Nobody provisions a build system per branch, and nobody queues behind a single global build anymore.

The front end followed. On Vercel, every push to a non-production branch gets its own preview deployment by default; Netlify works the same way, and the branch itself is one immutable build plus routing on shared hosting infrastructure. Reviewers stopped asking whether a change works on someone’s laptop, because the change is already running somewhere.

Both cases have the same shape: the expensive machinery is shared, the branch is thin, and creating one is cheap enough that nobody thinks about it. That’s what a layer feels like once it has a branch primitive.

Each of these primitives also changed behavior once it arrived. Per-branch CI made it normal to run the full test suite on every push instead of nightly. Preview deploys made it normal for a product manager to click through a change before merge. Cheap branches don’t just remove a queue; they raise the bar for what gets checked before merge.

The data layer was supposed to be the hard case

Databases carry state, so conventional wisdom said branching would never work there. Then Neon, PlanetScale and Xata shipped it anyway, and Neon’s documentation now makes the parallel explicit: branch your data the same way you branch your code.

A database branch is a copy-on-write view over shared storage pages, created in seconds regardless of how large the database is. Schema migrations and risky data changes get validated against production-shaped data instead of a stale seed script, and the branch disappears when the work merges.

“If the layer with the most state can hand out branches in seconds, statelessness was never the real requirement.”

The data layer matters to this story because it removed the best excuse. If the layer with the most state can hand out branches in seconds, statelessness was never the real requirement. Whatever is still unbranched is unbranched by choice.

The runtime is the last layer to learn the trick

The microservices runtime resisted longest because it looks nothing like a file tree. It has live traffic, a service graph and dozens of moving dependencies, and the naive branch, a full copy of the environment, is so expensive that most teams concluded branching did not apply here.

The copy-on-write move works anyway. Run one shared, stable version of the system that is continuously deployed from main. For each change, deploy only the services the change touches as a lightweight ephemeral environment, and route each test request through the changed services while everything else falls through to the shared stable versions. The environment branch costs roughly what the changed services cost, which is why one can exist for every change an agent produces.

Routing is the part that sounds exotic and isn’t. A request tagged with a label gets steered to the changed service versions at each hop, propagated through the call chain the same way trace context already flows through most instrumented systems. The shared stable environment plays the role of main, the changed services are the delta, and the label is the pointer that assembles a coherent view of the system per request.

This isn’t a hypothetical architecture. Uber built SLATE to give each developer an ephemeral environment routed against shared production-grade dependencies because contention over staging could not keep up with its developer count.

What an agent-native stack means

Put the layers together and a different development model appears. An agent picks up a task, and the change gets a worktree, a pipeline run, a preview, a data branch, and a running environment from the start. Validation stops being the scarce resource that serializes everything upstream of it.

Teams are already composing the lower layers. Bitso, a crypto exchange with 250-plus engineers, pairs an environment branch with a database branch for each change, so the runtime delta and the data delta travel together and shared staging stays out of the critical path.

That end-to-end branch is what the phrase agent-native software development lifecycle should mean. Not agents wired into yesterday’s pipeline, but a stack where any change, human or machine, can exist at every layer for as long as validation takes and disappear afterward.

The payoff compounds with agent count. When the branch primitive at every layer is a delta over something shared, validation concurrency scales with cluster capacity instead of with budget, and the number of changes a team can prove correct per day rises with the number it can generate. That is the ratio that decides whether agent adoption shows up as shipped software or as a longer queue.

The audit is cheap to run. Follow one change from worktree to validated and note the first layer where it waits on something shared. That’s where your stack stops branching. 

For most teams, the answer is the runtime, and if it’s yours, Signadot is a practical place to start.

The post Anthropic recommends a git worktree per agent. Your runtime infra makes that a problem. appeared first on The New Stack.

Thibault Sottiaux, who leads core products at OpenAI, believes that today’s version of Codex will seem outdated before the year ends.

Sottiaux posted on X late Monday, “Given some of the results I’m seeing recently, it’s pretty clear Codex is a good harness.” He continued, “But it will seem primitive in 2-3 months and we’re about to go through another major evolution in how we use AI at the frontier.” He also said, “The next generation of models need more than your laptop.”

“It will seem primitive in 2-3 months and we’re about to go through another major evolution in how we use AI at the frontier.”

Given some of the results I'm seeing recently, it's pretty clear Codex is a good harness.

But it will seem primitive in 2-3 months and we're about to go through another major evolution in how we use AI at the frontier. The next generation of models need more than your laptop.

— Tibo (@thsottiaux) August 4, 2026

Sottiaux did not share details about OpenAI’s plans for the coming months. However, his comments are timely since the company is already working to move Codex beyond tasks limited to a developer’s computer. Since launching a new GPT-5 model for Codex in early July and surpassing 8 million users shortly after, the product has been evolving quickly.

Ona fills the infrastructure gap

In June, OpenAI said it plans to buy Ona, a company that creates secure cloud development environments. OpenAI called this deal part of the “next phase of Codex,” where agents can keep working in a customer’s cloud even after the laptop that started the job is closed.

“The next generation of models need more than your laptop.”

Codex currently uses cloud infrastructure, but it might still need the developer’s laptop to access projects and run tools. If the laptop goes offline, the agent may lose what it needs to keep working.

OpenAI has already tested this approach. In an experiment published in February, Codex worked for about 25 hours straight, used around 13 million tokens, and generated about 30,000 lines of code while building a design tool from scratch. Alibaba has pushed even further — its Qwen3.8-Max agent recently coded autonomously for 16 days, producing 265 commits with zero human help. Ona could help solve this problem.

The company, which used to be called Gitpod, creates cloud environments that can be set up with the tools and dependencies needed for each project. OpenAI said Ona has helped 2 million developers use these environments.

Agents need persistent workspaces

If the acquisition goes through, Ona’s technology would let Codex have a permanent workspace in a customer’s cloud. Agents could get the context and tools they need for a task without relying on an active session on a local machine.

OpenAI says companies will still decide how Codex works in their cloud environments, including what sensitive systems it can access. The deal is not final yet, so OpenAI and Ona are still separate companies.

It is not clear if Sottiaux’s prediction is truly related to Ona. Although the acquisition shows OpenAI is looking beyond just the model, because for Codex to work on its own, it needs an environment that stays online even when the developer’s laptop is off.

Unfortunately, moving the execution environment to the cloud solves one problem but creates many new ones.

Security risks grow with access

Letting a coding agent have full access to a company’s network or a developer’s credentials is undoubtedly risky. OpenAI said Ona’s customer-controlled model will let agents work inside an organization’s own cloud, while OpenAI provides the model and orchestration. Even if the model gets stronger and can handle more complex tasks, it still needs a secure place to run commands, save its progress, and interact with other systems.

Developers can assign tasks like refactoring, upgrading dependencies, or investigating bugs to the agent and let it work remotely. They can track its progress, check terminal output, and step in if a human decision is needed. When the agent finishes, users can review the pull request and see which tests were run.

OpenAI is already heading this way. Codex has been folded into the ChatGPT desktop app and can handle parallel tasks. Its desktop app is increasingly built around managing agents, and its mobile features let developers monitor and guide tasks running on laptops, devboxes, or remote environments. It has also expanded with new plugins and tools aimed at knowledge workers beyond just developers.

Agent environments will use computing resources along with CI/CD systems.

Managing a new agent layer

This change means there is a new type of infrastructure to manage. Anthropic is already moving on this front — its acqui-hire of Mendral is aimed at automating CI/CD tasks like flaky tests and dependency reviews directly inside its platform. Agents will need their own identities and access rules, and their actions will need to be logged, reviewed, and linked back to them, just like with human developers and current automation.

Sottiaux’s prediction certainly has provoked curiosity. Two or three months is a very short time for a product to become “primitive.”

​

The post Today’s Codex will feel “primitive” by fall — and its own team’s roadmap backs it up appeared first on The New Stack.

OpenAI has detailed how the GPT-5.6 model family balances capability and cost across its stack, and the company‘s most important claim is a benchmark result showing that its flagship model, GPT-5.6 Sol, with maximum reasoning, outperforms Claude Fable 5 from Anthropic on the Artificial Analysis Coding Agent Index. The margin comes with 54% fewer output tokens. The findings were shared in a company blog post on Wednesday.

For developers, what matters most is how OpenAI arrived at the benchmark results and the role GPT-5.6 Sol played in optimizing the infrastructure that now serves it.

The family spans three models across the price curve. In addition to Sol, there is Terra, which performs as well as GPT-5.5 on intelligence benchmarks at half the price, and Luna, the fastest and most affordable, which is priced 80% below Sol.

The efficiencies come from optimizations at four layers, spanning the models, inference, the API stack, and the agentic harness behind Codex and ChatGPT Work.

According to the post reviewed by The New Stack ahead of its publication, the efficiencies come from optimizations across four layers: models, inference, the API stack, and the agentic harness behind Codex and ChatGPT Work. The architecture diagrams in the post draw the same separation as three planes: the local harness, CPU-bound API orchestration, and GPU-bound model inference.

For developers building and operating agents, the post is worth reading less as a product announcement and more as a systems paper. Nearly every technique it describes, from incremental tokenization to append-only context, applies to any team running a tool-calling loop at scale.

A model that rewrites its own serving code

The efficiency work starts in training. OpenAI says GPT-5.6 is trained to achieve more work per token, with training optimized for both task success and efficiency so the model takes a more direct path through a task.

With Codex, GPT-5.6 Sol autonomously rewrote and optimized OpenAI’s production kernels, the core code that executes the mathematical operations making up the model. OpenAI says this worked in part because GPT-5.6 is trained to write and improve kernels in Triton and Gluon. Both are open-source GPU programming languages maintained by OpenAI. These efforts, combined with broader kernel advancements from the model, reduced end-to-end serving costs by 20%.

Correctness is the obvious concern when a model rewrites the code it runs on. To address it, OpenAI reports heavy investment in verification tooling. That includes the open-source Floating-Point Sanitizer (FpSan), which validates the kernels GPT-5.6 Sol produces before they reach production.

The model went further with speculative decoding, a technique in which a smaller draft model proposes several tokens that the primary model verifies in parallel. The approach will feel familiar to anyone who understands how modern CPUs speculatively execute instructions ahead of a branch. Accepted proposals produce multiple output tokens from a single pass of the primary model. That reduces the expensive sequential computation the primary model would otherwise perform.

GPT-5.6 Sol in Codex improved its own draft model by designing and running hundreds of experiments on its architecture, with changes tested across size, structure, and features. The model also launched and monitored the speculative training process. It intervened autonomously when hardware failed or training became unstable. OpenAI reports the resulting improvements lifted token-generation efficiency by more than 15%.

More tokens from the same GPUs

OpenAI frames its inference work around a single objective – serving more tokens with the same hardware while preserving the intelligence, latency, availability, and reliability users expect. In a compute-constrained market where demand grows faster than capacity, that objective influences every design decision in the serving path.

Load balancing operates at three distinct levels. Globally, requests are routed based on geography, available capacity, and accelerator type. Within a cluster, work is distributed across model instances based on load, context length, and cache availability. Within each instance, work is partitioned across accelerators, the model’s experts, and computing cores. GPT-5.6 Sol in Codex helps OpenAI analyze production traffic and identify previously overlooked sources of imbalance. The same loop tests new routing strategies and helps engineers constantly tune the heuristics. OpenAI states that these load-balancing improvements alone dramatically reduced the cost of serving its models.

The key-value (KV) cache received the same treatment. When processing uncached input tokens, the model builds the KV cache in a single compute-intensive pass, then repeatedly reads from and extends it during generation. The optimal serving configuration depends heavily on prompt length, batch size, and cache hit rate. It covers batching, sharding, and cache management, and the configuration space was previously too large to tune systematically. With GPT-5.6 Sol in Codex, OpenAI analyzed production workloads and generated candidate configurations. The company says this makes workload-specific optimization practical at a level that broad heuristics could not reach earlier.

Process only what changed

The API team focuses on everything that happens around a model call. After a prompt is submitted, the API stack receives the request, loads context, and validates the input. Safety checks run next, and the text is converted into tokens for inference. OpenAI measures this overhead through time to first token (TTFT), time between tokens (TBT), and end-to-end time (E2E).

Tokenization is an O(n) operation, so longer prompts take longer to process. Codex would send the full conversation context after every tool call. That meant paying to tokenize the same conversation dozens of times per turn, even though only a small amount of context was new in each request. OpenAI solved this with a WebSocket integration that hoists tokenization state to the server. The first call renders and tokenizes the full prompt. Later calls send only the new input with a reference to the conversation, bringing the operation closer to O(1). The pattern mirrors an incremental build system that recompiles only the files that changed rather than the whole project.

These savings compound in tool-heavy workflows, where every tool result triggers another round trip through the API. For rollouts with 20 or more tool calls, OpenAI reports up to roughly 40% faster end-to-end execution.

Hardware turned out to matter as much as protocol design. All of OpenAI’s infrastructure runs on Kubernetes. The company found that nodes with the same instance type often carried different CPU generations, with many running outdated processors. In its measurements, the older processors consumed roughly twice the CPU resources for the same work. Reweighting traffic toward newer processors improved TTFT by about 20%, and CPU generation is now part of capacity planning.

OpenAI names four fates for application-layer overhead: delete it, overlap it with useful work, run it on faster hardware, or make the code consume fewer CPU cycles. Its asyncio changes move work off the critical path, while newer hardware and Rust implementations make the remaining work faster and more predictable.

An append-only harness

The agentic harness is a Rust-based orchestration layer that connects the models, tools, and the user’s environment. In a single turn, Codex might inspect source code, search deployment history, and read incident reports. Editing a file and running the tests each add another request. Since a task can require 30 model requests, an extra second per request adds up quickly.

Context bloat is the first target for the harness. As agents gain access to more tools, skills, plugins, and conversation history, context windows expand. The growth increases cost, distracts the model, and prompts unnecessary reasoning. The harness counters this with deferred discovery, which surfaces integrations, custom Model Context Protocol (MCP) tools, skills, and plugins only when needed. Tool output is capped at 10,000 tokens by default unless the model requests a different limit.

Prompt caching drives the second design choice. An agent loop resends the same instructions, tool definitions, and earlier results multiple times within a turn. The harness therefore treats all model-visible history as append-only, with new messages and tool results added at the end rather than inserted into earlier context. Tools are presented in a deterministic order, and runtime settings, such as approval policies, are applied during execution rather than embedded in tool definitions. OpenAI credits this design for the high prompt-cache hit rates in Codex and ChatGPT Work.

Platform teams building internal agents can adopt every one of these choices without OpenAI’s scale. Append-only context, deterministic tool ordering, and capped tool output attack token spend directly. That makes them the most portable lessons in the post for enterprises watching inference bills grow with each new agent deployment.

Where the gains come from

The post associates a number with most of its optimizations, and the figures are OpenAI’s own production measurements. Taken together, they show how modest individual wins compound across a serving stack.

The key takeaways

In summary, OpenAI describes the GPT-5.6 efficiency gains as the result of years of compounding improvements. They span research, inference, the API stack, and the agentic harness. The company states that the model’s role in landing many of them makes it optimistic that the pace of optimization will accelerate. Kernel work is called out as an area of continued investment.

The post positions efficiency, alongside raw intelligence, as the axis on which frontier labs now compete. The claimed 54% output-token advantage over Claude Fable 5 shows how OpenAI intends to fight that battle. The engineering blog makes a plausible case that software optimization is becoming an important lever alongside hardware improvements in reducing the cost of serving frontier models. The figures remain OpenAI’s own production measurements. The autonomy on display operates within Codex, with engineers in the loop. Developers and enterprises benefit either way, as these under-the-hood improvements reach them as more capable models at lower prices across the cost-intelligence curve.

The post Kernel of truth: GPT-5.6 Sol can cut its own costs, says OpenAI appeared first on The New Stack.