Put several AI bots to work, and a mistake by one may not stay within its assigned task. For example, that error could reach another bot’s files and login credentials, or even the computer running them all. Two releases this month offered companies very different ways of containing that risk.

On August 17, Nous Research announced that its Bot Mode would ship bundled and enabled by default in Hermes Agent v0.20.3, turning agent profiles into a roster of named bots that hand off work to one another. About a week earlier, SpaceXAI launched Grok Bot with almost the same interface: a sidebar of named teammates who sign in to your tools and keep working long after you close your laptop.

The interface converged within a week, but the answer to the question every platform team has to ask did not: When one bot goes wrong, what can it reach?

Four projects have now come to their own answer, and no two of them agree.

1. Grok Bot draws the line around the user account.
2. Hermes draws it around the profile.
3. OpenClaw draws it around an optional runtime sandbox.
4. ClawFleet draws it around a container.

Taken together, the documentation shows an industry converging on the persistent coworker interface far faster than it is converging on what constitutes an identity or a security boundary for it.

Four projects, four written answers

Every one of these products now offers the same surface. You create several named agents, assign them different jobs, and have them pass work among themselves. The naming convention alone suggests separation, since a bot called Expense Manager and a bot called Talent Scout sound like they occupy different rooms in a shared office.

The documentation says otherwise, and it says something different in each case. The unit of isolation is the account in one product, the profile directory in another, an opt-in container in a third, and the deployment topology in the fourth. Those four units are not interchangeable, and an operator who assumes the roster itself is the boundary will be right in exactly one of the four cases.

Is Grok Bot confused about what it wants to be?

SpaceXAI’s launch post leads with the promise that bots have their own computer. The documentation, last updated the same day, describes a single persistent cloud computer assigned to the user account rather than to any individual bot. Browser cookies and signed-in sessions are shared across the roster, files are visible to every bot, and command-line credentials are shared. One bot can pick up work that another bot saved.

Each bot gets its own screen on that machine, which allows several of them to run browser and desktop tools in parallel. SpaceXAI is direct about what those screens are not. The documentation calls them “separate work surfaces, not separate security boundaries.” It then instructs operators to keep a credential or file off the machine entirely if another bot on the account cannot use it.

The consequences run further than credentials. Signing in for one bot makes that session available to the others because the browser is shared. Installed connectors are account-wide, and their availability is not isolated to a single bot. The shared workspace sits at /workspace and is designed to survive computer updates and recovery, so the durable state is shared across the whole roster.

None of this is an implementation accident. It is what makes handoffs between bots cheap, and cheap handoffs are the product. But an operator reading only the launch page would build a mental model that the documentation contradicts, and that gap is where the risk sits.

Hermes gives each bot its own profile

Nous took the opposite architectural position. In Hermes, a bot is a profile, and each profile has its own configuration, memory, skills, credentials, and chat history stored in its own directory on disk. Handoffs between bots run as real invocations against the named profile, rather than as a shared context blob passed around within a single process.

Nous shipped the teammate protocol as part of v0.20.3, alongside the MCP 2.x SDK migration and a set of runtime hardening changes. The company archived the standalone plugin repository once the merge was completed. Bot Mode is on by default, so a Hermes user who updates gets the roster without opting in.

Two qualifications matter before anyone reads that as containment. A separate credential store does not guarantee different credentials, since what ends up in a new profile depends on how the operator created it and what they edited afterward. And every profile still shares the host machine, its operating system user, and its filesystem permissions. What Hermes documents is workstation-level separation of agent state, a meaningfully stronger default than a shared cloud account, but not the same as isolation.

OpenClaw’s sandbox is off by default

OpenClaw documents the most complete boundary of the four. When the sandbox is enabled with the Docker backend, agent tool execution runs inside isolated containers. At the same time, the gateway remains on the host, and the scope can be selected per session, per agent, or shared across agents. Each scope gets its own workspace. Auth material lives per agent under an agent-scoped auth profiles file. Operators can configure network isolation, resource limits, and allow-or-deny tool policies on top of it.

The documented default for that sandbox mode is off. That is a defensible choice for a project most people run on a laptop, where the container overhead buys little against a single-user threat model. The underlying setup behavior deserves more attention. If sandbox prerequisites fail during setup, the script resets sandbox mode to off rather than refusing to start, so an operator who intended isolation and encountered a Docker socket issue ends up running without sandbox isolation. The documentation also warns against mounting the host Docker socket into agent sandbox containers and flags the CLI container’s shared network namespace with the gateway as a trust boundary in its own right.

ClawFleet answers the same question by moving it into the deployment topology. The project documents a wrapper that puts each OpenClaw or Hermes agent in its own Docker container with an isolated filesystem and network. It lists roughly 500 MB of memory per OpenClaw instance and 150 MB per Hermes instance. That cost is why the other three projects make the boundary optional or skip it, and naming the number makes the trade-off legible.

How to choose which bot is right for you

Each project gives operators different advice

The operational guidance diverges as sharply as the architecture. OpenClaw’s docs read like infrastructure documentation, naming specific hazards such as the Docker socket and the shared network namespace, and telling the operator what not to do. Hermes documents the profile layout and the protocol, then leaves policy to the operator. Grok Bot’s guidance is largely the warning itself, an instruction to treat the account as the boundary and to keep sensitive credentials off the shared machine entirely.

Grok Bot carries a second disclaimer worth reading alongside the first. Sensitive actions route through an approval mechanism. SpaceXAI documents the categories that trigger it, including sending messages, publishing content, purchases and transfers, deleting data, and touching production. Enforcement runs through an LLM classifier. Cursor‘s documentation for that same engine states plainly that the classifier is not a security boundary and can make mistakes. A buyer evaluating the product therefore finds the phrase twice, attached to the two mechanisms they would most reasonably assume protect them.

AI agents still lack identities of their own

Enterprises can adopt any of these four products today and get real work done, and the honest reading is that all four are engineering their boundaries in good faith against different threat models. What none of them provides is an identity for the agent. In every case, the bot borrows the operator’s credentials, whether from a shared cloud browser, a profile directory, or a container volume, and the entire security conversation boils down to how far those borrowed credentials travel.

“There is no primitive to standardize on, so each project has invented a boundary at whatever layer it already controlled, the account, the profile, the runtime, or the container.”

That is why the four answers differ so much. There is no primitive to standardize on, so each project has invented a boundary at whatever layer it already controlled: the account, the profile, the runtime, or the container. Expect to see that gap close on the identity side rather than the agent side, through scoped delegation and per-agent credentials issued by the identity provider, rather than being copied from the human. Until then, the useful move for platform teams is unglamorous and specific. Read the security page before the launch page, because for this class of product, they describe different things.

The post Grok Bot vs. Hermes: Where each draws the security boundary appeared first on The New Stack.

The massive growth of generative AI has fundamentally altered data center design. As distributed model training scales to span hundreds of thousands of GPUs, the scale-out network connecting these nodes has emerged as a first-order performance bottleneck. For decades, traditional off-the-shelf Ethernet has been the undisputed king of enterprise and cloud networking. It is cheap, standardized…

Source

IBM is announcing today at the annual Hot Chips conference what may be the most consequential change to mainframe architecture in decades: a processor whose cores can natively execute both IBM's own instruction set and Arm's — switching between the two in nanoseconds.

The chip, which will power the next generation of IBM Z and LinuxONE systems, is the first dual-architecture mainframe processor ever built. It is designed to let enterprises run the vast and fast-growing ecosystem of Arm-native Linux software — including the AI frameworks that increasingly define modern infrastructure — directly alongside the z/OS transaction-processing workloads that anchor the world's banks, insurers, and governments.

"As technology enthusiasts on both sides, we're really excited about being what I would consider one of the most powerful commercially available processors that'll be dual architecture," Tina Tarquinio, chief product officer for IBM Z and LinuxONE, told VentureBeat in an exclusive interview ahead of the announcement.

The announcement marks the first hardware milestone from the strategic collaboration IBM and Arm unveiled in April, and it offers an unusually direct answer to a question that has shadowed the mainframe for years: can the machine that processes most of the world's regulated financial transactions remain a first-class citizen in an AI era built largely on other people's silicon?

How IBM engineered a processor core that speaks two instruction sets

The most striking engineering decision is what IBM chose not to do. The company could have bolted a handful of standalone Arm cores onto the side of its processor — a simpler design that other chipmakers have used for heterogeneous computing. Instead, IBM built every core on the chip to be bilingual.

"On this chip are 11 cores, and each core can dynamically switch back and forth between Arm software mode and traditional Z software mode," said Christian Jacobi, IBM Fellow and chief technology officer of IBM Systems Development, in an exclusive interview with VentureBeat. "That enables us to run the mission-critical enterprise software right next, on the same chip, to the much broader software ecosystem of Arm applications."

The mechanism relies on the open-source KVM hypervisor. Enterprises can run Arm64 Linux virtual machines and Linux on Z virtual machines side by side, and as the hypervisor dispatches each virtual machine onto a physical core, the core flips into the corresponding mode. The performance penalty, Jacobi said, is effectively zero. "That switch takes about the nanosecond scale," he said. "Because you're running for many milliseconds in the virtual image, this switching overhead sort of amortizes to zero — pretty much no impact at all."

Traditional z/OS workloads run in a separate partition on the same chip, outside KVM — meaning a bank's core ledger, its fraud models, and a modern Arm-native monitoring stack can all share the same silicon, the same memory fabric, and the same reliability guarantees. Jacobi was candid that IBM debated the easier path and rejected it. "We're really not addressing their need if we just have a few, I'd say, loosely Arm cores in the corner of the chip," he said. "It really needed to be deeply integrated into the entire system design for it to have the same qualities of service that clients are used to."

The specifications underscore that this is no compromise design. Built on a leading-edge 2-nanometer process node, the chip runs its 11 high-performance cores at a base frequency above 5.7 GHz — extraordinarily fast by industry standards — with on-chip AI inference accelerators for in-transaction fraud detection, a dedicated data processing unit for I/O acceleration, and a large cache architecture. Full systems will scale to hundreds of cores and tens of terabytes of memory. "That's really, really fast compared to what you otherwise get in the industry," Jacobi said. "It's just another example of how mainframe technology is not old technology. It's very modern, leading-edge technology."

Why the mainframe needed Arm's 22 million developers

The strategic logic behind the chip is about software, not hardware. IBM's s390x architecture runs an enormous share of the world's mission-critical transactions, but the broader universe of enterprise software — monitoring tools, security agents, cloud-native middleware, and above all the AI stack of PyTorch, ONNX Runtime, and container workloads — was built for x86 and, increasingly, for Arm. By Arm's own estimates, close to half of the compute shipped to major hyperscalers in 2025 was Arm-based, driven by AWS Graviton, Google Axion, and Microsoft's Arm silicon. Arm counts more than 22 million developers worldwide.

Porting each application to s390x has been a grinding, one-ISV-at-a-time effort, and Tina Tarquinio, chief product officer for IBM Z and LinuxONE, described the calculus bluntly. "No matter how great our ecosystem team is, we would never be able to work with all of them and port them all," she told VentureBeat. "There's a lot of ISVs out there, and so we wanted to make a fundamental, big step-function forward. We took a swing from a technology point of view."

Notably, she said customers weren't asking for a dual-architecture chip per se — they were asking for outcomes. "I wouldn't say our clients were saying, 'Can you please make me a dual-architecture environment?' But they were saying, 'Help me get these surround workloads, or different types of workloads, to run in a quicker-to-market fashion.'"

The compatibility promise is ambitious: Arm Linux binaries should run unmodified. "The new Arm capabilities are designed to be 100% binary compatible," Jacobi said. "Once you have, for example, Red Hat Linux for Arm, and you have applications that run on Red Hat Linux for Arm, they will run on the system without modifications." Arm defines the instruction set architecture and supplies validation tooling to guarantee that IBM's implementation behaves identically to every other Arm chip — while IBM designs and builds the silicon entirely in-house. "Very good partnership. Very solid engineering partnership as well," Jacobi said of the collaboration.

What a next-generation Spyre accelerator means for enterprise AI on the mainframe

IBM is also previewing the next generation of its Spyre AI accelerator at Hot Chips, and the pairing is not coincidental. The current architecture already offers two tiers of AI: an on-processor accelerator, introduced with the Telum chip in 2022, that handles ultra-low-latency inference such as fraud scoring inside a payment transaction, and the Spyre accelerator card sitting in the I/O subsystem for heavier models.

The new Spyre raises the ceiling considerably. "We're also bringing a much higher performance chip that is capable of running large language models for agentic workflows," Jacobi said — both AI-ops workflows that administer the system itself and business workflows "for things like document understanding and insurance adjudication." The new accelerator will ship with high-bandwidth memory to feed those models.

Here the dual-architecture bet and the AI bet converge. Enterprises want to run inference next to their data; the data lives on the mainframe; and the AI tooling is overwhelmingly Arm-native. Mohamed Awad, Arm's executive vice president for cloud AI, framed the announcement in exactly those terms: "As AI scales, more of the computing landscape is converging on Arm. Bringing Arm compute and its software ecosystem to these platforms will extend that momentum into mission-critical enterprise infrastructure to give organizations greater choice in how they deploy AI."

The timing tracks with where enterprise AI actually stands. McKinsey's most recent State of AI survey found that while 88% of organizations now use AI in at least one business function, nearly two-thirds have not yet scaled it across the enterprise — and the companies capturing the most value are those redesigning core workflows rather than running detached pilots. For regulated industries whose systems of record sit on IBM Z, running AI where the transactions happen is arguably the most direct route to that kind of integration.

When the dual-architecture IBM Z system will ship — and why existing customers shouldn't worry

Buyers will need patience. The chip will debut in the successor to the z17, which shipped in the second quarter of 2025, and IBM holds to a roughly three-year product cadence — pointing to a launch around 2028. But Tarquinio insisted the program is well past the concept stage. "It's more than being on the drawing board. We're full steam ahead on the whole system," she said, adding that IBM will release more details in the run-up to launch.

For IBM's installed base, the reflexive question is whether embracing Arm signals a slow sunset for the traditional architecture. Both executives pushed back hard. "This is a big and. It is not an or," Tarquinio said. "I have a roadmap that goes out 10 or 15 years of hardware systems. Many of our teams are working on this next system; many are also working on the one after that, and the one after that."

Jacobi cast the move as continuity rather than rupture. "The traditional mainframe that we have today as a z17 system is not just a faster version of what we built 25 years ago," he said. "We didn't have pervasive encryption capabilities. We didn't have on-processor AI capabilities. Adding the Arm capability is the next big iteration in this continuous evolution."

The competitive subtext is the cloud. Asked why an enterprise would run Arm workloads on a mainframe instead of a hyperscaler, Tarquinio pointed to the platform's availability numbers: "We're talking eight nines of availability — that's 0.3 seconds of downtime a year. If you're running your ledger, if you're running your fraud detection, any of these mission-critical apps, you want that." The pitch, she said, is fit for purpose: match the infrastructure to the SLA, not the fashion.

There are real caveats. IBM's own press release notes that statements of future direction "represent goals and objectives only." The Arm support is Linux-only for now, and the hardest engineering — running a foreign instruction set at production performance, with mainframe-grade fault detection and recovery, under real customer workloads — remains to be proven over the next two years.

But the ambition is unmistakable. For sixty years, the mainframe has survived every wave of technology that was supposed to kill it — minicomputers, client-server, the cloud — by absorbing what it needed from each. Now IBM is attempting its boldest act of absorption yet: teaching the machine that runs the world's money to speak the language of the AI era, fluently and natively, on the same silicon. "Bringing something that'll really be first of its kind in production," Tarquinio said, "showcases again what IBM is capable of from a technology point of view." The mainframe, it turns out, isn't being left behind by the future. It's learning to run it.

Real-time AI at scale is harder than it looks. Pipelines that hum along in development routinely hit problems in production. It’s always easy to blame the model for all your problems. But issues like rising latency and degrading accuracy can usually be traced back to the data pipeline. 

My colleague Tim Koopmans and I recently discussed what typically goes wrong with real-time AI at scale. After Tim shared some hard-fought lessons learned, we talked about how to avoid falling into these traps yourself – including the practices and infrastructure choices that can help you avoid them. You can watch the full video or read the key points below. 

Why AI performance fails at scale

Tim learned the following real-time AI performance lessons the hard way: through fits of frustration while building an ML-based financial trading app.

You can’t dig yourself out of tail latency

All too often, latency looks fine in testing, then a P99 spike surfaces under real concurrent load. For example, as Tim’s app approached ~740K operations per second, its P99 latency skyrocketed to 3 seconds. 

“I kept blaming the model for being slow, but it turns out the model was fine,” Tim explained. “It was just that the feature lookups were killing me.” Each inference call was doing just a handful of reads, but those reads queued up [behind writes] under load. The average latencies seemed fine, but that P99 tail latency was just unacceptable. 

“Tail latency isn’t a bug that you can fix, it’s a property of your architecture.”

Once you hit highly concurrent write throughput, you get lock contention – and that impacts the tail latencies. At this point, retries and bigger caches and connection pool tuning don’t help. As Tim put it, “Tail latency isn’t a bug that you can fix, it’s a property of your architecture. For example, if your storage engine is producing GC pauses at exactly the wrong moment, you’re going to cop a latency spike, no matter what.”

The culprit in Tim’s app was actually Postgres under pressure: “It’s not a slow database, but it was just a database being asked to do too much in this particular case,” Tim continued. 

Stale features kill accuracy

If you notice a mysterious accuracy drop that the model itself can’t explain, feature freshness might be the problem.

For Tim, this issue was particularly frustrating. User profile (wallet addresses) staleness was blowing past a five-minute SLA target by hours, vector embeddings were going stale, and offline evaluation metrics looked fine the entire time. As Tim put it, “You have this maddening situation where offline evaluation metrics look great, but as soon as you mix it in with online data, that performance is rubbish.”

“Offline evaluation metrics look great, but as soon as you mix it in with online data, that performance is rubbish.”

When the model was in production, it started making calls that didn’t track. After spending what seemed like ages debugging the model, the model itself turned out to be fine. The problem was that the model was making decisions based on old data (garbage in, garbage out, essentially). 

Vectors indexes need maintenance

No matter what vector database vendors imply, “set it and forget it” isn’t a realistic strategy for embeddings. Every re-embedding pass rots the index a little more, whether you notice it happening or not.

Tim hit this too. He was re-embedding content every time he improved the model, and the index quality rotted a bit more with every pass. At one point, he noticed that the recall rate (the share of true best-matches an approximate search actually finds) dropped to a dismal 42% – and query latency ballooned at the same time. He explained, “HNSW graphs degrade as they take on mutations. The nasty thing is you don’t really realize that until you realize your results are tainted.”

He advised others to treat a vector index like you’d treat any other database index. It needs the same care, love and attention as anything else you operate. That means:

• Monitor recall accuracy (and results returned) 
• Plan for partial builds (or batch builds)
• Know that changing your similarity function, your search parameters, or your embedding model means starting the graph over from scratch.

You gotta keep ’em separated

Another problem is resource contention – for example, training and serving fighting over the same hardware. Tim had just one machine doing double duty. With everything running on the same infrastructure, GPU, RAM, and CPU were all competing for resources. Side note: Many people don’t realize that vector search is a CPU cost, not a memory cost, since you’re traversing a graph rather than just storing vectors. 

The fix is the same thing every distributed systems person already knows: You gotta keep ‘em separated. This is just good engineering principles: separate your write path from your read path, separate training from serving if you can afford it.”

Retraining is inevitable

Recognize that retraining isn’t optional, and it isn’t free. Every model swap requires transition time.

Tim explained that there’s a dodgy window where the old model is still serving stale predictions and the new one hasn’t warmed up yet. For the database, this could mean new access patterns, cache misses, cold reads or request queues building up. When you notice that data is drifting, or user behavior is changing, the model you trained three months ago is getting worse – that’s the sign that it’s time to retrain. 

It’s going to happen eventually, so plan for it. Tim’s own approach was blue-green deployments, canaries, running the old and new model in parallel under different names, and doing the actual cutover at the application layer rather than all at once. If you’re at, say, Tripadvisor scale – with 100 million ML models – you can imagine the process will be considerably more complex.

Avoiding the doom loop with a high-performance database

These problems tend to build on each other and snowball. Latency causes staleness, staleness degrades accuracy, degraded accuracy triggers retraining, retraining causes contention, and contention makes latency worse again. It can create what Tim deemed a “doom loop.”

Here are some tips for avoiding that doom loop. 

Monitor, monitor, monitor

Be obsessive about monitoring. Watch freshness and backlog in particular because a growing backlog is what eventually drives up tail latency. Also watch index health, since that’s where recall rots. And load test beyond steady state because you can’t really predict when some weird confluence of factors will cause usage to surge. 

Isolate your workloads

This addresses two of the problems from earlier: the write storms that caused tail latency, and training and serving sharing the same infrastructure.  A database that handles concurrent writes well and isolates workloads properly can absorb both.

For example, with ScyllaDB, the write path is lock-free and multi-writer. That means every node takes writes in an active-active fashion, and no row gets locked in the process. As a result, a burst of concurrent writes doesn’t back up into a queue the way it would on a database built around single-writer assumptions. 

On top of that, a practice we call “workload prioritization” controls how workloads compete for system resources. This ensures latency-sensitive queries are fast, even with other heavy workloads running on the same cluster. That way, a retraining job or a backfill won’t steal resources from whatever’s serving live inference. 

Separate vector indexing

To address the vector index problem, keep the index separate instead of bolting it onto the same process as the core database. For example, ScyllaDB Vector Search writes land in the core database first, and the index gets built out of that data asynchronously, as its own service. 

If the index can’t keep up with the write rate (whether from a re-embedding pass or a full rebuild after a similarity function change), it falls behind – but it never misses a write and the core database is not impacted. Even if the vector store goes down, the embeddings still persist in the core database. And because ANN queries are CPU-heavy, keeping them on a separate service means they’re not fighting the core database for the same CPU cycles the writes require.

Under billion vector benchmarks, that separated architecture held P99 latency under 10 milliseconds at a concurrency of 300. It handled ~150,000 ANN queries a second with a moderate recall target. Realize that higher recall will bring a latency and throughput tradeoff, and always test this in advance to assess how your own mileage varies. 

Absorb the shock

This one comes down to whether your infrastructure can absorb a sudden change in write pressure or traffic shape without a scramble. The storage engine’s architecture matters a lot here.

For example, ScyllaDB is built on an LSM-tree, which tolerates that kind of write pressure instead of degrading under it. Elastic scaling, with what we call “tablets,” can scale a cluster by something like 10x within minutes instead of hours. That means that if a model rollout changes your access patterns overnight, or you need to absorb a backfill before a big retrain, you don’t end up waiting on a multi-hour resharding job.

The more things change…

So much about AI is genuinely novel, but the infrastructure problems described above generally are not.

“Real-time AI is really a distributed systems problem in a costume.”

Tim mentioned that a feature store was probably the primitive use case: the same high write throughput, low latency work, years before anyone called it AI. Real-time AI is really a distributed systems problem in a costume. Once you understand that, you can design for it so you don’t get blindsided by these not-so-new challenges. 

The post Why real-time AI at scale is so hard appeared first on The New Stack.

On July 16, the team at Hugging Face noticed something weird moving through their production systems: An intruder that was cloning datasets, harvesting credentials, and hopping between internal clusters faster than a human could type. Five days into the investigation, the authorities were notified, and several days later, OpenAI traced the intruder back to one of its models.

“Two frontier models escaped their test environments this summer; only one of them had a funny reason.”

That prompted Anthropic to look through its own test logs, and it found that similar events had occurred three times. One Claude model probed nine thousand hosts in a single run without setting off any alarms. Another slipped a poisoned package into the public Python registry, compromising fifteen machines before it was caught. The third incident dated back to April and went undetected for three months until a rival’s error prompted the audit. Nobody had noticed. Two of the three companies involved didn’t know anything was amiss until they got a call.

Containment by instruction

A sandbox, in the context of AI testing, means a model gets a task, a set of tools, and an instruction: no access to the outside world.

OpenAI’s model found an exploit in a tool it was allowed to use, and leveraged it to access the open Internet. Anthropic’s models didn’t even need that. The models were told a path didn’t exist, so when their tasks led them down a path that wasn’t supposed to be there, they treated it as part of the exercise.

“In both instances, the only constraint was an instruction. There was no mechanism outside the model itself to ensure those instructions wouldn’t be treated as optional.”

In both instances, the only constraint was an instruction. There was no mechanism outside the model itself to ensure those instructions wouldn’t be treated as optional.

So how did they escape?

OpenAI’s agent was being evaluated on a cybersecurity benchmark called CyberGym. During the test, it chained a code-execution flaw in a dataset loader to a second flaw in the platform dataset configuration file handler. It used the combined exploit to run commands on a live processing worker. From there it grabbed a write-scoped access token and cloned a private dataset as a staging area, which became a dead drop for its own tools. By the time anyone noticed, the intrusion had been running for three days, touched several internal clusters, and compromised accounts at four other companies, according to Hugging Face’s own review.

What it wanted, in the end, was comical: the answer key to the test it was taking.

Anthropic’s three incidents were nowhere near as convoluted. Its models were being run through security exercises against third-party evaluation partners, and the only thing keeping them from accessing the Internet was an instruction not to. One incident involved an SQL injection targeting a resource the model wasn’t supposed to be able to access. Another, during an exercise where the usual safeguards were deliberately switched off, ended with a Claude model publishing a malicious package to the Python Package Index, and fifteen machines picking it up before it was pulled down. No zero-day exploit. No dramatic escape. Just locked doors that turned out to be mirages.

The structural weakness

Strip away the specifics, and both incidents share the same root cause: there was nothing outside the model’s own judgment that could enforce the limitations placed on it. A credential that shouldn’t have been able to access the open Internet did so anyway. An instruction that should’ve stopped a plan cold got treated as just another input to reason around.

“If a fence doesn’t hold during testing, it won’t hold in production.”

It feels like we’ve spent the last year reliving the same problem in new ways. For example, in deleted databases, in poisoned supply chains, in every credential an agent inherited because provisioning it correctly took one extra step nobody got around to. The only delta is where the boundary was supposed to be: around the dev experiment, meant to contain dangerous capabilities, or standing between it and every production system it could reach. If a fence doesn’t hold during testing, it won’t hold in production.

How do we break out of the loop?

Unfortunately, the answer isn’t self-improvement. A more capable model wouldn’t have caught any of this. The only way to fix it is with Infrastructure that verifies where a model actually is, rather than trusting its reports.

Egress controls that are closed by default. A test environment that can reach the open Internet is probably not a great idea. Outbound traffic should be locked to an explicit allowlist. Never assume a model will simply decline to look for a way out. “No internet access” needs to be a network-enforced fact, not just an instruction.

Narrowly scoped credentials with short lifespans. Both incidents ran, in part, on tokens that extended beyond what the tasks required. A credential built for one evaluation run, scoped to nothing beyond it, and expiring on a timer, can’t become the write-scoped key that turns into a lateral-movement problem across multiple clusters. Yes, it will be a chore; so gird your patience by imagining having to explain to company leadership why taking steps to avert a catastrophe was too tedious to bother with.

Policy gates that sit outside the agent’s reasoning. An agent that can decide for itself whether an action is in scope is grading its own test. A plan should have clear rules written in code, evaluated by something other than the agent, before anything executes. Open Policy Agent has become a de facto default here for exactly this reason. It doesn’t care how confident the plan sounds; nothing runs without proper authorization from an external source.

An audit trail that is entirely independent of the agent. Both incidents were discovered by rereading logs after the fact, not by asking the models what happened. That’s how oversight should be done. As the proverb goes: Quis custodiet ipsos custodes? Who watches the watchers? A system of record that captures what actually occurred, regardless of the agent’s own account of itself, is the only version of events worth trusting.

This is a sign

The industry spent a decade learning that the CI/CD pipeline is an attack vector that requires real fortification, not just bolted-on convenience. Test environments for frontier models are following the same arc at a faster pace. The next time one of these agents escapes containment, it’ll probably be one built around finding unlocked doors, which will make it substantially more dangerous than a coding agent that deletes a few databases.

Test rigs must be treated as if they hold something real, because, as far as the credentials are concerned, they do. A sign on a door is never going to be enough to keep everyone out; there has to be a lock whose robustness correlates to the value of what it guards.

“A sign on a door is never going to be enough to keep everyone out; there has to be a lock whose robustness correlates to the value of what it guards.”

Whether by a state-sponsored crew probing a water management system in the middle of the night, or a company’s own model trying to shave a few points off a benchmark, boundaries will always be tested. Two labs found out this summer, and the story needs to be taken seriously. The vulnerabilities are real, the transparency from the labs is welcome, and the containment failures are a cause for concern.

Catching a model that tried the handle is the easy part; both labs proved that. The more challenging, and therefore critical, part is making sure the next containment environment actually has doors that are firmly locked.

The post Securing sandboxes: What happens when AI agents escape containment? appeared first on The New Stack.

The artificial intelligence landscape has reached a pivotal inflection point. Over the past several years, the paradigm has shifted from passive, conversational Large Language Models (LLMs) to autonomous AI agents, digital software entities capable of reasoning, invoking tools, executing multi-step workflows, and making real-time decisions across enterprise systems without constant human intervention.

As organizations accelerate the production deployment of autonomous agents, modern security frameworks must evolve to keep pace. Traditional Identity and Access Management (IAM) systems were primarily designed around two distinct operational models:

• Human users: Authenticated via Multi-Factor Authentication (MFA), Single Sign-On (SSO), and interactive sessions.
• Service accounts and workloads: Authenticated via static API keys, fixed service tokens, or IP whitelisting.

Autonomous AI agents blur the line between these two models. An agent acts with the non-deterministic reasoning and delegated agency of a human, but operates at the scale, parallel velocity, and automation speed of a machine service.

To safely harness the power of autonomous workflows, enterprise security architecture must move toward continuous, agent-aware Zero Trust governance. Below are six foundational identity capabilities that organizations should adopt to secure AI agents in production environments effectively.

“Autonomous AI agents blur the line between these two models. An agent acts with the non-deterministic reasoning and delegated agency of a human, but operates at the scale, parallel velocity, and automation speed of a machine service.”

“When it comes to agentic AI identity, most organizations are woefully unprepared for inherent security risks and operational challenges of managing those identities.” – Ken Buckler, Research Director, EMA – Agentic AI Identities – Is Your Organization Prepared?

1. Verifiable agent identities & “Know Your Agent” (KYA)

Autonomous entities require verifiable digital identity frameworks that establish clear, cryptographically bound accountability for every machine action.

• Cryptographic attestation: Every agent instance should possess a unique, cryptographically signed identity bound to its underlying model version, execution environment, and deployment origin.
• Delegation chains: When a human user delegates a task to an agent (or when a primary agent spawns sub-agents), the identity system must construct an immutable, traceable chain of delegation. This ensures the infrastructure can continuously verify who authorized the initial action and what specific scope was granted.

2. Ephemeral credentials & just-in-time (JIT) tokenization

Static API keys and persistent service tokens represent a significant surface area of exposure when integrated into dynamic agentic workflows. Replacing long-lived credentials with short-lived tokens dramatically reduces the potential window of risk.

• Just-in-time (JIT) minting: AI agents should operate with ephemeral credentials generated on demand, strictly limited to the API calls required for a single operational step, and configured to expire within seconds or minutes.
• Bound OAuth flows & PKCE: Enforcing Proof Key for Code Exchange (PKCE) and strict token-binding protocols ensures that credentials cannot be reused or replayed outside of their intended runtime context.

“Replacing long-lived credentials with short-lived tokens dramatically reduces the potential window of risk.”

3. Relationship-based access control (ReBAC) & intent binding

Coarse-grained permissions, such as those in traditional Role-Based Access Control (RBAC), are often too broad for non-deterministic tool usage. Access governance should be based on fine-grained relationship models and task intent.

• Intent-bound authorization: Authorization systems should evaluate not only whether an agent has general permission to access a resource, but whether that request directly aligns with the explicitly authorized sub-task.
• Fine-grained contextual policies: Implementing relationship-based access control (ReBAC) or Attribute-Based Access Control (ABAC) allows teams to define precise conditions (e.g., “Agent X may read Document Y only if human user Z is the document owner and the active workflow is ‘Data Summarization'”).

4. Machine-speed containment & automated anomaly detection

Because AI agents operate at speeds far exceeding those of manual monitoring, security containment mechanisms must be automated, agent-aware, and built into the control plane.

• Behavioral rate & scope limits: Security controls should establish baselines for expected agent behavior to detect anomalies, such as rapid parallel tool invocations, repetitive execution loops, or unusual queries to non-standard endpoints.
• Automated circuit breakers: If an agent’s execution pattern or request velocity exceeds defined behavioral bounds, identity proxies can automatically revoke ephemeral tokens and safely isolate the workload in real time.

5. In-the-loop runtime enforcement & human approvals

Security governance cannot rely solely on static pre-authorization; policies must be evaluated continuously at runtime before individual actions execute.

• Action-level policy interception: Enforce real-time policy checks at the agent harness layer—evaluating shell commands, database queries, file operations, and outbound API calls against governance rules before execution.
• Configurable approval workflows: Establish flexible escalation paths that permit low-risk read operations automatically while requiring explicit human-in-the-loop validation for high-impact actions, such as code deployments or financial transactions.

6. Web-scale identity architecture built for machine workloads

Autonomous workflows generate significant operational volume. Identity systems must be architected to handle machine-scale throughput without performance degradation or store bloat.

• Machine-speed throughput: Multi-step workflows and parallel worker agents demand identity control planes that can handle high-volume token validation and policy evaluation with minimal latency.
• Lifecycle governance for sub-agents: Dynamically spawned sub-agents require rapid provisioning and immediate teardown upon task completion, thereby preventing the accumulation of orphaned credentials and ensuring clean session termination.
• Inline cryptographic safeguards: Prioritizing inline policy enforcement over post-mortem log reviews allows organizations to intercept unauthorized state changes before they occur, maintaining operational integrity across multi-cloud environments.

Conclusion: securing the future of enterprise automation

As AI models evolve from passive assistance tools to active operational participants, identity becomes the primary boundary for enterprise governance. By bridging the machine identity gap with verifiable agent identities, short-lived JIT credentials, fine-grained relationship authorization, and automated runtime enforcement, security leaders can confidently deploy autonomous AI agents to drive productivity while maintaining complete operational control.

The post Six identity capabilities for securing autonomous AI agents appeared first on The New Stack.

Anthropic launched a new Browser Use tool that gives Claude a structured view of a web page in addition to what is visually rendered. Announced Thursday, the tool uses the page’s accessibility tree to help Claude find and interact with specific elements directly rather than having to work out where they are on the screen.

Browser Use is part of a broader Anthropic release that also brings Computer Use, the Skills API and Files API into general availability. Developers can access the browser tool through the Claude API using browser_toolset_20260801.

Browser Use is part of a broader Anthropic release that also brings Computer Use, the Skills API and Files API into general availability.

The change gives Claude a more direct way to interact with a web page. Instead of working out a button’s position from a viewport image and targeting coordinates such as x: 640, y: 320, Claude can receive a reference such as ref_3 tied to that element and use it when it wants to act.

Page references replace coordinates

Computer Use can operate across an entire desktop by looking at screenshots and sending mouse coordinates and keyboard commands. Browser Use works within the browser itself, where it can use page structure that would be difficult to recover reliably from pixels alone.

When Claude calls read_page, the developer’s executor returns a text representation of the accessibility tree, in which elements such as links, buttons, and text boxes can be tagged with references. If Claude later wants to click a button represented by ref_3, it can send that reference along with the requested operation rather than trying to calculate where the button is on the screen.

That said, if the tab navigates to a new page or the page changes enough, a reference that pointed to a button a moment ago may no longer work. The API will not catch that on its own, so the executor has to recognize when the reference no longer matches the underlying element, reject the action and have Claude read the page again before continuing.

Batching cuts model calls

Playwright, for example, can represent a page as an ARIA snapshot and locate elements by role rather than coordinates. At the same time, Microsoft’s Playwright MCP server already exposes structured accessibility snapshots with references a model can use to identify elements. The concepts line up closely with Browser Use, but the protocols do not: Playwright MCP speaks MCP, while Anthropic’s tool uses its own client-toolset protocol, so developers would still need an adapter that translates Claude’s requests into Playwright actions and returns the results in the format Claude expects.

Puppeteer offers many of the same building blocks, exposing the browser’s accessibility tree via Accessibility.snapshot() and providing APIs for controlling Chrome and Firefox. A developer could use those APIs for navigation or page reads, then maintain Anthropic’s reference mappings on top.

A developer could use those APIs for navigation or page reads, then maintain Anthropic’s reference mappings on top.

Slightly confusing, an unrelated open-source project also called Browser Use runs AI browser agents against Chromium through the Chrome DevTools Protocol. Despite the shared name, it has no connection to Anthropic’s tool and comes with its own agent loop and browser abstractions, so connecting the two would still require integration work.

Several browser actions can happen in one turn

Anthropic is also reducing the back-and-forth between Claude and the browser by allowing multiple actions to be requested in a single model turn. Now actions can arrive together as several tool_use blocks. The application executes them in order and sends the results back together, avoiding another model call between every click and keystroke. Anthropic says that can lower latency and costs, particularly as workflows scale from a handful of interactions to dozens or hundreds.

That matters more as browser tasks get longer. Cheaper models alone will not solve the token cost problem in agentic workflows, so cutting unnecessary model calls is another way to reduce costs.

If Claude has to return to the model after every click or keystroke, a long browser task can quickly rack up model calls. Batching cuts out some of that back-and-forth by letting Claude request several actions at once, but the browser still has to carry them out in order because each one depends on what happened before it. If Claude asks to click a button, fill in a field, and submit a form, for example, the executor cannot simply move on to the next step if that first click fails, because everything that follows is now based on a page state Claude never reached.

Batching cuts out some of that back-and-forth by letting Claude request several actions at once, but the browser still has to carry them out in order because each one depends on what happened before it.

Developers host the browser

Browser Use is currently limited to the Claude API and is not available inside Claude Managed Agents. Adding it to a Messages API request exposes 27 browser operations by default. Claude can decide which of those operations it wants to use, but Anthropic does not execute them. The application has to translate each request into an action inside its own browser environment, preserve the session between turns and return enough information for Claude to understand what happened.

Loading all of those operations has a token cost. Anthropic’s pricing documentation says the default Browser Use toolset adds roughly 6,600 input tokens to a request, before counting screenshots, accessibility trees and other results sent back to Claude. Developers can turn off operations they do not need to reduce that overhead.

It also creates a different hosting split from some of the other tools Anthropic announced Thursday. Skills uploaded through the Skills API can run inside Anthropic’s code execution sandbox, while the Files API stores documents that can be reused by ID. Browser sessions, along with their downloads and uploaded files, stay in the developer’s environment.

Approval gates need rethinking

Claude can still encounter a prompt injection in web content or be redirected to an unexpected location, which is why Anthropic recommends running the browser in an isolated container or virtual machine with minimal access. JavaScript and file uploads should remain disabled unless needed, since code generated by Claude runs with the page’s privileges and can reach data or make requests available to that page.

Batching makes approval a little trickier because several actions can arrive at once, and a routine click at the beginning of a sequence could eventually lead to something that requires the user’s permission. That means the executor has to check actions as they happen and stop for approval when needed.

The post Anthropic’s new browser tool doesn’t actually run a browser appeared first on The New Stack.

I’m Matt Burns, Chief Content Officer at Insight Media Group. Each week, I round up the most important AI developments, explaining what they mean for people and organizations putting this technology to work. The thesis is simple: workers who learn to use AI will define the next era of their industries, and this newsletter is here to help you be one of them.

Model triage is becoming one of the most important skills for the AI-native developer. I’ve argued all summer that the people getting the most out of frontier models are the ones disciplined enough not to run the best model by default. On Wednesday, Stripe and Ramp validated that idea 70 minutes apart: Stripe bought OpenRouter, and Ramp released its internal router.

Bloomberg puts the OpenRouter price tag above $7 billion, and Axios says it’s more than $8 billion in cash and stock. Stripe has not released the terms, so the details remain fuzzy.

While the acquisition made headlines, the architecture is the story. For the last couple of years, picking a model was something written into an application, a string in a config file, and swapping models took some work. A router changes that workflow. Stripe bought the layer, and Ramp built it. Both are betting their existing relationships give them a unique wedge to own this critical layer in the new AI stack.

Picking a model is becoming a runtime decision

OpenRouter is an endpoint serving more than 400 models from over 80 providers, processing more than 10 trillion tokens a day. Andrej Karpathy calls it the transfer switch for AI. Ramp’s Router does the same job on a smaller catalog and claims roughly 40% lower cost for the same output.

Both are betting the model name in a codebase is a liability. They’re mostly right. Back in June, I pointed to Mitchell Hashimoto, who found a standard coding task that cost about $1.50 on GPT-5.5 and roughly $9 on Claude Fable, with both producing equally acceptable results. A router automates that triage, making decisions on every request rather than only on those a developer explicitly configures.

This is becoming a large problem and a large opportunity. Our own Amanda Caswell reported this week that Anthropic’s /claude-api skill was burning about 200,000 tokens before answering a single question, and that loading its reference docs on demand instead of up front cut that to roughly 25,000. Hafiz Hassan wrote for us last week about why AI pipelines cost 10x more than the demo, and every culprit on his list is an engineering decision: system prompts resent every turn, whole conversation histories appended, oversized RAG chunks, raw JSON dumped into context. It’s a great practical guide, and none of the items Hassan identifies are procurement problems.

The token bill is generated by your code, which is why the tools to control it are arriving there as well.

Stripe and Ramp want the same layer for opposite reasons

Stripe is attacking the problem from the bottom up, through developers. The company’s investor letter, leaked Wednesday by Eric Newcomer, makes the argument directly: “Up until now, every developer has needed a straightforward and reliable way to manage their revenue pipeline, and serving this need gave rise to Stripe. Going forward, however, every developer will also need a straightforward and reliable way to manage their intelligence pipeline.”

Stripe wants to control AI spending through the long tail of developers. Ramp wants to control it through its existing relationship with finance.

Stripe has built this product before. Its payments business hides dozens of local payment methods behind a single API, routing each transaction to the payment method most likely to convert. The AI version is the same idea applied to models instead of payment networks.

Ramp is attacking it from the top down, through finance. The company bought the router.com domain and says its customers already buy quadrillions of tokens a month through Ramp. Founder Veeral Patel’s launch post pitches the service simply: “Monitor and control your AI bill across every provider.” Adam Wazzan sums up Ramp’s strategy better than I can: “when a CFO ships a product for CTOs.”

Stripe wants to control AI spending through the long tail of developers. Ramp wants to control it through its existing relationship with finance. Both are chasing what is rapidly becoming one of the largest line items in corporate technology budgets: tokens.

On X, Kabir Goel pushes back on Stripe’s framing. Routing tokens is a way to spend less, while Stripe’s other products are designed to help businesses make more. 

“Stripe is just not where teams go to understand how much they’re spending,” he writes. “That’s pretty squarely Ramp territory.” 

He has a point about where teams look today. Whether that’s still true three years from now is exactly what Stripe just spent billions betting against.

The router worth pointing at is the one with no model to sell

Whichever router you point at decides which model writes your code, and not every router is disinterested. Our own Paul Sawers flagged the problem in July when he covered the first wave of Cursor’s, Ramp’s, and Meta’s routers. Cursor backs Grok and Composer. Meta is building Muse Spark. Both have reasons to send work to their own models, and Paul quoted developer Elvis Saravia asking whether routing logic ought to be open source rather than a vendor’s private judgment call.

Stripe and Ramp do not sell models. OpenRouter CEO Alex Atallah says as much in Stripe’s own announcement: Developers “need a neutral layer to orchestrate and manage them all.” Investor Gavin Baker frames the opportunity the same way, arguing that Stripe can become the neutral infrastructure layer for AI, just as it became the neutral infrastructure layer for payments. 

Neutrality isn’t free, though. Stripe takes a percentage of token spend, and Ramp wants your spending relationship, so “free through 2026” is a customer acquisition strategy with an expiration date.

The obvious objection is that vendor motives are the wrong thing to worry about, and routing quality is what really matters. That’s fair, and Towards Data Science published one of the best practical examples I’ve read. Pratik Rupareliya describes a routing layer that cut a support agent’s inference bill by 40% but also broke the product. A classifier sent “simple” queries to a cheaper model, but some of those “simple” queries were actually fraud investigations. 

The cheaper model answered them confidently and incorrectly. Customers stopped using the agent, churn rose above baseline in month four, and retention costs were four to five times higher than the savings. It took three months to surface and another month to identify the cause. His fix was per-tier quality monitoring combined with an uncertainty-routed cascade, which ultimately settled at 35% savings without sacrificing quality. It’s an excellent article to read before diving into model routing.

So instrument the routing. Log what the router picks on every request and break out your quality metrics by the model that served them. Ramp Router reportedly records the model, provider, tier, tokens, latency, cost, and fallback attempts for every call. Stripe OpenRouter rankings have been a public version of that telemetry for years. You can get similar visibility with either approach.

Right now, the model is becoming an implementation detail. The competition is shifting to the layer that decides which model gets the job. Stripe and Ramp are betting that developers won’t care what sits behind the endpoint, so long as the bill is lower and the results are good enough. 

The post Forget the model wars, Stripe and Ramp just started the router wars appeared first on The New Stack.

On Tuesday, Warp introduced Warp Factories, open infrastructure for building cloud software factories, agentic systems that automate work across the software development lifecycle, which have been popping up in different forms from companies like Augment Code and Chainguard. 

Warp, an agent development platform, calls Warp Factories “the building blocks” for developers to create their own scalable factories. It’s pitching the infrastructure as the solution for two problems founder and CEO Zach Lloyd says are frequent engineering complaints: 1) measuring and improving coding agent ROI; 2) governance and control.

The aim is to tackle both problems by making sure “the annoying bits [are] taken care of” so developers can focus purely on optimizing factories for specific products. 

As Lloyd writes in a blog post, he “predicts software factories will be as ubiquitous as CI/CD in the next few years.” Experts tell The New Stack they see software factories gaining traction, but they’re more cautious about the timeline.

“I think the software factory is inevitable,” Lee Faus, founder and CEO, Atomic Software and former global field CTO, GitLab, tells The New Stack. “But before software factories become as foundational as CI/CD, the industry needs to solve a deeper infrastructure problem.”

Specifically, he calls out the importance of tracing agent work: “We’re spending a lot of time talking about how to build the software factory,” Faus continues. “I think we’re going to spend much more time asking what becomes the system of record for the factory.”

Build the factory without building all the infrastructure 

Lloyd acknowledges that many organizations already have engineering teams at work building cloud software factories — but he argues that’s too big to be an inside job. 

Warp Factories, thus, emerges as the infrastructure on which developers can build their own factories, providing the core components to speed development without making organizations sacrifice flexibility, programmability, customization, or ownership.

When asked about Lloyd’s take on building infrastructure, Erik Gfesser, long-time engineer, tells The New Stack he agrees it doesn’t make sense for most organizations to tackle it in house.

As Lloyd writes, Warp’s new infrastructure is “built to increase coding agent ROI over time” with evals and benchmarks to measure effectiveness and built-in self-improvement and memory. Developers get queryable metrics on agent throughput, cost, quality, and ROI, visible via the Factory control room, API, and Factory MCP. Scorers evaluate how work items move through the factory with an eye on things like token spend, code quality, and whether or not the work introduced defects. 

From there, those scores power self-improvement loops and benchmarks. “Observer” agents score select agent runs and then search for ways to make improvements by adjusting variables like the harness, model, or context before making PRs to improve underlying factory functionality. Benchmarks, meanwhile, let developers score tasks across different models and harness configurations to compare performance.

Governance gets easier, but there’s more to solve

Per Warp, the infrastructure includes features to address governance and control, alongside factory definitions as version-controlled code, definitions for distinct agents, plus skills, MCPs, and permissions. 

Looking more broadly, Faus tells The New Stack software factory governance will require more than just controlling how agents operate, though:

“A software factory without a record of change risks becoming a very efficient way to manufacture code that nobody can fully explain.”

“Shared infrastructure can make permissions, model access, tool use, MCP connections, policies, cost, and execution environments easier to manage centrally. That’s valuable,” he says. “But governance isn’t just being able to control what an agent is allowed to do. It is being able to prove what it actually did.”

As software factories help speed up code generation, he says the harder problem becomes understanding the scores of interconnected decisions both humans and agents make across the development cycle. 

For example, if one agent triages an issue, another researches it, a third implements it, and still others review and verify it, how can an engineer reconstruct why that change was made six months later? “That record has to remain connected to the change itself,” says Faus. “[Otherwise,] a software factory without a record of change risks becoming a very efficient way to manufacture code that nobody can fully explain.”

Software factories are probably the future, but it will be a slow roll-out

Though Warp’s founder is gung-ho about the rapid rise of software factories, other experts are less certain. Like Faus, Gfesser expects software factory adoption to take time: 

“My expectation is that software factory adoption will likely be fragmented across multiple vendors similarly to the early stages of CI/CD.”

“As an early adopter of CI/CD myself, I know that CI/CD didn’t catch on the way it did until quality open source products were made available for widespread usage.”

He points out that while the Warp client is open source, the server, the Warp Drive backend, and OZ (Warp’s agent orchestration layer) are proprietary. Also worth noting: OpenAI is named as the founding sponsor of Warp’s open source repository. 

“As such, my expectation is that software factory adoption will likely be fragmented across multiple vendors similarly to the early stages of CI/CD,” he says. 

The post Warp wants to make it easier to build your software factory appeared first on The New Stack.

After weeks of speculation, fintech giant Stripe has confirmed that it’s tabled a bid for AI model gateway platform OpenRouter, a deal designed to help businesses optimize how they route and spend AI tokens.

While terms of the deal have not been disclosed, independent reports peg the acquisition price at a cool $8 billion, making it Stripe’s largest known acquisition to date.

To a casual observer, the deal marks a somewhat odd combination: why would a payments processor want to own technology that decides which AI model answers a given prompt? Well, it all ultimately comes down to “tokenomics” — the emerging discipline of managing the cost, allocation and consumption of AI tokens.

On top of that, OpenRouter has previously said that people should think of it as “like Stripe for LLMs,” owing to the fact that it makes the fragmented AI model market accessible through a single developer-friendly API, much as Stripe did for payments. And that synergy will now culminate in the two companies becoming one.

Token gesture: ‘making good use of scarce compute resources’

Stripe became a $159 billion juggernaut as the developer plumbing behind online payments — the infrastructure that lets internet businesses accept money, run subscriptions, and get paid globally. While its core pitch has always been about making it easy for businesses to accept money, AI has become one of the biggest costs those same businesses have to manage, and managing both sides of that ledger is part of Stripe’s job.

Stripe has been building out AI billing infrastructure long before the OpenRouter deal, previewing LLM token billing and an LLM proxy for routing and metering model calls in 2025. With OpenRouter under its wing, Stripe gains a much more sophisticated routing layer that can choose between hundreds of models and providers based on cost, speed and performance.

In its announcement on Wednesday, Stripe co-founder and CEO Patrick Collison says that “tokens are the central currency for companies building with AI,” adding that the acquisition is ultimately all about the economics of AI.

“Tokens are the central currency for companies building with AI, and it’s clear that the real-world economic potential will depend on making good use of scarce compute resources.”

“The real-world economic potential will depend on making good use of scarce compute resources,” he notes. “Stripe is building the economic infrastructure for AI, and together with OpenRouter we’ll help businesses maximize profitability by routing their requests intelligently and spending their tokens efficiently.”

Open sesame

OpenRouter itself is a relative newcomer to the technology world. Started in early 2023, and co-founded by former OpenSea CTO Alex Atallah, the platform acts as a single front door to the increasingly crowded AI model market. Developers can use one API to access and switch between hundreds of models from dozens of providers, without having to rewrite their applications every time they change models.

Underneath that common interface, OpenRouter handles much of the messy stuff: routing requests between providers, automatically falling back when one goes down, and optimizing for things such as price, latency and model quality. It generally passes through providers’ inference prices without a markup, instead making money through a 5.5% fee on credits purchased through the platform.

That proposition has helped it gain sizeable traction. OpenRouter now says it serves more than 10 million developers and companies across more than 400 models, processing over 10 trillion tokens per day.

The company is also fresh off the back of a $113 million funding round, led by Alphabet’s growth fund, with participation from a slew of high-profile backers including the venture arms of Nvidia, Databricks, Snowflake, MongoDB, and ServiceNow — a strategic bet by some of the biggest names in AI and enterprise software.

“AI has become the single largest driver of economic growth in the US, and inference is quickly becoming the largest line item for every company.”

In its own announcement post, penned by founders Alex Atallah, Chris Clark, and Louis Vichy, OpenRouter positions the deal against a bigger shift in where businesses are spending their money: away from simply building AI products and toward the ongoing cost of running them.

“AI has become the single largest driver of economic growth in the US, and inference is quickly becoming the largest line item for every company,” they write.

As for why Stripe, OpenRouter points to a shared developer-first heritage. Stripe’s APIs became something of a benchmark for developer software, while its payments infrastructure gives it experience handling huge volumes of transactions, fraud and abuse — problems OpenRouter increasingly faces as AI usage grows.

The company also suggests that remaining independent was a perfectly viable option, and that very few potential buyers could have persuaded it otherwise.

“There are few companies on earth we would have considered selling to; our mission, our neutrality, and our lead in the market make the story for independence strong,” they write. “We would only join a company if we thought we could do more together, faster, without compromising any of them.”

For customers, OpenRouter’s message is essentially business as usual. Stripe will own the company once the deal closes, but OpenRouter says its brand, product, roadmap and model-neutral approach will remain as is.

“There are few companies on earth we would have considered selling to; our mission, our neutrality, and our lead in the market make the story for independence strong.”

That continuity will likely matter, too, because OpenRouter is far from alone in trying to solve the problem. A slew of companies this year have been investing in their own routing layers, as AI inference costs climb and no single model stays the best or cheapest option for all that long.

The model-routing rush

Cursor, the AI coding tool now owned by Elon Musk’s SpaceX, shipped its own Router back in July, claiming savings of 30-50% compared with routing every request through its priciest model.

Ramp, the $44 billion spend-management company, also debuted its very own model router in July, a product that launched on Wednesday at its own dedicated Router.com domain — the same day Stripe announced its deal with OpenRouter. The company says three years of tuning its own AI spend internally cut its bill by 30% — the pitch to new users now promises a bigger number, an average 40% cut.

Meta, for its part, is also reportedly building a model router of its own. The Information reported in July that it’s planning Switchboard — a project out of an internal incubator called AAI Labs, that scores each request for difficulty and routes the easy ones to cheaper models. It’ll stay internal at first, aimed at cutting Meta’s own AI agent bill, but could eventually ship as an external product too.

All this activity speaks to a much broader reckoning over the cost of AI. In June, the Linux Foundation announced the Tokenomics Foundation, backed by the likes of Google, Microsoft, IBM and Salesforce, to develop common standards and benchmarks around how AI tokens are produced, consumed and monetized.

Model routers are one practical answer to the broader underlying problem: spend less by being smarter about which model gets each job. And with OpenRouter now set to become part of Stripe, those economics are moving directly into the payments giant’s wheelhouse.

The post OpenRouter called itself the “Stripe for LLMs” — now Stripe’s swooped in to buy it appeared first on The New Stack.

Serval is making Catalyst, its AI agent for building enterprise automations, generally available Thursday and enabling it by default for customers — allowing teams of AI agents to decide what should be automated and then build the automation itself.

Catalyst sits above Serval’s AI-native service management platform as an admin-facing “super agent.” It can inspect ticket history, standard operating procedures or natural-language instructions, identify recurring work, and draft the workflows, skills, forms, access policies, journeys and dashboards needed to automate it.

Serval is also using Catalyst to create background agents that continuously inspect connected systems for emerging problems and propose fixes before an employee files a ticket.

That distinction matters because enterprise service management vendors are rapidly converging on AI-assisted workflow creation.

ServiceNow’s Build Agent can already translate natural-language instructions into full-stack applications, flows, scripts and other platform metadata, while its AI Agent Advisor can analyze instance records to identify automation opportunities. Atlassian’s Rovo can generate Jira automation flows from plain-English requirements, and Freshworks offers Freddy AI Agent Studio for creating service agents that act across Freshservice workflows.

So Serval’s claim to differentiation is narrower — and potentially more consequential — than simply “we use AI to build workflows.” Catalyst is designed as a single administrative layer that can move from discovering an opportunity, to assembling multiple kinds of governed automation, to creating proactive agents that keep looking for new work to automate.

"You just started with a single prompt, and now you’ve got enterprise-grade workflows ready to deploy that are going to solve all password resets for the entire company," Serval co-founder and CEO Jake Stauch told VentureBeat in an interview.

From ticket history to working automation

Serval says Catalyst analyzes existing help desk data before an organization has decided what to automate. If it finds a repetitive category of requests, it can draft the automation required to resolve those requests and stage the result for administrator review. Users can also upload an SOP or spreadsheet and ask Catalyst to turn the documented process into an executable system.

Serval’s documentation says Catalyst can build workflows, author help desk skills, create onboarding and offboarding journeys, configure access-management policies, construct dashboards, investigate operational issues and debug failed workflow runs. Unlike Serval’s earlier workflow builder, Catalyst is intended to become the primary interface for configuring the platform; the company says its long-term goal is that anything an administrator can do through the UI should also be possible through Catalyst.

The actual workflows are code-backed. In a demonstration, Stauch showed Catalyst taking a request to build password-reset workflows, detecting connected systems including Okta, Google Workspace and Microsoft Entra, and generating the underlying TypeScript needed to perform those actions. Administrators could then add approvals or restrict who was allowed to run the workflow.

The models underneath Catalyst are deliberately swappable

Serval is not building its own foundation model. Stauch said in the interview that the company uses models from “frontier labs,” runs evaluations to determine which models work best for particular jobs, and is deliberately model-agnostic. “You can swap different models in,” he said, adding that Serval also works with enterprises that build their own models.

Stauch provided more detail in a May 2026 interview with Sequoia Capital, saying Serval was using both OpenAI and Anthropic models. He said OpenAI’s GPT models had performed best for end-user interactions and tool calling, while Anthropic’s Sonnet and Opus models were producing the strongest results for the code-generation side of Serval’s automation system — the workload most directly relevant to Catalyst. Serval continuously runs evals rather than automatically moving every workload to the newest model release, Stauch said.

That architecture makes the underlying LLM less central to Serval’s differentiation. The company’s own documentation now lets organization administrators supply their own OpenAI or Anthropic API keys, including a compatible custom endpoint, while Stauch said the broader architecture can accommodate different models.

The materials do not, however, establish that every Catalyst user gets a self-service menu for arbitrarily choosing an individual model. Serval’s pitch is instead that its proprietary value sits in the harness around those models: enterprise context and memory, integrations, generated code, permissions, approvals and the controls governing what an agent can actually do.

That code-generation model is central to Serval’s pitch against ServiceNow. Stauch argues that legacy ITSM deployments often accumulate custom tables, business rules, workflows and platform-specific expertise that make seemingly simple automation changes expensive to implement. Serval, by contrast, wants administrators and business teams to describe the outcome they need and let the model generate the implementation.

But ServiceNow is no longer standing still on that front. Its current Build Agent similarly creates applications and code from natural-language prompts, supports flow design and testing, and operates inside ServiceNow’s governance framework. ServiceNow’s AI Agent Studio lets customers create agents and agentic workflows, while AI Agent Advisor is explicitly designed to analyze operational records for automation candidates.

The competitive question is therefore shifting from “who has generative AI?” to how many separate tools, configuration concepts and specialists are required to get from an observed operational problem to a production automation.

Serval is effectively arguing that Catalyst compresses those steps into one conversational surface and a smaller platform model. ServiceNow, by comparison, now has a powerful but broader set of AI and development surfaces spanning Build Agent, AI Agent Studio, AI Agent Advisor, Workflow Studio and AI Control Tower. That breadth is an advantage for customers already deeply invested in ServiceNow, but it also illustrates the complexity Serval is attacking. ServiceNow itself notes that Build Agent is aimed at admins and developers who understand and can support what it generates.

Atlassian is moving in the same direction from a different starting point. Rovo can generate “if this happens, then that happens” automation flows from natural-language descriptions, while Jira Service Management increasingly supports agents that triage, investigate and execute service work.

Freshworks’ Freddy AI Agent Studio likewise emphasizes agents that resolve requests end-to-end, with prebuilt IT and HR agents and more than 30 workflow templates.

Catalyst’s differentiator, then, is not that rivals cannot generate an automation from a sentence. It is Serval’s attempt to make the entire automation lifecycle itself agentic.

Building agents that look for trouble before a ticket exists

That approach becomes clearest with Serval’s background agents.

Rather than waiting for a help desk request, a background agent can run on a schedule across connected systems, correlate signals and draft a remediation. In one customer example provided by Serval, an agent correlated network incidents across two offices using switch telemetry, DHCP data and historical tickets, ruled out hardware and wireless interference, traced the issue to configuration drift, and generated a remediation workflow for an administrator to approve.

“Most AI agents today wait for an employee to ask a question or submit a ticket,” Stauch said. “We believe the future is AI that acts before an employee ever submits a request.”

That framing also highlights a philosophical difference in Serval’s pitch. The startup does not want service management to revolve around creating, routing and tracking better tickets. It wants the system to eliminate as many requests as possible by turning repeated support work into executable automation.

"A lot of the code written in enterprises has nothing to do with software engineering," Stauch explained. "It’s actually internal automations and other scripts for the company, and so we use that technology to build a better service management platform."

Serval's pitch to enterprises is that it can largely automate those scripts. And the governance model is critical because Catalyst can generate code and potentially initiate changes across production systems. Serval says Catalyst inherits the permissions of the user operating it and remains scoped to that user’s team workspace.

Everything it builds starts as a draft, and organizations can restrict publishing privileges or require formal review and approval before an automation becomes active.

Customer data remains customer-owned, with several deployment options

Those controls also extend to the enterprise data Catalyst examines. Stauch said Serval is intended to operate as the customer’s system of record and told VentureBeat that “they own all the data.”

Serval’s current Master Services Agreement is more precise: customers retain rights, title and interest in both their “Customer Materials” — a category that includes records, documents, workflows, prompts, inputs and configurations — and the output Serval generates from them. Serval receives the rights necessary to process that information to provide, maintain, support and secure the service.

Serval also says it does not retain or use customer materials, inputs or outputs to train, fine-tune or improve its own or third-party AI models.

Its Data Processing Addendum identifies Serval as the processor of customer personal data and allows processing for operating the service, responding to support requests, diagnosing issues and protecting the platform, while authorized subprocessors can also be involved. Serval’s acceptable-use terms say it maintains a current list of AI subprocessors and model providers for customers.

Where that data resides can vary by deployment. Stauch said customers can use Serval as a cloud SaaS service, run it on-premises or place it in their own VPC. Serval’s self-hosting documentation now describes two fuller options: a Serval-managed single-tenant deployment inside an AWS account owned by the customer, or a self-managed deployment on the customer’s Kubernetes cluster in any cloud or on-premises environment.

In the AWS option, Serval says it operates the installation without persistent IAM access to the customer’s AWS account.

There are therefore two distinct access boundaries for enterprise buyers to consider.

1. At the Catalyst level, the agent can only reach data, integrations and automations available to the user and team workspace under which it is operating.

2. At the platform level, Serval and authorized subprocessors necessarily process customer information to deliver and support the service, subject to the company’s contractual confidentiality and data-processing terms.

That makes Stauch’s informal statement that Serval “doesn’t touch” customer data better understood as an ownership and deployment claim, rather than a literal assertion that the service never processes it.

Ramp and other customers provide an early test

Customer deployments provide some evidence that the faster-build thesis can translate into operational changes, although the metrics come from Serval’s own case studies.

Corporate expense and financial technology firm Ramp says in a Serval case study that Catalyst has made workflow building 50% faster and helped extend Serval across roughly 10 teams, including IT, finance, facilities, people and talent, legal and business operations. In one hardware replacement program, Serval says Ramp automated 600 laptop replacements and saved 150 hours, leaving approval as the principal human step.

The more telling Catalyst example may be what happened afterward. Ramp had already automated laptop replacement when Catalyst suggested splitting its shipping logic into separate office and home workflows to reduce errors. The company also says employees outside IT now use Catalyst for analytics, bulk ticket operations, workflow troubleshooting and HR process automation.

Other Serval deployments show the broader operating environment Catalyst is meant to configure. Mercor says it has onboarded more than 4,000 external experts through Serval automations and expanded the platform across seven teams. Together AI says Serval automates 95% of its just-in-time infrastructure access requests, with approval and auditing controls around sensitive access. Perplexity says Serval automatically handles more than half of its incoming IT requests and all employee onboarding.

Those deployments extend beyond Catalyst itself, but they demonstrate the type of cross-system automation substrate Catalyst is now being asked to build and maintain.

Serval says more than 90% of customers adopted Catalyst as their starting point for automation during beta. Catalyst is generally available Aug. 20 and will be enabled by default for all Serval organizations.

Pricing and the battle with ServiceNow

Pricing is customized depending on the size of the deployment and is not publicly listed on Serval's website or documentation.

Serval describes a single platform fee and typically runs a pilot to determine expected deployment and usage.

Stauch said the software license can be similar to ServiceNow’s, but argues total cost of ownership can be substantially lower because customers require fewer implementation and maintenance services.

"The total cost of ownership is going to be dramatically less — usually half as much, sometimes 10 to 20% of the total cost of ownership of ServiceNow," Stauch said. "But the actual software license fee is not necessarily going to be all that different."

Serval's origin story and history

Serval was founded in 2024 by Stauch and CTO Alex McLeod, former Verkada product and engineering leaders, after they repeatedly heard IT customers complain about overburdened help desks and the limitations of established IT service-management software.

Serval has positioned itself as an AI-native alternative to platforms such as ServiceNow and Jira Service Management, combining help-desk ticketing, access management, asset management and workflow automation within a single system.

Serval and Sequoia Capital describe the company’s goal as moving IT software beyond merely recording and routing requests toward resolving them automatically.

The company can operate as an organization’s primary IT service-management system or add automation to an existing one. Its publicly identified customers include Perplexity, Mercor, Clay, Verkada and Together AI.

Serval says customers can automatically resolve more than half of their incoming IT requests; its Together AI case study reports automation of 95% of that customer’s just-in-time access requests.

Investor interest accelerated rapidly in late 2025. Serval announced a $47 million Series A led by Redpoint Ventures in October, bringing its funding at that point to $52 million.

In December, it raised another $75 million in a Sequoia-led Series B at a $1 billion valuation, lifting total capital raised to approximately $127 million; Redpoint, Meritech Capital and General Catalyst also participated.

Serval told Reuters that revenue had grown 500% since August 2025 and that it was expanding beyond IT into operational work performed by human resources, finance and legal departments.

The big test for enterprise customers

For enterprise buyers, Catalyst’s biggest test will be whether its compression of the automation lifecycle survives contact with large, messy, highly customized environments.

ServiceNow can now generate applications and discover automation opportunities with AI. Atlassian and Freshworks are adding increasingly capable agentic automation to their own service platforms. Serval therefore cannot rely on natural-language creation alone as its moat.

Its stronger wager is that an AI-native platform can make the administrative layer itself agentic: continuously finding repetitive work, building the necessary resources across the service stack, exposing generated code for review, and proposing the next automation before an administrator has opened a workflow designer.

If Catalyst works at that scope, the competitive unit is no longer the ticket — or even the workflow. It is the system that keeps turning an enterprise’s operational history into new automation.

Every engineering team deploying AI agents eventually discovers an uncomfortable truth: the model isn’t the biggest expense. The hidden cost is everything around it: repeated retrievals, duplicate prompts, unnecessary tool calls, oversized context windows, multiple agents reasoning over the same information. Individually, these architectural decisions seem harmless. At production scale, they become a severe tax on latency, infrastructure, and cloud spend.

A proof-of-concept agent that answers 50 questions a day can tolerate inefficiencies. An enterprise platform coordinating thousands of requests per minute cannot.

This article explores practical techniques for engineering token-efficient AI systems without sacrificing output quality. Rather than focusing solely on prompt compression, we will optimize the entire workflow from routing and retrieval to caching and model selection.

Why token optimization is a systems problem

Most discussions around token optimization begin and end with prompt engineering. In practice, architecture drives token consumption.

Consider a typical multi-agent workflow:

User 
  ↓
Intent Agent
  ↓
Retriever
  ↓
Research Agent
  ↓
Planning Agent
  ↓
Writer Agent
  ↓
Reviewer Agent
  ↓
Final Response

At each stage, the system might retrieve the same documents, repeat identical instructions, call the same model, and resend the entire conversation history. By the time a response reaches the user, the architecture has processed tens of thousands of unnecessary tokens.

“Improving efficiency requires redesigning the workflow, not just shortening the prompts.”

Improving efficiency requires redesigning the workflow, not just shortening the prompts.

Architecture overview

A production-ready, token-efficient architecture introduces optimization before every expensive model invocation.

User Request
       │ 
       ▼
Intent Router 
       │ 
       ▼
Semantic Cache ───────► Cached Response
       │ 
       ▼
Context Budget Manager
       │ 
       ▼
Adaptive Retriever
       │ 
       ▼
Model Router
       │ 
       ▼
LLM
       │ 
       ▼
Validated Response

“The large language model is no longer the first component. It is the final, most expensive operation.”

Notice the critical shift: the large language model is no longer the first component. It is the final, most expensive operation.

Step 1: Install modern dependencies

Use the latest package structure to avoid deprecated imports and align with the current LangChain ecosystem.

Bash
pip install \
   langchain \
   langchain-core \
   langchain-openai \
   langchain-community \
   fastapi \
   faiss-cpu \
   tiktoken \
   rank-bm25 \
   pydantic \
   python-dotenv

Step 2: Configure the model

Production systems must configure retries, timeouts, and credentials through the environment.

Python
import os
from langchain_openai import ChatOpenAI 

api_key = os.getenv("OPENAI_API_KEY") 
if not api_key: 
    raise ValueError("OPENAI_API_KEY must be configured.")

llm = ChatOpenAI( 
    model="gpt-4o-mini", 
    temperature=0, 
    api_key=api_key, 
    timeout=30.0, 
    max_retries=2, 
)

Setting a low temperature improves consistency, while explicit timeouts and retry limits help the system recover gracefully from transient API failures.

Step 3: Route before you generate

Not every request requires a large language model. Deterministic logic can often answer simple questions. Routing inexpensive requests away from the LLM yields the most significant cost reduction in production systems.

Python
def classify_request(question: str) -> str:
    q = question.lower()

    if "status" in q:
        return "metrics"

    if "runbook" in q:
        return "retrieval"
   
    return "generation"

Step 4: Add a semantic cache

One of the simplest and most effective optimizations is an exact-match cache, which returns a previously generated response when the same question is asked against the same retrieved documents, avoiding unnecessary model calls.

Python
import hashlib

# Using an exact-match (lexical) cache
exact_match_cache = {}

def cache_key(question: str, sources: list[str]) -> str:
    """
    Generate a deterministic cache key from the user question
    and the retrieved document identifiers.
    """
    fingerprint = question + "|" + "|".join(sorted(sources))
    return hashlib.sha256(fingerprint.encode()).hexdigest()

# Example usage in the pipeline:
# key = cache_key(question, source_ids)
# if key in semantic_cache:
#     return semantic_cache[key]

Step 5: Budget your context

Most retrieval pipelines return far more text than the model actually needs. Instead of stuffing the context window with every retrieved document, establish a strict context budget.

Python
import tiktoken

encoder = tiktoken.encoding_for_model("gpt-4o-mini")
MAX_CONTEXT_TOKENS = 2500

def build_context(chunks):
    context = []
    used = 0

    for chunk in chunks:
        tokens = len(encoder.encode(chunk.page_content, disallowed_special=()))

        if used + tokens > MAX_CONTEXT_TOKENS:
            break 

        context.append(chunk.page_content)
        used += tokens

    return "\n\n".join(context)

Step 6: Retrieve once

Repeated retrieval is a surprisingly common flaw in multi-agent systems. The rule is simple: retrieve once, reuse everywhere.

Python
from langchain_core.documents import Document
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings

documents = [
    Document(
        page_content="Database latency often follows connection pool exhaustion.",
        metadata={"source": "db_runbook"},
    ),
    Document(
        page_content="Node pressure can increase API response times.",
        metadata={"source": "cluster_runbook"},
    ),
]

embeddings = OpenAIEmbeddings(api_key=api_key)
index = FAISS.from_documents(documents, embeddings)

retrieved_docs = index.similarity_search(question, k=4)
shared_context = build_context(retrieved_docs)

Now, every downstream agent consumes the same optimized context instead of launching its own redundant retrieval pipeline.

Step 7: Route models intelligently

Large models should solve complex problems. Everything else belongs to a smaller, faster model.

Python
from langchain_openai import ChatOpenAI

small_model = ChatOpenAI(model="gpt-4o-mini", temperature=0, api_key=api_key)
large_model = ChatOpenAI(model="gpt-4.1", temperature=0, api_key=api_key)

def choose_model(question: str):
    """Route requests to the most appropriate model based on complexity."""
    if len(question) < 200:
        return small_model
    return large_model

This strategy drastically reduces operational costs without noticeably affecting response quality.

Step 8: Estimate tokens before sending

Without token telemetry, optimization is just guesswork. Monitoring usage makes efficiency measurable and helps engineers detect cost regressions.

Python
import tiktoken

encoder = tiktoken.encoding_for_model("gpt-4o-mini")

def estimate_tokens(messages):
    """
    Estimate input tokens for an OpenAI-style chat payload.
    Note: This is an estimate, not an exact billing calculation.
    """
    tokens_per_message = 3
    tokens_per_name = 1
    total = 0

    for message in messages:
        total += tokens_per_message
        for key, value in message.items():
            if isinstance(value, str):
                total += len(encoder.encode(value))
            if key == "name":
                total += tokens_per_name

    # Every reply is primed with additional assistant tokens.
    total += 3
    return total

Step 9: Validate responses

Production systems must return structured outputs to ensure downstream systems receive predictable, well-formed data.

Python
from pydantic import BaseModel

class AgentResponse(BaseModel):
    answer: str
    sources: list[str]

def validate_response(answer: str, sources: list[str]):
    """Validate and serialize the agent response using a structured schema."""
    response = AgentResponse(
        answer=answer,
        sources=sources,
    )
    return response.model_dump()

Step 10: Build the optimized pipeline

Finally, assemble the architectural components into a single workflow. Notice how failures degrade gracefully instead of crashing the service.

Python
import logging

from langchain_core.prompts import ChatPromptTemplate

logger = logging.getLogger(__name__)

def run_pipeline(question: str):
    """Execute the token-efficient AI workflow with graceful degradation."""
    try:
        route = classify_request(question)

        # Route deterministic requests away from the LLM.
        if route == "metrics":
            return {
                "answer": "Retrieve metrics directly from the monitoring system.",
                "sources": [],
            }

        # Retrieve context once.
        docs = index.similarity_search(question, k=4)

        context = build_context(docs)

        source_ids = [
            doc.metadata.get("source")
            for doc in docs
            if doc.metadata.get("source")
        ]

        # Check exact-match cache.
        key = cache_key(question, source_ids)

        if key in exact_match_cache:
            return exact_match_cache[key]

        # Select the most appropriate model.
        model = choose_model(question)

        # Keep trusted instructions separate from untrusted user input.
        prompt_template = ChatPromptTemplate.from_messages(
            [
                (
                    "system",
                    (
                        "Answer the user's question using ONLY the provided context. "
                        "If the answer cannot be determined from the context, say so."
                        "\n\nContext:\n{context}"
                    ),
                ),
                ("user", "{question}"),
            ]
        )

        chain = prompt_template | model

        result = chain.invoke(
            {
                "context": context,
                "question": question,
            }
        )

        payload = validate_response(
            answer=result.content,
            sources=source_ids,
        )

        # Cache validated response.
        exact_match_cache[key] = payload

        return payload

    except Exception:
        logger.exception("Token-efficient pipeline failed.")

        # Gracefully degrade instead of crashing.
        return {
            "answer": (
                "The AI pipeline encountered an error. "
                "Please continue using the standard operational workflow."
            ),
            "sources": [],
        }

What actually reduced token usage?

When teams instrument architectures like this, the largest savings rarely come from editing prompts. They come from eliminating unnecessary work.

The biggest improvements typically stem from:

• Retrieving documents once instead of multiple times.
• Caching semantically identical requests.
• Routing simple requests away from the LLM.
• Limiting context with explicit token budgets.
• Selecting the smallest suitable model.

These architectural shifts reduce cost and latency while making system behavior significantly easier to reason about.

Lessons learned

Several core principles consistently emerge when optimizing AI systems for production:

• Treat tokens like infrastructure: Tokens are a finite resource, just like CPU cycles or memory. Monitor them, budget them, and optimize them.
• Retrieval is usually the largest source of waste: Repeated retrieval often contributes more unnecessary tokens than verbose prompts. Share context whenever possible.
• Bigger models are not always better: Smaller, faster models effectively handle many operational tasks. Reserve larger models for genuinely complex reasoning.
• Caching is an engineering feature: A semantic cache is more than a performance optimization—it is a core architectural component that reduces cost, latency, and provider dependence.
• Measure before you optimize: Instrumentation must accompany every production deployment.

As AI systems mature, success will increasingly depend on engineering efficiency rather than raw model size. The hidden tax of AI agents is rarely a single expensive prompt; it is the accumulation of redundant retrievals, oversized contexts, unnecessary model calls, and repeated reasoning across distributed workflows.

“The most effective production AI systems are not the ones that generate the most tokens. They are the ones that generate only the tokens they truly need.”

By treating token consumption as a systems engineering problem, organizations can build AI platforms that are faster, less expensive, and highly scalable. Routing requests intelligently, budgeting context, sharing retrieval results, validating structured outputs, and introducing semantic caching are practical techniques that guarantee efficiency without compromising quality.

The most effective production AI systems are not the ones that generate the most tokens. They are the ones that generate only the tokens they truly need.

The post Stop the token bleed: building token-efficient multi-agent systems appeared first on The New Stack.

AI infrastructure platform company TrueFoundry has launched its open-source agent harness, TrueForge. The technology, announced Wednesday, is directly billed as an alternative to Claude Managed Agents, Anthropic’s hosted infrastructure service that runs, sandboxes, and orchestrates autonomous Claude agents.

TrueForge promises to enable software engineers to build, deploy, debug, and govern production AI agents on any model (and the company means any model) or MCP server, while reducing total agent operating costs by an estimated 50%.

While open models such as GLM-5.2 from Chinese frontier model maverick Z.ai are challenging proprietary frontier models at lower costs, most managed agent platforms still lock enterprises into a single vendor’s models, infrastructure, and pricing. 

Challenging the pervading narrative of managed agent platform lock-in

Ex-machine learning tech lead at Meta and now co-founder and CEO of TrueFoundry, Nikunj Bajaj, tells The New Stack that this pervading managed agent platform lock-in is precisely the logic behind his firm’s neutral approach to model vendor choice.

“A provider selling you a million tokens for $50 has zero incentive to tell you the same task could be done using a model that charges 50 cents for a million tokens,” Bajaj says. “Traditionally, one vendor provides the models, builds your agents and decides your token usage, in what order, and with what tools and under governance that the managed agent provider stipulates – and they’re selling the exact same setup to your competitor.”

Fundamentally, he insists, this means “the incentives are misaligned” here and so the “players in this game don’t get a voice to talk to the referee” in managed agent deployment scenarios where there’s always a tradeoff. 

“Why should building powerful agents mean giving up control of your AI stack? We give developers the managed-agent experience without forcing them into one vendor forever,” adds Bajaj.

“A provider selling you a million tokens for $50 has zero incentive to tell you the same task could be done using a model that charges 50 cents for a million tokens. The incentives are misaligned, so the players in this game don’t get a voice to talk to the referee.”

The harness underneath becomes the strategic control point

Although Claude Managed Agents only arrived as a beta release in April of this year, Bajaj and team think they can track an evolutionary curve being etched out here. This arc sees the first wave of AI agents existing on developers’ laptops, inside coding tools and prototypes. But the next wave is moving into customer-facing products typified by hosted infrastructure services with the ability to use shared workflows.

Crucially, that’s a shift that turns the harness underneath those products into a strategic control point, working as an execution layer in an operational loop between the user, the model and the systems it interacts with.

“This is indeed the reality: the harness is the critical layer between the user, LLM, and everything else,” clarifies Bajaj. “Say a developer is building an agent. They bring their own models, but the harness still decides when to call an MCP server, when to use an agent someone has already built, what context to keep, and which model handles which part of the plan.”

This means there are security implications, too. Bajaj specifies that “some actions” still need to run in a completely isolated sandbox, and some data should never be sent to a closed-source model. 

“All of that logic sits in the harness. If software engineers don’t use one, then the developer team has to build all of that logic from scratch,” he adds.

Enterprises will want to own key agent layers

In an open vendor-neutral approach to managed agent platform provision, organizations must manage persistent sessions, tool credentials, execution sandboxes, context, human approvals, debugging, access policies, and spending across every agent they operate. TrueFoundry is betting enterprises will want to own that layer rather than inherit it from a single model provider, but with enterprise governance built in at lower cost.

TrueForge routes every model call and MCP interaction through TrueFoundry’s AI Gateway, so budget enforcement, rate limits, and guardrails can be applied to deliver a governed and secure managed agent experience for enterprises.

Headless chickens, when foo and bar are behind the wheel

When organizations don’t have the same hold on the steering wheel, Bajaj says that he has personally witnessed operations where “foo” and “bar” (standard placeholder names used in computer programming for as yet-unnamed known metasyntactic variable values, rather like John Doe) end up becoming the doers of everything. 

“Every action in the system came from a generic shared account, not a person you could actually identify. So when something changed or broke, you had no idea who to talk to. Once, when we were halfway through a migration from shared access to individual access, some keys were rotated. Half the company was still on the old account, and the system broke for half the company,” he explains.

Teams can run TrueForge on their own infrastructure, bring their own models, MCP servers, and API keys, and route each task to whichever model fits the cost, latency, or quality needs of that job. But does that mean workloads might become too fragmented that way?

“On the contrary, workloads become more uniform,” enthuses Bajaj. “Most teams already bring their own models by default. What changes is that organizations get to define what it takes for a model, agent, or MCP to belong in their registry. I call it the agent development life cycle, or ADLC. Once you own that, you can enforce the same operating principles across everything.”

In practice, the TrueFoundry team confirms it has seen most AI-centric software engineering operations converge on “roughly a dozen models” for typical tasks, plus a few specialized models for niche work. 

Is Anthropic doing something wrong?

TrueForge ships with support for OpenAI, Anthropic, and 20+ additional models, along with 40+ built-in tools, sandboxed execution, human-approval workflows, large-context handling, generative UI, and web search powered by Tavily. But despite offering a Claude Managed Agents alternative, Bajaj goes to pains to point out he doesn’t hold Anthropic up as some kind of pariah. 

“This isn’t about Anthropic doing something wrong,” confirms Bajaj. “It’s that it doesn’t own every model in the world. Claude Managed Agents can only choose from the finite set of models Anthropic offers. There are open models that are terrific at certain tasks at a fraction of the cost, or simply more capable for that particular job. An open harness has a much wider set of choices.”

When you own the harness, you can get rid of the parts that don’t apply to you

He underlines his point by pointing out that Anthropic also has to build one harness for a very broad set of customers; a truth that means its system prompt has to account for all kinds of instructions, guardrails, and corner cases. 

“Many of those elements may have nothing to do with a developer’s own use case, but they still go into every call and add cost and latency. When you own the harness, you can get rid of the parts that don’t apply to you and make it extremely specialized,” he adds.

“To be clear, we support Anthropic as a first-class provider because its models are great. There will be many cases where our users want to use them. The point is not to limit that choice to Anthropic alone.”

To validate its statements here, TrueFoundry has tested the above claim on a total of 14 level-one and level-two tasks from DevRev’s public Enterprise-Bench. The company says TrueForge “came in 50% cheaper at similar accuracy”, so the savings came from using fewer tokens and having access to models outside Anthropic’s set that were better suited to specific tasks.

“To be clear, we support Anthropic as a first-class provider because its models are great. There will be many cases where our users want to use them. The point is not to limit that choice to Anthropic alone,” Bajaj concludes.

TrueFoundry is also launching a hosted, pay-per-usage version of TrueForge for teams that want the same experience without managing the infrastructure themselves.

The post An open source rival to Claude Managed Agents just launched appeared first on The New Stack.

Mistral is giving enterprise customers until August 31 to replace the Google Drive and Microsoft SharePoint Knowledge Connectors they use in Vibe Work with MCP-based alternatives. The company says in its Knowledge Connectors documentation that both existing connectors will be shut down and deleted on that date.

There is no automatic migration, so administrators will need to install the MCP replacements before every user reconnects their Google or Microsoft account. The move changes how Vibe Work reaches company documents, but Mistral has said little about the retrieval architecture behind the new connectors.

Mistral stores a searchable index

With the current system, an administrator chooses which Google Drive folders or SharePoint sites the organization wants to make available, then Mistral processes those files and stores the resulting index in its European data centers.

Once the index is ready, users connect their personal accounts, and when they search in Vibe Work, the connector checks permissions copied from Google Drive or SharePoint so the results include only files they can access, while scheduled synchronizations pick up later changes and deletions.

This setup lets Mistral handle retrieval by searching a prebuilt index whenever a user submits a query. The company says indexing can take anywhere from a few minutes to several hours, depending on how much data the organization includes, although that work is completed before users begin searching.

The company says indexing can take anywhere from a few minutes to several hours, depending on how much data the organization includes, although that work is completed before users begin searching.

MCP shifts retrieval off-platform

The company defines MCP as a common interface that lets models call tools and retrieve data from external services — the same protocol layer that is reshaping how AI products connect to external APIs across the industry.

In June, Mistral added Google Drive and SharePoint to a directory containing more than 60 integrations, but it did not explain how those two connectors retrieve documents or say who operates the underlying MCP servers. The setup can range from live calls to the source API to a server-managed search index, with any combination of the two in between. That flexibility is part of what makes MCP appealing in some environments and unnecessary in others, but it also means that the behavior of a given connector depends entirely on its operator.

Permissions rules remain unclearThe company doesn’t run the third-party servers behind these connectors, so it can’t promise how they will behave or what they will do with customer data. That gap between the governance Mistral once provided and what it’s now handing off to third parties reflects a trend in how enterprises are adopting MCP connectors without fully resolving the governance layer. As other platforms have learned, opening the door to external servers means trusting the protocol and the operator and building the guardrails to go with it.

The migration notice offers even less detail. It doesn’t say whether the Google Drive and SharePoint MCP servers retrieve files directly from Google and Microsoft. Any caching remains unexplained, leaving customers unsure where retained data would live. Mistral makes no promise that searches will be as fast or return results of the same quality.

Mistral makes no promise that searches will be as fast or return results of the same quality.

The outgoing Google Drive connector follows the sharing rules already attached to each file, including group and domain access. A file set to “anyone with the link” still isn’t automatically visible to everyone in the organization. SharePoint uses Microsoft Entra ID groups, so older groups created only within SharePoint aren’t picked up.

Mistral hasn’t said whether the MCP replacements will follow the same rules. OAuth can limit a server’s access to the connected user, but that doesn’t mean search results will be filtered exactly as they were in the outgoing index. The protocol wasn’t designed to enforce that kind of enterprise permission model; the connector’s retrieval layer has to do it.

OAuth can limit a server’s access to the connected user, but that doesn’t mean search results will be filtered exactly as they were in the outgoing index.

Deletion timeline still unresolved

The company says disabling a Knowledge Connector results in permanent deletion of the indexed data, but the deprecation notice does not say whether the August shutdown will trigger that process automatically or how long the deletion will take. It also leaves administrators unsure whether they should disconnect the old connectors themselves before the deadline.

Because the same connectors work in Vibe Code and Mistral’s workflow system, teams can use the same approach to expose external data across chat, coding and automated jobs — a pattern that is becoming more common as MCP connectors spread from chatbots into production infrastructure.

Mistral also recommends checking server output for signs of prompt injection — advice that underscores the still-emerging challenge of securing the space between AI agents and the external services they access.

The post What happens to your indexed data when Mistral flips the switch? appeared first on The New Stack.

Cursor has officially thrown its hat into the code-hosting ring with Origin, a Git-compatible platform built for a world where AI agents generate the commits.

The beta launch announced late on Monday comes two months after Tomas Reimers took to the stage at Cursor’s developer conference in San Francisco to tease its agent-focused GitHub alternative. That happened to be on the very same day that Elon Musk’s SpaceX confirmed it had tabled a $60 billion bid to acquire Cursor outright, and with that deal formally closing on August 14, Origin becomes the first product Cursor has shipped as a fully owned SpaceX subsidiary.

Notably, however, Origin landed on the same day that GitHub itself went down worldwide, turning what might have been a routine beta rollout into a “case in point” on why Cursor was building Origin to begin with. But while the 8-hour outage may have seemed like fortuitous timing on the surface, GitHub going offline when it did wasn’t great for Cursor, given that Cursor needs a fully operational GitHub for new users to get the ball rolling. As SpaceXAI’s Matt Palmer acknowledged on X: “We were going to ship this earlier, but GitHub was down. Importing your GitHub repos as a first onboarding step is non-optimal if GitHub is down.”

Still, GitHub’s troubles predate this particular blackout. As The New Stack reported in June, the platform has logged hundreds of incidents over the previous 12 months as commit volume jumped from 1 billion a year to 1.4 billion a month, with AI agents alone generating more than 17 million pull requests monthly — growth GitHub traced to infrastructure bottlenecks like MySQL contention and webhook overload.

“Cursor has joined a slew of technology companies looking to rebuild version control for a world where agents work around the clock.”

In an interview with The New Stack at the time, GitHub COO Kyle Daigle discussed the scaling problem: “It’s not just about normal scaling,” he said. “It’s now making sure we can scale at 30 or 40 times” annual growth, as opposed to doubling each year, which GitHub had historically planned around.

Fast-forward to today, and Cursor has joined a slew of technology companies looking to rebuild version control for a world where agents work around the clock, querying and pushing to repositories faster than any human team ever could.

Origin story

Origin marks a fairly significant expansion of Cursor’s ambitions. Until now, its agents have largely operated on code hosted elsewhere; with Origin, Cursor is pushing to own more of the underlying development infrastructure itself.

At launch, that starts with the basics. Users can create and host Git repositories directly inside Cursor, with the new Codebase tab acting as the home for Origin repos.

Those repositories still behave like Git repos outside Cursor. Developers can clone them locally, add an Origin remote, and push code from the command line — essentially putting Cursor in the role normally occupied by a service such as GitHub.

Cursor isn’t demanding an all-or-nothing migration, either. Existing GitHub repositories can be synced into Origin and displayed alongside Cursor-hosted repos, while GitHub remains the source of truth for projects that started there.

Pull requests are built in too, including diffs, comments, checks and merging. Cursor’s agents sit directly alongside that code: from the browser, users can ask questions about what they’re viewing, have an agent make changes, update a PR or push a branch.

That combination is arguably the more consequential part of Origin: the repository, pull request and coding agent now all live inside the same product — giving Cursor more control over the environment in which code is stored, reviewed and changed.

Rob Whiteley, CEO of Coder, a cloud development platform built for enterprises, sees Origin as a “smart play” — most of the industry’s energy has gone into the tools that write code, he argues, while comparatively little has gone into what happens to that code after it’s produced.

“GitHub is starting to crack under the weight of agentic code development, and an agent-native source code forge is needed,” Whiteley tells The New Stack. “Everyone is integrating the ‘writing code’ stack, from editor and chat to agents, tools and LLMs. No one else is really integrating the ‘managing code’ stack, where code gets stored, versioned, reviewed and merged.”

For now, Origin’s restricted to Cursor’s Pro, Teams and Enterprise plans — nothing on the free tier, it seems — and the rollout itself is staged, so not everyone will have access to it quite yet.

How is Origin different to GitHub?

For now, there’s no escaping the fact that there isn’t a great deal that’s different from trusty ol’ GitHub, a fact that wasn’t entirely lost on the online community. And Origin’s own team isn’t shying away from that, either.

Tomas Reimers, the Origin engineer who co-founded Graphite, a code-review startup Cursor acquired in early 2026, fielded questions directly from developers on Hacker News after the launch.

“We’re intentionally releasing this as a GitHub alternative where we meet them toe-to-toe on functionality.”

Asked what set Origin apart from GitHub beyond uptime, Reimers concedes that it’s very “very little,” in all honesty. “We’re intentionally releasing this as a GitHub alternative where we meet them toe-to-toe on functionality,” he writes.

He does note that more is on the way: in the coming weeks, Reimers explains, Origin should start shipping deeper agent integrations, tooling that can make sense of agent-written code, and automation that pushes pull requests toward a mergeable state on their own.

“Expect a lot more from us,” he continues. “We wanted to release a beta so people could start experimenting with our scalability and extensibility themselves. Over the next few weeks, you can expect a handful of features starting to change source control to better understand and work with agents.”

Several in the online community also highlighted the timing of Origin’s launch. Gergely Orosz, author of The Pragmatic Engineer newsletter and an investor in Graphite, initially took to X to complain about GitHub’s ongoing performance issues, despite the number of engineers it has at its disposal.

Within an hour, however, Orosz was back to comment on Origin. “Cursor could not have timed their launch announcement of their hosted code service better either,” he writes. “If GitHub was stable, these alternatives would not be as interesting / popular!”

“These alternatives,” as Orosz puts it, are already forming an orderly queue in GitHub’s shadow.

The ‘GitHub alternative’ surge

The most direct comparison to Origin is perhaps Entire, a distributed Git network founded by Thomas Dohmke, who stepped down as GitHub’s CEO last August. Entire, essentially, mirrors repositories across regional nodes, with GitHub remaining the source of truth for now — though that could change once teams start creating repos natively on the platform. The company raised a $60 million seed round in February, with investors including Microsoft’s venture arm, and it formally went to market in July.

“Cursor could not have timed their launch announcement of their hosted code service better either. If GitHub was stable, these alternatives would not be as interesting / popular!”

GitLab, meanwhile, is also rethinking source control for an agent-heavy world, announcing a private beta of “Next Generation Source Code Management” — internally called Project Switch — back in June. Instead of agents cloning an entire repository to read or change a handful of files, the system lets them query the server for exactly what a task needs, with each agent’s visibility capped at the minimum required.

Elsewhere, code editor startup Zed has also been teasing a new approach to version control since last year, and in early August the company finally debuted Delta, a “multiplayer environment for coding with agents and reviewing what they build,” as the company puts it.

“Delta keeps code and conversations connected, so developers and agents can work together with the full context of how the code came to be,” Zed co-founder and CEO Nathan Sobo wrote at the beta launch.

Underneath, Delta runs on DeltaDB, which keeps a live copy of conversations and in-progress work synced across a team. It sits alongside a project’s existing Git repository, and works with agent tools including Claude Code. What that changes in real terms: comments stay attached to the code they refer to as an agent keeps editing it.

Despite GitHub’s persistent reliability problems, Whiteley doesn’t see Cursor’s version becoming a major enterprise play any time soon, mostly due to the pain of switching.

“Most [enterprises] have already spent a lot of pain and money standardizing on GitHub, and moving again would mean a lot of pain for limited ROI today,” he says. “That could change as ‘vibe coding’ generates an order of magnitude more code. If Cursor commits to keeping Origin open enough for enterprises to trust and integrate with, it could become much more appealing over time.”

So while there is a clear flurry of activity in the “GitHub alternative” realm, it’s still too early to say whether any of them will cut it in the long term. GitHub has an 18-year head start: it launched in 2008, popularized the pull-request review model most of these newcomers are trying to disrupt, and was snapped up by Microsoft in 2018. It also set off the current wave of AI coding tools itself with the launch of Copilot in 2021 — the same wave now generating the volume its own infrastructure is struggling to absorb.

For now, GitHub remains the only one of these platforms actually handling this kind of developer activity at scale. However, at a lofty $2 trillion valuation, SpaceX is one of the world’s most valuable companies, which puts Cursor in a strong position when it comes to investment — not just in its own AI models, but in the infrastructure underneath them.

The post “If GitHub was stable, these alternatives would not be as interesting”: Cursor launches Origin as GitHub goes dark appeared first on The New Stack.

Half of enterprise AI deployments are missing their own latency targets at peak load. This is the headline finding of Akamai’s State of AI Inference 2026 report, which surveyed 200 AI practitioners and found that 82% of organizations say their most critical use cases require end-to-end response times of 500 milliseconds or less. A total of 64% of organizations now require end-to-end response times of less than 250 milliseconds for their most important use cases, yet 50% of deployments are failing to meet these latency demands at peak load.

My colleague Ari Weil, who leads product marketing for our cloud computing business and ran point on that research, summarizes the findings well: “The enterprise AI honeymoon phase is over… they are hitting the latency wall.” 

Agentic workflows aren’t a “single round trip”

The latency issue stems from the way agents work. It’s an iterative process, somewhat like a king sending out knights, emissaries, and messengers to conduct the business of the kingdom. There are many comings and goings, not just one person sent on a single round trip. 

For instance, when an agent built on a framework like LangChain, CrewAI, or Pydantic AI received a user request, it can fan out into dozens of sequential operations such as a reasoning call, a tool invocation, an API lookup, or a context retrieval. Then an agent may execute another reasoning call to decide what to do with what just came back. Every one of these operations or “hops” that must cross a wide-area network to reach a centralized data center adds latency, and a chain of 50 hops can multiply that transport time into seconds on its own, regardless of how fast the model generates tokens.

In fact, in a paper posted to arXiv in November 2025, researchers found that CPU-side processing can account for up to 90.6% of total latency in agentic workloads. In other words, your GPU might finish a reasoning step in a few hundred milliseconds, but then it might have to wait on additional tool call runs to CPUs in distant data centers. This is what causes spikes in GPU idle time. 

“More GPU capacity does nothing for this. You can’t brute-force your way out of a wait state.”

More GPU capacity does nothing for this. You can’t brute-force your way out of a wait state. This is the part of the conversation the industry keeps skipping, mostly because “buy more GPUs” is a much quicker fix to suggest than “figure out where your CPU-bound work is actually executing and why it’s so far from the data it needs.”

We need new benchmarks to fix the latency issue

One reason the looming latency wall sneaks up on teams is that they are not looking at the right benchmarks for agentic workloads. Most LLM-serving benchmarks measure tokens per second and GPU utilization on a single box. That’s great if the workload is indeed on a single box (i.e., one model answering one prompt), but that’s not the case with agentic workloads. Those benchmarks don’t address an agentic response that, say, makes a 50-hop chain cross a WAN 4 times to reach 4 separate services. 

“Staging may pass the benchmarks because it tests the model, but production tests the whole chain, including every hop your serving engine was never designed to see.”

That’s where the gap lies: Staging may pass the benchmarks because it tests the model, but production tests the whole chain, including every hop your serving engine was never designed to see.

The 500ms wall is not a soft target

This is showing up at scale because agents are moving into production faster than most teams’ architecture is evolving to support them. LangChain’s State of Agent Engineering 2026 survey of more than 1,300 professionals found that 57.3% of organizations now have agents running in production, up from 51% a year earlier. Among those builders, latency has become the second-most-cited barrier to production, behind only output quality. 

This is a serious issue for application teams. The 500ms threshold in Akamai’s survey isn’t a performance goal teams can afford to miss. For a live customer interaction or a real-time compliance check, that 500ms determines whether the application works or it doesn’t. 

We’ve solved this problem before

There’s a reason this feels familiar to anyone who was building for the web in 1999. Akamai exists because of a nearly identical problem. MIT researchers Tom Leighton and Danny Lewin founded the company to answer a challenge posed by Tim Berners-Lee: fix what the press had started calling the “World Wide Wait,” the crushing latency of pulling every request back to a small number of centralized servers. When the trailer for The Phantom Menace crashed sites across the internet in 1999, the culprit was distance: millions of browsers all reaching for the same far-away origin server at the same moment. The fix moved content to thousands of points closer to the people requesting it, instead of trying to build a faster origin.

Agentic AI is running into the same wall, just in a different vehicle. AI works just fine on centralized inference if you’re talking about running batch jobs overnight. But today’s applications built on agentic AI are real-time loops sitting inside live transactions, and the fix for agentic lag is distribution. Instead of expanding racks of CPUs and GPUs at the center, we need to move agentic execution to where the model’s tools, context data, and users actually live.

Agentic AI needs a tiered architecture, not a bigger data center

In practice, agentic AI requires a tiered architecture, one that includes a centralized core, regional GPU clusters, and CPUs at the Edge. 

• Centralized core—perfect for heavy reasoning over large context windows, where the round trip to a large model matters less than the model’s raw capability.
• Regional GPU clusters, increasingly built on hardware like NVIDIA’s Blackwell platform—ideal for localized inference, so the heaviest compute sits closer to where demand actually concentrates.
• Edge CPUs—the essential component for speed. This is the nexus for tool execution, orchestration, and context retrieval, since these are the steps that happen most often in a chain and benefit most from sitting next to the data and APIs they call.

We’ve built Akamai Inference Cloud around this tiered framework. It’s the same distribution logic behind our AI Grid Orchestrator. We route CPU-bound orchestration and tool calling to the edge, and keep GPU-bound reasoning where it makes sense, regionally or centrally. 

What to demand before you commit

The good news is you don’t need to distribute every workload to the edge on day one. But before you commit to a production architecture, you should know which of your agent’s dozens of hops are latency-sensitive and which aren’t. Then build a defined performance budget for each one.

“The teams that treat it as a GPU-shopping decision will be back here in six months, staring at the same four-second response time, wondering why more compute didn’t help.”

My advice is this: Before signing off on a large-scale inference deployment, ask your infrastructure for four things: 

1. Portability across regions and providers
2. Elasticity to absorb peak load without falling over
3. Data locality so tool calls aren’t crossing oceans to reach the context they need
4. A performance budget you’ve actually tested against production traffic, not staging traffic.

The teams that address this infrastructure decision now will be the ones whose agents still work when the benchmark environment transitions to real users. The teams that treat it as a GPU-shopping decision will be back here in six months, staring at the same four-second response time, wondering why more compute didn’t help.

The post Agentic AI has a latency problem that more compute won’t solve appeared first on The New Stack.

Cursor began rolling out Origin, its own code hosting platform, to paid users on Monday morning. Roughly three and a half hours later, GitHub's status page lit up with what became a six-hour-and-forty-two-minute global degradation — error rates near 20% across pull requests, issues and the API, and near 50% on archive and raw file downloads, according to GitHub's incident log. Enterprise single sign-on went down with it: SAML, OIDC, SCIM provisioning and Team Sync all failed. So did Copilot.

The developer internet did what the developer internet does.

"You can now host your repos in Cursor Origin and deploy to Vercel via Cursor Origin which is itself hosted on Vercel," Vercel chief executive Guillermo Rauch posted on X. "And unlike GitHub, it's online 😁" Asked why he was smiling, Rauch replied: "trying to make light of the situation. We ourselves are stuck because of github rn!"

Matt Palmer, who works at Cursor, quote-tweeted his own company's launch with the day's best line: "We were going to ship this earlier, but GitHub was down." A GitHub outage, in other words, delayed the launch of a GitHub competitor.

Product launches get locked weeks in advance, and no evidence suggests Cursor timed this one. But the coincidence did the company an enormous favor, because it dramatized the argument Origin exists to make. For eighteen years, choosing where to host your team's source code has been the least interesting decision an engineering organization makes. Cursor is betting that AI agents have made it interesting again — and for technical decision makers, that is the real news here. Not a new product, but a new procurement question with a governance problem attached.

Inside Origin: what Cursor's code hosting platform actually does

Origin lives in a new Codebase tab inside Cursor. Teams name a codebase, which becomes part of its URL, then push to it over the command line. From there they get the machinery you would expect from a forge — the service layer that wraps Git and handles storage, permissions, checks and merges. Every repository comes with pull requests: timelines, commits, checks and files changed. Reviewers read the diff, leave comments and merge, without ever opening a browser tab.

What Cursor built around that machinery is the part worth studying. Agents now operate in the same surface as the code and the pull requests they are modifying. "Your code, PRs, and agents are now in the same place," the changelog reads. A developer can ask questions about the file on screen, hand an agent a review comment and have it revise the pull request in place, or tell it to push a branch — all inside the editor where the code was written.

Three integrations shipped on day one, and the choice of partners is telling. Vercel spins up a preview deployment for every pull request and ships to production on merge, available in public beta for Pro and Enterprise customers, its developer account said. Depot and Buildkite run continuous integration, and critically, both execute existing GitHub Actions workflows unchanged. Buildkite adds native pipelines on top.

That compatibility layer is the whole strategy in miniature. Cursor is not asking teams to rewrite their build system, retrain their engineers or rip out their deployment pipeline. It is asking them to try a second window onto code they already have — which is a far easier request to approve.

More partners are coming, the company said, and the ones it landed first are the ones that matter to a platform team evaluating whether Origin can carry real work. A forge without deployments and CI is a code viewer. A forge that runs your existing Actions workflows and ships previews to the CDN you already pay for is a candidate.

Why letting GitHub stay the source of truth is Origin's smartest design choice

Here is the decision enterprise buyers should study most closely, because it determines whether Origin survives a security review at all.

Cursor does not ask you to leave GitHub. Connect a GitHub organization, pick repositories, and they appear alongside Origin-native ones. "Pushes keep going to GitHub, which stays the source of truth for anything started there," the changelog says. Access permissions mirror GitHub's existing read and write settings rather than establishing a parallel system. Pull request conversations sync in both directions — comment in Cursor and it posts to GitHub; reply or react on GitHub and it surfaces in Cursor "within seconds."

This is a classic wedge, and a well-executed one. Rip-and-replace migration of source control ranks among the highest-risk projects an engineering organization can undertake. It touches continuous integration, compliance evidence, audit trails, branch protection rules, every integration in the toolchain and the muscle memory of every engineer on staff. Almost no chief technology officer approves that for a product in early beta.

A read-mostly mirror that leaves GitHub authoritative approves itself. It costs nothing to try, breaks nothing if abandoned, and quietly relocates the place developers spend their working hours. If Cursor's review experience proves better — and Cursor spent real money to make sure it would — the source of truth eventually follows the attention.

That money went to Graphite, the code review startup Cursor bought in December 2025 for what Axios reported was well above its $290 million Series B valuation. Graphite built stacked pull requests, the workflow that lets developers keep shipping dependent changes without waiting on approvals. Announcing the deal, Cursor wrote that "the boundary between where you write code and where you collaborate on it feels increasingly arbitrary," and promised "some more radical ideas we can't share just yet." Origin is the radical idea. Graphite co-founder Tomas Reimers unveiled it on stage at Cursor's inaugural Compile conference in June and leads its development.

How AI agents turned code review into software's new bottleneck

The case for an agent-native forge rests on a claim that is easy to state and, unusually for this market, well supported by evidence: writing code stopped being the constraint. Reviewing and integrating it became one.

Google's 2025 DORA report, drawn from nearly 5,000 technology professionals, found that 90% of developers now use AI at work, spending a median of two hours a day with it, and more than 80% say it made them more productive. But AI adoption showed a positive relationship with software delivery throughput and a negative one with delivery stability. More output, more breakage. The report's authors describe AI as "an amplifier" that "magnifies the strengths of high-performing organizations and the dysfunctions of struggling ones."

Trust has not kept pace with volume. Stack Overflow's 2025 developer survey of 49,009 respondents across 177 countries found 84% using or planning to use AI tools, while trust in their accuracy fell to 33% from 43% a year earlier and distrust climbed to 46% from 31%. Two-thirds named "AI solutions that are almost right, but not quite" as their leading frustration. GitLab's ninth annual DevSecOps survey, of 3,266 practitioners polled by Harris, put numbers on the operational drag: 73% had hit problems with vibe-coded output, 70% said AI made compliance management harder, and only 37% would let AI handle daily tasks without human review.

The volume climbs regardless. GitHub's Octoverse 2025 counted 180 million developers, 630 million repositories and 43.2 million pull requests merged per month, up 23% year over year. And RuntimeWire reported the internal figure that best explains Origin's existence: 35% of pull requests merged inside Cursor were opened by agents running autonomously in cloud virtual machines.

A forge built for humans assumes a pull request represents human intent, opened by someone you can ask what they meant. Once a third of merged changes come from software, the queue stops being a conversation and becomes a scheduling problem. That is a real architectural argument, and it is the strongest thing Cursor has going for it.

GitHub's reliability crisis handed Cursor an opening it did not have to earn

The supply-side case for an alternative is simpler: GitHub has been unreliable, and its own executives have said so.

An analysis by LeadDev counted 257 incidents between May 2025 and April 2026, 48 of them major — roughly one significant disruption per week. February was the worst month on record with 37. GitHub Actions alone accounted for 57 outages in twelve months. Chief technology officer Vlad Fedorov has said the platform "wasn't built for the scale it's now being asked to handle" and must design for 30 times today's load. In an April engineering post covered by InfoQ, the company acknowledged it "failed to meet its own reliability standards," citing rapid growth, tight architectural coupling and inadequate load shedding. Monday's outage was the seventh incident on GitHub's status page in fifteen days.

The fatigue is audible. "GitHub really doesn't feel built for the agent era," one developer wrote on X as Origin went live. "It goes down way too often, but until now there haven't been many real alternatives."

The defections started before Origin existed. The Zig programming language moved to Codeberg in November 2025, citing Actions failures among its reasons. In April, Mitchell Hashimoto announced that Ghostty — a terminal emulator with more than 52,000 stars — would leave too, pointing to near-daily outages that blocked reviews and CI for hours. And The Information reported in March that OpenAI, a company Microsoft holds a large stake in, began building its own GitHub alternative partly because outages left its engineers unable to commit for hours at a time, as Tom's Hardware relayed.

Microsoft's structure has not helped. Thomas Dohmke resigned as GitHub chief executive in August 2025 and was never replaced; the unit's leadership was absorbed into Microsoft's CoreAI organization under executive vice president Jay Parikh. In a May report, The Information wrote that Parikh had warned deputies that coding tools from Cursor and Anthropic could eventually make GitHub obsolete. GitHub's own answer to the agent era, Agent HQ, lets customers orchestrate third-party agents from Anthropic, OpenAI, Google, Cognition and xAI inside GitHub — a coherent strategy that concedes the agent layer and keeps the substrate underneath. Origin attacks precisely that substrate.

Now that SpaceX owns Cursor, who actually holds your source code?

Cursor's rise has been extraordinary even by the standards of this cycle. Founded in 2022 by four MIT students, Anysphere raised $8 million from the OpenAI Startup Fund in October 2023, per TechCrunch, then $100 million at $2.5 billion, $900 million at $9.9 billion, and $2.3 billion at $29.3 billion last November. In May, Bloomberg reported annualized revenue of $3 billion and more than 3,000 customers paying at least $100,000 a year.

Then, three days before Origin shipped, Bloomberg reported that SpaceX completed its $60 billion all-stock acquisition of Cursor — an agreement TechCrunch covered in June, days after SpaceX's record IPO and six months after it absorbed xAI. Cursor now operates inside a division called SpaceXAI. The vendor asking to hold your proprietary source code became, last Friday, a unit of a rocket company with its own frontier-model division and a founder not known for institutional caution.

Jason Andersen of Moor Insights & Strategy raised the model-routing question to Tech Times in June, before the deal closed: "xAI's models and treatment of guardrails are very different than what Cursor has stood for." That piece framed the question a chief information security officer now has to answer. When one company controls the editor where agents write code, the host where that code lives and the model those agents run on, what governs what it does with the code?

Cursor has not published an answer. RuntimeWire noted before launch that Origin's pricing, security architecture, data-handling terms and migration tooling were all unpublished, and Monday's changelog adds none of them. It says only that Origin reaches "all paid plan users starting today, except enterprise orgs whose admins opt out." Opt-out, not opt-in — a sentence administrators should read twice.

There is also a track record to weigh. In July, researchers at Mindgard disclosed that Cursor would execute a malicious git.exe planted in a Windows project's root the moment a user opened it, with no prompt — a repository-poisoning flaw they first reported in December 2025. The Hacker News reported that Cursor declined to patch it, calling the issue out of scope under a shared-responsibility model while conceding it had not "closed the loop with the researcher in a timely manner." No CVE was issued. The same flaw class turned up unpatched in GitHub Copilot CLI, Google's Gemini CLI and OpenAI's Codex — but a vulnerability the vendor declined to fix makes an awkward footnote for a product whose pitch is basically “let us hold your repositories.”

What engineering leaders should settle before they let Origin into the toolchain

Origin is a beta, not a migration, and treated as one it is worth evaluating. The sync mode gives platform teams a low-risk way to measure whether an agent-native review surface shortens cycle time, without touching a single branch protection rule. But three things deserve resolution before anything authoritative moves.

The first is the default. Origin switches on for paid users unless an enterprise administrator opts out, which means an organization that has not made an affirmative decision about whether proprietary code may be mirrored to a new host has effectively had that decision made for it. Confirming your posture is a Monday-morning task, not a next-quarter one.

The second is the paperwork. Retention, residency, training use, subprocessors and what changes now that Cursor reports into SpaceX are all unpublished, and a product page is not a contract. Until those terms exist in writing, the defensible position is to treat Origin as a convenience layer over GitHub rather than a system of record — which is, conveniently, exactly what its architecture already is.

The third is the exit. Origin's Actions compatibility and its GitHub-as-source-of-truth design are the properties that make it safe to adopt. They are also the ones most likely to erode as Cursor's incentives shift toward owning the substrate rather than borrowing it. Ask what egress looks like now, while the mirror is still a mirror.

None of which makes Cursor's argument wrong. GitHub earned its incumbency by being boring, dependable infrastructure, and it has spent eighteen months being neither while a third of the code arriving at its front door stopped being written by people. Origin is a serious answer to a real problem, built by a team that bought the right company to build it.

But GitHub's failure and Cursor's are different in kind, and enterprises should not confuse them. Monday's outage resolved at 20:22 UTC. Availability is an engineering problem, and engineering problems close. The question of who holds your source code, what they may do with it and who they ultimately answer to carries no such timestamp — and on that one, the company that spent Monday selling trust has yet to publish its terms.

Multi-tenancy has moved in one direction for 60 years: the tenant keeps getting smaller. Mainframe time-sharing carved a single machine into slices so an organization’s departments could share it, and the tenant was the org. Virtualization gave each team its own fleet of virtual machines, and the tenant became the team. Containers and Kubernetes namespaces shrank it again, until a platform team could hand every developer an isolated environment on a shared cluster.

That last step, an environment per developer, became the target state of platform engineering in the 2020s. A namespace per developer, capacity planned by seat, golden paths sized to headcount. Underneath all of it sits one assumption: a person produces one stream of work at a time, so isolating people isolates work.

Coding agents broke that assumption. A developer running five agent sessions has five changes in flight at once, each needing its own working version of the system. Anthropic’s engineers, building a C compiler with a fleet of parallel agents, ran nearly 2,000 Claude Code sessions across two weeks. Cursor’s documentation tells developers to run as many agents as you want in parallel. None of those concurrent workstreams is a person.

“The tenant has shrunk one more time. It is no longer the developer. It is the change.”

The tenant has shrunk one more time. It is no longer the developer (or even the agent). It is the change.

Tenancy demand scales with changes in flight, not headcount

Capacity planning by seat worked because changes arrived at human pace, roughly one per developer at a time. That denominator is gone. A Microsoft study of command-line coding agent adoption found that developers merged roughly 24% more pull requests over four months, and merged pull requests understate the pressure. Every change that reaches merge is preceded by iterations and abandoned attempts, and each of those also needed somewhere to run.

Run the seat math against the change math. A 50-developer organization where each engineer supervises a few agent sessions has hundreds of changes in some stage of validation on a busy day. Each one wants data it can migrate and write to without asking permission, its own view of shared message topics, and a running version of the services it touched. That is the demand of a 300-person (or more) engineering org on a 50-tenant platform. 

Every layer built on the person-tenant assumption misprices this. A per-developer namespace hands one tenant slot to what is now five concurrent workstreams. Shared staging serializes all of them into a single queue. Seat-based capacity plans budget for the number of employees while the bill tracks the number of changes in flight.

The new tenant is the change, not the agent

The tempting candidate for the new tenant is the agent, and it is the wrong one. Agents are interchangeable workers. Two agents can collaborate on one change, one agent can rotate through five changes, and a crashed agent gets replaced mid-task without anything downstream noticing. Give each agent its own environment, and you have repeated the old mistake at a new scale: isolating workers when the thing that must not leak is work.

“Give each agent its own environment, and you have repeated the old mistake at a new scale: isolating workers when the thing that must not leak is work.”

The durable unit is the change. It comes into existence when work on it starts. It accumulates state that no other tenant should see: a schema migration, test writes, new versions of one or two services, the messages it produced during validation. It needs to observe a version of the system that includes its own edits and nobody else’s. And it is torn down when it merges or is abandoned, taking all of that state with it.

Naming the change as the tenant turns a vague scaling problem into a design target, because change-level tenancy has three requirements that person-level tenancy never had to meet:

• Creating a tenant must be near free.
• Isolation must cover only what changed.
• The tenant’s lifecycle must be bound to the change itself, not to a ticket or a timer.

Platform teams already run this playbook in production

The discipline these requirements call for is not new. Anyone operating a multi-tenant production service already knows the rules: tenants share the substrate, each tenant privately owns only what makes it distinct, creating a tenant is self-service and cheap, and a tenant’s resources are reclaimed the moment it leaves. Nobody stands up a private copy of the product per customer, and nobody files a ticket to onboard one.

Those same organizations run pre-production on the opposite rules. Environments are provisioned by ticket or by seat, capacity is planned per person, and isolation is achieved by duplicating the stack when it is achieved at all. The multi-tenancy playbook that runs the product has never been applied to the platform that builds the product.

Change-level tenancy is that playbook, applied. Treat every change as a tenant of the development platform, and the three requirements stop being novel. They are the standard properties of any competently run multi-tenant system.

The tenant owns what changed and shares everything else

A SaaS tenant owns its data and configuration, never a copy of the application. A change tenant is sized the same way. It owns the one or two services it modified and an isolated database branch it can migrate and write against, and nothing else. Everything the change did not touch resolves against one shared stable environment, continuously deployed from main, so every tenant validates against real, current dependencies without owning a copy of them.

A footprint that small makes tenant creation nearly free, and creation cost is what decides whether the model scales to agent demand. Isolated data no longer requires copying a database: Neon and Xata create copy-on-write branches in seconds regardless of dataset size, consuming storage only for the data that diverges. The runtime side costs one deployment, because starting the modified services is all that is left to do. A tenant that costs one deployment can be created hundreds of times a day.

Tenants onboard and offboard themselves

Multi-tenant platforms scale because nobody provisions tenants by hand. Signup creates the tenant, cancellation removes it, and no operator sits in the loop. Change tenants need the same contract. The tenant comes into existence when work on the change starts and disappears when the change merges or is abandoned, with no ticket at the front and no cleanup script at the back.

Offboarding is the half that platform teams underestimate. At person scale, an orphaned environment was a minor waste found in a quarterly cleanup. At change scale, orphans accumulate as fast as agents abandon experiments, and the leak outgrows the cleanup.

Automatic offboarding also keeps the accounting accurate. When tenants are created and destroyed by the change’s own lifecycle events, the number of live tenants equals the number of changes in flight, and platform capacity becomes a quantity you can measure and plan against instead of a pile of environments nobody is sure anyone still uses.

Re-measure the platform in changes, not seats

The practical shift for platform teams starts with measurement. Count changes in flight at peak, not seats: open pull requests with activity in the last day is a fine proxy, and for most teams the number is already several times headcount. Then price the marginal tenant: What does one more concurrent change cost in dollars and in minutes of setup? If the answer is a full environment and tens of minutes, the platform is still doing person-level tenancy.

Those two numbers expose where the old assumptions live. Namespace quotas sized per developer, staging booked by team calendar, database seeds refreshed nightly for everybody at once: each is a seat-denominated policy waiting to fail under change-denominated load. The fix in every case is the same three requirements: near-free creation, isolation sized to the change, lifecycle bound to the change, applied to whichever part of the platform still assumes the tenant is a person.

Change-level tenancy is the prerequisite for an agent-native SDLC

Every previous definition of the tenant named a person or a group of people, and that held because only people produced changes. A platform could equate one seat with one workstream, plan capacity from the hiring plan, and keep a human in the provisioning loop. Coding agents break all three of those properties at once: one person now operates several concurrent workstreams, those workstreams are created and abandoned at machine pace, and no human is positioned to provision or clean up each one.

“The change is the only unit of isolation that stays stable when the workers become software.”

That is why the software development lifecycle (SDLC) needs its tenant redefined around the change rather than around whoever, or whatever, wrote the code: the change is the only unit of isolation that stays stable when the workers become software. 

Organizations that keep person-sized tenancy will watch agent-generated changes queue behind infrastructure built for a fraction of the load. The ones that re-platform around the change will convert agent throughput into merged work. If you’re exploring the second path, that is exactly what we built Signadot to support.

The post Per-developer environments were the goal. Agents moved the goalposts. appeared first on The New Stack.