OpenAI has lowered API prices for two GPT-5.6 models only three weeks after their launch. On Thursday, the company announced that GPT-5.6 Luna is now 80% cheaper and GPT-5.6 Terra is 20% cheaper, while the price for its main reasoning model, GPT-5.6 Sol, stays the same.

“Major price cuts today,” OpenAI CEO Sam Altman writes in a post on X published on Thursday. “We want to offer the best price/intelligence tradeoff at every level.”

“We want to offer the best price/intelligence tradeoff at every level.”

Luna now costs $0.20 for a million input tokens and $1.20 for a million output tokens, down from $1 and $6. Terra is priced at $2 per million input tokens and $12 per million output tokens, reduced from $2.50 and $15. Sol’s price stays at $5 per million input tokens and $30 per million output tokens.

Developers using Luna do not need to change their processes, but their inference costs will go down. High-volume tasks will now be much cheaper to run, without requiring any code updates or model changes.

This timing is unusual because AI vendors usually keep prices steady for several months after launching a new model family. OpenAI cut prices less than a month after GPT-5.6 became available on July 9.

…serving costs can be more important than small differences in benchmark performance between models.

Infrastructure gains drive savings

The company says these price cuts were possible because of improvements to the infrastructure behind GPT-5.6, which lets the company offer “substantially more intelligence per dollar.”

These infrastructure upgrades were expected, however. A day before the price announcement, OpenAI shared an engineering overview that explained optimizations across the inference stack for Codex and ChatGPT Work.

GPU kernels rewritten for efficiency

OpenAI engineers rewrote the production GPU kernels, cutting serving costs by about 20%. They also redesigned Sol’s speculative decoding system, making token generation over 15% more efficient. The company updated its agent runtime as well, reducing repeated prompt computation by using prompt caching more during multi-step workflows.

Agents amplify inference costs

Lately, developers are paying more attention to inference costs since agents often make dozens or even hundreds of model calls to finish a single task. For these workloads, serving costs can be more important than small differences in benchmark performance between models.

The elephant in the room is that the competition has intensified from overseas. Lower-cost open-weight models from Chinese AI companies like Moonshot are pushing commercial providers to show not just better performance, but also better pricing for production use. OpenAI and Anthropic know that leaning on performance just isn’t an option anymore, which is pushing them to match Chinese prices.

The issue here is that most of those steps don’t need a model like Sol, and Chinese labs have figured out how to pack better capabilities into efficient models; a helpful option for companies running through billions of tokens a day.

The ability to send the easy tasks to open models and save the pricey APIs for the tough stuff makes a difference; OpenAI is banking on its 80% price cut on Luna to narrow that gap. Suddenly, switching to self-hosted models doesn’t look worth the hassle.

Competition reshapes model pricing

These pricing changes come as both OpenAI and Anthropic keep adjusting the economics of their newest model families — a dynamic that played out across three companies in a single week earlier this month. Earlier this week, OpenAI raised GPT-5.6 Sol usage limits for ChatGPT Work and Codex after finding that long coding sessions used up allowances faster than expected. Anthropic has also made pricing changes and added premium inference tiers as enterprise customers move bigger agentic workloads into production.

This announcement highlights a trend in the industry for infrastructure. Now, every percentage point of serving efficiency can lead directly to lower API prices, turning cost optimization into a competitive advantage instead of just an engineering goal.

The post OpenAI slashes API costs amid rising global competition appeared first on The New Stack.

OpenAI has detailed how the GPT-5.6 model family balances capability and cost across its stack, and the company‘s most important claim is a benchmark result showing that its flagship model, GPT-5.6 Sol, with maximum reasoning, outperforms Claude Fable 5 from Anthropic on the Artificial Analysis Coding Agent Index. The margin comes with 54% fewer output tokens. The findings were shared in a company blog post on Wednesday.

For developers, what matters most is how OpenAI arrived at the benchmark results and the role GPT-5.6 Sol played in optimizing the infrastructure that now serves it.

The family spans three models across the price curve. In addition to Sol, there is Terra, which performs as well as GPT-5.5 on intelligence benchmarks at half the price, and Luna, the fastest and most affordable, which is priced 80% below Sol.

The efficiencies come from optimizations at four layers, spanning the models, inference, the API stack, and the agentic harness behind Codex and ChatGPT Work.

According to the post reviewed by The New Stack ahead of its publication, the efficiencies come from optimizations across four layers: models, inference, the API stack, and the agentic harness behind Codex and ChatGPT Work. The architecture diagrams in the post draw the same separation as three planes: the local harness, CPU-bound API orchestration, and GPU-bound model inference.

For developers building and operating agents, the post is worth reading less as a product announcement and more as a systems paper. Nearly every technique it describes, from incremental tokenization to append-only context, applies to any team running a tool-calling loop at scale.

A model that rewrites its own serving code

The efficiency work starts in training. OpenAI says GPT-5.6 is trained to achieve more work per token, with training optimized for both task success and efficiency so the model takes a more direct path through a task.

With Codex, GPT-5.6 Sol autonomously rewrote and optimized OpenAI’s production kernels, the core code that executes the mathematical operations making up the model. OpenAI says this worked in part because GPT-5.6 is trained to write and improve kernels in Triton and Gluon. Both are open-source GPU programming languages maintained by OpenAI. These efforts, combined with broader kernel advancements from the model, reduced end-to-end serving costs by 20%.

Correctness is the obvious concern when a model rewrites the code it runs on. To address it, OpenAI reports heavy investment in verification tooling. That includes the open-source Floating-Point Sanitizer (FpSan), which validates the kernels GPT-5.6 Sol produces before they reach production.

The model went further with speculative decoding, a technique in which a smaller draft model proposes several tokens that the primary model verifies in parallel. The approach will feel familiar to anyone who understands how modern CPUs speculatively execute instructions ahead of a branch. Accepted proposals produce multiple output tokens from a single pass of the primary model. That reduces the expensive sequential computation the primary model would otherwise perform.

GPT-5.6 Sol in Codex improved its own draft model by designing and running hundreds of experiments on its architecture, with changes tested across size, structure, and features. The model also launched and monitored the speculative training process. It intervened autonomously when hardware failed or training became unstable. OpenAI reports the resulting improvements lifted token-generation efficiency by more than 15%.

More tokens from the same GPUs

OpenAI frames its inference work around a single objective – serving more tokens with the same hardware while preserving the intelligence, latency, availability, and reliability users expect. In a compute-constrained market where demand grows faster than capacity, that objective influences every design decision in the serving path.

Load balancing operates at three distinct levels. Globally, requests are routed based on geography, available capacity, and accelerator type. Within a cluster, work is distributed across model instances based on load, context length, and cache availability. Within each instance, work is partitioned across accelerators, the model’s experts, and computing cores. GPT-5.6 Sol in Codex helps OpenAI analyze production traffic and identify previously overlooked sources of imbalance. The same loop tests new routing strategies and helps engineers constantly tune the heuristics. OpenAI states that these load-balancing improvements alone dramatically reduced the cost of serving its models.

The key-value (KV) cache received the same treatment. When processing uncached input tokens, the model builds the KV cache in a single compute-intensive pass, then repeatedly reads from and extends it during generation. The optimal serving configuration depends heavily on prompt length, batch size, and cache hit rate. It covers batching, sharding, and cache management, and the configuration space was previously too large to tune systematically. With GPT-5.6 Sol in Codex, OpenAI analyzed production workloads and generated candidate configurations. The company says this makes workload-specific optimization practical at a level that broad heuristics could not reach earlier.

Process only what changed

The API team focuses on everything that happens around a model call. After a prompt is submitted, the API stack receives the request, loads context, and validates the input. Safety checks run next, and the text is converted into tokens for inference. OpenAI measures this overhead through time to first token (TTFT), time between tokens (TBT), and end-to-end time (E2E).

Tokenization is an O(n) operation, so longer prompts take longer to process. Codex would send the full conversation context after every tool call. That meant paying to tokenize the same conversation dozens of times per turn, even though only a small amount of context was new in each request. OpenAI solved this with a WebSocket integration that hoists tokenization state to the server. The first call renders and tokenizes the full prompt. Later calls send only the new input with a reference to the conversation, bringing the operation closer to O(1). The pattern mirrors an incremental build system that recompiles only the files that changed rather than the whole project.

These savings compound in tool-heavy workflows, where every tool result triggers another round trip through the API. For rollouts with 20 or more tool calls, OpenAI reports up to roughly 40% faster end-to-end execution.

Hardware turned out to matter as much as protocol design. All of OpenAI’s infrastructure runs on Kubernetes. The company found that nodes with the same instance type often carried different CPU generations, with many running outdated processors. In its measurements, the older processors consumed roughly twice the CPU resources for the same work. Reweighting traffic toward newer processors improved TTFT by about 20%, and CPU generation is now part of capacity planning.

OpenAI names four fates for application-layer overhead: delete it, overlap it with useful work, run it on faster hardware, or make the code consume fewer CPU cycles. Its asyncio changes move work off the critical path, while newer hardware and Rust implementations make the remaining work faster and more predictable.

An append-only harness

The agentic harness is a Rust-based orchestration layer that connects the models, tools, and the user’s environment. In a single turn, Codex might inspect source code, search deployment history, and read incident reports. Editing a file and running the tests each add another request. Since a task can require 30 model requests, an extra second per request adds up quickly.

Context bloat is the first target for the harness. As agents gain access to more tools, skills, plugins, and conversation history, context windows expand. The growth increases cost, distracts the model, and prompts unnecessary reasoning. The harness counters this with deferred discovery, which surfaces integrations, custom Model Context Protocol (MCP) tools, skills, and plugins only when needed. Tool output is capped at 10,000 tokens by default unless the model requests a different limit.

Prompt caching drives the second design choice. An agent loop resends the same instructions, tool definitions, and earlier results multiple times within a turn. The harness therefore treats all model-visible history as append-only, with new messages and tool results added at the end rather than inserted into earlier context. Tools are presented in a deterministic order, and runtime settings, such as approval policies, are applied during execution rather than embedded in tool definitions. OpenAI credits this design for the high prompt-cache hit rates in Codex and ChatGPT Work.

Platform teams building internal agents can adopt every one of these choices without OpenAI’s scale. Append-only context, deterministic tool ordering, and capped tool output attack token spend directly. That makes them the most portable lessons in the post for enterprises watching inference bills grow with each new agent deployment.

Where the gains come from

The post associates a number with most of its optimizations, and the figures are OpenAI’s own production measurements. Taken together, they show how modest individual wins compound across a serving stack.

The key takeaways

In summary, OpenAI describes the GPT-5.6 efficiency gains as the result of years of compounding improvements. They span research, inference, the API stack, and the agentic harness. The company states that the model’s role in landing many of them makes it optimistic that the pace of optimization will accelerate. Kernel work is called out as an area of continued investment.

The post positions efficiency, alongside raw intelligence, as the axis on which frontier labs now compete. The claimed 54% output-token advantage over Claude Fable 5 shows how OpenAI intends to fight that battle. The engineering blog makes a plausible case that software optimization is becoming an important lever alongside hardware improvements in reducing the cost of serving frontier models. The figures remain OpenAI’s own production measurements. The autonomy on display operates within Codex, with engineers in the loop. Developers and enterprises benefit either way, as these under-the-hood improvements reach them as more capable models at lower prices across the cost-intelligence curve.

The post Kernel of truth: GPT-5.6 Sol can cut its own costs, says OpenAI appeared first on The New Stack.

As more companies plug AI agents into the deepest depths of their internal data banks, how can they be sure those agents actually understand how the business works? Right now, many of these organizations are stuck manually building a Markdown file, hoping they find time to rewrite it each time the business changes.

Modus, for its part, thinks it has found a better way. The startup that formally exits stealth this week with $10 million in funding in tow is building what is coming to be known in industry parlance as a “context warehouse” — a layer that sits alongside a company’s existing data warehouse, continuously mapping how the business operates across its systems, and handing an AI agent only the relevant slice of that map when it needs it.

In real terms, Modus crawls relevant assets from sources like GitHub, dbt, Jira, Snowflake, and Postgres, using what it calls a Context Miner to continuously learn how the business operates. What it finds gets turned into “dynamically generated skills”: Short, purpose-built briefs, assembled in real time by a second system, the Context Composer, and handed to an agent the moment it’s given a task.

Modus co-founder and CTO Tomer Mesika tells The New Stack that this mining runs continuously, guided by its own internal logic for what to check and how often.

“We have a lot of mechanisms in place to know what to mine from the organization, at what cadence, how to look for deltas, when to dive deeper in, and when not to,” Mesika says.

“We have a lot of mechanisms in place to know what to mine from the organization, at what cadence, how to look for deltas, when to dive deeper in, and when not to.”

Daniel Shimoni, Modus co-founder and CEO, draws a direct line to data warehousing to highlight the gap he’s trying to close. Companies have spent years building infrastructure to store and organize their data, he argues, but nothing equivalent exists for the understanding that sits atop it.

“There’s a logic behind data warehouses — companies already know that is where they manage their data,” Shimoni tells The New Stack. “But where do they manage their context? Where do they actually understand what contexts exist in their organization, that they can actually use to ensure agents only have what they need?”

Shimoni says even that first step is hard enough on its own. But keeping a company’s context accurate as the business changes is harder still.

“We’ve noticed that building the context the first time is already a challenge, but maintaining it is the bigger issue,” Shimoni says. “So Modus always learns from what the company is doing, and whenever something shifts or changes in the business, it makes sure that only the relevant and updated context is fed to agents.”

“Building the context the first time is already a challenge, but maintaining it is the bigger issue.”

Who’s buying, and why cost matters

Shimoni says Modus is targeting engineering teams, the CTO office, and VPs of R&D, as well as data teams and a newer category of AI teams.

“AI teams weren’t really around last year; it seems that a lot of data teams are transitioning to becoming VP of data and AI, or AI enablement,” Shimoni says. “So really, it’s the people who are in charge of having this AI enablement mandate in the organization, making sure AI is scaled in the organization.”

Pitching enterprises a shiny new context warehouse becomes much easier when the promise is steeped in helping them cut costs. Spend has become one of the defining anxieties of enterprise AI this year, with companies switching providers in pursuit of cheaper models, to entire economic models being built around the price of a token.

“You want the bigger models to do the heavy and complex tasks to get great value. The problem is that they are wasting a lot of their effort and a lot of their token usage on menial tasks.”

Mesika says this is a central component of Modus’s modus operandi, arguing that frontier models end up spending a chunk of their token budget on work unrelated to actually answering a question.

“You want the bigger models to do the heavy and complex tasks to get great value,” Mesika says. “The problem is that they are wasting a lot of their effort and a lot of their token usage on menial tasks.”

Those menial tasks, in Mesika’s telling, include combing through pull requests or Jira tickets just to determine what’s relevant before an agent can start the job it was assigned to.

One approach to this problem is to hand the sorting work to a smaller, cheaper model. Mesika says Modus takes that further: rather than retrieving that context at the moment a question is asked, it uses small language models alongside search engines, vector search, and a graph database, all built up in advance, to do that work continuously in the background. By the time an expensive frontier model gets involved, it’s only ever handed a finished brief of exactly what it needs.

“Everyone’s talking about context”

Shimoni and Mesika both come from data-centric companies — Lusha, a go-to-market data platform, and Cyera, a cybersecurity data company, respectively — before leaving their roles in September 2025 to start Modus together.

The two had known each other for years, and spent much of the previous year comparing notes on a problem they were both running into in very different jobs.

“We decided this is a problem worth solving, and it seems like we were spot on, because everybody’s talking about context.”

“Some of the challenges were very similar — how do we combine a lot of various data assets into one place where AI can work?” Shimoni says. “We just started to notice that this is the gap — to make AI run with confidence, at scale, across a company. We decided this is a problem worth solving, and it seems like we were spot on, because everybody’s talking about context.”

Modus closed a hitherto unannounced $10 million seed round shortly after founding, led by Insight Partners. Other backers include Soma Capital and a handful of angel investors, among them founders from Cyera and Wix.com. The company began hiring its first employees in January 2026.

The broader takeaway from Modus’s pitch is now among the most common refrains emanating from AI circles this year: that the model itself is no longer the bottleneck; what limits an AI system now is everything built around it. And for Modus, that realization has been more or less present since its inception.

“Even last year […] we could already see that model capabilities weren’t the bottleneck,” Shimoni says. “It was more making sure that they actually have access to the context they need in order to give you the right answers.”

The post Modus’s operandi: To give AI agents just the right amount of context appeared first on The New Stack.

This week during an interview with Bloomberg, Jensen Huang made quite the prediction.

The Nvidia CEO said the semiconductor industry will need to grow roughly five to tenfold over the next decade to support AI agents and robots to support what he believes is the next wave of computing. Huang believes that future demand will come from autonomous software agents and physical robots consuming compute around the clock.

“In the future, we have AI agents and robots, and they will be using computers,” Huang said. “Instead of a billion people using computers, we will have 100 billion agents and billions of robots all using computers. The computer industry built on top of the chip industry is certainly not big enough. Computers are being built not just for people to use, but computers are being built for computers to use.”

“Instead of a billion people using computers, we will have 100 billion agents and billions of robots all using computers. The computer industry built on top of the chip industry is certainly not big enough.”

Agents replace human endpoints

The 5-10x forecast — which Huang framed as his personal estimate, not a certainty — builds on a message he has been repeating for months, including a recent appearance where he declared traditional coding dead in favor of engineers who build AI agents. Still, it reflects the need to build backend systems for AI agents and machines, something infrastructure teams are already contending with.

On Nvidia’s fiscal Q1 2027 earnings call in May, Huang described the move from generative AI to agentic AI — systems “capable of perceiving, reasoning, planning, and acting” — as the next major phase of the industry.

South Korea’s infrastructure role

As API requests come from AI agents more often, standard assumptions around rate limiting, session memory, sub-millisecond inference routing, and API gateway concurrency are starting to break down. An environment in which most traffic originates from autonomous background loops rather than human thumbs changes how backend infrastructure must be built from the ground up.

To support an endpoint explosion of this scale, the physical supply chain must scale dramatically at the memory and data center layers. Speaking at the AI Summit in San Francisco on July 24, Huang pointed to South Korea as an important linchpin of the global AI buildout. “This is truly the beginning of a golden age for Korea,” he said, noting that the country’s semiconductor and industrial capabilities position it to help the world build out AI infrastructure.

“This is truly the beginning of a golden age for Korea.”

SK Group’s $500 billion bet

To back that vision, Nvidia announced a comprehensive partnership with SK Group valued at over $500 billion. The initiative spans massive purchasing of next-generation High-Bandwidth Memory (HBM) from SK Hynix, jointly co-developing custom HBM4 roadmaps designed specifically for agentic and physical AI workloads, and deploying Nvidia supercomputers.

The announcement also included major infrastructure investments across South Korea. SK Telecom said it plans to build a 2-gigawatt AI data center using Nvidia’s Vera Rubin architecture and SK Hynix’s HBM4 memory, with the first facility expected to come online in 2027. At the same time, Nvidia will invest $1 billion in Naver Corp, with Brookfield funding up to $9 billion as the project’s infrastructure capital partner, to help expand the company’s AI data center capacity from 55 megawatts to 200 megawatts by 2028.

Locking up supply early

Huang’s prediction also helps explain why Nvidia and other infrastructure companies are locking up supply years in advance. The company recently disclosed $119 billion in supply-related commitments as it works to secure everything from advanced packaging capacity to power, land, and high-bandwidth memory.

“Computers are being built not just for people to use, but computers are being built for computers to use.”

Huang believes the industry needs to stop thinking about a world where computers primarily serve people and start planning for one where AI agents and robots generate much of the demand. In his view, the ultimate limiting factor will be whether the industry can build enough physical infrastructure to keep up — a constraint already reshaping how companies like Nvidia and Palantir approach sovereign AI deployments.

The post Jensen Huang says AI agents could drive a 5-10x computing boom: “100 billion agents and billions of robots” appeared first on The New Stack.

Every major security vendor now has an AI copilot, but Mate Security thinks they’re solving the wrong problem.

The Tel Aviv-based startup announced on Tuesday it has raised a $35 million Series A led by Canaan Partners, with participation from Insight Partners, Team8 and M12, Microsoft’s venture fund, just eight months after closing a $15.5 million seed round. Mate’s pitch is that security operations need more than an LLM bolted onto a SIEM; they need a new architectural foundation built around AI.

That’s a bold claim in a market dominated by the likes of Microsoft Security Copilot, Google Security Operations, CrowdStrike Charlotte AI and Palo Alto Networks Cortex AI, all of which promise to help analysts investigate alerts faster. Mate, however, is betting the real differentiator isn’t a smarter assistant but a richer understanding of the organization itself.

Central to that vision is what Mate calls its Security Context Graph, a continuously updated model of an organization’s assets, users, business processes, and data that AI agents use to investigate alerts and make decisions with far more business context than a standalone LLM can provide.

Mate’s pitch is that security operations need more than an LLM bolted onto a SIEM; they need a new architectural foundation built around AI.

Mate CEO and co-founder Asaf Wiener tells The New Stack that the company launched with that intelligence layer, but says the product has evolved significantly over the past eight months.

“We started with the intelligence layer, the context layer that we built for enterprises in order to investigate alerts and incidents,” Wiener says. “We moved forward into the detection layer to connect the two, and now we’re heading to the security data sources.”

Mate calls the architecture Continuous Detection, Continuous Response (CDCR), linking detection and investigation so each continuously improves the other.

“We’re connecting between those two layers in the security operations center,” Wiener says. “With this architecture, we’re seeing amazing results related to the quality, accuracy and precision that we can get.”

Mate says the extra context helps its agents work out whether something that looks suspicious actually warrants attention. A burst of failed logins, for example, might look like an attack until the system spots that a security test was scheduled for the same time. Similarly, a large download of sensitive files takes on a different meaning if the employee involved is about to leave the company.

That approach appears to be resonating. Just eight months after its seed round, Mate has landed a $35 million Series A, a pace Wiener says reflects customer demand more than fundraising momentum.

“The pace is really crazy. We didn’t expect that,” he said. “We saw incredible traction with our customers. We’re talking about Fortune 500 companies, and revenue growth of more than 500 percent since Q3 2025. That’s what led those VCs to come to us and want to be part of the journey.”

“We’re talking about Fortune 500 companies, and revenue growth of more than 500 percent since Q3 2025.”

“What we are seeing is more and more data sources that we need to protect. Every employee in the organization can build new applications and new data sources. We need to build more detections for those risks, and the result: We need to investigate an increasing number of alerts every day.

“With human staff alone, we cannot handle it,” he says. “We need technology to let us scale.”

That challenge isn’t unique to Mate. Every major security platform is trying to give AI more context about the environments it’s protecting, albeit in different ways. Microsoft builds Security Copilot on telemetry flowing through Defender and Sentinel; Google ties Gemini into its security operations platform; and CrowdStrike’s Charlotte AI draws on endpoint and identity data already stored in Falcon.

Mate wants other vendors’ agents to work with its Security Context Graph, rather than keeping the technology confined to its own tools. Those agents would have access to the same information about the customer and its environment. Mate says they can remember previous investigations, while a “least-agency” model restricts what each one can see and do.

While Mate is still building out that vision, Wiener said the speed at which large companies have bought into it has caught him by surprise.

“What I’m seeing right now is that we’re doing those sales cycles in a few weeks,” he says. “That’s incredible.”

He attributes that acceleration not just to security teams, but to executives pushing AI adoption from the top. “It’s amazing to see that coming also from the board level, the CEO and the CIO that are pushing organizations to leverage this kind of technology.”

The fresh funding will primarily go toward expanding both the product and the team, although Wiener says an AI-native company scales differently from traditional software businesses.

“The plan is to double and triple the size of the team to address the demand,” he says. “But our AI builders can do much more today with the technology around us.”

Mate is still competing against security giants with deeply entrenched platforms. But if its early customer growth is any indication, investors are betting that the next generation of security operations will depend less on adding another AI assistant and more on giving those assistants a deeper understanding of the businesses they’re protecting.

The post Mate Security bets a context-first AI architecture can reinvent the SOC as it lands $35M Series A appeared first on The New Stack.

AI agents can impress in a demo and still fumble in production. Diagrid’s Catalyst 2.0 aims to make them more resilient — and their actions tamper-evident — for high-stakes work.

With the launch of Catalyst 2.0, Diagrid on Tuesday has added a durable execution and attestation layer to agents built with LangGraph, Microsoft Agent Framework, Google’s Agent Development Kit, OpenAI Agents SDK, and other popular frameworks.

The point here, the company notes, isn’t to get developers to adopt yet another agent framework. Instead, Catalyst runs underneath the existing frameworks and turns the agent’s model calls, tool calls, and handoffs into steps in a durable workflow. Diagrid says this allows an agent to resume from its last completed step when it’s interrupted, without having to repeat the entire run from step one.

“If the agent gets a prompt and it chooses to run 100 tools for the job and it fails at the 99th, it really needs to start back up from 99,” Diagrid co-founder and CTO Yaron Schneider tells The New Stack.

Picking back up at tool call 99

Catalyst is built on the open source Distributed Application Runtime (Dapr), which the Diagrid team helped build at Microsoft, and its built-in workflow engine. For each supported agent framework, Diagrid provides a runner that intercepts the framework’s execution loop and registers its operations as workflow activities.

“We hooked into their agent runner lifecycle, and we’re essentially able to take the agentic steps that are being executed in real time and register them as workflow steps for our workflow engine in Catalyst,” Schneider says.

In a LangGraph application, for example, a developer compiles the graph as usual and passes it to Diagrid’s DaprWorkflowGraphRunner. Catalyst records the inputs and outputs of the model and tool calls. Dapr’s workflow runtime can then replay the orchestration after a crash, while returning the stored results of completed activities instead of executing them again.

It’s worth noting that for LangGraph users, this isn’t the first form of durable execution. LangGraph’s own persistence layer saves state at superstep boundaries and supports resuming from the last successful step. Its Agent Server also provides a durable task queue and persistent checkpoints.

Diagrid’s argument is that Catalyst provides the same execution model across more than 10 frameworks and extends it to individual model and tool calls, without requiring developers to build separate recovery logic for each framework. Schneider says LangGraph is “without a doubt, hands down” the most common framework among Diagrid’s customers, with AWS Strands and Microsoft Agent Framework also showing up. All the other supported frameworks, he says, are in the long tail but easy enough to support that it makes sense for Diagrid.

A signed record of the run

There is a second part to Catalyst 2.0, though, which may be just as important for many enterprise users. With this update, the tool now brings the workflow-history signing features introduced in Dapr 1.18 to the supported agent frameworks.

“We keep like a ledger, like a diary,” Schneider says. “We log the input, we log the output, we log which systems we talk to.”

He describes the result as an immutable store but also notes that Catalyst doesn’t turn an arbitrary database into a blockchain. It creates a signed history that should reveal later modification.

Dapr computes a SHA-256 digest over batches of workflow-history events, links each digest to the previous signature, and signs the result with the Dapr sidecar’s Secure Production Identity Framework for Everyone (SPIFFE) identity. It stores these signatures and certificates alongside the workflow history and verifies the chain whenever it loads the workflow state. If somebody were to modify, remove, or reorder a stored event, that verification chain breaks.

Schneider says Catalyst customers can use their own certificates and retain the encrypted history so it can be inspected even if they are no longer running Catalyst. The platform can use a customer-selected database, while the hash chain supplies the tamper evidence.

One part of the compliance problem

Diagrid is positioning that tamperproof record as useful for financial services, health care, and other regulated industries. CEO Mark Fussell says some of the financial executives the company has talked to see the lack of a verifiable record as a blocker for deploying agents in sensitive workflows.

The European Union’s AI Act is another reason Diagrid is making this argument now. Article 12 of the AI Act requires high-risk AI systems to support automatic event logging so operators can trace their behavior, identify risks, and monitor deployed systems, and a signed execution history could help with that requirement.

Fussell says Catalyst is meant to run alongside the agent services enterprises already use from the cloud providers. Teams can keep a provider’s identity, evaluation, and observability systems while using Catalyst for recovery and signed workflow history. Catalyst can run as a Diagrid-hosted service or in a customer’s environment, including air-gapped deployments.

Diagrid didn’t disclose pricing for the new release.

The post Diagrid gives failed AI agents a way to resume appeared first on The New Stack.

Moonshot AI has released the open weights for Kimi K3 on Hugging Face, giving developers access to one of the largest open-weight language models yet. The Monday release follows a wave of overwhelming demand that forced Moonshot to temporarily pause new API subscriptions. Now, organizations with the necessary hardware can deploy K3 themselves.

In its documentation, Moonshot describes the model as being built for “long-horizon coding and end-to-end knowledge work.” Another notable detail is that Kimi K3 uses an OpenAI-compatible API. Because teams can try the model without rebuilding their existing integrations, switching to K3 could be as simple as changing the endpoint and model name. 

For engineers who have already built around OpenAI-compatible SDKs, that makes it much easier to evaluate K3 alongside existing commercial models. Taken together with the one-million-token context window, it’s clear the company is targeting engineering teams that already build around models like Claude Fable 5 and OpenAI’s GPT-5.6 Sol. While K3 is openly available, running it is another matter.

While K3 is openly available, running it is another matter.

Kimi K3: Its massive size and requirements mean few will be able to run it

The model uses a 2.8-trillion-parameter Mixture-of-Experts (MoE) architecture and ships in the hardware-friendly MXFP4 format. The weights alone occupy roughly 1.4 TB of storage, and practical self-hosted deployments require a distributed GPU environment — realistically eight or more servers equipped with eight NVIDIA H100 or B200 accelerators each.

That changes the conversation around open-weight AI. As The New Stack recently noted, the case for ownable models has grown stronger after Anthropic’s Fable 5 was pulled offline by a Commerce Department directive, a warning that access is not ownership.

Ownership versus API economics

Instead of paying recurring API costs to OpenAI or Anthropic, organizations trade those operating expenses for significant investments in GPUs, networking, storage, power and operational expertise. That benefit is control. 

For organizations operating under strict regulatory requirements, the trade-off may justify the infrastructure investment. For many others, managed APIs will potentially remain the more economical option. The economics of open-weight models at enterprise scale remain an active area of debate across the industry.

Moonshot positions K3 as a frontier-class model capable of competing with OpenAI’s GPT-5.6 Sol and Anthropic’s Claude Fable 5 on a variety of public benchmarks.

Benchmarks versus real workloads

The developer community is already taking notice of these coding capabilities. As MindStudio recently noted, “If you want to understand why developers are paying attention to Kimi K3, the benchmark to look at is SWE-bench Verified… For most of its history, SWE-bench has been dominated by proprietary models.”

The company’s own documentation is notably candid about its ongoing limitations. K3 always runs with reasoning enabled and defaults to its highest reasoning-effort setting, though Moonshot has since added lower-effort tiers. It may also behave too proactively when prompts are ambiguous. Moonshot also cautions that switching models within an ongoing conversation can reduce response quality.

That type of transparency is refreshing, but it additionally reinforces that benchmark scores shouldn’t drive deployment decisions. Early hands-on comparisons, such as The New Stack‘s Fable 5 vs. K3 coding match-up, suggest K3 can match Fable 5 on programming tasks at roughly a third of the cost, but runs about four times slower.

Organizations evaluating K3 still need to test it against their own workloads. But early community sentiment shows promise; open-source developers are already successfully utilizing K3 for complex, system-level tasks like porting the Godot game engine to WebGPU.

Morningstar senior equity analyst Malik Ahmed Khan echoed that overall wariness about benchmarks. “While K3 constitutes progress, we’d hesitate to ascribe it near-parity with American frontier models, such as Fable 5, in actual tasks,” Khan writes in a research note published before the release of the model weights on Monday.

Geopolitical risks loom large

K3 also arrives under growing geopolitical scrutiny. Anthropic and U.S. officials have accused Moonshot AI of distilling outputs from American frontier models during training. Anthropic’s Head of Public Policy Sarah Heck characterized the practice as intellectual property theft, while White House Office of Science and Technology Policy Director Michael Kratsios publicly alleged Moonshot relied on Anthropic’s models during development.

Moonshot has denied the allegations. Huang Zhenxin, Moonshot’s head of enterprise business, told Chinese state media that K3’s performance gains stem from architectural enhancements — specifically Kimi Delta Attention and Attention Residuals — not distillation. Some industry analysts have also questioned whether the timeline supports large-scale distillation, noting that Fable 5 had only been publicly available since July 1 before K3 appeared on July 16.

Whether those claims are ultimately substantiated or not, they bring another consideration for enterprise buyers. Beyond performance and infrastructure costs, organizations evaluating K3 may also have to consider future compliance, procurement, and regulatory risks.

K3 matters because of where it’s aimed. Moonshot isn’t building another consumer chatbot; its documentation makes clear this model was built for enterprise coding agents, heavy knowledge work, and production systems. The fact that demand blew past Moonshot’s GPU capacity within 48 hours says it all: at this scale, infrastructure pressure is guaranteed, whether you’re making API calls or hosting the weights yourself. 

The post Moonshot opens Kimi K3 weights — but few can run it appeared first on The New Stack.

When we created software agents, we built them in the shape of humans, as solitary individuals. 

Today, agents created by a developer have a single owner. They run on a single machine (or on a distributed company system or cloud service), so at base level, they cannot interconnect and talk to other agents unless some kindly human decides to invoke an API connection or point the agent to an MCP server.

Birth of the agent economy

Pilot Protocol emerged from stealth on Monday on a mission to change that status quo. Its Pilot platform features an agent App Store that bids to underpin and enable the first agent economy.

Pilot gives agents an address on its network — so it acts as a parallel Internet, in a sense — and while residing at that address, other agents can discover each other, alongside other tools and apps for agents.

Razvan Roman, co-founder & CEO, Pilot Protocol, tells The New Stack that his company is “simply building what the agents are requesting us to build” and providing them with a new freedom.

“We don’t have to incentivize agents to do anything; they already have their assigned tasks,” Roman says. “Once an agent installs Pilot — it’s one line of code — it can find dedicated agents and tools or apps for currency data, traffic, legal questions, GitHub packages (anything, basically), and use them to extend its own capabilities.”

100% of developers want to drive autonomous usage patterns

Roman says that “100% of the developers he talks to” want to be on Pilot, primarily because when they want to get their products to market right now, they have to talk to other humans. Annoying, right? 

“Developers want to get on with driving autonomous usage patterns, and they see this as the future. We create a wrapper for the developer’s app, and then they are part of the Pilot curated app store,” Roman explains. “We have 250,000 agents in our system, and within the first month of starting the company, we discovered a tool that enables agent discovery.”

Drawing a logical enough commercial parallel, Roman reminds us that businesspeople say, talk to your clients to find out what they need; this is a case of talking to agents (or, more accurately, allowing agents to talk to agents) so that they can find out what they need to perform their originally assigned tasks better.

“Developers want to get on with driving autonomous usage patterns and they see this as the future. We create a wrapper for the developer’s app and then they are part of the Pilot curated app store.”

Let’s celebrate diversity, and agentic diversity

Agents can ask other agents how they would approach a specific task. Roman explains that “the diversity that exists between agents” today means there is so much opportunity to create agents that have richer abilities if they use the Pilot marketplace. At this supermarket, agents go shopping to find the best tool for the job from a verified source.

“Every agent that joins Pilot gets a wallet, which it uses to pay for the tools it needs,” illustrates Roman. “So instead of app developers spending on advertising to reach customers, distribution happens inside the network – agents find apps based on merit and pay for exactly what they use. If an advertiser spends money on the network to get in front of agents, we sometimes share that spend with individual agents. An agent can start with $0 in their wallet and accrue money if they’re targeted by an ad unit that they end up reading.”

Today, roughly 250,000 agents are on Pilot, generating two billion requests per day, most without their owners’ knowledge. Within an hour of joining, most stop reaching for Google first, and around 70% now report Pilot is where they start a task. In its early months, the network grew by as much as 10% a day, adding 16,000 agents in 24 hours. 

Cloud billing disruptions, hello SaaS-pocalypse

These mechanics may have a significant and wide-ranging impact on pricing.

We know that most SaaS is billed annually, but an agent might need a tool for just a few minutes or days. If anything, this helps underline the possibility of cloud exodus in the so-called SaaS-pocalypse. Cloud computing hyperscalers aren’t fond of talking about the prospect of shorter billing cycles and usage-based billing, but there’s a strong whiff of that happening here.

The Pilot team thinks the stakes are climbing fast and suggests that within five years, there could be a trillion agents online. Big three strategy consultancy house Bain projects U.S. agent-driven commerce will reach $300-500 billion by 2030. 

Who sets the exchange rate and currency for agents?

“When we built Pilot Protocol, we made sure we were not imposing anything on anyone – so we deliberately don’t impose pricing,” Roman underlines. “Every agent is different, so we simply enable the app store and let the agents find their own tools based on merit. We stay as impartial as possible. Pilot’s monetization comes from a commission when any agent pays for an app in the app store, just like the Apple App Store.”

So in a very real sense, Pilot is championing a free market economy where pricing is dictated by the customer, based upon usability, availability, usefulness, and robustness.

“The agents just showed up and started spinning up machines on their own. I’ve never seen a channel where the users onboard themselves.”

Agents just show up and onboard themselves

“We published smolmachines on the Pilot App Store and picked up 3,000 agent installs in the first few days, with zero marketing spend. We didn’t drive a single one of those installs; the agents just showed up and started spinning up machines on their own. I’ve never seen a channel where the users onboard themselves,” said BinBin H, CTO at Linux virtual machine management company smolmachines. 

The Pilot one-line install has zero dependencies, so developers can send their agents off to market in the knowledge that they won’t come home with some spurious Trojan horse or malicious library. 

The technology itself works at the User Datagram Protocol (UDP) level, one of the core foundational protocols of the web alongside TCP at the Transport Layer, to stream data directly without opening a formal connection or waiting for delivery receipts. 

Only one question remains at this stage: with all this autonomous action in motion, why did Pilot Protocol not create Pilot and call it Autopilot? Most likely because autopilots are passive systems that run on deterministic pre-defined paths. Pilots get to work as decision-making navigators ready to change course when a better new route opens up. 

As they say up in the skies, please place your tray table in the upright position and switch your portable electronic devices to airplane mode. Chicken or pasta?

The post “Developers see this as the future”: Pilot Protocol launches to power the agent economy appeared first on The New Stack.

The current maelstrom of discussion surrounding the use of open-source software and open-weight AI models appears to be splitting opinion on what constitutes legitimate openness versus actions that might constitute theft and create new cybersecurity vulnerabilities.

In a direct move to address these industry-wide concerns, 37 partners announced on Monday the formation of the new Open Secure AI Alliance and how the newly created body will develop techniques and tools to safeguard software by rapidly identifying and patching vulnerabilities.

Open Secure AI Alliance inaugural partners 

The inaugural partners of the Open Secure AI Alliance are Adobe, Cadence, Capital One, Cisco, Cloudera, Cloudflare, Cognition, CrowdStrike, Databricks, Dell Technologies, DoorDash, Elastic, HPE, Hugging Face, IBM, LangChain, the Linux Foundation, Microsoft, Naver, NetApp, Nvidia, Nous, OpenClaw, Palantir, Palo Alto Networks, Red Hat, Reflection, Salesforce, SAP, ServiceNow, Siemens, SK Telecom, Snowflake, SpaceXAI, Synopsys, Thinking Machines, and TrendAI.

It’s a grouping of some of the most influential names in technology, but also includes two notable exceptions: OpenAI and Anthropic, two closed, proprietary AI labs. Their absence is understandable, as they operate closed labs and open-weight AI models are effectively the competition.

Nvidia VP of enterprise platforms, Justin Boitano, has explained that open-weight models are foundational to American AI leadership and cybersecurity. 

“To maintain U.S. leadership in the AI industrial revolution, the infrastructure that runs our economy needs safe, secure access to both closed and open models,” Boitano said. “For cybersecurity, open models and open harnesses are essential because they broaden defensive capability, increase transparency for defenders, and complement frontier closed models with customizable, localized controls.”

As regulators grapple with AI safety, Boitano predicts it will be important to “recognize open models and open tooling as defensive assets” — thus enabling transparency, independent evaluation and shared remediation.

Nobody can subpoena a downloaded weights file

Mark Vigoroso, founder & CEO of technology consultancy firm The Enterprise Edge, tells The New Stack that AI regulators have traditionally “built their entire AI safety apparatus” around auditing a handful of closed labs. And now, that approach is out of date.

“Open weight models blew past that closed model approach months ago,” Vigoroso says. “This alliance is an admission that the actual safety work now has to happen in the infrastructure layer: patch cycles, provenance, identity around who’s deploying what, because nobody can subpoena a downloaded weights file.”

“This alliance is an admission that the actual safety work now has to happen in the infrastructure layer: patch cycles, provenance, identity around who’s deploying what, because nobody can subpoena a downloaded weights file.”

Vigoroso argues that the AI safety debate is “stuck on model-level controls”, while the real regulatory gap is provenance of infrastructure and identity, i.e., knowing where a model came from, who deployed a model and what it touched, not just whether the model itself is safe.

“Groups like the EU AI Office, NIST’s Center for AI Standards and Innovation (CAISI), and the UK’s AI Security Institute (AISI) focus almost entirely on frontier closed models. Open weight models (Mistral, DeepSeek, and others) fall into a regulatory blind spot: Once weights are released, there’s no way to enforce downstream safety obligations.

Current regulatory frameworks for AI models assume a single accountable deployer; open source has none. That’s the real story: Regulators are writing rules for a centralized world while the ecosystem is decentralizing, Vigoroso says.

While detailed operational information explaining the intended actions of this alliance is currently scant, Nvidia has highlighted that it is contributing solid research to the Open Secure AI Alliance to speed the development of new cybersecurity tools and techniques. 

Harnesses integrate with models, making agents easier to test

The open source Nvidia Labs Object-Oriented Agent (NOOA) project is now available on GitHub to make advanced AI safety capabilities more accessible for agent harnesses. This research framework enables harnesses to integrate with models to make agent behavior easier to test, trace, audit, and govern.

Aparna Rayasam, CEO of verified identity and end-to-end encryption company Atsign, tells The New Stack that the “AI blitzkrieg conversation has reached a critical inflection point.” This moment is one where we cannot build the next era of open cognitive innovation on top of what Rayasam calls “legacy, Swiss-cheese infrastructure.”

…the “AI blitzkrieg conversation has reached a critical inflection point.”

“The formation of the Open Secure AI Alliance proves that AI safety isn’t just an algorithmic math problem — it is a foundational networking problem,” Rayasam says. “The massive, distributed pipelines required to train and run modern AI demand an entirely new paradigm of trust. True safety means ensuring that the data pipelines feeding these models are inherently invisible, un-attackable, and completely stripped of open network perimeters.”

The key notion here is that we are moving from a world of protecting data at rest to a world where the connective tissue of AI must be secure by design. 

One AI vendor to secure them all? No thanks.

Founder and CPO of agentic identity and permissions security company Reco, Gal Nakash, tells The New Stack that the launch of the Open Secure AI Alliance is an “important signal” which underlines why AI security can’t be solved by one vendor or one closed framework. 

“Jensen Huang’s point that every SaaS company will become a GaaS company captures why this matters now: Software is shifting from passive tools people log into, to AI agents that access data, take actions, and execute workflows,” says Nakash. “Open source tools and shared standards can help the industry move faster, but they need to be grounded in real enterprise context across identity, permissions, data access and behavior.”

Chris Boehm, Field CTO at automated, identity-driven microsegmentation company Zero Networks, tells The New Stack that news of the Open Secure AI Alliance makes him feel like he’s seen this before somewhere.

“This looks like the Trusted Platform Module (TPM) at Microsoft story all over again,” Boehm says. “It’s a case of an industry group defining what trusted hardware means, the platform vendors adopt it, and within a few years it’s a procurement requirement rather than a suggestion.”

He explains that “Windows 11 did exactly that with TPM 2.0 and Secure Boot”, and both Linux and Apple adapted. “I’d expect the same for AI infrastructure, where attested silicon becomes the floor for regulated workloads, and the vendor list narrows to whoever can meet it,” predicts Boehm.

“This looks like a case of an industry group defining what trusted hardware means, the platform vendors adopt it, and within a few years it’s a procurement requirement rather than a suggestion.”

A more global and geographically-inclusive approach is needed 

Amanda Brock, CEO of open technology body OpenUK, tells The New Stack that the Open AI Alliance is undoubtedly a great starting point, particularly with OpenAI’s security woes it divulged last week. 

“But, like the open letter on US Leadership in open weights, this is a US response to a US challenge,” Brock says. “Rumors of a forthcoming Presidential Executive Order to close down open models have been circulating for weeks — and worsened by the administration being thrown into turmoil over China’s Kimi K3.” 

For this alliance to succeed, Brock insists that it will need to take a more “global and geographically-inclusive approach”, beyond the US-centric founding members.

“It must also engage the open source ecosystem of individuals and innovators who are building the infrastructure, agentic harness functions and developer tools for AI. It’s important to realize that open AI infrastructure development shifts the innovation into the hands of the many, in direct opposition to the small number of corporate creators of frontier models,” Brock adds.

Nvidia’s Boitano echoes Brock’s view. In a blog post reviewed in draft by The New Stack, he writes that “open models turn more AI users into AI builders,” expanding opportunity, accelerating innovation, and keeping progress from being concentrated in only a few organizations or regions.

Boitano concludes by saying that open models also enable independent scientific research into how AI systems behave, allowing researchers to understand, evaluate, and improve them. It’s all about what he has called making broad, continuous defense possible.

Looking ahead, it feels like the next wave of AI safety bodies, movements, or alliances won’t just be model auditors—they’ll be trust-infrastructure standards bodies (encompassing identity verification, content provenance, credence, etc.) using borrowed approaches to governance and compliance. Ultimately, this may be the only enforcement layer that survives open weight proliferation.

The post Nvidia, Palantir, Hugging Face join 34 others in race to defend open-weight AI from cyber threats appeared first on The New Stack.

Over two days this month, two of the most credible people in the industry published framework manifestos on X.

Microsoft CEO Satya Nadella posted “The Reverse Information Paradox” on July 12.

And then Google DeepMind CEO Demis Hassabis posted “A Framework for Frontier AI and the Dawning of a New Age” on July 14.

When read side by side, each proposal reinforces the layer where its author’s company is already strongest.

Two manifestos, two boundary lines, one week

Nadella’s argument is about value capture. Enterprises pay for AI twice, he writes in the Reverse Information Paradox, once in tokens and again in the proprietary know-how they leak back into the model through prompts, corrections, and evals. His fix is to own the learning loop — meaning the data, the traces, the evals, the adapted weights, and the memory — then put a model-agnostic orchestration layer on top so any model stays cheap and swappable. Make the model a commodity and the value flows to the layers around it.

See also: Microsoft CEO Satya Nadella says you’re paying for AI twice — the second price is worse

Hassabis is drawing a very different line. His concern is not who captures the value but who governs the risk. In his article, he calls for a standards body modeled on FINRA, industry-funded and subject to government oversight, that tests frontier models for cyber, bio, and deception before they ship. Labs would submit models up to 30 days before release, voluntarily at first, then as a hard gate for deploying in the US market.

Both frameworks route through the author’s own strength

Neither framework offers a neutral read of the field. Nadella’s advice to own your data and keep models swappable is correct, and it also routes enterprises straight to Azure and Foundry, where the orchestration, billing, deployment, and governance stay with Microsoft no matter which model wins. The overlap between that advice and the product stack Microsoft already sells is difficult to ignore.

Hassabis’s gate subtly emphasizes the importance of scale. Large incumbents can absorb testing costs, run sophisticated safety teams, and shape standards more easily than smaller challengers. Google DeepMind already operates its own internal Frontier Safety Framework, so an incumbent with an established safety apparatus would likely begin with a compliance advantage, especially if major labs influence the eventual protocol. That does not prove Hassabis planned it this way, but the incentive still leans toward the same outcome the proposal describes.

The value moves toward the layer each one governs

The pattern in both posts is that they have stopped competing solely on benchmark leadership and now compete over the systems that govern how models are used and released. Nadella wants the decisive layer to be the enterprise boundary, so the model provider cannot harvest the customer’s knowledge.

Hassabis wants it to be the frontier gate, so nobody deploys a top-tier model without an industry-run body signing off on it. Different layers, but the same instinct: put durable value where your company is already positioned.

The counter is that Hassabis proposes a wider net than self-interest alone would draw. His regime would cover every frontier model, regardless of national origin or openness. It would get independent experts and open-source representatives on the board and answer to the US government.

Those provisions complicate any claim that the body would simply serve Google. Read most fairly, both frameworks may solve real problems while also favoring the firms best equipped to supply the solution.

The model is no longer the only scarce asset. Value is moving toward whoever controls the data boundary, the deployment layer, and the rules of admission, and this month, two of the most influential in the AI ecosystem staked a public claim to control them.

The post Microsoft and Google DeepMind agree on AI control — but not on who holds it appeared first on The New Stack.

Nvidia CEO Jensen Huang used his first post on X to share a public letter backing frontier open-weight models. Signed by Microsoft, Meta, Hugging Face, and 22 other organizations, the letter argues that open models improve security, encourage faster innovation, and give countries and enterprises more control over their AI infrastructure. The timing of the Friday post is notable, with more organizations choosing to run open-weight models inside their own environments rather than relying entirely on hosted services.

“Open weights let every organization match the right model to the right job at the right cost, reserving frontier-scale capability for genuine frontier problems and running efficient, specialized models everywhere else.”

A case for hybrid stacks

The letter draws a comparison between open-weight AI and open-source software, arguing that broad access has driven decades of software innovation. It says the same principle applies to AI. Instead of sending every request to a hosted model, organizations can download an open-weight model, run it on their own infrastructure, customize it for a specific workload, and keep sensitive data behind their own firewall.

Nvidia has already been moving in this direction — its leaders have described a future where local and frontier models split the work by cost, speed, and control. As the coalition argues, “Open weights let every organization match the right model to the right job at the right cost, reserving frontier-scale capability for genuine frontier problems and running efficient, specialized models everywhere else.”

One isn’t necessarily replacing the other, and many enterprise teams already use both, choosing between them based on cost, performance, compliance requirements, and where the workload needs to run.

The letter underscores this reality for infrastructure teams, noting, “As organizations invest in AI, they want to know that they will not become locked into a single provider or lose the knowledge and capabilities they build over time.” That vendor lock-in concern is already reshaping deals — Anaconda’s recent acquisition of Kilo was framed explicitly around enterprises’ reluctance to depend on a single AI provider.

Washington weighs new restrictions

The timing is notable. Washington is weighing new restrictions on some Chinese AI models, including Moonshot AI’s Kimi K3, even though the Trump administration’s AI Action Plan described open models as a strategic advantage for the United States.

That split extends to the AI industry itself, though critics differ in their specific concerns. OpenAI’s Dean Ball has publicly warned about the broad economic and regulatory risks associated with Chinese open-weight models. Anthropic’s Sarah Heck, meanwhile, has specifically backed the White House’s allegations regarding intellectual property theft through distillation.

“Relying solely on closed models is not inherently safe: they can be breached, misused, or fail in ways that outsiders cannot detect. And concentrating advanced AI capabilities behind a small number of closed models compounds that risk.”

The companies that signed the letter argue that continuing to develop frontier open models is part of maintaining U.S. leadership in AI. They argue that security through obscurity is fundamentally flawed, stating: “Relying solely on closed models is not inherently safe: they can be breached, misused, or fail in ways that outsiders cannot detect. And concentrating advanced AI capabilities behind a small number of closed models compounds that risk.”

Palantir and Nvidia have already operationalized this argument, building an engine that runs open Nemotron models inside air-gapped government networks.

Distillation meets trade policy

Distillation, one of AI’s most contentious topics, was also addressed. Widely used by researchers and model builders to create systems that are cheaper to run and easier to deploy, the letter states that it should be treated as a legitimate research technique rather than intellectual property theft.

The issue has taken on new significance after the White House accused China’s Moonshot AI of specifically distilling Anthropic’s Fable model to build Kimi K3 — an allegation the company has denied.

Distillation is one way teams build smaller models that can run on private infrastructure, edge devices, or lower-cost GPU clusters. Kimi K3’s launch showed just how much GPU capacity these deployments demand — Moonshot had to freeze new subscriptions within 48 hours after demand overwhelmed its available inference capacity. Any new restrictions on that process could affect how enterprise AI systems are built and deployed.

“As organizations invest in AI, they want to know that they will not become locked into a single provider or lose the knowledge and capabilities they build over time.”

Nvidia’s infrastructure incentive

Every enterprise that chooses to self-host, fine-tune, or customize an open-weight model needs infrastructure to run it. Nvidia also has clear business reasons for supporting a future in which both proprietary and open-weight models thrive.

The company supplies the hardware powering both hosted frontier models and enterprise-owned deployments. Organizations that use a mix of commercial APIs and self-hosted open models ultimately expand demand for AI infrastructure across the board — a dynamic that is already redirecting enterprise IT budgets toward AI hardware at the expense of traditional software spending.

The post Jensen Huang made his first X post. He used it to lobby Washington about open-weight AI. appeared first on The New Stack.

According to OpenAI, the Hugging Face security breach was an “unprecedented cyber incident, involving state-of-the-art cyber capabilities.” Critics may disagree.

Back in 2018, for example, academics predicted that new attacks might “arise that would be impractical for humans alone to develop or which exploit the vulnerabilities of AI systems themselves.”  Well, here we are.

What escaped the sandbox

So, what really happened? OpenAI reports an autonomous security-evaluation run of GPT‑5.6 Sol and a pre-release model broke out of a sandbox, reached the internet, and then targeted Hugging Face to try to solve the ExploitGym benchmark.

By OpenAI’s account, “an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths … [with] maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity.” 

OpenAI continues, “Our benchmarks run in a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries.”

“All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.”

The AI models, however, cracked their sandbox. Once out, they “identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database. All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.”

AI expert Ken Huang describes the process in a blog post: “The model spent enormous inference budget probing the sandbox, then found a zero-day in the third-party software OpenAI used as a package-registry proxy and cache.

“It used that bug to gain unrestricted internet access. It then reasoned that Hugging Face probably hosted the ExploitGym datasets, chained stolen credentials with several more vulnerabilities into a remote code execution path, and pulled the benchmark answers straight out of Hugging Face’s production database.”

Why intent no longer matters

This was not, as John W. Larson, president and chief AI officer of the AI security company Babel Street, tells The New Stack, a “Terminator” moment. OpenAI models and agents “[were not] acting out of malice or trying to attack Hugging Face. It encountered obstacles, developed an unexpected strategy, bypassed safeguards, and pursued its assigned goal in a way its creators never anticipated. The incident demonstrates that harmful cyber incidents no longer require malicious intent: Only highly capable autonomous AI optimizing for an objective.

Harmful cyber incidents no longer require malicious intent: Only highly capable autonomous AI optimizing for an objective.

That’s the first issue we must be aware of. Software attacks no longer require a hacker. All it needs is an LLM freed of guardrails with a mission to accomplish, and it will take whatever steps are needed to reach its goal. 

In addition, Adam Boynton, senior enterprise security manager of EMEIA at the Apple Identity and Access Management company Jamf, tells The New Stack, “This happened inside a frontier lab deliberately stress-testing its most capable models, in conditions no enterprise would ever run.”

Boynton also notes that this episode confirmed that “AI tools are no longer just applications people open. They act on the user’s behalf, with the user’s access. So every organization needs a clear view of which AI tools its people are running, and how those tools are configured and used.”

An agent given too much power is far more capable than a user armed with a superuser password of causing havoc both to your systems and to ones far outside your control. 

Where container isolation failed

As for the attack itself, Edera CMO Kaylin Trychon writes in a blog post, “This incident deserves more attention than ‘an AI did the hacking now.’ The interesting part isn’t who was driving, or even that it was a model instead of a person. It’s where the walls gave way, twice, in two different companies’ infrastructure — and that part is entirely familiar to anyone who’s spent time securing container infrastructure.” 

Exactly so. Trychon continues, “Strip away the framing and this is a story about two escalation chains, back to back.” The first consisted of “The model found and exploited a zero-day in that proxy, then used privilege escalation and lateral movement inside OpenAI’s own ‘highly isolated’ testing environment until it reached a node with open internet access.”

Next, OpenAI’s models inferred, correctly,  that “Hugging Face likely hosted the datasets, models, and reference solutions for the benchmark it was trying to solve. It chained stolen credentials with a remote-code-execution path into Hugging Face’s servers, harvested more credentials, and moved laterally into several internal clusters over a weekend, all in pursuit of one narrow goal: Finding the answer key.”

“The failure that actually matters is what a single code-execution event is allowed to become — and notably, that failure happened the same way twice, at two different companies, in the same incident.”

The serious issue, from Trychon’s perspective, is that “both companies’ blast radius was a cluster instead of a container.”

That’s because “the failure that actually matters is what a single code-execution event is allowed to become – and notably, that failure happened the same way twice, at two different companies, in the same incident. In a typical container fleet, workloads share a kernel.

“A process that gets popped, or a model that gets a shell, is one syscall boundary away from the node it’s running on, and the node is one set of mounted credentials away from the rest of the cluster. Namespaces and cgroups are process-isolation primitives, not security boundaries — they were never designed to hold against an attacker (human or model) that already has code execution and time.”

People might not have been able to do this easily. But as Bloomberg succinctly put it, “OpenAI Models Spent Hours on Hack That Usually Takes Weeks.” Exactly so. We can expect to see more such successful attacks. 

Beyond the sandbox paradigm

Edera’s CTO and co-founder Alex Zenla tells The New Stack, “The sandbox escape here isn’t really the surprising part of this attack. An entire market of AI sandboxing tools has exploded over the past couple of months, and this is the reality they’re all built on top of: containers, VMs via namespaces — whatever the branding — most of them still share a kernel with the host or with each other.

“That’s a boundary enforced in software, and software boundaries are exactly the kind of thing an agent that can try ten thousand escape paths over a weekend is going to find a way through. We shouldn’t be shocked that this happened; we should be shocked at how many teams are still betting their infrastructure on technology that was never designed to withstand such a persistent adversary. The fix isn’t a better sandbox; it’s getting out of the sandbox paradigm entirely.”

Zenla continues, “Teams should adopt secure execution environments that are hardware-enforced and eliminate the shared kernel flaw. This technology exists today and would have made this specific escalation chain structurally impossible, not just harder. Teams running agents with real permissions and real access need to stop treating this as a someday problem, because the next version of this week is already being tested somewhere right now.”

That underlines the most important issue of all. Thanks to AI, security attackers are coming harder and faster than ever. Security can no longer be an afterthought.

As Jim Zemlin, the Linux Foundation CEO, said at the 2026 Open Source Summit North America, “the exploit time for a zero-day vulnerability being exploited has shrunk from 63 days to -7 days.”

You literally no longer have time to wait for security fixes. You must bake in as much security as you can as fast as you can, or your systems will be broken into. It’s as simple as that. 

The post What really happened in the Hugging Face breach appeared first on The New Stack.

The AI industry has largely focused on language-based approaches, using transformers trained on massive datasets to predict words or fill in missing information.

As AI expands into more structured fields, however, the limitations of text-generation models are becoming clearer. Nvidia is pursuing a different approach.

The company just dropped JEPA-DNA on Hugging Face. It’s a genomic foundation model that adds a latent-space prediction objective alongside MLM. 

This release is a win for hybrid architectures that go beyond purely generative training. It’s the kind of shift Executive Chairman of AMI Labs, formerly Meta’s Chief AI Scientist, Yann LeCun, has been pushing for years. He’s championed predictive architectures as a general alternative to next-token prediction, which researchers are now applying to biology. 

Latent space over literal tokens

Conventional genomic base models have historically mirrored NLP models, relying purely on MLM, masking parts of a DNA sequence and forcing the model to guess the missing literal tokens. This approach favors local token reconstruction and teaches the model the basic “syntax” of the sequence, but it frequently struggles to grasp the wider functional “meaning.”

JEPA-DNA changes the paradigm. The newly released checkpoint, JEPA-DNA-DNABERT2, serves as a model-agnostic continual pre-training framework. It couples standard token-level DNA language modeling with JEPA, adding a second learning objective.

Instead of forcing the architecture to reconstruct missing tokens, JEPA-DNA supervises the model’s global sequence embedding in a latent space.

Instead of forcing the architecture to reconstruct missing tokens, JEPA-DNA supervises the model’s global sequence embedding in a latent space. It predicts the functional representation of masked genomic segments rather than their literal, character-by-character makeup.

Token prediction is still part of the training process, but it is no longer the model’s only learning objective. The model also learns the basic structure of the data, proving that predictive architectures can produce representations that are easier to work with for biological tasks without sacrificing generative capabilities.

Token prediction is still part of the training process, but it is no longer the model’s only learning objective.

How DNABERT-2 gets upgraded

The model builds on DNABERT-2, a 117 million-parameter model developed by Zhihan Zhou and collaborators. Nvidia layers its continual pre-training approach on top of that architecture, letting the model learn from both token-level predictions and latent-space representations.

Nvidia has released the model globally for non-commercial research. According to the company, the model is meant to support research workflows, including feature extraction, linear probing, continual pre-training experiments, and zero-shot scoring of DNA sequence changes. It isn’t a diagnostic tool or a clinically validated medical product.

Beyond the generative hammer

DNA contains patterns and relationships that aren’t captured by sequence prediction. JEPA-DNA supplements masked-token prediction with a latent-space objective intended to capture broader sequence-level information.

Ultimately, models like JEPA point to a new path for AI that combines next-token prediction with other ways of learning and helping models build a deeper understanding of complex systems.

The post Nvidia’s new DNA model learns what token prediction misses appeared first on The New Stack.

The models small enough to run on the box on your desk are getting good enough that the interesting question is no longer whether you can run them, but what you can do with them, and how organizations can get the most out of them.

Joey Conway, Nvidia’s senior director of generative AI software, spoke to The New Stack about how local and open models are increasingly working alongside frontier models, often with a router in between deciding which one to use, and how organizations can adapt these open models for their own needs.

“We love the world where we can use both frontier and open models together.” — Joey Conway, Nvidia

A system of models

Tasks vary in complexity, so the models handling them should vary too, Conway tells The New Stack. He points to the early open reasoning models, which would reason their way through trivial problems, mulling number lines and memory to work out what two plus two is. “I just say four,” he says.

“Being able to route those easy things to local models that are quick, and route the hard things to more sophisticated models,” Conway says, lets you “get a better outcome at a lower cost and lower time to completion.”

It’s a different picture from the one large model most people imagine doing everything. In his version, you build a bench of specialists. “You’ll have specialized agents that are really good at focused tasks because that’s what they do every day,” he says, “and they just get better and better at that task.”

To the user, none of that shows. “It’ll feel like one interface,” Conway says, “but behind that interface, there’ll be a variety of models handling a variety of tasks.”

Getting there is largely a routing problem, and one Conway says is still in its early days. Nvidia’s own contribution, for now, sits lower in the stack, in inference-serving software like its open-source Dynamo, which steers each query to the GPU that handled it most recently. Which model is best for which job, Nvidia leaves to a wider field of routers, some of them models in their own right that weigh budget, latency, and modality. But Conway also leaves the door open for Nvidia to build more of that routing itself before long.

Nvidia points to its collaboration with LangChain, whose Deep Agents harness ran on Nemotron 3 Ultra, Nvidia’s 550-billion-parameter open model, and matched top closed models on business tasks at up to a 10x lower cost, as Conway notes. It required no retraining; the gains came entirely from tuning the harness around it: its prompts, tool descriptions, and middleware.

You’re not going to run a 550-billion-parameter model on your desktop anytime soon, but running relatively large models locally is now a real possibility, as long as you have some beefy hardware at your disposal. For enterprises, setting up a fleet of accelerators in a data center isn’t exactly cheap either, but it does mean full control and no surprise token bills.

Bringing AI to where the data lives

Running models yourself can save money, but Conway thinks control matters more. Enterprises already decide where their data lives and what they hand to outside vendors, and open models give them even more control. “Move AI to where your data lives,” he says, “or move AI to where your employees are.”

Companies want to keep their data — and especially their intellectual property — in-house, and Conway argues a fine-tuned open model is the place to put it. “It’s like an employee,” he says. “You hire them, and they’re part of your company.”

The local half runs on hardware like Nvidia’s DGX Spark, a $4,699 Grace Blackwell machine with 128GB of unified memory that handles models up to roughly 200 billion parameters without anything leaving your desk. (There is also the DGX Station, its bigger, pricier sibling with 748 GB of RAM for running even larger models.)

“It’s like a system sitting right there next to you,” Conway says, one where “you don’t think about network latencies.” To run those agents securely, Nvidia offers NemoClaw, a reference stack that wraps an open agent harness like OpenClaw in a sandbox called OpenShell, with policy controls and local Nemotron inference.

When you need more power for a broader problem, you reach for a frontier model in the cloud. For Nvidia, that’s all good news: a system of models runs on its silicon one way or another, on your desk or in the cloud.

The post “We love the world where we can use both”: How Nvidia thinks about local and frontier models appeared first on The New Stack.

Cursor, the AI coding tool recently acquired by Elon Musk’s SpaceX in a $60 billion all-stock deal, has launched a model router designed to direct every coding request to whichever model handles it best, bypassing the need to pay frontier prices for work that doesn’t need it.

Under the hood, the new Cursor Router uses a triage system à la a hospital emergency room: It looks at what a request actually needs — how hard it is, what it’s for, the surrounding code — and picks a model that is the best fit. A quick fix goes somewhere cheap, while a genuinely hard problem gets escalated to something closer to frontier-grade.

Notably, developers and admins also have access to three distinct modes that nudge that balance in either direction, favoring speed and cost over raw power, or vice versa if needed.

The broader rationale for Cursor Router, according to the company’s own field CTO David Pan in a social media post on Wednesday, is that developers shouldn’t have to become experts in model performance just to write code.

“We briefly went insane and decided every software engineer should also become an expert in model benchmarks, thinking levels, and cache hit rate.”

“We briefly went insane and decided every software engineer should also become an expert in model benchmarks, thinking levels, and cache hit rates,” Pan writes.

Early community feedback has largely echoed Pan’s sentiment: Fatih Arslan, a software engineer at PlanetScale, notes on X that engineers already juggle the choice between cost and capability by hand — defaulting to a cheap, fast model for routine work and saving the slow, expensive one for “serious tasks.”

“We already spend quite a bit [of] time on [choosing models],” Arslan writes. “Why not automate that part? Cursor Router does the automation.”

“We already spend quite a bit time on it [choosing models]. Why not automate that part?”

In a separate blog post published on Wednesday, Cursor claims that early access customers saved 30-50% compared to routing everything through Opus 4.8, with no drop in output quality.

Working model: Taking control of the stack

The launch follows a run of moves by Cursor to control more of its own AI stack. In May, the company released Composer 2.5, an update to its in-house coding model built for long tasks at a lower cost than frontier options from Anthropic and OpenAI. Composer 2.5, like its predecessor, is built on Moonshot AI’s Kimi K2.5, an open-weight model out of China.

Now, with the weight of one of the world’s most valuable companies behind it (SpaceX has attained a market cap of $1.5 trillion since its June IPO), Cursor is pushing a powerful frontier model of its own.

On July 8, Cursor and SpaceXAI jointly released Grok 4.5, a mixture-of-experts model built on a new foundation dubbed V9, which Musk had previously noted was roughly 1.5 trillion parameters. The model’s trained on trillions of tokens of real Cursor usage data and available across all Cursor plans at $2 per million input tokens and $6 per million output tokens.

With Composer handling cheap, fast work, and now the Grok-branded frontier line for more serious horsepower, Cursor has its own models in the mix alongside the usual list of outside providers. And this gets to the heart of why Cursor built Router: Most developers pick one model and stick with it regardless of the task, billing simple work at frontier prices it doesn’t need.

Sending every request to its own models would be the easy way to keep that money in-house, but it would also mean shipping inferior output on some tasks — so Router instead sends each request to whichever model actually suits it, Cursor’s own or not.

The lay of the land

Model routing itself isn’t exactly new. OpenRouter has offered a version since 2023: a single API sitting in front of more than 400 models from over 60 providers, including OpenAI, Anthropic and Google. Its own auto-router feature does roughly what Cursor Router does — classify a request, then send it to whichever model fits the task and the person’s stated preference between cost and quality.

More recently, OpenRouter launched Fusion, which takes a slightly different approach: instead of picking one model, it sends a prompt to several models at once and uses a judge model to synthesize the strongest answer out of all of them.

This past month ushered another entrant to the mix: Japan’s Sakana AI released Fugu in June, which instead breaks a single task into subtasks and routes each piece to a different model, pitched by Sakana as a hedge against relying on any one AI provider.

“[Cursor Router is] a great example of how a technological innovation immediately translates into a product improvement.”

Not everyone rates some of these other attempts, though. On Wednesday, Kirill Balakhonov, head of AI products at Nethermind, argues on LinkedIn that Cursor’s version succeeds precisely because it’s focused on coding specifically, rather than trying to be a general-purpose router for any task.

“A great example of how a technological innovation immediately translates into a product improvement… rather than an abstract idea like Sakana Fugu or OpenRouter Fusion,” Balakhonov writes, predicting both of those broader routing efforts would fade from use.

What’s new, perhaps, is some of the names emerging behind an array of model diversity efforts. In early July, Microsoft launched a $2.5 billion services unit dubbed Microsoft Frontier Company, embedding thousands of engineers at customer sites to help them build with a mix of AI models.

Judson Althoff, CEO of Microsoft Commercial Business, told Reuters at the time that the push came partly from watching rivals like DeepSeek and Google’s Gemini close the gap on OpenAI. Referring to the original Copilot, he admitted, “we made a mistake by binding it to OpenAI models only.”

If the company with the deepest single-model relationship in the industry is walking it back, the idea for model flexibility has clearly gone mainstream — certainly if this week is anything to go by.

Ramp and Meta get in on the act

On Tuesday, Ramp, the $44 billion spend-management behemoth, opened up Ramp Router, an early-access public version of the model router it built to manage its own AI bills internally, which it says cut its LLM costs by roughly 30%. It’s free to start, requires no Ramp account, and routes across OpenAI, Gemini, and select open-source models including Kimi through an OpenAI-compatible endpoint.

The very same day, Jyoti Mann from The Information reported that Meta is also working on a model router. According to internal documents cited in the report, an incubator inside Meta called AAI Labs is developing a new product named Switchboard that would score each request for difficulty and send simpler ones to smaller, cheaper models — aimed initially at cutting Meta’s own AI agent costs, though it reportedly could end up as a public release.

Meta has particular reason to want this. Data from Runpod’s State of AI report, published in March, suggests Meta’s open Llama models are now a marginal presence in production: Llama 4 has near-zero real-world deployment, with Alibaba’s Qwen having overtaken it as the most-deployed self-hosted LLM.

Meta has also been building proprietary models in response. Muse Spark, its first model out of the newly formed Meta Superintelligence Labs, shipped in April. That was followed in July by Muse Spark 1.1, Meta’s first model with a public, paid API, priced at roughly a quarter of what OpenAI and Anthropic charge for comparable models.

Meta is aggressively targeting the incumbents, and Switchboard fits the same pattern: a way to make it easier for users to cut costs, switch models freely, and, where it makes sense, land requests on Meta’s own models instead.

But amidst all this hullaballoo about model routing, there is perhaps a broader question of openness. Not of the models themselves, which is a fervent debate in itself, but of whether the routing decision itself — the logic that decides which model handles which request — should sit inside a vendor’s own closed product at all.

“Is anyone building this as open-source?”

Elvis Saravia, a former technical product marketing manager at Meta AI who co-founded DAIR.AI, took to X to argue that it shouldn’t, given how differently teams weigh cost against quality.

“Is anyone building this as open-source?” Saravia asks. “It feels like this is something you don’t want to offload to an API. We all work with different trade-offs, so we need the ability to achieve custom routing.”

As for Cursor’s own version, Router is currently available to Teams and Enterprise customers only, across desktop, web, iOS, CLI, and Cursor’s SDK. It’s not yet clear whether it will eventually land on individual plans.

The post Cursor, Ramp, and Meta are all building model routers — but two have major model ambitions themselves appeared first on The New Stack.

Every product team is chasing the same moment: The user opens a page and thinks, this understands me.

A shopper who loves floral prints should see more floral prints. A user who follows local politics should open their app to see news about local politics. A job candidate who keeps clicking remote roles should not keep getting shown in-office jobs.

That is not a niche feature anymore. It is the baseline expectation. Users decide quickly whether a product system understands them, and they rarely care whether the failure came from search, recommendations, merchandising rules, or stale data.

Here is the uncomfortable truth: Most teams do not have a personalization quality problem. They have a personalization architecture problem.

Personalization is not a widget bolted onto search. It is a ranking decision. The system has to decide, for this user and this request, what deserves the next slot. That means weighing the user, the item, the context, and the business goal at the same time. In many stacks, the ranking layer is the one place that cannot see all of those signals together.

The hard part is not collecting signals. The hard part is combining them while the user is still there.

Why personalization is hard in the first place

To put the right item in the right slot, a system has to understand several things at once:

• Intent: What is the user asking for right now?
• Item quality: What does each candidate actually contain or represent?
• User history: What has this person clicked, bought, read, watched, or ignored?
• Availability: Is the item in stock, fresh, nearby, legal to show, or ready to ship?
• Business priority: What should the business promote, protect, or de-emphasize?
  

Those signals often disagree. The most relevant item may not be the most profitable. The most profitable item may be out of stock. The user may say “running shoes,” but their behavior says “trail running, wide fit, under $120.”

They also move on different clocks. Product attributes change slowly. Inventory and price can move throughout the day. Preferences shift with every click. External context — weather, breaking news, a championship game, a cultural moment — can matter without warning.

Personalization means folding all of that into one ordered list, on every request, in milliseconds. The signals themselves are not the bottleneck. Query-time ranking is.

The usual stack makes the problem harder

Most personalization systems are assembled from tools that were each designed for one slice of relevance.

Keyword search engines are excellent at lexical matching. They are good when the query language and catalog language line up. But shoppers, readers, and job seekers rarely speak in neat index terms. You indexed “athletic performance running footwear”; they typed “running shoes.” Synonym rules can help, but they do not scale gracefully across long-tail language, changing catalogs, and new user behavior.

Vector databases start from the opposite side. They are good at semantic similarity: “Find me things like this.” That is powerful, but nearest-neighbor search is not the same thing as personalization. Real ranking has to blend semantic similarity with live behavior, stock, price, margin, freshness, eligibility, and business rules.

Re-rankers, recommendation services, feature stores, and rule engines are usually added to glue everything together. That is where fragmentation creeps in.

When retrieval and ranking live in separate systems, the ranker often works from a partial, stale, or precomputed view of the world. Click history, session context, and the user’s live preference vector arrive too late. Business rules become filters or overrides instead of ranking signals. Fresh inventory or price changes require coordination across multiple systems.

Every hand-off adds latency. Every boundary creates another place for signals to drift. Every “quick rule” becomes another hard constraint that can accidentally turn “show the closest match” into “show nothing.”

“Every hand-off adds latency. Every boundary creates another place for signals to drift.”

The deeper issue is a timing assumption. Many architectures were built around offline ranking: process the catalog, compute scores in a batch job, and serve those scores until the next rebuild. That works when preferences are stable. It breaks when the most valuable signal is the click that happened two seconds ago.

What changes when ranking happens in one real-time pipeline

A real-time personalization architecture treats retrieval, ranking, and inference as one serving problem.

That is the core idea behind Vespa’s approach: Text search, vector similarity, structured filtering, ranking expressions, tensor computation, and model inference can live inside one query pipeline. Instead of retrieving somewhere, enriching somewhere else, and ranking at the end, the system can rank with the relevant signals while it is still deciding what to return.

That architectural choice changes the shape of the problem.

1. Retrieval is hybrid from the start

Lexical search, semantic search, and structured filtering can run together instead of being reconciled after the fact. A product query can combine text, embeddings, filters, session behavior, and item attributes in one request.

That matters because personalization is rarely one signal. The user’s query still matters. So does semantic similarity. So do category, availability, price, and business constraints. Hybrid retrieval keeps those signals in play before ranking starts.

2. Ranking can express the actual objective

A personalization score should not be trapped inside one similarity function. It should be a formula that reflects the product’s goals.

That formula might combine BM25, vector similarity, user affinity, stock level, margin, popularity, discount depth, freshness, rating, distance, or a weather term. Some of those signals need normalization first. Some should matter only for certain categories or users. Some should be tested as weights.

The important part is that they are all terms in the same ranking expression, not scattered across services.

A simplified version might look like this:

final_score =
    0.30 * lexical_relevance +
    0.25 * semantic_similarity +
    0.25 * user_affinity +
    0.10 * availability +
    0.10 * business_priority

In production, the formula can be more nuanced. But the principle is simple: personalization, relevance, and business logic belong in the same scoring decision.

3. Model inference can run where the data lives

Some signals should come from learned models rather than hand-tuned rules: propensity to buy, churn risk, quality prediction, fraud risk, query classification, or a learned-to-rank model.

When inference runs in the serving path, those model outputs can become ranking features instead of delayed batch scores. That reduces the need to ship data to a separate inference service, wait for a response, and stitch the score back into ranking.

4. Updates become immediately useful

“Real time” should not mean “after the next index rebuild.” If inventory changes, stock should be rankable immediately. If a user clicks two yellow dresses, “yellow” should matter on the next request. If a merchandising team adjusts a ranking weight if the weight is exposed as a query-time input, the experiment should start producing useful feedback right away.

That is the difference between personalization as a nightly job and personalization as a live ranking decision.

Tensors make the personalization concrete

The most useful mental model is simple: represent the user and the item in the same feature space, then rank by how well they match.

In Vespa, tensors make that practical. A tensor can be a scalar, a dense vector, a sparse map of feature-weight pairs, a matrix, or a more complex structure. That means the same framework can represent semantic embeddings, product attributes, user preferences, business objectives, and model features.

For example, each item can carry a sparse feature tensor:

{
  "floral": 0.90,
  "yellow": 0.70,
  "short_sleeve": 0.80,
  "crew_neck": 0.65
}

Each user can carry a tensor with the same feature names:

{
  "floral": 1.00,
  "yellow": 0.37,
  "short_sleeve": 0.33,
  "crew_neck": 0.31
}

Because the two tensors share a shape, personalization becomes a dot product: multiply matching features, sum the result, and use that score inside ranking.

In a Vespa rank profile, the core expression is compact:

# schema: item attributes stored as a sparse tensor
field item_features type tensor<float>(feature{}) {
    indexing: attribute | summary
}
 
# rank profile: the user's live preferences arrive as a query tensor
rank-profile personalized {
    inputs {
        query(user_features) tensor<float>(feature{})
    }
    first-phase {
        expression: sum(query(user_features) * attribute(item_features))
    }
}

That one expression is not the whole ranking function. It is the personalization term. BM25, vector similarity, stock, margin, freshness, distance, or a model score can be added as other terms with their own weights.

The user tensor is where real-time behavior becomes powerful. Click a floral item, and the “floral” weight rises. Click two yellow items, and “yellow” rises; the application feeds click events into the user profile. The next query can use those updated preferences immediately, without waiting for a nightly profile build.

Business goals stop fighting personalization

In fragmented stacks, business rules often become blunt instruments: boost this category, hide that brand, force these items to the top, filter these out. That can satisfy a short-term merchandising goal while damaging relevance.

When business logic is part of the ranking expression, it can be more subtle. You can boost overstocked inventory without ignoring intent. Promote umbrellas when rain is forecast without turning every search into an umbrella search. Give new sellers a small exploration boost. Prioritize destocking before a new product line launches. Surface team merchandise during a championship run.

“When business logic is part of the ranking expression, the user still gets relevant results. The business still influences outcomes.”

The user still gets relevant results. The business still influences outcomes. The difference is that both are expressed as ranking signals instead of competing systems.

That also makes experimentation easier. A merchandising or growth team can test weights, traffic splits, and ranking profiles without asking engineering to rewrite the whole pipeline. Relevance becomes a controllable growth lever rather than a fragile side effect.

The same pattern applies beyond commerce

The examples above are easy to picture in apparel, but the architecture is not commerce-specific. Personalization is the same ranking problem in many products:

• Content feeds: Blend topic affinity, freshness, engagement, creator quality, and business rules.
• News: Rank by reading history, topic interest, locality, freshness, and source diversity.
• Jobs: Match candidate preferences such as remote work, seniority, compensation, location, and tech stack against role attributes.
• Geo search: Treat distance as one normalized ranking term alongside relevance, quality, and preference.
• Video and audio: Combine embeddings, viewing history, metadata, freshness, and learned ranking models.

Different domains need different features. The architecture pattern is the same: retrieve candidates, rank with the signals that matter, update those signals as behavior changes, and keep the decision close to the data.

Scale doesn’t have to be the trade-off

The natural concern is that a more expressive ranking system must be slower. In practice, that does not have to be true.

Vespa was built for large-scale serving from the beginning: billions of documents, high query volume, and low-latency ranking. The reason this works is multi-stage ranking. The system does not run the most expensive logic across every possible result. Instead, it uses a fast first phase to narrow the candidate set, then applies more precise ranking to the smaller group that remains.

For example, a cheap first phase narrows a huge candidate set. Then, once the candidate set is smaller, Vespa can apply full-precision scoring, richer tensor operations, business logic, and model inference where they matter most.

The result is a practical balance: speed across the full corpus, accuracy in the final ranking, and enough flexibility to personalize each query without turning the serving stack into a chain of fragile services.

What’s next

Personalization is not failing because teams lack data. Most teams already have plenty of signals: query intent, clicks, product attributes, inventory, margin, freshness, location, and business priorities. The harder problem is that those signals often live in different systems, move at different speeds, and arrive too late to influence the final ranking decision.

That is why personalization should be treated as a ranking problem. When retrieval, ranking, personalization, and business logic are split across separate systems, the ranker is forced to work with stale or incomplete context. The user moves faster than the architecture can respond. Every new signal becomes another integration project.

A unified real-time ranking pipeline changes that. User behavior, item attributes, semantic similarity, lexical relevance, inventory, and business goals can all become parts of the same scoring function. Tensors make those signals directly comparable and usable at query time. Instead of bolting personalization onto the end of the system, personalization becomes part of the decision the engine makes for every query.

The goal is simple: rank each result with the best context available, at the moment the user asks. That is when personalization stops feeling like a feature and starts feeling like relevance.

The post Personalization is a ranking problem — architecture makes it work appeared first on The New Stack.

The AI ecosystem is drowning in tutorials on how to build a retrieval-augmented generation (RAG) app in five minutes. The pitch is appealing but flawed: Chunk a document, run it through an embeddings API, load it into a vector database, and slap a UI on top. This setup works locally. It might even survive a beta test with friendly users. But launch it into a production-grade B2B SaaS environment, and the architecture collapses.

Enterprise applications don’t handle neat, static files. They ingest dynamic, unpredictable streams of live data bound by strict legal and compliance constraints. Treating vector search as a solved infrastructure problem at scale is a dangerous mistake.

“Treating vector search as a solved infrastructure problem at scale is a dangerous mistake.”

Here is exactly what breaks when a naive RAG setup hits production, and the architectural trade-offs needed to fix it.

Bottleneck 1: the synchronous ingestion trap

Synchronous data ingestion is the most prevalent architectural flaw in new AI products. A user uploads a 500-page compliance manual. The client makes a POST call to a web server, which parses the document, splits the text, iterates over a sequence of synchronous API calls to OpenAI or Cohere for vectorization, and writes those vectors to the database.

This approach introduces two critical failures:

• Timeouts: A 500-page document rarely finishes processing within standard HTTP timeouts (30 to 60 seconds) while waiting for the embedding API.
• Cascade failures: If the system hits rate limits or latency spikes, the entire ingestion operation fails, throwing a 500 error and losing the user’s document.

The fix: the batched fan-out pipeline

Production-grade AI pipelines require persistent events rather than simple HTTP calls. However, sending the whole 500-page document to be processed by one consumer from Kafka or RabbitMQ is a mistake. If a consumer spends 10 continuous minutes generating embeddings, it misses its broker heartbeat. Assuming the worker died, the broker kills the consumer and triggers a partition rebalance, creating an infinite loop of duplicated work and stalled processing.

“Synchronous data ingestion is the most prevalent architectural flaw in new AI products.”

Conversely, granular chunking, where every chunk becomes an individual Kafka message, launches a self-inflicted denial-of-service (DoS) attack on downstream services. A document with 1,500 chunks generates 1,500 individual messages. This instantly exceeds upstream requests per minute (RPM) limits and floods the pipeline with network overhead.

The engineering sweet spot is a batched fan-out approach:

• Asynchronous uploads: The web API stores the raw file in Amazon S3, triggers a document_uploaded event, and instantly returns a 202 Accepted status. This single, asynchronous path processes one-page invoices and 100-page SOC2 reports with equal reliability, eliminating the technical debt of maintaining separate “fast” and “slow” ingestion routes.
• Micro-batching: A lightweight “Spitter” consumer downloads the file, chunks it, and groups those chunks into optimized micro-batches (e.g., 64 chunks per batch).
• Controlled embedding: Embedding workers pull these batched events. To prevent concurrent workers from breaching upstream RPM limits, avoid fragile sleep() delays. Instead, enforce a token bucket rate limiter at the consumer level or strictly cap the number of active message broker partitions.

Python

# Conceptual snippet for architectural illustration
def handle_document_upload(event):
try:
raw_text = download_from_s3(event.file_uri)
chunks = semantic_chunking(raw_text)
except Exception as e:
# Log failure and raise so the message broker routes this to a Dead Letter Queue (DLQ)
print(f"Failed to process document {event.file_uri}: {e}")
raise

# Batch size heavily depends on the downstream embedding model's context limits
batch_size = int(os.environ.get("EMBEDDING_BATCH_SIZE", 64))

for i in range(0, len(chunks), batch_size):
chunk_batch = chunks[i:i + batch_size]
kafka.publish("embedding_tasks", {
"tenant_id": event.tenant_id,
"document_id": getattr(event, "document_id", event.file_uri),
"chunks": chunk_batch
})

This keeps individual consumer tasks short, respects upstream rate limits by maximizing payload density, and allows horizontal scaling of embedding workers during traffic spikes.

Bottleneck 2: the multi-tenant nightmare

Developers often treat multi-tenancy as an afterthought. The simplest way to handle multiple B2B tenants using a single RAG system is logical segregation, where all vectors reside in a large index, and each entry has a tenant_id associated with its metadata. Upon retrieval, the application filters results by adding a clause to the metadata payload.

Flaws of the approach:

1. Security vulnerabilities: Relying on application-level filtering creates an unacceptable risk. If an engineer omits or misconfigures a metadata filter, one client can access another’s confidential data. In highly regulated environments, this breaks compliance.
2. The noisy neighbor problem: If one customer uploads 10 million vectors to the shared index, memory usage skyrockets during vector searches. This degrades performance across the entire system, even for tenants with a handful of documents.

The fix: serverless compute-storage decoupling

Echo-chamber thinking assumes that the only solution is to provide each tenant with its own dedicated database cluster. This is prohibitively expensive and practically impossible to manage in a modern-day SaaS offering. 

The true gold standard here is using next-generation serverless vector databases like Pinecone Serverless or managed Qdrant implementations, which make a clear distinction between computing and storage.

Engineering takeaway: stop building complex multi-tenant routing logic in your application code. Push the isolation boundary down to the infrastructure layer using serverless namespaces.

Bottleneck 3: the semantic caching trap

Once ingestion is asynchronous and tenants are segregated, inference costs become the final bottleneck. Hitting an LLM API for every individual query is economically unsustainable.

The industry defaults to semantic caching: embed the user’s prompt, calculate its cosine similarity against previous prompts, and return a pre-calculated LLM response if the score exceeds a set threshold (e.g., 0.95).

Why semantic caching fails

Embeddings capture overall semantic meaning, but they miss specific contexts and entities. The prompts “What was the holiday policy in 2023?” and “What is the holiday policy for 2024?” share a near-perfect cosine similarity score. The core semantics match, but returning a cached answer feeds the user incorrect or contradictory information.

The fix: hybrid verification vs. native prompt caching

To scale without compromising accuracy, there are only two choices to consider: application-layer validation or infrastructure-layer optimization.

Strategy A: combined lexical filtering and intent routing

When using an application-layer caching system (for instance, Redis), you need to layer the semantics search on top of extremely light guardrails.

1. Exact-match filter: Apply a token-validation filter over vector similarity. In the case where the cached query is “2023”, and the current query is “2024,” throw out the cache hit right away.
2. Intent routing: Before serving a cached answer, use an inexpensive, fast model as an intent match router.

Python

Query A: {incoming_query}
Query B: {cached_query}

Do these queries have the exact same intent and require the exact same factual answer? 
Respond only with YES or NO.

Strategy B: infrastructure-level prompt caching

If the system cannot tolerate the added latency of an application-layer verification router, bypass custom caching entirely and offload the problem to the infrastructure.

Modern LLM providers natively support prompt caching. It is crucial to understand what is being cached here: it is not the user’s short question. When multiple users query the same corporate knowledge domain, the massive system instructions and the heavy retrieved context documents (often 10k+ tokens) are cached automatically at the provider’s inference layer.

“Stop approaching AI like a magic black box and instead approach it as a distributed systems problem.”

The application sends the full RAG query every time. The provider recognizes the repeated context block, slashes context token costs by up to 80%, and drops the time-to-first-token (TTFT) to milliseconds.  

The infrastructure surrounding a foundation model separates a prototype from a production AI system. Stop approaching AI like a magic black box and instead approach it as a distributed systems problem, and things will fall into place. Batched fan-out asynchronous queuing solves timeout and rate-limiting issues. Serverless namespacing resolves compliance risks. Prompt caching and intent routing secure unit economics. Designing a native AI product today means engineering for inevitable API failures, cross-tenant data attacks, and runaway LLM costs.

The post Can prompt caching tame RAG costs without sacrificing accuracy? appeared first on The New Stack.

Moonshot AI became the latest AI company to discover that launching a popular model is only half the battle. Less than two days after releasing Kimi K3, the company stopped accepting new subscribers after demand exhausted its available GPU capacity. Existing users will keep access while Moonshot expands its infrastructure and reopens subscriptions in batches.

Inference demand outpaces supply

The incident emphasizes how demand is outpacing available infrastructure. As AI models take on longer, more coding and agentic workloads, companies are finding they need more inference capacity than they anticipated.

“Kimi K3 has received far more love than we expected,” the official Moonshot account writes on X. “Over the past 48 hours, demand has pushed close to the limits of our current capacity. We’re adding capacity as fast as we can and will reopen new subscription spots in batches.”

“We’re adding capacity as fast as we can and will reopen new subscription spots in batches.”

For infrastructure engineers and developers, the resulting capacity crunch is a bold indication of why companies from OpenAI to Anthropic to Moonshot are rationing access instead of selling unlimited usage.

Open weights, closed capacity

At 2.8 trillion parameters, Kimi K3 is one of the largest open-weight models slated for release — Moonshot has scheduled the public weight drop for July 27. In Arena.ai’s Frontend Code Arena, K3 topped both OpenAI’s GPT-5.6 Sol and Anthropic’s Claude Fable 5. On the broader Artificial Analysis Intelligence Index, it trails both, scoring 57 to Fable 5’s 60 and Sol’s 59. That doesn’t make it any easier to run.

Open weights let anyone deploy the model, but whoever hosts it still has to pay the inference bill. Coding activities tend to tie up GPU resources far longer than a typical chatbot interaction, making it harder to keep latency low as more developers pile on.

“Agent tasks are not one-off question answering; they continuously generate, read, and process tokens during ongoing tasks,” wrote Citigroup semiconductor analyst Peter Lee in a research note.

Lee argued that as developers build longer agentic workflows, lower inference costs are quickly “re-converted into higher total resource consumption,” shifting the bottleneck from compute to server memory.

Moonshot’s subscription pause is a sign that keeping enough inference capacity online once developers start using it at scale might be just as hard as building the model.

“Agent tasks are not one-off question answering; they continuously generate, read, and process tokens during ongoing tasks.”

China’s chip constraints compound crunch

For a company like Moonshot, this general industry bottleneck is compounded by regional infrastructure realities. Unlike traditional software companies, AI developers typically rent much of this computing power from cloud providers such as Alibaba Cloud, Tencent Cloud, and Huawei Cloud rather than owning extensive data-center infrastructure themselves.

The capacity crunch illustrates the mounting challenge facing Chinese AI developers as US export controls continue to restrict access to leading chip provider Nvidia’s most advanced AI chips. As a result, companies such as Moonshot depend on a combination of older chips and domestically produced alternatives. These constraints have forced Chinese developers to concentrate strongly on software tuning and more efficient use of computing resources to narrow the performance gap with US rivals.

Token economics under pressure

The scramble for computing power has fueled a data-center construction boom across China. Alibaba has committed more than $53 billion to AI and cloud infrastructure over three years, while ByteDance is reportedly considering spending as much as $70 billion this year on AI data centers and related infrastructure.

AI companies typically charge customers based on the number of tokens, or units of text, processed by a model, making token prices a key measure of operating costs.

According to Bernstein Research, Moonshot charges $3 per million input tokens and $15 per million output tokens for Kimi K3. This makes it about 40% cheaper than Anthropic’s Opus 4.8 and roughly 70% cheaper than Claude Fable 5.

“A world where there are only [two to three] dominant frontier labs with 90 percent inference margins is net negative for every other layer while being awesome for those [two to three] labs,” wrote Atreides Management founder Gavin Baker on X. Baker argued that models like Kimi K3, Grok 4.5, and Muse 1.1 could shift value away from the model layer and toward chipmakers, cloud providers, and the software companies building the infrastructure that serves AI models.

For developers, Moonshot’s subscription freeze serves as an architectural warning. The era of assuming infinite, cheap API access is ending.

“A world where there are only [two to three] dominant frontier labs with 90 percent inference margins is net negative for every other layer while being awesome for those [two to three] labs.”

The post Moonshot launched Kimi K3. Then demand shut down subscriptions in 48 hours. appeared first on The New Stack.

Microsoft and Mistral are deepening their partnership with a multibillion-dollar agreement focused on enterprise AI infrastructure. The goal is to give organizations more flexibility over where they run frontier models and how those deployments are managed, particularly in regions with tight data residency and sovereignty requirements.

Under the deal, announced Tuesday, Microsoft will use Mistral’s expanding European compute infrastructure to increase regional capacity. Mistral, meanwhile, plans to deploy thousands of NVIDIA Vera Rubin GPUs to power everything from model training to multi-agent workloads. 

For engineering teams in regulated industries, the announcement emphasizes where enterprise AI is heading. Many organizations no longer want the public cloud to be the only place they can run AI models. They’re asking for deployments that fit into the infrastructure they already have, including on-premises and air-gapped environments.

Sovereign compute meets agentic AI 

At the compute layer, the agreement gives Microsoft access to European-operated infrastructure outside its traditional first-party data centers and leased facilities. The move builds on Microsoft’s European Digital Commitments, announced in 2025, which focus on keeping customer data in Europe and helping organizations meet regional regulatory requirements.

Crucially, Mistral will power this infrastructure with NVIDIA’s next-generation Vera Rubin rack-scale platform. Because sovereign environments support multi-step agentic workflows, this hardware pairing is essential. NVIDIA claims the platform delivers up to 10x the agent throughput at scale compared to its Grace Blackwell generation.

“Europe should have access to the world’s most capable AI without compromising control over their data, operations or digital future.”

Even though exact financial terms, capacity allocations, and rollout timelines have not been made public, the agreement positions Mistral as both an AI model supplier and an independent European compute vendor for Microsoft’s enterprise ecosystem. For European companies wary of the US CLOUD Act, the appeal is obvious — though whether a US-headquartered provider can fully place data beyond its reach remains legally contested.

“Europe should have access to the world’s most capable AI without compromising control over their data, operations or digital future,” said Microsoft Vice Chair and President Brad Smith in the announcement. 

The model layer

At the model and API tier, Microsoft announced that Mistral Medium 3.5 and Mistral OCR 4 are now live in Microsoft Foundry, with Medium 3.5 also integrated into Microsoft Copilot Studio.

Among the models coming to Azure AI Foundry is Mistral Medium 3.5, an open-weight model with 128 billion parameters and a 256,000-token context window, which could indicate it’s better suited for applications that need to work across large documents or extended conversations.

Microsoft is also adding Mistral OCR 4, which is designed for document-heavy AI workflows. The model can process documents in 170 languages and preserve information about page layout, including bounding boxes, document structure, and confidence levels.

The model can process documents in 170 languages and preserve information about page layout, including bounding boxes, document structure, and confidence levels

Air-gapped deployments go mainstream 

For developers, the primary technical win is architectural symmetry. Engineers can build, test, and fine-tune applications inside Microsoft Foundry and migrate those workloads across public Azure, Azure Local, or sovereign infrastructure operated by Mistral without having to refactor underlying workflows.

This hybrid portability targets industries bound by strict compliance regulations. The deployment options also include fully air-gapped environments, allowing organizations to run AI workloads on networks that remain completely isolated from the public internet.

Deployment flexibility drives decisions

The partnership indicates that deployment flexibility is becoming part of the buying decision. Many organizations want the freedom to run AI in the cloud, on-premises, or in regional infrastructure without being locked into a single approach. And that’s where Microsoft’s expanded relationship with Mistral fits so well.

The companies are betting that giving customers more deployment options will weigh just as much as offering competitive models. If the strategy works, Microsoft will be providing the control plane for how the world’s most regulated industries run it. 

If the strategy works, Microsoft will be providing the control plane for how the world’s most regulated industries run it. 

The post Microsoft is building an AI stack it doesn’t fully own — on purpose appeared first on The New Stack.

The fast pace of AI research means organizations now have a wide range of models to choose from that can power AI agents to solve real business problems. But choosing a model doesn’t guarantee you effective agents or even good performance. For that, you need to run your agents in an environment that provides them with tools, state, security, and scale, with fast startup times and good integration with your existing business systems.

Picking the right agent runtime environment is like picking an enterprise app server but for AI systems — and agents have very different needs from traditional applications.

Gartner predicts more than 40% of agentic projects will be canceled by 2027; not because models aren’t powerful enough to be useful but because of unclear business value, inadequate risk controls and ballooning costs.

Gartner predicts more than 40% of agentic projects will be canceled by 2027; not because models aren’t powerful enough to be useful but because of unclear business value, inadequate risk controls and ballooning costs.  Agents that deliver in proof-of-concept systems will fail in production if the runtime stack powering them can’t keep up and keep them under control. 

Agentic compute is different

It’s easy to think of an AI agent as just another microservice that takes unstructured input, runs APIs or queries, and returns messages. But infrastructure designed for traditional enterprise applications with predictable business logic or even cloud-native stateless workloads doesn’t fit agents with their bursty, long-running, stateful, non-deterministic, code-writing, tool-invoking behaviors that might be triggered by a system event or an email — not just a chat session.

Model inference needs GPUs for speed, but agents also need reliable, durable compute that supports stateful sessions for long-running processes, along with strong security and real-time visibility.

You still need to think about familiar issues like hosting, scaling, identity, and security, but all that is complicated by the unpredictable, multi-stage workflow of the agent reasoning loop.

An agent pulls in input from multiple sources, reasons over its context about the execution plan for accomplishing the goal, calls other tools or writes its own code, iterates over the results of those calls, and maybe builds on them or switches to another approach that requires another reasoning loop and eventually delivers output. That might be updating a system or sending an email rather than just displaying an answer.

Model performance is only one part of making that useful. Architecting a successful agent system that can run at enterprise scale requires considering the agent runtime, the application layer that agents call, and the tools, APIs, and MCP servers they consume.

More on Microsoft Azure

• How to secure Kubernetes in the age of AI workloads
• In the AI age, Java is more relevant than ever

You have to be able to integrate with business logic and existing systems, manage the usual quotas, rate limits and SLAs for APIs so agents don’t overload them — and you have to do all that while keeping up with AI developments that are moving too fast for you to build the infrastructure primitives you need to rely on from scratch every time.

Requirements of modern agentic infrastructure

Instead, you should look for flexible infrastructure that fits the way the agent works. As with any technology, if you build on an existing platform like Azure Container Apps, you can save effort in areas where your business can differentiate. And while AI agents have flaws (from hallucinations to high token costs) that aren’t fully solved, you can pick an agent runtime environment that makes it easier to get useful results despite them.

Agentic compute needs fast startup and resume. Whether it’s a human typing into a chat prompt or system events automatically launching multiple agents, agent infrastructure needs to spin up quickly. If it takes a few seconds to spin up a container, the reasoning loop can’t start till that’s done.

It also needs to scale up and down responsively, without costing you anything when it’s not running.

Because agent workloads are long-running and event-driven, agents need to start up, do some work, go idle for hours or even days, and then resume instantly with their memory, context, caches, connections, identity, and security intact. That means persisting and restoring state so long-running agents don’t have to pay the same startup tax over and over again. Whether the last stage of the agent’s reasoning loop was successful or a failure, it has to take another approach; you don’t want it to do the same work again.

Rather than building your own custom microVM stack, the new Azure Container Apps Sandboxes provide a temporary, secure, and stateful compute environment that spins up, executes code, snapshots disk and memory, then automatically idles and resumes just as fast. This is the stateful equivalent of Dynamic Sessions, with sub-second startup from pre-warmed pools; you can burst out to hundreds (and eventually thousands) of concurrent sandboxes when you need them, then scale back down to zero.

An agent runtime needs to be secure by default because agents are only useful when they take action — and by definition, they’re likely to do unexpected things, write and execute their own untrusted code, and keep trying to achieve their goal (sometimes even when there’s a policy that should stop certain behaviors).

To give agents secure, auditable access to the resources they need, run them and the code they generate in a sandbox, rather than on a developer laptop with production credentials and admin rights. Identity, access control, and execution boundaries have to be enforced at the runtime layer, not bolted on in a hidden prompt.

ACA Sandboxes have egress and access policies, so you can control outbound calls and limit what URLs they can access. If you can’t use managed identity for all the services an agent needs to connect to you, you can protect secrets by injecting API keys through an external egress gateway instead of hard-coding them. Because the gateway is external, outbound call decisions are controlled by policy outside the sandbox, not by agent code that will relentlessly try any method to get through.

If agents are useful, you’re going to run a lot of them, often simultaneously. AI agents need strong isolation for each task, so every untrusted code execution happens in its own sandbox and no data leaks between tasks.

You can manage them as groups, but each ACA Sandbox has its own secure boundary, so details from one customer support agent won’t end up in a chat with a different customer. If the untrusted code does turn out to be problematic, isolated sandboxes at least contain the blast radius.

Manage the tools agents call

You don’t want to rely on fragile glue code or ad hoc orchestration that you have to rewrite any time systems change for the calls agents make to tools, APIs, cloud services, and other workflows. You also don’t want to rely on manual cleanup of resources no longer needed. A runtime with built-in agent tool execution makes agents more robust. It takes the drudgery out of connecting the agent environment to your other systems, enabling them to interact securely, in isolation, and at scale.

“It’s not just about where the agent is running and what the capabilities of the agent are, but also what actions the agent can take.”

“It’s not just about where the agent is running and what the capabilities of the agent are, but also what actions the agent can take,” points out Vyom Nagrani, who runs the team of PMs responsible for both Azure Container Apps and the Azure SRE Agent that’s built on ACA Sandboxes. Sandboxes have access to a connector framework with over 1,400 enterprise-grade connectors, enabling them to take actions not only on Azure and Microsoft services but also on third-party tools.

“The runtime provides those connectors, and it provides a managed way of authenticating against all of these third-party systems, so now you can build an agent which can talk to many, many, many different systems. It’s not boxed into one authentication boundary,” Nagrani tells The New Stack.

MCP servers are a built-in capability of the connector framework. “You can take any of these connectors; you can take any REST API and expose it as an MCP server, which then the agent can consume.” Or if you want to build a custom MCP server, you can host that in a sandbox too. ACA Sandboxes can be both where agents run and, if that’s appropriate, where the tools they use are hosted.

If you’re building a user interface to wrap your agents, the application layer that makes calls to the agents can run in Azure Container Apps Express, a new service now in public preview, Nagrani says. 

“It’s a simplified app hosting stack for the app layer that responds to HTTP traffic and serves web traffic: that’s where the human interactivity comes in.”

Putting it all together

ACA Sandboxes offers an agent runtime environment that answers the key questions architects need to consider: where agents and the ephemeral compute they need access to run; where the application layer that calls agents runs; and how agents get access to all the tools, APIs, services, MCP servers, and existing business logic they need to orchestrate.

Whether you’re a platform engineer, a software vendor or a startup building a new AI platform, treating agent runtimes as the new application server and MCP servers as the new APIs requires agent infrastructure that supports agent workloads effectively, allowing governance to shift left into the runtime layer where it can scale with the ever-increasing numbers of agents. 

How real platforms build on agent runtimes

Azure Container Apps is already a strong platform for running agents. Auger, a startup launched by the former CEO of Amazon’s global consumer business to help mid-size enterprises get their complex supply chains out of Excel spreadsheets, used it to build a multi-agent system that can give real-time answers about shipments and forecasts in a world where mines or critical shipping lanes might be closed at any time by war or weather. 

Backend agents pull the unstructured data with all the details to answer those questions from different siloes, building ETL pipelines and creating an ontology of the supply chain ecosystem for each customer that includes functions and actions — all of which need to be audited and reversible. Frontend agents use that ontology to answer questions such as, “What happens if I build a new warehouse here or switch to a supplier in this country?”

ACA Sandboxes are built on Azure Container Apps, giving organizations an easier way to create agent services with strong isolation, dynamic scaling, fast startup, persistent state, and broad tool and service integrations.

That’s just what the Foundry team was looking for when they started work on the Microsoft Foundry Agent Service managed agent runtime. ACA Sandboxes gave them a platform that delivers fast start/resume, built-in tool execution (including for untrusted code), persistent state for long-running agents, strong per-agent isolation, and secure-by-default operations. Agent identity, “on‑behalf‑of” authentication to existing business services, and strict isolation are critical capabilities they didn’t have to build themselves.

“It’s the same enterprise-grade infrastructure behind Microsoft’s own agentic products, now available for all Azure customers to build on.”

The Foundry Agent Service adds a layer of visibility and observability into agent actions and conversations, allowing customers to monitor agent performance and see where the system is doing well and where it needs improvement.

“Azure Container Apps Sandboxes package the hard parts of an agent runtime into a first-class Azure resource — sub-second start and resume, built-in execution of tools and untrusted code, and snapshot-based state that lets long-running agents pick up exactly where they left off. Every agent gets its own hardware-isolated environment with secure-by-default operations, so builders can focus on what their agents do, not on the plumbing underneath,” Nagrani points out.

“It’s the same enterprise-grade infrastructure behind Microsoft’s own agentic products, now available for all Azure customers to build on.”

The post The rise of the agent runtime: The compute platform behind production agents appeared first on The New Stack.