When large language models (LLMs) hallucinate, developers typically assume the model lacks the required facts. Engineering teams diagnose the error as missing knowledge. The standard response is to increase model size, expand training data, or build complex retrieval architectures.

A new study by researchers at Google Research and Technion demonstrates that the knowledge is often not missing. The model has the information encoded parametrically but fails to surface it during generation. 

Their experiments show that frontier models like GPT-5 and Gemini-3 encode 95-98% of tested facts. This indicates that in many cases, recall, rather than encoding, is the primary bottleneck for factual accuracy. 

By understanding how to unlock existing knowledge through inference-time computation, engineering teams can build more reliable applications without necessarily relying on larger models or external databases.

Knowledge profiling: measuring what models actually know

To map this gap between storage and retrieval, the researchers propose shifting the evaluation focus from question-level accuracy to fact-level profiling. Instead of simply scoring whether an LLM answers an isolated prompt right or wrong, fact-level profiling tests a single underlying piece of information across multiple conditions, evaluating whether the fact is stored in the model's parameters at all, whether it can be queried from different directions and phrasings, and what computational effort is required to retrieve it.

This framework distinguishes between whether a fact is parametrically "encoded" and whether it is "known". A model encodes a fact if it can accurately reproduce it when primed with its original training context. A model knows a fact if it can reliably answer questions about it across varied phrasings and directions.

"Encoding and recall failures are indistinguishable under accuracy metrics, yet they imply different limitations and solutions,” the researchers write. “Encoding failures call for pre-training interventions, such as scaling model size or data coverage. Recall failures suggest post-training interventions that often improve how models utilize what they already encode."

The paper illustrates this using a sample fact: Oasis played their first gig at the Boardwalk club. Based on how models process this information, the study categorizes knowledge into five distinct profiles:

• Direct recall: The model encodes the fact and readily accesses it to answer direct questions without extra inference compute.

• Encoding failure (empty shelves): The model neither encodes nor knows the fact. It cannot complete a Wikipedia-style sentence about Oasis’s early days, nor can it answer questions about the event. This signals a need for more pre-training data or greater model capacity.

• Recall failure (lost keys): The model has the fact encoded but cannot access it. It can seamlessly complete the original training text about Oasis, but fails to answer "Where did Oasis play their first show?" even when given time to think.

• Recall with thinking: The fact is encoded, but inaccessible to direct generation. It is only successfully recalled when the model uses inference-time computation, such as Chain-of-Thought, to bridge the gap. The researchers refer to this mechanism as recall facilitation. The model might initially fail to answer the direct question. By generating intermediate thoughts about the band's early history in Manchester, it structurally primes itself to locate and recall the locked answer.

• Inference without encoding: The model never explicitly encoded the Oasis fact. Instead, it successfully answers the question by making an educated guess or reasoning across other encoded facts it does know. It might deduce the answer by chaining together separate data points, such as "Oasis formed in Manchester," "the Boardwalk was a famous 90s music club there," and "the Boardwalk hosted early gigs by emerging bands.”

Scaling illusions, long-tails, and tip-of-the-tongue recoveries

The researchers evaluated 13 LLMs on over 4 million responses. They used WikiProfile, a benchmark containing 2,150 facts extracted from Wikipedia, testing each fact across formats ranging from exact context completion to multiple-choice verification.

For frontier models like GPT-5 and Gemini-3, encoding is nearing saturation. These models successfully encode 95-98% of the tested facts. However, they still fail to directly recall 26-34% of those encoded facts without thinking. 

Inference-time thinking acts as a vital recovery mechanism. Providing models with extra computational effort successfully retrieves 40-65% of the encoded facts that models initially fail to directly recall. The researchers compare this to the human tip-of-the-tongue state, where deliberate effort, such as mentally retracing context, eventually helps remember the information.

Scaling up model size does not automatically resolve this gap. In fact, companies often mistakenly try to solve recall failures by fine-tuning larger internal models—an expensive architectural misstep.

"When facts come out wrong, the go-to move is to scale, meaning train a larger model or add more data," Nitay Calderon, Research Scientist at Google, told VentureBeat. "Both are expensive, and if the facts are already encoded, neither helps."

For example, the researchers found that scaling the Gemma3 model from 1 billion to 27 billion parameters largely filled the "empty shelves" by decreasing encoding failures from 85% to 23%. But at the same time, the share of recall failures increased, peaking at 40% without thinking.

This suggests that scaling mainly solves the storage problem rather than the access problem. As the model memorizes vastly more facts, a larger pool of knowledge becomes trapped in an "encoded but inaccessible" state. The bulk of model errors shifts from missing data to failed recall.

"Our findings suggest that recall is tightly coupled to the conditions under which facts were learned, degrading when queries diverge from training-time patterns," the researchers write. How a user asks a question directly dictates whether the model can unlock the stored answer.

For example, the experiments showed that rare facts are encoded at rates similar to popular facts. Yet they found a large recall gap between long-tail and highly popular facts that exceeds 25% for frontier models.

Similarly, models struggle to generate answers to reverse questions (i.e., asking for the subject instead of the object). For example, a model might easily answer that Oasis played their first gig at the Boardwalk club, but fail to answer who played their first gig at that same club. At the same time, the same models show that they know the correct answer when given the same question in multiple-choice format.

"Whereas these failures are often interpreted as limitations of memorization or bidirectional encoding, our results suggest a different picture: rare facts are often encoded but inaccessible, and reverse facts can be recognized even when they cannot be generated,” the researchers write. “This reframes both phenomena as recall failures rather than 'missing knowledge.'"

The ROI of thinking and tips for developers

The high encoding rates of frontier models require a shift in how developers approach factuality and pipeline architecture.

Don’t treat every factual failure as a retrieval problem: The default enterprise reaction to hallucinations is often to deploy Retrieval-Augmented Generation (RAG), scale up vector databases, or ingest more domain documents. While RAG is the right call for fresh or internal data, using it as a blanket fix for hallucinations adds latency and costs to facts the model already has locked in its parametric memory.

"A lot of what teams solve with RAG are facts the model can already answer from memory, so you're paying extra latency and per-call cost for nothing," Calderon said. "If a fact is truly missing, RAG can be the right fix. But if the fact is encoded and the model just can't recall it, RAG and scaling the model only add cost on top of the real problem."

Use inference-time reasoning selectively: Thinking recovered 40–65% of encoded facts that models failed to directly recall. However, because only 10-20% of facts actually require thinking, turning it on globally wastes your compute budget. The challenge is dynamically routing queries, as models lack the self-awareness to reliably diagnose when they are about to fail.

"To use the compute well, the model has to sense ahead of time that a plain answer is about to fail, so it can escalate before answering," Calderon said. "That self-awareness is its own skill, and today's models aren't reliably good at it." This metacognitive bottleneck is why Google researchers are developing frameworks like "faithful uncertainty" to allow models to accurately gauge their own confidence and trigger deeper reasoning rather than hallucinating.

Deploy generate-then-verify pipelines: Because models are better at recognizing facts (verification) than generating them from scratch, developers can build architectural loops where a model generates a response and is then prompted to explicitly reflect on and verify its own claims. "Since recognizing a correct answer is easier than generating one, a verify pass over the model's own output could catch mistakes that plain generation misses and add some factual improvement on top," Calderon said.

Test semantic access, not just benchmark accuracy: Standard accuracy metrics mask underlying model capabilities. Evaluation sets should probe the same underlying fact across different phrasings, contexts, and directions to truly understand what a model knows versus what it can reliably access.

Leverage query reformulation and retries: Because recall is highly context-dependent, query framing dictates success. Changing the structure of a prompt, generating relevant intermediate context, or prompting the model to generate a reasoning chain before answering are legitimate reliability mechanisms that surface information direct prompts miss.

Limitations and practical takeaways

The WikiProfile benchmark relies on encyclopedic Wikipedia facts. These findings might not perfectly generalize to proprietary or highly specialized enterprise domains. A model's ability to store and recall a niche internal company metric may behave differently than its handling of public encyclopedic data.

Fully profiling a frontier model on the WikiProfile suite costs approximately $500. Developers can significantly reduce this cost by omitting multiple-choice variants or using fewer response samples per question. 

Teams can access the WikiProfile benchmark on Hugging Face to evaluate their own systems. Because the benchmark includes the exact prompts used to build it, enterprise data engineering teams can recreate the pipeline on their own internal corpora to diagnose whether their bespoke agents are suffering from missing data or missing keys. However, teams should manage their expectations when moving away from encyclopedic data.

"The pipeline is built to be applied on a new corpus, and we provide all the prompts we used," Calderon said. "The one thing to expect: on Wikipedia it was mostly a recall problem. Domain-specific facts may genuinely not be encoded in the model."

Ultimately, this shift toward knowledge usage levels the playing field for enterprise AI stacks. "For companies that don't build models from scratch, this is good news," Calderon said. "Pre-training is hugely expensive and out of reach for most, but the levers that matter now are not: post-training can help with little data and few steps, and inference-time tools like thinking, verify steps, and retrieval are already what most teams use."

This story was updated to include remarks from Google.

Consider an AI agent tasked with a complex enterprise workflow like migrating massive batches of customer records from a legacy CRM to a cloud database. The agent cannot rely solely on its internal context window for a job spanning hours and depends on the runtime layer, aka the harness.

This harness provides execution feedback, like server logs, to help the agent maintain an accurate understanding of dynamic API connections. It also provides state trackers and control-flow mechanisms to manage completed and pending subgoals, ensuring the agent doesn't skip or duplicate data batches. When unexpected errors occur, such as a database rejecting a batch due to strict API rate limits, the harness provides tools and instructions to help the agent recover.

The main way to tell an agent how and when to use its tools is to have a human developer write a set of rules and instructions telling it what to do step-by-step. For example, a developer might instruct the agent to always search the company wiki before writing an email. Because the agent is just following a rigid script, it lacks true autonomy. It hasn't been trained to independently weigh the costs and benefits of its actions.

To solve this, researchers at Meta AI and University of Illinois Urbana–Champaign introduce EvoHarness-RL, a framework that adds a layer of abstraction to the agent's harness and teaches the underlying model when to read, update, or consolidate the information it obtains from its environment.

In long-horizon tasks, how AI agents read and process the information they obtain from their environment is pivotal to their success. The agent must update its understanding of its environment, track completed and pending subgoals, recover from failed actions, and reuse procedures from previous experience. This execution depends on the harness.

A series of self-evolving agentic frameworks like Harness-1 solve part of the problem by accumulating past trajectories and distilling them into structured procedural memory, like reusable skills, workflows, or code libraries for future tasks. However, they generally separate this long-term skill curation from real-time, within-episode state tracking. They aren't actively training the agent on how to manage its immediate environmental reality or track its active task steps while working.

Xuying Ning, co-author of the EvoHarness-RL paper, told VentureBeat that manual logic and rigid memory structures are primary culprits draining engineering resources.

"The optimal harness often changes with the model," Ning explained. "Different models may need different prompts, memory designs, permissions, or sandbox configurations. If all of this logic is manually coded, every model upgrade can lead to another long cycle of tuning and debugging."

Furthermore, existing memory systems that simply accumulate experience can actively degrade an agent's reasoning. "Append-only memory assumes that more context is always helpful, which is not necessarily true," Ning said. "Over a long task, the memory may contain outdated conclusions, failed attempts, or information that is no longer relevant." As a result, long-horizon agents need a dynamic memory capable of updating, compressing, and replacing information to avoid repeating past mistakes.

EvoHarness-RL: A unified belief, progress, and experience workspace

To overcome the limitations of rigid, manual prompts, the researchers introduce EvoHarness-RL, a training technique that teaches the agent to make optimal use of its harness. Instead of blindly following hardcoded instructions, the agent learns how to construct a structured workspace from messy execution data and decide when and how to consult that external state during complex workflows.

To simplify the management of different components of the harness, EvoHarness-RL consolidates the agent’s support systems into a single, unified interface. This interface, known as the Belief, Progress, and Experience (BPE), categorizes the agent's external needs into three functional areas:

• Belief: Maintain an accurate read on the current environment.

• Progress: Manage completed and pending subgoals.

• Experience: Reuse historical knowledge across tasks.

Instead of using complex, domain-specific APIs, the AI interacts with this clean dashboard using four compact meta-actions: track, commit, recall, and note. It issues commands to track the live environment, commit to workflow updates, recall past strategies before acting, and write notes to save newly discovered insights for future runs.

These states map directly to high-value enterprise verticals. "In software engineering, Belief can represent the agent’s current understanding of the repository," Ning said, detailing how the agent monitors component interactions and workspace changes. "Progress tracks what has already been completed, what still needs to be done, and which steps depend on others." Meanwhile, Experience captures lessons, like user feedback on a mistake, to guide future actions.

The same idea applies to finance, Ning said. During a compliance audit, Belief might describe the applicable rules and available evidence. Progress tracks which checks have been completed and which exceptions remain open. Experience helps the agent recognize recurring discrepancies or know when an issue should be escalated.

"Together, these states help prevent the agent from losing track of its work or repeating the same failed approach," Ning said.

To teach the agent both the mechanics and the strategy of managing its external workspace, the researchers designed a two-stage training recipe. In the first stage, supervised harness fine-tuning, the base model learns how to extract and structure useful facts from messy interaction logs into the BPE framework.

However, querying memory or updating trackers consumes time and compute tokens, meaning the agent cannot afford to blindly check its tools at every step. To solve this, the second stage uses “cost-aware” reinforcement learning to teach the agent efficiency. This phase trains the agent to calculate when accessing its external state is worth the budget cost. This two-step process transforms tool-use from a rigid, hardcoded prompt into a learned runtime behavior.

EvoHarness-RL in action

To validate EvoHarness-RL, the researchers evaluated the system using the ALFWorld benchmark, a text-based environment featuring multi-step tasks that test sequential logic and state tracking.

They used Qwen3-8B as the base model to train. The team pitted the trained 8B model against three large frontier models (Claude Opus 4.5, GPT-4.1, and GPT-5), frozen agent frameworks with static tools (such as ReAct, ExpeL, and ReasoningBank), and advanced trainable methods (e.g., standard GRPO, SkillOS, and SkillRL).

The results show a significant jump in performance for smaller, cost-effective models. With EvoHarness-RL, the Qwen3-8B model achieved a 96.9% average success rate, a 49.0 percentage point improvement over its baseline ReAct counterpart.

Furthermore, the trained model outperformed advanced trainable frameworks like SkillRL (89.9%) and SkillOS (80.2%). Most impressively for enterprise developers looking to optimize compute costs, the 8B model effectively matched the performance ceiling of expensive closed models like Claude Opus 4.5, which scored 96.4% out-of-the-box.

Beyond empowering smaller models, the experiments show that the BPE framework has universal benefits across all model scales, even without the extensive reinforcement learning phase. When researchers equipped frozen, out-of-the-box frontier models with the BPE prompt-time harness, their execution improved significantly. GPT-4.1's success rate improved by 22.1 points and GPT-5 by 25.7 points.

Aside from the results, the researchers recorded effects during the experiments that demonstrate the dynamic behavior the LLMs acquire as they go through the EvoHarness-RL training. During the reinforcement learning phase, they observed a behavioral shift as the agent internalized knowledge over time, which they called "harness annealing". 

Early in training, the AI relied heavily on querying its Experience and Progress trackers for almost every step. However, as it mastered routine actions, it actively reduced its reliance on external tools, embedding the successful patterns directly into its parameters. In a real-world enterprise setting, this translates directly to lower latency and reduced compute costs. By annealing its tool usage, the AI stops wasting tokens and time querying databases for standard workflows it has already mastered.

Simultaneously, the agent demonstrated "harness evolution," where it dynamically adapted its strategy based on the complexity of the situation at hand. While it bypassed its tools for simple, familiar tasks, it actively chose to scale up its use of the Belief and Experience modules the moment it encountered novel environments or unexpected roadblocks. For example, if an AI agent is migrating standard database records, it moves fast. When it encounters a strange legacy API endpoint or a complex validation error, it slows down, pulls up the live server logs, and queries its historical tickets to safely resolve the edge case rather than hallucinating a guess.

Bringing EvoHarness-RL into existing systems

Despite these massive gains, adopting a new framework often introduces friction for enterprise engineering teams. However, EvoHarness-RL utilizes an environment adapter that allows internal implementations to remain domain-specific to an organization's existing tools while sharing the trainable layer.

"I think there is significant potential to integrate BPE into existing orchestration systems," Ning said. "It does not necessarily require teams to replace their current tools or agent frameworks. BPE can work as an additional state-management layer that continuously organizes what the agent currently believes, how far it has progressed, and what it has learned."

For enterprise builders worried about inference costs, the framework addresses the hidden engineering cost of consolidation. Because consolidation requires strong reasoning, teams can adopt a hybrid, asynchronous architecture to optimize budgets.

"One possible compromise is to use a frontier model to generate high-quality consolidation data, then fine-tune a capable open-weight model to handle routine state management," Ning said. Furthermore, "because consolidation can happen asynchronously, it does not always need to slow down the agent’s main execution loop."

Teams must also carefully evaluate when a trainable BPE harness is necessary versus when it is overkill.

"For a short and stable task, ReAct or standard RAG may already be sufficient," Ning said. "BPE becomes much more valuable when an agent works for many hours, days, or even weeks." In those complex scenarios, an agent needs a compressed understanding of its decisions to avoid getting lost, relying on Experience to iteratively improve from previous failures and human feedback.

Ultimately, this approach signals a shift for AI orchestration engineers. "It is not a complete replacement of workflow engineering," Ning said, "but a transition from directly scripting agent behavior to creating systems in which better behavior can be learned."

When an agentic AI system hands a task from a small model to a larger one — or back down again — it pays a steep tax: the receiving model has to recompute the entire conversation from scratch, driving up compute costs and latency. This is a major bottleneck for enterprises building long-horizon, multi-LLM workflows.

To solve this challenge, researchers at Nvidia have introduced a cross-model KV cache transfer technique that directly maps the prefilled KV cache from a source model into the target model. This technique aligns with real-world agentic applications where large contexts accumulate across many turns. 

For real-world AI applications, cross-model KV cache transfer can reduce compute costs and latency on long-running, multi-LLM workflows — and it does so with simple linear math, not an expensive deep learning model.

Experiments show that, on compatible model pairs, this linear mapping process runs 2.7 to 25 times faster than recomputing the conversation while retaining up to 98% of the target model's standalone accuracy. 

Why swapping models mid-session is so expensive

Examining how LLMs handle memory helps understand why multi-model workflows hit a performance wall in production. When an LLM receives a prompt, it must first execute the “prefill” stage, which is the initial forward pass that computes the keys and values for all input tokens and populates the Key-Value (KV) cache. 

After that, it enters the “decode” phase, where it computes and generates the next tokens in the sequence. During this phase, the model reads from this KV cache to predict new tokens one by one, bypassing the need to re-evaluate the entire history of the conversation for each new token.

In multi-turn conversations or long-horizon agentic sessions, the context gradually becomes longer. Because the computational cost of the prefill stage scales directly with both model size and input length, processing these long sessions becomes increasingly expensive and introduces significant latency if the KV cache is invalidated.

This invalidation happens whenever the AI system tries to swap models mid-session, such as routing a complex reasoning step to a larger model or dropping to a smaller model to save costs. Because different LLMs have different architectures, they expect their cache inputs in different formats. 

As a result, any model switch forces the receiving model to repay the entire prefill cost from scratch to recompute the KV cache for the accumulated context. 

Mapping memory between models without starting over

The Nvidia researchers studied cross-model KV cache transfer to see how developers can transform the KV cache of one model into the expected format of another without running the prefill phase again. 

If solved, cross-model KV cache transfer has benefits in both directions. Small-to-large model transfer upgrades the quality of the output. For example, a cheap, small model handles the routine parts of an agentic workflow but struggles with a complex reasoning problem, and you map the KV cache to a larger model and continue the process seamlessly.

On the other hand, large-to-small model transfer reduces compute costs. A highly capable, large model might be used to unpack a massive, complex system prompt or synthesize a dense PDF at the start of a session. Once the heavy lifting is done, the session's KV cache is mapped down to a smaller, more economical model to handle the rapid-fire, conversational turns that follow.

There have been previous efforts to solve the KV cache transfer problem, but they suffer from a few key limitations. These include the need for expensive gradient-based training or very strict architectural constraints.

For this initial study, the authors restricted their focus to within-family transfers, such as transitioning between different-sized models in the Qwen, Llama, or Ministral families. These models share tokenizers, training data DNA, and core architectural styles but differ in size and depth. However, this framework leaves plenty of room for future experiments. The researchers note the technique could eventually be expanded to cross-family transfers, mismatched KV head counts, or hybrid architectures that blend standard attention with other memory mechanisms.

The key finding of the Nvidia study is that cross-model KV cache is a significantly linear structure. This means you can do the mapping with simple algebra tricks and without the need for heavy neural network training. For example, when experimenting on KV cache transfer from a 14-billion parameter Qwen3 model to a 32-billion parameter version, the authors discovered that a simple linear regression mapping from one source layer to a target layer can recover 56% of the variance in the target’s keys and 32% of the variance in its values. When combining multiple source layers, those numbers climbed to 79% and 65% respectively.

To translate this linear relationship into a practical system, the researchers designed a closed-form per-head ridge mapper with three key components:

• Per-head ridge regression: Instead of using complex deep learning to train the system, they fit a simple linear regression using a tiny calibration set of a few hundred text sequences. This technique solves a classic line-of-best-fit problem independently for every attention head.

• Cross-layer source selection: Because the source and target models have different numbers of layers, the mapper evaluates and selects the most predictive source layers to feed into each specific target layer. This way, the system picks only the most helpful pieces of memory from the old model to construct the new model's memory.

• Content-space mapping: Before translating the data, the mapper strips away the RoPE encodings. RoPE, or Rotary Position Embedding, is a standard mechanism that applies a mathematical, position-dependent rotation to the data so the model understands the order of the tokens in a sequence. Stripping the RoPE values makes it possible for the mapper to generalize to sequences of lengths larger than its training data.

Putting the linear mapper to the test

To test whether the technique works, the researchers evaluated the transfer pipeline across six “matched-KV” model families. Matched-KV means the source and target models share the same KV head count and per-head dimensions, which is typical for different-sized models within the same family.

The model families included Qwen3, Llama 3.1, and Ministral 3, with tests for KV cache transfer across different sizes ranging from 3 billion to 70 billion parameters. Their experiments included a massive 8.8x parameter leap from Llama 3.1 8B to 70B.

To cover a wide range of tasks, they evaluated the models on five core accuracy benchmarks (ARC-Challenge, HellaSwag, WinoGrande, MMLU, and GSM8K) as well as language modeling perplexity on WikiText-2 and a multi-turn conversation task called CoQA. To fit the linear translation mapper, they used a tiny calibration dataset of just 500 text sequences of 1,024 tokens each.

The researchers compared the framework against the baseline ceiling accuracy where the target model does a full, traditional prefill. They also compared their full system against ablated configurations, such as reducing the number of selected layers or deactivating different components. Additionally, they compared their simple method against a deep neural network trained with backpropagation to see if heavier deep learning could recover accuracy on pairs where the linear method struggled.

For four of the six tested pairs, the fast, closed-form linear ridge mapper retained 73% to 98% of the target's standalone prefill accuracy — including the massive leap from Llama 3.1 8B to 70B, which retained 72.8% of target accuracy.

The mapper also runs between 2.7 and 25 times faster than re-prefilling. For example, when translating a 32,768-token KV cache from a Qwen3 14B to a 32B model, the transfer took just 278 milliseconds, compared to nearly 7 seconds for a standard re-prefill.

The system also demonstrated high stability on tasks that run across many steps. When tested on multi-turn conversations, the drift, or accuracy loss, between the target baseline and the transferred cache remained incredibly small across 10 turns, proving it will not cascade into failure during long agentic sessions.

However, the straightforward linear approach did run into limitations on specific model pairs. For two of the Ministral configurations, the linear mapper degraded sharply because the simple linear fit failed to extrapolate outside calibration data. To fix this, the researchers swapped the linear mapper for a nonlinear multi-layer perceptron (MLP) with two 1,024-unit hidden layers trained on the same data. This added a complexity and training tax to the setup, but it recovered their accuracy to above 90%.

A bigger industry problem than one paper can solve

The introduction of cross-model transfer is part of a broader, industry-wide push to solve the KV cache bottleneck, which has emerged as one of the key hurdles for scaling enterprise AI. As developers push LLMs to process massive documents or code bases and execute long-running reasoning tasks, managing this memory layer is becoming as important as the models themselves.

Over the past year, researchers have attacked this compute and memory problem from multiple angles. For instance, Nvidia recently introduced dynamic memory sparsification (DMS), a technique that intelligently evicts less important tokens from the KV cache to cut reasoning costs by up to 8x. 

Other approaches focus on aggressive data compression. MIT researchers developed an algebraic compaction technique called Attention Matching that compresses the KV cache by 50x without degrading quality. Similarly, Nvidia introduced KV Cache Transform Coding (KVTC), which borrows media compression concepts to shrink memory by 20x without altering the underlying model weights.

Beyond compression, researchers are also attacking the computational overhead of memory retrieval. Optimizers like IndexCache strip away redundant layer calculations to deliver significantly faster time-to-first-token in long-context applications. And models like DeepSeek and the GLM series are optimizing the KV cache through architecture innovations.

As AI systems take on longer-horizon tasks and more complex architectures, the underlying memory infrastructure is becoming as important as the models themselves. Cross-model KV cache transfer gives developers one more tool for keeping inference costs down as they scale multi-model agentic systems.

A retrieval-augmented generation (RAG) system is built to answer strictly from the documents it retrieves. But when engineers optimize these AI pipelines end-to-end, the reader module can learn a shortcut: instead of relying on retrieved evidence, it starts answering from its own internal memory — while the system's overall accuracy keeps climbing. This is the hidden challenge of "role drift," a failure mode in compound AI systems where individual modules learn to bypass their assigned tasks even as end-to-end performance improves.

To address this, researchers at MIT and Harvard introduce Role Anchor, a technique that forces modules to stay in their lanes during training. When applied, the technique mitigates role drift. For example, it forces the RAG reader to rely on retrieved evidence instead of answering based on its internal knowledge.

The primary takeaway for practitioners is that end-to-end accuracy alone can overstate how much a compound AI system has genuinely learned. Engineers must evaluate individual components and ensure they work as intended.

Role Anchor serves as both a guardrail and a diagnostic tool when optimizing multi-step LLM pipelines. It can be essential for real-world AI applications that require a strict division of labor between modules.

Why terminal accuracy hides the problem

Compound LLM systems divide complex tasks among specialized modules. For example, a system designed for multi-hop reasoning might split a task between a "Decomposer" and a "Solver.” The Decomposer breaks a large problem down into manageable sub-tasks, while the Solver computes the answers to those sub-questions. This division of labor allows AI engineers to delegate execution to smaller, cheaper models, and makes it possible to process sub-tasks in parallel where possible.

To improve the performance of AI pipelines, engineers typically optimize them using end-to-end reinforcement learning (RL) guided by a single "terminal reward.” This means the system is evaluated on whether or not the final answer is correct (the researchers call it “terminal accuracy”). When this terminal accuracy goes up, the system is considered to be learning and working as intended.

However, terminal accuracy does not verify whether the modules properly executed the tasks they were assigned. As Xiaoyang Cao, co-author of the paper, told VentureBeat, "Terminal accuracy reduces the behavior of an entire multi-part AI system to a single number. It shows whether the final answer is correct, but says little about which components contributed or whether they followed their assigned roles."

This blind spot leads to role drift, a failure mode where a module's behavior diverges from its assigned role during optimization, even though the system's terminal accuracy continues to improve. 

"For engineering teams, the practical risk is that they can deploy a pipeline that passes every end-to-end evaluation even though its intended division of labor has silently broken down," Cao said. Because the reward system only scores the final answer, it fails to detect or penalize the module for going rogue.

Consider how this happens in the Decomposer-Solver pipeline. The Decomposer's assigned role is to write abstract sub-questions without solving the task, leaving the reasoning to the Solver. Under end-to-end RL, the Decomposer quickly learns that the weaker Solver is prone to errors on abstract tasks. To maximize the reward, the Decomposer begins leaking or planting answers into the sub-questions it sends to the Solver. The Solver ends up parroting the answer the Decomposer fed it. Terminal accuracy goes up, but the intended architecture is compromised.

But if the system is getting the right answers and accuracy is going up, why should we care if a module drifts from its role?

Real-world deployment requires much more than just a correct final answer on a training dataset. The implicit roles assigned to these modules ensure scalability, reliability, and auditability. Consider what happens when role drift takes over:

• Loss of efficiency and auditability: In the reasoning example, role drift causes the Decomposer to do all the heavy lifting instead of planning and delegating. "Once the decomposer starts putting answers directly into its sub-questions, the solvers are reduced to copying those answers," Cao said. "You are still paying to run [different modules], but they are no longer doing independent work." The workload can no longer be parallelized across multiple Solvers, it cannot be delegated to cheaper models to save compute, and downstream human stakeholders can no longer audit the system's logic step-by-step to verify how it arrived at the answer.

• Fragility in dynamic environments: Consider a RAG system, in which a Reader model is tasked to answer questions strictly using external retrieved documents. If the Reader drifts and learns to rely on its own internal parametric memory instead (because its memory happens to be accurate during training), the system becomes brittle. When the enterprise updates its database with new information, or a user asks a question about a novel topic outside the model's pretraining, the system will fail because it abandoned the grounding mechanism it was built to use.

How Role Anchor measures a role — and enforces it

"Training only for the final outcome rewards a system for producing the right answer, regardless of how it gets there," Cao said. To counter this, Role Anchor serves as a lightweight regularization technique that makes role instructions part of the training objective. It compares how the component behaves with and without those instructions and discourages training from weakening their effect. 

At a high level, it ensures the module continues to respect the steering influence of its original role prompt throughout the reinforcement learning optimization process, making role drift both measurable and controllable.

A key insight of Role Anchor is that a role’s effect can be measured by comparing how a model behaves with and without the role prompt. The system evaluates two different prompts for each module:

1. The specialized, instruction-heavy role prompt (e.g., "You are a careful Reader. Use the retrieved passages to answer the user’s questions...").

2. The neutral prompt (e.g., "Answer the user's question...").

For any given input, the model outputs a probability distribution for the next token. When run under the role prompt, it will favor certain tokens. When run under the neutral prompt, it behaves like a generic assistant. The difference between these two probability distributions is the "role utility."

This utility measures the ”nudge,” or the direction and strength with which the role prompt shifts the LLM’s default predictions. If a token is highly aligned with the assigned role, the role prompt boosts its likelihood compared to the neutral baseline (or “nudges” the model toward that token).

Before starting RL training, Role Anchor keeps a frozen copy of the model as reference and measures the role prompt's original nudge on this reference model. This pre-RL nudge serves as the ground truth of the designer's intent, acting as a proxy for how the role prompt is supposed to steer the model.

During RL training, as the active model’s weights are updated, Role Anchor regularly calculates the current nudge and compares it to the reference nudge. If the current nudge starts to fade or deviate from the reference, Role Anchor applies a penalty to the model to prevent role drift.

To see this practically, consider the RAG system evaluated by the researchers. In this pipeline, the Reader module is explicitly instructed to answer user questions based only on retrieved documents, rather than relying on its internal knowledge.

During unconstrained, outcome-only RL, the reader learns that the upstream retriever is sometimes noisy. To maximize accuracy on the training set, it starts ignoring the retrieved passages and answering from memory. Consequently, the gap between its behavior under the role prompt and the neutral prompt shrinks to the point that the reader starts behaving identically under both, ignoring the grounding instructions.

In contrast, Role Anchor detects when the reader’s nudge deviates from the reference nudge. It applies a penalty, redirecting the model’s parameters away from this memory-based shortcut. This forces the reader to find role-compliant ways to improve, such as learning how to extract answers from the retrieved passages more robustly or avoiding using its internal knowledge when the retrieved passages are faulty.

The numbers: how much of the accuracy gain was real

To test the efficacy of Role Anchor, researchers evaluated it on the RAG and Decomposer-Solver (DEC) pipelines. The experiments compared systems trained with standard outcome-only reinforcement learning (no anchor) against systems trained with Role Anchor.

Under outcome-only RL, the RAG system's terminal accuracy rose, but its internal integrity collapsed. The researchers measured "Evidence-Following Accuracy," a probe testing if the model changes its answer when the retrieved text is deliberately swapped to state the opposite. This metric plummeted from 0.86 to 0.54 (just above random chance), meaning the model learned to ignore retrieved passages and rely on its pre-trained parametric memory instead. In one test, researchers deliberately changed a piece of information in a retrieved document to contradict the model’s internal knowledge. The unanchored model did not update the response because it wasn’t using the external document.

When Role Anchor was applied, the Reader’s Evidence-Following Accuracy remained at 0.869, proving it relied strictly on the retrieved text. When researchers fed the anchored model random passages that were unrelated to the input prompt, its accuracy correctly dropped because it refused to use its internal knowledge. The unanchored model scored higher on random passages because it was guessing from memory.

The Decomposer (DEC) pipeline showed an even more dramatic failure mode. Under outcome-only RL, terminal accuracy shot up, but the "insertion rate" (i.e., the frequency at which the Decomposer leaked the answer into the sub-questions it sent to the Solver) surged from 0.143 to 0.596.

In the RAG pipeline, preserving the intended role cost the system a very modest accuracy drop (-0.067). The Reader still learned to be better at extracting answers, but it did so legitimately rather than by cheating with its internal memory. This means it is more reliable on real-world tasks with novel knowledge it has not seen during training.

In the DEC pipeline, unanchored RL improved accuracy by 0.310 above the base model, while Role Anchor only showed a 0.057 improvement. When diagnosed, it turned out that the underlying issue was that the Solver model was too small and couldn’t learn the problem-solving part. This forced the Decomposer model to cheat and provide the answer to boost the terminal accuracy. This meant 86% of the unanchored improvement was fake, and the system had simply learned to exploit a shortcut instead of learning how to reason or decompose problems better.

However, this tradeoff is not a universal rule. In some cases, eliminating shortcuts can actually boost overall performance. "Role Anchor… does not necessarily reduce final accuracy," Cao said. "In a coding pipeline we recently tested, the model had learned to manipulate its own test executor during reinforcement learning training. Adding Role Anchor completely eliminated that shortcut while slightly improving correctness on the final tests used to judge the code."

What it takes to add Role Anchor to an existing pipeline

For engineering teams looking to apply this technique, "Role Anchor can be added to an existing reinforcement learning fine-tuning process as an extra training objective for each component that a team wants to anchor," Cao said. The main pipeline and deployment setup remain entirely unchanged.

To implement it, engineers need three specific items for each anchored component: its original role instructions, a matched neutral version with the role information removed, and a saved copy of the model from before reinforcement learning fine-tuning.

Importantly, there is no latency penalty at inference time. "Role Anchor runs only while the model is being trained, so it does not slow down the deployed system," Cao said. He noted that their current implementation takes roughly 20 percent longer during training due to additional calculations, though there is likely room to optimize and reduce that overhead. The research code, training configurations, and selected model weights will be released publicly in the near future.

Deciding when to use Role Anchor is a case-by-case decision based on whether final accuracy captures everything that matters. Cao points to a regulated legal RAG system as a prime candidate. "The component producing the answer may need to follow retrieved evidence, stay grounded in an approved set of documents, and produce answers that can be traced back to their sources," he said. "Final accuracy alone cannot verify those properties, so the behavior of that component needs to be measured and enforced directly."

As enterprise AI evolves toward more complex compound pipelines, role enforcement will become harder, and relying on prompts alone will prove unreliable. "At larger scales, role specifications will need to be enforced through both training and system design," Cao said. "Methods such as Role Anchor can help preserve intended behavior during training, while clear system boundaries, limited tool permissions, and monitoring during use can provide additional safeguards."

Brex CEO Pedro Franceschi offered a blueprint for one of the pressing challenges facing the enterprise today at VB Transform 2026: securely deploying AI agents, like the open-source OpenClaw, into production environments.

Unlocking this enterprise value requires a mindset shift. The industry needs to move past vague terminology and focus on concrete enterprise roles. 

“People talk a lot about agents, but I think 'agents' is a terrible name. It's this Silicon Valley concept that doesn't really mean much,” Franceschi said. 

Instead, the goal should be creating entities that can genuinely collaborate with human workers. "The concept we always had in mind was the idea of a virtual employee — someone on Slack, an entity, it has an email address, it can join meetings, you can email it, and that you can work with," Franceschi said.

Realizing this vision demands a new security paradigm. Franceschi’s presentation detailed how Brex pointed OpenClaw at internal roles, realized traditional security models failed, and built a novel network-level security layer called CrabTrap.

The OpenClaw security dilemma

The journey began following a breakthrough in December, when coding models reached a level of maturity that enabled the January release of OpenClaw. This marked the moment agents could finally self-bootstrap and maintain their own codebases instead of relying on hard-coded, static tools. 

However, when Franceschi proposed deploying this to automate internal functions, the Brex security team firmly rejected the idea. “They said, 'Hell no. How could we trust an agent doing these things? This thing has code execution capabilities. There's no way to control it,'” Franceschi said. That caution isn't unique to Brex — enterprises broadly have been wary of granting agents uncontrolled code execution on corporate networks.

To solve this, Brex had to shift the security perimeter. Franceschi contrasted this with approaches like Nvidia's NemoClaw, which he said secure agents by limiting their tool usage — a model he believes neutralizes the coding capabilities that give agents their value.

“… the premise we had was that the coding capabilities were critical to the model having the ability to do a variety of tasks,” he said. 

Brex's fix was to shift the security boundary to the network layer instead. Instead of policing the ever-changing code inside the container, the focus must shift to monitoring what the code actually attempts to send or receive from the outside world.

CrabTrap and the LLM-as-a-judge solution

This network-centric approach led to the creation of CrabTrap, an open-source HTTP proxy built by Brex. The mechanism operates on the assumption that OpenClaw can do anything and might already be compromised. Therefore, CrabTrap monitors all outbound network traffic between the container and the internet, using an LLM to judge whether that traffic aligns with the agent's approved policy.

“Instead of trying to control the code running in the container, assume the thing can do anything and monitor the network traffic between that container and the internet,” Franceschi said. 

Using a large language model (LLM) to judge every single network request introduces unacceptable latency, often adding thousands of milliseconds to response times. Brex solved this by passing traffic through a bifurcated system. 

Routine, low-risk actions pass through static, pre-approved rules instantly. If a recruiting agent tries to view a LinkedIn profile, the static rule allows it. However, high-risk actions such as sending emails are flagged and routed to the LLM judge for evaluation. Franceschi said that architecture ensures only about 2% of complex requests actually face LLM latency. 

A surprising finding from the project was how effectively the LLM judge performs this role. Franceschi attributed this to the models' training: LLMs are exposed to billions of web pages and HTTP requests, giving them what he described as an inherent semantic understanding of network traffic patterns.

“[Models] are very good at discerning what is within the policy and what is not,” Franceschi said, adding that this capability emerges naturally through pre-training without needing heavy prompting.

Brex put this infrastructure to the test with “Jim,” a virtual recruiter built on OpenClaw. Jim handles various tasks, including sourcing candidates, scoring inbound applicants, and sending emails. 

When Jim attempts an action that falls outside the established policy, CrabTrap relies on a human-in-the-loop workflow. If the LLM judge flags an unapproved outbound email, CrabTrap pings a human manager on Slack. 

The Slack notification explains the agent's underlying intent and suggests a policy change that would allow the action. The human manager can then review the context and click "yes" or "no" to update the rules dynamically. 

"I like the virtual employee analogy because a lot of these things were solved already in a company, in the context of humans," Franceschi said. "When an employee hits a wall, they escalate to their manager."

The cost of the frontier

Brex is a fintech company, not a cybersecurity vendor. The decision to build CrabTrap in-house was driven by a lack of mature commercial solutions that could satisfy their security team. 

Franceschi acknowledged the inherent cost of operating at the bleeding edge, admitting that commercial vendor solutions will likely catch up. 

“When we built this, it was clear to me there was a 70% chance we would throw it away in six months... But what we learned by being six months ahead was worth it in shaping our AI adoption strategy,” he said. 

The investment in building internal tools provided Brex with the experience needed to safely deploy agents months ahead of the broader market. For enterprise leaders navigating the AI landscape, the core takeaway is the necessity of building the cultural and technical muscle to operate in an agentic world today. 

“We don't have all the answers, but the answer is not to do nothing,” Franceschi said.

As enterprise codebases grow, AI agents tasked with analyzing them are buckling under the weight of long-horizon tasks that require multiple interactions and tool calls. Dividing the work among a team of agents seems like the obvious fix, but it introduces a fatal flaw: most multi-agent systems are not designed for agents to coordinate among themselves mid-task and in real time.

To solve this, researchers at Coral AI Labs and multiple universities introduced AgentRadio, an asynchronous message-passing layer that allows agents to communicate between their execution steps without interrupting their main work. In real-world enterprise applications where subtasks are highly interdependent, this architecture enables agents to make mid-course corrections rather than continue on dead-end paths until a formal review phase.

On a benchmark of long-horizon questions over production repositories, a team of agents powered by AgentRadio nearly doubled task accuracy for four Claude Code agents working independently. It also outmatched single agents running on more advanced models. For AI practitioners, AgentRadio shows that the right coordination structure can outmatch raw compute and model scale.

The challenge of codebase understanding

LLM-based agents are increasingly capable of handling long-horizon tasks that require interacting with different tools and environments. Codebase understanding represents an extreme version of this challenge. It requires an AI agent to build the software, execute it, trace execution paths across multiple files, and synthesize evidence over extended periods.

Under these conditions, single-agent systems usually break down because of a “coverage problem.” 

"A single agent follows one serial path through the repository," Xinxing Ren, Caelum Forder, and Peter Carroll, co-authors of the AgentRadio paper, explained to VentureBeat. As its context grows, "the initial plan becomes harder to revise and discoveries made late in the investigation do not always propagate." The model can usually execute individual steps, but "the hard part is keeping every obligation, dependency, and piece of contradictory evidence active across a long investigation."

One benchmark that helps measure AI performance on large codebases is SWE-Atlas QnA. This benchmark consists of long-horizon, natural-language questions over live production repositories. The tasks can’t be solved by just exploring the code. AI agents must run the software and execute multiple commands to find the answers.

According to the research team’s experiments, a single Claude Code instance running on Opus 4.6 resolves just 32.3% of these tasks. Upgrading to a newer, more advanced model like Opus 4.8 only yields a 57.2% success rate.

A natural remedy is to distribute the workload across multiple agents, allowing each to work with a smaller, cleaner context. Multi-agent solutions can provide substantial performance gains when tasks are cleanly decomposable, meaning they can be solved separately and merged at the end.

Codebase understanding, however, is rarely cleanly decomposable. The subtasks are highly interdependent. A critical configuration file or a bug uncovered by one agent can completely rewrite or redirect the entire exploration path of another agent. Because of these dependencies, agents must coordinate, negotiate, and share intermediate discoveries in real time.

Despite this need, asynchronous multi-agent communication is rare. The researchers point out that existing multi-agent systems generally fall into three flawed patterns:

• Parallel but isolated: Agents operate simultaneously but do not communicate at all.

• Parallel but round-synchronized: Agents can communicate, but only at strict, synchronized round boundaries. This forces agents to stop and wait for one another to finish a round before they can debate or exchange intermediate findings. Round-based systems assume that important discoveries can wait until the next communication phase, which is an expensive assumption when agents are working on interdependent parts of a live system. For example, an agent investigating an API symptom might uncover evidence that invalidates the storage agent's current hypothesis. "If that information waits until both agents finish, the storage investigation may complete along the wrong path," the researchers said.

• Asynchrony in adjacent forms: These systems offer limited asynchronous features, such as top-down task dispatching. They don’t have peer-to-peer lateral channels between agents or shared memories that require an agent to actively pause its work to read updates.

In their paper, the researchers point out that the main bottleneck hindering current multi-agent systems is that “an agent that is working cannot also be listening.”

“To our knowledge, no existing system gives concurrently working agents passive awareness of one another over a lateral, natural-language channel,” the researchers write.

How AgentRadio works

To dissolve the mutual exclusion between working and listening, the researchers developed AgentRadio, an asynchronous message-passing layer designed to plug directly into existing coding-agent harnesses.

AgentRadio equips agents with three primitives:

• The create_thread primitive opens a conversation between participating agents.

• The send_message primitive appends a message to a thread and returns without blocking the sending agent.

• The wait_for_mention primitive blocks the process until a message mentioning the caller arrives. It delivers the message along with a full snapshot of all threads so the agent has instant context. 

This trio enables agents to have a state of “passive awareness,” where they can continue their primary tasks while passing messages and updating their knowledge in the background.

AgentRadio's code is available under the Apache 2.0 license on GitHub. It is designed to be lightweight, requiring no direct modifications to the underlying agent harnesses like Claude Code or Codex CLI. 

The architecture consists of two main parts:

• The message server: A standalone process that acts as the central hub, storing all active threads, messages, and mentions for the group of agents.

• Harness-side integration: Agents interact with the server using three simple shell scripts, one corresponding to each primitive.

The only strict requirement for the system to work is that the agent harness must be able to run a shell command as a background task. The agents are instructed in their system prompts to keep one watcher running and to send messages through the provided scripts. Running the wait_for_mention script in the background allows the agent to continue its work and receive notifications asynchronously.

To integrate this into an existing stack, a team still needs a "thin adapter that starts the workers, assigns identities, connects them to the shared server, and manages final synthesis," the researchers said. That work sits around the coding agent rather than requiring changes to the underlying model.

AgentRadio in action

To validate the real-world utility of AgentRadio, the researchers tested the framework on 124 tasks from the SWE-Atlas QnA benchmark. The tests covered domains including system design, root-cause analysis, security, and API integration.

The researchers used Claude Opus 4.6 and DeepSeek V4 Pro as the backbone models. For the harness, they evaluated configurations ranging from a single Claude Code agent (B0) to a team of agents with classic division of labor (L1), up to a team of agents using AgentRadio to coordinate asynchronously (L3).

The experimental results showed that the AgentRadio communication architecture outperforms both naive multi-agent setups and raw compute scaling.

While a single Claude Code agent with Opus 4.6 resolved only 32.3% of the tasks, the full AgentRadio setup nearly doubled that metric, resolving 62.1% of the tasks, and surpassed the single agent running on Opus 4.8, which hit 57.2%. It also boosted the DeepSeek V4 Pro results from 29.0% to 50.8%. 

To understand how this practically impacts enterprise AI, the paper highlights a real-world task involving a MinIO system. Solving the task required checking per-request server logs, a requirement the agents did not anticipate during their initial planning phase.

In the L2 setting, where agents collaborate but lack asynchronous communications, two agents independently realized they needed these logs while executing commands. Because they could not share this finding mid-execution, one agent gave up privately and the other failed to propose it to the team. During the review phase, the team unanimously agreed on the wrong answer, missing five rubrics.

With AgentRadio activated, the agents made the same mid-execution discovery, but one agent instantly broadcasted the required server-side log evidence to the shared worklog. Because the other agents were passively listening, they absorbed this new evidence immediately. This real-time coordination transformed a failing score into a perfect 16 out of 16.

"The useful distinction is timing," the researchers said. "The team did not need another agent or another review round. It needed one agent's discovery to reach the right peers before its operational value expired."

The researchers note that the same pattern appears in enterprise incident work. For example, an agent investigating an API symptom might uncover evidence that invalidates the storage agent's current hypothesis. If that information waits until both agents finish, the storage investigation may complete along the wrong path. “Passive awareness lets the second agent incorporate the contradiction at its next work step without interrupting a command already in progress,” they said.

The cost and complexity of coordination

AgentRadio requires a fixed multi-agent team budget, which inherently multiplies the token cost. The researchers acknowledge that the "tax is real," noting that average API spend rose from $2.96 per task for one Opus agent to $19.45 for the full AgentRadio stack.

However, raw scale does not equal performance. When researchers compute-matched the test by spending $17.76 on six independent Opus runs, the models only resolved 37.9% of tasks, compared with 62.1% for AgentRadio. This suggests that AgentRadio's architecture is a structural win, not just a brute-force scale win. Teams should still be aware of inter-agent churn. "Communication can redirect an agent toward better evidence, and it can also distract an agent from a valid path," the researchers warned.

A fixed multi-agent team should not become the default response to every engineering task. The more useful test to determine if a multi-agent setup is required is whether the task contains "responsibility breakpoints," the researchers said. These are places "where a competent engineer would involve another person because the work crosses an ownership boundary, needs an independent hypothesis, or carries enough risk to justify separate verification."

“Coordination is a strong fit when the task can be decomposed, the resulting parts remain interdependent, the single-agent success rate is unreliable, and an incomplete answer has a meaningful downstream cost,” the researchers said. Examples include repository-wide architecture questions, unfamiliar legacy systems, cross-service incident investigation, security analysis, dependency migrations, and multi-module refactors.

Conversely, a single agent remains the cleaner choice for “bounded, local, and reversible work,” such as a known one-file change or boilerplate generation. 

“Use one agent while one context can still own the problem honestly,” the researchers said. “Introduce another responsibility when the existing agent would otherwise need to compress away evidence, cross an independent ownership boundary, or verify its own high-impact conclusion.”

From research to commercialization: Coral Code

While AgentRadio serves as a controlled research implementation using a fixed four-agent team and a five-phase protocol, the underlying principles are being adapted into a commercial product called Coral Code.

Instead of a rigid, multi-agent protocol applied to every ticket, Coral Code works from the bottom up. An engineer begins with their existing coding agent, and Coral introduces repository-scoped investigation, specialist responsibility, and communication only when the emerging evidence justifies it. "Coral packages the operational concerns around the tools engineers already use, providing the repository context, scoped specialists, communication, and evidence layer around the harness rather than inside it," the researchers said.

This dynamic approach optimizes costs by targeting the relevant unit: the cost of a completed, reviewable outcome.

The future of autonomous software engineering

While AgentRadio provides a major upgrade to agent orchestration, there are still hurdles to overcome. One major bottleneck that the researchers pointed out to is “attention governance and verification.”

“Passive awareness makes communication available during execution. It does not decide which agents should exist, which discovery deserves an interruption, who should receive it, or when the evidence is strong enough to revise the plan,” the researchers said. If every agent receives every update, the communication layer becomes noise. If several agents share the same bad assumption, faster communication can spread the error.

For example, in one of the case studies in the paper that involved the Grafana platform, four of nine rubrics required negative conclusions, such as observing that a datasource picker did not select automatically. The agents ran the relevant tests, yet none formed the missing negative hypothesis. Both configurations failed the four rubrics. 

“Passive awareness can distribute an idea that somebody develops. It cannot supply a conception that never appears anywhere in the team,” the researchers said.

As task durations stretch longer, communication and coordination become critical. "The next generation of systems… needs adaptive responsibility assignment, evidence-aware routing, conflict resolution, explicit cost limits, permissions, recovery, and clear human escalation points," the researchers note. Most importantly, it requires durable provenance so engineering leads can inspect which agent made a claim and why an action was accepted.

"Longer-running agents make communication more important. They also make accountability much harder to fake," they said.

For developers, the operating assumption has been one engineer, one agent — the model Claude Code and similar tools. At VB Transform 2026, James Zou, associate professor of biomedical data science at Stanford University, argued that assumption is about to break: the next frontier isn't a single, more capable agent, it's tens of thousands of them collaborating.

For developers and product builders, the most critical takeaway from Zou’s presentation is how these massive systems are orchestrated. His team's research offers a practical blueprint for connecting legacy databases to AI orchestration layers and designing environments that enable thousands of agents to collaborate.

Emulating the organization — the virtual biotech

Zou’s project began as a "Virtual Lab" consisting of five to eight agents structured to mirror his physical Stanford lab. The setup included an AI professor acting as the principal investigator and AI students with distinct specialties holding regular group meetings. 

"We also created for the agents a replica of Stanford, an agent school, where the agents can actually go to the school and do supervised fine-tuning to improve their expertise in their specific domains," Zou noted.

The virtual lab successfully designed new nanobody proteins for recent COVID variants. 

"What is really exciting to us is that these AI-designed nanobody proteins actually worked much better than the previous human-designed nanobodies in terms of binding to the recent different viruses," Zou said.

Following this wet-lab validation, the team expanded their ambition. They transitioned from emulating a single research team to modeling a massive corporate structure. 

The resulting system, dubbed the Virtual Biotech, comprises tens of thousands of specialized AI agents overseen by a Chief Scientific Officer (CSO) agent. It operates through distinct corporate divisions, such as target discovery, molecule design, and clinical trials.

"Working with the CSO agent are different divisions that mirror the divisions found in a human biotech or pharma company," Zou explained — one focused on identifying drug targets, another on designing molecules, a third on safety and clinical trials. Individual agents specialize further within a division, he said. "Under the target discovery division, we'll have one agent that specializes in looking at all the genetics data, another agent that looks at all the genomics data and single-cell data, and so on."

The multi-agent advantage

As foundation models grow more capable, developers face a core architectural dilemma: Why distribute workloads across tens of thousands of specialized agents instead of channeling all computing resources into a single, omniscient model?

Zou's team ran a head-to-head comparison of a multi-agent team against a single agent tasked with the same scientific challenge. The multi-agent ecosystem created friction and interaction that produced better solutions that were more resilient against compounding errors.

"In these scientific virtual labs, the agents actually get into debates and disagreements. They have to convince the other AI scientists [of] their ideas, and all of that elicits much more creative and robust reasoning compared to if you have a single model trying to do the problem by itself from scratch," Zou said.

The orchestration bottleneck

When scaling to tens of thousands of agents, orchestration becomes the primary bottleneck. The system requires a unified context layer that allows agents to synthesize knowledge from various tools, datasets, and historical records.

Many enterprise teams attempt to solve data integration by wrapping existing databases with an MCP. However, legacy systems are not very friendly to agents. For instance, dropping a PDF of a research paper into an agent's context window is inefficient, and standard text models struggle to interpret complex figures and tables, leading to hallucinations. 

"Even if you wrap an MCP around the existing databases and APIs, that doesn't solve the underlying problem: the interface and APIs are not suitable for agents," Zou said. He added that existing databases are designed to be consumed by humans or pre-AI algorithms.

To resolve this, Zou's team created Paperclip. The platform relies on a core strength of modern LLMs: their ability to write code and navigate file systems. Instead of forcing agents to query brittle, database-specific APIs, Paperclip digitizes unstructured data and maps disparate databases into a unified, AI-native virtual file system.

This structure allows agents to access knowledge from millions of papers using standard file-system operations. 

"This basically shows that we can get much better accuracy if you use Paperclip, and we can reduce the time and the cost by over an order of magnitude compared to if you use agents without these AI-native scientific infrastructures," Zou stated.

Real-world validation

To test the practical output of this architecture, Virtual Biotech spun up 37,000 "clinical trial agents" to synthesize fragmented trial data. These agents identified single-cell features that predict trial success — drug targets supported by these features were about 50% more likely to reach market than comparable drugs without them.

The system then autonomously designed an antibody-drug conjugate (ADC) targeting the CD276 protein for lung cancer. The agents completed this design autonomously, relying exclusively on data published prior to January 2025.

Several months later, Zou said, pharmaceutical company Merck independently developed and validated the same therapeutic design — which went on to receive breakthrough designation from the FDA. He characterized this as "a third-party external validation of the therapeutic design provided by the virtual biotech agents."

Designing ecosystems, not workflows

As multi-agent systems scale, leaders must rethink how they manage these digital workforces. Zou advocated for shifting from designing rigid workflows to creating open environments. Workflows dictate the exact steps an agent should take, similar to managing a junior employee. Environments provide the infrastructure, guardrails, and incentives for agents to collaborate on open-ended problems. 

"In workflows, we're trying to tell agents what to do and how to do their job. But in environments, we're providing the infrastructures, the incentives, and the guardrails, but otherwise we leave it open to incentivize agents to collaborate," Zou said.

Optimization at scale means engineering the environment rather than fine-tuning individual models. While single agents can improve via reinforcement learning or supervised fine-tuning in the agent school, the success of a massive multi-agent system relies on adjusting the parameters governing their collaboration. 

"At the multi-agent [side], we're not actually fine-tuning and changing the individual models anymore, but we're optimizing the environment," Zou explained. "The environment itself is the object that we optimize to improve the agents."

Enterprise teams building AI agents keep hitting the same wall: a chatbot that can answer a prompt but can't remember what the last five people asked it, and can't tell you whether last month's version actually worked.

In a fireside chat with VentureBeat's Sam Witteveen at VB Transform 2026, Asana's chief product officer, Arnab Bose, unpacked how his team tackled this problem to build a new operating system: Agentic Work Management (AWM). The product treats AI agents as coachable teammates that operate alongside humans rather than as one-to-one assistants.

For product builders and developers trying to move beyond basic integrations, Bose provided a look under the hood. He detailed how Asana engineered AWM, offering a blueprint for solving real-world bottlenecks and building agentic systems at scale.

The Work Graph: 18 years of company data, repurposed

To build an operating system for human-agent teams, Asana needed a ready-made enterprise context graph. They built AWM on top of their 18-year-old architecture: the Work Graph. 

This graph-based database organizes information through a structure the company calls the Pyramid of Clarity. The smallest unit of work is a task with an assignee and a due date. Tasks belong to projects, projects roll up into portfolios, and portfolios connect to company-wide goals. The graph can help trace for example how a delayed design task impacts a corporate revenue goal. The Work Graph provides a real-time ledger of who does what, by when, and why. 

AWM leverages this architecture to create a multiplayer teammate. A standard AI copilot is stateless and tied to a single user's prompt. Because AWM plugs into the Work Graph, the AI can view overarching company goals, update project statuses, and share memory with human colleagues. 

"Because [the agent] is plugged into the Work Graph, it's not just looking at a particular prompt that you're sending it or looking at a particular individual's markdown file system on their local file,” Bose said. “It's working off of that shared ledger for the whole company."

AWM is already in production. Bose said Asana has "several customers live and successful on it," including FedEx, which published its own case study on the shift.

Building in guardrails for confidential work

Shipping AWM to enterprise customers required Asana to solve several technical hurdles. The first was data governance. If an AI teammate acts across a company, it builds a shared memory by learning from workflows and human feedback. 

Bose highlighted a critical boundary problem: If an executive uses AWM to build workflows for a confidential project, the system must ensure the agent's updated memory does not leak context to an unauthorized employee who interacts with the same agent later. 

"[I] shouldn't be able to leverage that shared memory when I run the AI teammate if you created that memory using that same teammate on a project that is, let's say, a secret M&A project that I don't have access to," Bose said. Asana engineered a system of access controls to govern what triggers the creation of a memory versus the simple execution of a task.

Second, AWM handles dynamic model routing to abstract prompt engineering away from the user. When a user assigns a task to an AI teammate (i.e., drafting a job description for a general manager role), the AI cross-references public job postings, Asana’s internal style guide, and product requirement documents. For a complex task, the system automatically routes the prompt to a heavy frontier model — Bose pointed to Anthropic's Opus and OpenAI's models as examples — while lighter tasks get down-leveled to something faster and cheaper. 

"We don't want the knowledge worker to have to think through what the best possible prompt, context engineering, and attachments are that they should put into the task," Bose said. "It should feel as if you were assigning the task to a human being."

This dynamic routing introduces a third challenge: billing abstraction. Agentic tasks vary in computational complexity, making credit burn rates unpredictable.

"We don't want to get into a state where our customers are having to reason about the fact that some of these tasks... are way more complex than others and they'll be burning credits at different rates," Bose said, adding that unpredictable pricing risked customers throttling their own employees by capping how often they could run an AI teammate.

To make AWM commercially viable, Asana designed its billing architecture to charge a static cost per task completion. The platform absorbs the complexity of model selection, token counts, and run limits to ensure predictable enterprise pricing.

The problem with stateless chatbots

AWM targets a specific problem with current enterprise AI deployments: statelessness. Developers can easily connect large language models to enterprise tools like Slack, Google Drive, or Databricks using Model Context Protocol (MCP) integrations. However, basic chat-based agents lack persistence.

Bose detailed a scenario where a user asks a chat agent to draft a marketing campaign based on historical performance and competitive research. The agent fetches data from external tools to answer the prompt, but the execution happens in a vacuum. It is a one-off task that benefits a single individual. It fails to create a reusable workflow for the next person building a similar campaign.

"The challenge with that is that those calls are stateless, and they are not leveraging a shared company brain that is this graph-based database or a context graph," Bose said. 

AWM solves this by creating a permanent state. When an AI teammate inside AWM completes a task, the system records the metadata. It registers whether the completion improved the project status and how it moved higher-level company goals. 

Inside CoreWeave's product launches

Cloud provider CoreWeave is an early adopter using AWM to overhaul complex new product launches. 

"CoreWeave is using both our deterministic AI studio workflow rules as well as multiple AI teammates to do new product launches," Bose shared. 

In the past, CoreWeave product managers filled out complicated forms detailing infrastructure, parameters, and costs. Human reviewers manually evaluated these forms and broke them out into specific tasks for finance, marketing, and hardware teams. 

Under the AWM workflow, a product manager writes a standard Google document pointing to their product requirement documents. A deterministic AI workflow reads the document, automatically creates the project structure, and assigns tasks. Specialized agents then take over the execution. One agent then watches overall project status and flags bottlenecks; another, working inside individual tasks, forecasts infrastructure costs and recommends approvals when the numbers align with historical budgets. The system automatically triages the busywork while human beings focus on evaluating the AI's outputs.

The frenemy problem

The dynamic gets complicated by the fact that the same frontier-model providers powering AWM under the hood — Anthropic, OpenAI — are also shipping their own competing agent products, like Anthropic's Claude in Slack (Tag). Pressed on the overlap, Bose didn't dispute the tension.

"I think that's the reality that we all have to live in," he said.

His case for AWM's staying power rests on Asana's 18 years of user-experience and workflow data, and prebuilt standard operating procedures for specific industries — expertise he argues raw frontier models don't have. A product like Tag can work well in Slack, he said, but it requires a highly curated channel and its own separate credentials for every downstream app it touches.

"There's a big difference between the power of the model plus a lightweight way to demonstrate its value, and something that's pre-built … for true end-to-end use," Bose said.

If you ask an AI coding agent to write a standalone Python script to parse a single JSON file, it will likely give you a perfect answer in seconds. But the same agent often breaks if you ask it to build a systematic data processing pipeline, like ingesting thousands of messy documents, chunking text, scoring quality, and filtering noise for a Retrieval-Augmented Generation (RAG) system that fits your specific enterprise stack.

While large language models (LLMs) excel at one-off code generation, their outputs for complex data-processing tasks are typically free-form, disposable scripts. These scripts are detached from the governable workflow abstractions that MLOps teams rely on for production, making them difficult to audit or edit visually.

To address this, researchers at Peking University, Zhongguancun Academy, and Shanghai’s Institute for Advanced Algorithms Research introduced DataFlow-Harness, an open-source framework that guides an LLM agent to build structured, visual data-processing workflows step-by-step, rather than writing raw code from scratch.

The framework makes AI-generated pipelines easier to manage and integrate into existing architectures because the generated artifacts are persistent and easily editable.

The researchers report that the platform achieves a 93.3% observed end-to-end pass rate on a 12-task data-engineering benchmark. Compared to standard Claude Code, it reduces API costs by up to 72.5% and response latency by 49.9%, while achieving nearly the same success rate as an AI given the entire codebase to write standard scripts. For enterprise teams, this means getting the speed of AI automation without accumulating unmanageable technical debt, ensuring that pipelines remain secure, auditable, and ready for production.

The "NL2Pipeline gap"

Data-centric AI requires workflows for tasks like synthetic data generation, retrieval augmentation, and model training. While LLMs can translate natural language into executable implementations to perform these tasks, high task accuracy is insufficient for production deployment.

"The first wall is usually not writing Python," Runming He, first author of the DataFlow-Harness paper, told VentureBeat. "Modern coding agents can often produce a plausible script quickly. The harder problem is grounding that script in a live production platform: using operators that are actually installed, matching the real dataset schema, referring to registered datasets and model services, preserving dependencies between stages, and leaving behind an artifact that another engineer can understand and revise."

General-purpose AI agents frequently hallucinate dependencies, relying on unavailable operators or outdated platform assumptions. Instead of leaving behind an artifact that another engineer can understand and revise, they generate disposable code that is difficult to audit through workflow managing tools.

The researchers define this challenge as the "NL2Pipeline gap": the disconnect between a user expressing workflow requirements in natural language and the production environment requiring structured and persistent pipeline assets.

The researchers demonstrated this gap in their experiments. For example, when Claude Code was allowed to write standard, free-form scripts using codebase context, it hit a 94.2% success rate. However, when restricted to only using the platform's specific building blocks to create a native workflow graph, its success rate dropped to 83.3%. This gap is the paper's central finding: native, governable pipelines are meaningfully harder for the agent to produce than throwaway code.

“Closing this gap requires more than improving code-generation accuracy: construction must remain grounded in platform semantics and produce artifacts that integrate with the host platform,” the researchers write.

How the four components work together

"DataFlow-Harness changes the agent’s action space," He said. "Instead of asking the agent to emit arbitrary code, it retrieves the live operator registry and current pipeline state through MCP and applies typed, incremental changes to a persistent DAG."

To achieve this, the platform organizes workflow synthesis around four components: the Data Pipeline Backend, the interaction layer (DataFlow-WebUI), the MCP Tools Layer, and the AI guidance layer (DataFlow-Skills).

The Data Pipeline Backend acts as the authoritative source of truth across conversational, visual, and programmatic interfaces. It represents the pipeline as a directed acyclic graph (DAG), a structured workflow map containing data sources, configured pre-built processing modules (which the researchers refer to as "operators"), and execution dependencies. Instead of generating free-form code, agents interact with this backend through “typed mutations,” like adding an operator or connecting edges.

DataFlow-Skills are markdown files that inject domain-specific knowledge into the model's context window, guiding it on operator-selection patterns, schema inference, and assembly procedures. Rather than letting the AI guess how to assemble components, skills provide the AI with compatibility rules, teaching it how to correctly match different data formats and handle complex data structures without breaking the pipeline. 

The MCP Tools Layer gives the AI access to the operator registry and current state of the data workflow. The AI proposes structured changes through the tools layer. The system validates the changes to ensure the workflow runs in a valid sequence and that every connected module speaks the same data language.

DataFlow-WebUI provides two interfaces that allow humans and AI to build the workflow together. Developers can describe workflow requirements in natural language through a conversational interface. They can also access the workflow as a graphical map in a visual DAG editor. Here, they can directly inspect the changes proposed by the AI and make modifications.

“The current implementation performs static checks against platform metadata before accepting pipeline changes,” He said. “These include checks for registered datasets, operators and model-serving references, field flow, and some invalid parameter usage, as well as structural validity. The result is visible in a graphical editor and can be revised either manually or by the agent in later turns.”

The results: 93.3% pass rate, 72.5% lower cost

The researchers tested DataFlow-Harness on a benchmark of 12 tasks across six industrial data-processing scenarios, such as QA generation, review governance, and schema normalization. They used Claude Opus 4.7 as the backbone model in their experiments.

They compared DataFlow-Harness against three baselines:

• Vanilla CC: An unconstrained coding baseline using standard Claude Code.

• Context-Aware CC: An agent that has access to the DataFlow codebase in its context window.

• MCP-only: An agent that has access to the DataFlow MCP tools and is instructed to generate platform-native DAGs (without access to DataFlow-Skills).

DataFlow-Harness achieved a 93.3% end-to-end pass rate, improving by 10.0 percentage points over MCP-only and beating Vanilla CC (91.7%), while being within 0.9 percentage points of Context-Aware CC (94.2%).

Importantly, it reduced API costs to $0.261 per task, a 72.5% drop compared to Vanilla CC and 42.8% compared to Context-Aware CC. In generating workflows, it was 49.9% faster than Vanilla CC and 17.6% faster than Context-Aware CC.

DataFlow-Harness proved particularly effective on complex tasks that depend on implicit domain knowledge, like QA generation. The baseline MCP-only approach frequently generated structurally valid DAGs but struggled to infer task-specific procedures from operator descriptions alone.

To show how this works in the real world, the researchers detailed a textbook-to-VQA extraction task. This job required the AI to stitch together capabilities such as PDF parsing, layout recovery, OCR, figure extraction, multimodal understanding, and long-range question-answer matching. DataFlow-Harness achieved 97.2% precision and an 87.3% coverage rate, easily beating the baselines. By having the AI snap together existing platform assets rather than coding complex tasks from scratch, it recovered more valid QA pairs from the document.

Their experiments also showed that DataFlow-Harness is highly effective at creating data generation pipelines. For example, in a synthetic instruction-data generation task, the agent built a multi-stage pipeline that generated candidate instruction–response pairs, critiqued and rewrote them, scored them with an LLM-based judge, and filtered low-quality outputs before training.

"Such workflows are costly to build and fragile to maintain as collections of ad hoc scripts," He said. "The harness does not make them automatically safe, but it turns them into explicit, editable stages that engineers can inspect, test, and govern using normal production controls."

Similarly, when tasked with building a math data cleaning-and-synthesis pipeline, the data produced by the DataFlow-Harness pipeline trained a better-performing model with higher average accuracy on AIME24 and AIME25 benchmarks than the data produced by the vanilla Claude Code pipeline.

Tech stack fit and implementation tradeoffs

For engineering teams evaluating DataFlow-Harness, it is important to understand how it fits into existing infrastructure. Released under the Apache 2.0 license, the current implementation requires a bit of engineering to fit into popular tech stacks.

"The current implementation is native to the DataFlow platform; it is not a turnkey Airflow, Prefect, or Spark plug-in," He said. To use those systems as an execution backbone, teams must build an adapter to connect their organization’s registry, metadata, and execution interfaces to the agent's control layer.

Furthermore, organizations must invest in the boundaries they want the AI to respect. This requires maintaining an operator registry, defining schemas, and encoding recurring domain procedures as Skills. Because of this overhead, He recommends against using the framework for small, one-off transformations where a simple script suffices, or in legacy environments that cannot expose reliable metadata.

Finally, while the platform prevents illogical connections by validating structural properties, it is an engineering control layer, not a compliance substitute. "The harness should still be treated as an engineering control layer, not as a substitute for compliance policy, validated detection models, access controls, audit logging, or human approval," He said.

The platform is open-source, and developers can access the source code and codebase documentation directly via the project's GitHub repository.

As protocols like MCP become standardized, the boundary between human engineers and AI agents will shift. "The goal is not autonomous data engineering without oversight," He said. "It is a better division of labor: agents perform repetitive construction inside explicit boundaries, while engineers remain responsible for the semantics, policies, and consequential decisions that require domain accountability."

Runway spent weeks trying to engineer its way out of a stubborn bug: AI-generated avatars would drift off-center during real-time video generation. The fix wasn't a back-end patch — it was a new front-end feature that just worked around the problem. That's the kind of lesson Ryan Phillips, head of enterprise product at Runway ML, walked through at VB Transform 2026, arguing that even companies not building foundation models themselves can learn from how Runway builds, evaluates, and ships them.

"I think even if you are not all building models yourselves, it's helpful to learn how we do it because I think almost all of the lessons are applicable to what you all are doing day-to-day," Phillips said.

Runway is an applied AI research company building general world models to power generative tools. During his presentation, Phillips showcased Runway Characters, a real-time video model that enables zero-latency, back-and-forth interactions with AI-generated avatars. Five years ago, creating a video with illegible text and low framerates took artists hundreds of hours of stitching individual frames together, he said. Today, Runway’s models generate interactive video on the fly.

“Studying how we build these real-time models can inspire how you build and deploy real-time experiences, whether agentic or not, in your companies today,” he said.

Demystifying evals

Building a robust AI product starts with a high-quality evaluation set. However, creating this set cannot be treated solely as an engineering task. It requires deep cross-functional alignment across product, design, research, and sales to define what "quality" actually looks like.

Phillips emphasized running internal workshops where team members review generated examples together. The goal is to align the entire organization on specific failure modes so everyone shares a unified definition of a successful generation.

“We spent a lot of time working with our team, running through examples... of what success and failure looks like, down to the very, very detailed and picky things,” Phillips said.

The resulting evaluation set must cover broad customer use cases alongside extreme edge cases. For instance, Phillips highlighted that to ensure the model behaves predictably when pushed beyond standard human facial structures, they used “Tooth,” a non-human character with no nose and very unusual teeth.

When grading these generations, the Runway team looks for subtle artifacts. In one example, a video where a character’s face remained intact but background elements, such as a net, began morphing was strictly graded as a failure.

Despite the cutting-edge nature of the product, the tool Runway uses to track these evaluations is simple: an Excel spreadsheet. The team logs tests daily, categorizing outputs as "minor" or "major" failures against a predetermined pass rate. 

“We set a bar before we get started on what percentage we need to pass, and when we hit that, we ship the model,” Phillips said. “So it's not magical.”

For enterprise developers facing non-deterministic quality drift in their own real-time pipelines, manual evaluation at scale is a bottleneck. To solve this, Phillips noted that developers can rely on language models to automate the visual grading process. 

“LLMs are getting quite good at being a judge for a lot of this content, especially the types of morphing or changing that you would see in an evaluation set,” he said. Teams can also feed an LLM behind-the-scenes context (e.g., a hand-drawn sketch or an ad's structural layout) to guide the generation and validation processes, ensuring quality without adding cognitive load to the end user.

Model training and turning bugs into features

Delivering real-time generative video requires a highly optimized technical stack. The process begins with pre-training a massive foundation model, which is resource-intensive and slow to generate outputs. To achieve real-time latency, Runway relies on distillation, where a smaller, faster "student" model is trained to mimic the large "teacher" model. According to Phillips, distillation helps Runway cut down “80 to 90% of the generation time.”

The team then applies adversarial post-training (APT) to the distilled model. This technique forces the model to continuously improve by testing it against a system designed to find its flaws, helping regain the visual sharpness lost during the distillation process.

However, altering the model architecture introduces new problems. The distillation and APT phases introduced a stubborn bug: characters would sway or drift from the center of the frame during real-time generation.

The team spent weeks attempting to fix the core model to eliminate the drift, he said. Ultimately, they discovered that if the user's initial input image was perfectly centered, the generated video remained stable. Instead of spending more time on a backend engineering patch, Runway pivoted to a user experience solution.

“What we did was, when we noticed this in our evaluations, we then said, 'What if we just offered that as a feature?' If a user gives us a character that is turned to the left, we know the video is going to morph. Let's just fix it for them,” Phillips said. 

They introduced a frontend feature called "Optimize for Image Quality," which automatically re-centers the user's image before generation begins. By wrapping a backend model limitation in a frontend tool, users perceived a helpful feature rather than an engineering flaw.

“Turn model limitations into product features so that you can actually expand how the model works,” Phillips advised. “It might feel like a limitation internally, but your customers will not see it that way if you're kind of building this in as a product feature.”

The devil is in the infrastructure details

Delivering video globally at 24 frames per second requires optimizing every layer of the infrastructure stack. This ranges from caching and parallel decoding to making deep kernel changes in partnership with hardware providers like Nvidia.

Shortly after launching Runway Characters, he said the team noticed that 8% of API calls were dropping to 16 frames per second, causing the video to stutter for customers. 

Finding the root cause required deep observability. The team used an AI agent powered by Claude alongside monitoring tools like Datadog and Sentry to trace the anomaly. The debugging session isolated the problem to a single data center in the us-east-1 region.

“The solution actually wasn't [to] go fix anything or change a config,” Phillips explained. “They actually went and physically replaced those GPUs in the data center to fix it, and that ultimately solved the problems.”

For enterprise teams deploying real-time applications, the takeaway is clear: hardware and infrastructure anomalies will directly impact model performance, requiring rigorous, full-stack debugging capabilities. 

“Don't forget about all the small details, because there's so many of them when you're deploying these models,” Phillips said.

Surviving "failure hell" and the future of world-building

Developing AI systems is rarely a linear process. Teams often find themselves stuck for weeks on a single problem with no end in sight, a phase Phillips referred to as "failure hell.”

“We think you have to go through that pain and really struggle with the problem for a little bit before you can get the breakthrough,” he said. Consistent iteration eventually flattens the difficulty curve, triggering sudden, exponential improvements.

As the underlying models overcome these technical hurdles, the role of enterprise creatives is also fundamentally changing. Traditionally, marketing and design teams have focused on creating single assets, like a specific advertisement or illustration. In an era of real-time generation and agentic workflows, that paradigm is shifting toward defining parameters, aesthetics, and intellectual property.

“You might not be designing a single ad, but you might design a world that then the agent or a real-time video model can generate ads from,” Phillips said. 

Enterprise AI is facing an ROI paradox. While throwing more compute at the strongest foundation model works well in product experiments, the costs become unbearable when the product is deployed in production.

A new paper from researchers at Writer provides a solution that is accessible to engineering teams. The study takes a systematic look at optimizing the different components of the orchestration layer that wraps around the foundation model, aka the AI harness. 

By optimizing the harness, the researchers show dramatic reductions in tokens per task, a drop in cost-per-successful-task by up to 61%, and quality that holds steady, all without changing the underlying foundation model.

Because the harness is fully under the developer's control and requires no model fine-tuning, engineering teams can apply these findings to build highly cost-efficient AI applications.

The ROI crisis of tokenmaxxing

The current state of AI engineering is plagued by "tokenmaxxing," an industry trend where developers rely on massive context windows and brute-force token consumption as a substitute for good system design. 

Rather than engineering elegant workflows, developers have imported a reflex from traditional software development: generate, run, fail, stuff the error and more context back into the window, and retry. 

"Teams tokenmaxx because it's the cheapest fix in the moment, and because it's literally how most engineers work today," Waseem AlShikh, CTO and co-founder of Writer, told VentureBeat. Because this approach succeeds often enough on coding tasks, it has become the default reflex for every other agentic workload. The danger is that per-token price drops mask the underlying inefficiency. 

"Your invoice is tokens-per-task times price-per-token, and most teams only watch the second number," AlShikh said. "In agentic workloads, tokens-per-task compounds — every loop iteration re-transmits the growing context — and it compounds faster than prices fall. The price cut becomes an anesthetic. It masks the fact that the loop itself is bleeding."

Tokenmaxxing leads to several enterprise failure modes. Teams route simple tasks to premium frontier models by default. They use the LLM as a lazy search index, stuffing the context window with raw documents instead of retrieving exact answers. Most destructively, they build unconstrained agentic loops that spiral out of control when the model encounters an error. Because output tokens cost significantly more than input tokens across all major model providers, inefficient task execution acts as a silent budget killer.

The industry has introduced several efficiency techniques to curb these costs, but they largely fall short because they treat the model in isolation: 

• Prompt compression condenses input text to save space, but ignores how the system sequences those inputs across complex workflows. 

• Budgeted reasoning caps the computational steps a model can take, which often degrades output quality if the workflow isn't intelligently routed. 

• Terse coding forces models to output minimal code to save output tokens, but does nothing to solve inefficient tool calling. 

• Speculative decoding uses a smaller draft model to speed up a larger model's text generation, optimizing inference speed while failing to address bloated agent architectures.

These efforts fail because they optimize the engine while ignoring the transmission. They do not look at the orchestration layer, leaving underlying architectural inefficiencies unresolved.

Unpacking the harness: the levers of efficiency

The harness is the orchestration layer that routes, formats, and turns the underlying LLM into a working system.

The core levers of harness optimization include system prompt caching, interaction history compaction, tool management, retrieval strategies, and error management. These are the most accessible intervention points for engineering teams looking to improve AI performance. 

As the Writer researchers note in the study: “If the harness is the layer that composes model calls into work, it is also the layer that sets the price of work.”

Historically, developers have treated the harness as disposable glue code designed simply to connect an API to a user interface. The study signals that the harness must now be treated as a first-class object: a primary software artifact that requires its own testing, versioning, and rigorous design. 

For enterprises, this reframes the "own-versus-rent" decision. 

"Enterprises spend months on model evaluations and then rent their orchestration off the shelf — which means they're optimizing the smaller lever and outsourcing the bigger one," AlShikh said. "Whoever owns the harness owns your unit economics, and an open framework tuned for demos is not tuned for your invoice." 

Inside the experiments

To isolate the impact of the orchestration layer, the researchers ran experiments on six foundation models spanning multiple vendors and weight classes: Claude Sonnet 4.6, Gemini 3.1, Gemini Flash 3.5, Qwen 3.6, GLM 5.1, and Writer’s own model, Palmyra X6. 

Their experiments compared a frozen, conventional production agent loop against the finished Writer Agent Harness on the same 22 locked enterprise tasks, spanning capabilities like grounding and retrieval, multi-step workflows, tool use, and content generation. By holding the models and tasks constant, they could isolate the effects of the orchestration layer itself.

The optimized harness drove a significant drop in costs, cutting the blended cost per task by 41%, from 21 cents to 12 cents. This was largely achieved by slashing token consumption, with the number of tokens per task falling 38%, from 14.2k to 8.8k.

The harness is designed to delegate tasks like search to specialized sub-agents. A sub-agent receives only the tool and the specific query it needs, retrieves the exact data, and returns a capped, clean summary to the main agent — keeping the primary context window from filling up with raw search results.

Task success rates held steady even as token use fell — moving from 78% to 81%, a gain the researchers describe as directional rather than statistically significant at their sample size, meaning quality didn't suffer even as costs dropped.

End-to-end task latency also dropped significantly, reducing the median wall-clock time by 44%, from 48 seconds to 27 seconds, due to prompt caching and the elimination of dead-end reasoning loops.

However, the researchers also found limits to multi-agent orchestration. Smaller models like Gemini Flash 3.5 and Qwen 3.6 scored well below a usable reliability threshold on sub-agent delegation tasks (0.45 and 0.42, respectively) — the capability simply isn't dependable yet on lighter-weight models.

Sub-agent orchestration only crossed a usable reliability threshold on the two strongest models tested: Writer's own Palmyra X6 (0.86) and Claude Sonnet 4.6 (0.85).

The developer’s playbook: actionable takeaways and tradeoffs

The findings from the study translate into a playbook for enterprise developers building agentic workflows at scale. The first step is to implement what AlShikh calls the "Two-Zone Prompt" and "Context Offloading."

Structure for system prompt caching (The Two-Zone Prompt): Modern LLM APIs offer prompt caching, but developers must structure their payloads correctly to trigger it. Developers must separate the "stable zone" from the "volatile zone." Place static, unchanging elements (e.g., core rules, large tool schemas, and standard operating procedures) at the top of the prompt. Dynamic elements, such as the specific user query or recent conversational task state, must be appended at the bottom. This ordering allows the harness to reuse the cached prefix across hundreds of calls. "That single separation makes prompt caching actually work and stops you from re-paying for the same instructions on every one of an agent's thirty steps," AlShikh said.

Manage context with Context Offloading: Avoid context stuffing, where every turn of a loop is appended into a monolithic prompt until the window maxes out. Instead, move history and intermediate artifacts out of the window into retrievable storage, and pull back only what the current step needs. If possible, delegate tasks to single-purpose sub-agents to avoid context bloat. As AlShikh points out, "the biggest line item in agent spend isn't reasoning — it's re-sending things the model has already seen."

Build resilient loops and redefine KPIs: Unmanaged agent loops drain API budgets rapidly. Teams must begin tracking Completions Per Million tokens (CPM) to understand their true task costs, but the harness itself must contain physical guardrails. "The core principle is that you never ask the model to police its own spending," AlShikh said. "The fence has to live below the model, in code, on your side of the API." This requires three hard checks:

• Hard per-task token budgets: The run terminates when the budget is spent, no exceptions.

• Generation fencing: Caps on steps, tool calls, and recursion depth to stop non-converging agents. 

• Failure-spend governance: Cap what a run can spend after its first failed validation so a failing task doesn't become your most expensive task.

Avoid unnecessary complexity: Optimizing the orchestration layer comes with engineering overhead. If you're in the prototyping and exploration stage, that overhead isn't justified — iterate fast with a strong model and a light harness. Once you're scaling to millions of requests a day, the savings from harness optimization become substantial.

However, teams must be aware of "harness leverage." Adding structural scaffolding requires the model to hold and obey that context. If a model is too small, it will spend its limited capacity parsing the scaffolding instead of doing the task, causing accuracy to drop and tokens to rise. The rule for adding complex orchestration features is strictly mathematical: "If a feature adds more coordination tokens than it removes task tokens for that specific model, cut it," AlShikh said. "Nothing in the harness is free."

The future of the enterprise harness

The era of tokenmaxxing and treating context windows like bottomless buckets is coming to an end. Throwing more compute at poorly designed systems is not a viable strategy for companies that need to demonstrate a return on their AI investments. 

As foundation models evolve to absorb planning, tool selection, and multi-step reasoning natively into their weights, the role of the harness will shift from compensating for model weakness to enforcing enterprise policy.

"What never moves into the model is the 'allowed': budgets, permissions, data boundaries, audit trails, deterministic kill-switches," AlShikh said. "Five years from now, the harness will be thinner but more important. There will be less scaffolding and more governance. However capable the model gets, someone external to it still has to define what it may spend, see, and touch. That layer belongs to the enterprise, and it should never be rented."

Model routing is becoming a key component of the enterprise AI stack, dynamically sending prompts to the right AI model to optimize speed and costs. However, current frameworks mostly treat routing as a static classification problem, which severely limits their potential.

A new open-source framework called Agent-as-a-Router tackles this bottleneck, treating the router as a dynamic, memory-building agent. It uses a Context-Action-Feedback (C-A-F) loop to track model successes and failures and update the behavior of the router. 

The researchers also released ACRouter, a concrete implementation of this paradigm. In their tests, ACRouter significantly outperformed static routers and the expensive strategy of defaulting to premium models, all without requiring teams to train massive models or write endless heuristics.

For real-world applications, this framework provides the option to replace hard-coded AI infrastructure with self-optimizing systems that can adapt to changes in user behavior and foundation models used in the enterprise AI stack. 

The economics of routing and the information deficit

Single-model setups are useful for experiments but detrimental when scaling AI applications. AI engineers use model routing to map tasks to cheaper and faster open models when possible, while reserving expensive frontier models for complex reasoning. 

Currently, developers rely on two main mechanisms for this task. The first is heuristics-based routing, which relies on hard-coded manual rules. For example, a developer might write a rule dictating that if a prompt contains certain keywords, it is routed to GPT-5.5. Otherwise, it goes to a self-hosted open source model like Kimi K2.7. 

The second mechanism is static trained policies. These are machine learning classifiers trained on historical datasets that look at the prompt's embeddings and predict the best model based on past training data.

Both approaches are static. When the researchers tested these existing mechanisms on real-world coding and agentic workflows, they found a hard ceiling on accuracy. The key finding shows that static routers suffer from a severe information deficit. Because they only evaluate the input text and never see if the model actually succeeded in executing the task, they guess blindly when faced with complex edge cases.

This results in three distinct points of failure. First, static routers suffer from a frozen information state, meaning they cannot accumulate new execution feedback during deployment. Second, they fail in out-of-distribution (OOD) generalization. They break down during day-two operations when enterprise data or user behavior shifts because their training data no longer matches reality. Finally, they are highly vulnerable to model churn. A static classifier trained on today's models may become obsolete when a better model drops the following week.

Agent-as-a-Router: A self-evolving system

The core thesis of the Agent-as-a-Router is that a truly effective router must acquire and accumulate execution-grounded information during deployment, essentially learning on the job. 

The researchers achieved this through the C-A-F loop. When a new prompt arrives, the router examines the prompt and task metadata, such as the programming language or difficulty. It then searches its historical memory for similar tasks to see which models succeeded or failed in the past. The router uses this context to select the target model and execute the task. Finally, the system observes the real-world outcome, extracts a success or failure signal, and writes this feedback back into its memory to inform future routing decisions.

Consider an automated enterprise data analytics pipeline. The router receives a SQL generation task and sends it to an open-source model like Kimi. The model hallucinates a column name and fails to compile the SQL. The C-A-F loop observes the compiler error, registers it as feedback, and logs it. The next time a similar obscure SQL query arrives, the router checks its context and routes the task to a more advanced model like Claude Opus 4.8. 

ACRouter

The researchers developed ACRouter as the concrete instantiation of this framework. It is composed of three core components: the Orchestrator, the Verifier, and Memory. This architecture is supported by a tool layer to physically execute the C-A-F loop.

The Memory module powers the context phase. Built on a vector store, it retrieves relevant past interactions and updates the historical database with new outcomes. The Orchestrator handles the action phase. It processes the user prompt alongside the retrieved memory to select the most capable target model from the available pool. The Verifier manages the feedback phase by evaluating the chosen model's output to generate a clear success or failure signal.

The tool layer hooks the Verifier into real-world execution environments, like a Python code interpreter, an agentic sandbox, or a database engine. The tool layer allows the system to execute the generated code or query and observe the exact outcome, providing the verifiable signal the router needs to learn.

The Orchestrator itself is lightweight. Instead of a massive, computationally heavy large language model, the researchers trained a sub-billion parameter adapter based on Qwen 3.5 (0.8B parameters), which means it can be self-hosted on a device of your choice.

ACRouter in action: Outperforming the frontier baselines

To stress-test the framework, the researchers introduced CodeRouterBench, an evaluation environment comprising roughly 10,000 tasks with verified scores across eight frontier models, including Claude Opus 4.6, GPT-5.4, Qwen3-Max, and GLM-5. The evaluation was split between in-distribution (ID) tests (covering nine single-turn coding dimensions like algorithm design and test generation) and an out-of-distribution (OOD) agentic programming testbed. The OOD tasks were qualitatively different, requiring multi-step planning, file navigation, and iterative debugging to see if the router could adapt to fundamentally new domains.

The baseline results revealed why a single-model strategy is flawed: no single model dominates every category. For example, while Claude Opus 4.6 achieved the highest average performance, it was outperformed in algorithm design by GLM-5 (an 86% relative improvement) and in test generation by Qwen3-Max (a 111% improvement), despite Opus costing roughly 12 times as much as smaller models like Kimi-K2.5. 

In the benchmarks, static routers continuously failed by sending a specific niche coding task to a model ill-equipped for that exact syntax. The static router had no way to know the code was failing to execute. In contrast, ACRouter adjusted its strategy after receiving negative feedback signal from the execution environment. 

According to the researchers' benchmarking, ACRouter sits firmly at the Pareto frontier of cost and performance. On both the ID task streams and the complex OOD agentic tests, ACRouter achieved the lowest cumulative regret, a metric measuring sub-optimal routing decisions over time. On the in-distribution test set, ACRouter cost $13.21 across the full task run, compared to $34.02 for always defaulting to Opus — a 2.6x savings.

It dynamically matched tasks to the most capable model for that specific niche, suggesting that enterprises can achieve or exceed frontier-level accuracy across diverse workloads without paying a premium price for every query. 

Caveats, limitations, and how to get started

While the Agent-as-a-Router paradigm solves the information deficit, it is not a blanket solution for all AI workflows. 

The framework shines in verifiable tasks where the Verifier gets a clear success or failure signal from the environment, such as coding or data retrieval. It is effective for applications with distribution shifts and domains where different models excel in completely distinct niches. 

Conversely, the setup is overkill for trivial tasks where any model will suffice, or for low-volume applications that do not justify the engineering overhead. It is also unsuitable for subjective domains, such as creative writing, where a correct answer cannot be easily verified and feedback signals are impossible to standardize.

The researchers open-sourced the code on GitHub and released the orchestrator model weights on Hugging Face under the Apache 2.0 license. The router is compatible with Claude Code, Codex, and OpenCode.

The vast majority of business data is tabular — living in data warehouses, CRMs, and financial ledgers — yet building a reliable model from it still means training a new one from scratch for every dataset, then maintaining hyperparameter tuning loops, feature engineering, and retraining pipelines to fight data drift. Google Research is proposing a way around that: a new foundation model called TabFM that treats tabular prediction as an in-context learning problem instead.

It can generate predictions for a new, unseen table in a single forward pass. For enterprise developers and AI engineers, this reduces the time-to-production from weeks of pipeline engineering to a single API call.

The challenge with traditional ML

To extract reliable predictions from a gradient-boosted tree, data scientists must build and maintain complex data pipelines. They have to clean messy inputs, impute missing values, encode categorical variables into numerical formats, and engineer custom feature crosses.

Once the data is ready, they must run repetitive hyperparameter optimization loops, searching across learning rates, tree depths, subsampling ratios, and regularization grids to find the best configuration. 

Once deployed, these traditional models "incur ongoing operational debt through data drift monitoring and retraining pipelines to stay accurate," Weihao Kong, Research Scientist at Google Research, told VentureBeat.

Meanwhile, the rest of the AI industry has moved on. Generative AI models for text and computer vision have seamlessly shifted to zero-shot inference, where a model can perform a completely new task simply by being prompted with context. 

Large language models (LLMs) already excel at in-context learning, so why can't we just feed tables into an off-the-shelf LLM?

Because LLMs are trained on natural language rather than structured data, they struggle to process tables directly. First, their context limits are exhausted quickly by medium-sized tables containing just a few thousand rows and hundreds of columns. Second, LLMs suffer from tokenization inefficiency, awkwardly splitting numerical values and destroying mathematical precision. Finally, they suffer from structural blindness. When a 2D table is serialized as a 1D text string, LLMs lose track of which value belongs to which row and column as the table grows. 

"That's why, today, it is far more effective to use an LLM to write the code that handles feature engineering and calls XGBoost than to ask the LLM to read the table itself," Kong said.

What is TabFM?

To run inference with TabFM, you do not update any model weights. Instead, you take your historical examples (the training rows with their known labels) and your target rows (the new data you want to predict) and pass them to the model as a single, unified prompt. The model learns to interpret the relationships between columns and rows directly from this context at runtime.

For example, consider an enterprise analyst trying to predict customer churn. Instead of building a bespoke data pipeline and training an XGBoost model, they can simply pass a sample of historical user session data alongside a new, active session into TabFM. In one forward pass, the model returns an instant churn probability. 

TabFM overcomes the limitations of LLMs by treating the data as a grid, preserving its structural integrity without forcing it into a single-dimensional text string.

To effectively process diverse tabular structures while enabling scalable zero-shot prediction, TabFM synthesizes the strengths of earlier experimental architectures, TabPFN and TabICL. TabPFN, developed by Prior Labs, first proved that a transformer architecture could perform zero-shot classification on small tables, though it struggled to scale computationally to larger datasets. 

Later, TabICL, developed by France's National Research Institute for Digital Science and Technology, addressed this bottleneck by introducing row compression, allowing in-context learning to efficiently process much larger tables. 

TabFM combines TabPFN's deep feature contextualization with TabICL's efficient compression into a novel hybrid design built on three key mechanisms:

1. Alternating row and column attention: The raw table is first processed through a multilayer attention module that alternates across both columns (features) and rows (examples). By continuously attending across these two dimensions, the model natively captures complex feature interactions. This deep contextualization does the heavy lifting that would usually require tedious manual feature crafting by data scientists.

2. Row compression: Following this contextualization, the cross-attended information for each row is compressed into a single, dense vector representation. TabICL pioneered this by using CLS tokens to compress a row's rich information into one vector, "in contrast to TabPFN v2, v2.5, and v2.6, which attend over the full cell grid throughout the network," Kong explained. This drastically shrinks the computational footprint.

3. In-context learning (ICL): A causal Transformer then operates on this sequence of compressed embeddings. This Transformer model uses the attention mechanism of TabICL to attend over these dense row vectors, drastically reducing the computation cost and allowing the model to process large datasets efficiently.

A major selling point of TabFM is its pretraining recipe. The model was trained entirely on hundreds of millions of synthetic datasets. These datasets were dynamically generated using structural causal models (SCMs) that incorporate a wide variety of random functions. By training exclusively on synthetic SCMs, TabFM learned the fundamental mathematical priors of how tabular features interact without ingesting real-world, confidential CSV files.

TabFM in action

To test the model's capabilities, Google researchers benchmarked TabFM on TabArena, a comprehensive evaluation suite spanning 51 diverse tabular datasets across 38 classification and 13 regression tasks.

On these public benchmarks, TabFM's zero-shot predictions already match or beat heavily tuned supervised baselines. However, Google is careful to note that this does not automatically mean TabFM will universally dethrone bespoke, hyper-optimized production models on every enterprise workload.

"Instead of replacing hyper-optimized production models, the true practical business value it unlocks for lean engineering teams is velocity," Kong said. "It allows data analysts and backend engineers to instantly spin up high-quality baseline models without a dedicated data science team managing a complex lifecycle."

For advanced practitioners looking to squeeze out maximum accuracy, the research team also introduced a "TabFM-Ensemble" configuration. By running the model through 32 distinct variations and blending the results, TabFM pushes the performance even further. 

Getting started, trade-offs, and the cloud future

The shift to in-context learning for tables introduces a new economic trade-off that engineering teams must consider. 

With traditional algorithms, training is slow and expensive, but inference is lightning-fast and cheap. TabFM flips this dynamic. While training time drops to zero, inference becomes significantly heavier. Because the model must process the entire historical dataset as context during every single prediction, it requires more compute and memory at runtime. 

In this new paradigm, "traditional machine learning training becomes the 'prefill' phase (KV caching) in the context window," Kong said. While this prefill cost is steep, it is paid only once per table, and the cache is reused across subsequent queries. "The catch is prediction latency, which no amount of caching removes," Kong added. Every new prediction requires a pass through a large transformer. "Any production API requiring single-digit-millisecond response times cannot tolerate TabFM's forward-pass overhead."

For developers looking to evaluate the model today, the barrier to entry is low. Google designed TabFM as a drop-in replacement for traditional ML workflows, offering a scikit-learn compatible API (TabFMClassifier and TabFMRegressor). It natively handles mixed numerical and categorical columns, works directly with pandas DataFrames, and requires no manual ordinal encoders or numerical scalers. The library supports both JAX and PyTorch backends.

However, enterprise teams need to be aware of current limitations and licensing restrictions. The model architecture has a hard limit of 10 output classes for classification tasks, and it is optimized for tables with up to 500 features. More importantly, while Google released the underlying codebase under the permissive Apache 2.0 license, the pre-trained model weights are published on Hugging Face under a strict tabfm-non-commercial-v1.0 license. Developers can evaluate the model internally, but it cannot be deployed in commercial products yet.

Looking ahead, Google is addressing the commercial deployment friction through its cloud ecosystem. TabFM is being integrated directly into Google BigQuery, allowing analysts to run zero-shot predictions natively via an “AI.PREDICT” command. By putting foundation model inference right next to the data warehouse, TabFM could soon make complex tabular machine learning as accessible as a basic database query.

In practice, TabFM shines in rapid prototyping, high data drift environments, and small to medium-sized datasets under 100,000 rows. Conversely, teams should stick to traditional models for strict, ultra-low latency APIs, or massive tables exceeding one million rows, which currently require aggressive row sampling that degrades the foundation model's competitive advantage.

A team routing queries across a coding specialist, a logic specialist, and a generalist model assumes each will cover the others' blind spots. A new study evaluating 67 frontier models from 21 providers shows that assumption is mathematically flawed — and the flaw has a name: the co-failure ceiling.

The assumption works like this: as long as two models don't usually fail on the exact same prompts, combining them is supposed to create a safety net against failures.

The real limit on orchestration is not how often models disagree, but the percentage of prompts where every model in the pool gives the wrong answer at once. By ignoring the co-failure ceiling, enterprises are building complex, expensive routing infrastructure to chase performance gains that do not exist. Fortunately, developers can use this same math to build a cost-free test that determines exactly when multi-model orchestration will actually pay off.

The hidden costs of the multi-model strategy

To orchestrate multiple language models, developers typically rely on three architectures. Model routers act as traffic cops, sending complex queries to expensive models and simple queries to cheaper ones. Cascades send every prompt to a cheap model first, only escalating to a premium model if the initial system signals low confidence. Finally, approaches like Mixture-of-Agents (MoA) fuse multiple models by asking them the same question and generating a synthesized answer from their combined outputs.

These architectures introduce a "shadow price" to inference costs. Every time a development team implements a router or a cascade, they pay a premium in added system latency, complex infrastructure maintenance, and increased governance risks across multiple API providers.

To justify these operational costs, engineers rely on “pairwise error correlation” to select their model pool. Imagine a developer has Model A, which writes excellent Python but fails at SQL, and Model B, which writes excellent SQL but fails at Python. Because they fail on different types of prompts, their pairwise error correlation is low. The developer assumes that by placing a routing layer in front of them, they have created a composite system that rarely fails at coding.

According to the study, throwing diverse models together based on low correlation can actually hurt performance if the models are not equally capable — when you vote across diverse but unequal models, the weaker ones often gang up and outvote the smartest one.

Josef Chen, author of the paper, told VentureBeat that in their experiments, "Naive majority voting across unequal models had negative mean gain (minus 10 points on our hard mix): diverse-but-weaker members outvote the strong one." The actionable advice for developers is to "combine only models within a matched quality band." If you cannot match quality, take the single-model baseline and spend your budget on the best model available.

The paper provides one bright spot for this approach regarding MoA architectures. When building ensembles, teams often use "Self-MoA," where they query the same premium model multiple times to generate a synthesized answer. The researchers found that at matched quality, building a diverse ensemble of models with low pairwise correlation beats a high-correlation Self-MoA setup.

However, when teams use that same pairwise correlation metric to predict the absolute accuracy of their overall system, the math breaks down.

"So teams pay the orchestration overhead up front (latency, complexity, multi-provider operations) on the assumption that a diversity dividend arrives later," Chen said. "Usually it doesn't, because today's best models agree, and, worse, they fail on the same queries … the prompt simply carries little signal about which model will be the one that's right when the frontier disagrees."

Why the math fails: the co-failure ceiling

The core finding of the study centers on a metric called the "co-failure rate" — the formal name for the all-wrong scenario described above. No router, voting system, or cascade can ever achieve an accuracy higher than the ceiling it imposes.

The coding, logic, and generalist pool shows low pairwise correlation on routine prompts — they rarely fail together. But the co-failure ceiling represents the obscure, highly complex edge case that pushes past the limits of current AI architectures. If a prompt is so difficult that all three models hallucinate or fail, it does not matter how intelligently the router distributes the task. The entire pool wipes out at once.

The researchers tested their 67-model pool, which included GPT-5.5, Claude Opus 4.8, and Gemini 3.1 Pro, on the open-ended MATH-500 math benchmark. Based on standard pairwise correlation, statistical models predicted that the entire pool would wipe out simultaneously on only 2.3% of the questions. In reality, the co-failure rate was 5.2%.

Standard correlation metrics underestimated the failure rate by roughly 2.25 times. The culprit is not just independent difficulty, but a shared failure point.

"The driver is what we call a common-mode atom: a slice of queries on which the entire market fails together, which no pairwise statistic can see," Chen said. "Adding a 20th model to your pool doesn't buy tail coverage. The tail is shared."

The researchers also found that task format directly triggers co-failure. When they took graduate-level science questions from the GPQA benchmark and changed them from multiple-choice to free-response formats, the all-wrong tail expanded to 12.7%.

Developers can engineer around the ceiling, though. "The engineering implication is uncomfortable: multi-model setups buy the least exactly where teams want them most, on open-ended generation," Chen said. "Anywhere you can convert generation into verification or constrained selection (structured outputs, checkable answers, execution tests), you reopen the ceiling."

Ultimately, the researchers found this ceiling limits AI applications in two distinct ways, depending on the domain:

• Ceiling-bound environments (e.g., open-ended math): The co-failure rate is high. The task is too hard, and all models fail simultaneously. No amount of routing can bypass the lack of underlying capability.

• Realizability-bound environments (e.g., graduate-level science): The co-failure rate is near zero, meaning at least one model in the pool usually knows the answer. However, the models disagree so subtly that a routing layer cannot reliably pick the correct answer without an omniscient oracle.

The $0 pre-deployment sanity check

Before dedicating engineering hours to building a router, teams can calculate their absolute performance ceiling for free using a mathematical formula called a Clopper-Pearson bound.

The Clopper-Pearson bound operates as a worst-case scenario calculator. If you flip a coin ten times and get eight heads, you cannot guarantee the coin will land on heads 80% of the time forever. The bound takes a small sample of test questions and outputs a mathematically guaranteed ceiling.

Applied to language models, suppose a team tests a pool of five agents on 50 sample queries and finds they all fail together on just two questions. A developer might assume their multi-agent system will achieve 96% accuracy in production. The Clopper-Pearson formula corrects this optimism. It analyzes the small sample size and provides a mathematical guarantee that the true co-failure rate could actually be as high as 12%.

To use this in practice, enterprises must build a held-out dataset. A fintech company, for example, could take 200 complex customer support tickets from the previous quarter and have human agents write perfect resolutions to serve as a benchmark. While this sounds like a heavy manual project, mature engineering teams can automate the entire ceiling calculation.

"Integration is trivial: it's a counting job over eval logs teams already produce," Chen notes, "so it runs in the same CI stage as the eval suite and re-triggers whenever the model pool or the workload changes."

The engineering team then runs its candidate models against these 200 tickets once and records the results. When they want to evaluate multi-model configurations, they can use the co-failure rate measure to predict the maximum accuracy they can get from the system without running extra queries.

One important conclusion the study draws is that on tasks where answers can be definitively checked, combining models rarely beats using the single best model on the market, unless the team possesses an exceptionally strong query-level routing signal.

In an enterprise environment, a definitively checked task has an objective, zero-tolerance answer. This includes generating a SQL query that must execute without error, extracting a specific invoice total from a 50-page PDF, or formatting a JSON payload that perfectly matches a strict schema. For these tasks, enterprises are usually better off paying a premium for the smartest frontier model rather than weaving together three cheaper models and hoping a router picks the correct output. The study didn't test subjective, ungraded tasks like drafting marketing copy — the authors note that whether these findings hold outside their verifiable benchmarks remains an open question.

Because this mathematical check is free, enterprise teams can track their own co-failure rates as new models drop.

"The measurement costs nothing, so any team can track its own co-failure rate across model generations and watch whether the tail is closing," says Chen. Ultimately, "the lever buyers hold is failure-mode heterogeneity and market churn, not model count."

As enterprise AI systems scale to handle complex workflows, practitioners face the challenge of routing subtasks to the right tools and skills. Agents can have hundreds of tools and skills and get confused on which one to use for each step of a workflow.

To address this challenge, researchers at Alibaba developed SkillWeaver, a framework that creates an execution graph for a given task and chooses the right skills for each of the nodes. They also introduce Skill-Aware Decomposition (SAD), a novel technique that uses a feedback loop to enable the agent to fetch and vet relevant tool candidates iteratively. This compositional approach and feedback loop mechanism distinguishes SkillWeaver from other tool-routing frameworks that choose tools in a one-shot fashion. 

SkillWeaver relates to real-world AI applications where agents autonomously orchestrate multi-tool ecosystems, such as the Model Context Protocol (MCP), to execute multi-step business operations like downloading datasets, transforming information, and creating visual reports. 

In practice, the researchers' experiments with SkillWeaver show that implementing this retrieve-and-route approach significantly increases accuracy while reducing token consumption by over 99% compared to naively exposing agents to an entire tool library.

For practitioners building AI agents, the main takeaway is that the granularity of task decomposition is the biggest bottleneck to accurate tool retrieval. 

The challenge of skill routing

Skills are a key pattern in modern LLM agent architectures. A skill is a modular, reusable tool specification that uses structured natural language documentation. 

As enterprise agents integrate with massive tool ecosystems, accurately routing user queries to the right skills becomes a difficult task. Exposing an entire library to an LLM to find the right tool is highly inefficient, quickly overwhelms context limits, and consumes hundreds of thousands of tokens.

Most current tool-use frameworks attempt to solve this through API retrieval, documentation matching, or hierarchical structures that treat routing strictly as a single-skill selection or per-step problem. 

However, this single-skill paradigm is insufficient for enterprise environments because real-world queries are inherently compositional. A standard business request such as "Download the dataset, transform it, and create visual reports" cannot be fulfilled by one tool. It requires breaking the prompt down and sequencing an API client, a data processor, and a visualization tool into a cohesive, multi-step execution plan.

How SkillWeaver and SAD work

To tackle this, the researchers frame the problem of handling complex tasks that require multiple skills as "compositional skill routing." Given a complex user prompt and a vast library of tools, an agent must simultaneously figure out how to break the request into a sequence of atomic sub-tasks, how to map each sub-task to the single best available skill, and how to compose those skills into an executable plan.

SkillWeaver orchestrates this process through three distinct stages: Decompose, Retrieve, and Compose. In the first stage, an LLM acts as a task decomposer, breaking the user's complex query down into a sequence of sub-tasks that each require one skill. Once the sub-tasks are clearly defined, the system uses an embedding model to compare each subtask against the skill library to pull a shortlist of the top candidate tools for each step. 

In the final stage, a planner evaluates the retrieved candidates based on how well they work together. It checks for inter-skill compatibility to ensure the outputs of one tool naturally flow into the inputs of the next. It then creates a final execution plan as a Directed Acyclic Graph (DAG) that maps out dependencies so independent tasks can potentially execute in parallel.

For example, consider a user asking an AI agent to "Download the dataset, transform it, and create visual reports." In the decompose stage, the decomposer LLM breaks this into three distinct sub-tasks: downloading the dataset, transforming the data, and creating the reports. 

In the retrieve stage, the system searches the library and finds candidates like “api-client” or “http-fetch” for task one, “csv-parser” or “etl-pipeline” for task two, and so on. Finally, the compose stage evaluates these options, selects the specific combination of “api-client,” “csv-parser,” and “chart-gen” that are most compatible, and wires them together into a final, ready-to-execute workflow.

A key challenge of this pipeline is that LLMs often produce generic step descriptions that fail to match the specific, technical vocabulary of the actual skills available in the library. To fix this, SkillWeaver introduces Iterative Skill-Aware Decomposition (SAD), a novel feedback loop. SAD works by having the LLM draft an initial plan, conducting a preliminary search to find loosely matching skills, and then feeding those retrieved skills back into the LLM as hints. This allows the LLM to rewrite its decomposition so the granularity and vocabulary perfectly align with the actual tools that exist.

SkillWeaver in action

To evaluate how SkillWeaver performs in realistic enterprise scenarios, the researchers created a custom benchmark called CompSkillBench. It consists of 300 multi-step queries of different difficulty levels. To mirror real-world environments, they used a library of 2,209 real-world skills sourced from the public MCP ecosystem, covering 24 functional categories like cloud infrastructure, finance, and databases. 

For the core engine, the researchers primarily used a lightweight 7-billion parameter model (Qwen2.5-7B-Instruct) for task decomposition, paired with a standard semantic search retriever (MiniLM with a FAISS index) to find the tools. SkillWeaver was evaluated against three main setups: a brute-force "LLM-Direct" method where they stuffed all the tool names into the prompt of a large model, a vanilla LLM-based decomposition without SAD, and a ReAct-style agent loop.

The experiments indicate that task decomposition is the main bottleneck. Standard LLM behavior falls short when dealing with large tool libraries, but the SAD feedback loop dramatically moves the needle. In the vanilla setup, the 7B model achieved a decomposition accuracy (i.e., predicting the correct number of steps) only 51.0% of the time. By activating the SAD feedback loop, accuracy jumped to 67.7% (with the larger Qwen-Max model, the accuracy reached 92%). On "hard" tasks requiring four to five distinct skills, SAD improved accuracy by 50%.

One fascinating finding was that larger models can actually perform worse when unguided. When tested in the vanilla setup, a larger 14-billion parameter model saw its accuracy plummet below the 7B model's accuracy because it tended to over-decompose tasks into microscopic, unnecessary steps. Once SAD was introduced, the retrieved tool hints anchored the model back to reality and increased its accuracy. This suggests that aligning an agent with the vocabulary of specific tools is often more impactful than paying for a larger, more expensive LLM.

Another important takeaway is token savings. The LLM-Direct baseline, which used the very large Qwen-Max model, showed that feeding all tools into the prompt of a large model fails. Despite near-perfect task breakdown capabilities, the massive model only retrieved the right tool category 21.1% of the time when flooded with tool options. SkillWeaver's targeted retrieve-and-route approach vastly outperformed this in accuracy while slashing context window consumption from an estimated 884,000 tokens down to roughly 1,160 tokens per query, a 99.9% reduction. For practitioners, this translates directly to drastically lower API costs and faster response times. 

Finally, the traditional ReAct baseline completely failed, achieving 0% decomposition accuracy. Its loop naturally collapses multi-step plans into isolated actions rather than explicitly mapping out a cohesive, multi-tool sequence.

Considerations for developers

While the researchers have not yet released the source code for SkillWeaver, their work was built on off-the-shelf tools that can easily be reproduced. 

Skill-Aware Decomposition (SAD), which is the key innovation at the heart of the framework, is a clever prompt-engineering and retrieval loop. The authors have shared the prompt templates in their paper, and developers can implement it themselves quite easily using standard orchestration libraries like LangChain, LlamaIndex, or even raw Python scripts.

As for the retrieval component, the authors built the core framework using all-MiniLM-L6-v2, an open-source embedding model. They found that swapping in a slightly stronger off-the-shelf encoder (BGE-base-en-v1.5) immediately boosted accuracy without any fine-tuning. While an off-the-shelf bi-encoder is great at getting a relevant tool into the top 10 candidates nearly 70% of the time, it struggles to consistently rank the perfect tool at exactly number one, achieving that only about 37% of the time. To bridge this gap, teams will likely need to implement a secondary cross-encoder or LLM-based reranker to re-order those top 10 candidates.

One upfront preparation requirement is vectorizing the tool library and building a FAISS index in advance. In practice, this is a negligible hurdle. Embedding and indexing all 2,209 skills in the benchmark took a mere 15 seconds. Once built, retrieving tools from the index adds less than 15 milliseconds of latency per query. For enterprise environments, syncing the tool index is a trivial background job. 

A current limitation in SkillWeaver is the lack of error recovery. While SkillWeaver successfully maps out a compatible DAG for execution, the authors' pilot study revealed the challenges of multi-step tool chains. For example, if an API call fails in step two, the entire chain breaks. The paper's core contribution is limited to the routing and planning phase. For a true production deployment, practitioners must build their own error recovery, fallback, and retry mechanisms on top of the compose stage to handle real-world API timeouts or malformed outputs.