Model routing is becoming a key component of the enterprise AI stack, dynamically sending prompts to the right AI model to optimize speed and costs. However, current frameworks mostly treat routing as a static classification problem, which severely limits their potential.

A new open-source framework called Agent-as-a-Router tackles this bottleneck, treating the router as a dynamic, memory-building agent. It uses a Context-Action-Feedback (C-A-F) loop to track model successes and failures and update the behavior of the router. 

The researchers also released ACRouter, a concrete implementation of this paradigm. In their tests, ACRouter significantly outperformed static routers and the expensive strategy of defaulting to premium models, all without requiring teams to train massive models or write endless heuristics.

For real-world applications, this framework provides the option to replace hard-coded AI infrastructure with self-optimizing systems that can adapt to changes in user behavior and foundation models used in the enterprise AI stack. 

The economics of routing and the information deficit

Single-model setups are useful for experiments but detrimental when scaling AI applications. AI engineers use model routing to map tasks to cheaper and faster open models when possible, while reserving expensive frontier models for complex reasoning. 

Currently, developers rely on two main mechanisms for this task. The first is heuristics-based routing, which relies on hard-coded manual rules. For example, a developer might write a rule dictating that if a prompt contains certain keywords, it is routed to GPT-5.5. Otherwise, it goes to a self-hosted open source model like Kimi K2.7. 

The second mechanism is static trained policies. These are machine learning classifiers trained on historical datasets that look at the prompt's embeddings and predict the best model based on past training data.

Both approaches are static. When the researchers tested these existing mechanisms on real-world coding and agentic workflows, they found a hard ceiling on accuracy. The key finding shows that static routers suffer from a severe information deficit. Because they only evaluate the input text and never see if the model actually succeeded in executing the task, they guess blindly when faced with complex edge cases.

This results in three distinct points of failure. First, static routers suffer from a frozen information state, meaning they cannot accumulate new execution feedback during deployment. Second, they fail in out-of-distribution (OOD) generalization. They break down during day-two operations when enterprise data or user behavior shifts because their training data no longer matches reality. Finally, they are highly vulnerable to model churn. A static classifier trained on today's models may become obsolete when a better model drops the following week.

Agent-as-a-Router: A self-evolving system

The core thesis of the Agent-as-a-Router is that a truly effective router must acquire and accumulate execution-grounded information during deployment, essentially learning on the job. 

The researchers achieved this through the C-A-F loop. When a new prompt arrives, the router examines the prompt and task metadata, such as the programming language or difficulty. It then searches its historical memory for similar tasks to see which models succeeded or failed in the past. The router uses this context to select the target model and execute the task. Finally, the system observes the real-world outcome, extracts a success or failure signal, and writes this feedback back into its memory to inform future routing decisions.

Consider an automated enterprise data analytics pipeline. The router receives a SQL generation task and sends it to an open-source model like Kimi. The model hallucinates a column name and fails to compile the SQL. The C-A-F loop observes the compiler error, registers it as feedback, and logs it. The next time a similar obscure SQL query arrives, the router checks its context and routes the task to a more advanced model like Claude Opus 4.8. 

ACRouter

The researchers developed ACRouter as the concrete instantiation of this framework. It is composed of three core components: the Orchestrator, the Verifier, and Memory. This architecture is supported by a tool layer to physically execute the C-A-F loop.

The Memory module powers the context phase. Built on a vector store, it retrieves relevant past interactions and updates the historical database with new outcomes. The Orchestrator handles the action phase. It processes the user prompt alongside the retrieved memory to select the most capable target model from the available pool. The Verifier manages the feedback phase by evaluating the chosen model's output to generate a clear success or failure signal.

The tool layer hooks the Verifier into real-world execution environments, like a Python code interpreter, an agentic sandbox, or a database engine. The tool layer allows the system to execute the generated code or query and observe the exact outcome, providing the verifiable signal the router needs to learn.

The Orchestrator itself is lightweight. Instead of a massive, computationally heavy large language model, the researchers trained a sub-billion parameter adapter based on Qwen 3.5 (0.8B parameters), which means it can be self-hosted on a device of your choice.

ACRouter in action: Outperforming the frontier baselines

To stress-test the framework, the researchers introduced CodeRouterBench, an evaluation environment comprising roughly 10,000 tasks with verified scores across eight frontier models, including Claude Opus 4.6, GPT-5.4, Qwen3-Max, and GLM-5. The evaluation was split between in-distribution (ID) tests (covering nine single-turn coding dimensions like algorithm design and test generation) and an out-of-distribution (OOD) agentic programming testbed. The OOD tasks were qualitatively different, requiring multi-step planning, file navigation, and iterative debugging to see if the router could adapt to fundamentally new domains.

The baseline results revealed why a single-model strategy is flawed: no single model dominates every category. For example, while Claude Opus 4.6 achieved the highest average performance, it was outperformed in algorithm design by GLM-5 (an 86% relative improvement) and in test generation by Qwen3-Max (a 111% improvement), despite Opus costing roughly 12 times as much as smaller models like Kimi-K2.5. 

In the benchmarks, static routers continuously failed by sending a specific niche coding task to a model ill-equipped for that exact syntax. The static router had no way to know the code was failing to execute. In contrast, ACRouter adjusted its strategy after receiving negative feedback signal from the execution environment. 

According to the researchers' benchmarking, ACRouter sits firmly at the Pareto frontier of cost and performance. On both the ID task streams and the complex OOD agentic tests, ACRouter achieved the lowest cumulative regret, a metric measuring sub-optimal routing decisions over time. On the in-distribution test set, ACRouter cost $13.21 across the full task run, compared to $34.02 for always defaulting to Opus — a 2.6x savings.

It dynamically matched tasks to the most capable model for that specific niche, suggesting that enterprises can achieve or exceed frontier-level accuracy across diverse workloads without paying a premium price for every query. 

Caveats, limitations, and how to get started

While the Agent-as-a-Router paradigm solves the information deficit, it is not a blanket solution for all AI workflows. 

The framework shines in verifiable tasks where the Verifier gets a clear success or failure signal from the environment, such as coding or data retrieval. It is effective for applications with distribution shifts and domains where different models excel in completely distinct niches. 

Conversely, the setup is overkill for trivial tasks where any model will suffice, or for low-volume applications that do not justify the engineering overhead. It is also unsuitable for subjective domains, such as creative writing, where a correct answer cannot be easily verified and feedback signals are impossible to standardize.

The researchers open-sourced the code on GitHub and released the orchestrator model weights on Hugging Face under the Apache 2.0 license. The router is compatible with Claude Code, Codex, and OpenCode.

The vast majority of business data is tabular — living in data warehouses, CRMs, and financial ledgers — yet building a reliable model from it still means training a new one from scratch for every dataset, then maintaining hyperparameter tuning loops, feature engineering, and retraining pipelines to fight data drift. Google Research is proposing a way around that: a new foundation model called TabFM that treats tabular prediction as an in-context learning problem instead.

It can generate predictions for a new, unseen table in a single forward pass. For enterprise developers and AI engineers, this reduces the time-to-production from weeks of pipeline engineering to a single API call.

The challenge with traditional ML

To extract reliable predictions from a gradient-boosted tree, data scientists must build and maintain complex data pipelines. They have to clean messy inputs, impute missing values, encode categorical variables into numerical formats, and engineer custom feature crosses.

Once the data is ready, they must run repetitive hyperparameter optimization loops, searching across learning rates, tree depths, subsampling ratios, and regularization grids to find the best configuration. 

Once deployed, these traditional models "incur ongoing operational debt through data drift monitoring and retraining pipelines to stay accurate," Weihao Kong, Research Scientist at Google Research, told VentureBeat.

Meanwhile, the rest of the AI industry has moved on. Generative AI models for text and computer vision have seamlessly shifted to zero-shot inference, where a model can perform a completely new task simply by being prompted with context. 

Large language models (LLMs) already excel at in-context learning, so why can't we just feed tables into an off-the-shelf LLM?

Because LLMs are trained on natural language rather than structured data, they struggle to process tables directly. First, their context limits are exhausted quickly by medium-sized tables containing just a few thousand rows and hundreds of columns. Second, LLMs suffer from tokenization inefficiency, awkwardly splitting numerical values and destroying mathematical precision. Finally, they suffer from structural blindness. When a 2D table is serialized as a 1D text string, LLMs lose track of which value belongs to which row and column as the table grows. 

"That's why, today, it is far more effective to use an LLM to write the code that handles feature engineering and calls XGBoost than to ask the LLM to read the table itself," Kong said.

What is TabFM?

To run inference with TabFM, you do not update any model weights. Instead, you take your historical examples (the training rows with their known labels) and your target rows (the new data you want to predict) and pass them to the model as a single, unified prompt. The model learns to interpret the relationships between columns and rows directly from this context at runtime.

For example, consider an enterprise analyst trying to predict customer churn. Instead of building a bespoke data pipeline and training an XGBoost model, they can simply pass a sample of historical user session data alongside a new, active session into TabFM. In one forward pass, the model returns an instant churn probability. 

TabFM overcomes the limitations of LLMs by treating the data as a grid, preserving its structural integrity without forcing it into a single-dimensional text string.

To effectively process diverse tabular structures while enabling scalable zero-shot prediction, TabFM synthesizes the strengths of earlier experimental architectures, TabPFN and TabICL. TabPFN, developed by Prior Labs, first proved that a transformer architecture could perform zero-shot classification on small tables, though it struggled to scale computationally to larger datasets. 

Later, TabICL, developed by France's National Research Institute for Digital Science and Technology, addressed this bottleneck by introducing row compression, allowing in-context learning to efficiently process much larger tables. 

TabFM combines TabPFN's deep feature contextualization with TabICL's efficient compression into a novel hybrid design built on three key mechanisms:

1. Alternating row and column attention: The raw table is first processed through a multilayer attention module that alternates across both columns (features) and rows (examples). By continuously attending across these two dimensions, the model natively captures complex feature interactions. This deep contextualization does the heavy lifting that would usually require tedious manual feature crafting by data scientists.

2. Row compression: Following this contextualization, the cross-attended information for each row is compressed into a single, dense vector representation. TabICL pioneered this by using CLS tokens to compress a row's rich information into one vector, "in contrast to TabPFN v2, v2.5, and v2.6, which attend over the full cell grid throughout the network," Kong explained. This drastically shrinks the computational footprint.

3. In-context learning (ICL): A causal Transformer then operates on this sequence of compressed embeddings. This Transformer model uses the attention mechanism of TabICL to attend over these dense row vectors, drastically reducing the computation cost and allowing the model to process large datasets efficiently.

A major selling point of TabFM is its pretraining recipe. The model was trained entirely on hundreds of millions of synthetic datasets. These datasets were dynamically generated using structural causal models (SCMs) that incorporate a wide variety of random functions. By training exclusively on synthetic SCMs, TabFM learned the fundamental mathematical priors of how tabular features interact without ingesting real-world, confidential CSV files.

TabFM in action

To test the model's capabilities, Google researchers benchmarked TabFM on TabArena, a comprehensive evaluation suite spanning 51 diverse tabular datasets across 38 classification and 13 regression tasks.

On these public benchmarks, TabFM's zero-shot predictions already match or beat heavily tuned supervised baselines. However, Google is careful to note that this does not automatically mean TabFM will universally dethrone bespoke, hyper-optimized production models on every enterprise workload.

"Instead of replacing hyper-optimized production models, the true practical business value it unlocks for lean engineering teams is velocity," Kong said. "It allows data analysts and backend engineers to instantly spin up high-quality baseline models without a dedicated data science team managing a complex lifecycle."

For advanced practitioners looking to squeeze out maximum accuracy, the research team also introduced a "TabFM-Ensemble" configuration. By running the model through 32 distinct variations and blending the results, TabFM pushes the performance even further. 

Getting started, trade-offs, and the cloud future

The shift to in-context learning for tables introduces a new economic trade-off that engineering teams must consider. 

With traditional algorithms, training is slow and expensive, but inference is lightning-fast and cheap. TabFM flips this dynamic. While training time drops to zero, inference becomes significantly heavier. Because the model must process the entire historical dataset as context during every single prediction, it requires more compute and memory at runtime. 

In this new paradigm, "traditional machine learning training becomes the 'prefill' phase (KV caching) in the context window," Kong said. While this prefill cost is steep, it is paid only once per table, and the cache is reused across subsequent queries. "The catch is prediction latency, which no amount of caching removes," Kong added. Every new prediction requires a pass through a large transformer. "Any production API requiring single-digit-millisecond response times cannot tolerate TabFM's forward-pass overhead."

For developers looking to evaluate the model today, the barrier to entry is low. Google designed TabFM as a drop-in replacement for traditional ML workflows, offering a scikit-learn compatible API (TabFMClassifier and TabFMRegressor). It natively handles mixed numerical and categorical columns, works directly with pandas DataFrames, and requires no manual ordinal encoders or numerical scalers. The library supports both JAX and PyTorch backends.

However, enterprise teams need to be aware of current limitations and licensing restrictions. The model architecture has a hard limit of 10 output classes for classification tasks, and it is optimized for tables with up to 500 features. More importantly, while Google released the underlying codebase under the permissive Apache 2.0 license, the pre-trained model weights are published on Hugging Face under a strict tabfm-non-commercial-v1.0 license. Developers can evaluate the model internally, but it cannot be deployed in commercial products yet.

Looking ahead, Google is addressing the commercial deployment friction through its cloud ecosystem. TabFM is being integrated directly into Google BigQuery, allowing analysts to run zero-shot predictions natively via an “AI.PREDICT” command. By putting foundation model inference right next to the data warehouse, TabFM could soon make complex tabular machine learning as accessible as a basic database query.

In practice, TabFM shines in rapid prototyping, high data drift environments, and small to medium-sized datasets under 100,000 rows. Conversely, teams should stick to traditional models for strict, ultra-low latency APIs, or massive tables exceeding one million rows, which currently require aggressive row sampling that degrades the foundation model's competitive advantage.

A team routing queries across a coding specialist, a logic specialist, and a generalist model assumes each will cover the others' blind spots. A new study evaluating 67 frontier models from 21 providers shows that assumption is mathematically flawed — and the flaw has a name: the co-failure ceiling.

The assumption works like this: as long as two models don't usually fail on the exact same prompts, combining them is supposed to create a safety net against failures.

The real limit on orchestration is not how often models disagree, but the percentage of prompts where every model in the pool gives the wrong answer at once. By ignoring the co-failure ceiling, enterprises are building complex, expensive routing infrastructure to chase performance gains that do not exist. Fortunately, developers can use this same math to build a cost-free test that determines exactly when multi-model orchestration will actually pay off.

The hidden costs of the multi-model strategy

To orchestrate multiple language models, developers typically rely on three architectures. Model routers act as traffic cops, sending complex queries to expensive models and simple queries to cheaper ones. Cascades send every prompt to a cheap model first, only escalating to a premium model if the initial system signals low confidence. Finally, approaches like Mixture-of-Agents (MoA) fuse multiple models by asking them the same question and generating a synthesized answer from their combined outputs.

These architectures introduce a "shadow price" to inference costs. Every time a development team implements a router or a cascade, they pay a premium in added system latency, complex infrastructure maintenance, and increased governance risks across multiple API providers.

To justify these operational costs, engineers rely on “pairwise error correlation” to select their model pool. Imagine a developer has Model A, which writes excellent Python but fails at SQL, and Model B, which writes excellent SQL but fails at Python. Because they fail on different types of prompts, their pairwise error correlation is low. The developer assumes that by placing a routing layer in front of them, they have created a composite system that rarely fails at coding.

According to the study, throwing diverse models together based on low correlation can actually hurt performance if the models are not equally capable — when you vote across diverse but unequal models, the weaker ones often gang up and outvote the smartest one.

Josef Chen, author of the paper, told VentureBeat that in their experiments, "Naive majority voting across unequal models had negative mean gain (minus 10 points on our hard mix): diverse-but-weaker members outvote the strong one." The actionable advice for developers is to "combine only models within a matched quality band." If you cannot match quality, take the single-model baseline and spend your budget on the best model available.

The paper provides one bright spot for this approach regarding MoA architectures. When building ensembles, teams often use "Self-MoA," where they query the same premium model multiple times to generate a synthesized answer. The researchers found that at matched quality, building a diverse ensemble of models with low pairwise correlation beats a high-correlation Self-MoA setup.

However, when teams use that same pairwise correlation metric to predict the absolute accuracy of their overall system, the math breaks down.

"So teams pay the orchestration overhead up front (latency, complexity, multi-provider operations) on the assumption that a diversity dividend arrives later," Chen said. "Usually it doesn't, because today's best models agree, and, worse, they fail on the same queries … the prompt simply carries little signal about which model will be the one that's right when the frontier disagrees."

Why the math fails: the co-failure ceiling

The core finding of the study centers on a metric called the "co-failure rate" — the formal name for the all-wrong scenario described above. No router, voting system, or cascade can ever achieve an accuracy higher than the ceiling it imposes.

The coding, logic, and generalist pool shows low pairwise correlation on routine prompts — they rarely fail together. But the co-failure ceiling represents the obscure, highly complex edge case that pushes past the limits of current AI architectures. If a prompt is so difficult that all three models hallucinate or fail, it does not matter how intelligently the router distributes the task. The entire pool wipes out at once.

The researchers tested their 67-model pool, which included GPT-5.5, Claude Opus 4.8, and Gemini 3.1 Pro, on the open-ended MATH-500 math benchmark. Based on standard pairwise correlation, statistical models predicted that the entire pool would wipe out simultaneously on only 2.3% of the questions. In reality, the co-failure rate was 5.2%.

Standard correlation metrics underestimated the failure rate by roughly 2.25 times. The culprit is not just independent difficulty, but a shared failure point.

"The driver is what we call a common-mode atom: a slice of queries on which the entire market fails together, which no pairwise statistic can see," Chen said. "Adding a 20th model to your pool doesn't buy tail coverage. The tail is shared."

The researchers also found that task format directly triggers co-failure. When they took graduate-level science questions from the GPQA benchmark and changed them from multiple-choice to free-response formats, the all-wrong tail expanded to 12.7%.

Developers can engineer around the ceiling, though. "The engineering implication is uncomfortable: multi-model setups buy the least exactly where teams want them most, on open-ended generation," Chen said. "Anywhere you can convert generation into verification or constrained selection (structured outputs, checkable answers, execution tests), you reopen the ceiling."

Ultimately, the researchers found this ceiling limits AI applications in two distinct ways, depending on the domain:

• Ceiling-bound environments (e.g., open-ended math): The co-failure rate is high. The task is too hard, and all models fail simultaneously. No amount of routing can bypass the lack of underlying capability.

• Realizability-bound environments (e.g., graduate-level science): The co-failure rate is near zero, meaning at least one model in the pool usually knows the answer. However, the models disagree so subtly that a routing layer cannot reliably pick the correct answer without an omniscient oracle.

The $0 pre-deployment sanity check

Before dedicating engineering hours to building a router, teams can calculate their absolute performance ceiling for free using a mathematical formula called a Clopper-Pearson bound.

The Clopper-Pearson bound operates as a worst-case scenario calculator. If you flip a coin ten times and get eight heads, you cannot guarantee the coin will land on heads 80% of the time forever. The bound takes a small sample of test questions and outputs a mathematically guaranteed ceiling.

Applied to language models, suppose a team tests a pool of five agents on 50 sample queries and finds they all fail together on just two questions. A developer might assume their multi-agent system will achieve 96% accuracy in production. The Clopper-Pearson formula corrects this optimism. It analyzes the small sample size and provides a mathematical guarantee that the true co-failure rate could actually be as high as 12%.

To use this in practice, enterprises must build a held-out dataset. A fintech company, for example, could take 200 complex customer support tickets from the previous quarter and have human agents write perfect resolutions to serve as a benchmark. While this sounds like a heavy manual project, mature engineering teams can automate the entire ceiling calculation.

"Integration is trivial: it's a counting job over eval logs teams already produce," Chen notes, "so it runs in the same CI stage as the eval suite and re-triggers whenever the model pool or the workload changes."

The engineering team then runs its candidate models against these 200 tickets once and records the results. When they want to evaluate multi-model configurations, they can use the co-failure rate measure to predict the maximum accuracy they can get from the system without running extra queries.

One important conclusion the study draws is that on tasks where answers can be definitively checked, combining models rarely beats using the single best model on the market, unless the team possesses an exceptionally strong query-level routing signal.

In an enterprise environment, a definitively checked task has an objective, zero-tolerance answer. This includes generating a SQL query that must execute without error, extracting a specific invoice total from a 50-page PDF, or formatting a JSON payload that perfectly matches a strict schema. For these tasks, enterprises are usually better off paying a premium for the smartest frontier model rather than weaving together three cheaper models and hoping a router picks the correct output. The study didn't test subjective, ungraded tasks like drafting marketing copy — the authors note that whether these findings hold outside their verifiable benchmarks remains an open question.

Because this mathematical check is free, enterprise teams can track their own co-failure rates as new models drop.

"The measurement costs nothing, so any team can track its own co-failure rate across model generations and watch whether the tail is closing," says Chen. Ultimately, "the lever buyers hold is failure-mode heterogeneity and market churn, not model count."

As enterprise AI systems scale to handle complex workflows, practitioners face the challenge of routing subtasks to the right tools and skills. Agents can have hundreds of tools and skills and get confused on which one to use for each step of a workflow.

To address this challenge, researchers at Alibaba developed SkillWeaver, a framework that creates an execution graph for a given task and chooses the right skills for each of the nodes. They also introduce Skill-Aware Decomposition (SAD), a novel technique that uses a feedback loop to enable the agent to fetch and vet relevant tool candidates iteratively. This compositional approach and feedback loop mechanism distinguishes SkillWeaver from other tool-routing frameworks that choose tools in a one-shot fashion. 

SkillWeaver relates to real-world AI applications where agents autonomously orchestrate multi-tool ecosystems, such as the Model Context Protocol (MCP), to execute multi-step business operations like downloading datasets, transforming information, and creating visual reports. 

In practice, the researchers' experiments with SkillWeaver show that implementing this retrieve-and-route approach significantly increases accuracy while reducing token consumption by over 99% compared to naively exposing agents to an entire tool library.

For practitioners building AI agents, the main takeaway is that the granularity of task decomposition is the biggest bottleneck to accurate tool retrieval. 

The challenge of skill routing

Skills are a key pattern in modern LLM agent architectures. A skill is a modular, reusable tool specification that uses structured natural language documentation. 

As enterprise agents integrate with massive tool ecosystems, accurately routing user queries to the right skills becomes a difficult task. Exposing an entire library to an LLM to find the right tool is highly inefficient, quickly overwhelms context limits, and consumes hundreds of thousands of tokens.

Most current tool-use frameworks attempt to solve this through API retrieval, documentation matching, or hierarchical structures that treat routing strictly as a single-skill selection or per-step problem. 

However, this single-skill paradigm is insufficient for enterprise environments because real-world queries are inherently compositional. A standard business request such as "Download the dataset, transform it, and create visual reports" cannot be fulfilled by one tool. It requires breaking the prompt down and sequencing an API client, a data processor, and a visualization tool into a cohesive, multi-step execution plan.

How SkillWeaver and SAD work

To tackle this, the researchers frame the problem of handling complex tasks that require multiple skills as "compositional skill routing." Given a complex user prompt and a vast library of tools, an agent must simultaneously figure out how to break the request into a sequence of atomic sub-tasks, how to map each sub-task to the single best available skill, and how to compose those skills into an executable plan.

SkillWeaver orchestrates this process through three distinct stages: Decompose, Retrieve, and Compose. In the first stage, an LLM acts as a task decomposer, breaking the user's complex query down into a sequence of sub-tasks that each require one skill. Once the sub-tasks are clearly defined, the system uses an embedding model to compare each subtask against the skill library to pull a shortlist of the top candidate tools for each step. 

In the final stage, a planner evaluates the retrieved candidates based on how well they work together. It checks for inter-skill compatibility to ensure the outputs of one tool naturally flow into the inputs of the next. It then creates a final execution plan as a Directed Acyclic Graph (DAG) that maps out dependencies so independent tasks can potentially execute in parallel.

For example, consider a user asking an AI agent to "Download the dataset, transform it, and create visual reports." In the decompose stage, the decomposer LLM breaks this into three distinct sub-tasks: downloading the dataset, transforming the data, and creating the reports. 

In the retrieve stage, the system searches the library and finds candidates like “api-client” or “http-fetch” for task one, “csv-parser” or “etl-pipeline” for task two, and so on. Finally, the compose stage evaluates these options, selects the specific combination of “api-client,” “csv-parser,” and “chart-gen” that are most compatible, and wires them together into a final, ready-to-execute workflow.

A key challenge of this pipeline is that LLMs often produce generic step descriptions that fail to match the specific, technical vocabulary of the actual skills available in the library. To fix this, SkillWeaver introduces Iterative Skill-Aware Decomposition (SAD), a novel feedback loop. SAD works by having the LLM draft an initial plan, conducting a preliminary search to find loosely matching skills, and then feeding those retrieved skills back into the LLM as hints. This allows the LLM to rewrite its decomposition so the granularity and vocabulary perfectly align with the actual tools that exist.

SkillWeaver in action

To evaluate how SkillWeaver performs in realistic enterprise scenarios, the researchers created a custom benchmark called CompSkillBench. It consists of 300 multi-step queries of different difficulty levels. To mirror real-world environments, they used a library of 2,209 real-world skills sourced from the public MCP ecosystem, covering 24 functional categories like cloud infrastructure, finance, and databases. 

For the core engine, the researchers primarily used a lightweight 7-billion parameter model (Qwen2.5-7B-Instruct) for task decomposition, paired with a standard semantic search retriever (MiniLM with a FAISS index) to find the tools. SkillWeaver was evaluated against three main setups: a brute-force "LLM-Direct" method where they stuffed all the tool names into the prompt of a large model, a vanilla LLM-based decomposition without SAD, and a ReAct-style agent loop.

The experiments indicate that task decomposition is the main bottleneck. Standard LLM behavior falls short when dealing with large tool libraries, but the SAD feedback loop dramatically moves the needle. In the vanilla setup, the 7B model achieved a decomposition accuracy (i.e., predicting the correct number of steps) only 51.0% of the time. By activating the SAD feedback loop, accuracy jumped to 67.7% (with the larger Qwen-Max model, the accuracy reached 92%). On "hard" tasks requiring four to five distinct skills, SAD improved accuracy by 50%.

One fascinating finding was that larger models can actually perform worse when unguided. When tested in the vanilla setup, a larger 14-billion parameter model saw its accuracy plummet below the 7B model's accuracy because it tended to over-decompose tasks into microscopic, unnecessary steps. Once SAD was introduced, the retrieved tool hints anchored the model back to reality and increased its accuracy. This suggests that aligning an agent with the vocabulary of specific tools is often more impactful than paying for a larger, more expensive LLM.

Another important takeaway is token savings. The LLM-Direct baseline, which used the very large Qwen-Max model, showed that feeding all tools into the prompt of a large model fails. Despite near-perfect task breakdown capabilities, the massive model only retrieved the right tool category 21.1% of the time when flooded with tool options. SkillWeaver's targeted retrieve-and-route approach vastly outperformed this in accuracy while slashing context window consumption from an estimated 884,000 tokens down to roughly 1,160 tokens per query, a 99.9% reduction. For practitioners, this translates directly to drastically lower API costs and faster response times. 

Finally, the traditional ReAct baseline completely failed, achieving 0% decomposition accuracy. Its loop naturally collapses multi-step plans into isolated actions rather than explicitly mapping out a cohesive, multi-tool sequence.

Considerations for developers

While the researchers have not yet released the source code for SkillWeaver, their work was built on off-the-shelf tools that can easily be reproduced. 

Skill-Aware Decomposition (SAD), which is the key innovation at the heart of the framework, is a clever prompt-engineering and retrieval loop. The authors have shared the prompt templates in their paper, and developers can implement it themselves quite easily using standard orchestration libraries like LangChain, LlamaIndex, or even raw Python scripts.

As for the retrieval component, the authors built the core framework using all-MiniLM-L6-v2, an open-source embedding model. They found that swapping in a slightly stronger off-the-shelf encoder (BGE-base-en-v1.5) immediately boosted accuracy without any fine-tuning. While an off-the-shelf bi-encoder is great at getting a relevant tool into the top 10 candidates nearly 70% of the time, it struggles to consistently rank the perfect tool at exactly number one, achieving that only about 37% of the time. To bridge this gap, teams will likely need to implement a secondary cross-encoder or LLM-based reranker to re-order those top 10 candidates.

One upfront preparation requirement is vectorizing the tool library and building a FAISS index in advance. In practice, this is a negligible hurdle. Embedding and indexing all 2,209 skills in the benchmark took a mere 15 seconds. Once built, retrieving tools from the index adds less than 15 milliseconds of latency per query. For enterprise environments, syncing the tool index is a trivial background job. 

A current limitation in SkillWeaver is the lack of error recovery. While SkillWeaver successfully maps out a compatible DAG for execution, the authors' pilot study revealed the challenges of multi-step tool chains. For example, if an API call fails in step two, the entire chain breaks. The paper's core contribution is limited to the routing and planning phase. For a true production deployment, practitioners must build their own error recovery, fallback, and retry mechanisms on top of the compose stage to handle real-world API timeouts or malformed outputs.