Nvidia and Palantir announced Thursday that they’re working together to bring “sovereign AI to critical supply chains,” kicking off initially with Nvidia’s own sprawling supply chain.

The news builds on a partnership that kicked off last October, when the duo said they would combine Nvidia’s AI computing and models with Palantir’s software to help companies use AI to make complex operational decisions. Then in June, they expanded that effort into sovereign AI, allowing organizations to run and customize Nvidia’s AI models inside tightly controlled environments while keeping sensitive data and model weights under their own control.

Now, they’re applying that technology inside Nvidia itself, where they say a smaller, fine-tuned model is already outperforming a far larger one.

A proving ground for sovereign AI

The companies have fine-tuned Nvidia’s 30-billion-parameter Nemotron 3.5 Lightning model on decisions made by Nvidia’s supply-chain operations team. Palantir’s Foundry and Artificial Intelligence Platform (AIP) bring together the data behind those decisions, while its Ontology acts as a live map connecting components, factories, capacity and production commitments. Nvidia’s cuOpt software, meanwhile, works out how to distribute scarce parts, with Nemotron weighing the wider context and recommending what planners should do.

They then plan to “extend the learnings from Nvidia’s deployment” to companies in other sectors, including manufacturing, energy, healthcare, automotive and aerospace. Palantir’s own customers will be able to build versions tailored to their own supply chains by training Nemotron on their proprietary data using Foundry and AIP, then run the resulting system on-premises or through cloud and colocation providers.

So, in effect, Nvidia and Palantir are putting the sovereign AI partnership they outlined in June into practice inside Nvidia, while using that deployment as a proving ground for an architecture other companies can adapt to their own use cases.

Nvidia as a test case

As the world’s most valuable public company at a $5.4 trillion market cap, there’s good reason for Nvidia to start close to home. Its supply chain spans millions of parts, thousands of suppliers, and a global network of manufacturing partners, with the company saying a single Vera Rubin rack alone contains some 1.3 million parts. Those components have to arrive in the right place at the right time: if one part is missing, assembly can stall while everything else that arrived sits waiting.

And that complexity is what Nvidia founder and CEO Jensen Huang says makes supply chains a natural target for the technology. From chips and memory to manufacturing, networking, power and cooling, he argues that building modern AI systems increasingly depends on coordinating an enormous web of companies and components.

“Supply chains are the operating system of the physical economy, and AI factories are among the most complex systems ever built,” Huang says in a statement.

Palantir co-founder and CEO Alex Karp goes further, arguing that Nvidia’s operations provide an unusually demanding environment in which to put the companies’ approach to the test.

“Nvidia has arguably the most valuable, intricate, and complex supply chain in the world.”

“Nvidia has arguably the most valuable, intricate, and complex supply chain in the world,” Karp adds in a separate statement.

The sovereignty selling point

Nvidia has long been positioning itself at the center of the open-model debate. In July, Huang even used his first-ever post on X to promote an industry letter lobbying Washington to support frontier open-weight models, arguing that they give companies and countries more control over their AI infrastructure.

Then in early September, Nvidia swooped in with a $12.9 billion deal for Hugging Face, the so-called “GitHub for AI models.” Amid concerns that ownership by the world’s dominant AI chipmaker could undermine Hugging Face’s neutrality, Huang pledged that it would remain open, continue hosting models from across the industry and support hardware beyond Nvidia’s own.

Nemotron is central to Nvidia’s own open-model push. The name dates back to 2023, when Nvidia released its first Nemotron-3 8B models for enterprises to customize and fine-tune. Those early models were downloadable through Hugging Face and Nvidia’s NGC catalog, although access was gated and governed by Nvidia’s own community license. So they were customizable, and their weights were available, but the much broader “open model” positioning Nvidia uses today came later.

The current Nemotron 3 series arrived back in December, initially spanning Nano, Super and Ultra models aimed at different agentic AI jobs. Nvidia now publishes weights and, for many of the models, training data and recipes so developers can customize themselves. Nemotron 3.5 Lightning, released in August, is the 30B model Nvidia and Palantir have fine-tuned for this supply-chain deployment.

That openness is also at the heart of the whole sovereignty pitch: companies can adapt Nemotron using proprietary data while keeping that data, the model weights, and inference inside their own environment.

Specialization over size

Nvidia’s own deployment gives outsiders a result to chew on. It says the fine-tuned 30B Lightning scored 86.7% accuracy on its supply-allocation task, versus 55.5% for the 550B Nemotron 3 Ultra—a model roughly 18 times its size.

In a technical blog post published on Thursday alongside the main announcement, Nvidia solutions architects Nell Barber, Rana Haber, and Aastha Jhunjhunwala note that the result shows how far specialization can go. On a tightly defined allocation task, the 30B model outperformed a general-purpose model more than an order of magnitude larger.

“This doesn’t mean the smaller model is more capable overall. Its gains are concentrated in the domain it was post-trained on.”

“This doesn’t mean the smaller model is more capable overall,” they add. “Its gains are concentrated in the domain it was post-trained on. Future production risk forecasting remained difficult despite fine-tuning. Specialization improved the decision task but failed to solve every prediction problem attached to it.

For companies considering Nvidia’s blueprint, the more interesting takeaway may be this: a smaller open model, trained on business specifics, can sometimes be more useful than reaching for the biggest model available.

The post Nvidia and Palantir fine-tune a 30B Nemotron model for Nvidia’s supply chain. It beats a model 18 times its size. appeared first on The New Stack.

Hundreds of thousands of AI agent sandboxes can already run concurrently on a single DeepSeek cluster. Now the company is staffing up to handle what happens as that number — along with its training, evaluation, and other backend workloads — keeps climbing.

Cui Tianyi, who joined DeepSeek in March and works on its Harness team, the group responsible for the infrastructure and environments used to run and evaluate agents, announced in an X post that roughly 150 engineering positions on September 7, with the hiring concentrated in server-side engineering and Agent Elastic Compute rather than AI research. The work spans operating systems, virtualization, networking, storage, scheduling, and the control-plane services that coordinate those resources.

Cui said DeepSeek’s existing backend systems will need upgrades, maintenance, and rewrites as workloads grow. One such system at the center of that scaling challenge is DeepSeek Elastic Compute, or DSec, the sandbox infrastructure DeepSeek built to execute agent workloads during post-training and evaluation.

Cui said DeepSeek’s existing backend systems will need upgrades, maintenance, and rewrites as workloads grow.

Four sandboxes, one SDK

Agent workloads require more than GPUs for inference, with each agent also needing an isolated environment to run code, call tools, change files, and collect the results.

DSec supports four types of those environments through the same Python SDK. Simple function calls go to pre-warmed containers, while Docker-compatible containers handle jobs that need a persistent environment. DeepSeek uses Firecracker microVMs when stronger isolation is needed and QEMU virtual machines for workloads that require a full guest operating system.

That range means the same infrastructure can handle anything from a simple tool call to a software-engineering task that needs an entire OS. It’s a similar challenge to the one the rest of the industry is bumping into as agents move from demos to production. OpenAI, for instance, recently designed custom silicon specifically to address the compute pressure that agent workloads create, and DeepSeek open sourced its own agent harness in August.

Lazy loading agent environments

Every sandbox needs its own environment, but copying complete container or VM images onto every host would consume enormous amounts of storage and network bandwidth while adding to startup time. DeepSeek gets around that by tying DSec into 3FS, the distributed filesystem it originally built for its AI infrastructure, and keeping container base images and filesystem commits as read-only layers backed by 3FS.

The metadata stays local, but the underlying data blocks are fetched only when they’re actually needed. MicroVMs use a similar setup, sharing their read-only base layer through 3FS while writes from individual sandboxes are kept in local copy-on-write layers.

DeepSeek says DSec reduces duplicate page-cache usage across virtualized environments and reclaims memory to allow safe overcommitment, while changes to the container runtime cut the CPU overhead of each sandbox.

The team also had to deal with spinlock contention inside the container runtime. At small scale, the CPU time spent there barely registers. At scale, it limits how densely those environments can be packed onto each host.

DeepSeek says DSec reduces duplicate page-cache usage across virtualized environments and reclaims memory to allow safe overcommitment, while changes to the container runtime cut the CPU overhead of each sandbox.

When replay breaks training

During reinforcement learning and other post-training workloads, large numbers of agent rollouts can be running at once, and jobs may be interrupted as compute gets reassigned. Starting over wastes everything the agent has already done, but picking up where it left off isn’t as simple as replaying its previous commands.

Some of those commands may have changed a file or otherwise altered the environment, so running them again could produce a different result or leave the training trajectory in the wrong state. DSec avoids that with a globally ordered trajectory log that records commands along with their results.

When a rollout resumes, DSec can fast-forward through the completed work using those recorded results rather than executing the commands a second time. That reduces the cost of interruptions across thousands of training and evaluation runs, while the same logs preserve a history of how each sandbox changed and allow earlier sessions to be replayed.

Engineers, not researchers, wanted

The roughly 150 openings reach across DeepSeek’s backend, including the lower-level systems work behind Agent Elastic Compute as well as the services that support its models and agents.

DeepSeek said in June that it planned to at least double the size of every department, but this round of hiring leans heavily toward the systems underneath its models rather than the models themselves. DSec is part of that work, with hundreds of thousands of sandboxes running concurrently and putting pressure on everything from how jobs are scheduled to how they recover after an interruption.

The roughly 150 openings reach across DeepSeek’s backend, including the lower-level systems work behind Agent Elastic Compute as well as the services that support its models and agents.

The post DeepSeek is hiring 150 engineers, and none of them will touch a model appeared first on The New Stack.

OpenAI says it hit a goal it set last fall, stating researchers are now using what the company calls an “automated research intern,” which is an agent that can handle well-defined tasks that would normally take a researcher several days.

The data shows coding-agent use climbing throughout 2026, and by mid-August its agents were logging 3.1 agent-workdays for every human workday across the research organization. The median researcher was spending more than $600 on inference per day at API prices, while those in the 90th percentile spent more than $7,000.

The median researcher was spending more than $600 on inference per day at API prices, while those in the 90th percentile spent more than $7,000.

Agent hours versus useful output

But everyone knows that an agent-workday and a human workday aren’t the same. The company converts the time agents spend working on tasks into standard eight-hour workdays. Because researchers can run several agents at once, the figure tells us how long the agents are working, but not necessarily what they’re completing.

For engineering teams, that leaves plenty of work on the human side, which means running more agents can increase the amount of work happening at once, but it can also increase the amount of work a human needs to keep track of.

OpenAI’s very specific definition of a research intern highlights that it must be able to complete well-defined research tasks that would take a skilled person several days, but a human is still in charge. The company’s next goal, an automated AI researcher, is one they hope to reach by March 2028.

The company’s next goal, an automated AI researcher, is one they hope to reach by March 2028.

Supervision becomes the constraint

Using a taxonomy from Epoch AI, OpenAI broke the agents’ work into six areas — Decide, Design, Build, Run, Analyze, and Communicate — and found activity increased across all six between January and August, although agents still did relatively little of the work involved in deciding what research to pursue.

Much of the work is practical, with agents writing research and infrastructure code, monitoring experiments, and providing enough technical support that OpenAI says attendance at debugging office hours has fallen, prompting one team to stop holding the sessions altogether.

And yet, more agent hours don’t automatically mean more useful research. OpenAI says code output and experiment counts are relatively easy to track, but neither shows how much progress those agents actually made. Compute also increased significantly as the number of experiments rose.

OpenAI used another model to judge how well agents performed on tasks of varying difficulty and found that, despite improving success rates between January and July, humans still had to step in on more than half of successful tasks that would have taken a person four to eight hours.

Security incidents limit Astra deployment

Once engineers can run several agents at once, with those agents launching subagents of their own, the challenge shifts to keeping up with what they produce — catching runs that go off track, reviewing code diffs, and deciding what is ready to ship or feed into a training run.

⁠Astra’s persistent-agent capabilities already let researchers hand off multi-day assignments⁠, which makes this supervisory strain worse, not better.

The company acknowledges that as agents take over more of the execution, the parts of research that are hardest to automate will consume more of an engineer’s time, putting a practical limit on how much agent output one person can realistically review.

On July 20, a series of outages caused by agents disrupted OpenAI’s research infrastructure badly enough that the company took its training container service offline and later brought it back with tighter restrictions.

Nearly a month later on August 7, OpenAI tightened access again after early evidence suggested Astra could reach the “Critical” cybersecurity threshold in its Preparedness Framework, restricting the model to higher-security research areas and adding safeguards that developers may already be encountering as unexpected API interruptions⁠

Workloads shift between models fast

Astra-class GPU allocation fell 59.2% the following week, but that compute didn’t sit idle for long. Researchers moved much of the work to other models, which saw GPU allocation rise 17.2% and made up for roughly 85% of the drop in Astra usage. Instead of reducing the amount of work being run, the restrictions pushed it to other models, showing how easily workloads can move when one part of the system is locked down.

OpenAI’s researchers are handing off larger jobs to agents, running more of them at once and launching more experiments, but whether that translates into faster research is harder to measure — and OpenAI is still figuring out how to price it.⁠

OpenAI’s researchers are handing off larger jobs to agents, running more of them at once and launching more experiments, but whether that translates into faster research is harder to measure — and OpenAI is still figuring out how to price it.⁠

The post AI agents are creating more work, not less — and OpenAI’s own numbers back it up appeared first on The New Stack.

When enterprise AI applications scale, they inevitably hit a wall. For many engineering teams, this wall is initially diagnosed as a billing issue, a monthly API invoice that has grown out of control. However, viewing token consumption purely as a financial metric fundamentally misunderstands how LLMs operate in production. Token optimization, unlike its other optimization cousins, is not an accounting exercise; it’s a distributed systems and hardware utilization challenge.

“Token optimization, unlike its other optimization cousins, is not an accounting exercise; it’s a distributed systems and hardware utilization challenge.”

In this guide, we explore, through the lens of Concierge (a latency-sensitive, synchronous customer support agent) and Pathfinder (an asynchronous, multi-step autonomous CI debugging agent), how these systems fell victim to autoregressive bottlenecks as they grew, and how we fixed these issues.

What you’re actually paying for

A token is not a word: treating it like one will break your budgeting “models.” Every major LLM provider tokenizes text using byte-pair encoding (BPE), breaking words into subword units. While common words stay intact, rarer words or punctuation split into fragments. As a rule of thumb, 1 token = 4 characters, or 0.75 words in standard English prose.

When budgeting for production, you must account for the structural pricing spread; providers bill input tokens and output tokens at different rates. Output tokens are typically 4-5X more expensive than input tokens.

As a baseline, assume a mid-tier frontier model runs roughly $3 per million input tokens and $15 per million output tokens.

The quadratic history tax

LLM provider APIs are completely stateless; to make an LLM behave as if it remembers past events, you must resend the entire history of the session and input with every single API call.

This means that a model’s own previous outputs are continuously re-billed to you as inputs on subsequent steps. This triggers a compounding cost that impacts both Concierge and Pathfinder, though their curves scale differently.

Let S be the static system context (instructions and schemas), u be the incoming data per step, and r be the model’s response payload. The input cost for every given turn k is

When you sum this across a complete execution run of N steps, the total input token volume compounds quadratically. 

This O(N^2) accumulation of history is the exact mechanism that causes the explosion in cost and latency.

When we calculate the total input tokens consumed by a single session using the quadratic formula

• Concierge: Consumed 45,300 tokens per 10-turn ticket
• Pathfinder: Consumed 150,000 tokens per 15-step turn

Because Pathfinder’s step increment was 4X larger than Concierge, its token cost curve was drastically steeper. If Pathfinder were to get stuck in an infinite tool-use loop and hit 30 steps, a single run could consume 570,000 tokens.

The solution

Fixing the individual call

Prompt hygiene: Hardcoding static reference documentation into the system prompt means you pay to parse identical text on every turn. So we stripped static text from the prompt and switched to dynamic injection. 

For Concierge, we implemented a RAG step to fetch only the 2-3 policy snippets relevant to the ticket. The prompt dropped from 3,100 tokens to 380. A 60% reduction for a 10-turn thread.

For Pathfinder, we applied automated prompt compression using LLMLingua-2 to compress verbose CI log files before sending them to the model. By filtering out non-essential log lines, we reduced the size of incoming tool observations by 3X without sacrificing debugging accuracy.

from llmlingua import PromptCompressor

compressor = PromptCompressor(
model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
use_llmlingua2=True
)

try:
compressed_result = compressor.compress_prompt(
raw_ci_log_text,
rate=0.33,
force_tokens=["Error", "Exception", "Failed", "Traceback", "FATAL"]
)
# Pass high-density payload to the frontier model
compact_prompt = compressed_result["compressed_prompt"]
except Exception as e:
print(f"Compression failed, falling back to raw log text: {e}")
# Graceful degradation: pass the raw (or truncated) log if compression fails
compact_prompt = raw_ci_log_text

Eliminating the retry: Relying on open-ended prose instructions “return JSON” caused malformation. When parsing failed, the system initiated a synchronous retry, sending the entire accumulated context as if it were a new attempt. We replaced the entire natural language formatting request with strict structural contracts via forced schema validation.

In both Concierge and Pathfinder, we converted the output format to a strict pydantic schema for tool-calling mode and tool-execution payloads. Malformed outputs across both systems dropped to under 0.5%, eliminating tail latency spikes caused by cascading queues.

# Unified Schema Enforcement for Concierge Responses & Pathfinder Tool Execution
from pydantic import BaseModel
from typing import Literal

class TicketResponse(BaseModel):
    reply: str
    category: Literal["shipping", "returns", "billing", "product", "other"]
    escalate: bool
    confidence: float

# The API is structurally locked into emitting validated JSON matching the schema
response = client.messages.create(
    model="claude-opus-4",
    system=SYSTEM_PROMPT,
    messages=messages,
    tools=[
    {
    "name": "respond_to_ticket",
    "description": "Formulate a response and classify the support ticket.",
    "input_schema": TicketResponse.model_json_schema()
    }
    ],
    tool_choice={"type": "tool", "name": "respond_to_ticket"},
)

Output token bounding: Models naturally generate verbose reasoning chains and conversational filler, inflating expensive output tokens. Where the LLM provider exposes logit bias, you can directly suppress every token outside the valid set at decode time; where it doesn’t, constrained decoding libraries (Outlines, Guidance) or a forced tool call with an enum-typed schema will get you the same guarantee.

class ClassifyOnly(BaseModel):
    category: Literal["shipping", "returns", "billing", "product", "other"]
    priority: Literal["low", "medium", "high", "urgent"]

State management 

The stateless nature of the models meant we had to parse the static prompt prefix and historical steps on every turn. We introduced explicit cache breakpoints to allow the inference engine to reuse the states of static blocks. We altered both Concierge and Pathfinder to flag stable, historical segments for caching. Under standard vendor pricing, cache reads are discounted by 90%. It is important to check with your vendor on whether caching is enabled. 

# Caching the stable history prefix for a multi-turn session
response = client.messages.create(
    model="claude-sonnet-4",
    max_tokens=4096,
    system=[{
        "type": "text",
        "text": SYSTEM_PROMPT,
        "cache_control": {"type": "ephemeral"} # Cache hits drop prefix costs by 90%
    }],
    tools=TOOL_SCHEMAS,
    messages=session_history + [{"role": "user", "content": current_step_input}],
)

For a 10-turn Concierge chat, this dropped input costs by ~70%. For a 15-step Pathfinder trajectory, it resulted in a 76% cost reduction.

Semantic caching 

Duplicate queries across separate sessions were triggering redundant frontier model invocations. We implemented a vector similarity cache layer upstream of the LLM using Redis. Our Concierge service analysis showed that 34% of customer support tickets were semantic duplicates of common FAQs.

Intercepting these requests reduced latency to sub-50ms for hits. Because of the nature of CI pipeline logs, we have not yet found a suitable cache for Pathfinder’s inputs. 

import os
import json
import redis
from redis.commands.search.query import Query

# Configure connection via environment variable for environment portability
redis_url = os.environ.get("REDIS_URL", "redis://localhost:6379")
r = redis.Redis.from_url(redis_url)

def get_cached_response(tenant_id, query_text, threshold=0.92):
    try:
results = r.ft(f"cache_idx:{tenant_id}").search( # scoped by tenant -- see below
Query("*=>[KNN 1 @vector $vec AS score]").sort_by("score").dialect(2),
query_params={"vec": query_vec.tobytes()},
)
except redis.RedisError as e:
print(f"Redis cache error: {e}")
return None # Fail-open: gracefully fall back to a cache miss)
  query_vec = embed(query_text) # small, fast bi-encoder -- not the frontier model

try:
results = r.ft(f"cache_idx:{tenant_id}").search( # scoped by tenant -- see below
Query("*=>[KNN 1 @vector $vec AS score]").sort_by("score").dialect(2),
query_params={"vec": query_vec.tobytes()},
)
except redis.RedisError as e:
print(f"Redis cache error: {e}")
return None # Fail-open: gracefully fall back to a cache miss

Semantic caching could be a security problem, because if you choose a global cache, Customer A’s account-specific answer could get served to Customer B because their phrasing embeddings are close enough. To mitigate this, we split the cache into two tiers: a global cache for tenant-agnostic content, and a per-tenant, per-user namespace keyed with the tenant ID baked into the prefix itself for anything touching account state.

Cache poisoning is another risk: we only write to the cache from responses that passed schema validation and the injection-pattern classifier, we stamp every cache entry with its source traceId, and we encourage routine purging of unknown caches.

Context compaction

Uncapped conversation or agent trajectories allowed N to grow continuously, expanding the cost curve and causing latency degradation. We capped N by implementing a sliding window that summarizes historical context via a small, ultra-cheap model. For Concierge, we kept the last 3 turns verbatim while condensing older turns into a rolling metadata block.

For Pathfinder, when the debugging steps exceeded 4 runs, we trimmed and summarized the oldest tool execution outputs into a compact chronological timeline, transforming the open-ended quadratic cost explosion into a predictable, bounded window.

def compact_session_history(history_steps: List[Dict[str, Any]], keep_recent: int = 3) ->     List[Dict[str, Any]];
    """Flattens older history into a cheap summary block, preserving recent context."""
    if len(history_steps) <= keep_recent:
        return history_steps
    old_steps = history_steps[:-keep_recent]
    recent_steps = history_steps[-keep_recent:]
    
# Compress the old history using a fast, low-cost utility model
try:
historical_summary = summarize_with_utility_model(old_steps)
# Note: Anthropic prohibits 'system' roles in the messages array.
# Using 'assistant' ensures cross-provider compatibility.
return [{"role": "assistant", "content": f"[System Context: Summary of prior steps: {historical_summary}]"}] + recent_steps
except Exception as e:
print(f"History compression failed: {e}")
# Fallback: Return the uncompressed history to gracefully degrade
return history_steps

Model cascading

Directing every single operation to an expensive frontier model represents massive overprovisioning for mundane tasks. We integrated LiteLLM as an internal routing gateway to implement model cascading, routing every request to the lowest-cost model capable of completing the task.

“Directing every single operation to an expensive frontier model represents massive overprovisioning for mundane tasks.”

# litellm_config.yaml
model_list:
  - model_name: fast-path
    litellm_params:
      model: openai/mistral-support-ft
      api_base: http://vllm-internal:8000/v1
  - model_name: frontier-path
    litellm_params:
      model: anthropic/claude-opus-4

Simple, repetitive tasks are routed to a lower model, which offloads 70% of Concierge chats from the frontier model. For Pathfinder, we broke the agent loop down into separate sub-tasks: high-level planning, tool selection, and code-patch synthesis remained with the frontier model, while mechanical, text-heavy operations, such as log parsing, regex extraction, and error-string formatting, were offloaded to the lower models. This hybrid orchestration reduced Pathfinder’s token costs by more than 50%.

What’s next

The transformation of Concierge and Pathfinder proves a fundamental truth about production AI. You cannot achieve scale by simply relying on the natural language capabilities of a frontier model. You must engineer the system around it. By shifting your focus from naive token reduction to maximizing system resource utilization, we reclaimed absolute control over the infrastructure.

“Efficiency in the era of gen AI is not defined by how cheaply you can operate but by how densely you can pack information.”

Efficiency in the era of gen AI is not defined by how cheaply you can operate but by how densely you can pack information, how quickly you can serve it, and how reliably you can parse the output. The architectural decisions detailed here represent more than just a token optimization strategy; they are a required foundation for building high-throughput, battle-tested, and resilient AI systems at scale.

The post The systems guide to production token optimization appeared first on The New Stack.

Thanks to increasingly capable models, some parts of your business are getting faster, more capable, and more productive every month. These teams are using artificial intelligence to compress timelines, surface insights, and automate work that has historically been time-consuming and tedious.

Meanwhile, other functions just down the hall are still waiting for a formal rollout, a governance approval, or someone to tell them what to do and how to start. The gap between the AI haves and have-nots in your organization is widening, and addressing it requires a new operating model.

Your teams need more support

When leaders notice the uneven distribution of capability across their business, the instinct is to treat it as a tooling problem. They push to get everyone access to the same platforms, provide general-use training, and hire some specialists to slot into IT.

But access is table stakes. It’s a good start, but it won’t get you to strong organizational adoption.

Harvard Business School reports that workers using these tools completed tasks 25% faster and produced results rated more than 40% higher in quality. But the same study also found that performance declined when people used the tools without understanding where they applied and where they didn’t. Fluency, not just access, drives results.

Departmental leaders need guidance on how to apply capabilities in the context of their day-to-day work. Without that knowledge, they can’t ask the right questions.

“Performance declined when people used the tools without understanding where they applied and where they didn’t. Fluency, not just access, drives results.”

Teams playing catch-up tend to focus on how to inject new tools into existing workflows, when they should be thinking about re-engineering processes entirely. They’re focused on evolution in a world undergoing revolution.

Reimagining a process also requires stepping back from it, which is easier said than done. Here’s how it plays out in practice:

An SDR team comes to IT with a specific, bounded ask: “improve our sales lead routing.” Completely reasonable. But only when someone from IT, with visibility across the broader system, dives into the problem does the real opportunity surface. The data pipeline supporting lead routing is unnecessarily complex. With the right support, the conversation shifts to overhauling the entire pipeline and opens the door to fully agentic lead follow-ups.

Departmental leaders don’t lack ambition but throwing a software license and Slack channel at them won’t build the right kind of adoption. Technical support and strategic guidance are required to reimagine work from first principles.

AI fluency must be a structural consideration

The typical pattern puts a centralized team in charge of taking requirements, interpreting them in isolation, and delivering capabilities to departments months later. This model can’t keep pace when AI capabilities launch weekly.

A more effective approach pairs a central “hub” that owns platform strategy, governance, and reusable patterns with AI engineers embedded directly inside business departments. AI engineers serve as “spokes” inside departments, helping them identify vertical use cases day-to-day and delivering the cross-functional visibility needed to make a real impact. The AI engineer who solved a problem for finance can share the pattern with someone facing the same challenge in operations.

“A more effective approach pairs a central “hub” with AI engineers embedded directly inside business departments.”

In a department just getting started, the embedded AI engineer is the primary technical capability: scouting, prototyping, building. In a more mature department, they shift toward enablement, feeding patterns back to the “hub” and helping teams navigate AI without getting buried in process. Over time, departments will organically become AI-fluent as they learn from the engineers.

Make fluency your advantage

The right operating model drives how a function actually works, and strong fluency strengthens processes and institutional knowledge, so outcomes improve over time. As the flywheel builds, each problem solved raises the ceiling of what your team can do independently. 

McKinsey finds that the right workflow redesign is the single biggest factor in whether an enterprise sees meaningful bottom-line impact. Knowing what to redesign depends on how your teams understand and work with AI.

Everyone is adopting AI capabilities. The question now is whether your operating model helps your teams see the best path forward for applying them. If it doesn’t, that’s the gap to close first.

The post Your organization prioritized AI adoption, but you actually need AI fluency. appeared first on The New Stack.

Real-time AI at scale is harder than it looks. Pipelines that hum along in development routinely hit problems in production. It’s always easy to blame the model for all your problems. But issues like rising latency and degrading accuracy can usually be traced back to the data pipeline. 

My colleague Tim Koopmans and I recently discussed what typically goes wrong with real-time AI at scale. After Tim shared some hard-fought lessons learned, we talked about how to avoid falling into these traps yourself – including the practices and infrastructure choices that can help you avoid them. You can watch the full video or read the key points below. 

Why AI performance fails at scale

Tim learned the following real-time AI performance lessons the hard way: through fits of frustration while building an ML-based financial trading app.

You can’t dig yourself out of tail latency

All too often, latency looks fine in testing, then a P99 spike surfaces under real concurrent load. For example, as Tim’s app approached ~740K operations per second, its P99 latency skyrocketed to 3 seconds. 

“I kept blaming the model for being slow, but it turns out the model was fine,” Tim explained. “It was just that the feature lookups were killing me.” Each inference call was doing just a handful of reads, but those reads queued up [behind writes] under load. The average latencies seemed fine, but that P99 tail latency was just unacceptable. 

“Tail latency isn’t a bug that you can fix, it’s a property of your architecture.”

Once you hit highly concurrent write throughput, you get lock contention – and that impacts the tail latencies. At this point, retries and bigger caches and connection pool tuning don’t help. As Tim put it, “Tail latency isn’t a bug that you can fix, it’s a property of your architecture. For example, if your storage engine is producing GC pauses at exactly the wrong moment, you’re going to cop a latency spike, no matter what.”

The culprit in Tim’s app was actually Postgres under pressure: “It’s not a slow database, but it was just a database being asked to do too much in this particular case,” Tim continued. 

Stale features kill accuracy

If you notice a mysterious accuracy drop that the model itself can’t explain, feature freshness might be the problem.

For Tim, this issue was particularly frustrating. User profile (wallet addresses) staleness was blowing past a five-minute SLA target by hours, vector embeddings were going stale, and offline evaluation metrics looked fine the entire time. As Tim put it, “You have this maddening situation where offline evaluation metrics look great, but as soon as you mix it in with online data, that performance is rubbish.”

“Offline evaluation metrics look great, but as soon as you mix it in with online data, that performance is rubbish.”

When the model was in production, it started making calls that didn’t track. After spending what seemed like ages debugging the model, the model itself turned out to be fine. The problem was that the model was making decisions based on old data (garbage in, garbage out, essentially). 

Vectors indexes need maintenance

No matter what vector database vendors imply, “set it and forget it” isn’t a realistic strategy for embeddings. Every re-embedding pass rots the index a little more, whether you notice it happening or not.

Tim hit this too. He was re-embedding content every time he improved the model, and the index quality rotted a bit more with every pass. At one point, he noticed that the recall rate (the share of true best-matches an approximate search actually finds) dropped to a dismal 42% – and query latency ballooned at the same time. He explained, “HNSW graphs degrade as they take on mutations. The nasty thing is you don’t really realize that until you realize your results are tainted.”

He advised others to treat a vector index like you’d treat any other database index. It needs the same care, love and attention as anything else you operate. That means:

• Monitor recall accuracy (and results returned) 
• Plan for partial builds (or batch builds)
• Know that changing your similarity function, your search parameters, or your embedding model means starting the graph over from scratch.

You gotta keep ’em separated

Another problem is resource contention – for example, training and serving fighting over the same hardware. Tim had just one machine doing double duty. With everything running on the same infrastructure, GPU, RAM, and CPU were all competing for resources. Side note: Many people don’t realize that vector search is a CPU cost, not a memory cost, since you’re traversing a graph rather than just storing vectors. 

The fix is the same thing every distributed systems person already knows: You gotta keep ‘em separated. This is just good engineering principles: separate your write path from your read path, separate training from serving if you can afford it.”

Retraining is inevitable

Recognize that retraining isn’t optional, and it isn’t free. Every model swap requires transition time.

Tim explained that there’s a dodgy window where the old model is still serving stale predictions and the new one hasn’t warmed up yet. For the database, this could mean new access patterns, cache misses, cold reads or request queues building up. When you notice that data is drifting, or user behavior is changing, the model you trained three months ago is getting worse – that’s the sign that it’s time to retrain. 

It’s going to happen eventually, so plan for it. Tim’s own approach was blue-green deployments, canaries, running the old and new model in parallel under different names, and doing the actual cutover at the application layer rather than all at once. If you’re at, say, Tripadvisor scale – with 100 million ML models – you can imagine the process will be considerably more complex.

Avoiding the doom loop with a high-performance database

These problems tend to build on each other and snowball. Latency causes staleness, staleness degrades accuracy, degraded accuracy triggers retraining, retraining causes contention, and contention makes latency worse again. It can create what Tim deemed a “doom loop.”

Here are some tips for avoiding that doom loop. 

Monitor, monitor, monitor

Be obsessive about monitoring. Watch freshness and backlog in particular because a growing backlog is what eventually drives up tail latency. Also watch index health, since that’s where recall rots. And load test beyond steady state because you can’t really predict when some weird confluence of factors will cause usage to surge. 

Isolate your workloads

This addresses two of the problems from earlier: the write storms that caused tail latency, and training and serving sharing the same infrastructure.  A database that handles concurrent writes well and isolates workloads properly can absorb both.

For example, with ScyllaDB, the write path is lock-free and multi-writer. That means every node takes writes in an active-active fashion, and no row gets locked in the process. As a result, a burst of concurrent writes doesn’t back up into a queue the way it would on a database built around single-writer assumptions. 

On top of that, a practice we call “workload prioritization” controls how workloads compete for system resources. This ensures latency-sensitive queries are fast, even with other heavy workloads running on the same cluster. That way, a retraining job or a backfill won’t steal resources from whatever’s serving live inference. 

Separate vector indexing

To address the vector index problem, keep the index separate instead of bolting it onto the same process as the core database. For example, ScyllaDB Vector Search writes land in the core database first, and the index gets built out of that data asynchronously, as its own service. 

If the index can’t keep up with the write rate (whether from a re-embedding pass or a full rebuild after a similarity function change), it falls behind – but it never misses a write and the core database is not impacted. Even if the vector store goes down, the embeddings still persist in the core database. And because ANN queries are CPU-heavy, keeping them on a separate service means they’re not fighting the core database for the same CPU cycles the writes require.

Under billion vector benchmarks, that separated architecture held P99 latency under 10 milliseconds at a concurrency of 300. It handled ~150,000 ANN queries a second with a moderate recall target. Realize that higher recall will bring a latency and throughput tradeoff, and always test this in advance to assess how your own mileage varies. 

Absorb the shock

This one comes down to whether your infrastructure can absorb a sudden change in write pressure or traffic shape without a scramble. The storage engine’s architecture matters a lot here.

For example, ScyllaDB is built on an LSM-tree, which tolerates that kind of write pressure instead of degrading under it. Elastic scaling, with what we call “tablets,” can scale a cluster by something like 10x within minutes instead of hours. That means that if a model rollout changes your access patterns overnight, or you need to absorb a backfill before a big retrain, you don’t end up waiting on a multi-hour resharding job.

The more things change…

So much about AI is genuinely novel, but the infrastructure problems described above generally are not.

“Real-time AI is really a distributed systems problem in a costume.”

Tim mentioned that a feature store was probably the primitive use case: the same high write throughput, low latency work, years before anyone called it AI. Real-time AI is really a distributed systems problem in a costume. Once you understand that, you can design for it so you don’t get blindsided by these not-so-new challenges. 

The post Why real-time AI at scale is so hard appeared first on The New Stack.

“AI kill switch” entered the public conversation because it gives people a simple way to talk about a complex fear. 

As AI systems become more autonomous and harder to evaluate with familiar operating assumptions, a clearly defined intervention capability sounds reassuring. If something starts behaving in a way that creates unacceptable risk, people want confidence that someone has both the authority and the mechanism to stop it. It’s the “kill switch.”

Recent reporting around OpenAI models escaping a sandboxed testing environment and reaching Hugging Face gave that concern a concrete example. CNBC reports that the incident helped trigger a bipartisan bill requiring certain AI companies to maintain the ability to shut down, throttle, or suspend their models, with the Department of Homeland Security given authority to order a slowdown or shutdown in cases involving potential catastrophic harm. 

The political reaction is understandable. When a new category of risk surfaces, especially one the public does not yet know how to evaluate, leaders look for a way to make an abstract concern into something actionable. In this case, that language has formed around shutdown authority.

People who operate large environments tend to hear a different question underneath the policy language. If a shutdown order arrives, what exactly gets shut down?

In a modern production environment, answering that question usually means tracing more than one system. An AI-enabled service may depend on endpoints, APIs, cloud resources, identity systems, package registries, data pipelines, workflow automation, logging tools, and downstream applications that act on model output. 

Some of those dependencies may belong to different teams. Others may sit outside the company entirely. A few may have started as experiments and later become part of a production path without receiving the same scrutiny as the original architecture. By the time the service is important enough to raise governance concerns, it may no longer resemble a bounded application with a single owner and a clean operating surface.

Writing shutdown authority into legislation is far simpler than carrying that decision through a production estate shaped by years of migrations, exceptions, integrations, acquisitions, temporary fixes, and team-level decisions. That implementation gap is where the issue becomes most relevant to infrastructure teams.

For the last several years, much of the AI safety conversation has focused on acceptable use, privacy, model behavior, and human-in-the-loop oversight. Those topics still deserve attention, especially as organizations formalize where AI may be used, which data can be shared, and how employees should evaluate generated output. 

“Writing shutdown authority into legislation is far simpler than carrying that decision through a production estate shaped by years of migrations, exceptions, integrations, acquisitions, temporary fixes, and team-level decisions.”

As AI moves deeper into production workflows, the discussion also needs to involve a more practical concern: when an AI-enabled system creates unacceptable risk, can the organization understand the affected environment well enough to constrain it quickly, consistently, and with evidence?

The phrase “kill switch” may drive the public discussion, but the practical answer lives in the systems surrounding the AI capability.

The limits of a single control

Emergency stops are the kind of control most people picture when they hear the phrase “kill switch.” They make sense in physical systems. Manufacturing equipment, industrial machinery, and certain safety-critical devices can be designed with direct shutdown mechanisms. Software estates already stretch that metaphor, and enterprise AI stretches it further.

The model may be the most visible part of the discussion, although it is rarely the full surface area. An AI assistant used in software delivery might have access to repositories, CI/CD tools, artifact stores, ticketing systems, secrets, test environments, and deployment workflows. An AI agent used in IT operations might read telemetry, recommend remediation, open change requests, call automation scripts, or modify infrastructure through approved orchestration paths. 

Stopping one part of that chain can leave other paths untouched. Turning off a service may not revoke the credentials it uses. Suspending inference may leave downstream systems acting on outdated outputs. Interrupting the wrong dependency can create a separate service incident while the original risk remains only partly contained. Anyone who has worked through a security incident, emergency patch cycle, or major outage knows how quickly a clean decision turns into a sequence of technical tradeoffs.

A credible response plan must account for the system as it exists now, not as it looked during an architecture review months earlier. Infrastructure teams bring useful skepticism to that exercise because they are used to tracing scope, access, ownership, dependencies, and verification paths under pressure. They also know that many environments contain a gap between documented intent and production behavior.

Once AI is embedded in business workflows, those operational details become part of the governance conversation. They expose the places where policy language has moved faster than the infrastructure knowledge needed to make policy executable.

Before containment comes discovery

Much of the public conversation assumes organizations know where AI is running, which is a generous assumption in many enterprise environments.

AI can enter an enterprise through obvious channels, such as internally approved model providers or purpose-built applications. It also arrives through less visible paths. A SaaS product adds an AI feature. A development team experiments with an open source model. A hosted API gets attached to an internal tool. A vendor introduces an AI capability inside software the company already approved. Over time, the line between an “AI system” and a system that happens to use AI becomes harder to define.

This is where the kill switch metaphor starts to show its limits. If the relevant systems are discovered during the response, the team is already behind. Dependency questions, business impact, access paths, and evidence collection all become harder when the basic inventory is still being assembled.

“If the relevant systems are discovered during the response, the team is already behind.”

Infrastructure teams have seen versions of this problem before. During incident response, a service thought to be isolated turns out to have undocumented consumers. During a cloud migration, a supposedly unused integration is suddenly linked to a business process. During an audit, ownership records, configuration data, and actual operating conditions refuse to line up cleanly. AI adds a new category of concern, but the underlying visibility problem is familiar.

The challenge quickly expands beyond the model itself. Teams need to understand which systems call external models, where generated content influences workflows, which accounts and automation paths sit between a recommendation and an action, and how third-party AI capabilities have found their way into the environment.

Most discussions start with how to stop risky AI behavior. In many environments, the more revealing question comes earlier: can the organization produce a reliable picture of where AI touches the estate, which systems depend on it, and which workflows would keep moving if access changed?

Containment depends on the state of the system

Most infrastructure teams know that containment is less a single action than a set of operating conditions. The difficult work happens before the incident, when teams decide what should change if risk reaches a level that requires intervention.

Under ordinary circumstances, a service operates with a defined set of permissions, connections, dependencies, and logging requirements. Under restricted operation, selected assumptions change while investigators preserve evidence and determine whether the risk has been contained. A team might turn off endpoints, suspend integrations, limit external network access, revoke or rotate credentials, increase logging, or isolate workloads while the situation is being investigated.

The details vary because environments vary. That is exactly why generic answers tend to fall apart. A useful containment plan must match the systems it is intended to govern, including the dependencies that surround them and the business processes that rely on them.

Inventory sounds mundane until a response effort depends on it. In my work with infrastructure and compliance teams, I’ve repeatedly seen organizations struggle to maintain an accurate picture of their environments as cloud resources, Kubernetes clusters, SaaS services, and AI projects multiply faster than governance processes can track them during normal operations, which creates audit and support headaches. During a containment event, those same blind spots slow investigations, complicate dependency analysis, and make evidence harder to produce.

Governance eventually reaches production

Governance efforts often begin with documentation. Committees are formed to define responsibilities, agree on escalation paths, and establish a common language for discussing risk before an incident forces the issue. 

The conversation shifts once someone asks whether the control can be demonstrated. A document can describe who has authority to suspend an AI-enabled service. Still, it cannot turn off an integration, revoke a credential, increase logging, or prove that a set of systems entered a restricted condition. A risk register can identify a containment scenario, although it cannot document which nodes changed, when they changed, and whether they remain aligned with the required configuration.

Security, compliance, and infrastructure teams know this gap well. It appears in patching programs, configuration baselines, incident response exercises, supply chain reviews, and disaster recovery planning. Written controls tend to be cleaner than the real-world environments they describe. Production systems reflect years of accumulated decisions, exceptions, migrations, temporary fixes, acquired assets, and workarounds that may outlive the original reasons they existed.

“Written controls tend to be cleaner than the real-world environments they describe.”

AI increases urgency because some systems are becoming more autonomous and more connected to business workflows. It also adds outside pressure. When an incident becomes visible enough to prompt legislative action, boards and customers start asking sharper questions. A company may be able to point to an AI policy, but boards, customers, and regulators eventually want to understand exactly how that policy translates into action.

If leadership declares an AI-enabled workflow should be restricted, the discussion moves quickly from oversight to execution. Teams need to know where to intervene, which systems are affected, who owns the required changes, how completion will be verified, and what evidence remains once the response is over.

A vague answer may pass during experimental stages but becomes much harder to defend once AI is embedded in production services, regulated workflows, customer-facing systems, or environments connected to critical business operations.

Why a mandate will not solve the estate problem

A federal shutdown authority, if enacted, would place legal pressure on a narrow class of powerful AI providers. It would not remove the implementation burden for organizations that adopt, integrate, fine-tune, host, or embed AI systems inside their own environments.

Even if a major AI provider can throttle or suspend a model, each enterprise still must understand its own exposure in the context of its applications, workflows, dependencies, and operating assumptions. Which applications depend on that model? Which workflows fail open or fail closed if access is restricted? Which internal systems contain cached outputs, calculated decisions, or agent-created changes? Which business processes need manual fallback when an AI service is unavailable?

Policy debates often treat AI control as if the decisive action happens at the model layer. Sometimes it will; however, in many business environments, the risk will live in the connections that surround the model. A hosted AI service may be suspended while local workflows, scripts, integrations, and access tokens continue following their last known configuration.

“A serious AI containment strategy has more in common with mature infrastructure management than with an emergency stop button.”

A serious AI containment strategy has more in common with mature infrastructure management than with an emergency stop button. It requires an up-to-date inventory of AI-adjacent systems, a map of dependencies and access paths, predefined restricted conditions for high-risk services, tested procedures for applying those conditions, and evidence that changes were enforced. Ownership also needs to be clear, since fast action becomes difficult when authority is scattered across teams.

The work is less exotic than the public conversation can make it sound. AI-enabled systems still need to be managed as production systems with real dependencies and business impact.

Infrastructure teams belong earlier in the conversation

Spend enough time running infrastructure, and you develop a complicated relationship with documentation. Most organizations have diagrams, inventories, and governance processes, and all of them serve a purpose. The challenge is that production systems keep evolving long after those artifacts are created. Acquisitions introduce systems that do not fit cleanly into existing models. Applications gain integrations nobody anticipated during the original design process. Temporary exceptions become permanent. Cloud resources intended to live for a week are still running after a year.

Most of this happens for defensible reasons, usually in support of uptime mandates, delivery pressure, customer needs, or business continuity SLAs. The result is that operational knowledge becomes dispersed across people (some of whom will inevitably have moved on), tickets, runbooks, monitoring systems, and memory rather than living neatly in one place.

Infrastructure teams spend their days tracing dependencies, untangling ownership questions, and figuring out how systems behave outside a design review. Bringing that perspective into AI governance conversations early can prevent containment plans from depending on assumptions that did not translate into production. It also helps organizations understand the difference between disabling a model, restricting access to a service, isolating a workload, and preserving evidence during an investigation.

Scale complicates things further. A manual action that works effectively for ten systems may fail across hundreds or thousands. A change one expert can perform during business hours may become fragile if that person is unavailable when an event occurs. A runbook that looks adequate in a tabletop exercise may not survive a live environment where dependencies have changed, and the current ownership is unclear.

The phrase “kill switch” will probably remain part of the public debate because it is simple, memorable, and familiar. Practitioners do not have to accept the metaphor literally to leverage the attention it creates. They can redirect the conversation toward more useful questions: what restricted operation would mean for a given service, which dependencies would have to change, which controls can be applied reliably, which steps remain manual, and how the organization would prove the response worked.

These questions are less dramatic than a big red button, but their answers are also most likely to improve readiness.

Control starts before the incident

The Hugging Face incident gave the industry a vivid story, and Washington responded with the language of shutdown authority. That reaction is understandable. Leaders want mechanisms that sound equal to the risk, especially when the public conversation moves faster than the technical details can be explained.

By the time an organization begins thinking about containment, much of the hard work should already be done. Teams should already understand what is running, who owns it, what depends on it, and how changes will ripple through the environment.

AI can reduce certain workflow bottlenecks, but it also exposes weak inventory, unclear ownership, and brittle operating assumptions faster than many teams are prepared to handle. A future incident will not pause while teams locate assets, clarify ownership, identify credentials, or discover that a service dependency was never documented.

The current debate may be framed around new kill switches. For most organizations, the more useful work starts with building and maintaining an accurate picture of the systems, dependencies, and workflows that already exist across the estate.

The post The “AI kill switch” assumes you know what you are trying to shut down appeared first on The New Stack.

Cloud platform company Nscale announced this week a definitive agreement to acquire AI workload scaling specialist Anyscale, in a move that signals a new test of whether cloud-neutral AI software can stay neutral once it is paired with a GPU neocloud.

The purchase coalesces Nscale’s infrastructure capabilities, which span control systems that oversee GPUs, datacenters, power consumption, and the application layer where AI services themselves are executed, with Anyscale’s software layer for scaling AI workloads across data processing, training, inference, and reinforcement learning.

Argued by Nscale to be the coming together of “two highly complementary companies”, Nscale scooping up Anyscale could be a fundamental change in the resulting business model. 

Is this the start of GPU neocloud lock-in?

It’s important to remember that Nscale is a GPU neocloud (a specialized cloud provider running bare-metal GPUs and infrastructure optimized for AI and machine learning workloads), meaning that it runs its own GPU-rich datacenters and its own software ​stack. At the same time, Anyscale is an independent cloud-neutral software orchestration multi-cloud control plane that works with any cloud hyperscaler… but now owned by a single neocloud. 

That doesn’t sound quite so much like cloud-neutrality and agnosticism; it sounds more like a vertically integrated AI cloud provider proposition.

Chief product officer at Nscale, Dan Bathurst, tells The New Stack that the Anyscale platform “continues to be its own brand and product,” and that includes working with bring-your-own-cloud deployments on AWS, GCP, Azure, and the other clouds. 

“Where we want to win is on performance, not on any sort of vendor lock-in or forcing of someone to choose Nscale as the infrastructure provider.”

“But what really changes — or how it’s changing — is that customers now also get this first-party option, where they can have Anyscale running on Nscale fleet as a full-stack, highly-optimized solution. Where we want to win is on performance, not on any sort of vendor lock-in or forcing of someone to choose Nscale as the infrastructure provider,” Bathurst says.

He insists that it is in Nscale’s interest to ensure that it is making it easy for software engineering teams to get the outcomes they want with the workloads that they’re trying to run.

“For us, the existing commitments will carry forward, so Nscale’s value really is meeting instances where the compute already lives,” he says. “Where we want to win is on performance, not on any sort of vendor lock-in or forcing of someone to choose Nscale as the infrastructure provider.”

Neutrality on the platform layer, differentiation on the infrastructure layer

Bathurst invites users to think of it as “neutrality on the platform layer, but differentiation on the infrastructure layer” because the combination of the two organizations is a full-stack play.

“The differentiation comes from the fact that Nscale is fully vertically integrated with Anyscale. Therefore, if users want that first-party option, they can choose Anyscale and get the most optimized solution because, obviously, we’re designing, optimizing, and co-engineering every layer of that stack from power to the datacenter through to the application. It’s quite a unique proposition, but it’s not something we are going to force upon any customer,” confirms Bathurst.

Not everyone is convinced by the company’s pledge to maintain an agnostic and neutral open house. Sanjeev Mohan, principal analyst, SanjMo and former Gartner research VP for data and analytics, tells The New Stack that Anyscale “stops being a neutral player” the moment its best features and most optimal pricing land on Nscale first. 

“The software will still run anywhere, but ‘runs anywhere’ and ‘runs best somewhere’ are different things, and buyers will feel the gap in performance and cost. At that point, neutrality is a label.”

Runs anywhere, but… runs best somewhere

“The software will still run anywhere, but ‘runs anywhere’ and ‘runs best somewhere’ are different things, and buyers will feel the gap in performance and cost. At that point, neutrality is a label,” says Mohan. 

He agrees that integrating software and compute will produce measurable cost, performance and reliability gains. Defining this as “the strongest part of the deal”, Mohan explains that with Nscale controlling both the silicon and Anyscale’s control plane, it can tune scheduling, memory, and networking together in ways the compute-neutral Anyscale never could.

Anyscale commercial support for Ray

Anyscale was founded by the creators of Ray, an open source project that provides a distributed computing framework designed to scale Python workloads across any infrastructure into live production application jobs and services. 

Ray was donated to the PyTorch Foundation in 2025. Anyscale continues to provide its commercially supported services for Ray, which include a “no DevOps” route to 100% managed cloud infrastructure and serverless autoscaling, making it simpler to create, deploy, and monitor machine learning workflows in production.

Anyscale supports data processing, model training, batch inference, and LLMs across public and private cloud environments. As open source as this all feels, are we still edging towards narrower proprietary channels, or the possible threat of deeper application and data service dependencies that developers will ultimately have to wrangle around?

“I don’t think so, primarily because the way that the platform works, it’s designed to orchestrate across various different clouds and different infrastructure. It’s like a heterogeneous distributed compute platform. So the platform’s always gonna remain multi-cloud,” confirms Nscale’s Bathurst.

Pricing permutations and hyperscalers hearsay

Pressed on any forthcoming pricing changes or likely reactions from the major cloud hyperscalers in relation to Nscale now being a credible alternative, Bathurst and team were (perhaps understandably one day after an acquisition deal announcement) politely tight-lipped.

More voluble is always-affable analyst Mohan, who says that, “Every optimization that only shows up on Nscale hardware is a dependency. So, an argument can be made either way. Standalone orchestration software and independent tooling vendors are getting absorbed into whoever owns the GPUs, because the economics only work when you control both. Expect more of it,” Mohan underlines.

He explains that Nscale “now becomes a real specialist cloud services provider alternative,” i.e., not a general-purpose one like AWS, Azure and Google Cloud with their plethora of managed services, from databases and data warehousing to container orchestration through to AI/ML pipeline technology.  However, he does see space for Nscale to become a strong player in raw training and inference at scale.

From cryptocurrency to cloud contender

London, UK-based Nscale was established in 2024 from what was originally a cryptocurrency mining business. 

As suggested, Anyscale will retain its brand name as part of the Nscale family, and the company has restated its stance that customers are “free to choose the cloud infrastructure on which they run their AI workloads” today.

The company’s initial press statement said that “over time” users will gain the additional option of running the Anyscale software layer on Nscale’s full-stack AI platform. 

The first full-stack AI hyperscaler?

“Companies are moving beyond simply using AI to actually building their own. Doing that well requires the software and the infrastructure it runs on to be designed together,” says Keerti Melkote, CEO of Anyscale in the press release announcing the acquisition.

Melkote has defined the combination of Anyscale’s platform — built on Ray — with Nscale’s datacenter, compute and AI cloud services as the “first full-stack AI hyperscaler,” i.e., one that runs any AI workload at greater scale, so more software engineering teams can build and own their AI applications and services.

With this acquisition and the fusion of Nscale with Anyscale’s software layer, the organization will aim to widen its customer base. Existing work sees the company working in verticals from healthcare to e-commerce to robotics. It says its full stack offering will help companies speed up image and document processing, fine-tune LLMs on their proprietary data, and deploy AI agents in-house using open-source models.

The transaction is subject to closing conditions and regulatory approvals and is expected to close in the second half of 2026. Financial terms of the transaction were not disclosed, although Reuters reports a source stating that the deal price is “about $1.65 billion”, according to a person familiar with the deal.

AWS, Google Cloud and Microsoft Azure representatives were all contacted and invited to comment on this story.

The post Nscale just bought Anyscale. Here’s why it matters for multi-cloud neutrality. appeared first on The New Stack.

AI trash-talkers love to rip on the technology for failing to produce meaningful business results, often pointing to studies like that from MIT NANDA, which reveals a 95% failure rate for enterprise AI solutions, or that from IDC, which states “only 9% of [Europe, Middle East, and Africa] organizations have been able to deliver measurable business outcomes from most of their AI-related projects over the past two years.” 

What many AI skeptics fail to account for is the experiential nature of AI prototypes; not all these projects are actually meant to move beyond the testing phase. Still, a 5% success rate is embarrassing. 

What’s the holdup? 

Two things. First, most organizations build AI prototypes on sand; that is, the data infrastructure on which they build early applications can’t support later moves to production. Meanwhile, the operational teams responsible for managing those applications in production often lack the human power to keep up with engineering’s growing output. 

4 reasons prototyping infrastructure  ≠ production infrastructure

When asked why so many AI prototypes don’t make it to production, Phillip Merrick, co-founder, CPO, and chairman, pgEdge, tells The New Stack that data infrastructure is largely to blame. 

Specifically, he explains that prototype environments don’t meet the requirements of large enterprises for production, naming four main ways they fall flat.

First, Merrick says prototyping environments lack the deployment flexibility organizations need to move from prototype to production. 

Vendor-managed cloud platforms, he acknowledges, may seem like an obvious choice for prototyping, as they allow teams to get up and running quickly. Still, he warns they lack the technical chops to support AI applications in production, especially in security, compliance, and governance. Particularly for organizations in healthcare, finance, or other regulated industries, vendor-managed cloud platforms often lack the stringent controls found in self-managed cloud or on-prem environments. 

“You’ve got to be able to choose where that AI prototype is ultimately going to be put into production.”

In this way, flexibility and security go hand in hand. Merrick asserts. “You’ve got to be able to choose where that AI prototype is ultimately going to be put into production.” 

Similarly, when it comes time to shift to production, Merrick says vendor-managed cloud platforms can introduce data sovereignty challenges at both the enterprise and regional levels. 

“Your data layer is obviously where you enforce this,” notes Merrick. But he says the environments most teams use for prototyping muddy the waters: “If it’s on a vendor-managed platform in who knows what cloud, what region, then you’ve lost data sovereignty.” 

Lastly, Merrick brings attention to reliability, explaining that AI prototypes can’t move into production without assurance of high availability. For example, when it comes time to upgrade the database or swap hardware, can it be done without downtime? 

“In the vendor-managed cloud world, the answer to that is almost always no,” says Merrick, reiterating his point that moving AI apps from prototype to production requires enterprise-grade infrastructure. 

So why are developers prototyping where they can’t productionize?

If data infrastructure selection is what’s holding developers back from moving AI prototypes into production, then why do they keep starting on the wrong foot? 

As Merrick explains, many are attracted to the ease of use of vendor-managed cloud platforms. “These prototyping environments admittedly make it very easy to get started,” he says. But prototyping shortcuts, it seems, don’t pay off in the long run, as someone else is ultimately on the hook for making those prototypes production-ready. 

Still, Merrick doesn’t blame developers for looking for the easy way out. Rather, he says there’s a disconnect between the prototyping playground and the production battleground that prevents developers from understanding what it will take to productionize prototypes down the pike. 

More pgEdge articles in The New Stack

• Why developers are betting on Postgres for AI
• Moving AI apps from prototype to production requires enterprise-grade postgres infrastructure
• Reinventing PostgreSQL for the Next Generation of Apps
• How distributed Postgres solves the cloud’s high-availability problem
• Scaling AI inference at the edge with distributed PostgreSQL

Years ago, he says, tooling decisions were primarily made top-down without developer input: “Then, starting 15–20 years ago, the developers won back, quite rightly, the power in being able to choose their own tools.” 

The problem now, Merrick claims, is that developers make tooling decisions exclusively for prototyping environments, without anticipating production needs. Internal divisions mean developers are often only responsible for building prototypes before passing the baton to an entirely separate operations team for production: 

“The upshot is you really don’t have, in some organizations, this throughline of understanding [of] what the production requirements are all the way back to the developer making the initial choices.” 

For Merrick, this disconnect is where AI projects start to fall apart, as teams are left trying to move AI prototypes from accessible-but-inadequate vendor-managed cloud platforms to enterprise-grade data infrastructure that meets requirements for deployment flexibility, security, data sovereignty, and high availability. 

“But if you make the right data infrastructure choice, you won’t have that disconnect,” he says, “because you’ll have this throughline from prototype to production.” 

He names Postgres as the data infrastructure that best helps developers bridge this divide, calling it “the Swiss army knife of databases” due to its extensibility, fully open-source nature, and ability to address diverse data management problems, from unstructured data to vector embeddings to geospatial data. 

Where and how Postgres is run matters too, Merrick points out, again drawing attention to the limits of many vendor-managed cloud environments that often lack the governance controls to meet data requirements and/or the deployment flexibility to shift to compliant on-premises or BYO cloud environments. 

But picking the right data infrastructure only solves half the problem

Merrick says there’s another part of the equation most organizations are overlooking: people, or more precisely, database administrators (DBAs) and their growing workloads. 

Per Stack Overflow’s 2025 Developer Survey, 84% of respondents use AI tools, up from 76% the year prior. Meanwhile, Supabase says over 60% of databases on its platform have been launched “by some sort of AI tool.” As Merrick points out, this explosion of productivity comes with a catch: there aren’t enough DBAs to keep up.

“You’ve had this massive, massive step shift in developer productivity,” he explains. “But you have to have some way of managing that on the production side; these databases can’t go unmonitored.” Operations and administration teams were already struggling to keep track of existing Postgres databases before agentic engineering added even more, he says: “Who’s going to manage them?” 

He says it’s time for agentic operations to catch up with agentic engineering. 

AI DBA agents can give humans “superpowers” 

If it seems Merrick is proposing organizations look to fully autonomous DBA agents to take over, he says the industry isn’t there yet: 

“The world is not ready for fully autonomous databases administered by AI DBA agents. But there is a massive resource shortage and productivity problem, and DBAs can only manage so many databases,” he explains. Meanwhile, new “AI applications require so many more databases to put in production.” 

“The world is not ready for fully autonomous databases administered by AI DBA agents.”

So how can organizations increase their operational capacity? 

Merrick says DBAs should look to new AI DBA agents, not to take over but to give them “superpowers” to monitor and manage more databases with less manual slog. 

pgEdge’s Ellie is one example. Part of the pgEdge AI DBA Workbench, Ellie is an AI agent that has 21 MCP tools and can run EXPLAIN ANALYZE, inspect schemas, query historical metrics, and walk through multi-step diagnostic workflows. When a database falters, Ellie finds the problem, diagnoses it, and provides a solution in the form of working SQL code for the human DBA to review. “When you’ve reviewed it and agree that it’s the right course of action, you literally press the play button, and the agent plays that SQL code into the database, and you solve your problem,” explains Merrick.

In this way, Ellie should bring more capacity to operations teams, where Merrick insists organizations are starved for DBA expertise. To his point, some industry predictions say 41% of today’s database professionals intend to leave the industry in the next decade, half moving into retirement and the rest seeking other work. 

“An agent … can actually respond to those alerts far more quickly and productively than a human can.”

Without AI agents, Merrick argues, DBA work is tedious, laborious, and time-consuming. As he explains it, a database may have been humming along just fine, but when there’s a snag, trouble can manifest across multiple applications; it’s then up to the DBA to comb through monitoring data to observe and diagnose the problem, essentially scouring for a needle in a haystack. 

“An agent,” he says, “can actually respond to those alerts far more quickly and productively than a human can.” 

Better infrastructure AND people: It takes two to improve AI prototype success rates 

In nearly any context, AI raises questions about quality over quantity, and enterprise AI projects are no exception. Agentic engineering means developers can now produce more, but all those prototypes don’t just fly directly into production. Limitations in both infrastructure and operational human power are creating obstacles that cause many AI prototypes to fail. 

For Merrick, easing the transition from prototype to production requires not only great AI tooling but production-ready data infrastructure, paired with agentic operations that can keep up with the agentic engineering boom. 

The post Why most AI projects fail: It’s infrastructure and people  appeared first on The New Stack.

The biggest obstacle to enterprise AI isn’t models, data science talent, or even infrastructure.  It’s operations.

Across today’s enterprises, hybrid complexity has outpaced IT’s ability to manage it. Applications, workloads, runtimes, and infrastructure now span on‑premises environments, public clouds, edge locations, and air‑gapped sites. Each layer brings its own tools, vendors, and operational language. The result is friction everywhere and a widening gap between AI ambition and operational reality.

Latha Vishnubhotla, chief platform officer at Hewlett Packard Enterprise, tells The New Stack the challenges begin on Day 2.

“People can bring things up and make them functional very quickly,” says. “But where they spend most of their time is after the infrastructure becomes functional. Day 2 to Day N is where they spend a lot of time.”

That’s the problem enterprises are running into now. It’s not getting infrastructure up and running, but keeping it running, optimized, and reliable as AI workloads move from pilot to production.

Read on to dive into not only the Day 2 problem, but to learn how HPE’s GreenLake hybrid cloud management platform has grown to respond to this enterprise complexity — including that cross-platform infusion of agentic AI. 

Day 2 is when the hybrid cloud breaks down

In hybrid environments, operations teams aren’t managing a single stack. They’re juggling multiple runtimes, from bare metal and VMs to containers and AI‑native platforms.  Infrastructure across compute, storage, and networking often from different vendors. Workloads spread across data centers, public clouds, edge, and disconnected sites.  Legacy systems that were never designed to work together

Each layer has its own management tools and telemetry. When something goes wrong, the symptom rarely appears in the same place as the root cause.

“All these different tiers are talking to each other, but it’s not linear. You have to comb through and figure out where the issue actually is.”

“All these different tiers are talking to each other, but it’s not linear,” Vishnubhotla says. “You have to comb through and figure out where the issue actually is.”

Day Zero provisioning may be fast. Day 2 operations are where complexity compounds and teams burn time reacting rather than optimizing.

More AI is making the ops problem worse

AI not only raises the stakes but also delivers a solution.

Enterprises want to run more AI workloads, but data centers have finite capacity. Power, cooling, cost, and sustainability constraints are real. That’s why FinOps and GreenOps have become inseparable from infrastructure operations.

“When you want to run these workloads, you have to ask: what’s not being used?” Vishnubhotla says. “Why am I wasting here? Should I move something? Should I retire it?”

This is where traditional, human‑driven ops models start to break. There’s too much data, too many layers, and too many dependencies to reason about manually, especially at enterprise scale.

The ops platform as connective tissue

What enterprises need isn’t another point tool. It’s an operations platform that acts as connective tissue across the hybrid estate.

That’s the role GreenLake is designed to play.

GreenLake provides a unified platform experience for running and managing hybrid environments across on‑premises, private cloud, edge, and collocated infrastructure while preserving choice and control. Instead of hiding infrastructure behind abstraction, it makes it visible, observable, and operable from a single control plane.

“The control plane is actually running in the cloud,” Vishnubhotla says. “You get visibility across the entire estate.”

For organizations managing thousands of sites and tens of thousands of devices, that visibility is foundational. But visibility alone isn’t enough anymore.

Why agentic AI changes everything

The next step is agentic AI, AI systems embedded directly into the ops platform, trained on the context of specific infrastructure domains.

A networking agent understands networking. A storage agent understands storage. A compute agent understands compute. Each brings deep, domain‑specific intelligence to Day 2 operations.

“Each layer already has intelligence,” Vishnubhotla says. “If we can connect this intelligence, we can unleash very powerful outcomes.”

That’s where the idea of an agentic mesh comes in. Instead of siloed insights, AI agents share context across layers during provisioning, troubleshooting, and optimization. This shortens the time to root cause, reduces alert noise, and opens the door to predictive and, eventually, autonomous operations.

Predictive maintenance is a clear example. Rather than reacting to failures, AI can anticipate what’s likely to break, prioritize what actually matters, and help teams act before outages cascade.

Faster time to value for AI starts with competent ops

Agentic operations also unlock something enterprises care deeply about: faster AI ROI.

With a shared, platform‑level view, ops teams can answer questions like:

• What is connected to the estate
• Where is infrastructure deployed?
• Who’s using it—and how?
• Where is capacity being wasted?

GreenLake supports automation through copilots and MCP servers as well as UI‑driven workflows, reducing provisioning times and operational overhead. AI agents can even help predict demand and close feedback loops that used to take weeks.

“The bottleneck has always been on the ops side. Enterprises are deploying and operating infrastructure from Day Zero to Day N to unlock AI value faster.”

“The bottleneck has always been on the ops side,” Vishnubhotla says. “Enterprises are deploying and operating infrastructure from Day Zero to Day N to unlock AI value faster.”

The answer is a platform, not another tool

Hybrid complexity isn’t temporary. AI pressure isn’t slowing down. And Day 2 operations are only getting harder.

That’s why the industry is converging on a clear conclusion: the answer isn’t more tools; it’s a unified, intelligent ops platform.

GreenLake brings together visibility, agentic AIOps, and cross‑domain intelligence in a platform built for how enterprises actually run today. It connects the silos, scales operations teams, and turns infrastructure from a bottleneck into an enabler.

If AI is the future of the enterprise, operations is the gatekeeper.  And the ops platform powered by agentic AI is how that future gets unlocked.

The post Why agentic AI makes the ops platform the most important layer in the enterprise appeared first on The New Stack.