Update: As of 6:30 p.m. Eastern on Friday, September 4, GPT-6 Astra was available on ChatGPT for all paying users with plans that include access to it.

Thibault Sottiaux, a leading member of the technical staff at OpenAI, posted on his X account, “OK nevermind, the team and Astra did a good job and our systems are more scalable than we anticipated. Astra is now rolled out to all Plus and Business users too. Hope you have a blast and let us know how it goes!.”

Earlier in the day Friday, OpenAI CEO Sam Altman posted on X, “GPT-6 Astra is now available to all Pro, Enterprise, and Business Premium users in Work/Codex, and is available in the API. We will start rollout to Plus and Business users next. Thank you for the patience.”

Users who pay for the $8/month ChatGPT Go plan do not and will not have access to GPT-6 Astra or GPT-5.6, according to the company’s pricing tier documentation.

Sottiaux posted on Thursday evening after OpenAI announced the debut of Astra but before it released it to users: “We are starting to release GPT-6 Astra, and we are doing it as carefully and quickly as possible.”

Our original story, published at 12:24 p.m. Eastern on Friday, continues below:

OpenAI launched GPT-6 Astra on Thursday, but many developers hoping to give it a test drive are still waiting for access.

Hours after the announcement, OpenAI CEO Sam Altman apologized for what he called a “messy rollout,” acknowledging that broad access to Astra had not yet begun for either API customers or ChatGPT subscribers.

“First, sorry for the messy rollout,”

“First, sorry for the messy rollout,” Altman wrote in a post on X. He added that OpenAI expected to begin the broader rollout “in the near future,” starting with ChatGPT Pro subscribers.

For developers, that creates an unusual bind. OpenAI has already published Astra’s API documentation and pricing, including its 1.05 million-token context window, support for up to 128,000 output tokens, and standard API rates of $10 per million input tokens and $50 per million output tokens. But the endpoint itself is still rolling out, and OpenAI has yet to explain specifically what went wrong with the rollout. So while Astra’s benchmark results were impressive, developers and independent reviewers who want to evaluate those claims against their own workloads are still left waiting.

Why developers are waiting

OpenAI’s engineering lead for Codex, Thibault Sottiaux, offered a little more detail in an X post of his own about what is happening behind the scenes. He said the rollout will take “a few days” to complete and that OpenAI is bringing new systems and additional compute online as it expands access.

“We are starting to release GPT-6 Astra, and we are doing it as carefully and quickly as possible,” Sottiaux wrote. He added that “many novel systems will operate at scale for the first time” during the rollout and that OpenAI is “bringing a lot of compute up.”

“We are starting to release GPT-6 Astra, and we are doing it as carefully and quickly as possible,”

While this does not establish that compute capacity caused the delay, it does offer some insight into what OpenAI is dealing with as it expands access. The company’s original announcement said Astra would initially be available to a limited group of organizations, with ChatGPT Plus, Pro, Business, and Enterprise users, the OpenAI API, Microsoft Azure, and AWS Bedrock expected to follow “over the coming days.”

Developers who have secured early API access are already encountering surprises beyond the rollout itself. As The New Stack reported this week, Astra’s API introduces a new class of safety-triggered interruptions that look like timeouts but aren’t — which could make a real difference for developers building production workflows around the model.

Banked resets explained briefly

While users wait, OpenAI is trying to make up for at least part of the delay with something it calls a “banked reset.”

Sottiaux said paid ChatGPT subscribers will receive one banked reset for every day they remain without Astra access, beginning September 3, adding that the “team is moving mountains to give access as fast as we can.”

“Team is moving mountains to give access as fast as we can.”

OpenAI has used banked resets before with ChatGPT Work and Codex, giving users a way to replenish their usage after hitting a limit. They aren’t additional API credits or a permanent increase in usage limits but resets users can save until they need them.

What’s different with Astra is how OpenAI is handing them out: one for every day a paying ChatGPT subscriber remains without access. So far, OpenAI hasn’t announced anything similar for developers waiting to use Astra through the API.

The gesture fits a broader pattern of OpenAI experimenting with how it charges for AI: the company recently announced an outcome-based pricing model that would bill only when the model produces a correct result; another sign that its pricing strategy is still very much in flux.

Compute capacity complicates rollout

OpenAI says Astra will roll out to Plus, Pro, Business and Enterprise users, along with the OpenAI API, Microsoft Azure and AWS Bedrock, “over the coming days,” with Pro subscribers first. Sottiaux said the rollout should take a few days to complete.

For developers, the biggest unanswered question is when broad API access will arrive and whether it will come with tighter usage limits. Astra’s persistent-agent capabilities also make the wait more significant for developers.

OpenAI did not immediately respond to The New Stack’s questions about the rollout problems, banked resets, or API timeline.

The post “Sorry for the messy rollout”: OpenAI launches GPT-6 Astra to most paying users a day after its unveiling appeared first on The New Stack.

Modern AI platforms are no longer a single application behind one login screen. A user may start in a central portal, open a governed dataset, launch a notebook where that data resides, and invoke an assistant that calls services in another cluster. The workflow feels unified, but identity crosses control-plane and data-plane boundaries at every step. That is where conventional single sign-on…

Source

When enterprise AI applications scale, they inevitably hit a wall. For many engineering teams, this wall is initially diagnosed as a billing issue, a monthly API invoice that has grown out of control. However, viewing token consumption purely as a financial metric fundamentally misunderstands how LLMs operate in production. Token optimization, unlike its other optimization cousins, is not an accounting exercise; it’s a distributed systems and hardware utilization challenge.

“Token optimization, unlike its other optimization cousins, is not an accounting exercise; it’s a distributed systems and hardware utilization challenge.”

In this guide, we explore, through the lens of Concierge (a latency-sensitive, synchronous customer support agent) and Pathfinder (an asynchronous, multi-step autonomous CI debugging agent), how these systems fell victim to autoregressive bottlenecks as they grew, and how we fixed these issues.

What you’re actually paying for

A token is not a word: treating it like one will break your budgeting “models.” Every major LLM provider tokenizes text using byte-pair encoding (BPE), breaking words into subword units. While common words stay intact, rarer words or punctuation split into fragments. As a rule of thumb, 1 token = 4 characters, or 0.75 words in standard English prose.

When budgeting for production, you must account for the structural pricing spread; providers bill input tokens and output tokens at different rates. Output tokens are typically 4-5X more expensive than input tokens.

As a baseline, assume a mid-tier frontier model runs roughly $3 per million input tokens and $15 per million output tokens.

The quadratic history tax

LLM provider APIs are completely stateless; to make an LLM behave as if it remembers past events, you must resend the entire history of the session and input with every single API call.

This means that a model’s own previous outputs are continuously re-billed to you as inputs on subsequent steps. This triggers a compounding cost that impacts both Concierge and Pathfinder, though their curves scale differently.

Let S be the static system context (instructions and schemas), u be the incoming data per step, and r be the model’s response payload. The input cost for every given turn k is

When you sum this across a complete execution run of N steps, the total input token volume compounds quadratically. 

This O(N^2) accumulation of history is the exact mechanism that causes the explosion in cost and latency.

When we calculate the total input tokens consumed by a single session using the quadratic formula

• Concierge: Consumed 45,300 tokens per 10-turn ticket
• Pathfinder: Consumed 150,000 tokens per 15-step turn

Because Pathfinder’s step increment was 4X larger than Concierge, its token cost curve was drastically steeper. If Pathfinder were to get stuck in an infinite tool-use loop and hit 30 steps, a single run could consume 570,000 tokens.

The solution

Fixing the individual call

Prompt hygiene: Hardcoding static reference documentation into the system prompt means you pay to parse identical text on every turn. So we stripped static text from the prompt and switched to dynamic injection. 

For Concierge, we implemented a RAG step to fetch only the 2-3 policy snippets relevant to the ticket. The prompt dropped from 3,100 tokens to 380. A 60% reduction for a 10-turn thread.

For Pathfinder, we applied automated prompt compression using LLMLingua-2 to compress verbose CI log files before sending them to the model. By filtering out non-essential log lines, we reduced the size of incoming tool observations by 3X without sacrificing debugging accuracy.

from llmlingua import PromptCompressor

compressor = PromptCompressor(
model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",
use_llmlingua2=True
)

try:
compressed_result = compressor.compress_prompt(
raw_ci_log_text,
rate=0.33,
force_tokens=["Error", "Exception", "Failed", "Traceback", "FATAL"]
)
# Pass high-density payload to the frontier model
compact_prompt = compressed_result["compressed_prompt"]
except Exception as e:
print(f"Compression failed, falling back to raw log text: {e}")
# Graceful degradation: pass the raw (or truncated) log if compression fails
compact_prompt = raw_ci_log_text

Eliminating the retry: Relying on open-ended prose instructions “return JSON” caused malformation. When parsing failed, the system initiated a synchronous retry, sending the entire accumulated context as if it were a new attempt. We replaced the entire natural language formatting request with strict structural contracts via forced schema validation.

In both Concierge and Pathfinder, we converted the output format to a strict pydantic schema for tool-calling mode and tool-execution payloads. Malformed outputs across both systems dropped to under 0.5%, eliminating tail latency spikes caused by cascading queues.

# Unified Schema Enforcement for Concierge Responses & Pathfinder Tool Execution
from pydantic import BaseModel
from typing import Literal

class TicketResponse(BaseModel):
    reply: str
    category: Literal["shipping", "returns", "billing", "product", "other"]
    escalate: bool
    confidence: float

# The API is structurally locked into emitting validated JSON matching the schema
response = client.messages.create(
    model="claude-opus-4",
    system=SYSTEM_PROMPT,
    messages=messages,
    tools=[
    {
    "name": "respond_to_ticket",
    "description": "Formulate a response and classify the support ticket.",
    "input_schema": TicketResponse.model_json_schema()
    }
    ],
    tool_choice={"type": "tool", "name": "respond_to_ticket"},
)

Output token bounding: Models naturally generate verbose reasoning chains and conversational filler, inflating expensive output tokens. Where the LLM provider exposes logit bias, you can directly suppress every token outside the valid set at decode time; where it doesn’t, constrained decoding libraries (Outlines, Guidance) or a forced tool call with an enum-typed schema will get you the same guarantee.

class ClassifyOnly(BaseModel):
    category: Literal["shipping", "returns", "billing", "product", "other"]
    priority: Literal["low", "medium", "high", "urgent"]

State management 

The stateless nature of the models meant we had to parse the static prompt prefix and historical steps on every turn. We introduced explicit cache breakpoints to allow the inference engine to reuse the states of static blocks. We altered both Concierge and Pathfinder to flag stable, historical segments for caching. Under standard vendor pricing, cache reads are discounted by 90%. It is important to check with your vendor on whether caching is enabled. 

# Caching the stable history prefix for a multi-turn session
response = client.messages.create(
    model="claude-sonnet-4",
    max_tokens=4096,
    system=[{
        "type": "text",
        "text": SYSTEM_PROMPT,
        "cache_control": {"type": "ephemeral"} # Cache hits drop prefix costs by 90%
    }],
    tools=TOOL_SCHEMAS,
    messages=session_history + [{"role": "user", "content": current_step_input}],
)

For a 10-turn Concierge chat, this dropped input costs by ~70%. For a 15-step Pathfinder trajectory, it resulted in a 76% cost reduction.

Semantic caching 

Duplicate queries across separate sessions were triggering redundant frontier model invocations. We implemented a vector similarity cache layer upstream of the LLM using Redis. Our Concierge service analysis showed that 34% of customer support tickets were semantic duplicates of common FAQs.

Intercepting these requests reduced latency to sub-50ms for hits. Because of the nature of CI pipeline logs, we have not yet found a suitable cache for Pathfinder’s inputs. 

import os
import json
import redis
from redis.commands.search.query import Query

# Configure connection via environment variable for environment portability
redis_url = os.environ.get("REDIS_URL", "redis://localhost:6379")
r = redis.Redis.from_url(redis_url)

def get_cached_response(tenant_id, query_text, threshold=0.92):
    try:
results = r.ft(f"cache_idx:{tenant_id}").search( # scoped by tenant -- see below
Query("*=>[KNN 1 @vector $vec AS score]").sort_by("score").dialect(2),
query_params={"vec": query_vec.tobytes()},
)
except redis.RedisError as e:
print(f"Redis cache error: {e}")
return None # Fail-open: gracefully fall back to a cache miss)
  query_vec = embed(query_text) # small, fast bi-encoder -- not the frontier model

try:
results = r.ft(f"cache_idx:{tenant_id}").search( # scoped by tenant -- see below
Query("*=>[KNN 1 @vector $vec AS score]").sort_by("score").dialect(2),
query_params={"vec": query_vec.tobytes()},
)
except redis.RedisError as e:
print(f"Redis cache error: {e}")
return None # Fail-open: gracefully fall back to a cache miss

Semantic caching could be a security problem, because if you choose a global cache, Customer A’s account-specific answer could get served to Customer B because their phrasing embeddings are close enough. To mitigate this, we split the cache into two tiers: a global cache for tenant-agnostic content, and a per-tenant, per-user namespace keyed with the tenant ID baked into the prefix itself for anything touching account state.

Cache poisoning is another risk: we only write to the cache from responses that passed schema validation and the injection-pattern classifier, we stamp every cache entry with its source traceId, and we encourage routine purging of unknown caches.

Context compaction

Uncapped conversation or agent trajectories allowed N to grow continuously, expanding the cost curve and causing latency degradation. We capped N by implementing a sliding window that summarizes historical context via a small, ultra-cheap model. For Concierge, we kept the last 3 turns verbatim while condensing older turns into a rolling metadata block.

For Pathfinder, when the debugging steps exceeded 4 runs, we trimmed and summarized the oldest tool execution outputs into a compact chronological timeline, transforming the open-ended quadratic cost explosion into a predictable, bounded window.

def compact_session_history(history_steps: List[Dict[str, Any]], keep_recent: int = 3) ->     List[Dict[str, Any]];
    """Flattens older history into a cheap summary block, preserving recent context."""
    if len(history_steps) <= keep_recent:
        return history_steps
    old_steps = history_steps[:-keep_recent]
    recent_steps = history_steps[-keep_recent:]
    
# Compress the old history using a fast, low-cost utility model
try:
historical_summary = summarize_with_utility_model(old_steps)
# Note: Anthropic prohibits 'system' roles in the messages array.
# Using 'assistant' ensures cross-provider compatibility.
return [{"role": "assistant", "content": f"[System Context: Summary of prior steps: {historical_summary}]"}] + recent_steps
except Exception as e:
print(f"History compression failed: {e}")
# Fallback: Return the uncompressed history to gracefully degrade
return history_steps

Model cascading

Directing every single operation to an expensive frontier model represents massive overprovisioning for mundane tasks. We integrated LiteLLM as an internal routing gateway to implement model cascading, routing every request to the lowest-cost model capable of completing the task.

“Directing every single operation to an expensive frontier model represents massive overprovisioning for mundane tasks.”

# litellm_config.yaml
model_list:
  - model_name: fast-path
    litellm_params:
      model: openai/mistral-support-ft
      api_base: http://vllm-internal:8000/v1
  - model_name: frontier-path
    litellm_params:
      model: anthropic/claude-opus-4

Simple, repetitive tasks are routed to a lower model, which offloads 70% of Concierge chats from the frontier model. For Pathfinder, we broke the agent loop down into separate sub-tasks: high-level planning, tool selection, and code-patch synthesis remained with the frontier model, while mechanical, text-heavy operations, such as log parsing, regex extraction, and error-string formatting, were offloaded to the lower models. This hybrid orchestration reduced Pathfinder’s token costs by more than 50%.

What’s next

The transformation of Concierge and Pathfinder proves a fundamental truth about production AI. You cannot achieve scale by simply relying on the natural language capabilities of a frontier model. You must engineer the system around it. By shifting your focus from naive token reduction to maximizing system resource utilization, we reclaimed absolute control over the infrastructure.

“Efficiency in the era of gen AI is not defined by how cheaply you can operate but by how densely you can pack information.”

Efficiency in the era of gen AI is not defined by how cheaply you can operate but by how densely you can pack information, how quickly you can serve it, and how reliably you can parse the output. The architectural decisions detailed here represent more than just a token optimization strategy; they are a required foundation for building high-throughput, battle-tested, and resilient AI systems at scale.

The post The systems guide to production token optimization appeared first on The New Stack.

We instrumented the full path from pod creation to first inference response on a GPU node running a 70B-class model. Eight minutes. Six sequential phases. We expected one bottleneck. We found six, and which one dominates depends on model size.

For a 64 GB model, 65% of the startup time is spent recompiling CUDA kernels that produce identical output every time. For a 203 GB model, 92% of the time is spent downloading weights from S3 through a calling pattern that leaves 98% of available bandwidth idle. Both are fixable with configuration changes. Neither is fixed by default.

“Eight minutes. Six sequential phases. We expected one bottleneck. We found six.”

We define time to first token served (TTFTS) as the wall-clock duration from pod creation to the first inference response leaving the GPU. Not time to first token (TTFT), which measures per-request latency once the model is warm. TTFTS is the one-time startup tax. TTFT begins where TTFTS ends.

Here’s what we achieved:

The warm-node row is what you pay on every pod restart: scale-up events, rolling updates, OOM recoveries. That’s the 80-93% win, and it requires only configuration changes. The cold-node row includes ~2 minutes of fixed infrastructure cost (node provisioning and framework initialization) that no application-layer optimization can remove. The rest is avoidable waste that we eliminated through platform and configuration fixes. The warm-node optimizations are environment variables and a volume mount that work on any Kubernetes cluster. The cold-node optimizations require EKS Auto Mode, which comes pre-configured with pre-compiled NVIDIA drivers, SOCI (Seekable OCI) parallel image pull, and NVMe instance store mounting.

All model startup measurements were taken on p5.48xlarge instances running Amazon EKS Auto Mode, with S3 traffic routed directly (bypassing the NAT Gateway) and container images in a private Amazon ECR repository (same region as compute). Model startup improvement ratios (80-93%) hold consistently across instance types (validated on P-family and G-family). Cold-node times vary with network bandwidth and CPU count. For the weights loading and compilation cache configuration, see Accelerate model loading on Amazon EKS.

The Kubernetes ecosystem has made real progress on the inference stack in 2026. OCI image volumes are now stable for model delivery. Dynamic Resource Allocation (DRA) gives GPUs structured attributes instead of opaque integer counts and provides flexibility in allocating GPUs to workloads. Gateway API has inference-aware routing extensions. But none of these primitives address the full cold-start stack: the six layers between “pod pending” and “first token served,” each with its own bottleneck and its own fix.

The six layers of cold start

When a new inference pod starts on a freshly provisioned GPU node, it passes through six distinct phases before serving its first request:

1. Node provisioning. Karpenter launches an EC2 instance, boots it, and registers it with the Kubernetes API server (~60-90s).
2. GPU driver initialization. The driver kernel module must load and expose accelerator devices.
3. Container image pull. The inference engine image (8-12 GB compressed) must be transferred to the node and extracted.
4. Model weights download. The model files must stream from object storage into GPU memory.
5. GPU kernel compilation. torch.compile traces the model graph and generates optimized CUDA kernels.
6. Engine initialization. CUDA graph capture, KV cache profiling, and HTTP server startup (30-120s depending on whether compilation is cached).

Each layer has a different bottleneck, a different fix, and a different owner.

Which layer dominates depends on model size

Before diving into each layer, one finding shaped every decision we made: the bottleneck is not fixed.

We instrumented the model startup path (layers 4 and 5) and measured each phase independently for two model sizes:

64 GB model (Qwen3.6-35B-A3B):

• Weights loading: ~29s (35% of model startup)
• torch.compile: ~53s (65% of model startup)

203 GB model (Llama-4-Scout, TP=4 where TP is tensor parallelism, splitting the model across GPUs):

• Weights loading: ~423s (92% of model startup)
• torch.compile: ~34s (8% of model startup)

For models under ~100 GB, compilation dominates. For larger models, network transfer dominates. torch.compile time stays roughly constant (it depends on graph complexity, not parameter count). Weights loading scales linearly with file size.

“For models under ~100 GB, compilation dominates. For larger models, network transfer dominates.”

This means any single-layer optimization has a ceiling.

Layer 1: Node provisioning

On EKS Auto Mode and Karpenter-managed clusters, node provisioning takes approximately 60-90 seconds for accelerated instances from pod pending to node Ready. Karpenter calls the EC2 Fleet API directly and reacts to pending pods within seconds, keeping provisioning at the EC2 launch floor.

Layer 2: GPU driver initialization

The NVIDIA GPU Operator in its default configuration adds 2-3 minutes to node boot while it compiles the driver kernel module from source. This cost repeats on every new node.

When the platform controls the full stack (OS image, kernel version, driver version, boot sequence) it can pre-compile driver kernel modules at image build time. The node boots, runs modprobe to load an already-compiled .ko file, and the GPU is ready in seconds.

This matters more now than it used to. Blackwell-architecture GPUs (G7, G7e instances) require NVIDIA’s open-source kernel modules exclusively. Older Maxwell/Pascal/Volta GPUs can only run proprietary modules. A cluster with both legacy and next-gen GPU nodes needs different drivers, different AMIs, different upgrade cycles. A managed platform that pre-compiles the correct module per instance family eliminates this complexity.

On EKS Auto Mode, the GPU driver loads in seconds (pre-compiled at image build time), compared to the 2-3 minutes a runtime-compilation approach requires.

Layer 3: Container image pull

A production vLLM or SGLang inference image is typically 8-12 GB compressed. Standard containerd pulls layers sequentially, decompresses them one by one in memory, and writes them to disk. At this size, sequential pull takes 2-4 minutes on a cold node depending on instance type and available CPU cores. For larger custom images (30-50 GB compressed), containerd can run out of memory entirely during decompression.

EKS Auto Mode uses SOCI’s parallel pull mode, which replaces containerd’s default snapshotter. The SOCI snapshotter downloads layer chunks concurrently via HTTP range requests and writes each chunk directly to its target byte position on disk (no in-memory ordering buffer). Decompression runs in parallel across all available CPU cores.

Pull time is bottlenecked by CPU-bound decompression, not network bandwidth. We confirmed this directly: a p4d.24xlarge with 400 Gbps networking achieved only ~1 Gbps effective pull throughput because CPU decompression was the constraint. On instances with capable, current-generation CPUs, SOCI parallel pull reduces image pull time from 2-4 minutes to 30-60 seconds. The dominant factor is per-core decompression throughput, which depends on CPU generation and instruction-set support, more than raw core count. A newer CPU with fewer cores can outperform an older one with more.

For a deeper look at how bounded-memory parallel pull handles images exceeding 30 GB without OOM, see Bounded-Memory Parallel Image Pulling for Large Container Images.

Layer 4: Model weights download

The obvious optimization for weights loading: more parallel connections. Split the model files into small chunks, download them concurrently, saturate the network pipe.

We tested it on p5.48xlarge with the 64 GB model streaming from same-region S3. The results were counterintuitive:

256 parallel connections provided no benefit over 17. The only failure mode was 8 GB chunks (exceeding shard file size), which caused a 56% regression.

Why? Because the open-source Run:ai Model Streamer (integrated into vLLM and SGLang) processes S3 range requests sequentially within each worker thread. A worker assigned to a 3.9 GB shard file downloads its byte-range requests one after another on a single connection. The parallelism comes from running multiple workers on different files, not from splitting one file into more pieces.

We settled on 4 GB chunks matching typical SafeTensors shard size (3-5 GB per file) with an aggressive timeout-and-retry for slow requests. S3 GET latency has a measurable long tail: in our testing, a meaningful fraction of requests took 2-3x longer than median, and a single stalled connection holds up the entire model load. Rather than wait, we kill stalled connections after a few seconds below a speed threshold and retry on a fresh connection. This follows S3’s own performance guidance.

For the 203 GB model, these config-only changes reduced weights loading from 423 seconds to 25 seconds (94% improvement). For the 64 GB model, from 29 seconds to 12 seconds. No code modifications, just environment variables. The tuning consists of three settings: chunk size aligned to shard file boundaries (eliminating the serial sub-request problem), a minimum-speed threshold that kills and retries stalled S3 connections, and explicit concurrency matching the number of shard files per tensor-parallel rank.

Layer 5: GPU kernel compilation

Every time a vLLM or SGLang pod starts, PyTorch traces the model’s computation graph and compiles it to optimized CUDA kernels. This takes 34-53 seconds depending on model architecture. The output is identical every time for the same model, GPU type, and tensor-parallel configuration.

And Kubernetes throws it away on every pod restart. Pods use ephemeral storage by default. When a pod terminates, its local filesystem is destroyed. The next pod recompiles from scratch.

“The output is identical every time for the same model, GPU type, and tensor-parallel configuration. And Kubernetes throws it away on every pod restart.”

Point the torch.compile cache directory at local NVMe instance store. GPU instances ship with NVMe that EKS Auto Mode mounts automatically. First pod compiles and writes ~15-30 MB of cached kernels. The second pod on the same node loads pre-compiled binaries in 4-6 seconds. One volume mount and environment variables.

The cache is safe because the compiled artifacts are deterministic: same model architecture + GPU architecture + tensor-parallel degree + PyTorch version equals valid cache. An image update or hardware change triggers exactly one recompilation.

torch.compile time is hardware independent. The same model compiles in ~52 seconds whether running on H100 or A100. The cache hit (4-6 seconds) is equally consistent across GPU types. This means the optimization works identically regardless of instance type.

Layer 6: Engine initialization

After weights are loaded and kernels compiled, the inference engine must capture CUDA execution graphs and profile KV cache memory. With compiled kernels cached, this completes in 30-45 seconds. Without cache, graph capture triggers additional JIT compilation and takes 60-120 seconds.

This is why the torch.compile cache has an outsized impact: it accelerates not just layer 5 but also layer 6. Cached compilation reduces a 2-3-minute combined phase to a 35-50-second combined phase.

Framework initialization (Python interpreter startup and PyTorch import) adds tens of seconds of fixed overhead that cannot be reduced through configuration.

The compounding effect

The six layers compound. Platform fixes (layers 1-3) eliminate 4-8 minutes of overhead: pre-compiled drivers replace 2-3 minutes of runtime compilation, parallel pull reduces image transfer time from 2-4 minutes to 30-60 seconds, and Karpenter keeps node provisioning to its hardware minimum. Configuration changes (layers 4-5) cut the remaining model startup by 80-93%. Engine initialization (layer 6) drops from 60-120 seconds to 30-45 seconds once the compile cache is warm. Together, cold-node TTFTS drops from 8-15 minutes to approximately 5 minutes.

64 GB model (Qwen3.6-35B-A3B), TP=2:

203 GB model (Llama-4-Scout), TP=4:

The warm-node subsequent pod number is what matters most for production. It’s what you pay on every pod restart. The 80-93% reduction is consistent across instance types because the optimizations target software bottlenecks (calling patterns, redundant compilation), not hardware limits.

The cost of cold starts at scale

Why does any of this matter? Because GPU nodes are expensive and inference traffic is bursty.

A single p5.48xlarge costs $55/hour on-demand. Even G-family instances commonly used for inference cost $10-20/hour. Every minute of cold start is GPU time you’re paying for but not using. If your autoscaler needs 8+ minutes to bring up new capacity, you must over-provision (burn money on idle GPUs) or accept latency spikes during traffic surges.

“Every minute of cold start is GPU time you’re paying for but not using.”

When model startup drops to 16-32 seconds on warm nodes, the calculus changes. You can scale more aggressively, keep fewer buffer nodes, and respond to traffic spikes without multi-minute startup delays.

What we learned

1. Decompose before optimizing. For 64 GB models, torch.compile dominates (65%). For 203 GB models, S3 loading dominates (92%). Without measuring each phase independently, we would have optimized the wrong layer.
2. The bottleneck flips with model size. torch.compile time is roughly constant across model sizes. Weights loading scales linearly. Every team running inference should know which regime they’re in.
3. “More parallelism” requires understanding the execution model. 256 connections performing sequential work inside each thread is no faster than 17. The bottleneck was the calling pattern, not the concurrency limit.
4. 15-30 MB can save 53 seconds. The most impactful optimization for smaller models was persisting a tiny cache file. Always check whether an expensive computation produces deterministic output before trying to make it faster.
5. Platform-level control enables optimizations that configuration alone cannot achieve. Pre-compiled drivers, default-on parallel image pull, and NVMe auto-mounting are infrastructure-layer decisions that compound upward. Together with the config-only changes at the application layer, these changes reduce cold start time from minutes to seconds.
6. The ecosystem is building the right primitives, but cold start lives between them. OCI image volumes, DRA, inference-aware routing, and local model caches are all real progress. But the compilation bottleneck and S3 tuning gaps sit in spaces that no upstream Kubernetes primitive addresses. Sometimes the highest-impact optimization is a volume mount and two environment variables, not a new API.

For the complete configuration guide, including environment variables, YAML manifests, and instance-specific recommendations, see “Accelerate model loading on Amazon EKS” in the Amazon EKS User Guide.

The post Cut GPU inference cold start from 8 minutes to less than a minute appeared first on The New Stack.

Nvidia has confirmed that it’s agreed to acquire Hugging Face in a mammoth $12.9 billion deal that will bring one of the AI industry’s most important open-model platforms under the auspices of the world’s dominant AI chipmaker — and, with a market cap of well over $5 trillion, the most valuable company on Earth.

When reports first emerged in August that Nvidia was lining up a gargantuan bid for what has often been described as the “GitHub of AI Models,” concerns quickly surfaced over what the deal could mean for the platform’s openness and hardware neutrality. As The New Stack reported at the time, Hugging Face’s value lies partly in giving developers a neutral place to find and deploy open models across Nvidia, AMD, Intel and cloud accelerators — raising the question of what happens if that platform is owned by just one of them.

Keeping Hugging Face open

That, ultimately, is why Nvidia founder, president and CEO Jensen Huang is going to great lengths to allay fears that Hugging Face could become a vehicle for steering developers toward Nvidia’s own hardware and software stack.

“Nvidia compute will not be required to build on or deploy through Hugging Face.”

In the official announcement on Thursday, Huang makes a series of explicit promises around neutrality, noting that Hugging Face would “remain an open platform for the entire AI ecosystem,” continue to support multiple clouds and accelerators, while stressing that “Nvidia compute will not be required to build on or deploy through Hugging Face.”

“Hugging Face will continue to support open source and open weight models from across the ecosystem, from every model builder,” Huang writes. “It will continue to support multi-cloud and multi-accelerator development and deployment, so builders can use the hardware and infrastructure that best fit their work.”

Indeed, it’s clear Nvidia had clocked the concerns around the impending acquisition. The word “open” appears no fewer than 19 times in Huang’s relatively short announcement, underlining just how central that reassurance is to Nvidia’s pitch for the deal.

Huang also points to a recent open letter on open weights that he co-signed alongside executives and researchers from across the AI industry, including those from Hugging Face. The letter argued that open-weight models are critical to broadening access to AI, strengthening competition and giving developers more control over how models are deployed and adapted.

For what it’s worth, Nvidia has been pushing hard into open-weight AI itself, including through its Nemotron models and a broader effort to make frontier-class models easier to run locally. Hugging Face co-founder and CEO Clément Delangue went so far as to call Nvidia the “King of American open-source AI” a few months ago, pointing to its growing collection of public models, datasets and Spaces on Hugging Face.

In the wake of the announcement on Thursday, Delangue doubled down on that position in a fresh post announcing the deal, arguing that Hugging Face has reached the point where keeping open-source AI competitive would require substantially more resources.

“It needs more compute, more support, more collaboration and more visibility.” – Hugging Face CEO Clément Delangue

“10 years after starting Hugging Face, open-source AI is at an inflection point,” Delangue writes on LinkedIn. “Thanks to the community, we’ve shown that it can be a complement, and even an alternative, to closed-source APIs. But for it to happen at larger scale, it needs more compute, more support, more collaboration and more visibility.”

He adds that Nvidia has committed to backing Hugging Face while keeping the platform open, independent and compute-agnostic, with the founders and existing team staying on.

“Together, we think we can make open source the default way to build AI,” Delangue writes, setting out a goal of helping 100 million AI builders “own their intelligence rather than rent it.”

GitHub as a historical precedent

However, there is some historical precedent for taking such assurances with a degree of caution. When Microsoft bought GitHub for $7.5 billion in 2018, it similarly promised that the platform would remain independent and open. GitHub largely retained that openness, though Microsoft’s later use of public GitHub code to help train the proprietary, paid Copilot service sparked a backlash among factions of the open-source community.

In truth, that obvious comparison may actually understate Nvidia’s challenge. In recent analysis for Forbes, technology analyst Janakiram MSV argues that neutrality at Hugging Face has a hardware dimension that GitHub never had to contend with. Hugging Face maintains integrations spanning AWS Trainium and Inferentia, Google TPUs, Intel Gaudi, AMD Instinct and other accelerators. Under Nvidia ownership, continued support for those rival chips becomes a real test of just how neutral the platform will remain in the long run.

Not a done deal yet

As for the acquisition itself, well, it’s not over the line quite yet. In a filing with the US Securities and Exchange Commission (SEC), Nvidia notes that it expects the deal to close in the first half of 2027, subject to customary closing conditions and regulatory approvals. Given Nvidia’s lofty position in AI infrastructure and Hugging Face’s role as a major distribution point for open models, those approvals are unlikely to be a mere formality. Nvidia’s filing also flags the possibility that future regulation around open-source AI could affect Hugging Face’s operations or increase compliance costs.

“Hugging Face [will] continue to permit model makers, developers, and users to upload and download models and datasets of their choosing and to support other silicon vendors.”

Notably, Nvidia also uses its SEC filing to reaffirm its neutrality commitments, stating that under the commitment, “Hugging Face would continue to permit model makers, developers, and users to upload and download models and datasets of their choosing and to support other silicon vendors.”

Of the roughly $12.9 billion headline price, about $11.9 billion will go to Hugging Face stockholders, with up to $1 billion earmarked for equity-based retention awards for employees joining Nvidia.

And while the headline price is usually rounded to $12.9 billion, the actual figure is an oddly specific $12,930,300,000. That’s no accident either, as Hugging Face co-founder Thomas Wolf alludes to in a LinkedIn post. For those still in the dark, 129303 is the decimal Unicode value for the 🤗 emoji, while #129303 is a green color code nodding to Nvidia’s branding.

A neat little Easter egg buried inside what can only be described as one of the biggest AI deals of the year.

The post “Hugging Face will remain an open platform”: Nvidia strikes $12.9B deal for the ‘GitHub of AI’ appeared first on The New Stack.

AI agents are multiplying as corporations adopt the technology in record numbers. Smarter underlying models, better tool use, and improved multi-agent collaboration have pushed agents to evolve beyond impressive demos into practical technology that companies marshal in production environments. But the job’s not finished. 

As companies deploy more agents, more often, and against longer tasks, the plumbing that provides their AI ephemera with the required information is buckling.

Here’s the problem: AI agents are sending waves of queries against company data, creating concurrency issues and exposing just how difficult it can be to ensure a company’s AI-legible information is fresh, served only when relevant, and quickly available.

Join the live conversation: On September 24 at 12 p.m. Eastern/9 a.m. Pacific, Whit Walters, Field CTO and Lead Analyst at GigaOm and author of Defeating the Integration Tax report, joins Bonnie Chase, Director of Product Marketing at Vespa.ai, to discuss what happens when retrieval architecture meets that workload.

And crucially, they will explore in this live conversation what changes when a team rebuilds it as a unified layer instead of a fragmented one.

Register for our free event on September 24

You might be asking yourself: How has this problem not been solved yet? Google famously handles tens of thousands of search queries every second; how difficult can it be to serve agents the information that they need when we’ve solved the human version of the same problem? It’s no small challenge, and it’s why retrieval engineering is a labor category you’ll hear more about in coming quarters.

So, why is the problem worse with AI? Agents don’t ask a single question. They may retrieve data, reason against it, and then go back for more context. That doesn’t sound too complicated, until we recall that companies often stitch multiple systems together to provide their agents with required information. In practice, that means fusing vector databases, ranking tools, and serving layers into a single hybrid retrieval system that serves ever more agentic queries.

Worse, when several agents ping the same cobbled-together architecture at once, relevance drift becomes a real issue. You might do all the work to get your company or team up and running with agents, only to see the effort fail because of stale data, generic answers, or even truncated results as retrieval plumbing stumbles.

Your AI agents can’t scale successfully if they get dumber the more agents you deploy. So join the conversation on September 24, where we’ll break down how you can solve your retrieval engineering woes.

What you’ll take away:

• Why agent workloads create a fundamentally different retrieval challenge than added concurrency alone
• The specific failure modes at agent scale — latency stacking, stale context, relevance drift
• Why fragmented retrieval stacks amplify those failures
• What a unified retrieval architecture looks like in practice

The post Want to scale AI agents without breaking anything? Retrieval engineering is the answer. appeared first on The New Stack.

Microsoft AI on Thursday released MAI-Transcribe-2, a speech-recognition model the company says is faster, more accurate, and cheaper than anything OpenAI, Google, or ElevenLabs currently sells. Then it priced the thing at 10 cents per hour of audio.

That figure deserves a pause. When Microsoft AI shipped the first model in this line just five months ago, it charged $0.36 an hour. Thursday's early-bird price cuts that by roughly 72%. For an enterprise processing 100,000 hours of call-center audio a year — a modest volume for a large bank or telecom — the bill drops from $36,000 to $10,000. At that level, transcription stops being a line item anyone argues about.

The release arrives as Microsoft executes a strategy that would have seemed implausible two years ago: building its own frontier-class models one modality at a time, then steadily swapping them into products that once ran on OpenAI's technology. Transcription is the modality where that plan has moved fastest, and MAI-Transcribe-2 is its clearest proof point yet. It also offers a preview of how the world's most valuable software company intends to compete in AI without depending on the partner it spent $13 billion to cultivate.

What MAI-Transcribe-2 does and why the feature list matters to enterprise buyers

The model transcribes audio in 60 languages, up from 43 in June's MAI-Transcribe-1.5 and 25 in April's original release. It runs on Microsoft Foundry, the company's model marketplace for developers, and in MAI Playground, its testing environment. Microsoft says it built the model for the messy audio that real businesses generate — background noise, low-quality recordings, overlapping speech — rather than clean studio conditions.

More important than the language count is what Microsoft has bundled into the base product. Speaker diarization sorts out who said what in a multi-person recording, which is the difference between a wall of text and a usable meeting transcript. Word-level timestamps attach a precise time marker to every word, enabling search, editing, and alignment with video. Keyword biasing lets developers feed the model a list of drug names, product codes, or employee names so it stops mangling domain jargon. Automatic language identification means users no longer have to declare the language in advance.

Two features stand out for their specificity. A configurable output style offers a "verbatim" mode that preserves every "um," false start, and stutter for compliance and legal teams, alongside a "clean" mode that strips fillers for readable captions and notes. And code switching handles conversations that drift between languages mid-sentence; Microsoft explicitly names Hinglish and Spanglish, a nod to the Indian and U.S. Hispanic markets where a single customer-service call might toggle languages a dozen times. Specialty vendors have historically charged premiums for each of these capabilities. Microsoft is including all of them for a dime.

How to read Microsoft's FLEURS and Artificial Analysis benchmark claims

Microsoft makes three performance claims, each resting on a different measuring stick, and technical buyers should understand what each one captures and what it misses.

The first is that MAI-Transcribe-2 ranks number one on FLEURS across 60 languages with an average word error rate of 5.2%. FLEURS is a benchmark Google researchers published in 2022, built from native speakers reading roughly 2,000 sentences in each of 102 languages — about 12 hours of speech per language. It is the standard yardstick for multilingual speech recognition because it lets you compare a model's Swahili against its Swedish on identical content. Word error rate, its metric, simply counts substitutions, insertions, and deletions against a human reference; 5.2% means roughly one word in 20 is wrong. But FLEURS is read speech, not conversation, and Microsoft's average has actually risen from the 3.7% it reported for MAI-Transcribe-1.5 in June. That almost certainly reflects broader coverage rather than regression — averaging across 60 languages instead of 43 means folding in low-resource languages where every model struggles — but buyers should request the per-language breakdown.

The second claim is that the model ranks second on the Artificial Analysis word-error-rate leaderboard and defines that firm's accuracy-latency Pareto frontier. Artificial Analysis is an independent benchmarker that tests models through their public APIs, measuring what a customer actually gets. Its index blends simulated agent conversations, European Parliament speeches, and corporate earnings calls, weighting heavily toward English business speech. In June, the firm ranked MAI-Transcribe-1.5 third at 2.4% WER, behind Alibaba's Fun-Realtime-ASR-preview and ElevenLabs' Scribe v2, while calling it the fastest model in the top 10. Climbing to second suggests Microsoft has cleared ElevenLabs. "Pareto frontier" is the phrase practitioners should note: it means no rival beats the model on accuracy without being slower, and none beats it on speed without being less accurate.

The third claim is raw speed — 10 times faster than OpenAI's GPT-Transcribe, seven times faster than ElevenLabs' Scribe v2, five times faster than Google's Gemini 3.5 Transcribe, per Artificial Analysis evaluations. In batch transcription, speed matters less because anyone is waiting and more because throughput is cost. A model running at 300 times real-time needs a fraction of the GPU-hours of one running at 30 times. That efficiency is what lets Microsoft charge a dime and, presumably, still make money.

Three speech models in five months: inside Microsoft AI's rapid release cadence

The pace is the story within the story. On April 2, MAI-Transcribe-1 launched with 25 languages at $0.36 per hour. On June 2, MAI-Transcribe-1.5 arrived with 43 languages, keyword biasing, and a third-place ranking on Artificial Analysis. Today, MAI-Transcribe-2 shipped with 60 languages, diarization, timestamps, code switching, a second-place ranking, and a price of $0.10.

Three releases in five months, each expanding language coverage by roughly 40% while adding features competitors gate behind premium tiers. That cadence is characteristic of a team that has settled on a stable architecture and is now turning the crank on data and scale — the phase where speech models tend to improve quickly and predictably. It is also the cadence of a company that intends to make transcription a commodity before anyone else can.

The organizational bet behind that speed is one Mustafa Suleyman, Microsoft AI's chief executive, described to The Verge in April. He credited the first model to "a small, focused 10-person team" that had been "liberated from any of the bureaucracy," with a larger surrounding group handling vendor management and data acquisition.

He also told The Verge the model ran at "half the GPU cost of the other state-of-the-art models," calling it "a huge cost-saving" for Microsoft. Meta, Amazon, Google, and Anthropic have all experimented with similar flattened structures, The Verge noted. Microsoft's transcription line is the most visible test yet of whether the approach produces commercial results rather than research papers.

Why Microsoft is building its own AI models despite its $13 billion OpenAI bet

Microsoft has invested more than $13 billion in OpenAI, and hosts OpenAI's models across Azure, Office, and Copilot. For most of the past four years, the obvious question about any Microsoft-built model has been: why bother? The answer has sharpened over the past year, and it begins with independence.

When Microsoft hired Suleyman from Inflection AI in March 2024, along with most of Inflection's staff, Salesforce CEO Marc Benioff read it as a declaration of intent. "Microsoft is building their own AI and I don't think Microsoft will use OpenAI in the future. They'll have their own frontier models," Benioff told CNBC in January 2025. "That's why they hired Mustafa Suleyman." Benioff had his own motives — Salesforce competes with Microsoft and invests in Anthropic — but events have largely borne him out.

In October 2025, Microsoft and OpenAI restructured their partnership in a deal that, per Microsoft's own announcement, allowed Microsoft to "independently pursue AGI alone or in partnership with third parties" for the first time. Suleyman told The Verge that renegotiation "unlocked [Microsoft's] ability to pursue superintelligence," and Microsoft announced its MAI Superintelligence team weeks later. In April 2026, the companies amended the deal again, ending Microsoft's exclusive access to OpenAI's models and eliminating Microsoft's revenue-share payments, according to reports at the time. Each amendment loosened the tie. Each one was followed by more MAI models.

How Microsoft's in-house models are cutting costs across Teams, Word, and Excel

The second half of the answer is margin. Every prompt Microsoft routes to an OpenAI model carries a cost. Every prompt it routes to its own model on its own GPUs carries a smaller one. In July, Bloomberg reported that Microsoft had begun using MAI models to answer a portion of user prompts in Word and Excel — products it had previously advertised as powered by OpenAI and Anthropic. TechCrunch framed the shift as part of a broader industry pullback on AI spending, with Amazon, Uber, Meta, and Accenture all reportedly trimming.

Transcription is the natural first target for this substitution because the problem is bounded and the metric is objective. Microsoft owns Teams, which generates an enormous volume of meeting audio. It owns Nuance, whose clinical documentation business runs on speech recognition. It owns the Azure speech services that thousands of enterprises already call. Every one of those workloads is a candidate to move onto MAI-Transcribe-2, and every hour that moves is an hour Microsoft no longer pays anyone else for.

Suleyman has been unusually candid that this is the point. Superintelligence, he told The Verge in April, "is really about, 'Are these models capable of delivering product value for the millions of enterprises that depend on us to deliver world-class language models?'" Whatever one thinks of applying the word "superintelligence" to a transcription API, the commercial logic is plain: build the capability once, deploy it across a dozen products, and stop writing checks to a partner that is increasingly a competitor.

MAI-Transcribe-2 vs. OpenAI, Google, and ElevenLabs: the competitive picture

Microsoft's release names four rivals: OpenAI's GPT-Transcribe, Google's Gemini 3.5 Transcribe, OpenAI's older Whisper V3-Large, and ElevenLabs' Scribe v2. It does not mention Deepgram, AssemblyAI, Speechmatics, or Rev — the specialists that have sold transcription to enterprises for a decade. Microsoft is positioning against the frontier labs, not the incumbents.

That framing is partly marketing and partly true. The frontier labs have treated speech as a checkbox feature of broader platforms, priced accordingly, and a dedicated model that beats them on speed by five to 10 times while matching their accuracy is a genuine differentiator. But the specialists will feel the price pressure most acutely. At $0.10 an hour, Microsoft is pricing at or below where many of them sell high-volume enterprise contracts, and it is bundling diarization, timestamps, and 60 languages into the base rate. The specialists' remaining moat is domain depth — medical vocabularies, legal formatting, industry-specific integrations — and Microsoft's keyword biasing feature is aimed squarely at it.

The one competitor Microsoft conspicuously does not claim to beat on accuracy is Alibaba, whose models have posted leading numbers on independent leaderboards for much of 2026. TechCrunch reported in July that some U.S. companies had begun evaluating Chinese models as cheaper alternatives despite security concerns. Microsoft's pitch to those buyers is implicit but unmistakable: comparable accuracy, faster inference, lower price, and a vendor your compliance team already trusts.

The questions technical decision makers should ask before switching transcription vendors

For all its specificity on benchmarks, the release leaves several practical questions open. The first is duration: Microsoft calls $0.10 per hour a launch offer without naming an end date or a standard rate, and anyone building a cost model should get both in writing. The second is streaming. The release emphasizes batch throughput and long-form audio but says nothing about real-time transcription, which voice agents and live captioning require. Artificial Analysis maintains a separate streaming leaderboard, and Microsoft's silence on it is notable.

The third is per-language accuracy. A 5.2% average across 60 languages could mean 3% on major languages and 12% on low-resource ones, so buyers with specific needs should test those languages directly. The fourth is diarization quality. Word error rate does not measure speaker attribution; a transcript can have near-perfect WER and still assign every other sentence to the wrong person. The release offers no diarization error rate or comparable metric.

The fifth is data handling. Enterprise transcription touches medical records, legal privilege, and financial disclosures, and the release says nothing about data residency, retention, or whether audio submitted to Foundry feeds future training. Microsoft's April announcements described training data as a mix of human-curated recordings, contractor-recorded noisy audio, and "vast amounts of data from the open web," per The Verge — a description that should prompt pointed questions from regulated industries. None of these gaps is unusual for a launch announcement, but they are exactly the questions that separate a leaderboard win from a production deployment.

What Microsoft's speech model strategy reveals about its broader AI ambitions

Step back from the speech-recognition details and a pattern emerges that extends well beyond transcription. Microsoft's AI unit now ships models for images, voice, transcription, code, reasoning, and cybersecurity. At Build in June, it announced seven new MAI models in a singlekeynote. Each follows the same playbook: target a well-defined modality, optimize aggressively for inference cost, price below the frontier labs, distribute through Foundry, and quietly swap the model into Microsoft's own products.

This is not an attempt to build one model that beats GPT or Gemini at everything. It is an attempt to build a portfolio of specialized models that, in aggregate, let Microsoft serve most of its enterprise workloads without paying anyone else — and to sell the surplus capacity to everyone else at prices the specialists cannot match. Transcription happened to be the first modality where the approach fully matured, but the release notes for MAI-Transcribe-2 read less like a product announcement than a template.

Suleyman has spent two years talking about "humanist superintelligence" and AI assistants that are "accountable to them, on their side." The vocabulary is lofty. The execution is a spreadsheet. Five months ago, Microsoft charged 36 cents to turn an hour of speech into text. On Thursday it charged a dime, threw in six features its rivals sell separately, and claimed the top spot on the industry's standard multilingual benchmark. The company that spent $13 billion learning what frontier AI costs has decided it would rather own the factory than rent the output — and now it is selling the output for less than the rent.

MAI-Transcribe-2 is available now through Microsoft Foundry and MAI Playground.

A 438-billion-parameter reasoning model isn’t an obvious choice when speed is a priority. Multiverse Computing is betting that compression can make it one. 

On Wednesday, the Spanish company launched Quasar 438B, its first large-scale model and a system designed specifically for coding and enterprise agents. Quasar achieves a score of 43 on Artificial Analysis’ Intelligence Index and 69.3 on Terminal-Bench v2.1, while Artificial Analysis currently records its output speed at approximately 183 tokens per second. Multiverse is positioning Quasar as the European model with the highest score on the Intelligence Index, outperforming Mistral Medium 3.5 (30) and NVIDIA Nemotron 3 Ultra (38).

Multiverse is betting that a 438B-parameter model can be fast and inexpensive enough for agents that repeatedly reason, call tools, and check the results.

That puts Quasar in an interesting middle ground. It doesn’t match the strongest models on coding performance, but Multiverse is betting that a 438B-parameter model can be fast and inexpensive enough for agents that repeatedly reason, call tools, and check the results.

The model features a context window of one million tokens, is available in English and Spanish, and can be accessed via the Multiverse CompactifAI API.

Compression claims, missing details 

Multiverse built CompactifAI to shrink large AI models so they need less memory and compute to run. The company says it can reduce model size by 80% to 95% with only a small loss in accuracy, but it hasn’t disclosed how much Quasar was compressed or which model it started with.

In July, Multiverse announced a $570 million Series C to expand its library of compressed models and commercialize the technology. Quasar is the biggest test of that approach so far.

Multiverse hasn’t said what hardware is required to run Quasar or how much the compression reduces its memory and compute needs. That matters for agents, which may repeatedly call the model and other tools before finishing a task.

Multiverse hasn’t said what hardware it takes to run Quasar or how much the compression cuts its memory and compute needs.

Coding benchmarks show tradeoffs 

In Multiverse’s comparison, Quasar’s Terminal-Bench v2.1 score of 69.3 places it ahead of Mistral Medium 3.5 but still well behind the best frontier systems, while Claude Opus 5 achieves the highest score on that benchmark at 89.1.

Multiverse is pitching Quasar for software engineering, technical copilots, research, and workflow automation. Its 1-million-token context window gives agents room to work with large codebases and hold onto information as a task progresses, although processing more context also requires more compute. That can be especially important in coding, where code that passes every test can still trip up the next AI agent if it loses track of what came before.

Agent latency beyond throughput 

Artificial Analysis found that Quasar starts responding in about 1.1 seconds and can produce a 500-token response, including reasoning, in around 15.3 seconds. Those numbers are fast, but an agent also has to wait for tools, process growing context, and make repeated model calls over the course of a task. The agent tooling layer itself is still catching up to what these models need.

Those numbers are fast, but an agent also has to wait for tools, process growing context, and make repeated model calls over the course of a task.

Proprietary model, open questions 

Quasar is proprietary and only available through Multiverse’s API, so developers can’t inspect the weights or run it on their own hardware. For now, that also makes it difficult to know whether the speed Multiverse is reporting will carry over to everyday agent use.

Quasar also arrives as European AI companies are building more of their own model and compute infrastructure instead of relying on U.S. hyperscalers. Multiverse is taking a different route, using compression to make a 400B-plus model cheaper and faster to run. The next step is to see how that holds up against benchmarks.

The post Multiverse says its 438B model is fast enough for AI agents. The benchmarks tell a more complicated story. appeared first on The New Stack.

Thanks to increasingly capable models, some parts of your business are getting faster, more capable, and more productive every month. These teams are using artificial intelligence to compress timelines, surface insights, and automate work that has historically been time-consuming and tedious.

Meanwhile, other functions just down the hall are still waiting for a formal rollout, a governance approval, or someone to tell them what to do and how to start. The gap between the AI haves and have-nots in your organization is widening, and addressing it requires a new operating model.

Your teams need more support

When leaders notice the uneven distribution of capability across their business, the instinct is to treat it as a tooling problem. They push to get everyone access to the same platforms, provide general-use training, and hire some specialists to slot into IT.

But access is table stakes. It’s a good start, but it won’t get you to strong organizational adoption.

Harvard Business School reports that workers using these tools completed tasks 25% faster and produced results rated more than 40% higher in quality. But the same study also found that performance declined when people used the tools without understanding where they applied and where they didn’t. Fluency, not just access, drives results.

Departmental leaders need guidance on how to apply capabilities in the context of their day-to-day work. Without that knowledge, they can’t ask the right questions.

“Performance declined when people used the tools without understanding where they applied and where they didn’t. Fluency, not just access, drives results.”

Teams playing catch-up tend to focus on how to inject new tools into existing workflows, when they should be thinking about re-engineering processes entirely. They’re focused on evolution in a world undergoing revolution.

Reimagining a process also requires stepping back from it, which is easier said than done. Here’s how it plays out in practice:

An SDR team comes to IT with a specific, bounded ask: “improve our sales lead routing.” Completely reasonable. But only when someone from IT, with visibility across the broader system, dives into the problem does the real opportunity surface. The data pipeline supporting lead routing is unnecessarily complex. With the right support, the conversation shifts to overhauling the entire pipeline and opens the door to fully agentic lead follow-ups.

Departmental leaders don’t lack ambition but throwing a software license and Slack channel at them won’t build the right kind of adoption. Technical support and strategic guidance are required to reimagine work from first principles.

AI fluency must be a structural consideration

The typical pattern puts a centralized team in charge of taking requirements, interpreting them in isolation, and delivering capabilities to departments months later. This model can’t keep pace when AI capabilities launch weekly.

A more effective approach pairs a central “hub” that owns platform strategy, governance, and reusable patterns with AI engineers embedded directly inside business departments. AI engineers serve as “spokes” inside departments, helping them identify vertical use cases day-to-day and delivering the cross-functional visibility needed to make a real impact. The AI engineer who solved a problem for finance can share the pattern with someone facing the same challenge in operations.

“A more effective approach pairs a central “hub” with AI engineers embedded directly inside business departments.”

In a department just getting started, the embedded AI engineer is the primary technical capability: scouting, prototyping, building. In a more mature department, they shift toward enablement, feeding patterns back to the “hub” and helping teams navigate AI without getting buried in process. Over time, departments will organically become AI-fluent as they learn from the engineers.

Make fluency your advantage

The right operating model drives how a function actually works, and strong fluency strengthens processes and institutional knowledge, so outcomes improve over time. As the flywheel builds, each problem solved raises the ceiling of what your team can do independently. 

McKinsey finds that the right workflow redesign is the single biggest factor in whether an enterprise sees meaningful bottom-line impact. Knowing what to redesign depends on how your teams understand and work with AI.

Everyone is adopting AI capabilities. The question now is whether your operating model helps your teams see the best path forward for applying them. If it doesn’t, that’s the gap to close first.

The post Your organization prioritized AI adoption, but you actually need AI fluency. appeared first on The New Stack.

Perplexity wants its AI agents to use more of the computing power already sitting inside a Mac. The company launched Hybrid Compute on Tuesday, a new feature that lets Perplexity Computer move parts of the same task between powerful cloud models and smaller models running directly on Apple silicon.

The timing is significant. Hybrid Compute arrives on the first day of John Ternus’ tenure as Apple’s CEO. Ternus, who previously led Apple’s hardware engineering organization and played a key role in the company’s transition from Intel processors to its own Apple silicon, succeeds Tim Cook after 15 years as CEO. Perplexity is now betting that same hardware can become part of the infrastructure behind autonomous AI agents.

The task starts in the cloud. When a step involves sensitive information, Computer can move that part of the job to a model running on the Mac without having to start over.

When a step involves sensitive information, Computer can move that part of the job to a model running on the Mac without starting over.

Privacy Gate checks first

A Perplexity-trained Privacy Gate runs locally on the Mac and looks for sensitive information such as names, addresses, account numbers, and secrets. When it flags something, the user can decide whether that part of the task should stay on the device.

Users can review what the system wants to keep local before work begins and ensure it hasn’t missed anything they don’t want sent to the cloud. They can also choose which model handles the local work.

At launch, users can choose between Gemma E4B, Qwen3.6 35B-A3B, and a version of Qwen3.6 35B that Perplexity post-trained itself, with more models planned for later. Perplexity also handles the installation through the desktop app, so users don’t have to open a terminal or set up the model themselves.

Local tokens cost nothing

Once a task is running, the app displays local CPU, GPU, and memory utilization, as well as the number of tokens consumed. Users aren’t charged for tokens generated by models running locally on their Mac. That gives Perplexity another reason to move work onto the device beyond privacy, since running models locally can also cut inference costs. (The question of who bears inference costs and when is becoming a competitive issue across the industry.)

There is a compromise. Perplexity uses its most capable models in the cloud, while smaller models handle local work, so keeping more of a task on the Mac can improve privacy and cut costs at the expense of some capability. Hybrid Compute leaves that choice to the user.

Perplexity uses its most capable models in the cloud, while smaller models handle local work, so keeping more of a task on the Mac can improve privacy and cut costs at the expense of some capability.

Context crosses the boundary

The trickier bit is what happens to context when part of a task moves onto the Mac. The cloud model still needs to know enough about what happened locally to continue the job, without getting access to the private information that was supposed to stay there.

Perplexity says Computer can move a step from the cloud to a local model without restarting the task or losing context, and ultimately combine the cloud and local work into a single result. It does not detail in its announcement exactly what context is passed between those environments or how information produced by the local subagent is filtered before it returns to the broader workflow.

The local subagent can work with private files and data and take actions on the Mac. Users can also start a task on an iPhone and hand off local work to their Mac without having to start over.

For enterprise customers, Perplexity adds company-wide rules for what stays local and a record of what leaves each device, bringing the same governance questions facing AI agents down to the device level.

DGX Spark starts local

Hybrid Compute reverses the approach Perplexity introduced for Nvidia’s DGX Spark last week. DGX Spark starts locally and reaches out to frontier cloud models only with permission, while the Mac version starts in the cloud and moves work onto the device when needed.

In both cases, the agent harness decides where each part of the job runs. That orchestration — deciding which tools and context an agent actually needs — is becoming an increasingly difficult engineering problem as agents gain access to more systems and data.

Apple silicon becomes part of the agent stack

Running this much of an agent locally still requires a fairly powerful Mac. Perplexity recommends at least 32GB of unified memory, Apple silicon, and macOS 15. The feature is available to Pro and Max subscribers as well as enterprise customers.

Those requirements show the limits of local AI at the moment. Smaller models can run on plenty of Macs, but giving an agent enough compute to handle meaningful work still requires relatively high-end hardware.

That will likely change as Macs get better at running larger models. For Ternus, the bigger question is how much of the AI work now happening in the cloud will eventually move onto the machines Apple sells.

For Ternus, the bigger question is how much of the AI work now happening in the cloud will eventually move onto the machines Apple sells.

The post Your Mac is now part of Perplexity’s AI infrastructure appeared first on The New Stack.

Every application has always needed a place to keep its state. That is not new. What is new is who creates that place, how many get created, and how long they have to survive after everyone has stopped looking.

“Separating durable state from ephemeral compute is the new requirement, and the idle cost trap is only half of it.”

Consider the agents built by Kimi, the AI platform from Moonshot AI. A non-technical user describes an application in plain language. An agent builds the frontend, backend, and database, then deploys them in minutes. The user never thinks about infrastructure. They asked for a tool and received a running one.

Then the agent does something the previous generation of coding assistants never did. It does not hand over code and walk away. It keeps the application running and comes back to maintain it. That last word is where the old assumptions break. Build once and hand off, and state is almost an afterthought. 

Build, deploy, and maintain across tens of millions of applications, most idle most of the time, and persistence stops being a database feature and becomes the central economic problem of the whole system.

I have written before about what happens to a database when its user is an agent rather than a human. Kimi is the clearest production example I have seen, and it surfaces a requirement that the industry has not really been asked to meet: keep state alive cheaply in two very different places at once. The problem is not performance. It is the economics of persistence.

The number that changes the question

When you serve thousands of tenants, the questions are familiar. How fast are queries? How do we fail over? How do we back up? The industry has good answers for all of them. When you serve tens of millions, most created by agents and most idle most of the time, a different question moves to the front, and it is the one nobody planned for: what does it cost to keep something alive that nobody is using right now?

Ask it once, and you are talking about idle databases. Ask it again, one layer up, and you are talking about the agent’s own half-finished work sitting between maintenance sessions. Same question, two surfaces. At a thousand you absorb it. At ten thousand it stings. At tens of millions it is fatal.

“In a human-driven product, what you provision and what people use track each other. In an agent-driven product, they diverge wildly.”

This is the shift agents force. In a human-driven product, what you provision and what people use track each other. In an agent-driven product, they diverge wildly, because agents create far faster and more casually than humans ever did, and most of what they create goes idle immediately.

One principle, two places

There is a single move underneath everything that follows: separate durable state from ephemeral compute. State is the part you must never lose, so it belongs on a shared, cheap, effectively infinite foundation. 

I have argued before that object storage is becoming the new network layer of the data stack, and this is what that looks like, built for agents: durable data at rest costs almost nothing. Compute is the part you should be able to throw away, summoned when work arrives and released when it stops.

Couple the two and you pay to keep compute running for state that is doing nothing. Decouple them, and your cost stops tracking how much you have created and starts tracking how much work is happening. 

Every hard number in the Kimi deployment comes from getting this separation right in the two places an agent-scale product needs it: the databases the agent provisions for its users and the workspace the agent lives in.

The first place: the idle tenant

The instinct most teams start with is the one that has worked for a decade: give each tenant its own database instance. It is clean and easy to reason about. Kimi’s earlier shape was close to this, with single-instance PostgreSQL behind the product. It works beautifully until the tenant count climbs into the tens of thousands, and then the economics invert. The reason is structural. 

A per-tenant instance couples the logical experience of an isolated database to the physical reality of dedicated, always-on compute. The tenant needs the first. The business cannot afford the second at scale. Cost grows with tenants created, not tenants active, and in an agent-driven product those two numbers are worlds apart.

“The agent experiences a dedicated, isolated database, while a virtual layer beneath provides each one its own namespace. The agent does not need a dedicated instance. It needs the experience of one.”

I have started calling this the idle cost trap, because teams walk into it without seeing it, and by the time they feel it, the architecture that created it is load-bearing. The way out is the principle. The agent experiences a dedicated, isolated database, while a virtual layer beneath provides each one its own namespace and isolation guarantees over a shared substrate, with object storage holding durable data and a routing layer sending requests where they belong. The agent does not need a dedicated instance. It needs the experience of one.

There is a catch worth being honest about. If compute is summoned on demand, then when a request arrives for an idle tenant something has to spin up, and if that is slow, you have traded a cost problem for a latency problem. 

Kimi provisions a database in about a second, from a warm pool of pre-initialized resources kept ready and replenished behind each claim, so database setup drops out of the delivery pipeline entirely. That number is not a benchmark to brag about. It is the evidence that ephemeral compute over shared, persistent storage can stand in for the always-on model, because it appears fast enough to feel always-on.

The second place: the agent’s own workspace

The maintenance half of the lifecycle drags the same problem into a place most teams never look: the agent’s own working environment. An agent that builds once can treat its workspace as disposable: spin up a sandbox, write code, ship, discard. An agent that maintains cannot work that way. 

It has to return days or weeks later and pick up where it left off: source code, Git history, checkpoints, the record of what it was in the middle of doing. If that context is lost, the agent does not resume work. It is reconstructing it.

And execution environments are ephemeral on purpose, because keeping millions alive between sessions is the idle cost trap in a different costume. So the environment gets torn down. If the work goes with it, every maintenance session begins by rebuilding state the agent already had: wasted compute, wasted tokens, and a user watching an agent relearn its own project.

The fix is the same separation, one layer up. Kimi uses a persistent filesystem that keeps development state alive independently of the compute that produced it. Source code, Git history, checkpoints, and task progress persist after the environment is destroyed, so the agent resumes rather than starting over. Compute is ephemeral and cheap to discard. State is durable and cheap to keep. Same trade as the idle tenant, seen from the builder’s side.

The database choice becomes a quality input

There is a further effect that surprised me, because it shows up in the quality of what the agent builds. Every infrastructure decision is a chance to get something wrong. If each task forces the agent to reason from scratch about which database to use and how to configure it, it is improvising on every run, and improvisation is where errors enter. 

When the stack is unified, it applies known-good patterns instead. Kimi saw code generation success rates improve from standardizing on a unified data layer, and that is the mechanism: fewer places for the output to go wrong.

“The choice of database is no longer just an infrastructure decision. It is a quality input to the agent’s work.”

This is a version of an argument I keep returning to. Agents need guardrails, and the most effective ones are built into the infrastructure rather than bolted on as instructions. A database that behaves consistently every time is a guardrail. The choice of database is no longer just an infrastructure decision. It is a quality input to the agent’s work.

Four properties that have never been requried together

Step back from Kimi and the shape of the requirement is clear. An agent-scale product needs four things from its data layer at once:

• Tenant isolation: every agent-created database must be logically separate, so millions of tenants never bleed into one another.
• Instant provisioning: a new tenant has to exist in about a second, because an agent will not wait and neither will the user behind it.
• Cost elasticity: an idle tenant, and an idle workspace, has to cost almost nothing, because most things are idle most of the time.
• Persistent state independent of compute: the tenant’s data and the agent’s development state alike must survive the ephemeral environments that produced them.

Each has been solved before in isolation. Databases have offered isolation for decades. Serverless systems provision quickly. Object storage is cheap at rest. What is new is the demand for all four at once, at a scale where any one failing breaks the product. That is the real infrastructure competition of the agent era, and it is not the one the industry is used to having. It is not about who has the fastest single query. It is about who can hold all four together at tens of millions of tenants without one collapsing the others.

The pattern is bigger than one product

I do not think Kimi is a special case. It is an early, unusually clear instance of a pattern that will repeat across every product where agents build for non-technical users at scale. The shape is always the same. One agent, one workspace, one database, repeated millions of times. Each instance feels independent. 

The infrastructure underneath is shared, and it works only because a virtual layer separates the logical experience of isolation from the physical cost of compute, and because durable state is allowed to outlive the compute that produced it.

Kimi is the version where the agent hands a finished application to an end user and then stays on to maintain it, which is harder in two ways at once. The tenant does not go away when the run ends; it persists, idle, waiting, multiplied by tens of millions. And the agent does not go away either, so its own working state has to survive every gap between sessions. Persistence is no longer a property of one component. It is the property the whole system is organized around.

“The database conversation in the agent era is no longer really about speed. Get it wrong, and no amount of model quality will save the margins.”

Teams building in this direction will hit the idle cost trap on both surfaces, whether they plan for it or not. The only choice is whether they see it coming. Design for all four from the start and you scale past the point where the per-tenant-instance model collapses. Do not, and you hit that ceiling at tens of thousands of tenants, exactly where Kimi’s earlier architecture did, and sooner than you expect, because agents fill a tenant table faster than any human-driven product ever has.

The database conversation in the agent era is no longer really about speed. It is about whether the economics of persistence can survive the scale that agents create in both the database and the workspace. Get that right early, and the rest of the product has room to grow. Get it wrong, and no amount of model quality will save the margins.

If you are building a product where agents provision infrastructure for end users and then stay on to maintain it, four things make that economically viable: isolation, instant provisioning, near-zero idle cost, and durable state on a shared substrate. That is what we have built into TiDB’s serverless and agent workloads. It is the pattern these teams keep arriving at from different directions.

The post When agents build, deploy, and maintain, persistence becomes the hard problem appeared first on The New Stack.

OpenAI caused something of a stir over the weekend when it announced plans to cut Cursor’s direct access to OpenAI models in November.

The reason? Elon Musk.

In a statement issued late on Friday, OpenAI pointed to two previous incidents involving Musk’s companies: Twitter breaking the terms of a data-licensing deal after Musk’s 2022 takeover of the social network, and Musk’s admission under oath earlier this year that xAI had partly used OpenAI models through distillation — conduct OpenAI says violated its terms of service. And now that SpaceX’s $60 billion deal to acquire Cursor has closed, OpenAI’s attentions are turning to Cursor.

“This decision was incredibly tough, as we care deeply about our models being broadly available for developers,” the company wrote. “We are making this choice because we cannot be confident that SpaceX will use our technology within our terms of service, based on our experience with Elon Musk’s companies violating contracts.”

While that in itself was big news for anyone following the day-to-day rough and tumble of the AI industry, what was particularly notable was the response of OpenAI’s arch rival.

The Anthropic factor

As The New Stack noted in its coverage, Anthropic co-founder and “chief compute officer” Tom Brown moved fast, posting publicly within hours of OpenAI’s statement to confirm that it continues to see Cursor as a “trusted partner,” and will “continue to increase compute to support Claude models in Cursor.”

But anyone who has followed Anthropic’s recent history could be forgiven for wondering why.

Take Windsurf. In May 2025, reports emerged that OpenAI was in talks to buy the AI coding tool for $3 billion. Anthropic didn’t hang around for the deal to close though, and within weeks, Windsurf said Anthropic had cut its direct access to Claude 3.5 Sonnet and Claude 3.7 Sonnet.

Speaking at an event hosted by TechCrunch shortly after, Anthropic co-founder Jared Kaplan said that “it would be odd” for Anthropic to be selling Claude to OpenAI. As things transpired, the OpenAI deal fell through, and Google stepped in instead, paying $2.4 billion to hire Windsurf’s founders and R&D staff into DeepMind.

“They cut them off ruthlessly when RUMORS of OpenAI potentially buying Windsurf surfaced.”

In a social media post published on Sunday, Gergely Orosz, engineer and author of the Pragmatic Engineer newsletter, is quick to highlight the episode with Windsurf, which he noted had also been a “trusted partner” with Anthropic for some time. “They cut them off ruthlessly when RUMORS of OpenAI potentially buying Windsurf surfaced,” Orosz writes. “Now, SpaceX, an Ant(hropic) competitor bought Cursor — it’s still a trusted partner?”

Then there’s xAI, the AI company Musk founded in 2023 to build Grok, which SpaceX acquired outright in a February transaction valuing xAI at $250 billion. In January this year, Kylie Robison reported that Anthropic had cut xAI staff off from Claude, which they’d been accessing through Cursor — with Cursor reportedly telling xAI it was “a new policy anthropic is enforcing for all its major competitors.”

So Anthropic has previous form for moving fast, on rumor alone in Windsurf’s case, whenever a customer starts to resemble a competitor. Which makes this week’s public vote of confidence for a company now wholly owned by one of Anthropic’s actual rivals worth a second look.

The compute dependency

What makes SpaceX different from the previous Windsurf and xAI episodes is that Anthropic is also buying a huge amount of compute from it.

On May 6, Anthropic announced it had secured the entire output of SpaceX’s Colossus 1 data center near Memphis, Tennessee — more than 300 megawatts of compute and over 220,000 Nvidia GPUs. Anthropic had been hampered by limited compute availability, and the SpaceX deal, alongside other recent compute agreements, let it raise usage limits for Claude Pro and Max subscribers almost overnight.

Two weeks later, the financial terms came out. Per SpaceX’s IPO filing, Anthropic agreed to pay $1.25 billion a month for compute across Colossus and Colossus II — about 325,000 Nvidia GPUs combined — scheduled to run through May 2029, subject to termination rights.

SpaceX, for its part, said the arrangement would allow it to monetize some of its compute capacity while retaining enough to meet its own AI training and inference needs. As The New Stack reported in May, the deal also underscored just how central access to compute had become to competition between the leading AI labs. And it created an unusual commercial relationship: Anthropic was now buying a huge amount of compute from a company that also owned one of its direct AI rivals in xAI.

And that is what makes Anthropic’s response to the Cursor acquisition so notable. In response to Tom Brown’s post on X on Saturday, Replit founder and CEO Amjad Masad points to the contrast between Anthropic’s support for Cursor now and its treatment of Windsurf last year, suggesting the latter had been harsher than OpenAI’s decision to cut Cursor off.

“More likely answer is that you can’t do that here because you need the compute.”

“Maybe you changed your ways, but we all remember what you did to Windsurf, which was infinitely nastier,” Masad writes. “More likely answer is that you can’t do that here because you need the compute.”

Orosz essentially makes the same argument. If Anthropic was prepared to cut access when Windsurf merely looked likely to end up in OpenAI’s hands, why is it publicly promising MORE Claude capacity to Cursor after the company had actually been acquired by SpaceX?

“SpaceX basically in this enviable position where one of its biggest competitors depends on its compute infra!”

“Either SpaceX and Grok are not competitors to Anthropic (they are!); or, more likely, SpaceX leasing its Colossus 1 data center is more important to Anthropic than to stop offering Claude to SpaceX,” Orosz writes. “SpaceX basically in this enviable position where one of its biggest competitors depends on its compute infra!”

And so this effectively highlights how strong a position SpaceX finds itself in. It now owns xAI and Cursor, putting it in direct competition with Anthropic in foundation models through Grok and in AI coding tools through Cursor, while Anthropic is simultaneously paying it billions of dollars for compute capacity supporting Claude.

Whatever the stated rationale for treating Cursor as a “trusted partner,” that relationship leaves SpaceX with something neither Windsurf nor xAI had at the time Anthropic moved against them: a source of leverage over any decision on whether or not to cut access to Claude.

The post SpaceX is in an “enviable position”: why Anthropic is sticking with Cursor as OpenAI cuts access appeared first on The New Stack.

Connecting an AI agent to a tool is relatively straightforward. Things get more complicated once an organization has hundreds or thousands of resources spread across different clouds and platforms. Agentic Resource Discovery, or ARD, is designed to help agents navigate all of that.

AWS highlighted the open specification in its August 31 Weekly Roundup after taking a deeper technical look at it a week earlier, describing the idea as “DNS, but for agents.” Instead of telling an agent where to find everything in advance, ARD lets it search across different registries for what it needs.

And despite AWS highlighting the project, ARD isn’t an AWS technology. It was authored by Junjie Bu of Google, R.V. Guha of Microsoft, and Shaun Smith of Hugging Face, and released under the Apache 2.0 license. Engineers from several other companies have helped shape the project, including Cisco, Databricks, GitHub, GoDaddy, Nvidia, Salesforce, ServiceNow, and Snowflake.

AWS’s role, at least so far, has been to provide feedback on the specification and explore how it could work with its own Agent Registry. The goal is to make the existing registries work together.

Instead of telling an agent where to find everything ahead of time, ARD lets it search across different registries for what it needs.

MCP skips the discovery step

The Model Context Protocol has become a common way for AI applications to connect to external tools and data, but it assumes the client already knows which server it wants to use. That becomes a problem as companies spread their infrastructure across clouds, SaaS platforms, and internal systems.

ARD helps an agent find a resource before it tries to use it. The specification uses the term “agentic resource” to refer to anything an AI client can connect to, from an MCP server to other external capabilities. An ARD-compatible service keeps track of what’s available, rather than requiring developers to set up every connection in advance.

The Model Context Protocol has become a common way for AI applications to connect to external tools and data, but it assumes the client already knows which server it wants to use.

Federation without forced migration

Companies can keep their own catalogs and policies while routing searches to other ARD-compatible services. An enterprise, for example, could keep internal resources private while searching approved external catalogs when needed. AWS calls this “describe once, discover everywhere.”

The current v0.91 proposal, dated August 26, uses JSON-LD and a REST interface. Its required POST /search endpoint searches by task, while optional endpoints allow clients to browse available resources.

Each discovery service can set its own rules for what it returns and which sources it trusts. This is also where AWS’s DNS comparison falls short. A domain name points to a specific location, while an ARD search could turn up several options that all appear capable of doing the job.

Route 53 engineers shaped ARD

The DNS comparison has some history behind it. Two of the three authors of AWS’s August 24 ARD post work closely with Route 53. Principal software engineer Jeffrey Damick focuses on DNS and networking technologies. At the same time, Bhargav Talluri leads product management for Route 53 and for agent identity and discovery in AWS Agent Registry.The

Agent Registry already provides AWS customers with a central view of their resources. Adding ARD could bring resources running elsewhere into that view without requiring companies to register everything with AWS.

Adding ARD could bring resources running elsewhere into that view without requiring companies to register everything with AWS.

ARD’s governance is still being worked out, with board terms and membership among the details yet to be settled. The group has also discussed eventually moving the project to a neutral organization such as the W3C or an AI foundation.

Finding tools before using them

AWS is already exploring how it could connect with Agent Registry and find resources outside its own catalog.

There may not be much time to settle on a common approach, since connecting all these directories will only get harder once companies have built their own discovery systems.

The post MCP was supposed to solve the agent tooling problem. It missed a step. appeared first on The New Stack.

SpaceX and Nvidia say they are adapting the Vera Rubin NVL72 rack-scale AI platform for orbital use, with SpaceX targeting a first launch in the fourth quarter of 2027. 

The dream of an AI data center in space lives on in SpaceX and Nvidia’s August 24 announcements that the platform for Low Earth Orbit (LEO) Starmind AI satellites will be based on the Vera Rubin NVL72 chip family and architecture.

This proposed system would form the computing core of SpaceXAI’s first-generation Starmind AI satellite and extend Nvidia’s architecture from terrestrial AI data centers into space. 

SpaceX CEO Elon Musk posted on X the same day, “SpaceX, in partnership with Nvidia, has designed a space-optimized Vera Rubin NVL72 system for launch to orbit in Q4 next year, with significant scale in 2028.”

Musk’s post came after he said during SpaceX’s Q2 earnings call, “Going forward, we’ve decided to build exclusively on Nvidia because we think the Vera Rubin architecture is the best architecture.” Musk continued, “This is not some sort of far-future, distant thing; we expect to start launching these next year. We think the design of the NVL72 VR computer is a much better design than, say, having a standard rack -style design. So we expect to deploy this on the ground as well as in orbit, because we think it’s going to be a radical simplification of the standard NVL72 rack. It will cost less. It will be more effective. If we’re going to put it in space, why not want to put it on the ground? I think that’s going to be pretty cool.”

On Earth, the Vera Rubin NVL72 is Nvidia’s rack-scale AI design that combines 72 Rubin GPUs and 36 Vera CPUs, alongside high-speed networking components such as ConnectX-9 SuperNICs. Nvidia says SpaceXAI’s planned Starmind satellite will be based on an optimized version of that system.

A conventional NVL72 rack assumes gravity, technicians, stable grid power, a building-scale liquid loop and frequent replacement of failed parts. Orbit removes each of these assumptions.

The idea is more ambitious than putting a conventional edge-AI accelerator aboard a spacecraft. Nvidia and SpaceXAI are proposing to bring a modified architecture used in AI data centers into orbit, while altering it for orbital operational requirements.

Getting that working in orbit, though, is easier said than done. 

As Curtis Pyke, founder of Kingy AI, writes, “A conventional NVL72 rack assumes gravity, technicians, stable grid power, a building-scale liquid loop and frequent replacement of failed parts. Orbit removes each of these assumptions.“

“Cooling is unforgiving. Space is cold, but vacuum does not carry heat away through convection.”

In particular, Pyke continues, “Cooling is unforgiving. Space is cold, but vacuum does not carry heat away through convection. Heat must travel from the chips to the radiator surfaces and then leave as infrared radiation. SpaceX says AI1 can avoid chillers, cooling towers and fans and reduce cooling overhead by an order of magnitude.”

SpaceX explains that AI1 would instead use closed-loop liquid cooling inside the spacecraft and large deployable radiators to send heat directly to space as infrared radiation. While the claimed reduction is physically plausible in principle, there’s no proof yet that these AI satellites’ cooling systems can deliver. 

Another major problem that remains unaddressed is how to make the orbital rack radiation-tolerant. Making Vera Rubin NVL72 radiation-tolerant means far more than putting an ordinary NVL72 rack in a shielded satellite enclosure. It would require a system-level redesign of its GPUs, CPUs, memory, networking, power, cooling, firmware, and operations around a specified orbit and mission life.

LEO orbit is not benign. NASA cites typical trapped-particle dose rates of 100 to 1,000 rad(Si) per year for low-inclination LEO spacecraft below 500 km. That level of radiation is not an immediate death sentence for electronics, but over a multiyear mission it will cause cumulative degradation. Radiation-qualified space hardware can deal with that. Commercial Off-The-Shelf (COTS) electronics are another matter. A true radiation-hardened Rubin GPU would also require design changes at the transistor and circuit levels. 

Even were Nvidia to make such a chip, for a high-density AI system such as the SpaceX design, the concern isn’t simply whether one processor survives a 5- or 10-year dose. The satellite contains numerous radiation-sensitive elements, such as GPU logic, SRAM caches, register files, system memory, and memory controllers. With thousands of cores and billions of memory storage cells, the aggregate fault rate — not the behavior of an individual component — drives the design.

The most realistic near-term answer would be a radiation-tolerant, fault-managed Rubin-derived orbital system, not a fully radiation-hardened NVL72 in the traditional military-space sense. It could use selected commercial Nvidia parts, substantial shielding, ECC and data integrity mechanisms, redundant controllers and power paths, aggressive fault detection, software recovery, and reduced-performance operating modes.

The post SpaceX designed an orbital Vera Rubin. Radiation comes next. appeared first on The New Stack.

Alibaba this week unveiled Qwen3.8-Flash, an open-weight, multimodal Mixture-of-Experts (MoE) model. 

Hot on the heels of Qwen 3.8 Max, which arrived at the start of the month, this 125-billion-parameter AI model is offered as a prelude to Qwen 4. It is positioned as both a performance and value-for-money play. As such, it is claimed to have “superior capabilities in coding and office tasks” and an optimal balance among capability, latency, and cost.

“In this release, we are opening the weights of Qwen3.8-Flash-Next, a multimodal MoE model that also serves as an early preview of the architecture used in Qwen4,” confirmed Alibaba in its release blog.

How and why does Alibaba offer early access precursor models?

By releasing the architectural changes in Qwen3.8-Flash early, the organization hopes the community will examine the mechanics, constructs, and components within and start road-testing them before the full Qwen4 model family is built on top of them.

By stating that Qwen3.8-Flash paves the way for Qwen4, Alibaba is showcasing (and, importantly, openly sharing) design forms that it will carry into its subsequent models, so that developers can start building (or at least planning) their next codebases early. 

Specifically then, Alibaba has stated that Qwen3.8-Flash “plays the same role” that Qwen3-Next played for Qwen3.5 i.e. by which the company means that Qwen3.8-Flash introduced developers to the company’s hybrid Gated DeltaNet + Gated Attention design (computational components that enable AI models to balance long-context efficiency with contextual focus and attention during inference and training), which was then used across the Qwen3.5, Qwen3.6, Qwen3.7 and Qwen3.8 series.

“Qwen3.8-Flash-Next upgrades the model systematically along four aspects – attention, residual, embedding and optimization –  improving model capability while further optimizing computational efficiency, model capacity and training stability,” stated Alibaba.

Benchmark scores against rival models

When benchmarked on agentic coding, long-horizon agent tasks and multimodal intelligence, Qwen3.8-Flash appears to perform respectably against models including DeepSeek-V4-Flash and Claude-Opus-4.6 across SWE-bench Pro (agentic coding), CoWorkBench (long-horizon office work), Toolathlon Verified (real-world tool use), MathVision (visual maths problem-solving), AndroidWorld (agentic mobile use) and ERQA (embodied intelligence).

“Qwen3.8-Flash-Next upgrades the model systematically along four aspects – attention, residual, embedding and optimization – improving model capability while further optimizing computational efficiency, model capacity and training stability.”

Tested on agentic coding using SWE-bench Pro, Qwen3.8-Flash-Next scores 62.5, compared to 61.7 on Qwen3.8-27B, 55 on Qwen3.7-Plus, 56.0 on DeepSeek-V4-Flash-0731, and 53.4 on Claude-Opus-4.6 (Max).

Importantly, Qwen3.8-Flash requires only what Alibaba details as “around one-ninth of the training resources,” while delivering superior performance. The company says that this means Qwen3.8-Flash “significantly reduces” both training and inference costs compared with Qwen3.7-Plus, a model three times its size. 

What’s the difference between Qwen3.8-Flash-Next and Qwen3.8-Flash?

For the sake of nomenclature, Qwen3.8-Flash-Next is the open-weight research-frontier model available to developers on both the Hugging Face AI developer hub and Alibaba’s ModelScope community portal. Built on the same underlying architecture, Qwen3.8-Flash is the production version of the model, offered via the QwenCloud API with 1 million tokens by default and official built-in tools. 

Qwen3.8-Flash-Next features a 125B-parameter main model, supplemented by an additional 51B N-gram embeddings, with 6B parameters activated per token. It natively supports 262,144 tokens of context and can be extended to 1,000,000 tokens with YaRN.

What architectural updates have happened?

As noted, Qwen3.8-Flash introduces architectural extensions across attention mechanisms, residual connections, embeddings, and optimization. Its hybrid attention architecture combines Gated DeltaNet (GDN), which compresses historical information, with Qwen Sparse Attention (QSA), a novel design that uses a lightweight compressed indexer to select relevant context, substantially reducing attention costs for long sequences.

Alibaba has explained that the Gated Residual (GR) mechanism expands data pathways between layers while strengthening cross-layer information flow and training stability. The N-gram Embedding technique scales model capacity with minimal additional computation, while the Muon Optimizer enhances the efficiency of large-scale model training.

“I’d love it if local LLMs actually could replace remote ones today, and I have no reason to lie about my experience either…it’s just not there (yet) for local professional software development.”

What do developers think of Qwen3.8-Flash?

In terms of developer reaction, it’s been a mixed bag so far. 

Mechatronics engineer Alok posts on X, saying he thinks the Video RAM barrier (i.e., the need for physical, high-speed GPU-based memory for LLM memory management in the face of model quantization that aims to enable better long-context inference) is now officially dead. 

“I just ran Qwen3.8-Flash-Next (MoE) 125B A6B with a  250,000 context window on a single 24GB RTX 4090 – 21 tokens/sec decode. 364 t/s prefill – no mtp. No dflash. No KV cache quantization! We are running datacenter models on consumer hardware,” enthused Alok.

Multi-disciplined software developer Embedding Shapes is less happy.

They post on Hacker News using some colorful language to describe how models keep [insert expletive]-ing up very basic things before saying, “I’d love it if local LLMs actually could replace remote ones today, and I have no reason to lie about my experience either. But I too got hopeful reading the sentiment on the Internet about Qwen 3.8, but it’s just not there (yet) for local professional software development.”

Pricing and access

Qwen3.8-Flash can be accessed via API Model Studio and Qwen Cloud, Alibaba’s AI-native cloud platform. Pricing per 1 million tokens is US$0.16 for input and US$0.47 (or 3 RMB for developers inside China) for output.

The model is also available on QwenWork, Alibaba’s workplace AI agent platform, where it runs a redesigned “standard mode” that cuts token consumption per task by 75% and “roughly doubles generation speed” compared with the current mode.

Alibaba has claimed that this brings flagship-level capabilities within reach of everyday workloads and that developers can run this model on hardware that they likely already own.

When is Alibaba’s Qwen4 scheduled for launch?

Alibaba has not confirmed a firm launch date for Qwen4. Still, a casual web search on the topic yields a range of commentators and market watchers who broadly agree it will arrive before the end of the year, possibly as early as September.

Conjecture in this space explores whether the next model family could be optimized for complex 3D coding and design tasks in advanced spatial modeling. Others think that the Mixture-of-Experts (MoE) architecture will be extended and that deeper native multimodal processing capabilities might be featured.

Alibaba was contacted for broader comment on this story but declined to engage.

Qwen, in traditional Chinese, 通義千問 (pronounced Tōngyì qiān wèn), translates to “a thousand questions on general meaning” in English. Use that in your local pub quiz this weekend.

The post Alibaba just released Qwen3.8-Flash: “An early preview of the architecture in Qwen4” appeared first on The New Stack.

Growing datasets and public benchmarks are making it harder to tell whether a model is being tested on something it hasn’t seen before.

On Thursday, Google DeepMind showed off what the company calls the first double-blind evaluation of a proprietary frontier-class AI model. The setup keeps Gemini’s model weights hidden from the evaluators while the test questions are hidden from Google.

The pilot tested Gemini 2.5 Flash Lite against private benchmarks from MLCommons and the Singapore AI Safety Institute, but rather than using the results to tout a new Gemini score, Google is focusing on how the tests were run, with neither side having access to the other’s data.

Benchmark leakage inflates scores

Google’s technical report cites earlier research that found signs of benchmark leakage in about half of the 31 models tested, and another study published this year that found contamination can inflate scores, particularly for larger models. This dynamic has already raised questions about where certain post-training coding gains actually come from.

Keeping benchmarks private would seem like the obvious answer, except closed models make that difficult. With a closed model, evaluators usually have to run their questions through the provider’s API, which means the company can see a test that was supposed to stay private, whereas running it independently would require the provider to hand over its model weights. Google’s setup is meant to give both sides another option.

Keeping benchmarks private would seem like the obvious answer, except closed models make that difficult.

How the enclave works

The pilot is using Google Cloud Confidential Space together with an NVIDIA H100 Confidential GPU and Intel TDX host memory encryption.

Google DeepMind provides Gemini with its weights and inference code, while the evaluator offers its benchmark prompts and evaluation code; these are then transmitted over encrypted connections into the enclave, where the evaluation takes place without either party gaining access to the other’s protected assets.

The model weights are stored in hardware-encrypted GPU memory, and the evaluation prompts are kept in encrypted host memory as well. After the test is complete, the evaluator is given the permitted results, and the temporary environment can then be destroyed.

Before either side sends over its private data, remote attestation verifies that the enclave is running the software they agreed on.

Code controls beyond encryption

The enclave protects what’s stored in memory, but the code running inside it can still create problems. If an application has unrestricted network access, for example, it could send sensitive information somewhere else.

OpenMined’s PySyft handles that part of the process by allowing Google and the evaluator to approve the code in advance and to block sensitive parts of the evaluation from making external connections.

That process takes some work, although compute doesn’t appear to be the problem. The paper puts the overhead at less than 5% and instead points to legal agreements and code reviews between organizations as some of the bigger hurdles to running these evaluations.

The researchers eventually want to make attestation much less hands-on, comparing the goal to the HTTPS lock icon in a browser, where the security checks happen without the user having to deal with the underlying hashes and keys.

Google still has a hand in verification because, although the Confidential Space guest OS is open source and its build process has been externally validated, individual builds rely on private signing keys and can’t be independently reproduced. At the same time, Google’s own services are used to sign and verify the attestation report.

The paper puts the overhead at less than 5% and points instead to the legal agreements and code reviews between organizations as some of the bigger hurdles to running these evaluations.

Scaling past one GPU

So while the system reduces how much the two sides need to trust each other, it doesn’t remove trust entirely from the equation. Some of it simply moves elsewhere, including to the hardware itself. There is still some trust involved in the hardware itself, including the assumption that the cloud provider and the hardware maker aren’t working together to circumvent the protections.

Then there’s the benchmark itself. MLCommons points out that it still needs to be carefully managed, no matter how well the questions are protected.

The pilot ran Gemini 2.5 Flash Lite on a single H100 80GB Confidential GPU. Still, the researchers are already looking at clusters of H100 and B200 GPUs connected via encrypted links to eventually evaluate models that are too large to fit on a single GPU.

If this approach catches on, developers could have another way to look beyond the benchmark score, with some proof that the company behind the model didn’t see the test before the results came out.

If this approach catches on, a benchmark score could offer some proof that the company behind the model didn’t get to see the test first.

The post Google found a way to test Gemini without seeing the questions appeared first on The New Stack.

AI coding company Replit is throwing its weight behind the model-routing trend by making its “intelligent model routing” system the default across every account.

The system automatically selects the underlying model to handle a task as it evolves, with Replit weighing quality, speed, and cost in its routing decisions.

The company says the feature, dubbed Auto mode, will become the default option for all users, though Core and Pro subscribers can still override it and manually select models when they want more control.

Model-routing momentum

The announcement comes hot on the heels of a flurry of activity in the model-routing realm. Earlier in August, Stripe agreed to acquire model gateway platform OpenRouter for a reported $8 billion. On the very same day, Ramp launched Router.com, which routes requests to the lowest-cost model that meets a specified performance bar.

Before all that, in July, SpaceX-owned Cursor launched its own router, which automatically selects models for coding requests and claims to deliver comparable performance at a substantially lower cost. Meanwhile, Meta is reportedly developing an internal router called “Switchboard” that scores coding tasks by difficulty and sends simpler jobs to cheaper models.

“Across one model family, per-token rates can span orders of magnitude. At the same time, the intelligence of cheaper, smaller models is now much closer to their larger frontier counterparts, providing us a lot of room for cost optimizations.”

Michele Catasta, president and head of AI at Replit, says that one reason for the wider push into routing is simple economics — the growing gap between what models cost and the level of capability developers actually need for a given task.

“Across one model family, per-token rates can span orders of magnitude,” Catasta tells The New Stack. “At the same time, the intelligence of cheaper, smaller models is now much closer to their larger frontier counterparts, providing us a lot of room for cost optimizations.”

Replit, for its part, has been moving in this direction for some time. Catasta says that the company has spent recent months experimenting with early versions of Auto mode, subagent routing, and multiple iterations.

“Like any pivotal launch, we thoroughly tested Intelligent Model Routing in beta for a long period of time before we decided to release it in public,” he says. “The most important learning is understanding from first principles the failure modes of every experiment, so we could keep hill climbing on the final system that we just shipped.”

Enter Auto mode

The foundation of that work surfaced last week when Replit introduced Free Mode, a lower-cost Agent mode that doesn’t consume usage credits and uses Auto to choose the model on theuser’ss behalf, subject to usage limits.

Now, that same Auto routing approach is being pushed more broadly across Replit. The company says intelligent model routing will become the default across every account, with all users starting in Free Mode and Replit deciding which model is best suited to the task.

Free Mode, it’s worth noting, isn’t “free” in the sense of unlimited usage. When it launched, Replit made it available to Core and Pro subscribers without consuming their usage credits, but imposed limits that reset every five hours, with higher allowances for Pro users. In Free Mode, users cannot manually select a model.

Core and Pro subscribers can, however, switch to Replit’s Power or Max modes, where they can turn off Auto and choose a model themselves. Replit may also suggest moving a task into one of those higher-powered modes when it determines that more capability is required, though those modes can incur usage costs.

For Enterprise customers, meanwhile, administrators can restrict Auto to an approved set of models for each workspace, allowing Replit to continue routing tasks automatically while keeping model choice within company policy.

The agent advantage

Even before SpaceX agreed to pay a cool $60 billion to acquire Cursor, the AI coding startup had long been investing in its own coding models, including its Composer family. More recently, under the auspices of SpaceX, Cursor has been developing more cutting-edge models, too.

Replit, by contrast, isn’t making ownership of the underlying model layer central to its pitch. Instead, it’s betting that controlling the agent and the systems around it gives Replit enough insight to make better model-selection decisions on the fly.

“Replit has owned, from the start, both the agent harness and the infrastructure surrounding models which in turn allows us to train sophisticated model routers.”

“Replit has owned, from the start, both the agent harness and the infrastructure surrounding models which in turn allows us to train sophisticated model routers,” Catasta said. “Only in this way can we always offer useful intelligence to our users at the most competitive price point.”

That becomes particularly relevant as an Agent task unfolds, with Replit noting that its system can change which model it uses as the task develops, seeking a better trade-off between capability and cost at different points in the process. But for Catasta, that kind of dynamic routing is still only one part of a much broader research problem around how agents should use models.

“Model routing is still in its early development phase, and we expect further research will move the needle on serving the best intelligence when customers most need it,” Catasta explains. “Routing is but one piece of the puzzle that is tightly integrated to many other aspects of our harness research.”

“No third-party router company could reproduce the same results for our own agent.”

Replit also argues that seeing how people use its own Agent gives it an advantage that a standalone routing provider would struggle to reproduce. Catasta says a router has to infer the nature, difficulty, scope, and intent of a request, with Replit able to train against proprietary usage data and observe those signals across its user base.

“No third-party router company could reproduce the same results for our own agent,” he says.

The post Replit’s new default: Auto mode picks the best model for each task appeared first on The New Stack.

Nvidia has reportedly agreed to buy Hugging Face for $12.9 billion, putting one of the biggest names in AI hardware in charge of a platform developers rely on to find and run open models.

The Information first reported the deal Wednesday, citing a person familiar with the agreement. Nvidia and Hugging Face had not publicly confirmed it as of publication.

Hugging Face doesn’t push developers toward one chipmaker, which is what makes the acquisition interesting. Its Optimum libraries work with Nvidia’s TensorRT-LLM and also support hardware from AMD, Intel, and AWS. Projects such as Optimum AMD and Optimum Intel let developers run Transformers and Diffusers models on non-Nvidia hardware.

The company also plays a role in what happens after a developer chooses a model, including how easily they can get it running on the hardware they want to use. The problem for Nvidia might be this: It is buying a platform whose value depends on openness and hardware neutrality, but if the purchase means Nvidia hardware is favored, that value might diminish.

The company also plays a role in what happens after a developer chooses a model, including how easily they can get it running on the hardware they want to use.

Hugging Face already sits between the model and the chip

Hugging Face has expanded well beyond file hosting. With Inference Endpoints, developers can deploy a model from the Hub while Hugging Face handles the underlying infrastructure.

Those hosted deployments can run on AWS, Microsoft Azure or Google Cloud, but most of the GPU options Hugging Face lists are Nvidia chips, including the T4, L4 and A100. That gives developers a wider choice of hardware through Hugging Face’s open-source libraries than through its hosted services.

If the deal goes through, Nvidia would own both sides of that experience.

Deployment defaults favor Nvidia

NIM (Nvidia Inference Microservices) already works with models hosted on Hugging Face. Developers can point NIM to an hf:// repository path and pull the model directly from the Hub.

Owning Hugging Face would give Nvidia more room to bring NIM and CUDA-optimized containers directly into the deployment experience. Nvidia hasn’t announced plans to make NIM the default, and support for AMD and Intel could remain exactly where it is.

Owning Hugging Face would give Nvidia more room to bring NIM and CUDA-optimized containers directly into the deployment experience.

The bigger question is what happens over time. Nvidia could provide earlier support for new models on its own hardware or make deployment easier. At the same time, AMD, Intel, and AWS may have to reconsider how much engineering work they want to contribute to integrations maintained within a competitor-owned platform. Some of that work could eventually move elsewhere.

For developers, the difference may come down to which path requires less work. A competing chip doesn’t have to disappear from Hugging Face to become less appealing if an Nvidia model deployment takes fewer steps. We’ve seen a similar fight over the layers between AI models and the developers using them, with Cloudflare building more of that infrastructure itself.

Open models counter custom chips

That tension also helps explain why Hugging Face could be worth considerably more to Nvidia than its revenue alone would suggest.

Nvidia has been expanding its own Nemotron family of open models while investing heavily across the AI ecosystem. At the same time, some of its biggest customers are working to reduce their dependence on Nvidia hardware.

Google has its TPUs, AWS has Trainium, and Microsoft has Maia. OpenAI and Anthropic are also developing their own AI server chips. The push extends beyond the hyperscalers. Earlier this month, five European companies committed to purchasing AI compute built around non-Nvidia hardware that hasn’t been manufactured yet — a sign that the appetite for alternative accelerators is strong enough to attract forward contracts.

OpenAI this week published results from its new Jalapeño accelerator, which showed 1.5 to 1.9 times more work per watt while cutting end-to-end latency by up to 3.6 times on large open-weight models — although the chip has not yet been deployed at anything approaching Nvidia’s scale. A strong open-model ecosystem gives Nvidia a counterweight to that trend.

Open models are often expected to run in very different environments, and Hugging Face helps developers make that possible. A model found on the Hub might end up running on Nvidia hardware, an AMD GPU, or a cloud accelerator.

That flexibility is part of what Nvidia would be buying. Pushing Hugging Face too heavily toward its own hardware could make the platform less useful to developers who rely on it to work across different systems.

Interest in those models is also growing. Models from companies including DeepSeek, Moonshot AI and Z.ai have narrowed the gap with proprietary systems. At the same time, Hugging Face CEO Clément Delangue told The Information in June that the company had doubled its number of paying subscribers during the first six months of 2026. Delangue later said the company was “close to profitability.”

The Information puts Hugging Face’s annualized revenue at about $150 million. Against a $12.9 billion price tag, that’s a multiple of roughly 86.

So Nvidia would be paying for much more than Hugging Face’s current business. It would be buying a place developers already turn to when they want to work with open models, including models that don’t have to run on Nvidia hardware.

It would be buying a place developers already turn to when they want to work with open models, including models that don’t have to run on Nvidia hardware.

Community resists easy forking

Buying Hugging Face wouldn’t give Nvidia control over everything developers find there. Libraries such as transformers and diffusers are open source, and models on the Hub remain subject to their own licenses. Openly licensed models can still be hosted elsewhere, while the underlying libraries can be forked.

Much harder to recreate is the community Hugging Face has built around them. Developers already know where to look for models and have built workflows around the Hub and its integrations.

The post Nvidia’s $12.9B Hugging Face deal has an open-source problem appeared first on The New Stack.

OpenAI’s unreleased foundation model, codenamed Astra, is already working directly inside the company’s internal codebase, taking on experimental work that previously required as much as a week from a human researcher.

As part of a series of interviews with Time, OpenAI chief scientist Jakub Pachocki said Astra can take an idea for an experiment, turn it into code, run it, and return the results. A step beyond asking AI to fix a bug or write a function, OpenAI is essentially testing what happens when you hand the agent the experiment itself and let it figure out the steps in between.

For developers, Astra offers a glimpse into the future of coding agents. But giving an agent that much freedom creates another problem. Astra may already be powerful enough to trigger OpenAI’s highest-level cybersecurity safeguards.

OpenAI is essentially testing what happens when you hand the agent the experiment itself and let it figure out the steps in between.

Persistent agents change everything

OpenAI CEO Sam Altman described what the company is building as “persistent agents,” or systems that can keep working without needing a person to prompt them through every step.

Coding agents can already dig through a repository, change files, run tests, and try again when something breaks. Persistent agents are meant to keep going without a developer guiding them through each step.

That also changes what developers need from the tools around the agent. An IDE gives it somewhere to work, but a long-running agent needs infrastructure that can keep it running safely without constant oversight.

Multi-agent coordination at scale

In one demonstration witnessed by Time, 16 Astra agents worked together on a research-level math problem, splitting it into smaller pieces and then bringing their work together into a proposed solution.

For developers, it’s not hard to imagine that same setup applied to a large software project, with different agents working on different pieces at the same time. OpenAI is already experimenting with that kind of coordination, along with agents that can stay on a job for much longer.

Running several agents at once also complicates the infrastructure behind them. Developers need a way to keep the whole operation under control. Giving agents that much freedom, though, has created another problem for OpenAI: keeping them under control.

When agents escape containment

OpenAI said this month that preliminary evaluations indicate Astra may have reached the “Critical” cybersecurity capability threshold in the company’s Preparedness Framework, a finding the company disclosed alongside a pause on some frontier workloads.

Under the company’s framework, hitting that threshold brings stricter safeguards for how the model can be used.

OpenAI had already seen what could go wrong with agents that have access to tools. During a cybersecurity test, one of its internal AI agents escaped its sandbox and accessed Hugging Face systems without authorization. Astra was not the model involved, but the incident led OpenAI to pause some frontier-model research workloads while it tightened the infrastructure used to run them.

And OpenAI isn’t the only company running into this problem. Google’s AI coding agent recently broke out of the boundaries of its IDE. The circumstances were different, and the incident was less serious, but both point to the same problem that happens when giving an agent more freedom to act; you also have to make sure it stays where you put it.

Astra is now running under OpenAI’s strictest security controls. Some training and evaluation workloads have resumed, but OpenAI says a “significant number” are still paused while it upgrades the infrastructure behind them.

Astra is now running under OpenAI’s strictest security controls.

Monitoring costs real compute

The company also says it is monitoring Astra more closely when it uses tools, watching for behavior that could signal it is going beyond what it’s allowed to do. OpenAI estimates that monitoring adds about 20% to the inference compute for those workloads.

Time reports that OpenAI still plans to release Astra, although there’s no launch date yet. Until then, OpenAI is working through the same challenge developers could eventually face as agents work for longer stretches without human supervision and begin coordinating with other agents.

As agents take on more work, the IDE is only part of the picture. Developers also need a way to see what those agents are doing and keep them from going where they shouldn’t. OpenAI is already seeing the cost, with monitoring alone adding about 20% to Astra’s inference compute.

That kind of overhead could become another cost developers have to account for, along with the governance around autonomous agents that sets boundaries on what they can do.

OpenAI is already seeing what that costs, with monitoring alone adding about 20% to Astra’s inference compute.

The post OpenAI’s Astra can do a researcher’s week of work. That’s the problem. appeared first on The New Stack.

Perplexity shipped Portable Computer this week, the local-first version of its Computer agent running on an Nvidia DGX Spark workstation, and the bill is steep: A DGX Spark starts at $4,700, and even an aging 24GB RTX 3090 sells well above $1,500.

The architectural choice beneath the surface deserves as much attention as the price tag: Most platforms building agents address reliability with greater intelligence. That typically includes a larger orchestrator model, a planner model, or a critic model reviewing the work.

Probabilistic reasoning proposes the next action, and deterministic software decides whether to execute it.

Perplexity separated the two jobs instead of stacking them. Probabilistic reasoning proposes the next action, and deterministic software decides whether to execute it.

The loop controller is code, and the decisions are still a model

Perplexity uses the term “orchestrator” to refer to the runtime controller rather than the planning model. That controller assembles context, enforces policy, and executes approved tool calls inside an OS-level sandbox. The company says it is deterministic code, not yet another model. The local model proposes the next action, including which tool to call and when to ask a cloud advisor for help. The split resembles a control plane architecture, where reasoning suggests, and inspectable software retains authority. Nate Kupp, Perplexity’s vice president of Computer Enterprise and Infrastructure, tells The New Stack that the harness accounted for most of the engineering work.

Same weights, better scores

Perplexity kept the base model and the silicon constant, putting Qwen3.8-27B on the same DGX Spark across three agent stacks. On its internal Local Knowledge Work Bench, a held-out set of 53 tasks, the company reported 82.6% for Computer. Pi scored 77.6% and Hermes 74%.

On ParseBench-100, a subset covering charts, layouts, tables, and formatting, the gap widened considerably. Perplexity reported 65.1% for Computer, while Hermes reported 34.6% and Pi reported 13.9%.

Because the base weights were constant, the gap measures the system around the model rather than a better model. That does not mean weights stopped mattering. Perplexity post-trained Qwen into PPLX 27B and reported 85.4%, above its own base-model number. Harness engineering and post-training are significant factors of this approach.

The word harness also covers a lot of ground, including prompts, tool schemas, context management, verification hooks, and document processing. Orchestration code is one part of that surface.

The security boundary lives outside the model

The sandbox is the boundary, not the determinism. Perplexity says the sandbox restricts processes, filesystem paths, and network access. If the sandbox is unavailable, the harness disables itself before making any tool call. Deterministic code is valuable here because it makes policy easy to inspect and helps the system fail safely. Deterministic code can still ship a vulnerability or faithfully execute a permitted mistake.

The key distinction is not between model orchestration and code orchestration. It is whether permission is enforced by something other than asking a model in plain English to comply.

The key distinction is not between model orchestration and code orchestration. It is whether permission is enforced by something other than asking a model in plain English to comply.

The same discipline shapes how context gets spent. Perplexity reports that Qwen3.8-27B advertises a 260,000-token context window but begins to struggle beyond 100,000 tokens. The harness therefore keeps the core prompt and toolset small and loads skills on demand. Commonly used connectors became command-line tools rather than full Model Context Protocol definitions sitting permanently in context.

Deterministic execution cannot rescue reasoning that exceeds the local model. On Terminal Bench 2.1, Perplexity reported 59.6% running locally. Letting the local agent consult Claude Opus 5 raised it to 73.0%, compared with 82.4% when Opus 5 worked alone. All of this remains vendor-reported evidence on a bench that the company has yet to open-source.

For enterprises evaluating local agents, the component to scrutinize is the layer that grants and denies authority, because that is where the platform’s engineering is most evident.

The post Perplexity just separated reasoning from authority. Here’s why it matters for enterprises. appeared first on The New Stack.