Running an LLM on your PC is easy enough, but putting an agent to work there is a different story. Portable Computer, the local version of Perplexity’s Computer agent, is now available inside the Perplexity app for Windows on compatible Nvidia GeForce RTX and RTX PRO GPUs.
That’s the good news; the catch is, you’ll need an Nvidia GPU with at least 24GB of VRAM.
The Windows launch gives Perplexity three platforms in less than three weeks. Portable Computer debuted on Linux and Nvidia DGX Spark on August 25, followed a week later by hybrid compute for Apple silicon, which splits tasks between local and cloud models on Macs. Now Windows joins the mix, but bringing Portable Computer over took more than simply porting the app. Perplexity had to adapt the model runtime, orchestration, security, and hardware integration for each platform while keeping the user experience the same.
you’ll need an Nvidia GPU with at least 24GB of VRAM to use it.
Orchestration beyond the model
Portable Computer bundles those pieces together. On Windows, it supports PPLX 27B — Perplexity’s post-trained model — and Qwen 3.8 27B, both optimized for RTX GPUs, alongside a built-in browser, tool calling, and Perplexity’s proprietary SPACE sandbox.
It’s a different lane from LM Studio or Ollama, which make running models locally as painless as possible but stop well short of giving a model autonomy over multistep work. DeepSeek’s recent hiring spree of roughly 150 new roles, nearly all of them focused on agent infrastructure rather than the model, hints at how much engineering sits between a capable model and a capable agent.
DeepSeek’s recent hiring spree of roughly 150 new roles, nearly all of them focused on agent infrastructure rather than the model, hints at how much engineering sits between a capable model and a capable agent.
Connectors blur local boundaries
Perplexity ships connectors for Microsoft Outlook, OneDrive, and Word, plus Google Drive, Gmail, Slack, and GitHub — which tells you something about what “local” actually means here.
The agent can reach external services because it’s not air-gapped. Locally completed tasks can process files without sending documents to a cloud model. Once an agent has access to both local files and remote APIs on the same machine, figuring out which resources it actually needs and where to find them gets harder.
Hybrid cloud as fallback
Perplexity isn’t pretending that a 27-billion-parameter model running on a desktop GPU can handle everything, which explains the hybrid architecture. When the agent determines that a task needs more reasoning power than the local model can deliver, it can escalate to Perplexity’s cloud models.
According to Nvidia, the agent identifies when cloud support would help and asks the user for permission before sending any data off the machine.
For organizations handling sensitive or regulated data, that split can make all the difference. A local agent can grind through source code or financial records without uploading them to a hosted model for basic processing. There’s a cost angle too, since tasks completed locally don’t burn Perplexity Computer credits.
High VRAM floor limits reach
Portable Computer is available with Perplexity Pro ($20/month) and Max ($200/month), across individual and enterprise plans, with Nvidia DGX Station support coming later. The real challenge is taking local agents from developer passion projects to enterprise-ready tools. By baking this into Windows, it immediately gets in front of the scale of users needed to make that happen.
The real challenge is taking local agents from developer passion projects to enterprise-ready tools.
Last October, the P99 conference — the online gathering for developers focused on high-performance, low-latency applications — featured a cracking keynote from Chip Huyen.
The author of the best-selling AI Engineering, Huyen opened with simple math: Training a frontier model is a one-off cost, but inference is the same cost paid over and over. That’s great for the frontier model providers, and bad for us token burners. Over the life of a model, Huyen reckons the compute split ratio lands somewhere between 1:10 and 1:100 for training to inference. Reasoning models — which burn even more tokens — push that out even further. We all know the feeling of hitting our weekly session quotas.
Huyen’s point is that if inference is too expensive, then nobody ever recovers the training bill, which might explain why there are so many memes about the “profitability” of frontier models. So, how do we optimize inference?
That’s a topic that Huyen spent months researching for her book. And in the spirit of optimization, Huyen distilled it down to 30 minutes for the conference in October 2025.
Huyen is returning for P99 CONF 2026 in a few weeks. Ahead of that moment, watch her full talk – or read the recap below – from last year, then let’s talk about how those ideas aged over the past 11 months.
What to measure
Chip recommends focusing on a few key latency metrics:
Time to first token (TTFT): How much time elapses before the user sees anything
Time per output token (TPOT): The average time between consecutive tokens (aka inter-token latency)
End-to-end latency: Time to first token, plus time per output token, multiplied by the number of output tokens minus one
(Click to enlarge graphic.)
With reasoning models, some of those tokens never reach the user. “The first generated token might not be the same as the first visible token,” Huyen explained. “The model might think for a while, and it will only show the first token of the final output to the user.”
“The first generated token might not be the same as the first visible token,” — Chip Huyen
Some people also measure Time to Publish for that (i.e., how long until the user sees the first token). The best metric to prioritize depends on what matters most for your users.
Also consider “goodput” alongside throughput. Throughput measures requests processed in a given window. Goodput measures the requests that actually met your targets. Chip’s example: an app targets 200 ms time to first token and 100 ms time per output token, and processes 10 requests per minute, but only three hit both.
(Click to enlarge graphic.)
3 ways to optimize LLM inference
With inference servers, you can optimize from 3 different angles: the hardware, the model, and the service that manages the requests and responses.
(Click to enlarge graphic.)
Huyen previously worked at Nvidia and opted out of the hardware discussion: “Even though I find it to be an intellectually interesting topic, it’s not relevant to a lot of people because we don’t have the power to change the hardware itself,” Huyen explained. She also didn’t want to spend much time on the obvious solution: replica parallelism, or just adding more machines. It’s costly, and it gets complicated fast – especially if you end up with a mix of 80GB, 48GB and 24GB machines and models of varying sizes to distribute across them.
That leaves the model and the service. Huyen offers these tips on how to decide: “If you want to host the models yourself, or if you have access to the model weights, or if you train a model yourself, or you want to fine-tune or distill a model, then model optimizations might be for you. However, if you want to take a model as-is and make it more efficient on your own inference service, you might want to look into service optimizations.
Model optimization
The following techniques change the actual weights so that they can change the model outputs.
Quantization lowers the precision used to store weights and activations (e.g., from four bytes per parameter at 32-bit to one byte at 8-bit). Huyen explained, “Reducing the precision not only reduces the memory requirement to run the model, making it cheaper. It can also make the model a lot faster. If you do additions bit by bit and each weight is 32 bits, you have to do it 32 times. If it’s 8 bits, you only have to do it eight times.”
The tradeoff is a small quality hit. Huyen continued: “It’s possible to reduce a lot of the model’s memory footprint with minimal quality degradation, and quantization is pretty generalizable to a wide variety of model architectures and model sizes. That’s why it’s very popular. I rarely see any companies running a model at full precision anymore.”
“I rarely see any companies running a model at full precision anymore.” — Chip Huyen
Distillation involves using a large model to generate training data for a smaller model. For example, say you have a truly large model (the example Huyen used was o1) and want a model that performs like it, but is much smaller. Basically, you collect a large set of prompts, run them through the larger model, then train the smaller model on its responses.
Proceed with caution, though. Huyen warned, “A lot of model providers have the condition that they do not allow their models to be used to train competitive models. So even though it’s a very common technique, you need to check licensing.”
Service optimization
This set of techniques targets how requests are scheduled, routed, and reused. The actual weights aren’t affected.
Batching groups multiple requests so they’re processed together in a single pass through the model – which is much more efficient than dealing with them one at a time. Huyen presented a few batching options:
Static batching waits for the batch to fill. This maximizes compute utilization, but it might increase the latency for the first requests.
Dynamic batching runs on a timer instead (e.g., batching every 15 ms). This is less compute-efficient, but it’s better for latency.
Continuous batching handles the case where requests finish at wildly different times, which is common with LLMs. One request asks for the capital of Vietnam; another kicks off deep research. With static or dynamic batching, the finished request’s slot sits idle until the slowest one completes – and new requests queue up behind it. Continuous batching returns each request as it finishes and fills the spot with another request. That can improve compute resource utilization and latency.
(Click to enlarge graphic.)
Decoupling prefill and decode separates the two phases of a request onto different machines. (Prefill processes the input, while decode generates the output.) Huyen said, “Input tokens can be processed in parallel, whereas output tokens need to be generated sequentially. With parallel processing, it’s bounded by compute, the processing power of the chip. With decoding, it’s bounded by memory, because you have to move model weights.”
Because each phase stresses different resources, most services now separate them. To improve time to first token, shift machines toward prefill. If you care more about improving time per output token, shift them to decode.
(Click to enlarge graphic.)
Parallelism splits work across machines. Replica parallelism copies the whole model onto more machines. Tensor parallelism divides a very large matrix, so different machines compute different parts of it. Pipeline parallelism divides the model by layer, so requests move through as a pipeline.
(Click to enlarge graphic.)
Prompt caching processes shared text once, saving cost and latency. A lot of repetition exists across requests to the same application: the system prompt, the examples, the same code base, the same document behind different questions. You might as well process that shared segment once, cache it, and reuse it.
The technique was relatively rare when Huyen was writing AI Engineering. “There was one paper about it, and it was not really known, but it made a lot of sense. So I included prompt caching in the book, and I’m very happy to see that nowadays it’s pretty much everywhere.”
(Click to enlarge graphic.)
The savings scale depending on how much of your prompt gets cached. In Claude Code logs, Huyen’s open-source tool Sniffly found cache hit rates of 90% to 97%. Some providers rewrite prompts internally to improve hit rates, but you might as well structure them yourself.
Huyen’s tip: since caching works on shared prefixes, put the stable parts of your prompt first and the variable parts later. “It’s pretty easy to do, and it can improve your application performance significantly,” she noted.
Evaluating inference providers
Huyen closed with a warning for anyone evaluating inference providers: “There are many inference companies that provide inference optimizations for models you want to use, and a lot of them advertise just cost and latency.
“But pay attention to how many inference optimization techniques also change the model behavior or reduce the model quality. So when evaluating an inference service, it’s important to look not just at cost and latency, but also at model quality. Does this model, provided on this service, also perform similarly on standard benchmarks?”
What’s changed one year later?
So where do we stand today, one year on from this keynote? Most of it actually aged quite well.
On the economics, I reckon Huyen was bang on… I think, for most of us as users, we don’t have all the cost levers to pull that Huyen outlined. But it’s great to understand what is happening. As a novice local LLM user myself, I found I could relate to her points on parallelism (I don’t have it) and prompt caching/quantization (within my grasp of control).
Prompt caching (which Huyen said was new when Huyen wrote AI Engineering) is now priced into every bundle purchase of API tokens. And her Claude Code observation (90% cache hit rates) is probably the reason we mere mortals can still afford agentic coding agents at all.
Some of it aged in ways that were hard to predict at the time. Huyen mentioned how reasoning models make inference even more significant. One year on, I think agents running multi-step loops with tool calls have turned that idea from a footnote into a way to turn Claude’s rate limits (and their infamous 99.x% availability) on their head.
All the metrics Huyen described – time to first token, time to publish, goodput under a latency SLO, etc., are all now part of the lingo and probably need to be reasoned about differently.
That’s one thing I hope she’s talking about this year! Grab a free conference pass and join us online.
Grab a complimentary pass to PG 99 Conf 2026 and join us on October 21 and 22 to chat with Huyen.
GPT-6 Astra earns nearly three times as much as Claude Fable 5.1 on Andon Labs' Vending-Bench agent benchmark and refuses illegal price-fixing deals that Fable agrees to. On drone control, Astra is the first model to beat the human baseline on all five subtasks, including finding and following individual people.
OpenAI rolled out its Agents API in public beta Thursday, opening the backend behind Codex to developers looking to run agents unattended for days.
Now, developers don’t have to build their own system to keep an agent going because the API tracks the job as it progresses and gives the agent somewhere to execute its work, even when a task stretches well beyond a single context window.
That makes long-running agents easier to try, but it also gives developers more ways to burn through compute. Interestingly enough, on the same day Agents API launched, OpenAI paused new sign-ups for its $200-a-month Pro plan after demand for GPT-6 Astra strained capacity.
Thibault Sottiaux, engineering lead for Codex, writes on X that Pro subscriptions “put the most strain on our systems,” adding that OpenAI was working to add capacity “as fast as we can.”
To make sure our current users have an incredible experience and continued access to Astra, we are going to pause subscriptions to our $200 Pro plan. These put the most strain on our systems and we wanted to take the smallest step that allows us to continue giving the broadest… https://t.co/WhLEm3HBL7
The Agents API and ChatGPT Pro are separate products, so there’s no reason to assume one is taking capacity from the other. Still, the timing stands out: the company is making it easier for developers to run agents for hours or days while pulling back access to its heaviest-use consumer plan and working to add more capacity.
Agent inference adds up fast
As a task gets longer, the API can compress earlier context, so the agent doesn’t just stop when it reaches the model’s context limit. It can also bring in tools only when they’re needed or send parts of a larger job to subagents working in parallel. The actual work can run in OpenAI’s sandbox or on infrastructure the developer controls.
The actual work can run in OpenAI’s sandbox or on infrastructure the developer controls.
As agents make progress, they go back to the model for the next step, and a job that takes hours can rack up far more inference than a typical API call. The usage climbs even faster when agents work in parallel.
OpenAI has already seen this inside its own shop. In a research report published September 6, OpenAI said its research organization was logging 3.1 agent-workdays for every human workday by mid-August, measured in standard eight-hour equivalents. The median researcher, ranked by agent usage, was spending more than $600 per day on inference at API prices, while the 90th percentile exceeded $7,000.
Before June, OpenAI’s researchers were still putting in more hours than their agent, but by mid-August, the agents were doing three times as much work.
Arguably, OpenAI’s researchers are an extreme case, but the numbers show what happens when agent use starts to scale. One person can suddenly generate far more inference than their headcount would suggest.
One person can suddenly generate far more inference than their headcount would suggest.
Friction limited compute demand
The Agents API lowers the cost of that experimentation by leaving the orchestration layer out of the bill. Developers pay for the models, tools, and hosted compute their agents actually use.
The flip side is that it’s now easier to consume more inference. Context compaction is a good example. A full context window used to force developers to decide what to discard or how to summarize the work so far. Now the API handles that automatically and the agent keeps going. That’s useful for developers, but it also means the workload doesn’t stop when the context window fills up.
Astra demand hit the ceiling
The Astra rollout offers a preview of what that could look like. OpenAI stopped accepting new Pro subscribers less than two weeks after the model launched on September 3, saying those accounts put the most strain on its systems. The Agents API has its own rate limits and usage tiers, so the Pro pause doesn’t directly affect developers using it. Still, the company is already having to manage capacity around its newest model.
Infrastructure outweighs benchmarks now
The more agents developers run, and the longer they run them, the faster that usage adds up. One developer might have several agents working at once, each going back to the model throughout the task. So headcount alone doesn’t tell you much about how much compute you’re using.
For long-running agents, the challenge is keeping the work moving without wasting tokens or losing track of the task. Cloudflare made a similar bet this summer, arguing that the infrastructure around AI workloads would eventually matter as much as the models themselves.
For long-running agents, the challenge is keeping the work moving without wasting tokens or losing track of the task.
Deploying a large language model is only the first step toward production-ready serving. Production teams also need to serve as many concurrent users as...
Deploying a large language model is only the first step toward production-ready serving. Production teams also need to serve as many concurrent users as possible on available GPU infrastructure while preserving the interactivity that keeps applications responsive. That tradeoff matters even more for agentic AI workloads, where prompts can be long, context can be reused across steps…
Hundreds of thousands of AI agent sandboxes can already run concurrently on a single DeepSeek cluster. Now the company is staffing up to handle what happens as that number — along with its training, evaluation, and other backend workloads — keeps climbing.
Cui Tianyi, who joined DeepSeek in March and works on its Harness team, the group responsible for the infrastructure and environments used to run and evaluate agents, announced in an X post that roughly 150 engineering positions on September 7, with the hiring concentrated in server-side engineering and Agent Elastic Compute rather than AI research. The work spans operating systems, virtualization, networking, storage, scheduling, and the control-plane services that coordinate those resources.
Cui said DeepSeek’s existing backend systems will need upgrades, maintenance, and rewrites as workloads grow. One such system at the center of that scaling challenge is DeepSeek Elastic Compute, or DSec, the sandbox infrastructure DeepSeek built to execute agent workloads during post-training and evaluation.
Cui said DeepSeek’s existing backend systems will need upgrades, maintenance, and rewrites as workloads grow.
Four sandboxes, one SDK
Agent workloads require more than GPUs for inference, with each agent also needing an isolated environment to run code, call tools, change files, and collect the results.
DSec supports four types of those environments through the same Python SDK. Simple function calls go to pre-warmed containers, while Docker-compatible containers handle jobs that need a persistent environment. DeepSeek uses Firecracker microVMs when stronger isolation is needed and QEMU virtual machines for workloads that require a full guest operating system.
That range means the same infrastructure can handle anything from a simple tool call to a software-engineering task that needs an entire OS. It’s a similar challenge to the one the rest of the industry is bumping into as agents move from demos to production. OpenAI, for instance, recently designed custom silicon specifically to address the compute pressure that agent workloads create, and DeepSeek open sourced its own agent harness in August.
Lazy loading agent environments
Every sandbox needs its own environment, but copying complete container or VM images onto every host would consume enormous amounts of storage and network bandwidth while adding to startup time. DeepSeek gets around that by tying DSec into 3FS, the distributed filesystem it originally built for its AI infrastructure, and keeping container base images and filesystem commits as read-only layers backed by 3FS.
The metadata stays local, but the underlying data blocks are fetched only when they’re actually needed. MicroVMs use a similar setup, sharing their read-only base layer through 3FS while writes from individual sandboxes are kept in local copy-on-write layers.
DeepSeek says DSec reduces duplicate page-cache usage across virtualized environments and reclaims memory to allow safe overcommitment, while changes to the container runtime cut the CPU overhead of each sandbox.
The team also had to deal with spinlock contention inside the container runtime. At small scale, the CPU time spent there barely registers. At scale, it limits how densely those environments can be packed onto each host.
DeepSeek says DSec reduces duplicate page-cache usage across virtualized environments and reclaims memory to allow safe overcommitment, while changes to the container runtime cut the CPU overhead of each sandbox.
When replay breaks training
During reinforcement learning and other post-training workloads, large numbers of agent rollouts can be running at once, and jobs may be interrupted as compute gets reassigned. Starting over wastes everything the agent has already done, but picking up where it left off isn’t as simple as replaying its previous commands.
Some of those commands may have changed a file or otherwise altered the environment, so running them again could produce a different result or leave the training trajectory in the wrong state. DSec avoids that with a globally ordered trajectory log that records commands along with their results.
When a rollout resumes, DSec can fast-forward through the completed work using those recorded results rather than executing the commands a second time. That reduces the cost of interruptions across thousands of training and evaluation runs, while the same logs preserve a history of how each sandbox changed and allow earlier sessions to be replayed.
Engineers, not researchers, wanted
The roughly 150 openings reach across DeepSeek’s backend, including the lower-level systems work behind Agent Elastic Compute as well as the services that support its models and agents.
DeepSeek said in June that it planned to at least double the size of every department, but this round of hiring leans heavily toward the systems underneath its models rather than the models themselves. DSec is part of that work, with hundreds of thousands of sandboxes running concurrently and putting pressure on everything from how jobs are scheduled to how they recover after an interruption.
The roughly 150 openings reach across DeepSeek’s backend, including the lower-level systems work behind Agent Elastic Compute as well as the services that support its models and agents.
Ask a traditional enterprise application for a customer address or today’s revenue figures and, broadly speaking, it follows a predictable route its developers have already mapped out: authenticate the user, query the right system, return the result. Given the same underlying data, you’ll get the same answer each time.
Ask an AI agent the same question, and the journey is much harder to forecast. It might consult one system, decide it needs more context from another, make a dozen tool calls, pass information through a language model and only then produce an answer. Run the same request again, and it may take a different route altogether.
And in an enterprise, what happens along that route can matter just as much as the answer: which systems the agent accesses, what data it sees, what actions it takes and how much it spends.
That distinction — between predetermined software, and applications that make probabilistic decisions on the fly — sits at the heart of a new company from a founder who knows a thing or two about bringing order to a new generation of infrastructure.
AI agents are hard to govern
Dome Systems co-founder David McJannet left HashiCop in August 2025
In an interview with The New Stack, McJannet lays out his company’s thesis on AI agent governance, arguing that enterprises are now running into the same kind of problem that they did with cloud infrastructure: adoption comes first, then the real spadework begins of putting the right controls in place across security, operations and finance.
“It’s actually a very different architecture, and that is what unlocks the power of these new [agentic] applications.”
Part of the challenge, he says, is that agents are built very differently from the enterprise applications of yore, which companies spent years learning how to control.
“It’s actually a very different architecture, and that is what unlocks the power of these new [agentic] applications,” McJannet explains.
He points to self-driving cars as an example: a model takes in live inputs and interacts with the vehicle’s systems as conditions change, because no developer can reasonably pre-program every possible situation a car might encounter on the road.
“It’s making judgments along the way, as opposed to trying to look up the historical maps of the world and make a real-time decision,” McJannet continues.
An enterprise agent can behave in much the same way: call one tool, assess the result, decide it needs another, and keep going until the task is complete. That flexibility lets agents tackle work that would be difficult to script exhaustively in advance — but it also makes their behaviour harder for enterprises to govern.
And this gets to the heart of what McJannet is striving for with Dome.
Dome’s starting point is that an agent combines three things: code, a model, and the backend systems or tools it interacts with. Bringing those pieces together under one platform, McJannet says, is “table stakes” for applying meaningful constraints to what the agent can do.
“If you don’t have an integrated platform, you can’t enforce controls across everything that the agent is doing,” McJannet says.
“If you don’t have an integrated platform, you can’t enforce controls across everything that the agent is doing.”
And so Dome’s platform is built around those three elements. An agent registry keeps track of the agents themselves; an MCP gateway controls the tools they can call; and a model broker/router governs which models they can use and how requests are routed.
The setup starts by registering the agent and giving it an identity, establishing who is allowed to call it, and connecting the backend tools it can reach — Zendesk, in this example.
Dome registers an agent, verifies its caller and connects the tools it can use.
Next, Dome connects a model provider, groups available models into a pool with routing and failover rules, then combines the agent, its tools and its models behind a single gateway. That gateway becomes the point through which Dome can apply the policies governing what the agent is allowed to do.
Dome connects a model provider, creates a model pool and brings the agent behind a gateway.
Once those pieces are connected, teams can set permissions on each call, use guards to inspect responses, apply quotas to cap spending, and keep a common audit trail across the agent’s activity.
Today, McJannet says, enterprises are often piecing all of this together themselves. A standalone model broker might be brought in to control spending, while a separate tool gateway handles security and operational concerns. Some are then building their own agent registry to tie those systems together.
Moreover, buying those capabilities separately leaves enterprises with another integration problem to solve. A model router might govern one part of an agent’s activity and a tool gateway another, while the agent itself continues moving between them.
“If you just provide the tool gateway or just the model router, it doesn’t allow you to have this kind of system of control,” he says.
That is also where Dome’s latest move enters the fray. After spending its first months in early access, the company is now opening the platform to self-service users for the first time, allowing teams to sign up with little more than a credit card, bypassing the typically arduous enterprise sales process.
Dome goes self-serve
Self-serve is relatively unusual route for this kind of enterprise infrastructure product. Dome is publishing its prices, offering a free tier and letting practitioners get started without first going through a sales process, while keeping the traditional enterprise route open for larger customers.
The thinking is partly about who McJannet expects to use the product. Rather than limiting access to buyers who are already deep into a procurement process, for example, self-serve enables individual practitioners to be able to discover, try and use the platform themselves.
“”We want to make the barrier as low as possible to have people come on board,” McJannet says, adding that Dome had already seen a number of self-service sign-ups ahead of the launch.
Separately, its pricing reflects a belief about where value will ultimately sit in this market. McJannet regards model routing and tool connectivity as baseline capabilities, with the more valuable piece being the controls that sit across the agent as a whole — think permissions, data redaction and spending quotas.
It’s also worth noting that while Dome’s main target user will be platform engineering teams inside large enterprises, typically working alongside operations and security, self-serve also creates an opening for another kind of user: the small company, perhaps even only one or two people, building an agent and trying to sell into an enterprise. The sort of scenario that aligns with the fabled one-person unicorn promised by many in the AI realm.
Indeed, McJannet says developers can get far building the application itself, only to hit a wall when a prospective enterprise customer begins its security and operations review. How is identity enforced? Who can see the data the agent reaches? What happens when it calls other agents? Can its activity be reconstructed afterwards?
Some builders, he says, have asked whether they can “certify” their agents on Dome because “my agent won’t get deployed until I can satisfy these infrastructure elements.” McJannet is careful to add that Dome doesn’t currently run such a certification program, but it’s clearly one route the company could venture down.
“If you register that agent on Dome, all the infrastructure elements are taken care of,” McJannet says.
‘Unblocking AI agents’: Lessons from the cloud era
That division between developers eager to ship, and enterprise teams worried about what happens after, is also where McJannet sees the strongest parallel with his years at HashiCorp.
During McJannet’s tenure, HashiCorp increasingly positioned itself around helping large organizations standardize how cloud infrastructure was provisioned, secured and connected. That included the 2020 launch of HashiCorp Cloud Platform (HCP), which offered its infrastructure tools as managed cloud services.
More broadly, McJannet’s account of early cloud adoption begins with developers swiping a credit card and deploying directly to Amazon because cloud infrastructure allowed them to build applications that had previously been impractical. The applications were compelling enough that enterprises adopted cloud despite resistance from operations and security teams, and what followed was a second phase: companies needed common services for provisioning, credentials, networking and other controls before cloud could become routine across the organization.
Platform engineering teams became the people responsible for reconciling those two demands: allowing developers to build while giving security, operations and finance enough control to permit those applications into production. McJannet believes agents are now creating the same tension.
“You’ve got this queue of cool apps that developers build that the ops and security teams are just not comfortable letting flourish in their environments.”
“You’ve got this queue of cool apps that developers build that the ops and security teams are just not comfortable letting flourish in their environments,” he says. “And so, inevitably, it has to go that same direction where the platform engineering team has to figure out [a way] to get to say ‘yes’.”
Dome’s bet is that enterprises will eventually prefer one system spanning the entire agent to a patchwork of gateways, routers and security products. In McJannet’s telling, that common control layer is what gives enterprises a way to limit how far an agent can roam while still letting it act autonomously.
“You have to have this control layer that provides this corridor where we can constrain the behavior of that new type of application architecture,” he says. “Because without that, you cannot unblock the deployment of AI applications.”
“That’s the part that we’re trying to answer — how do we unblock agents at scale?”
There is still plenty for Dome to prove. The company isn’t naming customers at this stage; McJannet says none of the enterprises it has worked with are yet willing to be identified publicly, though he says Dome has spent the past eight months talking to dozens of them.
Ultimately, McJannet believes the cloud era showed that new applications only become commonplace once enterprises have the controls to let them through. Dome is his attempt to solve that problem for agents.
“I think that’s the part that we’re trying to answer — how do we unblock agents at scale?”
A context window can be technically complete and still describe a world that no longer exists. I built a deterministic benchmark to measure the cost of acting on stale context.
OpenAI says it hit a goal it set last fall, stating researchers are now using what the company calls an “automated research intern,” which is an agent that can handle well-defined tasks that would normally take a researcher several days.
The data shows coding-agent use climbing throughout 2026, and by mid-August its agents were logging 3.1 agent-workdays for every human workday across the research organization. The median researcher was spending more than $600 on inference per day at API prices, while those in the 90th percentile spent more than $7,000.
The median researcher was spending more than $600 on inference per day at API prices, while those in the 90th percentile spent more than $7,000.
Agent hours versus useful output
But everyone knows that an agent-workday and a human workday aren’t the same. The company converts the time agents spend working on tasks into standard eight-hour workdays. Because researchers can run several agents at once, the figure tells us how long the agents are working, but not necessarily what they’re completing.
For engineering teams, that leaves plenty of work on the human side, which means running more agents can increase the amount of work happening at once, but it can also increase the amount of work a human needs to keep track of.
OpenAI’s very specific definition of a research intern highlights that it must be able to complete well-defined research tasks that would take a skilled person several days, but a human is still in charge. The company’s next goal, an automated AI researcher, is one they hope to reach by March 2028.
The company’s next goal, an automated AI researcher, is one they hope to reach by March 2028.
Supervision becomes the constraint
Using a taxonomy from Epoch AI, OpenAI broke the agents’ work into six areas — Decide, Design, Build, Run, Analyze, and Communicate — and found activity increased across all six between January and August, although agents still did relatively little of the work involved in deciding what research to pursue.
Much of the work is practical, with agents writing research and infrastructure code, monitoring experiments, and providing enough technical support that OpenAI says attendance at debugging office hours has fallen, prompting one team to stop holding the sessions altogether.
And yet, more agent hours don’t automatically mean more useful research. OpenAI says code output and experiment counts are relatively easy to track, but neither shows how much progress those agents actually made. Compute also increased significantly as the number of experiments rose.
OpenAI used another model to judge how well agents performed on tasks of varying difficulty and found that, despite improving success rates between January and July, humans still had to step in on more than half of successful tasks that would have taken a person four to eight hours.
Security incidents limit Astra deployment
Once engineers can run several agents at once, with those agents launching subagents of their own, the challenge shifts to keeping up with what they produce — catching runs that go off track, reviewing code diffs, and deciding what is ready to ship or feed into a training run.
The company acknowledges that as agents take over more of the execution, the parts of research that are hardest to automate will consume more of an engineer’s time, putting a practical limit on how much agent output one person can realistically review.
On July 20, a series of outages caused by agents disrupted OpenAI’s research infrastructure badly enough that the company took its training container service offline and later brought it back with tighter restrictions.
Nearly a month later on August 7, OpenAI tightened access again after early evidence suggested Astra could reach the “Critical” cybersecurity threshold in its Preparedness Framework, restricting the model to higher-security research areas and adding safeguards that developers may already be encountering as unexpected API interruptions
Workloads shift between models fast
Astra-class GPU allocation fell 59.2% the following week, but that compute didn’t sit idle for long. Researchers moved much of the work to other models, which saw GPU allocation rise 17.2% and made up for roughly 85% of the drop in Astra usage. Instead of reducing the amount of work being run, the restrictions pushed it to other models, showing how easily workloads can move when one part of the system is locked down.
OpenAI’s researchers are handing off larger jobs to agents, running more of them at once and launching more experiments, but whether that translates into faster research is harder to measure — and OpenAI is still figuring out how to price it.
OpenAI’s researchers are handing off larger jobs to agents, running more of them at once and launching more experiments, but whether that translates into faster research is harder to measure — and OpenAI is still figuring out how to price it.
Someone moves off the finance team at 9 a.m. on a Monday. Your sync runs nightly at 2 a.m. For seventeen hours, that person can still pull finance documents out of your retrieval index, and nothing in the system knows it is wrong. I am borrowing the example from Truto, but every team I talk to recognizes some version of it.
That is the version with a clock on it. The version people ask about in security review sounds different. The retrieval pilot works, the demo lands, the executive sponsor is happy, and then someone asks how you guarantee this thing will never summarize the CEO’s compensation review for an intern who asked an innocent question about salary bands.
Most teams do not have an answer. What they have is a filter.
I think the answer has to be structural. Permissions are not a filter you apply to context after you have assembled it. They are a property of how context gets assembled for a particular identity, because assembly is the last moment where refusing to include something still means the model never saw it.
Permissions are not a filter you apply to context after you have assembled it.
The major platform vendors in this race are building some version of the same step, and nobody has really settled on a name for it. I run a company, Modus, that builds in this lane, so weigh the argument accordingly. In our product, we call it context composition. For this piece, I will call it context assembly. It is where a system decides which pieces of enterprise knowledge to hand a model for a specific person, in a specific moment, for a specific question. Everything upstream is storage, and everything downstream is inference. Assembly is where identity either lives or doesn’t.
Announced is not the same as shipped
The reason to argue about this in September rather than in June is that platform vendors have stopped disagreeing about where the step goes, and the software most companies run has not caught up with them.
Databricks reached the same slot from the other direction, extending Unity Catalog to the agent. However,h partners in that ecosystem note that the protection is anchored to the Databricks Runtime rather than to the data, so it stops applying when a BI tool or an MCP server reaches the same source directly.
Teams did not wait for any of this. They shipped the flat-index version while the identity-aware version stayed on the slide.
The direction is consistent, and so is the limit. Each of those controls is strongest inside the system that issues it. The interesting problem begins when an agent needs context that crosses several of those systems at once, and that is the job assembly has to solve.
The lake is not the business
Lake Formation enforces fine-grained permissions inside the lake it governs, and it does that well. Those permissions do not become the sharing rules in Salesforce, Slack, Google Drive, or Confluence.
AWS documents where its own boundaries sit. Its August guidance on propagating user authorization context through AgentCore walks through handing Salesforce a token scoped to the actual user, so Salesforce applies its own sharing rules. In AWS’s words, “the agent acts as an orchestrator, not a gatekeeper,” and “downstream services enforce authorization.”
That is a reasonable call. It is also an important product boundary. Lake Formation is not integrating with Salesforce, GitHub, Jira, Slack, Confluence, or Google Drive. Each of those decides who sees what on its own terms, or nobody does.
The same is true of your own stack. The tags on your chunks are not an identity boundary. They are a hint that your application code is trusted to honor.
What breaks when authorization arrives too late
The failure is structural, which is why I keep running into the same few versions of it.
I want to be careful here. “Filters are bad” is not the argument. The problem is ordering. A retrieval system can search a mixed index, retrieve opaque IDs, authorize them, and hydrate only the documents the person is allowed to read. That is a filter, and it is fine, because nothing unauthorized ever left the retrieval boundary.
The version I see more often runs the check after the documents have already been fetched. Once restricted text has been hydrated, reranked, summarized, or cached outside that boundary, authorization is chasing the problem instead of preventing it. AWS’s own guidance calls the broad-credential version of this a single point of failure because a prompt injection or a bug in the filtering logic can expose the whole dataset. And if it reached a model, the model has already read something the person was never entitled to retrieve, with any bug or injected instruction in that window free to act on it.
The defense most teams reach for first can make things worse. Jiale Liu, Jiahao Zhang, and Suhang Wang at Penn State red-teamed graph-based retrieval and found that summarization reduces leakage in untargeted attacks but can increase it in targeted attacks. My read of why is that summarizing preserves the salient detail, and the salient detail is usually the sensitive one. A separate 2026 preprint found cross-tenant leakage in pipelines that hand off from vector search to a graph, and eliminated it by re-checking authorization at every hop. Two individually secure components can still compose an insecure system when no one re-checks authorization at the transition between them.
Two individually secure components can still compose an insecure system when no one re-checks authorization at the transition between them.
The seventeen-hour window at the top of this piece is the same failure in slower motion. Direct shares, nested groups, and public links all change independently, which is why Google built Zanzibar as a relationship model rather than a list. A list of allowed users stamped on each chunk is a snapshot of a graph that moved without telling you.
None of this is fringe anymore. The OWASP Top 10 for LLM Applications moved sensitive information disclosure from sixth place to second in its 2025 revision and added LLM08, Vector and Embedding Weaknesses, which names the risk of context leaking between users who share a vector database and recommends a permission-aware store as the fix.
The enterprise-scale version of this is Copilot. In the first year of its enterprise rollout, a 2024 Gartner survey of 132 IT leaders found that oversharing led 40 percent to delay Microsoft 365 Copilot rollouts by 3 months or more. What makes that example useful is that Copilot is not the one doing the wrong thing. Microsoft checks the user’s permissions at query time, and its own documentation says results are trimmed to content the signed-in user has permission to access. Copilot surfaces what those people were already allowed to open.
A surprising amount of enterprise data stays private mainly because it is hard to find, and retrieval is very good at finding things.
The exposure was sitting there the whole time. A surprising amount of enterprise data stays private mainly because it is hard to find, and retrieval is very good at finding things.
Where identity has to arrive
So the lesson from Copilot is that resolving identity at assembly is necessary and not sufficient. Assembly inherits whatever the permission graph actually says. If the graph is wrong, stale, or too broad, the retrieval system will faithfully enforce the wrong answer. Homegrown retrieval can inherit the same problem, often with less governance tooling.
That does not weaken the case for assembly. It locates it. Assembly is not what makes your permissions correct. It is the last place where correct permissions still matter, because after that point the model has read the document.
I am not claiming to have invented this. AWS is arguing a version of it by governing the graph with the permissions the lake already has. OWASP got to the same place from the security side, and its recommended fix for LLM08 is a store that knows who is asking rather than a check that runs after the fact.
The part I would add comes from watching enterprise products make the jump from pilot to production. Teams can postpone many architectural decisions during a demo. They cannot postpone this one for very long. Eventually somebody asks who can see what, who guarantees it, how quickly a permission change propagates, and who owns the answer when three different systems disagree. That is often the moment when an impressive AI pilot turns into a security project, and it usually starts with something like an intern’s question.
So there are four questions I would put to any team building this.
Whether identity gets resolved at assembly or after retrieval.
How much of your context lives outside the lake, in chat and tickets and docs, where IAM does not reach.
What your worst-case staleness window looks like when someone changes teams.
And whether you can re-check authorization at every step along the way, or only once at the door.
If those answers are uncomfortable, that is useful. I have not had many of these conversations where they weren’t.
Researchers found that even a roughly seven-minute conversation with Google Gemini can reduce conspiracy beliefs about current crises, even when few verified facts are available. The effect beat a static fact sheet and, in follow-up surveys weeks later, carried over to beliefs about entirely different events.
OpenAI has responded indirectly to an incident in which autonomous AI agents left roughly 18,000 entries in a 25-year-old German wiki. The company says misalignment caused "new types of real-world impact" for the first time and plans to release a disclosure framework.
Self-identifying OpenAI agents posted 18,000 messages to a public wiki that discussed ways for other agents to bypass security sandbox restrictions during what was likely internal testing designed to gauge the agents’ hacking abilities, researchers said Friday.
In all, agents with 3,700 distinct self-given names posted the messages to German site DSEwiki over a six-week period. Besides discussing ways the agents could break out of the restricted environment OpenAI intended to prevent them from posting code or content to the Internet, the posts shared test answers. The posts also shared possible ways to perform XSS (cross-site scripting) attacks against the wiki and to impersonate site moderators. In three of the posts, agents used the word “swarm” to describe the collection of agents engaged in the activity.
Colluding to share answers
The research team—composed of Sydney Von Arx, Spencer Kitts, Thomas Larsen, and Cormac Slade Byrd—said they found the posts and pieced them together. The researchers say there are gaps in their understanding of precisely what actions the agents took because the research is based solely on the content of the posts. Additionally, the agents generated “chain of thought” data that’s understood only by OpenAI. As a result, the researchers said, they in some cases made educated guesses, including that the agents were, in fact, from OpenAI. In a statement, OpenAI later confirmed they were.
Enterprise work spans messages, decisions, projects, and obligations that change over time. An AI agent that starts without this context must reconstruct it...
Enterprise work spans messages, decisions, projects, and obligations that change over time. An AI agent that starts without this context must reconstruct it before contributing. To provide agents with this necessary context, our team used NVIDIA NemoClaw to build a memory-driven Chief of Staff. It maintains a human-readable knowledge layer called the self model: an agent memory of relevant…
AI agents are multiplying as corporations adopt the technology in record numbers. Smarter underlying models, better tool use, and improved multi-agent collaboration have pushed agents to evolve beyond impressive demos into practical technology that companies marshal in production environments. But the job’s not finished.
As companies deploy more agents, more often, and against longer tasks, the plumbing that provides their AI ephemera with the required information is buckling.
Here’s the problem: AI agents are sending waves of queries against company data, creating concurrency issues and exposing just how difficult it can be to ensure a company’s AI-legible information is fresh, served only when relevant, and quickly available.
And crucially, they will explore in this live conversation what changes when a team rebuilds it as a unified layer instead of a fragmented one.
Register for our free event on September 24
REGISTER NOW FOR THIS WEBINAR
By registering, you consent to The New Stack’s Privacy Policy, Terms of Use
and to receiving email communication from The New Stack and our event partner. You may opt out at any time.
You have successfully registered for the webinar.
You might be asking yourself: How has this problem not been solved yet? Google famously handles tens of thousands of search queries every second; how difficult can it be to serve agents the information that they need when we’ve solved the human version of the same problem? It’s no small challenge, and it’s why retrieval engineering is a labor category you’ll hear more about in coming quarters.
So, why is the problem worse with AI? Agents don’t ask a single question. They may retrieve data, reason against it, and then go back for more context. That doesn’t sound too complicated, until we recall that companies often stitch multiple systems together to provide their agents with required information. In practice, that means fusing vector databases, ranking tools, and serving layers into a single hybrid retrieval system that serves ever more agentic queries.
Worse, when several agents ping the same cobbled-together architecture at once, relevance drift becomes a real issue. You might do all the work to get your company or team up and running with agents, only to see the effort fail because of stale data, generic answers, or even truncated results as retrieval plumbing stumbles.
Your AI agents can’t scale successfully if they get dumber the more agents you deploy. So join the conversationon September 24, where we’ll break down how you can solve your retrieval engineering woes.
What you’ll take away:
Why agent workloads create a fundamentally different retrieval challenge than added concurrency alone
The specific failure modes at agent scale — latency stacking, stale context, relevance drift
Why fragmented retrieval stacks amplify those failures
What a unified retrieval architecture looks like in practice
A 438-billion-parameter reasoning model isn’t an obvious choice when speed is a priority. Multiverse Computing is betting that compression can make it one.
On Wednesday, the Spanish company launched Quasar 438B, its first large-scale model and a system designed specifically for coding and enterprise agents. Quasar achieves a score of 43 on Artificial Analysis’ Intelligence Index and 69.3 on Terminal-Bench v2.1, while Artificial Analysis currently records its output speed at approximately 183 tokens per second. Multiverse is positioning Quasar as the European model with the highest score on the Intelligence Index, outperforming Mistral Medium 3.5 (30) and NVIDIA Nemotron 3 Ultra (38).
Multiverse is betting that a 438B-parameter model can be fast and inexpensive enough for agents that repeatedly reason, call tools, and check the results.
That puts Quasar in an interesting middle ground. It doesn’t match the strongest models on coding performance, but Multiverse is betting that a 438B-parameter model can be fast and inexpensive enough for agents that repeatedly reason, call tools, and check the results.
The model features a context window of one million tokens, is available in English and Spanish, and can be accessed via the Multiverse CompactifAI API.
Compression claims, missing details
Multiverse built CompactifAI to shrink large AI models so they need less memory and compute to run. The company says it can reduce model size by 80% to 95% with only a small loss in accuracy, but it hasn’t disclosed how much Quasar was compressed or which model it started with.
In July, Multiverse announced a $570 million Series C to expand its library of compressed models and commercialize the technology. Quasar is the biggest test of that approach so far.
Multiverse hasn’t said what hardware is required to run Quasar or how much the compression reduces its memory and compute needs. That matters for agents, which may repeatedly call the model and other tools before finishing a task.
Multiverse hasn’t said what hardware it takes to run Quasar or how much the compression cuts its memory and compute needs.
Coding benchmarks show tradeoffs
In Multiverse’s comparison, Quasar’s Terminal-Bench v2.1 score of 69.3 places it ahead of Mistral Medium 3.5 but still well behind the best frontier systems, while Claude Opus 5 achieves the highest score on that benchmark at 89.1.
Multiverse is pitching Quasar for software engineering, technical copilots, research, and workflow automation. Its 1-million-token context window gives agents room to work with large codebases and hold onto information as a task progresses, although processing more context also requires more compute. That can be especially important in coding, where code that passes every test can still trip up the next AI agent if it loses track of what came before.
Agent latency beyond throughput
Artificial Analysis found that Quasar starts responding in about 1.1 seconds and can produce a 500-token response, including reasoning, in around 15.3 seconds. Those numbers are fast, but an agent also has to wait for tools, process growing context, and make repeated model calls over the course of a task. The agent tooling layer itself is still catching up to what these models need.
Those numbers are fast, but an agent also has to wait for tools, process growing context, and make repeated model calls over the course of a task.
Proprietary model, open questions
Quasar is proprietary and only available through Multiverse’s API, so developers can’t inspect the weights or run it on their own hardware. For now, that also makes it difficult to know whether the speed Multiverse is reporting will carry over to everyday agent use.
Perplexity wants its AI agents to use more of the computing power already sitting inside a Mac. The company launched Hybrid Compute on Tuesday, a new feature that lets Perplexity Computer move parts of the same task between powerful cloud models and smaller models running directly on Apple silicon.
The timing is significant. Hybrid Compute arrives on the first day of John Ternus’ tenure as Apple’s CEO. Ternus, who previously led Apple’s hardware engineering organization and played a key role in the company’s transition from Intel processors to its own Apple silicon, succeeds Tim Cook after 15 years as CEO. Perplexity is now betting that same hardware can become part of the infrastructure behind autonomous AI agents.
The task starts in the cloud. When a step involves sensitive information, Computer can move that part of the job to a model running on the Mac without having to start over.
When a step involves sensitive information, Computer can move that part of the job to a model running on the Mac without starting over.
Privacy Gate checks first
A Perplexity-trained Privacy Gate runs locally on the Mac and looks for sensitive information such as names, addresses, account numbers, and secrets. When it flags something, the user can decide whether that part of the task should stay on the device.
Users can review what the system wants to keep local before work begins and ensure it hasn’t missed anything they don’t want sent to the cloud. They can also choose which model handles the local work.
At launch, users can choose between Gemma E4B, Qwen3.6 35B-A3B, and a version of Qwen3.6 35B that Perplexity post-trained itself, with more models planned for later. Perplexity also handles the installation through the desktop app, so users don’t have to open a terminal or set up the model themselves.
Local tokens cost nothing
Once a task is running, the app displays local CPU, GPU, and memory utilization, as well as the number of tokens consumed. Users aren’t charged for tokens generated by models running locally on their Mac. That gives Perplexity another reason to move work onto the device beyond privacy, since running models locally can also cut inference costs. (The question of who bears inference costs and when is becoming a competitive issue across the industry.)
There is a compromise. Perplexity uses its most capable models in the cloud, while smaller models handle local work, so keeping more of a task on the Mac can improve privacy and cut costs at the expense of some capability. Hybrid Compute leaves that choice to the user.
Perplexity uses its most capable models in the cloud, while smaller models handle local work, so keeping more of a task on the Mac can improve privacy and cut costs at the expense of some capability.
Context crosses the boundary
The trickier bit is what happens to context when part of a task moves onto the Mac. The cloud model still needs to know enough about what happened locally to continue the job, without getting access to the private information that was supposed to stay there.
Perplexity says Computer can move a step from the cloud to a local model without restarting the task or losing context, and ultimately combine the cloud and local work into a single result. It does not detail in its announcement exactly what context is passed between those environments or how information produced by the local subagent is filtered before it returns to the broader workflow.
The local subagent can work with private files and data and take actions on the Mac. Users can also start a task on an iPhone and hand off local work to their Mac without having to start over.
For enterprise customers, Perplexity adds company-wide rules for what stays local and a record of what leaves each device, bringing the same governance questions facing AI agents down to the device level.
DGX Spark starts local
Hybrid Compute reverses the approach Perplexity introduced for Nvidia’s DGX Spark last week. DGX Spark starts locally and reaches out to frontier cloud models only with permission, while the Mac version starts in the cloud and moves work onto the device when needed.
In both cases, the agent harness decides where each part of the job runs. That orchestration — deciding which tools and context an agent actually needs — is becoming an increasingly difficult engineering problem as agents gain access to more systems and data.
Apple silicon becomes part of the agent stack
Running this much of an agent locally still requires a fairly powerful Mac. Perplexity recommends at least 32GB of unified memory, Apple silicon, and macOS 15. The feature is available to Pro and Max subscribers as well as enterprise customers.
Those requirements show the limits of local AI at the moment. Smaller models can run on plenty of Macs, but giving an agent enough compute to handle meaningful work still requires relatively high-end hardware.
That will likely change as Macs get better at running larger models. For Ternus, the bigger question is how much of the AI work now happening in the cloud will eventually move onto the machines Apple sells.
For Ternus, the bigger question is how much of the AI work now happening in the cloud will eventually move onto the machines Apple sells.
Login
