Google’s eighth-generation TPU announcement on Monday included more than new chips, introducing TPU Sync, an open-source system designed to speed up AI inference by tackling a problem that worsens as context windows grow: moving data between different types of memory.
The software is made for developers running large models, particularly ones handling million-token contexts and reinforcement-learning workloads. The company claims it can cut KV-cache transfer latency and time to first token by more than 90% in some tests.
The company claims it can cut KV-cache transfer latency and time to first token by more than 90% in some tests.
Inside the memory hierarchy
Written in C++, the framework moves data between accelerator memory, host DRAM, storage, and the network. It uses zero-copy transfers and native PjRtBuffer hardware descriptors to avoid unnecessary copies.
Combined host-to-device and device-to-host throughput reached 223 GB/s per Ironwood TPU host, approaching the theoretical 256 GB/s PCIe Gen5 limit.
Google’s numbers, with caveats
In Google’s benchmarks, moving KV caches for the 235-billion-parameter Qwen3 model went from taking several seconds to about 230 milliseconds. That cut the average time to first token (TTFT) by 60% when relatively few requests were running at once. With a larger 480-billion-parameter model, the reduction reached 90%.
Google saw similar gains when testing prefix caching with 64,000-token contexts. TTFT dropped by as much as 90%, throughput increased by more than 50%, and P99 inter-token latency fell 30%. Developers may see different results depending on what they’re running, how much cached data they can reuse, and how much network bandwidth they have available.
SemiAnalysis, offers some context for the gains. Its analysis suggests that external TPU serving previously lacked the infrastructure needed to separate prefill and decode workloads effectively.
Where Nvidia still leads
Nvidia may still have an advantage in some workloads. SemiAnalysis’s InferenceX data shows its GB200 and GB300 NVL72 systems ahead in parts of the disaggregated-serving performance curve, although a direct comparison with a fully tuned TPU v7 setup has yet to be published.
Chipping away at the CUDA moat
Developers can use the new software with the inference tools they already have since the company released the project alongside TorchTPU, with support for vLLM and SGLang, two widely used open-source inference engines. Developers already using those frameworks can run disaggregated workloads on TPUs without rebuilding their serving stack from scratch.
Developers already using those frameworks can run disaggregated workloads on TPUs without rebuilding their serving stack from scratch.
This also fits with Google’s broader effort to make its custom chips more competitive. The company has emphasized reducing inference costs through hardware changes, including reportedly designing chips around Gemini’s architecture.
Now it’s addressing the software problems that can leave those chips waiting for data. Making TPU infrastructure easier to use with familiar frameworks could give developers another reason to consider Google’s hardware instead of defaulting to Nvidia and CUDA; whether it can match Nvidia’s inference performance remains an open question.
Not production-ready yet
The software isn’t production-ready for everyone just yet. Google warns in its GitHub repository that the APIs are still changing and recommends sticking to a verified commit or tagged release. Support is also limited to Cloud TPU v7x and v6e, with specific JAX and PyTorch version requirements.
Support is also limited to Cloud TPU v7x and v6e, with specific JAX and PyTorch version requirements.
Together AI is putting AI coding agents to work on the GPU code that runs its inference infrastructure, automating performance tuning that used to take engineers days or weeks of manual work.
The company unveiled Together Compiler on Wednesday, a system that generates, tests, and improves GPU kernels without an engineer directing each experiment, and Together AI says it can deliver gains overnight. The process runs without a person steering each step, but it doesn’t ship anything on its own.
Asked how much people remain in the loop, Together AI tells The New Stack, “The optimization loop is autonomous,” adding that “the final evaluations and deployment process are automatic but must be kicked off by a human, who reviews the changes before deploying to production.”
“The optimization loop is autonomous. The final evaluations and deployment process … must be kicked off by a human, who reviews the changes before deploying to production.”
Why GPU kernels fall behind
New AI models arrive so frequently that keeping inference infrastructure optimized has become a challenge. Each model has different requirements, and GPU code tuned for one workload may not perform nearly as well on another.
Together AI says 80% of OpenRouter traffic now comes from models released within the past three months. That leaves engineers constantly adjusting code to keep up; now Together Compiler automates much of that work.
Multiple coding agents run experiments simultaneously, writing and testing GPU kernels while sharing results to help guide the next round of improvements.
Faster testing, more experiments
The company has also found a way to speed up testing. Instead of restarting an inference engine every time an agent changes the code, Together Compiler keeps the engine running and loads new code. According to Together AI, this cuts initialization time from roughly 467 seconds to just nine seconds.
Instead of restarting an inference engine every time an agent changes the code, Together Compiler keeps the engine running and loads new code.
Reading the 252x claim
Together AI says its compiler can run 252 times as many optimization experiments as existing agent-based systems. The company tells The New Stack that it used Apple’s RSI Inference Bench harness as its baseline, comparing both systems on the same inference workload and hardware, with only the optimization approach changing.
The 252x figure refers to the number of experiments Together Compiler can run compared with Apple’s system, not how much faster the AI models themselves perform.
The company separately reports inference throughput improvements of up to 2.6x compared with published benchmark configurations. Those gains depend on the model and hardware, and the results haven’t been independently verified.
More experiments give Together Compiler more chances to find improvements. And because each test takes less time, the agents can do overnight what might otherwise take engineers weeks of manual tuning.
Better performance, same cost
Faster inference doesn’t necessarily mean lower prices. Asked whether the performance gains would translate into customer savings or primarily improve its own infrastructure efficiency, Together AI tells The New Stack, “Together Compiler improves Together AI’s infrastructure efficiency. For the same dollar cost, customers will get better performance.”
“For the same dollar cost, customers will get better performance.”
Developers may get more work out of the same GPUs, but that doesn’t always mean lower per-token prices. The company tells The New Stack that customers will get better performance for the same cost, rather than a price reduction.
Together AI says its compiler is already serving trillions of tokens daily. It’s another sign that coding agents are moving beyond writing application code and into GPU performance optimization, with engineers reviewing work that agents can now complete overnight.
While it's not yet clear what kind of product the smartphone startup is cooking up, Nothing's CEO said the product is designed to leverage AI for productivity gains.
OpenAI's hardware team used GPT-6 Astra to design a device for the Codex coding agent in under a week, handling everything from the enclosure to the circuit board to the firmware. Several revision rounds were needed before production because the model didn't always account for manufacturing tolerances. A newer internal model scores significantly higher on OpenAI's engineering benchmark, though the company isn't sharing details.
Misumi Americas has launched Misumi Ventures, a $50 million early-stage investment fund targeting robotics, automation, advanced hardware, physical AI and industrial software companies. The San Francisco Bay Area-based fund plans to build a portfolio of 20 to 30 seed and early-stage companies, typically investing between $500,000 and $1.5 million initially and reserving capital for follow-on […]
At PyTorch Conference North America 2026 this week, Google open-sourced TorchTPU, a native PyTorch stack for Google TPUs, now available on GitHub. AWS also launched TorchNeuron in public beta, its open-source native PyTorch backend for AWS Trainium, also available on GitHub.
Both releases seem aimed at making it easier for developers to move beyond Nvidia GPUs, whose CUDA software ecosystem has long dominated AI development. But while Google and AWS’s new offerings do ease some of the challenges of switching hardware, getting top-tier performance still isn’t straightforward.
For developers eager for more computing options, it’s promising but somewhat disappointing.
Google wants to make TPUs more PyTorch-friendly
Google put TorchTPU in preview with select customers in April (PyTorch says Google’s been working on the project since mid-2025), teasing a simpler way to run PyTorch workloads on Google’s AI chips. PyTorch support has long been an awkward dance, so the new backend could make it easier to run existing models on TPUs.
Google says it wants to make its TPU software ecosystem more open and developer-friendly for the PyTorch community, as it works to make the chips a more practical option for developers already using PyTorch.
Developed in collaboration with the PyTorch team at Meta, TorchTPU allows developers to run existing PyTorch workloads without JAX dependencies in the runtime. This way, rather than requiring extensive code changes, developers can migrate existing workloads with only “trivial code changes,” according to PyTorch.
Specifically, developers can use eager execution to debug models and torch.compile to optimize TPU utilization and reduce memory overhead.
The new PyTorch stack from Google and Meta comes alongside a slew of other products and integrations announced at the conference, which Google says are “designed to help you run PyTorch effectively on Google Cloud or in your own infrastructure.” Google also announced integrations involving Hugging Face, vLLM, SGLang, llm-d, TorchTitan, Ray and TorchRec, while contributing LiteRT to the PyTorch ecosystem.
AWS wants in, too
AWS’s release extends the same PyTorch-native approach to its own AI chips.
Like TorchTPU, TorchNeuron gives developers a way to natively experiment and build models with PyTorch — this time for Trainium, AWS’s portfolio of AI chips designed for deep learning training and inference workloads. The public beta supports Trainium2 and Trainium3 instances.
AWS says moving a model to Trainium can be as simple as switching to .to(‘neuron’) for eager execution. Compiled models also need the Neuron backend selected, either as the default or in a torch.compile call.
TorchNeuron also integrates with Neuron Kernel Interface (NKI) — a bare-metal language and compiler for programming AWS Trainium and Inferentia NeuronDevices — so developers can create custom operators and optimize performance on Trainium. AWS and PyTorch have published more detailed guidance on the backend’s execution modes and performance tradeoffs.
Both are getting closer, but not there yet
While TorchTPU and TorchNeuron are notable steps forward in opening up more hardware choices for PyTorch developers, neither is a silver bullet.
PyTorch says a simple code change can get models running on TPUs, but peak performance can require TPU-specific optimization.
For distributed workloads, PyTorch says TorchTPU’s default Skinny Collectives approach can incur an end-to-end performance penalty of roughly 30–40% compared with an optimized approach because it prevents the XLA compiler from overlapping computation and communication.
For workloads seeking peak performance, PyTorch recommends additional tuning and production-grade stacks:
“Of course, for production workloads seeking peak performance we would absolutely recommend pursuing an optimal approach based on Pallas kernels and production-grade stacks such as vLLM-TorchTPU, SGLang-TorchTPU, or TorchTitan-TorchTPU, as examples.”
TorchNeuron is in public beta. In a separate blog post, AWS says teams can try Trainium with a configuration change instead of a code or model rewrite “in most cases,” leaving open which workloads need more changes.
In a case study, AWS describes how its Generative AI Innovation Center worked with AI research and technology company i14 to adapt its Laplace 3B model for Trainium2. The implementation required three changes: 1) using native PyTorch for the GLA attention kernel; 2) redesigning how Muon communicates; 3) compiling the sequential backward pass.
The releases give PyTorch developers more options beyond Nvidia GPUs. For now, moving existing code is easier than getting peak performance on unfamiliar chips.
DeepSeek has open-sourced the inference engine it uses to serve its models in production, a step it said last year wasn’t yet practical.
Dubbed DeepInfer, the framework is designed to let developers run DeepSeek’s V4.1 Flash model on their own infrastructure, with many of the same optimizations the company uses behind its API. Support for future DeepSeek models is also planned.
“Running a model efficiently is a different matter altogether.”
DeepSeek has, of course, long made its model weights available for developers to download and run using third-party software such as vLLM. But running a model efficiently is a different matter altogether. The software that manages requests, distributes work across chips, and generates responses can have a significant impact on speed, throughput and computing costs. And so DeepInfer essentially gives developers access to DeepSeek’s own approach to those challenges.
“We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.”
‘Making AI more affordable’
Taking to X on Tuesday, DeepSeek said it was making its internal inference technology available to outside developers in an effort to bring down the expense of running large AI models.
“We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable,” the company writes.
🚀 Introducing DeepInfer, our in-house inference framework, now fully open source!
Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more…
Released under the Apache 2.0 license, DeepInfer includes optimized computing kernels and communication libraries, including DeepSeek’s previously open-sourced FlashMLA, DeepGEMM, and DeepEP, alongside mechanisms for distributing inference across multiple chips. It also supports speculative decoding, a technique that speeds up response generation by predicting and verifying multiple tokens at once, and asynchronous scheduling to reduce the time processors spend waiting for new work. The framework also supports structured outputs, allowing developers to specify how model responses should be formatted.
Alongside DeepInfer, the company is also open-sourcing deepseek-recipe, a companion library that handles how API requests are converted into model inputs, and how the resulting output is processed into responses. Available for Rust and Python, it’s designed to help developers reproduce the behavior of DeepSeek’s official API when running their own deployments. The library was quietly released in September, with an updated version published on Tuesday.
The announcement comes just days after DeepSeek released an updated version of V4.1 Flash, called DeepSeek-V4.1-Flash-1017, touting improvements in coding performance and efficiency. That release coincided with the company making its in-house web search technology available to developers through new Search and Fetch APIs, as per The New Stack‘s report on Monday.
Why DeepSeek held back in 2025
DeepInfer goes further than DeepSeek’s previous plans to share its inference technology. Back in April 2025, the company published a document outlining its plans for its inference engine, explaining that it had considered releasing the entire system before deciding to contribute individual components and improvements to existing projects instead.
At the time, DeepSeek identified several obstacles. Its inference software had evolved from an early version of vLLM into a highly customized system, and the company questioned whether its relatively small research team could maintain another major open-source project. There were also practical difficulties in making the software usable outside DeepSeek’s own infrastructure.
“The engine is tightly coupled with our internal infrastructure, including cluster management tools, making it impractical for public deployment without significant modifications,” the company wrote at the time.
So what’s changed now, exactly? Well, DeepSeek hasn’t explained how it overcame those earlier difficulties, although the scope of the new release offers some clues. The DeepInfer documentation makes clear that the publicly available software currently supports offline inference, with “the core features needed for serving at scale,” as the company puts it. However, several components needed to operate a complete production service are still missing. These include a KV cache management system, infrastructure for separating different stages of inference across machines, and load balancing across replicas.
In other words, DeepSeek has opened up much of the engine responsible for running its models, while leaving developers to supply some of the surrounding infrastructure required to operate a full-fledged service. That narrower release may help explain how the company has made its technology available despite the challenges it identified last year.
Nvidia and Huawei, side by side
While the release of DeepSeek’s inference engine is significant enough in its own right, arguably the more interesting tidbit is the hardware supported from day one.
“Arguably the more interesting tidbit is the hardware supported from day one.”
Nvidia is included, as expected, though support is restricted to its SM100 architecture, used in Blackwell data center GPUs. Notably, the other supported platform is Huawei’s Ascend 950 NPU series, putting the Chinese chipmaker alongside Nvidia in DeepSeek’s list of compatible hardware. Older Nvidia architectures, including Hopper, are conspicuously absent.
This also ties back to DeepSeek’s published open-source plans from 2025. While the company was reluctant to release its entire inference engine at the time, it committed to working with hardware manufacturers ahead of future model launches, with the aim of making its models available across different chip architectures from day one.
US export restrictions have complicated Chinese companies’ access to Nvidia’s most advanced chips, adding another dimension to DeepSeek’s growing ties with Huawei. In September, Bloomberg reported that DeepSeek was planning to deploy at least 160,000 Huawei Ascend 950DT chips at a new data center in Inner Mongolia, specifically to run its models. The company has largely relied on Nvidia for training, although The Information reported in September that DeepSeek also plans to increase its use of Huawei chips for training, with deliveries expected as early as the fourth quarter.
Later that month, Reuters reported that DeepSeek and Huawei were collaborating on software tools intended to make Ascend processors easier to program, further underlining the company’s investment in Chinese AI hardware.
The Huawei inclusion has already attracted attention among the wider developer community. In a discussion on Reddit, one commenter singled out the two supported hardware platforms as a highlight of the news, while another pointed to DeepSeek’s origins in quantitative finance as a possible explanation for its technical expertise.
“Once again Deepseek shows how good high-frequency traders are at optimizing performance… On Ascend NPUs nonetheless!!,” the commenter wrote.
Another user seemingly poked fun at Anthropic’s Claude Opus 5.5, following reports that requests involving kernel programming for Huawei’s AI chips could trigger safeguards that redirect certain advanced AI development tasks to an older model.
“Don’t let Opus 5.5 see this! It will delete the Ascend support,” the user joked.
It is worth noting that feature support differs between the Nvidia and Huawei implementations. DeepInfer’s batch-invariant inference and deterministic sampling features, which help ensure reproducible outputs under specified conditions, are available only on Nvidia hardware. DeepSeek’s published performance measurements also use Nvidia GPUs, with four SM100 GPUs used for prefill and 64 for decoding, leaving open the question of how the framework performs on Huawei’s chips by comparison.
For developers without access to either chip family, the immediate value may lie in studying the newly available code and adapting its optimizations to other inference engines. For now, DeepSeek has given the wider open-source community considerably more of its inference technology to experiment with.
Microsoft is positioning the new Surface Laptop Ultra as an alternative to Apple’s MacBook Pros for creatives and developers — and especially AI developers.
Built on Nvidia’s N1X-based RTX Spark platform, with a Blackwell RTX GPU with up to 6,144 cores, a CPU with up to 20 cores, up to 128GB of unified memory, and a price that undercuts Apple, it’s a tempting alternative if you want to run local AI models and also want to have a powerful machine for editing videos and gaming on the go.
This is The New Stack, so I’m leaving the gaming benchmarks and creator use cases to others. My focus in testing the unit was on running AI workloads.
I’ve been using the Surface Laptop Ultra in its high-end $4,300 GPU and CPU configuration with 64GB of RAM and a 1TB SSD for the last week, including while on the road.
Typically, I use a Mac Studio with the M4 Max chip at home, which happily runs most local models I throw at it (as long as they fit into the 48 GB of RAM). I installed all the usual tools I use to run local models on the laptop. Think Unsloth Studio, Ollama, Hermes Agent, etc.
At no point did I feel like I was making a compromise in running those on the laptop. Indeed, if anything, the Blackwell GPU was generally a bit faster than my Mac Studio.
Fast local inference (with the right models)
Since the new Nemotron 3.5 Bolt model launched earlier this week, I also took that one for a spin. It answered queries at an impressive 56 to 60 tokens per second — and sometimes more. That makes it a great model for running in Hermes Agent or OpenCode, for example. It’s a heavily quantized model with 9B active parameters, so you would expect it to be fast, of course.
Yet, while the laptop has the unified memory to run it, the dense Qwen 3.8 27B at 8-bit quantization never achieved more than 8 or 9 tokens per second in Unsloth Studio. That’s not quite usable, but with these dense models, that’s virtually always due to memory bandwidth. But bringing it down to a 4-bit quantization brought the speed up to a respectable 25 to 28 tokens per second.
Using the 8-bit Gemma 4 26B-A4B shows how the mixture-of-experts technique is really what you need to run virtually any model locally at reasonable speeds. For this model, the Surface Laptop Ultra generally hit around 40 tokens per second.
I don’t have an M5-based MacBook or Mac Studio at hand, but Tom’s Hardware reports that the Ultra generally outperforms the M5 Max in a MacBook Pro in AI use cases.
Unsloth on the Surface Pro Ultra. Credit: The New Stack.
Performance drops when you unplug
All of those previous numbers were with the laptop on its 140W charger. Microsoft says it tuned the system so that running on battery would still give users most of the full performance. Microsoft claims its benchmarks showed that the laptop retained 99.6% of its performance when running on battery, but at least for the AI workloads I tested, that drop was far more noticeable.
The 4-bit Qwen 3.8 model, for example, dropped from its 25-28 tokens per second to around 18 tokens per second on battery. For Nemotron 3.5 Bolt, the performance dropped to around 42 to 45 tokens per second. Overall, it seems fair to expect around a 20% to 25% loss in performance when on battery when running AI workloads (and games show a very similar drop).
I did not run any formal battery life tests. Microsoft says users can expect 12 hours of active web usage. If you are consistently running local AI workloads, though, I wouldn’t expect more than a few hours of battery life based on my observations.
The RAM questions
Having 64GB of RAM puts you in a bit of an odd situation given that most smaller models right now are in the 27B to 30B league. The extra RAM gives you plenty of headroom for the KV cache and other applications, but there aren’t a lot of models in the 32-48GB range, for example, that the machine could also handle.
Going for 128GB of RAM brings the price of the Surface Laptop Ultra to $5,900
The trade-off for all that power: heat and noise
Unsurprisingly, all of the compute power (Microsoft says you get up to 1 petaflop of FP4 AI performance) creates a bit of heat.
Nobody is going to argue that the Surface Laptop Ultra is whisper-quiet when you run AI applications. The noise its dual fans make isn’t extreme, but enough to get a bit on your nerves while you are working. That’s especially noticeable when Hermes or your favorite coding agent is chugging along for a while on a more complex query.
There is a reason the Surface RTX Spark Dev Box, which is built on top of the same N1X platform, has a thousand air vents, after all (which will also likely allow it to push its performance beyond the N1X in the laptop).
Outside of that, the laptop is very quiet, though, and the fans remain off for most day-to-day use cases. It also quiets down very fast once the GPU load drops again.
The new Surface Magnetic Connect and 140W charging brick. Credit: The New Stack.
A few more observations
I would feel remiss if I didn’t highlight a few other things I found while testing the Ultra. This is, after all, Microsoft’s most ambitious laptop yet.
The touchpad is huge and has haptic feedback. There are only a few applications that support it so far, including DaVinci Resolve, for example. Having that feedback makes editing faster, as you can feel when the playhead snaps into place, for example. I want more touchpads with that feature.
Whatever Microsoft did with the speakers works wonders. There are side vents near the front of the laptop that create a surprisingly wide stereo field, something you will definitely notice in games.
The new magnetic USB-C charging connector is nice. It feels a lot like the magnetic Connect on other Surface devices. I just worry I’ll lose the cable sooner rather than later.
Microsoft is not positioning this as a gaming PC, but with that GPU, people will try to play a few games, and so did I. Gaming on a Windows ARM machine is still in its early days, but thanks to Microsoft’s emulation, it generally works without too much overhead. Games that rely more on the GPU than the CPU will fare better, but I was able to play Cyberpunk 2077 in 2K with ray tracing turned on at around 60fps. With frame generation and a few settings tweaks, even 4K gaming is an option.
With ray tracing off, the remaster of The Witcher 3 also happily ran at 60fps in 4K.
But since the 15-inch, 3:2 touchscreen (3270×2180) is essentially a better 3K screen, you wouldn’t want to run a game in 4K anyway unless you plug in an external monitor.
The screen looks great and can get very bright, but it is also highly reflective. That’s often distracting.
Overall, Windows on ARM feels a lot more solid now than it did a year ago. Surely there are some applications and games that won’t run (or run well), but I haven’t come across any issues. Increasingly, developers (like OpenCode) offer Windows ARM64 versions of the desktop apps.
Having two USB-C, one USB-A, and an HDMI port is such a joy if you have any older accessories. Add the headphone jack and SD card reader, and you don’t even need to carry a USB-C hub.
I’ve gotten used to my MacBook Air. The Laptop Ultra is heavy at 4.41lbs (2kg).
The SSD is replaceable. There are four screws underneath the laptop, and you don’t void your warrenty of you open the machine up. Apple should pay attention.
Open models and llm-d enabled IBM Research to explore quantum error correction hardware at 11.5× lower inference cost than an all-proprietary configuration.
Q.ANT, a startup out of Stuttgart, Germany, builds processors that use light instead of electricity to do some of the math behind AI. The company pitches them as a way to run AI on a fraction of the power today’s chips need.
Now developers can start writing software for those chips without owning one. Q.ANT pushed a free, open-source software kit to GitHub this week that lets developers build and test programs on a normal computer, then run them on the real chips once they get access.
This is a move out of Nvidia’s playbook. Nvidia owes its lead in AI as much to CUDA, the software developers use to program its GPUs, as it does to the chips themselves.
But with Q.ANT, the catch is the hardware. Q.ANT’s chips are running at a few research computing centers, and everyone else has to wait “the coming months” for cloud access through German provider IONOS or an on-site server from Q.ANT.
The kit, called the Q.ANT Native Computing Toolkit, is free on GitHub under a license that allows commercial use. Developers can work in Python or C. The key piece is a simulator that mimics the chip on a regular computer, with no Q.ANT drivers required.
What can it do today? The AI tools in this first version focus on running models that have already been trained. The examples read handwritten numbers, identify objects in photos and outline shapes in images. Training still happens on regular CPUs and GPUs.
The pitch for photonic computing is power. AI chips burn a lot of energy moving data back and forth between memory and the processor. Q.ANT’s chips do part of the math with light, specifically wave-shaped functions similar to a cosine, which regular chips calculate digitally. Q.ANT says AI models built around those functions get better results with fewer parameters, the settings a model learns during training. Fewer parameters means a smaller model, less data to move and less power. The kit includes examples comparing a standard model with one built Q.ANT’s way. Those comparisons are the company’s own.
“An ecosystem isn’t created by hardware alone. It emerges when the software layer is open and others can build on it,” said Michael Förtsch, Q.ANT’s founder and CEO. He calls the release the “Linux moment” of photonic computing.
Q.ANT is betting light can do the math itself. Lightmatter, one of the best-known companies in the field, now puts its focus on Passage, which uses light to move data between chips. The idea of light-based AI isn’t new, either. TNS covered MIT’s photonic processor for building optical neural networks back in 2017.
Q.ANT raised €62 million in July 2025 in a round led by Cherry Ventures, UVC Partners and imec.xpand. In March, it said its second-generation chips were running at the Leibniz Supercomputing Centre near Munich. The results it published from there compare the new chip with its old one: more than 50 times faster at the kind of math that does most of the work in AI models, and six times less energy on typical jobs, by the company’s numbers. Its bigger claims, like up to 30 times better energy efficiency, don’t say what they’re measured against.
Good software alone won’t carry a new chip. Nvidia has been building CUDA for nearly 20 years and is still adding to it, including deeper native Python support last year. Graphcore, the British AI chip startup, had its own software kit and still ended up being sold to SoftBank in 2024.
Q.ANT calls this the first openly available software kit for programming a photonic processor. That depends on how you count. Xanadu has offered free, open software for its light-based quantum computers since 2018. For now, developers can play with the simulator. What they can’t do yet is test Q.ANT’s power-saving claims on their own models. That has to wait until the chips open up.
The 1.58 in a 1.58-bit language model sounds like a hard limit, but Intel researchers pushed a ternary model below it by changing how its weights are stored rather than changing the model itself.
Their new BITCOS format compressed one checkpoint to 1.485 bits per weight and improved decoding throughput by as much as 18% on CPUs and 27% on GPUs.
The key is that the familiar 1.58-bit figure assumes a model uses its three possible weight values equally, while real ternary models contain far more zeros than that calculation accounts for. BITCOS stores the location and sign of each nonzero weight separately, allowing zeros to take up less space without retraining the model or altering its output — the equivalent of packing the same contents into a smaller box.
Where the 1.58-bit figure comes from
Ternary models use only three weight values — -1, 0, and +1 — and 1.58 bits is the theoretical minimum needed to represent three equally likely options. That number is cleaner than the reality of storing the weights, where the standard approach fits five ternary values into an eight-bit byte for an average of 1.6 bits each. Models commonly store weights in blocks of 128; however, this leaves the final byte partly unused and pushes the actual rate to 1.625 bits per weight.
When Intel’s researchers measured the distribution of weights across 29 checkpoints from seven ternary model families, they found that zeros accounted for between 29.7% and 51.5% of the weights. In 26 of those checkpoints, there were enough zeros for BITCOS to beat five-trit packing.
The sparsest was a ternary version of Qwen3-1.7B produced with CAT-Q post-training quantization, where 51.48% of the weights were zero, and BITCOS brought the storage cost down to 1.485 bits per weight.
Models commonly store weights in blocks of 128; however, this leaves the final byte partly unused and pushes the actual rate to 1.625 bits per weight.
How zeros save space
BITCOS stands for “BITmap and COmpacted Signs” and divides a model’s weights into two streams. The first assigns one bit to every weight to record whether it is zero or nonzero, while the second assigns a sign bit only to nonzero weights.
A positive or negative weight therefore consumes two bits, but a zero needs only the presence bit because it has no sign to record.
If z is the proportion of zero weights, BITCOS uses 2 − z bits per weight, dropping from 1.6 bits at 40% zeros to 1.485 bits at 51.5%. Because it changes only the storage format, unpacking restores the original -1, 0 and +1 values without affecting accuracy.
BITCOS becomes smaller than five-trit packing once more than 37.5% of a model’s weights are zero, a threshold reached by 26 of the 29 checkpoints Intel examined.
BITCOS becomes smaller than five-trit packing once more than 37.5% of a model’s weights are zero, a threshold reached by 26 of the 29 checkpoints Intel examined.
Making smaller weights run faster
Built for token-by-token decoding with small batch sizes, the format reduces the weight data moving through memory. Intel developed separate unpacking kernels for AVX-512 and AVX2 CPUs as well as Xe2 GPUs, joining other efforts to fit compressed models into faster inference pipelines for AI agents.
On AVX-512 hardware, the kernel uses the presence bitmap as a mask and pdep to scatter the compacted sign bits across the nonzero weight positions. Because Xe2 GPUs lack an equivalent instruction, Intel implemented the same operation with a 2KB lookup table.
Benchmarks across five systems
Compared with the 2-bit kernels, BITCOS ran 10% to 18% faster on the 64-core Xeon server and 2% to 15% faster on the 24-core Core Ultra 9. Performance improved by 9% to 22% on the integrated Arc 140V and by 2% to 27% on the discrete Arc Pro B70. These results measure decoding after the model has loaded, separate from efforts to cut GPU inference cold starts from minutes to seconds.
On the eight-core Lunar Lake CPU, Intel’s fixed 2-bit kernel beat BITCOS on every model because the system had enough bandwidth to make unpacking the bottleneck.
Format limits and open questions
The paper has not been peer-reviewed; all five test systems used Intel hardware, and the end-to-end benchmarks covered seven models at batch size one. Intel has yet to test the format on Nvidia, AMD, or Arm hardware.
Running an LLM on your PC is easy enough, but putting an agent to work there is a different story. Portable Computer, the local version of Perplexity’s Computer agent, is now available inside the Perplexity app for Windows on compatible Nvidia GeForce RTX and RTX PRO GPUs.
That’s the good news; the catch is, you’ll need an Nvidia GPU with at least 24GB of VRAM.
The Windows launch gives Perplexity three platforms in less than three weeks. Portable Computer debuted on Linux and Nvidia DGX Spark on August 25, followed a week later by hybrid compute for Apple silicon, which splits tasks between local and cloud models on Macs. Now Windows joins the mix, but bringing Portable Computer over took more than simply porting the app. Perplexity had to adapt the model runtime, orchestration, security, and hardware integration for each platform while keeping the user experience the same.
you’ll need an Nvidia GPU with at least 24GB of VRAM to use it.
Orchestration beyond the model
Portable Computer bundles those pieces together. On Windows, it supports PPLX 27B — Perplexity’s post-trained model — and Qwen 3.8 27B, both optimized for RTX GPUs, alongside a built-in browser, tool calling, and Perplexity’s proprietary SPACE sandbox.
It’s a different lane from LM Studio or Ollama, which make running models locally as painless as possible but stop well short of giving a model autonomy over multistep work. DeepSeek’s recent hiring spree of roughly 150 new roles, nearly all of them focused on agent infrastructure rather than the model, hints at how much engineering sits between a capable model and a capable agent.
DeepSeek’s recent hiring spree of roughly 150 new roles, nearly all of them focused on agent infrastructure rather than the model, hints at how much engineering sits between a capable model and a capable agent.
Connectors blur local boundaries
Perplexity ships connectors for Microsoft Outlook, OneDrive, and Word, plus Google Drive, Gmail, Slack, and GitHub — which tells you something about what “local” actually means here.
The agent can reach external services because it’s not air-gapped. Locally completed tasks can process files without sending documents to a cloud model. Once an agent has access to both local files and remote APIs on the same machine, figuring out which resources it actually needs and where to find them gets harder.
Hybrid cloud as fallback
Perplexity isn’t pretending that a 27-billion-parameter model running on a desktop GPU can handle everything, which explains the hybrid architecture. When the agent determines that a task needs more reasoning power than the local model can deliver, it can escalate to Perplexity’s cloud models.
According to Nvidia, the agent identifies when cloud support would help and asks the user for permission before sending any data off the machine.
For organizations handling sensitive or regulated data, that split can make all the difference. A local agent can grind through source code or financial records without uploading them to a hosted model for basic processing. There’s a cost angle too, since tasks completed locally don’t burn Perplexity Computer credits.
High VRAM floor limits reach
Portable Computer is available with Perplexity Pro ($20/month) and Max ($200/month), across individual and enterprise plans, with Nvidia DGX Station support coming later. The real challenge is taking local agents from developer passion projects to enterprise-ready tools. By baking this into Windows, it immediately gets in front of the scale of users needed to make that happen.
The real challenge is taking local agents from developer passion projects to enterprise-ready tools.
Perplexity wants its AI agents to use more of the computing power already sitting inside a Mac. The company launched Hybrid Compute on Tuesday, a new feature that lets Perplexity Computer move parts of the same task between powerful cloud models and smaller models running directly on Apple silicon.
The timing is significant. Hybrid Compute arrives on the first day of John Ternus’ tenure as Apple’s CEO. Ternus, who previously led Apple’s hardware engineering organization and played a key role in the company’s transition from Intel processors to its own Apple silicon, succeeds Tim Cook after 15 years as CEO. Perplexity is now betting that same hardware can become part of the infrastructure behind autonomous AI agents.
The task starts in the cloud. When a step involves sensitive information, Computer can move that part of the job to a model running on the Mac without having to start over.
When a step involves sensitive information, Computer can move that part of the job to a model running on the Mac without starting over.
Privacy Gate checks first
A Perplexity-trained Privacy Gate runs locally on the Mac and looks for sensitive information such as names, addresses, account numbers, and secrets. When it flags something, the user can decide whether that part of the task should stay on the device.
Users can review what the system wants to keep local before work begins and ensure it hasn’t missed anything they don’t want sent to the cloud. They can also choose which model handles the local work.
At launch, users can choose between Gemma E4B, Qwen3.6 35B-A3B, and a version of Qwen3.6 35B that Perplexity post-trained itself, with more models planned for later. Perplexity also handles the installation through the desktop app, so users don’t have to open a terminal or set up the model themselves.
Local tokens cost nothing
Once a task is running, the app displays local CPU, GPU, and memory utilization, as well as the number of tokens consumed. Users aren’t charged for tokens generated by models running locally on their Mac. That gives Perplexity another reason to move work onto the device beyond privacy, since running models locally can also cut inference costs. (The question of who bears inference costs and when is becoming a competitive issue across the industry.)
There is a compromise. Perplexity uses its most capable models in the cloud, while smaller models handle local work, so keeping more of a task on the Mac can improve privacy and cut costs at the expense of some capability. Hybrid Compute leaves that choice to the user.
Perplexity uses its most capable models in the cloud, while smaller models handle local work, so keeping more of a task on the Mac can improve privacy and cut costs at the expense of some capability.
Context crosses the boundary
The trickier bit is what happens to context when part of a task moves onto the Mac. The cloud model still needs to know enough about what happened locally to continue the job, without getting access to the private information that was supposed to stay there.
Perplexity says Computer can move a step from the cloud to a local model without restarting the task or losing context, and ultimately combine the cloud and local work into a single result. It does not detail in its announcement exactly what context is passed between those environments or how information produced by the local subagent is filtered before it returns to the broader workflow.
The local subagent can work with private files and data and take actions on the Mac. Users can also start a task on an iPhone and hand off local work to their Mac without having to start over.
For enterprise customers, Perplexity adds company-wide rules for what stays local and a record of what leaves each device, bringing the same governance questions facing AI agents down to the device level.
DGX Spark starts local
Hybrid Compute reverses the approach Perplexity introduced for Nvidia’s DGX Spark last week. DGX Spark starts locally and reaches out to frontier cloud models only with permission, while the Mac version starts in the cloud and moves work onto the device when needed.
In both cases, the agent harness decides where each part of the job runs. That orchestration — deciding which tools and context an agent actually needs — is becoming an increasingly difficult engineering problem as agents gain access to more systems and data.
Apple silicon becomes part of the agent stack
Running this much of an agent locally still requires a fairly powerful Mac. Perplexity recommends at least 32GB of unified memory, Apple silicon, and macOS 15. The feature is available to Pro and Max subscribers as well as enterprise customers.
Those requirements show the limits of local AI at the moment. Smaller models can run on plenty of Macs, but giving an agent enough compute to handle meaningful work still requires relatively high-end hardware.
That will likely change as Macs get better at running larger models. For Ternus, the bigger question is how much of the AI work now happening in the cloud will eventually move onto the machines Apple sells.
For Ternus, the bigger question is how much of the AI work now happening in the cloud will eventually move onto the machines Apple sells.
SpaceX and Nvidia say they are adapting the Vera Rubin NVL72 rack-scale AI platform for orbital use, with SpaceX targeting a first launch in the fourth quarter of 2027.
The dream of an AI data center in space lives on in SpaceX and Nvidia’s August 24 announcements that the platform for Low Earth Orbit (LEO) Starmind AI satellites will be based on the Vera Rubin NVL72 chip family and architecture.
SpaceX CEO Elon Musk posted on X the same day, “SpaceX, in partnership with Nvidia, has designed a space-optimized Vera Rubin NVL72 system for launch to orbit in Q4 next year, with significant scale in 2028.”
SpaceX, in partnership with Nvidia, has designed a space-optimized Vera Rubin NVL72 system for launch to orbit in Q4 next year, with significant scale in 2028 https://t.co/qdDq8YBkzl
Musk’s post came after he said during SpaceX’s Q2 earnings call, “Going forward, we’ve decided to build exclusively on Nvidia because we think the Vera Rubin architecture is the best architecture.” Musk continued, “This is not some sort of far-future, distant thing; we expect to start launching these next year. We think the design of the NVL72 VR computer is a much better design than, say, having a standard rack -style design. So we expect to deploy this on the ground as well as in orbit, because we think it’s going to be a radical simplification of the standard NVL72 rack. It will cost less. It will be more effective. If we’re going to put it in space, why not want to put it on the ground? I think that’s going to be pretty cool.”
On Earth, the Vera Rubin NVL72 is Nvidia’s rack-scale AI design that combines 72 Rubin GPUs and 36 Vera CPUs, alongside high-speed networking components such as ConnectX-9 SuperNICs. Nvidia says SpaceXAI’s planned Starmind satellite will be based on an optimized version of that system.
A conventional NVL72 rack assumes gravity, technicians, stable grid power, a building-scale liquid loop and frequent replacement of failed parts. Orbit removes each of these assumptions.
The idea is more ambitious than putting a conventional edge-AI accelerator aboard a spacecraft. Nvidia and SpaceXAI are proposing to bring a modified architecture used in AI data centers into orbit, while altering it for orbital operational requirements.
Getting that working in orbit, though, is easier said than done.
“Cooling is unforgiving. Space is cold, but vacuum does not carry heat away through convection.”
In particular, Pyke continues, “Cooling is unforgiving. Space is cold, but vacuum does not carry heat away through convection. Heat must travel from the chips to the radiator surfaces and then leave as infrared radiation. SpaceX says AI1 can avoid chillers, cooling towers and fans and reduce cooling overhead by an order of magnitude.”
SpaceX explains that AI1 would instead use closed-loop liquid cooling inside the spacecraft and large deployable radiators to send heat directly to space as infrared radiation. While the claimed reduction is physically plausible in principle, there’s no proof yet that these AI satellites’ cooling systems can deliver.
Another major problem that remains unaddressed is how to make the orbital rack radiation-tolerant. Making Vera Rubin NVL72 radiation-tolerant means far more than putting an ordinary NVL72 rack in a shielded satellite enclosure. It would require a system-level redesign of its GPUs, CPUs, memory, networking, power, cooling, firmware, and operations around a specified orbit and mission life.
Even were Nvidia to make such a chip, for a high-density AI system such as the SpaceX design, the concern isn’t simply whether one processor survives a 5- or 10-year dose. The satellite contains numerous radiation-sensitive elements, such as GPU logic, SRAM caches, register files, system memory, and memory controllers. With thousands of cores and billions of memory storage cells, the aggregate fault rate — not the behavior of an individual component — drives the design.
The most realistic near-term answer would be a radiation-tolerant, fault-managed Rubin-derived orbital system, not a fully radiation-hardened NVL72 in the traditional military-space sense. It could use selected commercial Nvidia parts, substantial shielding, ECC and data integrity mechanisms, redundant controllers and power paths, aggressive fault detection, software recovery, and reduced-performance operating modes.
Nvidia has reportedly agreed to buy Hugging Face for $12.9 billion, putting one of the biggest names in AI hardware in charge of a platform developers rely on to find and run open models.
The Information first reported the deal Wednesday, citing a person familiar with the agreement. Nvidia and Hugging Face had not publicly confirmed it as of publication.
Hugging Face doesn’t push developers toward one chipmaker, which is what makes the acquisition interesting. Its Optimum libraries work with Nvidia’s TensorRT-LLM and also support hardware from AMD, Intel, and AWS. Projects such as Optimum AMD and Optimum Intel let developers run Transformers and Diffusers models on non-Nvidia hardware.
The company also plays a role in what happens after a developer chooses a model, including how easily they can get it running on the hardware they want to use. The problem for Nvidia might be this: It is buying a platform whose value depends on openness and hardware neutrality, but if the purchase means Nvidia hardware is favored, that value might diminish.
The company also plays a role in what happens after a developer chooses a model, including how easily they can get it running on the hardware they want to use.
Hugging Face already sits between the model and the chip
Hugging Face has expanded well beyond file hosting. With Inference Endpoints, developers can deploy a model from the Hub while Hugging Face handles the underlying infrastructure.
Those hosted deployments can run on AWS, Microsoft Azure or Google Cloud, but most of the GPU options Hugging Face lists are Nvidia chips, including the T4, L4 and A100. That gives developers a wider choice of hardware through Hugging Face’s open-source libraries than through its hosted services.
If the deal goes through, Nvidia would own both sides of that experience.
Deployment defaults favor Nvidia
NIM (Nvidia Inference Microservices) already works with models hosted on Hugging Face. Developers can point NIM to an hf:// repository path and pull the model directly from the Hub.
Owning Hugging Face would give Nvidia more room to bring NIM and CUDA-optimized containers directly into the deployment experience. Nvidia hasn’t announced plans to make NIM the default, and support for AMD and Intel could remain exactly where it is.
Owning Hugging Face would give Nvidia more room to bring NIM and CUDA-optimized containers directly into the deployment experience.
The bigger question is what happens over time. Nvidia could provide earlier support for new models on its own hardware or make deployment easier. At the same time, AMD, Intel, and AWS may have to reconsider how much engineering work they want to contribute to integrations maintained within a competitor-owned platform. Some of that work could eventually move elsewhere.
For developers, the difference may come down to which path requires less work. A competing chip doesn’t have to disappear from Hugging Face to become less appealing if an Nvidia model deployment takes fewer steps. We’ve seen a similar fight over the layers between AI models and the developers using them, with Cloudflare building more of that infrastructure itself.
Open models counter custom chips
That tension also helps explain why Hugging Face could be worth considerably more to Nvidia than its revenue alone would suggest.
Nvidia has been expanding its own Nemotron family of open models while investing heavily across the AI ecosystem. At the same time, some of its biggest customers are working to reduce their dependence on Nvidia hardware.
Open models are often expected to run in very different environments, and Hugging Face helps developers make that possible. A model found on the Hub might end up running on Nvidia hardware, an AMD GPU, or a cloud accelerator.
That flexibility is part of what Nvidia would be buying. Pushing Hugging Face too heavily toward its own hardware could make the platform less useful to developers who rely on it to work across different systems.
Interest in those models is also growing. Models from companies including DeepSeek, Moonshot AI and Z.ai have narrowed the gap with proprietary systems. At the same time, Hugging Face CEO Clément Delangue told The Information in June that the company had doubled its number of paying subscribers during the first six months of 2026. Delangue later said the company was “close to profitability.”
The Information puts Hugging Face’s annualized revenue at about $150 million. Against a $12.9 billion price tag, that’s a multiple of roughly 86.
So Nvidia would be paying for much more than Hugging Face’s current business. It would be buying a place developers already turn to when they want to work with open models, including models that don’t have to run on Nvidia hardware.
It would be buying a place developers already turn to when they want to work with open models, including models that don’t have to run on Nvidia hardware.
Community resists easy forking
Buying Hugging Face wouldn’t give Nvidia control over everything developers find there. Libraries such as transformers and diffusers are open source, and models on the Hub remain subject to their own licenses. Openly licensed models can still be hosted elsewhere, while the underlying libraries can be forked.
Much harder to recreate is the community Hugging Face has built around them. Developers already know where to look for models and have built workflows around the Hub and its integrations.