❌

Reading view

AI needs fresh data. Vector indexing is getting in the way

Red, yellow, and white light trails curving across dark asphalt, evoking fast data flow for AI vector indexing

It’s not a revelation to say AI workflows are changing enterprise infrastructure. What gets less attention, though, is how they’re also shifting the fundamentals of data processing and search.

The complication lies in the data. As organizations build generative AI (GenAI) applications, retrieval-augmented generation (RAG) systems, and AI agents, they’re generating an unwieldy number of vectors from growing volumes of unstructured data. And with all that vectorized data comes a challenge: how to quickly find the right information to power AI applications and workflows. 

“After all, a model can only reason over the context you can find and feed it.”

Whether it’s a RAG system answering a question or an AI agent taking action, enterprise AI applications succeed on retrieval. After all, a model can only reason over the context you can find and feed it. If those systems rely on wrong or outdated information, even the best model can’t fix the underlying context problem. 

But finding the right context inside petabytes of enterprise data is a difficult search problem — one keyword search alone is no match. 

Search has moved beyond keywords

In recent years, search has come a long way from basic keyword-based matching. For one, user behavior has spearheaded a push from simple keyword entries to natural language queries. Agent workloads, meanwhile, which query 10 to 100 times more than humans, bring a new dimension to search — and traditional databases are not prepared to keep up.

While full-text search uses analyzers and relevance tuning to find exact words and phrases that match the index, semantic search goes beyond keywords to focus on meaning and intent, letting users search with everyday language or synonyms instead of exact index terms. Vector search, meanwhile, enables similarity search, using vector embeddings (i.e., numerical representations of data) stored in vector fields to find items that are semantically similar rather than exact matches. 

Hybrid search joins keyword and vector search, letting users combine exact keyword matches with meaning-based searches in a single result set. 

In the new phase of AI adoption, where applications increasingly rely on swathes of unstructured data to retrieve context, traditional keyword search is no longer enough to surface the right information on its own. As a result, organizations are increasingly turning to high-performance vector databases, semantic search, and hybrid search to support new AI workloads. 

Therein lies the next challenge: indexing all those vectors so they can be searched.

Why indexing is the new bottleneck

As organizations move from gigabyte-scale search to petabyte-scale AI workloads, the ability to quickly index vectors has become a major factor in determining whether AI applications have current information to work with. 

The problem is, indexing is often a slow, compute-intensive process, which means it can take days or even weeks for incoming vectors to become searchable. 

“For customers who want to search across petabytes of data, they may notice that it takes days to weeks to index.”

As Rachael Wade, global tech alliances manager at Elastic, tells The New Stack, “For customers who want to search across petabytes of data, they may notice that it takes days to weeks to index.” While a longer indexing timetable may work fine for smaller AI pilots, once it’s time to move those projects into production, indexing delays can significantly stall deployment timelines. 

A big part of the challenge lies in building the vector index, or HNSW (Hierarchical Navigable Small World) graph, a layered data structure that connects similar vectors to enable efficient search. During construction, vectors must be compared to determine how they should connect in the graph. Because this process can involve millions or even billions of vectors, building a vector index at enterprise scale is compute-intensive and requires a dizzying number of mathematical operations. 

What happens when enterprises don’t have the compute to keep up? Indexing is slow, so new information takes longer to become searchable, leaving RAG systems, queries, and agents potentially working on stale data. 

“…the new constraint is getting vectors indexed fast enough to keep that search current.”

One example Wade points out to The New Stack is a product catalog, where e-commerce companies rely on fresh data to keep production information up to date and searchable. Manas Singh, technical product manager for accelerated ML and vector search at Nvidia, agrees, adding that, for many companies, “the new constraint is getting vectors indexed fast enough to keep that search current.”

It’s time for GPU-accelerated indexing

While CPUs have been the default for building vector indexes, IT leaders are starting to realize that CPU-based indexing can struggle to meet the scale and data-freshness requirements of modern AI workloads. 

For one, indexing vectorized data at petabyte scale can be painstakingly slow, taking days or even weeks. It’s also costly; when you need to index hundreds of millions or billions of vectors, CPU hardware expenses can add up fast. 

But where CPU-based indexing can be slow and expensive at scale, GPUs are designed for this very kind of large-scale, high-intensity work. For AI workloads with large data volumes and high ingest rates, that makes accelerated computing GPUs built for massive parallelism and high-throughput math ideally positioned for the job. 

At least that’s the idea behind Elastic and Nvidia’s collaboration on GPU-accelerated vector indexing for Elasticsearch. 

The integration is powered by Nvidia cuVS, an open-source NVIDIA CUDA-XTM library for GPU-accelerated vector search and data clustering that enables faster index building and embedding retrieval for AI and recommendation workloads. Elasticsearch accesses cuVS through cuvs-java, another open-source library maintained by Nvidia.

Elastic, for its part, brings semantic and hybrid search to the table via its enterprise-focused vector database — one of the most widely used in the world that powers semantic search, AI assistants, and recommendations for both startups and enterprises. Developers can trial Elasticsearch for free.

By accelerating vector indexing, the Elastic and Nvidia collaboration aims to help enterprises index massive volumes of vectorized data faster so it becomes available sooner for semantic search and AI workloads, including GenAI and AI agents, at speed and scale. 

The payoff of faster indexing can be substantial. Per Elastic, moving indexing construction to GPU with Nvidia cuVS delivers nearly a 12x boost in indexing throughput and a 7x reduction in force-merge latency, i.e., the time it takes to consolidate small index segments.

To put the cost difference in perspective, Wade points to a cost-model example comparing a GPU-based setup with CPU infrastructure. As she tells The New Stack, one GPU, depending on the model, can index hundreds of millions of vectors within a roughly 10-minute SLA. Doing the same work with CPUs, she says, would take about 10 CPUs and require roughly two hours — amounting to what she says is a baseline total cost of ownership savings of 78%. 

The benefits of GPU acceleration become even more pronounced as vector dimensions increase. Because CPUs take significantly longer to process vectors at higher dimensions, GPU acceleration — whether that’s 5x, 6x, 7x, or even 12x — becomes more valuable at larger scales. 

Why faster indexing matters now

For on-premises deployments, Singh also highlights the indirect benefit of lower search latency. 

By shifting indexing work from CPUs to faster, more efficient GPUs, enterprises can free up CPUs for other work, thereby allowing the same on-premises hardware to dedicate more resources to search operations. As Singh puts it: “You free up the CPU resources for other things,” which he says can ultimately lead to lower search latency. 

Meanwhile, because GPU-accelerated vector indexing is available in self-managed Elasticsearch environments, including on-premises, developers can take advantage of faster indexing without having to move workloads off infrastructure they control. 

Altogether, for IT leaders evaluating hardware with an eye on maximizing ROI and maintaining data sovereignty, that means moving indexing from CPUs to GPUs is a triple win: It can complete more indexing work in less time with less infrastructure, while also meeting self-managed, on-premises deployment requirements and freeing up CPU capacity for search. 

In the bigger picture, the move toward GPU-accelerated indexing reflects how AI is changing data processing and search. For RAG systems, AI agents, and other retrieval-dependent applications, getting new data searchable quickly is critical to making those systems truly useful. And that starts with faster indexing. 

The post AI needs fresh data. Vector indexing is getting in the way appeared first on The New Stack.

  •  

Opinion: Moravec’s paradox explains why physical AI needs a sense of touch

Ron Zukerman, innovation director at Vishay Precision Group When people first started trying to build human‑like robots, the roadmap seemed clear. The assumption was simple: if something feels hard for us, it must be hard for machines too. So, we treated things like playing chess, doing advanced math, or analysing legal arguments as the “real” […]
  •  

How backlash and red tape continue to slow data centers post-Blue Wave

The data center boom is running into serious real-world obstacles, from widespread local opposition and permitting delays to a scramble for electricity. Before those challenges even come into play, though, there’s a more basic problem: A lot of the projects being announced may never get built in the first place. With billions of dollars in AI infrastructure […]
  •  

Google claims its new AI software cuts inference latency by over 90% — but there’s a catch

server rack

Google’s eighth-generation TPU announcement on Monday included more than new chips, introducing TPU Sync, an open-source system designed to speed up AI inference by tackling a problem that worsens as context windows grow: moving data between different types of memory.

The software is made for developers running large models, particularly ones handling million-token contexts and reinforcement-learning workloads. The company claims it can cut KV-cache transfer latency and time to first token by more than 90% in some tests.

The company claims it can cut KV-cache transfer latency and time to first token by more than 90% in some tests.

Inside the memory hierarchy

Written in C++, the framework moves data between accelerator memory, host DRAM, storage, and the network. It uses zero-copy transfers and native PjRtBuffer hardware descriptors to avoid unnecessary copies.

Combined host-to-device and device-to-host throughput reached 223 GB/s per Ironwood TPU host, approaching the theoretical 256 GB/s PCIe Gen5 limit.

Google’s numbers, with caveats

In Google’s benchmarks, moving KV caches for the 235-billion-parameter Qwen3 model went from taking several seconds to about 230 milliseconds. That cut the average time to first token (TTFT) by 60% when relatively few requests were running at once. With a larger 480-billion-parameter model, the reduction reached 90%.

Google saw similar gains when testing prefix caching with 64,000-token contexts. TTFT dropped by as much as 90%, throughput increased by more than 50%, and P99 inter-token latency fell 30%. Developers may see different results depending on what they’re running, how much cached data they can reuse, and how much network bandwidth they have available.

SemiAnalysis, offers some context for the gains. Its analysis suggests that external TPU serving previously lacked the infrastructure needed to separate prefill and decode workloads effectively.

Where Nvidia still leads

Nvidia may still have an advantage in some workloads. SemiAnalysis’s InferenceX data shows its GB200 and GB300 NVL72 systems ahead in parts of the disaggregated-serving performance curve, although a direct comparison with a fully tuned TPU v7 setup has yet to be published.

Chipping away at the CUDA moat

Developers can use the new software with the inference tools they already have since the company released the project alongside TorchTPU, with support for vLLM and SGLang, two widely used open-source inference engines. Developers already using those frameworks can run disaggregated workloads on TPUs without rebuilding their serving stack from scratch.

Developers already using those frameworks can run disaggregated workloads on TPUs without rebuilding their serving stack from scratch.

This also fits with Google’s broader effort to make its custom chips more competitive. The company has emphasized reducing inference costs through hardware changes, including reportedly designing chips around Gemini’s architecture.

Now it’s addressing the software problems that can leave those chips waiting for data. Making TPU infrastructure easier to use with familiar frameworks could give developers another reason to consider Google’s hardware instead of defaulting to Nvidia and CUDA; whether it can match Nvidia’s inference performance remains an open question.

Not production-ready yet

The software isn’t production-ready for everyone just yet. Google warns in its GitHub repository that the APIs are still changing and recommends sticking to a verified commit or tagged release. Support is also limited to Cloud TPU v7x and v6e, with specific JAX and PyTorch version requirements.

Support is also limited to Cloud TPU v7x and v6e, with specific JAX and PyTorch version requirements.

The post Google claims its new AI software cuts inference latency by over 90% — but there’s a catch appeared first on The New Stack.

  •  

Together AI let coding agents rewrite its GPU code. Then it kept humans on the deploy button.

Illustration of an open head with circuits

Together AI is putting AI coding agents to work on the GPU code that runs its inference infrastructure, automating performance tuning that used to take engineers days or weeks of manual work.

The company unveiled Together Compiler on Wednesday, a system that generates, tests, and improves GPU kernels without an engineer directing each experiment, and Together AI says it can deliver gains overnight. The process runs without a person steering each step, but it doesn’t ship anything on its own.

Asked how much people remain in the loop, Together AI tells The New Stack, “The optimization loop is autonomous,” adding that “the final evaluations and deployment process are automatic but must be kicked off by a human, who reviews the changes before deploying to production.”

“The optimization loop is autonomous. The final evaluations and deployment process … must be kicked off by a human, who reviews the changes before deploying to production.”

Why GPU kernels fall behind

New AI models arrive so frequently that keeping inference infrastructure optimized has become a challenge. Each model has different requirements, and GPU code tuned for one workload may not perform nearly as well on another.

Together AI says 80% of OpenRouter traffic now comes from models released within the past three months. That leaves engineers constantly adjusting code to keep up; now Together Compiler automates much of that work.

Multiple coding agents run experiments simultaneously, writing and testing GPU kernels while sharing results to help guide the next round of improvements.

Faster testing, more experiments

The company has also found a way to speed up testing. Instead of restarting an inference engine every time an agent changes the code, Together Compiler keeps the engine running and loads new code. According to Together AI, this cuts initialization time from roughly 467 seconds to just nine seconds.

Instead of restarting an inference engine every time an agent changes the code, Together Compiler keeps the engine running and loads new code.

Reading the 252x claim

Together AI says its compiler can run 252 times as many optimization experiments as existing agent-based systems. The company tells The New Stack that it used Apple’s RSI Inference Bench harness as its baseline, comparing both systems on the same inference workload and hardware, with only the optimization approach changing.

The 252x figure refers to the number of experiments Together Compiler can run compared with Apple’s system, not how much faster the AI models themselves perform.

The company separately reports inference throughput improvements of up to 2.6x compared with published benchmark configurations. Those gains depend on the model and hardware, and the results haven’t been independently verified.

More experiments give Together Compiler more chances to find improvements. And because each test takes less time, the agents can do overnight what might otherwise take engineers weeks of manual tuning.

Better performance, same cost

Faster inference doesn’t necessarily mean lower prices. Asked whether the performance gains would translate into customer savings or primarily improve its own infrastructure efficiency, Together AI tells The New Stack, “Together Compiler improves Together AI’s infrastructure efficiency. For the same dollar cost, customers will get better performance.”

“For the same dollar cost, customers will get better performance.”

Developers may get more work out of the same GPUs, but that doesn’t always mean lower per-token prices. The company tells The New Stack that customers will get better performance for the same cost, rather than a price reduction.

Together AI says its compiler is already serving trillions of tokens daily. It’s another sign that coding agents are moving beyond writing application code and into GPU performance optimization, with engineers reviewing work that agents can now complete overnight.

The post Together AI let coding agents rewrite its GPU code. Then it kept humans on the deploy button. appeared first on The New Stack.

  •  

OpenAI says GPT-6 Astra designed a hardware device from scratch in under a week

OpenAI's hardware team used GPT-6 Astra to design a device for the Codex coding agent in under a week, handling everything from the enclosure to the circuit board to the firmware. Several revision rounds were needed before production because the model didn't always account for manufacturing tolerances. A newer internal model scores significantly higher on OpenAI's engineering benchmark, though the company isn't sharing details.

The article OpenAI says GPT-6 Astra designed a hardware device from scratch in under a week appeared first on The Decoder.

💾

  •  

Misumi launches $50 million early-stage investment fund targeting robotics

Misumi Americas has launched Misumi Ventures, a $50 million early-stage investment fund targeting robotics, automation, advanced hardware, physical AI and industrial software companies. The San Francisco Bay Area-based fund plans to build a portfolio of 20 to 30 seed and early-stage companies, typically investing between $500,000 and $1.5 million initially and reserving capital for follow-on […]
  •  

Google and AWS open new paths for PyTorch beyond Nvidia

At PyTorch Conference North America 2026 this week, Google open-sourced TorchTPU, a native PyTorch stack for Google TPUs, now available on GitHub. AWS also launched TorchNeuron in public beta, its open-source native PyTorch backend for AWS Trainium, also available on GitHub.

Both releases seem aimed at making it easier for developers to move beyond Nvidia GPUs, whose CUDA software ecosystem has long dominated AI development. But while Google and AWS’s new offerings do ease some of the challenges of switching hardware, getting top-tier performance still isn’t straightforward. 

For developers eager for more computing options, it’s promising but somewhat disappointing.

Google wants to make TPUs more PyTorch-friendly

Google put TorchTPU in preview with select customers in April (PyTorch says Google’s been working on the project since mid-2025), teasing a simpler way to run PyTorch workloads on Google’s AI chips. PyTorch support has long been an awkward dance, so the new backend could make it easier to run existing models on TPUs.

Google says it wants to make its TPU software ecosystem more open and developer-friendly for the PyTorch community, as it works to make the chips a more practical option for developers already using PyTorch. 

Developed in collaboration with the PyTorch team at Meta, TorchTPU allows developers to run existing PyTorch workloads without JAX dependencies in the runtime. This way, rather than requiring extensive code changes, developers can migrate existing workloads with only “trivial code changes,” according to PyTorch. 

Specifically, developers can use eager execution to debug models and torch.compile to optimize TPU utilization and reduce memory overhead. 

The new PyTorch stack from Google and Meta comes alongside a slew of other products and integrations announced at the conference, which Google says are “designed to help you run PyTorch effectively on Google Cloud or in your own infrastructure.” Google also announced integrations involving Hugging Face, vLLM, SGLang, llm-d, TorchTitan, Ray and TorchRec, while contributing LiteRT to the PyTorch ecosystem. 

AWS wants in, too

AWS’s release extends the same PyTorch-native approach to its own AI chips.

Like TorchTPU, TorchNeuron gives developers a way to natively experiment and build models with PyTorch — this time for Trainium, AWS’s portfolio of AI chips designed for deep learning training and inference workloads. The public beta supports Trainium2 and Trainium3 instances. 

AWS says moving a model to Trainium can be as simple as switching to .to(‘neuron’) for eager execution. Compiled models also need the Neuron backend selected, either as the default or in a torch.compile call.

TorchNeuron also integrates with Neuron Kernel Interface (NKI) — a bare-metal language and compiler for programming AWS Trainium and Inferentia NeuronDevices — so developers can create custom operators and optimize performance on Trainium. AWS and PyTorch have published more detailed guidance on the backend’s execution modes and performance tradeoffs.

Both are getting closer, but not there yet

While TorchTPU and TorchNeuron are notable steps forward in opening up more hardware choices for PyTorch developers, neither is a silver bullet.

PyTorch says a simple code change can get models running on TPUs, but peak performance can require TPU-specific optimization.

For distributed workloads, PyTorch says TorchTPU’s default Skinny Collectives approach can incur an end-to-end performance penalty of roughly 30–40% compared with an optimized approach because it prevents the XLA compiler from overlapping computation and communication.

For workloads seeking peak performance, PyTorch recommends additional tuning and production-grade stacks: 

“Of course, for production workloads seeking peak performance we would absolutely recommend pursuing an optimal approach based on Pallas kernels and production-grade stacks such as vLLM-TorchTPU, SGLang-TorchTPU, or TorchTitan-TorchTPU, as examples.”

TorchNeuron is in public beta. In a separate blog post, AWS says teams can try Trainium with a configuration change instead of a code or model rewrite “in most cases,” leaving open which workloads need more changes.

In a case study, AWS describes how its Generative AI Innovation Center worked with AI research and technology company i14 to adapt its Laplace 3B model for Trainium2. The implementation required three changes: 1) using native PyTorch for the GLA attention kernel; 2) redesigning how Muon communicates; 3) compiling the sequential backward pass.

The releases give PyTorch developers more options beyond Nvidia GPUs. For now, moving existing code is easier than getting peak performance on unfamiliar chips. 

The post Google and AWS open new paths for PyTorch beyond Nvidia appeared first on The New Stack.

  •  

DeepSeek open-sources the inference engine behind its API. And Huawei gets equal billing with Nvidia.

DeepSeek mobile app

DeepSeek has open-sourced the inference engine it uses to serve its models in production, a step it said last year wasn’t yet practical.

Dubbed DeepInfer, the framework is designed to let developers run DeepSeek’s V4.1 Flash model on their own infrastructure, with many of the same optimizations the company uses behind its API. Support for future DeepSeek models is also planned.

“Running a model efficiently is a different matter altogether.”

DeepSeek has, of course, long made its model weights available for developers to download and run using third-party software such as vLLM. But running a model efficiently is a different matter altogether. The software that manages requests, distributes work across chips, and generates responses can have a significant impact on speed, throughput and computing costs. And so DeepInfer essentially gives developers access to DeepSeek’s own approach to those challenges.

“We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.”

‘Making AI more affordable’

Taking to X on Tuesday, DeepSeek said it was making its internal inference technology available to outside developers in an effort to bring down the expense of running large AI models.

“We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable,” the company writes.

🚀 Introducing DeepInfer, our in-house inference framework, now fully open source!

Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more…

— DeepSeek (@deepseek_ai) October 20, 2026

Released under the Apache 2.0 license, DeepInfer includes optimized computing kernels and communication libraries, including DeepSeek’s previously open-sourced FlashMLA, DeepGEMM, and DeepEP, alongside mechanisms for distributing inference across multiple chips. It also supports speculative decoding, a technique that speeds up response generation by predicting and verifying multiple tokens at once, and asynchronous scheduling to reduce the time processors spend waiting for new work. The framework also supports structured outputs, allowing developers to specify how model responses should be formatted.

Alongside DeepInfer, the company is also open-sourcing deepseek-recipe, a companion library that handles how API requests are converted into model inputs, and how the resulting output is processed into responses. Available for Rust and Python, it’s designed to help developers reproduce the behavior of DeepSeek’s official API when running their own deployments. The library was quietly released in September, with an updated version published on Tuesday.

The announcement comes just days after DeepSeek released an updated version of V4.1 Flash, called DeepSeek-V4.1-Flash-1017, touting improvements in coding performance and efficiency. That release coincided with the company making its in-house web search technology available to developers through new Search and Fetch APIs, as per The New Stack‘s report on Monday.

Why DeepSeek held back in 2025

DeepInfer goes further than DeepSeek’s previous plans to share its inference technology. Back in April 2025, the company published a document outlining its plans for its inference engine, explaining that it had considered releasing the entire system before deciding to contribute individual components and improvements to existing projects instead.

At the time, DeepSeek identified several obstacles. Its inference software had evolved from an early version of vLLM into a highly customized system, and the company questioned whether its relatively small research team could maintain another major open-source project. There were also practical difficulties in making the software usable outside DeepSeek’s own infrastructure.

“The engine is tightly coupled with our internal infrastructure, including cluster management tools, making it impractical for public deployment without significant modifications,” the company wrote at the time.

So what’s changed now, exactly? Well, DeepSeek hasn’t explained how it overcame those earlier difficulties, although the scope of the new release offers some clues. The DeepInfer documentation makes clear that the publicly available software currently supports offline inference, with “the core features needed for serving at scale,” as the company puts it. However, several components needed to operate a complete production service are still missing. These include a KV cache management system, infrastructure for separating different stages of inference across machines, and load balancing across replicas.

In other words, DeepSeek has opened up much of the engine responsible for running its models, while leaving developers to supply some of the surrounding infrastructure required to operate a full-fledged service. That narrower release may help explain how the company has made its technology available despite the challenges it identified last year.

Nvidia and Huawei, side by side

While the release of DeepSeek’s inference engine is significant enough in its own right, arguably the more interesting tidbit is the hardware supported from day one.

“Arguably the more interesting tidbit is the hardware supported from day one.”

Nvidia is included, as expected, though support is restricted to its SM100 architecture, used in Blackwell data center GPUs. Notably, the other supported platform is Huawei’s Ascend 950 NPU series, putting the Chinese chipmaker alongside Nvidia in DeepSeek’s list of compatible hardware. Older Nvidia architectures, including Hopper, are conspicuously absent.

This also ties back to DeepSeek’s published open-source plans from 2025. While the company was reluctant to release its entire inference engine at the time, it committed to working with hardware manufacturers ahead of future model launches, with the aim of making its models available across different chip architectures from day one.

US export restrictions have complicated Chinese companies’ access to Nvidia’s most advanced chips, adding another dimension to DeepSeek’s growing ties with Huawei. In September, Bloomberg reported that DeepSeek was planning to deploy at least 160,000 Huawei Ascend 950DT chips at a new data center in Inner Mongolia, specifically to run its models. The company has largely relied on Nvidia for training, although The Information reported in September that DeepSeek also plans to increase its use of Huawei chips for training, with deliveries expected as early as the fourth quarter.

Later that month, Reuters reported that DeepSeek and Huawei were collaborating on software tools intended to make Ascend processors easier to program, further underlining the company’s investment in Chinese AI hardware.

The Huawei inclusion has already attracted attention among the wider developer community. In a discussion on Reddit, one commenter singled out the two supported hardware platforms as a highlight of the news, while another pointed to DeepSeek’s origins in quantitative finance as a possible explanation for its technical expertise.

“Once again Deepseek shows how good high-frequency traders are at optimizing performance… On Ascend NPUs nonetheless!!,” the commenter wrote.

Another user seemingly poked fun at Anthropic’s Claude Opus 5.5, following reports that requests involving kernel programming for Huawei’s AI chips could trigger safeguards that redirect certain advanced AI development tasks to an older model.

“Don’t let Opus 5.5 see this! It will delete the Ascend support,” the user joked.

It is worth noting that feature support differs between the Nvidia and Huawei implementations. DeepInfer’s batch-invariant inference and deterministic sampling features, which help ensure reproducible outputs under specified conditions, are available only on Nvidia hardware. DeepSeek’s published performance measurements also use Nvidia GPUs, with four SM100 GPUs used for prefill and 64 for decoding, leaving open the question of how the framework performs on Huawei’s chips by comparison.

For developers without access to either chip family, the immediate value may lie in studying the newly available code and adapting its optimizations to other inference engines. For now, DeepSeek has given the wider open-source community considerably more of its inference technology to experiment with.

The post DeepSeek open-sources the inference engine behind its API. And Huawei gets equal billing with Nvidia. appeared first on The New Stack.

  •  
❌