❌

Normal view

Diagnose Distributed AI Workload Failures with NVIDIA Attribution

2 November 2026 at 23:39
In distributed AI workloads, a single error can produce a cascade of symptoms. A GPU or fabric event may surface as NCCL errors on other ranks, scheduler state...

In distributed AI workloads, a single error can produce a cascade of symptoms. A GPU or fabric event may surface as NCCL errors on other ranks, scheduler state changes, and thousands of downstream messages. The reported exception shows where the workload stopped, not where the failure began. In this post, you’ll learn how to run NVIDIA Attribution on sample workload evidence…

Source

Keep Distributed Training Running with Automated Workload Recovery

2 November 2026 at 22:43
As distributed AI training scales, failures across hardware and software dependencies increasingly disrupt training progress, making fast, reliable recovery...

As distributed AI training scales, failures across hardware and software dependencies increasingly disrupt training progress, making fast, reliable recovery essential. A stalled training rank can trigger an NVIDIA Collective Communications Library (NCCL) timeout, a GPU node can stop responding to the scheduler, or a straggler can slow the entire job. Each disruption is manageable on its own…

Source

πŸ’Ύ

Making Kubernetes AI-Ready for NVIDIA Vera Rubin on Day One

2 November 2026 at 18:00
Configuring and optimizing Kubernetes for new GPU hardware such as NVIDIA Vera Rubin is complex and error-prone. Drivers, kernels, runtimes, networking,...

Configuring and optimizing Kubernetes for new GPU hardware such as NVIDIA Vera Rubin is complex and error-prone. Drivers, kernels, runtimes, networking, operators, and schedulers must work together, with settings tuned to the hardware and workload. Small mismatches can prevent workloads from running or limit performance. The NVIDIA AI Cluster Runtime (AICR)β€”part of the NVIDIA DSX AI Factory…

Source

Support New AI Frameworks Faster Using NVIDIA Workload Map

2 November 2026 at 16:00
NVIDIA DSX AI Factory Platform brings together technologies to help the ecosystem design, build, and operate more efficient AI factories, including open source,...

NVIDIA DSX AI Factory Platform brings together technologies to help the ecosystem design, build, and operate more efficient AI factories, including open source, modular software components built to support multi-tenant operations and scaling. Operating an AI factory requires a shared understanding of distributed workloads: their components, pod templates, replica counts, and status.

Source

Build a Validated Kubernetes Runtime for NVIDIA Dynamo

30 October 2026 at 21:35
Glowing green AI network illustration in front of server racks.Teams building AI applications need a repeatable path from a model configuration to a working inference endpoint. NVIDIA Dynamo is a distributed inference...Glowing green AI network illustration in front of server racks.

Teams building AI applications need a repeatable path from a model configuration to a working inference endpoint. NVIDIA Dynamo is a distributed inference framework that coordinates model serving across GPU workers. Deploying it on Kubernetes is only a first step. A successful model deployment requires more: GPU drivers, resource allocation, scheduling, networking, and monitoring must all work…

Source

How to Check and Configure NVIDIA GPUDirect Storage for Your Platform

30 October 2026 at 15:01
Data center server racks with network cables connecting rack-mounted equipment.NVIDIA GPUDirect Storage (GDS) accelerates GPU storage access in on-premises and cloud environments. It provides a robust, portable, and efficient data path...Data center server racks with network cables connecting rack-mounted equipment.

NVIDIA GPUDirect Storage (GDS) accelerates GPU storage access in on-premises and cloud environments. It provides a robust, portable, and efficient data path between NVIDIA GPUs and storage targets to increase bandwidth, minimize latency, and reduce CPU and memory utilization across a broad range of platforms, application frameworks, and workload patterns. Configuring NVIDIA GPUDirect Storage…

Source

Memory-Enforced GPU Sharing in Kubernetes with NVIDIA CUDA 13.4

27 October 2026 at 20:50
Many AI workloads need only a fraction of a GPU. Giving each one an entire device can leave capacity idle; sharing GPUs lets platform teams serve more workloads...

Many AI workloads need only a fraction of a GPU. Giving each one an entire device can leave capacity idle; sharing GPUs lets platform teams serve more workloads with the infrastructure they already have. NVIDIA GPUs already support several approaches to sharing. Multi-instance GPU (MIG) splits a GPU into hardware-isolated instances that are transparent to the application…

Source

How NVIDIA Minerva Turns Telemetry Chaos Into Peak GPU Yield

26 October 2026 at 16:00
AI Factories are some of the most technologically complex operational environments ever created. Hundreds of thousands of GPUs, NVIDIA Spectrum-X Ethernet...

AI Factories are some of the most technologically complex operational environments ever created. Hundreds of thousands of GPUs, NVIDIA Spectrum-X Ethernet scale-out, scale-across and scale-in fabrics, NVIDIA NVLink scale-up fabrics, distributed schedulers, and multi-week training runs operate in continuous lockstep.These layers function as one production system even when different teams own their…

Source

How NVIDIA Mission Control 2.5 Brings Autonomous Recovery to Kubernetes

26 October 2026 at 15:00
Training large-scale AI models on thousands of GPUs is a high-stakes operation where monitoring and automation must be built-in to optimize cluster resiliency...

Training large-scale AI models on thousands of GPUs is a high-stakes operation where monitoring and automation must be built-in to optimize cluster resiliency and uptime at scale. This concept isn’t new to data center operators who have measured mean-time-to-repair (MTTR) and mean-time-between-failure (MTBF) long before AI factories existed. Now the stakes are higher. Modern AI factories run…

Source

Scaling NVIDIA TensorRT Performance Optimization Beyond a Single Build Route with the Global Performance Tuner, Part 2

23 October 2026 at 17:27
The Global Performance Tuner is a feature for automatic performance tuning released in NVIDIA TensorRT 11.1. It provides automatic, accuracy-aware and...

The Global Performance Tuner is a feature for automatic performance tuning released in NVIDIA TensorRT 11.1. It provides automatic, accuracy-aware and end-to-end performance tuning within TensorRT by customizing different optimizations for different models. For more details, see Part 1 of this series, which introduces the challenges associated with scheduling (a critical stage in deep learning…

Source

Scaling NVIDIA TensorRT Performance Optimization Beyond a Single Build Route with the Global Performance Tuner, Part 1

23 October 2026 at 17:27
Deploying a trained neural network involves more than translating each operator into a GPU kernel. It is a compilation problem with many legitimate answers. For...

Deploying a trained neural network involves more than translating each operator into a GPU kernel. It is a compilation problem with many legitimate answers. For example, when NVIDIA TensorRT builds an engine, it makes interdependent decisions about graph-IR transformations, fusion, implementation selection, memory placement, and quantization. This process is called scheduling.

Source

How to Read Data into GPU Memory in Python with NVIDIA GPUDirect Storage

22 October 2026 at 16:39
Deocrative image.GPU applications are processing ever-growing datasets with increasingly powerful accelerators. According to the 2025 Stanford AI Index, compute for notable AI...Deocrative image.

GPU applications are processing ever-growing datasets with increasingly powerful accelerators. According to the 2025 Stanford AI Index, compute for notable AI models has been doubling approximately every five months, while the size of datasets used to train LLMs has been doubling approximately every eight months. As computation and data scale together, moving data efficiently from storage becomes…

Source

Design Proteins with RFD4-Proteina Using Reward-Guided Inference-Time Search

21 October 2026 at 12:30
Protein design is a search problem. Choose amino acids for an eight-residue protein sequence and more than 25 billion combinations are possible. Choose amino...

Protein design is a search problem. Choose amino acids for an eight-residue protein sequence and more than 25 billion combinations are possible. Choose amino acids for a 100-residue protein sequence and the number of possibilities exceeds the number of atoms in the observable universe. Somewhere in that space is a protein that opens a path to a new therapy. There are too many combinations to…

Source

Developing an AI Factory Digital Twin with an Agent-Assisted Workflow

21 October 2026 at 09:38
Building an AI factory digital twin requires developers to bring site plans, equipment assets, and engineering requirements into a consistent model that teams...

Building an AI factory digital twin requires developers to bring site plans, equipment assets, and engineering requirements into a consistent model that teams can inspect, revise, and prepare for simulation. Connecting these inputs into an editable workflow helps teams explore design alternatives and incorporate engineering feedback as the design evolves. The NVIDIA Omniverse DSX Blueprint…

Source

Achieve Productive AI Capacity Faster and More Predictably with NVIDIA DSX POD

21 October 2026 at 09:35
AI factories turn energy into intelligence. But before an AI workload can produce its first token, operators must bring an interconnected system of compute,...

AI factories turn energy into intelligence. But before an AI workload can produce its first token, operators must bring an interconnected system of compute, networking, storage, power, cooling, and software online as one productive unit. In a conventional data center build, teams must first select the site, design power and cooling for the deployment, lay out the data center…

Source

The neocloud shakeout is coming, and Nvidia has the most to lose

19 October 2026 at 11:33

After stock crashes and Firmus's scrapped IPO, industry executives expect only seven to ten of the hundreds of AI cloud providers to survive. CoreWeave lost a quarter of its market value, and Microsoft pulled out of a deal with Nscale for 23,000 Nvidia chips. Nvidia has a lot riding on the outcome, since customers outside the hyperscalers make up 45 percent of its data center revenue.

The article The neocloud shakeout is coming, and Nvidia has the most to lose appeared first on The Decoder.

NVIDIA Run:ai Is Now Open Source as Part of NVIDIA DSX AI Factory Platform

13 October 2026 at 14:19
NVIDIA is making the full NVIDIA Run:ai platform available as open source software, giving customers, developers, and ecosystem partners access to...

NVIDIA is making the full NVIDIA Run:ai platform available as open source software, giving customers, developers, and ecosystem partners access to production-grade AI workload and GPU orchestration capabilities. This release expands the NVIDIA DSX AI Factory Platform by establishing NVIDIA Run:ai as an open foundation that enables the broader ecosystem to bring differentiated solutions to market…

Source

Scale Bitwise-Deterministic Pretraining with NVIDIA Megatron Core

6 October 2026 at 19:58
Bitwise determinism makes large-scale pretraining easier to debug, validate, and resume reproducibly. These benefits become especially valuable when training...

Bitwise determinism makes large-scale pretraining easier to debug, validate, and resume reproducibly. These benefits become especially valuable when training models with trillions of parameters across thousands of GPUs, where multiple parallelism dimensions, low-precision computation, and distributed checkpointing complicate failure reproduction and fix validation.

Source

Validate AI Factory Changes with Digital Twins and AI Agents

7 October 2026 at 16:00
Three monitors display a network simulation and infrastructure monitoring dashboards in front of data center server racks.AI factories are some of the most complex operations in the world, combining GPUs, CPUs, switches, DPUs, and SuperNICs alongside schedulers, orchestration...Three monitors display a network simulation and infrastructure monitoring dashboards in front of data center server racks.

AI factories are some of the most complex operations in the world, combining GPUs, CPUs, switches, DPUs, and SuperNICs alongside schedulers, orchestration services, security controls, and a rapidly changing software stack. Deploying this infrastructure effectively is a challenge. But so is being able to validate that the infrastructure, software, and policies work together for the workloads the…

Source

❌