When the image is the bottleneck: Machine learning changed what a container image looks like. A typical application ships in a few hundred MB and starts in seconds. A modern ML inference image carries a deep-learning framework, the CUDA stack, and sometimes the model weights. These images roughly reach 20 to 30 GB, with some even higher. On the GPU and accelerated instances these run on, pulling one of those images takes several minutes before the application can serve its first request: minutes during which provisioned accelerators are ready to process real work but waiting for images to be pulled.

We hit this bottleneck on a production ML platform running on Amazon EKS. The team needed pods ready within two minutes, but image pull alone consumed several minutes. Each pod pulled a roughly 30 GB container image on top of loading model data from a shared filesystem. The images were rebuilt on a regular cadence, so worker nodes faced cold pulls with no usable local cache. While the image was pulled, accelerators sat idle, autoscaling lagged demand, and request queues built up.

The natural first suspect for these large image pull times was the network or the registry. After all, 30 GB is a lot of data. However, profiling the image pull path showed neither was the bottleneck on accelerated instances with 100 to 400 Gbps of network bandwidth available. The real constraint was how the software used the hardware already available.

“The real constraint was how the software used the hardware already available.”

By rethinking the pull pipeline to leverage the network bandwidth, storage throughput, and compute these instances already had, we got those multi-minute pulls down to seconds. The improvements are available by default on EKS Auto Mode today, and we contributed the core changes upstream to containerd and the SOCI snapshotter. This is the story of how we dove into the internals of the image pull path, identified where time was being lost, and rebuilt those stages. It starts where we started: understanding what a large container image looks like and how it gets onto a node.

What a container image looks like at scale

A container image is not one file. It is a stack of layers plus a small JSON manifest listing them. Each layer is a tar archive of part of the filesystem (the base OS in one, CUDA libraries in another, your code in a third), gzip-compressed. For each layer, the manifest records a digest, a SHA-256 hash of the compressed bytes, so the node can prove it received exactly what was published. When the container runs, these layers are stacked and mounted together into a single unified filesystem.

Here is what real ML images look like when measured directly from their registry manifests:

Notice that layers within an image are not roughly equal in size. A single layer can account for more than half the total image, with individual layers often reaching 9 GB or larger, while the remaining layers are comparatively small. This size disparity has direct consequences for how long a pull takes, as the next sections explain.

Getting each of these layers onto a node involves six stages, and traditionally containerd, the industry-standard container runtime, performed most of these operations sequentially.

The stages of a pull

For each layer, containerd performs six operations in sequence. The first three are the download phase: fetch the compressed bytes from the registry over a single HTTP connection, verify the bytes by computing their SHA-256 and comparing it against the manifest digest, and write the compressed blob to local disk. 

The next three are the unpack phase: decompress the gzip archive, verify the decompressed content by computing a second SHA-256, and extract the files into the local snapshot directory managed by a containerd snapshotter, the component that stores and serves the on-disk representation of each layer. By default, containerd downloads up to three layers in parallel, but each layer uses a single connection, and unpacking remains strictly sequential across layers.

Two details are worth noting. Both SHA-256 checks mentioned above are mandated by the Open Container Initiative (OCI) specification, and on a multi-gigabyte layer, each of these hashes requires real work. The first is computed over the compressed bytes and proves the download was not corrupted or tampered with. 

The second is computed over the decompressed content. It gives the runtime a stable fingerprint of the layer’s actual filesystem data, which is what allows it to recognize shared layers across images and avoid redundant unpacking. Decompression is another deceptively expensive operation on large layers: gzip often triples a layer’s size on expansion, and because each block depends on the previous one, it runs on a single core. At the same time, the rest of the instance sits idle. 

Downloading layers can be parallelized, but the unpack sequence runs layer by layer. Layer two cannot begin until layer one finishes all six stages. The result is that at any given moment during a pull, the node is bottlenecked on only one resource: network bandwidth during download, CPU during decompression and hash verification, or disk throughput during extraction. The other resources sit idle, waiting their turn in the pipeline.

“Because layers are not equally sized, the single largest layer becomes the long pole in the pipeline.”

Because layers are not equally sized, the single largest layer becomes the long pole in the pipeline. A 10 GB layer that takes a minute to decompress on one core holds up the entire image, even if the other layers finish in seconds. Total pull time is effectively bounded by that one dominant layer moving through all six stages.

Existing approaches: working around the pull

There are well-known approaches to reducing image pull times. Each works around the bottleneck differently, whether by altering images, caching them, or relying on assistance from other parts of the stack.

Image size reduction: The most direct approach is to make images smaller through multi-stage builds, distroless base images, and stripping unused packages. For ML workloads, this hits hard limits. The GPU software stack alone (PyTorch, cuDNN, CUDA) imposes a compressed floor of roughly 3 to 4 GB that no build optimization can remove. Beyond that, ML images are assembled across organizational boundaries: a platform team provides the OS and drivers, a frameworks team adds deep-learning libraries, and researchers contribute application code and model weights. No single team controls the final artifact; model weights frequently end up baked in because serving frameworks expect local paths, and multi-stage builds yield only single-digit percentage savings on images whose bulk is irreducible.

Image caching (pre-pulling): Cache images on nodes by snapshotting image content into a volume and mounting it at provisioning time, so subsequent launches skip the pull entirely. This works for stable images but adds a dedicated pipeline stage for each image version. For ML workloads, the challenge goes beyond frequent rebuilds: a common pattern is a single pod consuming an entire node, so nodes scale in and out with each scheduling decision and every new node faces the full cold pull. This limits pre-caching to workloads with long-lived, static node pools.

“The GPU software stack alone (PyTorch, cuDNN, CUDA) imposes a compressed floor of roughly 3 to 4 GB that no build optimization can remove.”

Registry-side optimization: Some approaches serve image content remotely rather than pulling it to the node. Alibaba’s DADI, for example, presents container images as remote block devices that the node mounts on demand without a discrete pull step. This eliminates startup latency for workloads that access only a fraction of their image. Still, the container depends on the network throughout its lifetime and requires purpose-built serving infrastructure that may not be portable across providers.

Lazy loading: Instead of pulling the whole image up front, start the container immediately and fetch file content on demand. Projects in this space include eStargz and Nydus (which require converting the image to a new format) and AWS’s SOCI (Seekable OCI), which adds a seekable index alongside the unmodified image. These techniques work well when containers touch only a fraction of their data at startup. Still, ML images densely access the framework, CUDA libraries, and model weights before serving the first request, so nearly all the data ends up being fetched anyway. For these workloads, the pull pipeline itself needs to be faster.

Peer-to-peer distribution within the cluster: Tools like Dragonfly (a CNCF graduated project) and Spegel turn nodes that already have an image into seeders for nodes that need it, reducing registry egress and accelerating rolling deployments. For ML workloads, though, the first node still faces the full cold pull, and when images are rebuilt frequently, no node has the new version cached yet. P2P distribution complements rather than replaces improvements to the pull pipeline itself.

Fixing the image pull pipeline

Out of the six stages in the pull pipeline, we focused on two: downloading the layer blob from the registry, and unpacking the layers on the node. These are where the most time is spent and where the serialization cost is highest.

Since SOCI was already an open-source containerd snapshotter plugin with the plumbing to intercept and customize the pull path, it served as a staging ground where we could develop and validate these changes before contributing them upstream to containerd. 

Download: sharding a single layer into multiple requests

containerd traditionally uses a single HTTP connection to download each layer. We identified that splitting a layer into fixed-size chunks and fetching them concurrently over separate connections using HTTP range requests was significantly faster. At the same time, the containerd community also independently added parallel chunked download support in containerd 2.1, and we built on the same principle in the SOCI snapshotter.

However, there is one significant difference that allows us to keep the runtime’s memory footprint constant regardless of image size. The difference is where chunks live between arrival and final assembly. Our implementation writes each chunk directly to the local disk the moment it arrives, while containerd holds it in memory as the layer is assembled. This means the runtime’s memory stays flat whether you are pulling a 1 GB layer or a 15 GB layer, which matters on GPU nodes where system memory is shared with model weights and CUDA contexts.

Once download completes in seconds rather than minutes, the compressed layer hash that previously hid behind it becomes visible. Because the layer is now a complete file on disk rather than a stream consumed once, integrity verification and unpacking can proceed at the same time.

Unpack: All layers concurrently

After downloading, containerd decompresses and extracts each layer one at a time. This sequencing exists because in some filesystem backends, a later layer can overwrite files from an earlier one, so the order matters. The overlay snapshotter, which is the default on EKS and most Kubernetes clusters, sidesteps this constraint by keeping each layer in its own separate directory. The kernel mounts them together into one unified view only when the container starts. Because each layer extracts to an independent directory, unpacking one layer does not need to depend on another to finish.

With that established, we built an unpack path that decompresses and extracts all layers concurrently. The snapshotter detects whether the backend actually requires ordering and falls back to sequential if it does. For images with multiple large layers, total unpack time goes from the sum of all layers to roughly the time of the single largest one. We contributed this parallel unpack capability upstream to containerd v2.2, so the improvement is available to the broader community.

What this looks like in practice

With chunked parallel download, a large layer that previously took over a minute on a single connection finishes in single-digit seconds. When you then unpack all layers concurrently rather than sequentially, the full pipeline for a large ML image compresses from several minutes to well under a minute on instances with fast local NVMe storage. The machine spends its available compute and bandwidth actively pulling rather than waiting on serial stages. On larger images and instances with faster storage, the gains are more pronounced because the gap between available hardware capacity and what a single connection can use is wider.

What’s next

The two changes above address download and layer sequencing, but decompression of a single large layer remains serial. On a 64 vCPU machine, decompressing a layer that expands to 18 GB means a lot of compute sitting idle. Two opportunities stand out:

Parallel decompression within a single layer. Libraries like rapidgzip can locate block boundaries and inflate blocks across cores in parallel.

Parallelizable integrity verification. Today, the layer hash requires a single sequential read over the entire compressed blob after all bytes have landed. A tree-structured hash like BLAKE3 would allow computing the layer digest from independently hashed chunks so that verification could run in parallel with download rather than as a separate pass afterward.

Addressing these opportunities will squeeze out the last remaining serial stages in the pull path, bringing total pull time closer to what the raw hardware is capable of delivering.

Using parallel download and unpack with Amazon EKS

This image pull optimization is enabled by default on G/P/Trn instances with EKS Auto Mode. You can also benefit from this on other node types through one of two mechanisms:

• Native containerd 2.2: parallel download and unpack built into the runtime. Use when your node already runs containerd 2.2.
• SOCI snapshotter: parallel download and unpack with a memory-bounded download path. Use on older nodes without containerd 2.2 or memory-constrained instances.

EKS AL2023 and Bottlerocket AMIs that ship with containerd 2.2 do not enable this feature by default, but you can set the containerd config explicitly as shown below:

AL2023: add the containerd config through the nodeadm NodeConfig in user data:

apiVersion: node.eks.aws/v1alpha1
  kind: NodeConfig
  spec:
    containerd:
      config: |
        [plugins.'io.containerd.transfer.v1.local']
          max_concurrent_downloads = 20
          concurrent_layer_fetch_buffer = 16777216
          max_concurrent_unpacks = 5

Bottlerocket (K8s 1.36): Bottlerocket generates the same containerd config from its settings API, so set the equivalent keys in user data:

[settings.container-runtime]
  max-concurrent-downloads = 20
  concurrent-download-chunk-size = 16777216
  max-concurrent-unpacks = 5

SOCI snapshotter is bundled in the optimized EKS AMIs (AL2023 and Bottlerocket).

Bottlerocket: enable SOCI through EC2 user data:

[settings.container-runtime]
  snapshotter = "soci"
 [settings.container-runtime-plugins.soci-snapshotter.parallel-pull-unpack]
  max-concurrent-downloads = 20
 concurrent-download-chunk-size = "16mb"
 max-concurrent-unpacks-per-image = 5

Tune chunk size and concurrency under [settings.container-runtime-plugins.soci-snapshotter.parallel-pull-unpack]; the right values depend on your instance type and images.

AL2023: enable SOCI through the nodeadm FastImagePull feature gate, which switches image pulls to SOCI’s parallel-pull-unpack mode, and you can override the SOCI tuning parameters through user-data:

apiVersion: node.eks.aws/v1alpha1
  kind: NodeConfig
  spec:
    featureGates:
      FastImagePull: true

The image pull problem was never really about the registry, and it was never about needing faster hardware. The network, the storage, and the CPU were always there. When containerd’s pull pipeline took shape, images were measured in hundreds of megabytes, and the sequential approach served that world well. 

“The image pull problem was never really about the registry, and it was never about needing faster hardware.”

As AI and ML workloads pushed images past 20 GB, the gap between available hardware throughput and what the pull path was utilizing became impossible to ignore. For workloads running on the overlay snapshotter with high-bandwidth instances and fast local storage, parallelizing download and unpack closes most of that gap today. Other snapshotters and storage backends may have different constraints, and opportunities like parallel decompression and parallelizable integrity verification remain open.

We have been contributing these changes upstream because this is where they belong: in the runtime itself, available to everyone by default rather than locked behind additional software. Some of that work has already landed in containerd 2.2, and more is in progress. 

If you are working on container runtimes, image formats, or compression tooling and any of the open problems described here interest you, we would welcome collaboration. The faster we can collectively close the remaining serial stages, the sooner multi-gigabyte images stop being a deployment bottleneck for the entire ecosystem.

The post Pulling multi-gigabyte container images in seconds on Amazon EKS appeared first on The New Stack.

When you run Kubernetes at the scale we do on Amazon EKS, nodes break constantly. GPUs fall off the PCIe bus. Container runtimes wedge. Network interfaces disappear. Across tens of thousands of clusters, “rare” hardware failures happen multiple times a day, somewhere in the fleet.

For years, everyone responded the same way: an operator wakes up, reads a dashboard, SSHes into the node, cordons it, drains it, terminates the instance, and waits for a replacement. Every step is human-paced. Every step is toil. And if the failure lands at 3 a.m. on a weekend, the workload sits degraded for hours before anyone looks.

We built the EKS Node Monitoring Agent to help close that gap, which we open-sourced in April earlier this year. It detects node failures and writes Kubernetes NodeConditions that signal the problem to Karpenter, which then automatically replaces the node if required. The agent is one piece of a larger system. To understand where it fits, you need to understand what manages the nodes it monitors.

“Across tens of thousands of clusters, ‘rare’ hardware failures happen multiple times a day, somewhere in the fleet.”

AWS launched Amazon EKS Auto Mode that fully automates Kubernetes cluster infrastructure: compute provisioning, scaling, networking, storage, OS patching, and security hardening, so teams focus on applications, not cluster operations. It dynamically selects optimal EC2 instances (including GPU instances like P5, P6, and G6 families), scales based on workload demand, consolidates underutilized nodes, and keeps the operating system patched. EKS Auto Mode ships with automatic node repair as the default behavior: detection, severity classification, and Karpenter-driven node replacement all run out of the box with no add-on to install, no controller to configure, and no repair policy to write.

This is the story of how we built automatic node repair, the design decisions that shaped the system, and the hard lessons that came from operating it at GPU scale.

Six lessons from building self-healing Kubernetes nodes at scale

After operating this across thousands of clusters, the lessons compress into a short list. These are not unique to our system. The same patterns show up in NPD, NVSentinel, AKS Periscope, GKE’s auto-repair, and anyone building a custom node controller. They are folklore that should be a checklist. The open-source repo reflects each of these lessons in code, from the reason-code stability guarantees in the API to the jitter implementation that solved GPU workload interference.

“Your reason codes are an API contract. Additions are features. Renames are breaking changes.”

1. Your reason codes are an API contract. Every downstream consumer (repair controllers, dashboards, customer automation) keys on them by literal string match. Additions are features. Renames are breaking changes. Severity changes are breaking changes. Plan for them the way you plan for API versioning.
2. Absent and Unknown are not the same thing. “We are not watching” and “we are watching but cannot tell” require different responses from downstream automation. If your disabled monitor writes Unknown, some controller somewhere will eventually act on it. Emit nothing when you are not watching.
3. Don’t cross ownership boundaries. The kubelet owns workload-driven conditions. Your node-health agent owns hardware and infrastructure failures. Crossing that boundary means your repair system is fighting the kubelet’s eviction system, and one of them will make the wrong call.
4. Measure latency from the source. The detection SLO includes every hop in the signal chain: hardware event to driver log, driver log to journald, journald to agent poll, agent poll to NodeCondition write. The longest hop dominates. For kernel-level signals, journald flush cadence is the bottleneck. For GPU telemetry through DCGM, push-based policy violations (DBE, XID, NVLink) are near-instant, but polled field watches (NVSwitch fabric health, clock throttle) have a 5-minute floor. Know which path each detection uses.
5. Detection and diagnosis are separate systems with separate consumers. Detection feeds automation (fast, continuous, minimal data). Diagnosis feeds humans (on-demand, detailed, heavyweight). Conflating them degrades both.
6. Test telemetry interpretation against the spec, not empirical values. Hardware telemetry interfaces are not boolean. We read a DCGM bitfield for GPU fabric health and treated non-zero as failure. When a driver update changed the healthy return value from zero to a spec-defined non-zero mask, every GPU node was flagged unhealthy at once. The safety breaker held (by design), giving us time to ship the fix. The lesson: if you’re parsing packed enums or bitfields from GPU firmware, your test fixtures must come from the vendor documentation, not from what the field happened to return on previous hardware.

Node health detection in Kubernetes: Traps no one warns you about

Every node health agent in the Kubernetes ecosystem performs the same translation. Node Problem Detector (NPD), NVSentinel, GKE’s auto-repair, AKS’s Linux Extension, and the EKS Node Monitoring Agent all take noisy, low-level signals from a machine and translate them into a set of Kubernetes primitives: NodeCondition, Event, sometimes a CRD. The translation looks simple. It isn’t.

The output is a NodeCondition, which is just a type, a status (True/False/Unknown), a reason code, and a message. Four fields. But that surface area hides decisions that determine whether a repair action helps or hurts.

Reason codes are a public API. We learned this the hard way. In version 1.6.2, we changed NvidiaDeviceCountMismatch from Warning severity to Fatal. The technical reasoning was sound: once a GPU drops off the PCIe bus, it doesn’t come back without a node reboot or replacement. Leaving it as Warning meant GPU workloads kept getting scheduled onto degraded nodes, wasting expensive accelerator capacity. So we shipped the fix. Downstream automation broke. Customers had repair configurations keyed on the old severity. Dashboards that filtered on Warning stopped showing the fault. Automation that only acted on Fatal suddenly started draining nodes it hadn’t touched before. Dashboards that filtered on Warning stopped showing the fault. Automation that only acted on Fatal suddenly started draining nodes it hadn’t touched before. From that point, we treat every reason code addition as feature work and every rename or severity change as a breaking change.

“Absent” must not equal “healthy.” When we shipped per-monitor configurability in v1.6.0, we had to make a choice. A disabled monitor needs to produce some output (or no output). The three options: write True (your auto-repair now thinks the node is healthy because you’re not watching), write Unknown (ambiguous, might trigger repair depending on downstream logic), or omit the condition entirely. Only the third is safe.

This seems obvious in retrospect, but consider that NPD achieves the same result through a completely different mechanism: compile-time disable via build tags. NVSentinel delegates it to operator-authored CEL rules. The upstream Kubernetes spec defines what Unknown means, but if your repair automation treats Unknown as actionable, you will lose nodes for no reason. We chose to emit nothing when a monitor is off, and documented it as a hard contract.

Detection latency is bounded by the source, not by the agent. We originally told customers, “We detect kernel panics within 30 seconds.” This was wrong. Our agent’s detection time was under 30 seconds. But the kernel panic shows up in journald, and journald’s flush cadence is the actual bottleneck. If journald takes 45 seconds to write the line, our 30-second claim was incomplete.

For GPU faults, the picture is more nuanced because we use two detection paths with very different latency characteristics. The critical faults (double-bit ECC errors, XID errors, NVLink failures, page retirements, thermal and power violations) go through DCGM’s push-based policy violation channel. DCGM notifies our agent the moment it detects the violation; there is no polling interval. Detection of these faults is near-instant (sub-second in practice). A separate path uses a 5-minute field-value window to monitor NVSwitch fabric health, Fabric Manager status, and clock-throttle reasons. That window is the floor for those specific detections, but it does not apply to the critical GPU faults that trigger automatic repair. The lesson: the customer-facing SLO must include source-of-truth latency, and different signal paths within the same subsystem can have radically different floors.

Two severities, one switch: How auto-repair decides which nodes to replace

The kubelet already reports DiskPressure, MemoryPressure, and PIDPressure. NMA complements those with five additional conditions covering domains the kubelet does not monitor: kernel health, container runtime, networking, storage, and accelerated hardware. Every detection carries one of two severities, and severity is the switch that decides whether the repair cycle fires.

Condition severity is a terminal fault. It flips the matching condition to False and makes the node eligible for automatic repair. GPU device-count mismatches, critical XID and double-bit ECC errors, NVLink and NVSwitch fabric failures, a missing Fabric Manager, and Neuron DMA and HBM uncorrectable errors. On the networking and runtime side: VPC CNI process down, IPAMD unable to reach the API server, fork failures due to PID exhaustion, and pods wedged, terminating behind a broken container runtime. These are faults that won’t recover on their own. On GPU nodes, a single degraded accelerator can corrupt training checkpoints or waste thousands of dollars in compute per hour.

Event severity is informational. It posts a Kubernetes event, the NodeCondition remains True, and operators get visibility without disruption. Bandwidth ceilings, connection-tracking limits, Amazon Elastic Block Store (Amazon EBS) IOPS throttling, I/O delays, filesystem fragmentation, clock drift, liveness and readiness probe failures, kube-proxy anomalies, GPU thermal and power warnings, PCIe link degradation, and page-retirement thresholds. These signal trouble building before it turns terminal.

Getting severity wrong in either direction is expensive. Too aggressive, and you terminate healthy nodes and needlessly displace workloads. Too conservative, and degraded nodes serve traffic for hours while a GPU with a failing memory bank corrupts training checkpoints. The classification principle: if the failure is deterministic and infrastructure-owned (hardware broke, firmware crashed, a physical link went down), it triggers replacement. If the signal could be application-induced or transient, it stays informational. You never want to terminate a healthy node because a misbehaving pod saturated a resource.

“Getting severity wrong in either direction is expensive. Too aggressive, and you terminate healthy nodes. Too conservative, and degraded nodes serve traffic for hours.”

DiskPressure, MemoryPressure, and PIDPressure are the canonical examples. Every major auto-repair system (GKE, AKS, NPD) has independently converged on the same answer: don’t touch them. These are workload-driven conditions, not node-level faults. Replacing the node just moves the misbehaving workload to a fresh machine, where it will eat memory again. The correct response is kubelet-level pod eviction, not node replacement. If you’re building a node-health system, draw this boundary early and document it publicly.

The agent that hurt what it was protecting: GPU workload interference from health monitoring

The hardest lesson came from a customer running large-scale distributed GPU training. Their workload used NCCL collectives across hundreds of GPU nodes, where every node in a communication group must complete its step before any can proceed. One slow node makes every node wait.

They found that NMA itself was causing periodic slowdowns. The agent’s monitors all ran on independent goroutines, and when their polling intervals aligned, dozens of goroutines would wake simultaneously and burst onto many CPU cores at once. On a general-purpose web service, this would be invisible. In a distributed training job, microseconds of jitter on one node can cascade across the entire GPU cluster, causing measurable throughput loss.

The customer disabled NMA entirely and saw an immediate improvement. That was the worst possible outcome for us: a health agent that interferes with the workload it exists to protect is worse than no agent at all.

The fix was straightforward once we understood the problem. We added a startup jitter to every monitor’s polling interval. Each goroutine delays its first tick by a random offset (up to 20% of its base interval), staggering the wake times so they don’t align on boot. We cached system calls that hit /proc on every poll. We consolidated handlers that shared an interval into a single sequential work queue, reducing the goroutine count for monitors that didn’t need their own thread. The result was an agent whose CPU profile is flat and predictable rather than bursty.

The lesson generalized: if your health agent runs on the same host as the workload, its resource consumption pattern matters as much as its resource consumption total. A process that uses 0.5% CPU spread evenly is invisible. A process that uses 0.5% CPU in concentrated bursts can disrupt latency-sensitive distributed GPU workloads in ways that show up as lost training time rather than a CPU alarm.

This is why per-monitor configurability matters. Not every monitor is relevant to every workload. A dedicated GPU training cluster with one pod per node and no pod churn doesn’t need IPAMD monitoring or environment scanning. We shipped the ability to disable individual monitors so customers can keep the health coverage they need without paying the overhead of coverage they don’t.

How the repair cycle works

Karpenter is the compute controller that provisions and scales EKS Auto Mode nodes. It already owns the lifecycle of every node it launched, and consuming our NodeConditions for repair is a natural extension of that ownership. There’s no separate repair backend, no sidecar controller, no webhook chain. The same system that created the node is the one that replaces it.

Karpenter’s AWS cloud provider declares repair policies: each one pairs a condition type with a status that means “replace this node.” The policies include toleration windows that prevent reacting to transient blips:

• Accelerated hardware faults: 10 minutes. These are unambiguous (a GPU is either present or absent) and expensive to leave running (a training job on a degraded node wastes GPU-hours).
• Everything else (kernel, runtime, networking, storage, kubelet NotReady): 30 minutes. Enough time for a transient network blip or a temporary runtime hiccup to resolve on its own.

The flow:

1. The agent detects a terminal fault and flips the matching condition to False with a reason code.
2. Karpenter’s health controller sees the transition and starts a timer.
3. If the condition clears before the window expires, the timer resets silently. The node was never touched.
4. Past the toleration window, a safety gate checks fleet health. Karpenter will not repair more than 20% of nodes in a NodePool simultaneously. If a correlated event (a bad AMI rollout, a control-plane hiccup, a zonal impairment) trips conditions across many nodes at once, the system holds. Auto-repair also stands down while an Amazon Application Recovery Controller zonal shift is active, so deliberate traffic movement away from an impaired Availability Zone is not mistaken for a fleet of broken nodes.
5. Inside the safety threshold, Karpenter taints the node to block new scheduling, gracefully drains running pods (respecting PodDisruptionBudgets), terminates the instance, and provisions a replacement sized for the displaced workload.

The replacement node comes up with a fresh agent monitoring it from boot. No operator in the path. In our testing, the full cycle from fault injection to replacement node running workloads took under 12 minutes. Detection landed in under a second (critical GPU faults use DCGM’s push-based policy channel, not polling). Then 10 minutes of toleration, and roughly 90 seconds for the replacement to launch and register.

The part that surprised us: detection and diagnosis are not the same problem

Auto-repair handles the common case: broken node gets replaced, workload keeps running. But “why did that node fail?” is a different question, and one we initially tried to answer inside the detection path. That was a mistake.

Detection answers “is this node healthy?” It runs continuously with minimal overhead, and it needs to be fast: a condition flip that takes 5 minutes to produce is 5 minutes of degraded workload. Diagnosis answers “what went wrong?” It needs to collect detailed artifacts: full journald output, containerd state, network configuration, dmesg, GPU driver logs. In our testing, that collection completes in about 7 seconds and produces a compressed log bundle. Baking it into the detection hot path would have slowed down the thing customers care most about: how fast the system reacts.

We built them as separate concerns sharing an agent binary. The NodeDiagnostic CRD lets you request a full log bundle from any node through kubectl, without SSH. On EKS Auto Mode, where nodes are Amazon Elastic Compute Cloud (Amazon EC2) managed instances with no shell access by design, this is the only way to investigate after a GPU failure or any other node-level fault.

The experience is one command:

kubectl ekslogs <node-name>

The plugin creates a NodeDiagnostic resource. The agent on the target node detects it via a watch, collects system state into a compressed tarball, and stores it temporarily (available for 10 minutes). The plugin then downloads it through the kubelet’s Node Log Query API (KEP-2258, GA in Kubernetes 1.36). No SSH, no security groups, no key pairs.

This separation means detection doesn’t slow down to collect evidence, diagnosis doesn’t need to be always-on (saving node resources), and you can diagnose a node that auto-repair has already flagged but hasn’t yet terminated. The 10-minute window for accelerated hardware faults gives you exactly enough time to grab the logs before the node is gone. If you’re interested in further improvements, engage with us on EKS public roadmap.

What this means if you’re running EKS

On EKS Auto Mode, all of this is on by default. Auto Mode fully manages your cluster infrastructure (compute, networking, storage, patching, and security hardening) so you focus on applications, not cluster operations. The agent runs as a systemd service in the node image (not a DaemonSet you manage), Karpenter consumes its conditions as part of the compute lifecycle it already owns, and kubectl ekslogs gives you diagnostic access without SSH. There is nothing to install, configure, or operate. For GPU workloads, this means your expensive accelerator nodes are automatically monitored, classified, and replaced without any operator intervention.

On managed node groups or self-managed Karpenter, you can assemble the same loop: install the Node Monitoring Agent as an EKS add-on and opt each node group into auto-repair. The architecture is the same, just not pre-assembled.

The EKS Node Monitoring Agent is Apache 2.0 open source at github.com/aws/eks-node-monitoring-agent. 

The failure modes we hit when running it at scale, and the fixes that come out of them, flow back to anyone using it. If you’re building a node-health system or running ours and hitting an edge case, come build with us!

The post Self-healing GPU nodes in Kubernetes: What we learned building the EKS node monitoring agent appeared first on The New Stack.

Amazon EKS runs hundreds of thousands of Kubernetes clusters across more than thirty AWS regions. Operating at that scale has taught us something that has shaped how we build the service and that we think is useful to anyone running Kubernetes at scale: most availability problems do not stem from a component failing. They come from a component reacting to a problem in a way that makes it worse. A cache that goes stale and serves wrong answers. A health check that restarts the very process keeping a cluster alive.

What separates a resilient control plane from a fragile one is not the number of faults. It is whether a fault stays a fault or becomes an outage. This post is the story of how we keep the EKS-managed Kubernetes control plane on the right side of that line at ever-growing scale: the foundational changes we made and why, and what operating at fleet scale taught us about building systems that tolerate faults rather than spreading them. These are the reasons our most demanding customers confidently run their mission-critical workloads on EKS.

How AI and analytics workloads reshaped what “scale” means

Kubernetes was built for a particular rhythm of work. Pods came and went at predictable rates, and controllers had seconds or minutes to reconcile. The system’s design reflected that pace: strong data consistency, ordered watches, and consensus-replicated storage that puts correctness first. It worked beautifully for what it was designed to do, and it still does.

“What separates a resilient control plane from a fragile one is not the number of faults. It is whether a fault stays a fault or becomes an outage.”

But the workloads evolved faster than anyone anticipated. Foundation model training runs scale-up training jobs on thousands of GPU nodes in minutes. Real-time inference services scale from a warm baseline to thousands of replicas, then drop back within the hour. Apache Spark analytics pipelines burst from zero to tens of thousands of executor pods, chew through a dataset, and vanish. 

Emerging agentic AI workloads add yet another dimension: autonomous agents that spin up, fan out, execute tasks, and tear down in seconds or less. These workloads share a trait that distinguishes them from traditional microservices: they generate enormous volumes of state transitions within compressed time windows and are deeply intolerant of delays. This velocity of state change pushed us to reinvent some of the mechanics to support a scale that was previously impossible, and to contribute what we could upstream.

How EKS reimagined Kubernetes storage foundation

Every Kubernetes cluster depends on etcd as its source of truth. Every application, every service endpoint, every scheduling decision is stored there. If etcd loses data, the cluster forgets everything it knows. Protecting that state is the most important job for a managed Kubernetes service.

Operating etcd for one cluster is well understood. Operating it for a fleet of millions is a different problem entirely. Hardware fails, networks blip, and disks degrade, so something has to handle those events without a human in the loop. And the operations etcd needs most, like replacing a failed member or recovering after a zonal event, are exactly the ones where a person acting under pressure can make a mistake that causes permanent data loss.

From the beginning, we built an operator agent that runs alongside every etcd instance and automates its entire lifecycle. The agent has two jobs. First, backup and recovery: it takes point-in-time snapshots and stores them durably outside the cluster. If too many instances are lost at once and the survivors cannot form a majority, the agent automatically detects the condition and rebuilds from the latest snapshot. Second, membership management: when an instance fails, the agent removes the terminated member and adds its replacement in an order that protects quorum and prevents split-brain.

A recent, more fundamental change was replacing etcd’s consensus mechanism, Raft, with a purpose-built journal that provides durable, ordered storage independently of etcd. In traditional etcd, a majority of members must agree on every write before it is committed. If two of three are unhealthy, the cluster becomes unavailable. 

By offloading durability to the journal, etcd peers no longer negotiate quorum among themselves. Writes commit as soon as the journal acknowledges persistence, and that entire class of etcd quorum-loss failures disappeared. Since the journal handles persistence, etcd no longer needs to fsync writes to local disk, so its data store has moved to an in-memory filesystem. What was a disk-bound system became a compute-bound one, and storage latency was removed entirely from the critical path. For a deeper look at this architecture, read “Under the hood: Amazon EKS ultra scale clusters.”

“What was a disk-bound system became a compute-bound one, and storage latency was removed entirely from the critical path.”

For ultra-scale clusters, we went further and partitioned etcd into resource-specific shards. Each partition operates independently with its own storage budget and throughput capacity. The primary value is failure isolation. In a monolithic deployment, if the events keyspace exceeds its quota because a misbehaving controller creates objects faster than garbage collection can remove them, it blocks writes to everything, including node leases. 

Suddenly, healthy nodes appear unhealthy because their lease renewals are being rejected. With partitioned etcd, the events partition hits its quota, but the leases partition continues operating normally. Nodes remain healthy. The scheduler keeps running.

What replacing etcd’s consensus mechanism unlocked

Removing the quorum requirement allowed us to make a change we had wanted for a long time: running etcd on the same host as the API server. In the traditional layout, every read and write crosses the network between separate machines. Each trip is fast on its own, but at thousands per second, the travel time adds up. With collocation, the API server talks to its local etcd over a loopback interface, and pod scheduling and controller reconciliation get measurably faster. For workloads where job controller queue depth is the binding constraint, shaving milliseconds off each API call means many more jobs are processed per second before the queue starts growing.

This is where the operator agent paid off. When etcd runs on the same host, an etcd member comes and goes whenever a control-plane host is replaced, which happens routinely. That only works if membership management is completely safe and automatic, which is exactly what the agent was already doing. We did not have to build a colocation from scratch; we built it on top of infrastructure that had been managing etcd membership safely since day one.

Collocation also taught us a lesson worth passing on: the convenient path needs a failover in case it breaks. The local etcd is the fast path, but if it becomes impaired, the API server fails over to another etcd member that is actively serving other API servers from the same journal. When you optimize for the common case, design just as deliberately for the moment that optimization is not available.

Fixing bottlenecks across the stack

At extreme scale, you have to address bottlenecks across the entire Kubernetes stack, and most of them are not bugs in the traditional sense. They are design choices that were correct at the scale Kubernetes originally targeted and break down only when the numbers get large. Rather than working around them internally, we fix them upstream so the entire community benefits.

One example involved the watch cache, the in-memory layer that distributes state changes from etcd to every controller watching for updates. When a controller starts, it requests a full snapshot of the current state via a mechanism called WatchList, and the existing implementation holds a shared read lock for the duration of the response build. 

At hundreds of thousands of objects, that work runs long enough to starve the writer that needs exclusive access, so the cache’s resource version cannot advance. Consistent reads see a stale cache and fail over to etcd, while the response building churns through hundreds of thousands of allocations under the lock. We identified this as a limitation in the watch-cache’s locking model and are working with the community to refactor the underlying data structures and interfaces to eliminate the contention.

The same shape appears elsewhere. In the Horizontal Pod Autoscaler, a single mutex protecting the scaling state becomes a serialization point at high HPA counts, where workers spend nearly all their time blocked rather than doing useful work. A redesigned data store (PR #139142) restores parallelism and raises reconciliation throughput by orders of magnitude. In the scheduler, we identified a bottleneck (issue #138426): every scheduling cycle rebuilds a set of in-use persistent volumes by scanning every node in the cluster, even for pods that do not use storage at all. The fix computes that information lazily, and only for pods that actually need it, restoring throughput at scale.

Each of these started from a real production workload hitting a cliff, and we are working on the fixes upstream so the improvements reach every Kubernetes user.

From engineering to guarantees: EKS Provisioned Control Plane

The engineering described above made the EKS control plane more resilient and performant. But customers had a different problem: they could observe that the control plane kept up today, but they could not reserve its capacity the way they reserve compute or GPU capacity. 

A team planning a thousand-node training run could secure the instances weeks in advance, yet had no equivalent mechanism for the orchestration layer that would coordinate them. EKS Provisioned Control Plane fills that gap. It exposes the control plane’s performance as dimensions you size explicitly, backed by the same kind of commitment you expect from the rest of your infrastructure.

You choose a scaling tier that maps to concrete, measurable capabilities: API request concurrency, pod scheduling rate, and cluster database size. The tiers range from XL through 8XL. At the top end, 8XL on Kubernetes 1.34 provides 16,000 concurrent API request seats, 400 pods-per-second scheduling rate, and 16 GB of cluster database storage, all backed by a 99.99% availability SLA measured in one-minute intervals.

Tiers are not static. You step up before a GPU training run or a large sales event, step back down during quiet periods, or grow permanently as your platform matures. Configuration happens through the console, CLI, eksctl, CloudFormation, or Terraform on any cluster, without recreation or downtime. 

For AI workloads, orchestration capacity is planned alongside GPU capacity, available when the compute comes online. For analytics platforms submitting hundreds of jobs per minute, the control plane is ready for the burst before it arrives. And for organizations that need environmental consistency across staging, production, and disaster recovery, the same tier guarantees consistent performance characteristics everywhere.


Taking the same foundation to the edge

Some workloads cannot move to the cloud, whether due to data sovereignty requirements, latency constraints, or unreliable connectivity to the Region. Running Kubernetes in these disconnected environments introduces unique challenges: etcd must remain durable on hardware with only a few machines, the cluster must self-heal without reaching the cloud, and observability must survive network partitions that last days. 

With the updated architecture for EKS local clusters on instance store Outposts, we brought edge clusters onto the same management plane and software stack as EKS clusters in the cloud.

The control plane lives in an EKS-managed account on the Outpost rather than in the customer’s account, so customers never manage control plane instances, etcd backups, or logging agents themselves, and they cannot accidentally break the thing keeping their cluster alive. The same machine images, container images, and operator agent run in both places, with edge-specific behaviors selected by configuration. 

Because it is the same stack, new Kubernetes and EKS platform versions arrive in lockstep with their cloud release, and features like EKS add-ons, Pod Identity, and access entries work the same way they do in a Region.

The hardest part was keeping etcd healthy on hardware with only a few machines that may be cut off from the cloud for days at a time. We solved it by extending the same agent. It keeps a spare copy of the data continuously up to date and promotes it the instant a machine fails, so the cluster heals itself with no human involvement and no connection to the cloud. 

Observability survives the disconnect, too: the metrics agent continues collecting and writing to local disk, shedding the least critical data first when space runs short, so the signals that matter most are the last to go. When the link returns, the buffered data is flushed back with its original timestamps.

All of this only works because the system was designed from the start to operate without anyone logged in. That same design is what makes it possible to deploy changes safely across the entire fleet.

Operating safely at fleet scale

Every one of these changes was deployed to a running fleet of hundreds of thousands of clusters. The journal migration and collocation required transitioning each cluster individually. Every migration follows a strict sequence: validate pre-conditions, create a point-in-time snapshot, perform the switchover, validate post-conditions. If any step fails, the system rolls back automatically. 

Rollouts proceed cell by cell, zone by zone, region by region, with automated monitoring comparing latency, error rates, and throughput between updated and non-updated clusters. Any statistically significant deviation triggers an automatic halt.

What made all of this possible is that EKS is built to operate without human intervention at the individual cluster level. Through Zero Operator Access, the architecture prevents AWS personnel from having technical pathways to access customer content in the managed control plane. A system designed to work without human access must be observable, recoverable, and automatable from the start, and that same discipline is what enables operating at extreme scale.

Three operational lessons shaped how we approach this work.

The first is that a healthy leader is not the same as a working one. The control plane’s controllers run in an active-passive configuration, and early on, we treated an unhealthy standby as if cluster operations had halted. They had not; what matters is whether a leader exists. But the harder lesson: a leader can quietly stop making progress while still renewing its lease and passing every health check. The signal that caught this was watching the controller’s work queue depth. If the queue fills while the leader looks healthy, the system is falling behind in ways no liveness probe will catch.

“A leader can quietly stop making progress while still renewing its lease and passing every health check. The signal that caught this was watching the controller’s work queue depth.”

The second is that maintenance ordering matters as much as the maintenance itself. etcd defragmentation is blocking, and the pause grows with database size. When it hit the leader, every write stalled. We taught the agent to move leadership to a healthy node before defragmenting, so the disruptive work always lands on a follower while writes keep flowing.

The third is that liveness is not readiness. A process can be alive but not ready while it warms caches, and routing based solely on liveness sends requests to an instance that cannot handle them. Equally, readiness flapping during graceful draining should never trigger a restart. We keep the two signals strictly separate: one decides recovery; the other decides routing.

None of this work is visible from the outside, and that is the point. The largest clusters taught us lessons that made every cluster faster. The riskiest migrations produced safety machinery that protects every upgrade. The upstream fixes we contributed for workloads at the edge of what Kubernetes can handle flow back to every user of the project.

“None of this work is visible from the outside, and that is the point.”

When you deploy on EKS and your pods come up in seconds, even during a burst, even when something behind the scenes goes wrong, that speed is not accidental. It is the accumulated result of years of operating at scales where small problems can become big ones fast, and engineering the system to contain them before they do.

To explore the architectures referenced in this post, see EKS Provisioned Control Plane and local Amazon EKS clusters on AWS Outposts.

The post Operating Kubernetes at scale: a few stories from running Amazon EKS appeared first on The New Stack.