We instrumented the full path from pod creation to first inference response on a GPU node running a 70B-class model. Eight minutes. Six sequential phases. We expected one bottleneck. We found six, and which one dominates depends on model size.

For a 64 GB model, 65% of the startup time is spent recompiling CUDA kernels that produce identical output every time. For a 203 GB model, 92% of the time is spent downloading weights from S3 through a calling pattern that leaves 98% of available bandwidth idle. Both are fixable with configuration changes. Neither is fixed by default.

“Eight minutes. Six sequential phases. We expected one bottleneck. We found six.”

We define time to first token served (TTFTS) as the wall-clock duration from pod creation to the first inference response leaving the GPU. Not time to first token (TTFT), which measures per-request latency once the model is warm. TTFTS is the one-time startup tax. TTFT begins where TTFTS ends.

Here’s what we achieved:

The warm-node row is what you pay on every pod restart: scale-up events, rolling updates, OOM recoveries. That’s the 80-93% win, and it requires only configuration changes. The cold-node row includes ~2 minutes of fixed infrastructure cost (node provisioning and framework initialization) that no application-layer optimization can remove. The rest is avoidable waste that we eliminated through platform and configuration fixes. The warm-node optimizations are environment variables and a volume mount that work on any Kubernetes cluster. The cold-node optimizations require EKS Auto Mode, which comes pre-configured with pre-compiled NVIDIA drivers, SOCI (Seekable OCI) parallel image pull, and NVMe instance store mounting.

All model startup measurements were taken on p5.48xlarge instances running Amazon EKS Auto Mode, with S3 traffic routed directly (bypassing the NAT Gateway) and container images in a private Amazon ECR repository (same region as compute). Model startup improvement ratios (80-93%) hold consistently across instance types (validated on P-family and G-family). Cold-node times vary with network bandwidth and CPU count. For the weights loading and compilation cache configuration, see Accelerate model loading on Amazon EKS.

The Kubernetes ecosystem has made real progress on the inference stack in 2026. OCI image volumes are now stable for model delivery. Dynamic Resource Allocation (DRA) gives GPUs structured attributes instead of opaque integer counts and provides flexibility in allocating GPUs to workloads. Gateway API has inference-aware routing extensions. But none of these primitives address the full cold-start stack: the six layers between “pod pending” and “first token served,” each with its own bottleneck and its own fix.

The six layers of cold start

When a new inference pod starts on a freshly provisioned GPU node, it passes through six distinct phases before serving its first request:

1. Node provisioning. Karpenter launches an EC2 instance, boots it, and registers it with the Kubernetes API server (~60-90s).
2. GPU driver initialization. The driver kernel module must load and expose accelerator devices.
3. Container image pull. The inference engine image (8-12 GB compressed) must be transferred to the node and extracted.
4. Model weights download. The model files must stream from object storage into GPU memory.
5. GPU kernel compilation. torch.compile traces the model graph and generates optimized CUDA kernels.
6. Engine initialization. CUDA graph capture, KV cache profiling, and HTTP server startup (30-120s depending on whether compilation is cached).

Each layer has a different bottleneck, a different fix, and a different owner.

Which layer dominates depends on model size

Before diving into each layer, one finding shaped every decision we made: the bottleneck is not fixed.

We instrumented the model startup path (layers 4 and 5) and measured each phase independently for two model sizes:

64 GB model (Qwen3.6-35B-A3B):

• Weights loading: ~29s (35% of model startup)
• torch.compile: ~53s (65% of model startup)

203 GB model (Llama-4-Scout, TP=4 where TP is tensor parallelism, splitting the model across GPUs):

• Weights loading: ~423s (92% of model startup)
• torch.compile: ~34s (8% of model startup)

For models under ~100 GB, compilation dominates. For larger models, network transfer dominates. torch.compile time stays roughly constant (it depends on graph complexity, not parameter count). Weights loading scales linearly with file size.

“For models under ~100 GB, compilation dominates. For larger models, network transfer dominates.”

This means any single-layer optimization has a ceiling.

Layer 1: Node provisioning

On EKS Auto Mode and Karpenter-managed clusters, node provisioning takes approximately 60-90 seconds for accelerated instances from pod pending to node Ready. Karpenter calls the EC2 Fleet API directly and reacts to pending pods within seconds, keeping provisioning at the EC2 launch floor.

Layer 2: GPU driver initialization

The NVIDIA GPU Operator in its default configuration adds 2-3 minutes to node boot while it compiles the driver kernel module from source. This cost repeats on every new node.

When the platform controls the full stack (OS image, kernel version, driver version, boot sequence) it can pre-compile driver kernel modules at image build time. The node boots, runs modprobe to load an already-compiled .ko file, and the GPU is ready in seconds.

This matters more now than it used to. Blackwell-architecture GPUs (G7, G7e instances) require NVIDIA’s open-source kernel modules exclusively. Older Maxwell/Pascal/Volta GPUs can only run proprietary modules. A cluster with both legacy and next-gen GPU nodes needs different drivers, different AMIs, different upgrade cycles. A managed platform that pre-compiles the correct module per instance family eliminates this complexity.

On EKS Auto Mode, the GPU driver loads in seconds (pre-compiled at image build time), compared to the 2-3 minutes a runtime-compilation approach requires.

Layer 3: Container image pull

A production vLLM or SGLang inference image is typically 8-12 GB compressed. Standard containerd pulls layers sequentially, decompresses them one by one in memory, and writes them to disk. At this size, sequential pull takes 2-4 minutes on a cold node depending on instance type and available CPU cores. For larger custom images (30-50 GB compressed), containerd can run out of memory entirely during decompression.

EKS Auto Mode uses SOCI’s parallel pull mode, which replaces containerd’s default snapshotter. The SOCI snapshotter downloads layer chunks concurrently via HTTP range requests and writes each chunk directly to its target byte position on disk (no in-memory ordering buffer). Decompression runs in parallel across all available CPU cores.

Pull time is bottlenecked by CPU-bound decompression, not network bandwidth. We confirmed this directly: a p4d.24xlarge with 400 Gbps networking achieved only ~1 Gbps effective pull throughput because CPU decompression was the constraint. On instances with capable, current-generation CPUs, SOCI parallel pull reduces image pull time from 2-4 minutes to 30-60 seconds. The dominant factor is per-core decompression throughput, which depends on CPU generation and instruction-set support, more than raw core count. A newer CPU with fewer cores can outperform an older one with more.

For a deeper look at how bounded-memory parallel pull handles images exceeding 30 GB without OOM, see Bounded-Memory Parallel Image Pulling for Large Container Images.

Layer 4: Model weights download

The obvious optimization for weights loading: more parallel connections. Split the model files into small chunks, download them concurrently, saturate the network pipe.

We tested it on p5.48xlarge with the 64 GB model streaming from same-region S3. The results were counterintuitive:

256 parallel connections provided no benefit over 17. The only failure mode was 8 GB chunks (exceeding shard file size), which caused a 56% regression.

Why? Because the open-source Run:ai Model Streamer (integrated into vLLM and SGLang) processes S3 range requests sequentially within each worker thread. A worker assigned to a 3.9 GB shard file downloads its byte-range requests one after another on a single connection. The parallelism comes from running multiple workers on different files, not from splitting one file into more pieces.

We settled on 4 GB chunks matching typical SafeTensors shard size (3-5 GB per file) with an aggressive timeout-and-retry for slow requests. S3 GET latency has a measurable long tail: in our testing, a meaningful fraction of requests took 2-3x longer than median, and a single stalled connection holds up the entire model load. Rather than wait, we kill stalled connections after a few seconds below a speed threshold and retry on a fresh connection. This follows S3’s own performance guidance.

For the 203 GB model, these config-only changes reduced weights loading from 423 seconds to 25 seconds (94% improvement). For the 64 GB model, from 29 seconds to 12 seconds. No code modifications, just environment variables. The tuning consists of three settings: chunk size aligned to shard file boundaries (eliminating the serial sub-request problem), a minimum-speed threshold that kills and retries stalled S3 connections, and explicit concurrency matching the number of shard files per tensor-parallel rank.

Layer 5: GPU kernel compilation

Every time a vLLM or SGLang pod starts, PyTorch traces the model’s computation graph and compiles it to optimized CUDA kernels. This takes 34-53 seconds depending on model architecture. The output is identical every time for the same model, GPU type, and tensor-parallel configuration.

And Kubernetes throws it away on every pod restart. Pods use ephemeral storage by default. When a pod terminates, its local filesystem is destroyed. The next pod recompiles from scratch.

“The output is identical every time for the same model, GPU type, and tensor-parallel configuration. And Kubernetes throws it away on every pod restart.”

Point the torch.compile cache directory at local NVMe instance store. GPU instances ship with NVMe that EKS Auto Mode mounts automatically. First pod compiles and writes ~15-30 MB of cached kernels. The second pod on the same node loads pre-compiled binaries in 4-6 seconds. One volume mount and environment variables.

The cache is safe because the compiled artifacts are deterministic: same model architecture + GPU architecture + tensor-parallel degree + PyTorch version equals valid cache. An image update or hardware change triggers exactly one recompilation.

torch.compile time is hardware independent. The same model compiles in ~52 seconds whether running on H100 or A100. The cache hit (4-6 seconds) is equally consistent across GPU types. This means the optimization works identically regardless of instance type.

Layer 6: Engine initialization

After weights are loaded and kernels compiled, the inference engine must capture CUDA execution graphs and profile KV cache memory. With compiled kernels cached, this completes in 30-45 seconds. Without cache, graph capture triggers additional JIT compilation and takes 60-120 seconds.

This is why the torch.compile cache has an outsized impact: it accelerates not just layer 5 but also layer 6. Cached compilation reduces a 2-3-minute combined phase to a 35-50-second combined phase.

Framework initialization (Python interpreter startup and PyTorch import) adds tens of seconds of fixed overhead that cannot be reduced through configuration.

The compounding effect

The six layers compound. Platform fixes (layers 1-3) eliminate 4-8 minutes of overhead: pre-compiled drivers replace 2-3 minutes of runtime compilation, parallel pull reduces image transfer time from 2-4 minutes to 30-60 seconds, and Karpenter keeps node provisioning to its hardware minimum. Configuration changes (layers 4-5) cut the remaining model startup by 80-93%. Engine initialization (layer 6) drops from 60-120 seconds to 30-45 seconds once the compile cache is warm. Together, cold-node TTFTS drops from 8-15 minutes to approximately 5 minutes.

64 GB model (Qwen3.6-35B-A3B), TP=2:

203 GB model (Llama-4-Scout), TP=4:

The warm-node subsequent pod number is what matters most for production. It’s what you pay on every pod restart. The 80-93% reduction is consistent across instance types because the optimizations target software bottlenecks (calling patterns, redundant compilation), not hardware limits.

The cost of cold starts at scale

Why does any of this matter? Because GPU nodes are expensive and inference traffic is bursty.

A single p5.48xlarge costs $55/hour on-demand. Even G-family instances commonly used for inference cost $10-20/hour. Every minute of cold start is GPU time you’re paying for but not using. If your autoscaler needs 8+ minutes to bring up new capacity, you must over-provision (burn money on idle GPUs) or accept latency spikes during traffic surges.

“Every minute of cold start is GPU time you’re paying for but not using.”

When model startup drops to 16-32 seconds on warm nodes, the calculus changes. You can scale more aggressively, keep fewer buffer nodes, and respond to traffic spikes without multi-minute startup delays.

What we learned

1. Decompose before optimizing. For 64 GB models, torch.compile dominates (65%). For 203 GB models, S3 loading dominates (92%). Without measuring each phase independently, we would have optimized the wrong layer.
2. The bottleneck flips with model size. torch.compile time is roughly constant across model sizes. Weights loading scales linearly. Every team running inference should know which regime they’re in.
3. “More parallelism” requires understanding the execution model. 256 connections performing sequential work inside each thread is no faster than 17. The bottleneck was the calling pattern, not the concurrency limit.
4. 15-30 MB can save 53 seconds. The most impactful optimization for smaller models was persisting a tiny cache file. Always check whether an expensive computation produces deterministic output before trying to make it faster.
5. Platform-level control enables optimizations that configuration alone cannot achieve. Pre-compiled drivers, default-on parallel image pull, and NVMe auto-mounting are infrastructure-layer decisions that compound upward. Together with the config-only changes at the application layer, these changes reduce cold start time from minutes to seconds.
6. The ecosystem is building the right primitives, but cold start lives between them. OCI image volumes, DRA, inference-aware routing, and local model caches are all real progress. But the compilation bottleneck and S3 tuning gaps sit in spaces that no upstream Kubernetes primitive addresses. Sometimes the highest-impact optimization is a volume mount and two environment variables, not a new API.

For the complete configuration guide, including environment variables, YAML manifests, and instance-specific recommendations, see “Accelerate model loading on Amazon EKS” in the Amazon EKS User Guide.

The post Cut GPU inference cold start from 8 minutes to less than a minute appeared first on The New Stack.

Most organizations that know they should sign their images still don’t. Not because they disagree, but because the path to doing it well has been too long. The result is a delivery pipeline built on trust that nobody can verify.

The problem space

Unsigned container images create an open door for attackers at every stage of the delivery pipeline. Malicious images masquerade as legitimate packages, waiting to be pulled by an unsuspecting team. Compromised CI/CD pipelines silently inject tampered artifacts into production builds with no cryptographic evidence of modification. Stolen credentials let a bad actor impersonate a trusted publisher. Even within a single organization, inconsistent practice means some teams sign while others skip the step entirely, leaving gaps in the chain of trust that nobody has mapped. Compounding all of it is base image inheritance. Every container image inherits the security posture of its parent, so one compromised base image can propagate across dozens of downstream services before anyone notices.

“Scanning is fundamentally reactive. One tells you what is inside. The other tells you whether you can trust it.”

Scanning is fundamentally reactive. It answers, “what vulnerabilities exist in this image?” It cannot answer the question that matters more as artifacts get harder to inspect: “who built this, and has it been modified since it left the build system?” That is the domain of cryptographic signing, which provides proactive provenance. The two are complementary, not interchangeable. One tells you what is inside. The other tells you whether you can trust it. 

Why the AI era makes this urgent

The workloads have changed faster than the tooling. Model weights, training datasets, inference runtimes, and agent tooling increasingly ship as OCI artifacts. A pickled PyTorch checkpoint itself has no CVE to match against. Safer serialization formats like .safetensors remove the code execution path, but they say nothing about who produced the weights or whether they’re the ones you meant to load. There is no vulnerability database for a set of trained weights, and the CVE and SCA based scanning that registries run has nothing to compare them to. 

This is not theoretical. In February 2024, JFrog researchers found a malicious PyTorch model on Hugging Face that opened a reverse shell the moment it loaded, abusing pickle’s __reduce__ hook to execute arbitrary code on torch.load(). Their analysis surfaced roughly 100 models on the hub carrying genuinely malicious payloads. No CVE fired, because there was nothing for a CVE to describe. The malice lived in the serialized weights. Model-specific scanning has since appeared to close that gap. Hugging Face runs ClamAV plus a pickle import scan on every file pushed to the Hub, statically disassembling the pickle’s opcode stream to flag dangerous imports. While they help, they are also already being evaded. In February 2025, ReversingLabs described nullifAI, two models that slipped past picklescan by compressing with 7z instead of ZIP and by corrupting the pickle stream immediately after the payload ran, so static analysis errored out on a file whose reverse shell would have already run. Hugging Face removed the models inside 24 hours and patched picklescan. That is the shape of the problem. Pattern matching scanners are a line that keeps moving, and each one answers whether a file resembles something known to be bad. None of them answers where the file came from.

“A tampered application image defaces a page. A tampered AI model artifact corrupts predictions at scale.”

AI is widening the attack surface in the same motion. Coding assistants suggest dependencies that never pass a human threat model, and that code gets containerized and shipped faster than review can keep up. The blast radius changed too. A tampered application image defaces a page. A tampered AI model artifact corrupts predictions at scale, poisons recommendations served to millions, or in the agentic case takes actions in production: API calls, tool invocations, spend. And when you consume a pre-trained model, you inherit every upstream decision about its training data and its security with zero visibility into any of them. Provenance stopped being a question about your application code. It became a question about the model, the agent, and the tooling that carries them.

But signing is not a checkbox. It is a chain. It only works if every link holds.

Why registry is the right layer

Operating the registry at the scale of Amazon ECR has taught us something that shaped how we think about supply chain security. Most teams don’t verify images. They verify addresses. An admission policy allows images from your registry account, push credentials belong to the pipeline rather than to people, and a scanner blocks critical CVEs. That stops a lot of attacks. What it can’t do is tell a good image from a bad one once it’s inside the boundary, because registry provenance is a claim about location, not origin. Anything that can write to the repository produces an image that looks legitimate: a leaked CI token, a misconfigured cross-account role, a compromised build step. Digest pinning tells you that you got the bytes you asked for, not that those were the right bytes to ask for. 

Every container image passes through a registry before it runs. It is the last system in the path that sees every artifact, knows who pushed it, and controls who can pull it. It already holds identity context, already enforces access policy, and already stores the metadata that describes what an image contains. The hard part of image signing is doing it consistently across every team and every pipeline without slowing anyone down. The registry is the only layer that can make it invisible.

“The hard part of image signing is doing it consistently across every team. The registry is the only layer that can make it invisible.”

Signing does not make forgery impossible. An attacker who fully compromises a trusted signing identity, stealing both the credential and the permission to sign, can produce a validly signed malicious image that passes verification. What signing does is shrink the attack surface. Without it, tampering anywhere in the path works, because nothing downstream checks. With signing and enforcement, none of it works unless the attacker compromises one narrowly scoped signer, and that rogue signature is an auditable event tied to an identity instead of an anonymous overwrite. Revoke the identity and the whole fleet stops trusting it in one change. Signing turns an invisible, unbounded problem into a scoped, attributable, revocable one.

The operational tax we set out to remove

Signing is a three-step process:

Sign: Generate a signature at build or push time, binding the image digest to a verifiable identity. The hard question is custody: who holds the private key, and how is it rotated and protected?

Verify: At pull time, and critically before the workload is admitted, check the signature against a trust policy which is a declared list of the identities you trust to have signed what you are about to run.

Enforce: A Kubernetes admission controller like Kyverno blocks any image not signed by a trusted identity from ever running. Signing without enforcement changes nothing.

Enabling signing comes with operational cost. Engineers had to install and configure client-side tooling like Notation CLI or Cosign, then own their signing keys, certificates, rotation schedules, and revocation lists, then build custom automation to wire signing into every pipeline. Across an enterprise with thousands of uniquely configured pipelines, that rollout took weeks to months. What we wanted to know was whether the registry itself could absorb the cost, so that signing could become a property of pushing an image rather than a project each team takes on. The answer to that question became Amazon ECR Managed Signing.

The mechanics are deliberately boring, which took some doing. You create a registry level signing configuration with up to ten rules, each pairing a signing profile with repository filters, and every matching push gets signed from then on.

Managed Signing answers the custody question by not giving you the keys. You configure a signing profile in AWS Signer, which pins the signing algorithm, a validity period, and the identity that appears in the signature. Signer keeps the certificate and the private key. This means no signing key ever sits in a repo, a runner, or a build log. Validity defaults to 135 months, so signatures won’t expire on you. Revocation is what you’ll actually use when you find out a build was compromised.

Then what gets signed, which is narrower than people assume. Signer signs a small Notary payload whose targetArtifact describes the image manifest: media type, digest, size. Not the image bytes directly. Because the signed material is content addressed, verification becomes a statement about exact bytes. The signature itself lands in the same repository as a detached OCI artifact, typed application/vnd.cncf.notary.signature, with a subject descriptor pointing at the image manifest digest. One image can carry signatures from several profiles as your trust requirements change. 

Signing happens asynchronously, which keeps Signer off the push path. A synchronous call would turn an availability dip or a throttle into a failed docker push for a developer, and it would put signing latency in front of every pipeline. The push commits first, and ECR calls SignPayload after. 

Verification and enforcement happen downstream, and the trust policy is where the whole design becomes legible. Your cluster operator writes it and imports it with notation policy import. It’s a short reviewable file:

{
"version": "1.0",
"trustPolicies": [
      {
        "name": "aws-signer-tp",
        "registryScopes": ["*"],
        "signatureVerification": { "level": "strict" },
        "trustStores": ["signingAuthority:aws-signer-ts"],
        "trustedIdentities": [
          "arn:aws:signer:us-east-1:111122223333:/signing-profiles/platform_images"
        ]
      }
  ]
}

That policy says a workload runs only if it carries a signature chaining to the AWS Signer root and produced by that specific profile. Admission does the work in order: resolve the reference to a digest, fetch the signature via OCI Referrers API, validate the envelope against its embedded certificate chain, walk that chain to the root in the trust store, check the signing identity against trustedIdentities, and check revocation. Revoking a profile makes verification fail wherever that profile is trusted. New admissions stop immediately and running pods pick it up when they’re next rescheduled. On EKS you get there with Gatekeeper and Ratify, or with Kyverno. Both paths use the AWS Signer plugin. Every link is checkable by the cluster itself, from the artifact plus a root certificate without asking the verifier to trust the registry it pulled from, or the pipeline that pushed.

Conclusion

Vulnerability scanning answers a question that mattered in the application era: what is broken inside this image? The AI era asks a harder one that scanning was never built to answer. Can you prove where this came from, and that no one touched it?

The cryptography was never the hard part. Making it the path of least resistance was. Sign, verify, and enforce, and let the registry carry the tax so your teams don’t have to.

To explore what’s referenced here, see Amazon ECR managed signing and signature verification on Amazon EKS.

The post Your container images are unsigned. In the AI era, that’s a ticking time bomb. appeared first on The New Stack.

When the image is the bottleneck: Machine learning changed what a container image looks like. A typical application ships in a few hundred MB and starts in seconds. A modern ML inference image carries a deep-learning framework, the CUDA stack, and sometimes the model weights. These images roughly reach 20 to 30 GB, with some even higher. On the GPU and accelerated instances these run on, pulling one of those images takes several minutes before the application can serve its first request: minutes during which provisioned accelerators are ready to process real work but waiting for images to be pulled.

We hit this bottleneck on a production ML platform running on Amazon EKS. The team needed pods ready within two minutes, but image pull alone consumed several minutes. Each pod pulled a roughly 30 GB container image on top of loading model data from a shared filesystem. The images were rebuilt on a regular cadence, so worker nodes faced cold pulls with no usable local cache. While the image was pulled, accelerators sat idle, autoscaling lagged demand, and request queues built up.

The natural first suspect for these large image pull times was the network or the registry. After all, 30 GB is a lot of data. However, profiling the image pull path showed neither was the bottleneck on accelerated instances with 100 to 400 Gbps of network bandwidth available. The real constraint was how the software used the hardware already available.

“The real constraint was how the software used the hardware already available.”

By rethinking the pull pipeline to leverage the network bandwidth, storage throughput, and compute these instances already had, we got those multi-minute pulls down to seconds. The improvements are available by default on EKS Auto Mode today, and we contributed the core changes upstream to containerd and the SOCI snapshotter. This is the story of how we dove into the internals of the image pull path, identified where time was being lost, and rebuilt those stages. It starts where we started: understanding what a large container image looks like and how it gets onto a node.

What a container image looks like at scale

A container image is not one file. It is a stack of layers plus a small JSON manifest listing them. Each layer is a tar archive of part of the filesystem (the base OS in one, CUDA libraries in another, your code in a third), gzip-compressed. For each layer, the manifest records a digest, a SHA-256 hash of the compressed bytes, so the node can prove it received exactly what was published. When the container runs, these layers are stacked and mounted together into a single unified filesystem.

Here is what real ML images look like when measured directly from their registry manifests:

Notice that layers within an image are not roughly equal in size. A single layer can account for more than half the total image, with individual layers often reaching 9 GB or larger, while the remaining layers are comparatively small. This size disparity has direct consequences for how long a pull takes, as the next sections explain.

Getting each of these layers onto a node involves six stages, and traditionally containerd, the industry-standard container runtime, performed most of these operations sequentially.

The stages of a pull

For each layer, containerd performs six operations in sequence. The first three are the download phase: fetch the compressed bytes from the registry over a single HTTP connection, verify the bytes by computing their SHA-256 and comparing it against the manifest digest, and write the compressed blob to local disk. 

The next three are the unpack phase: decompress the gzip archive, verify the decompressed content by computing a second SHA-256, and extract the files into the local snapshot directory managed by a containerd snapshotter, the component that stores and serves the on-disk representation of each layer. By default, containerd downloads up to three layers in parallel, but each layer uses a single connection, and unpacking remains strictly sequential across layers.

Two details are worth noting. Both SHA-256 checks mentioned above are mandated by the Open Container Initiative (OCI) specification, and on a multi-gigabyte layer, each of these hashes requires real work. The first is computed over the compressed bytes and proves the download was not corrupted or tampered with. 

The second is computed over the decompressed content. It gives the runtime a stable fingerprint of the layer’s actual filesystem data, which is what allows it to recognize shared layers across images and avoid redundant unpacking. Decompression is another deceptively expensive operation on large layers: gzip often triples a layer’s size on expansion, and because each block depends on the previous one, it runs on a single core. At the same time, the rest of the instance sits idle. 

Downloading layers can be parallelized, but the unpack sequence runs layer by layer. Layer two cannot begin until layer one finishes all six stages. The result is that at any given moment during a pull, the node is bottlenecked on only one resource: network bandwidth during download, CPU during decompression and hash verification, or disk throughput during extraction. The other resources sit idle, waiting their turn in the pipeline.

“Because layers are not equally sized, the single largest layer becomes the long pole in the pipeline.”

Because layers are not equally sized, the single largest layer becomes the long pole in the pipeline. A 10 GB layer that takes a minute to decompress on one core holds up the entire image, even if the other layers finish in seconds. Total pull time is effectively bounded by that one dominant layer moving through all six stages.

Existing approaches: working around the pull

There are well-known approaches to reducing image pull times. Each works around the bottleneck differently, whether by altering images, caching them, or relying on assistance from other parts of the stack.

Image size reduction: The most direct approach is to make images smaller through multi-stage builds, distroless base images, and stripping unused packages. For ML workloads, this hits hard limits. The GPU software stack alone (PyTorch, cuDNN, CUDA) imposes a compressed floor of roughly 3 to 4 GB that no build optimization can remove. Beyond that, ML images are assembled across organizational boundaries: a platform team provides the OS and drivers, a frameworks team adds deep-learning libraries, and researchers contribute application code and model weights. No single team controls the final artifact; model weights frequently end up baked in because serving frameworks expect local paths, and multi-stage builds yield only single-digit percentage savings on images whose bulk is irreducible.

Image caching (pre-pulling): Cache images on nodes by snapshotting image content into a volume and mounting it at provisioning time, so subsequent launches skip the pull entirely. This works for stable images but adds a dedicated pipeline stage for each image version. For ML workloads, the challenge goes beyond frequent rebuilds: a common pattern is a single pod consuming an entire node, so nodes scale in and out with each scheduling decision and every new node faces the full cold pull. This limits pre-caching to workloads with long-lived, static node pools.

“The GPU software stack alone (PyTorch, cuDNN, CUDA) imposes a compressed floor of roughly 3 to 4 GB that no build optimization can remove.”

Registry-side optimization: Some approaches serve image content remotely rather than pulling it to the node. Alibaba’s DADI, for example, presents container images as remote block devices that the node mounts on demand without a discrete pull step. This eliminates startup latency for workloads that access only a fraction of their image. Still, the container depends on the network throughout its lifetime and requires purpose-built serving infrastructure that may not be portable across providers.

Lazy loading: Instead of pulling the whole image up front, start the container immediately and fetch file content on demand. Projects in this space include eStargz and Nydus (which require converting the image to a new format) and AWS’s SOCI (Seekable OCI), which adds a seekable index alongside the unmodified image. These techniques work well when containers touch only a fraction of their data at startup. Still, ML images densely access the framework, CUDA libraries, and model weights before serving the first request, so nearly all the data ends up being fetched anyway. For these workloads, the pull pipeline itself needs to be faster.

Peer-to-peer distribution within the cluster: Tools like Dragonfly (a CNCF graduated project) and Spegel turn nodes that already have an image into seeders for nodes that need it, reducing registry egress and accelerating rolling deployments. For ML workloads, though, the first node still faces the full cold pull, and when images are rebuilt frequently, no node has the new version cached yet. P2P distribution complements rather than replaces improvements to the pull pipeline itself.

Fixing the image pull pipeline

Out of the six stages in the pull pipeline, we focused on two: downloading the layer blob from the registry, and unpacking the layers on the node. These are where the most time is spent and where the serialization cost is highest.

Since SOCI was already an open-source containerd snapshotter plugin with the plumbing to intercept and customize the pull path, it served as a staging ground where we could develop and validate these changes before contributing them upstream to containerd. 

Download: sharding a single layer into multiple requests

containerd traditionally uses a single HTTP connection to download each layer. We identified that splitting a layer into fixed-size chunks and fetching them concurrently over separate connections using HTTP range requests was significantly faster. At the same time, the containerd community also independently added parallel chunked download support in containerd 2.1, and we built on the same principle in the SOCI snapshotter.

However, there is one significant difference that allows us to keep the runtime’s memory footprint constant regardless of image size. The difference is where chunks live between arrival and final assembly. Our implementation writes each chunk directly to the local disk the moment it arrives, while containerd holds it in memory as the layer is assembled. This means the runtime’s memory stays flat whether you are pulling a 1 GB layer or a 15 GB layer, which matters on GPU nodes where system memory is shared with model weights and CUDA contexts.

Once download completes in seconds rather than minutes, the compressed layer hash that previously hid behind it becomes visible. Because the layer is now a complete file on disk rather than a stream consumed once, integrity verification and unpacking can proceed at the same time.

Unpack: All layers concurrently

After downloading, containerd decompresses and extracts each layer one at a time. This sequencing exists because in some filesystem backends, a later layer can overwrite files from an earlier one, so the order matters. The overlay snapshotter, which is the default on EKS and most Kubernetes clusters, sidesteps this constraint by keeping each layer in its own separate directory. The kernel mounts them together into one unified view only when the container starts. Because each layer extracts to an independent directory, unpacking one layer does not need to depend on another to finish.

With that established, we built an unpack path that decompresses and extracts all layers concurrently. The snapshotter detects whether the backend actually requires ordering and falls back to sequential if it does. For images with multiple large layers, total unpack time goes from the sum of all layers to roughly the time of the single largest one. We contributed this parallel unpack capability upstream to containerd v2.2, so the improvement is available to the broader community.

What this looks like in practice

With chunked parallel download, a large layer that previously took over a minute on a single connection finishes in single-digit seconds. When you then unpack all layers concurrently rather than sequentially, the full pipeline for a large ML image compresses from several minutes to well under a minute on instances with fast local NVMe storage. The machine spends its available compute and bandwidth actively pulling rather than waiting on serial stages. On larger images and instances with faster storage, the gains are more pronounced because the gap between available hardware capacity and what a single connection can use is wider.

What’s next

The two changes above address download and layer sequencing, but decompression of a single large layer remains serial. On a 64 vCPU machine, decompressing a layer that expands to 18 GB means a lot of compute sitting idle. Two opportunities stand out:

Parallel decompression within a single layer. Libraries like rapidgzip can locate block boundaries and inflate blocks across cores in parallel.

Parallelizable integrity verification. Today, the layer hash requires a single sequential read over the entire compressed blob after all bytes have landed. A tree-structured hash like BLAKE3 would allow computing the layer digest from independently hashed chunks so that verification could run in parallel with download rather than as a separate pass afterward.

Addressing these opportunities will squeeze out the last remaining serial stages in the pull path, bringing total pull time closer to what the raw hardware is capable of delivering.

Using parallel download and unpack with Amazon EKS

This image pull optimization is enabled by default on G/P/Trn instances with EKS Auto Mode. You can also benefit from this on other node types through one of two mechanisms:

• Native containerd 2.2: parallel download and unpack built into the runtime. Use when your node already runs containerd 2.2.
• SOCI snapshotter: parallel download and unpack with a memory-bounded download path. Use on older nodes without containerd 2.2 or memory-constrained instances.

EKS AL2023 and Bottlerocket AMIs that ship with containerd 2.2 do not enable this feature by default, but you can set the containerd config explicitly as shown below:

AL2023: add the containerd config through the nodeadm NodeConfig in user data:

apiVersion: node.eks.aws/v1alpha1
  kind: NodeConfig
  spec:
    containerd:
      config: |
        [plugins.'io.containerd.transfer.v1.local']
          max_concurrent_downloads = 20
          concurrent_layer_fetch_buffer = 16777216
          max_concurrent_unpacks = 5

Bottlerocket (K8s 1.36): Bottlerocket generates the same containerd config from its settings API, so set the equivalent keys in user data:

[settings.container-runtime]
  max-concurrent-downloads = 20
  concurrent-download-chunk-size = 16777216
  max-concurrent-unpacks = 5

SOCI snapshotter is bundled in the optimized EKS AMIs (AL2023 and Bottlerocket).

Bottlerocket: enable SOCI through EC2 user data:

[settings.container-runtime]
  snapshotter = "soci"
 [settings.container-runtime-plugins.soci-snapshotter.parallel-pull-unpack]
  max-concurrent-downloads = 20
 concurrent-download-chunk-size = "16mb"
 max-concurrent-unpacks-per-image = 5

Tune chunk size and concurrency under [settings.container-runtime-plugins.soci-snapshotter.parallel-pull-unpack]; the right values depend on your instance type and images.

AL2023: enable SOCI through the nodeadm FastImagePull feature gate, which switches image pulls to SOCI’s parallel-pull-unpack mode, and you can override the SOCI tuning parameters through user-data:

apiVersion: node.eks.aws/v1alpha1
  kind: NodeConfig
  spec:
    featureGates:
      FastImagePull: true

The image pull problem was never really about the registry, and it was never about needing faster hardware. The network, the storage, and the CPU were always there. When containerd’s pull pipeline took shape, images were measured in hundreds of megabytes, and the sequential approach served that world well. 

“The image pull problem was never really about the registry, and it was never about needing faster hardware.”

As AI and ML workloads pushed images past 20 GB, the gap between available hardware throughput and what the pull path was utilizing became impossible to ignore. For workloads running on the overlay snapshotter with high-bandwidth instances and fast local storage, parallelizing download and unpack closes most of that gap today. Other snapshotters and storage backends may have different constraints, and opportunities like parallel decompression and parallelizable integrity verification remain open.

We have been contributing these changes upstream because this is where they belong: in the runtime itself, available to everyone by default rather than locked behind additional software. Some of that work has already landed in containerd 2.2, and more is in progress. 

If you are working on container runtimes, image formats, or compression tooling and any of the open problems described here interest you, we would welcome collaboration. The faster we can collectively close the remaining serial stages, the sooner multi-gigabyte images stop being a deployment bottleneck for the entire ecosystem.

The post Pulling multi-gigabyte container images in seconds on Amazon EKS appeared first on The New Stack.

When you run Kubernetes at the scale we do on Amazon EKS, nodes break constantly. GPUs fall off the PCIe bus. Container runtimes wedge. Network interfaces disappear. Across tens of thousands of clusters, “rare” hardware failures happen multiple times a day, somewhere in the fleet.

For years, everyone responded the same way: an operator wakes up, reads a dashboard, SSHes into the node, cordons it, drains it, terminates the instance, and waits for a replacement. Every step is human-paced. Every step is toil. And if the failure lands at 3 a.m. on a weekend, the workload sits degraded for hours before anyone looks.

We built the EKS Node Monitoring Agent to help close that gap, which we open-sourced in April earlier this year. It detects node failures and writes Kubernetes NodeConditions that signal the problem to Karpenter, which then automatically replaces the node if required. The agent is one piece of a larger system. To understand where it fits, you need to understand what manages the nodes it monitors.

“Across tens of thousands of clusters, ‘rare’ hardware failures happen multiple times a day, somewhere in the fleet.”

AWS launched Amazon EKS Auto Mode that fully automates Kubernetes cluster infrastructure: compute provisioning, scaling, networking, storage, OS patching, and security hardening, so teams focus on applications, not cluster operations. It dynamically selects optimal EC2 instances (including GPU instances like P5, P6, and G6 families), scales based on workload demand, consolidates underutilized nodes, and keeps the operating system patched. EKS Auto Mode ships with automatic node repair as the default behavior: detection, severity classification, and Karpenter-driven node replacement all run out of the box with no add-on to install, no controller to configure, and no repair policy to write.

This is the story of how we built automatic node repair, the design decisions that shaped the system, and the hard lessons that came from operating it at GPU scale.

Six lessons from building self-healing Kubernetes nodes at scale

After operating this across thousands of clusters, the lessons compress into a short list. These are not unique to our system. The same patterns show up in NPD, NVSentinel, AKS Periscope, GKE’s auto-repair, and anyone building a custom node controller. They are folklore that should be a checklist. The open-source repo reflects each of these lessons in code, from the reason-code stability guarantees in the API to the jitter implementation that solved GPU workload interference.

“Your reason codes are an API contract. Additions are features. Renames are breaking changes.”

1. Your reason codes are an API contract. Every downstream consumer (repair controllers, dashboards, customer automation) keys on them by literal string match. Additions are features. Renames are breaking changes. Severity changes are breaking changes. Plan for them the way you plan for API versioning.
2. Absent and Unknown are not the same thing. “We are not watching” and “we are watching but cannot tell” require different responses from downstream automation. If your disabled monitor writes Unknown, some controller somewhere will eventually act on it. Emit nothing when you are not watching.
3. Don’t cross ownership boundaries. The kubelet owns workload-driven conditions. Your node-health agent owns hardware and infrastructure failures. Crossing that boundary means your repair system is fighting the kubelet’s eviction system, and one of them will make the wrong call.
4. Measure latency from the source. The detection SLO includes every hop in the signal chain: hardware event to driver log, driver log to journald, journald to agent poll, agent poll to NodeCondition write. The longest hop dominates. For kernel-level signals, journald flush cadence is the bottleneck. For GPU telemetry through DCGM, push-based policy violations (DBE, XID, NVLink) are near-instant, but polled field watches (NVSwitch fabric health, clock throttle) have a 5-minute floor. Know which path each detection uses.
5. Detection and diagnosis are separate systems with separate consumers. Detection feeds automation (fast, continuous, minimal data). Diagnosis feeds humans (on-demand, detailed, heavyweight). Conflating them degrades both.
6. Test telemetry interpretation against the spec, not empirical values. Hardware telemetry interfaces are not boolean. We read a DCGM bitfield for GPU fabric health and treated non-zero as failure. When a driver update changed the healthy return value from zero to a spec-defined non-zero mask, every GPU node was flagged unhealthy at once. The safety breaker held (by design), giving us time to ship the fix. The lesson: if you’re parsing packed enums or bitfields from GPU firmware, your test fixtures must come from the vendor documentation, not from what the field happened to return on previous hardware.

Node health detection in Kubernetes: Traps no one warns you about

Every node health agent in the Kubernetes ecosystem performs the same translation. Node Problem Detector (NPD), NVSentinel, GKE’s auto-repair, AKS’s Linux Extension, and the EKS Node Monitoring Agent all take noisy, low-level signals from a machine and translate them into a set of Kubernetes primitives: NodeCondition, Event, sometimes a CRD. The translation looks simple. It isn’t.

The output is a NodeCondition, which is just a type, a status (True/False/Unknown), a reason code, and a message. Four fields. But that surface area hides decisions that determine whether a repair action helps or hurts.

Reason codes are a public API. We learned this the hard way. In version 1.6.2, we changed NvidiaDeviceCountMismatch from Warning severity to Fatal. The technical reasoning was sound: once a GPU drops off the PCIe bus, it doesn’t come back without a node reboot or replacement. Leaving it as Warning meant GPU workloads kept getting scheduled onto degraded nodes, wasting expensive accelerator capacity. So we shipped the fix. Downstream automation broke. Customers had repair configurations keyed on the old severity. Dashboards that filtered on Warning stopped showing the fault. Automation that only acted on Fatal suddenly started draining nodes it hadn’t touched before. Dashboards that filtered on Warning stopped showing the fault. Automation that only acted on Fatal suddenly started draining nodes it hadn’t touched before. From that point, we treat every reason code addition as feature work and every rename or severity change as a breaking change.

“Absent” must not equal “healthy.” When we shipped per-monitor configurability in v1.6.0, we had to make a choice. A disabled monitor needs to produce some output (or no output). The three options: write True (your auto-repair now thinks the node is healthy because you’re not watching), write Unknown (ambiguous, might trigger repair depending on downstream logic), or omit the condition entirely. Only the third is safe.

This seems obvious in retrospect, but consider that NPD achieves the same result through a completely different mechanism: compile-time disable via build tags. NVSentinel delegates it to operator-authored CEL rules. The upstream Kubernetes spec defines what Unknown means, but if your repair automation treats Unknown as actionable, you will lose nodes for no reason. We chose to emit nothing when a monitor is off, and documented it as a hard contract.

Detection latency is bounded by the source, not by the agent. We originally told customers, “We detect kernel panics within 30 seconds.” This was wrong. Our agent’s detection time was under 30 seconds. But the kernel panic shows up in journald, and journald’s flush cadence is the actual bottleneck. If journald takes 45 seconds to write the line, our 30-second claim was incomplete.

For GPU faults, the picture is more nuanced because we use two detection paths with very different latency characteristics. The critical faults (double-bit ECC errors, XID errors, NVLink failures, page retirements, thermal and power violations) go through DCGM’s push-based policy violation channel. DCGM notifies our agent the moment it detects the violation; there is no polling interval. Detection of these faults is near-instant (sub-second in practice). A separate path uses a 5-minute field-value window to monitor NVSwitch fabric health, Fabric Manager status, and clock-throttle reasons. That window is the floor for those specific detections, but it does not apply to the critical GPU faults that trigger automatic repair. The lesson: the customer-facing SLO must include source-of-truth latency, and different signal paths within the same subsystem can have radically different floors.

Two severities, one switch: How auto-repair decides which nodes to replace

The kubelet already reports DiskPressure, MemoryPressure, and PIDPressure. NMA complements those with five additional conditions covering domains the kubelet does not monitor: kernel health, container runtime, networking, storage, and accelerated hardware. Every detection carries one of two severities, and severity is the switch that decides whether the repair cycle fires.

Condition severity is a terminal fault. It flips the matching condition to False and makes the node eligible for automatic repair. GPU device-count mismatches, critical XID and double-bit ECC errors, NVLink and NVSwitch fabric failures, a missing Fabric Manager, and Neuron DMA and HBM uncorrectable errors. On the networking and runtime side: VPC CNI process down, IPAMD unable to reach the API server, fork failures due to PID exhaustion, and pods wedged, terminating behind a broken container runtime. These are faults that won’t recover on their own. On GPU nodes, a single degraded accelerator can corrupt training checkpoints or waste thousands of dollars in compute per hour.

Event severity is informational. It posts a Kubernetes event, the NodeCondition remains True, and operators get visibility without disruption. Bandwidth ceilings, connection-tracking limits, Amazon Elastic Block Store (Amazon EBS) IOPS throttling, I/O delays, filesystem fragmentation, clock drift, liveness and readiness probe failures, kube-proxy anomalies, GPU thermal and power warnings, PCIe link degradation, and page-retirement thresholds. These signal trouble building before it turns terminal.

Getting severity wrong in either direction is expensive. Too aggressive, and you terminate healthy nodes and needlessly displace workloads. Too conservative, and degraded nodes serve traffic for hours while a GPU with a failing memory bank corrupts training checkpoints. The classification principle: if the failure is deterministic and infrastructure-owned (hardware broke, firmware crashed, a physical link went down), it triggers replacement. If the signal could be application-induced or transient, it stays informational. You never want to terminate a healthy node because a misbehaving pod saturated a resource.

“Getting severity wrong in either direction is expensive. Too aggressive, and you terminate healthy nodes. Too conservative, and degraded nodes serve traffic for hours.”

DiskPressure, MemoryPressure, and PIDPressure are the canonical examples. Every major auto-repair system (GKE, AKS, NPD) has independently converged on the same answer: don’t touch them. These are workload-driven conditions, not node-level faults. Replacing the node just moves the misbehaving workload to a fresh machine, where it will eat memory again. The correct response is kubelet-level pod eviction, not node replacement. If you’re building a node-health system, draw this boundary early and document it publicly.

The agent that hurt what it was protecting: GPU workload interference from health monitoring

The hardest lesson came from a customer running large-scale distributed GPU training. Their workload used NCCL collectives across hundreds of GPU nodes, where every node in a communication group must complete its step before any can proceed. One slow node makes every node wait.

They found that NMA itself was causing periodic slowdowns. The agent’s monitors all ran on independent goroutines, and when their polling intervals aligned, dozens of goroutines would wake simultaneously and burst onto many CPU cores at once. On a general-purpose web service, this would be invisible. In a distributed training job, microseconds of jitter on one node can cascade across the entire GPU cluster, causing measurable throughput loss.

The customer disabled NMA entirely and saw an immediate improvement. That was the worst possible outcome for us: a health agent that interferes with the workload it exists to protect is worse than no agent at all.

The fix was straightforward once we understood the problem. We added a startup jitter to every monitor’s polling interval. Each goroutine delays its first tick by a random offset (up to 20% of its base interval), staggering the wake times so they don’t align on boot. We cached system calls that hit /proc on every poll. We consolidated handlers that shared an interval into a single sequential work queue, reducing the goroutine count for monitors that didn’t need their own thread. The result was an agent whose CPU profile is flat and predictable rather than bursty.

The lesson generalized: if your health agent runs on the same host as the workload, its resource consumption pattern matters as much as its resource consumption total. A process that uses 0.5% CPU spread evenly is invisible. A process that uses 0.5% CPU in concentrated bursts can disrupt latency-sensitive distributed GPU workloads in ways that show up as lost training time rather than a CPU alarm.

This is why per-monitor configurability matters. Not every monitor is relevant to every workload. A dedicated GPU training cluster with one pod per node and no pod churn doesn’t need IPAMD monitoring or environment scanning. We shipped the ability to disable individual monitors so customers can keep the health coverage they need without paying the overhead of coverage they don’t.

How the repair cycle works

Karpenter is the compute controller that provisions and scales EKS Auto Mode nodes. It already owns the lifecycle of every node it launched, and consuming our NodeConditions for repair is a natural extension of that ownership. There’s no separate repair backend, no sidecar controller, no webhook chain. The same system that created the node is the one that replaces it.

Karpenter’s AWS cloud provider declares repair policies: each one pairs a condition type with a status that means “replace this node.” The policies include toleration windows that prevent reacting to transient blips:

• Accelerated hardware faults: 10 minutes. These are unambiguous (a GPU is either present or absent) and expensive to leave running (a training job on a degraded node wastes GPU-hours).
• Everything else (kernel, runtime, networking, storage, kubelet NotReady): 30 minutes. Enough time for a transient network blip or a temporary runtime hiccup to resolve on its own.

The flow:

1. The agent detects a terminal fault and flips the matching condition to False with a reason code.
2. Karpenter’s health controller sees the transition and starts a timer.
3. If the condition clears before the window expires, the timer resets silently. The node was never touched.
4. Past the toleration window, a safety gate checks fleet health. Karpenter will not repair more than 20% of nodes in a NodePool simultaneously. If a correlated event (a bad AMI rollout, a control-plane hiccup, a zonal impairment) trips conditions across many nodes at once, the system holds. Auto-repair also stands down while an Amazon Application Recovery Controller zonal shift is active, so deliberate traffic movement away from an impaired Availability Zone is not mistaken for a fleet of broken nodes.
5. Inside the safety threshold, Karpenter taints the node to block new scheduling, gracefully drains running pods (respecting PodDisruptionBudgets), terminates the instance, and provisions a replacement sized for the displaced workload.

The replacement node comes up with a fresh agent monitoring it from boot. No operator in the path. In our testing, the full cycle from fault injection to replacement node running workloads took under 12 minutes. Detection landed in under a second (critical GPU faults use DCGM’s push-based policy channel, not polling). Then 10 minutes of toleration, and roughly 90 seconds for the replacement to launch and register.

The part that surprised us: detection and diagnosis are not the same problem

Auto-repair handles the common case: broken node gets replaced, workload keeps running. But “why did that node fail?” is a different question, and one we initially tried to answer inside the detection path. That was a mistake.

Detection answers “is this node healthy?” It runs continuously with minimal overhead, and it needs to be fast: a condition flip that takes 5 minutes to produce is 5 minutes of degraded workload. Diagnosis answers “what went wrong?” It needs to collect detailed artifacts: full journald output, containerd state, network configuration, dmesg, GPU driver logs. In our testing, that collection completes in about 7 seconds and produces a compressed log bundle. Baking it into the detection hot path would have slowed down the thing customers care most about: how fast the system reacts.

We built them as separate concerns sharing an agent binary. The NodeDiagnostic CRD lets you request a full log bundle from any node through kubectl, without SSH. On EKS Auto Mode, where nodes are Amazon Elastic Compute Cloud (Amazon EC2) managed instances with no shell access by design, this is the only way to investigate after a GPU failure or any other node-level fault.

The experience is one command:

kubectl ekslogs <node-name>

The plugin creates a NodeDiagnostic resource. The agent on the target node detects it via a watch, collects system state into a compressed tarball, and stores it temporarily (available for 10 minutes). The plugin then downloads it through the kubelet’s Node Log Query API (KEP-2258, GA in Kubernetes 1.36). No SSH, no security groups, no key pairs.

This separation means detection doesn’t slow down to collect evidence, diagnosis doesn’t need to be always-on (saving node resources), and you can diagnose a node that auto-repair has already flagged but hasn’t yet terminated. The 10-minute window for accelerated hardware faults gives you exactly enough time to grab the logs before the node is gone. If you’re interested in further improvements, engage with us on EKS public roadmap.

What this means if you’re running EKS

On EKS Auto Mode, all of this is on by default. Auto Mode fully manages your cluster infrastructure (compute, networking, storage, patching, and security hardening) so you focus on applications, not cluster operations. The agent runs as a systemd service in the node image (not a DaemonSet you manage), Karpenter consumes its conditions as part of the compute lifecycle it already owns, and kubectl ekslogs gives you diagnostic access without SSH. There is nothing to install, configure, or operate. For GPU workloads, this means your expensive accelerator nodes are automatically monitored, classified, and replaced without any operator intervention.

On managed node groups or self-managed Karpenter, you can assemble the same loop: install the Node Monitoring Agent as an EKS add-on and opt each node group into auto-repair. The architecture is the same, just not pre-assembled.

The EKS Node Monitoring Agent is Apache 2.0 open source at github.com/aws/eks-node-monitoring-agent. 

The failure modes we hit when running it at scale, and the fixes that come out of them, flow back to anyone using it. If you’re building a node-health system or running ours and hitting an edge case, come build with us!

The post Self-healing GPU nodes in Kubernetes: What we learned building the EKS node monitoring agent appeared first on The New Stack.

Amazon EKS runs hundreds of thousands of Kubernetes clusters across more than thirty AWS regions. Operating at that scale has taught us something that has shaped how we build the service and that we think is useful to anyone running Kubernetes at scale: most availability problems do not stem from a component failing. They come from a component reacting to a problem in a way that makes it worse. A cache that goes stale and serves wrong answers. A health check that restarts the very process keeping a cluster alive.

What separates a resilient control plane from a fragile one is not the number of faults. It is whether a fault stays a fault or becomes an outage. This post is the story of how we keep the EKS-managed Kubernetes control plane on the right side of that line at ever-growing scale: the foundational changes we made and why, and what operating at fleet scale taught us about building systems that tolerate faults rather than spreading them. These are the reasons our most demanding customers confidently run their mission-critical workloads on EKS.

How AI and analytics workloads reshaped what “scale” means

Kubernetes was built for a particular rhythm of work. Pods came and went at predictable rates, and controllers had seconds or minutes to reconcile. The system’s design reflected that pace: strong data consistency, ordered watches, and consensus-replicated storage that puts correctness first. It worked beautifully for what it was designed to do, and it still does.

“What separates a resilient control plane from a fragile one is not the number of faults. It is whether a fault stays a fault or becomes an outage.”

But the workloads evolved faster than anyone anticipated. Foundation model training runs scale-up training jobs on thousands of GPU nodes in minutes. Real-time inference services scale from a warm baseline to thousands of replicas, then drop back within the hour. Apache Spark analytics pipelines burst from zero to tens of thousands of executor pods, chew through a dataset, and vanish. 

Emerging agentic AI workloads add yet another dimension: autonomous agents that spin up, fan out, execute tasks, and tear down in seconds or less. These workloads share a trait that distinguishes them from traditional microservices: they generate enormous volumes of state transitions within compressed time windows and are deeply intolerant of delays. This velocity of state change pushed us to reinvent some of the mechanics to support a scale that was previously impossible, and to contribute what we could upstream.

How EKS reimagined Kubernetes storage foundation

Every Kubernetes cluster depends on etcd as its source of truth. Every application, every service endpoint, every scheduling decision is stored there. If etcd loses data, the cluster forgets everything it knows. Protecting that state is the most important job for a managed Kubernetes service.

Operating etcd for one cluster is well understood. Operating it for a fleet of millions is a different problem entirely. Hardware fails, networks blip, and disks degrade, so something has to handle those events without a human in the loop. And the operations etcd needs most, like replacing a failed member or recovering after a zonal event, are exactly the ones where a person acting under pressure can make a mistake that causes permanent data loss.

From the beginning, we built an operator agent that runs alongside every etcd instance and automates its entire lifecycle. The agent has two jobs. First, backup and recovery: it takes point-in-time snapshots and stores them durably outside the cluster. If too many instances are lost at once and the survivors cannot form a majority, the agent automatically detects the condition and rebuilds from the latest snapshot. Second, membership management: when an instance fails, the agent removes the terminated member and adds its replacement in an order that protects quorum and prevents split-brain.

A recent, more fundamental change was replacing etcd’s consensus mechanism, Raft, with a purpose-built journal that provides durable, ordered storage independently of etcd. In traditional etcd, a majority of members must agree on every write before it is committed. If two of three are unhealthy, the cluster becomes unavailable. 

By offloading durability to the journal, etcd peers no longer negotiate quorum among themselves. Writes commit as soon as the journal acknowledges persistence, and that entire class of etcd quorum-loss failures disappeared. Since the journal handles persistence, etcd no longer needs to fsync writes to local disk, so its data store has moved to an in-memory filesystem. What was a disk-bound system became a compute-bound one, and storage latency was removed entirely from the critical path. For a deeper look at this architecture, read “Under the hood: Amazon EKS ultra scale clusters.”

“What was a disk-bound system became a compute-bound one, and storage latency was removed entirely from the critical path.”

For ultra-scale clusters, we went further and partitioned etcd into resource-specific shards. Each partition operates independently with its own storage budget and throughput capacity. The primary value is failure isolation. In a monolithic deployment, if the events keyspace exceeds its quota because a misbehaving controller creates objects faster than garbage collection can remove them, it blocks writes to everything, including node leases. 

Suddenly, healthy nodes appear unhealthy because their lease renewals are being rejected. With partitioned etcd, the events partition hits its quota, but the leases partition continues operating normally. Nodes remain healthy. The scheduler keeps running.

What replacing etcd’s consensus mechanism unlocked

Removing the quorum requirement allowed us to make a change we had wanted for a long time: running etcd on the same host as the API server. In the traditional layout, every read and write crosses the network between separate machines. Each trip is fast on its own, but at thousands per second, the travel time adds up. With collocation, the API server talks to its local etcd over a loopback interface, and pod scheduling and controller reconciliation get measurably faster. For workloads where job controller queue depth is the binding constraint, shaving milliseconds off each API call means many more jobs are processed per second before the queue starts growing.

This is where the operator agent paid off. When etcd runs on the same host, an etcd member comes and goes whenever a control-plane host is replaced, which happens routinely. That only works if membership management is completely safe and automatic, which is exactly what the agent was already doing. We did not have to build a colocation from scratch; we built it on top of infrastructure that had been managing etcd membership safely since day one.

Collocation also taught us a lesson worth passing on: the convenient path needs a failover in case it breaks. The local etcd is the fast path, but if it becomes impaired, the API server fails over to another etcd member that is actively serving other API servers from the same journal. When you optimize for the common case, design just as deliberately for the moment that optimization is not available.

Fixing bottlenecks across the stack

At extreme scale, you have to address bottlenecks across the entire Kubernetes stack, and most of them are not bugs in the traditional sense. They are design choices that were correct at the scale Kubernetes originally targeted and break down only when the numbers get large. Rather than working around them internally, we fix them upstream so the entire community benefits.

One example involved the watch cache, the in-memory layer that distributes state changes from etcd to every controller watching for updates. When a controller starts, it requests a full snapshot of the current state via a mechanism called WatchList, and the existing implementation holds a shared read lock for the duration of the response build. 

At hundreds of thousands of objects, that work runs long enough to starve the writer that needs exclusive access, so the cache’s resource version cannot advance. Consistent reads see a stale cache and fail over to etcd, while the response building churns through hundreds of thousands of allocations under the lock. We identified this as a limitation in the watch-cache’s locking model and are working with the community to refactor the underlying data structures and interfaces to eliminate the contention.

The same shape appears elsewhere. In the Horizontal Pod Autoscaler, a single mutex protecting the scaling state becomes a serialization point at high HPA counts, where workers spend nearly all their time blocked rather than doing useful work. A redesigned data store (PR #139142) restores parallelism and raises reconciliation throughput by orders of magnitude. In the scheduler, we identified a bottleneck (issue #138426): every scheduling cycle rebuilds a set of in-use persistent volumes by scanning every node in the cluster, even for pods that do not use storage at all. The fix computes that information lazily, and only for pods that actually need it, restoring throughput at scale.

Each of these started from a real production workload hitting a cliff, and we are working on the fixes upstream so the improvements reach every Kubernetes user.

From engineering to guarantees: EKS Provisioned Control Plane

The engineering described above made the EKS control plane more resilient and performant. But customers had a different problem: they could observe that the control plane kept up today, but they could not reserve its capacity the way they reserve compute or GPU capacity. 

A team planning a thousand-node training run could secure the instances weeks in advance, yet had no equivalent mechanism for the orchestration layer that would coordinate them. EKS Provisioned Control Plane fills that gap. It exposes the control plane’s performance as dimensions you size explicitly, backed by the same kind of commitment you expect from the rest of your infrastructure.

You choose a scaling tier that maps to concrete, measurable capabilities: API request concurrency, pod scheduling rate, and cluster database size. The tiers range from XL through 8XL. At the top end, 8XL on Kubernetes 1.34 provides 16,000 concurrent API request seats, 400 pods-per-second scheduling rate, and 16 GB of cluster database storage, all backed by a 99.99% availability SLA measured in one-minute intervals.

Tiers are not static. You step up before a GPU training run or a large sales event, step back down during quiet periods, or grow permanently as your platform matures. Configuration happens through the console, CLI, eksctl, CloudFormation, or Terraform on any cluster, without recreation or downtime. 

For AI workloads, orchestration capacity is planned alongside GPU capacity, available when the compute comes online. For analytics platforms submitting hundreds of jobs per minute, the control plane is ready for the burst before it arrives. And for organizations that need environmental consistency across staging, production, and disaster recovery, the same tier guarantees consistent performance characteristics everywhere.


Taking the same foundation to the edge

Some workloads cannot move to the cloud, whether due to data sovereignty requirements, latency constraints, or unreliable connectivity to the Region. Running Kubernetes in these disconnected environments introduces unique challenges: etcd must remain durable on hardware with only a few machines, the cluster must self-heal without reaching the cloud, and observability must survive network partitions that last days. 

With the updated architecture for EKS local clusters on instance store Outposts, we brought edge clusters onto the same management plane and software stack as EKS clusters in the cloud.

The control plane lives in an EKS-managed account on the Outpost rather than in the customer’s account, so customers never manage control plane instances, etcd backups, or logging agents themselves, and they cannot accidentally break the thing keeping their cluster alive. The same machine images, container images, and operator agent run in both places, with edge-specific behaviors selected by configuration. 

Because it is the same stack, new Kubernetes and EKS platform versions arrive in lockstep with their cloud release, and features like EKS add-ons, Pod Identity, and access entries work the same way they do in a Region.

The hardest part was keeping etcd healthy on hardware with only a few machines that may be cut off from the cloud for days at a time. We solved it by extending the same agent. It keeps a spare copy of the data continuously up to date and promotes it the instant a machine fails, so the cluster heals itself with no human involvement and no connection to the cloud. 

Observability survives the disconnect, too: the metrics agent continues collecting and writing to local disk, shedding the least critical data first when space runs short, so the signals that matter most are the last to go. When the link returns, the buffered data is flushed back with its original timestamps.

All of this only works because the system was designed from the start to operate without anyone logged in. That same design is what makes it possible to deploy changes safely across the entire fleet.

Operating safely at fleet scale

Every one of these changes was deployed to a running fleet of hundreds of thousands of clusters. The journal migration and collocation required transitioning each cluster individually. Every migration follows a strict sequence: validate pre-conditions, create a point-in-time snapshot, perform the switchover, validate post-conditions. If any step fails, the system rolls back automatically. 

Rollouts proceed cell by cell, zone by zone, region by region, with automated monitoring comparing latency, error rates, and throughput between updated and non-updated clusters. Any statistically significant deviation triggers an automatic halt.

What made all of this possible is that EKS is built to operate without human intervention at the individual cluster level. Through Zero Operator Access, the architecture prevents AWS personnel from having technical pathways to access customer content in the managed control plane. A system designed to work without human access must be observable, recoverable, and automatable from the start, and that same discipline is what enables operating at extreme scale.

Three operational lessons shaped how we approach this work.

The first is that a healthy leader is not the same as a working one. The control plane’s controllers run in an active-passive configuration, and early on, we treated an unhealthy standby as if cluster operations had halted. They had not; what matters is whether a leader exists. But the harder lesson: a leader can quietly stop making progress while still renewing its lease and passing every health check. The signal that caught this was watching the controller’s work queue depth. If the queue fills while the leader looks healthy, the system is falling behind in ways no liveness probe will catch.

“A leader can quietly stop making progress while still renewing its lease and passing every health check. The signal that caught this was watching the controller’s work queue depth.”

The second is that maintenance ordering matters as much as the maintenance itself. etcd defragmentation is blocking, and the pause grows with database size. When it hit the leader, every write stalled. We taught the agent to move leadership to a healthy node before defragmenting, so the disruptive work always lands on a follower while writes keep flowing.

The third is that liveness is not readiness. A process can be alive but not ready while it warms caches, and routing based solely on liveness sends requests to an instance that cannot handle them. Equally, readiness flapping during graceful draining should never trigger a restart. We keep the two signals strictly separate: one decides recovery; the other decides routing.

None of this work is visible from the outside, and that is the point. The largest clusters taught us lessons that made every cluster faster. The riskiest migrations produced safety machinery that protects every upgrade. The upstream fixes we contributed for workloads at the edge of what Kubernetes can handle flow back to every user of the project.

“None of this work is visible from the outside, and that is the point.”

When you deploy on EKS and your pods come up in seconds, even during a burst, even when something behind the scenes goes wrong, that speed is not accidental. It is the accumulated result of years of operating at scales where small problems can become big ones fast, and engineering the system to contain them before they do.

To explore the architectures referenced in this post, see EKS Provisioned Control Plane and local Amazon EKS clusters on AWS Outposts.

The post Operating Kubernetes at scale: a few stories from running Amazon EKS appeared first on The New Stack.