We instrumented the full path from pod creation to first inference response on a GPU node running a 70B-class model. Eight minutes. Six sequential phases. We expected one bottleneck. We found six, and which one dominates depends on model size.
For a 64 GB model, 65% of the startup time is spent recompiling CUDA kernels that produce identical output every time. For a 203 GB model, 92% of the time is spent downloading weights from S3 through a calling pattern that leaves 98% of available bandwidth idle. Both are fixable with configuration changes. Neither is fixed by default.
“Eight minutes. Six sequential phases. We expected one bottleneck. We found six.”
We define time to first token served (TTFTS) as the wall-clock duration from pod creation to the first inference response leaving the GPU. Not time to first token (TTFT), which measures per-request latency once the model is warm. TTFTS is the one-time startup tax. TTFT begins where TTFTS ends.
Here’s what we achieved:
| Scenario | Description | Before | After | Reduction |
| Pod restart on warm node | Weights loading + compilation on existing node | 1.5-8 min | under 30s | 80-93% |
| New node from scratch | Fresh node provisioned, nothing cached | 8-15 min | ~5 min | 40-65% |
The warm-node row is what you pay on every pod restart: scale-up events, rolling updates, OOM recoveries. That’s the 80-93% win, and it requires only configuration changes. The cold-node row includes ~2 minutes of fixed infrastructure cost (node provisioning and framework initialization) that no application-layer optimization can remove. The rest is avoidable waste that we eliminated through platform and configuration fixes. The warm-node optimizations are environment variables and a volume mount that work on any Kubernetes cluster. The cold-node optimizations require EKS Auto Mode, which comes pre-configured with pre-compiled NVIDIA drivers, SOCI (Seekable OCI) parallel image pull, and NVMe instance store mounting.
All model startup measurements were taken on p5.48xlarge instances running Amazon EKS Auto Mode, with S3 traffic routed directly (bypassing the NAT Gateway) and container images in a private Amazon ECR repository (same region as compute). Model startup improvement ratios (80-93%) hold consistently across instance types (validated on P-family and G-family). Cold-node times vary with network bandwidth and CPU count. For the weights loading and compilation cache configuration, see Accelerate model loading on Amazon EKS.
The Kubernetes ecosystem has made real progress on the inference stack in 2026. OCI image volumes are now stable for model delivery. Dynamic Resource Allocation (DRA) gives GPUs structured attributes instead of opaque integer counts and provides flexibility in allocating GPUs to workloads. Gateway API has inference-aware routing extensions. But none of these primitives address the full cold-start stack: the six layers between “pod pending” and “first token served,” each with its own bottleneck and its own fix.
When a new inference pod starts on a freshly provisioned GPU node, it passes through six distinct phases before serving its first request:
Each layer has a different bottleneck, a different fix, and a different owner.
Before diving into each layer, one finding shaped every decision we made: the bottleneck is not fixed.
We instrumented the model startup path (layers 4 and 5) and measured each phase independently for two model sizes:
64 GB model (Qwen3.6-35B-A3B):
203 GB model (Llama-4-Scout, TP=4 where TP is tensor parallelism, splitting the model across GPUs):
For models under ~100 GB, compilation dominates. For larger models, network transfer dominates. torch.compile time stays roughly constant (it depends on graph complexity, not parameter count). Weights loading scales linearly with file size.
“For models under ~100 GB, compilation dominates. For larger models, network transfer dominates.”
This means any single-layer optimization has a ceiling.
On EKS Auto Mode and Karpenter-managed clusters, node provisioning takes approximately 60-90 seconds for accelerated instances from pod pending to node Ready. Karpenter calls the EC2 Fleet API directly and reacts to pending pods within seconds, keeping provisioning at the EC2 launch floor.
The NVIDIA GPU Operator in its default configuration adds 2-3 minutes to node boot while it compiles the driver kernel module from source. This cost repeats on every new node.
When the platform controls the full stack (OS image, kernel version, driver version, boot sequence) it can pre-compile driver kernel modules at image build time. The node boots, runs modprobe to load an already-compiled .ko file, and the GPU is ready in seconds.
This matters more now than it used to. Blackwell-architecture GPUs (G7, G7e instances) require NVIDIA’s open-source kernel modules exclusively. Older Maxwell/Pascal/Volta GPUs can only run proprietary modules. A cluster with both legacy and next-gen GPU nodes needs different drivers, different AMIs, different upgrade cycles. A managed platform that pre-compiles the correct module per instance family eliminates this complexity.
On EKS Auto Mode, the GPU driver loads in seconds (pre-compiled at image build time), compared to the 2-3 minutes a runtime-compilation approach requires.
A production vLLM or SGLang inference image is typically 8-12 GB compressed. Standard containerd pulls layers sequentially, decompresses them one by one in memory, and writes them to disk. At this size, sequential pull takes 2-4 minutes on a cold node depending on instance type and available CPU cores. For larger custom images (30-50 GB compressed), containerd can run out of memory entirely during decompression.
EKS Auto Mode uses SOCI’s parallel pull mode, which replaces containerd’s default snapshotter. The SOCI snapshotter downloads layer chunks concurrently via HTTP range requests and writes each chunk directly to its target byte position on disk (no in-memory ordering buffer). Decompression runs in parallel across all available CPU cores.
Pull time is bottlenecked by CPU-bound decompression, not network bandwidth. We confirmed this directly: a p4d.24xlarge with 400 Gbps networking achieved only ~1 Gbps effective pull throughput because CPU decompression was the constraint. On instances with capable, current-generation CPUs, SOCI parallel pull reduces image pull time from 2-4 minutes to 30-60 seconds. The dominant factor is per-core decompression throughput, which depends on CPU generation and instruction-set support, more than raw core count. A newer CPU with fewer cores can outperform an older one with more.
For a deeper look at how bounded-memory parallel pull handles images exceeding 30 GB without OOM, see Bounded-Memory Parallel Image Pulling for Large Container Images.
The obvious optimization for weights loading: more parallel connections. Split the model files into small chunks, download them concurrently, saturate the network pipe.
We tested it on p5.48xlarge with the 64 GB model streaming from same-region S3. The results were counterintuitive:
| Chunk size | Connections needed | Weights load time |
| 256 MB | 256 | 13.98s |
| 512 MB | 128 | 14.20s |
| 2 GB | 34 | 13.62s |
| 4 GB | 17 | 13.35s |
| 8 GB | 9 | 21.80s (+56%) |
256 parallel connections provided no benefit over 17. The only failure mode was 8 GB chunks (exceeding shard file size), which caused a 56% regression.
Why? Because the open-source Run:ai Model Streamer (integrated into vLLM and SGLang) processes S3 range requests sequentially within each worker thread. A worker assigned to a 3.9 GB shard file downloads its byte-range requests one after another on a single connection. The parallelism comes from running multiple workers on different files, not from splitting one file into more pieces.
We settled on 4 GB chunks matching typical SafeTensors shard size (3-5 GB per file) with an aggressive timeout-and-retry for slow requests. S3 GET latency has a measurable long tail: in our testing, a meaningful fraction of requests took 2-3x longer than median, and a single stalled connection holds up the entire model load. Rather than wait, we kill stalled connections after a few seconds below a speed threshold and retry on a fresh connection. This follows S3’s own performance guidance.
For the 203 GB model, these config-only changes reduced weights loading from 423 seconds to 25 seconds (94% improvement). For the 64 GB model, from 29 seconds to 12 seconds. No code modifications, just environment variables. The tuning consists of three settings: chunk size aligned to shard file boundaries (eliminating the serial sub-request problem), a minimum-speed threshold that kills and retries stalled S3 connections, and explicit concurrency matching the number of shard files per tensor-parallel rank.
Every time a vLLM or SGLang pod starts, PyTorch traces the model’s computation graph and compiles it to optimized CUDA kernels. This takes 34-53 seconds depending on model architecture. The output is identical every time for the same model, GPU type, and tensor-parallel configuration.
And Kubernetes throws it away on every pod restart. Pods use ephemeral storage by default. When a pod terminates, its local filesystem is destroyed. The next pod recompiles from scratch.
“The output is identical every time for the same model, GPU type, and tensor-parallel configuration. And Kubernetes throws it away on every pod restart.”
Point the torch.compile cache directory at local NVMe instance store. GPU instances ship with NVMe that EKS Auto Mode mounts automatically. First pod compiles and writes ~15-30 MB of cached kernels. The second pod on the same node loads pre-compiled binaries in 4-6 seconds. One volume mount and environment variables.
The cache is safe because the compiled artifacts are deterministic: same model architecture + GPU architecture + tensor-parallel degree + PyTorch version equals valid cache. An image update or hardware change triggers exactly one recompilation.
torch.compile time is hardware independent. The same model compiles in ~52 seconds whether running on H100 or A100. The cache hit (4-6 seconds) is equally consistent across GPU types. This means the optimization works identically regardless of instance type.
After weights are loaded and kernels compiled, the inference engine must capture CUDA execution graphs and profile KV cache memory. With compiled kernels cached, this completes in 30-45 seconds. Without cache, graph capture triggers additional JIT compilation and takes 60-120 seconds.
This is why the torch.compile cache has an outsized impact: it accelerates not just layer 5 but also layer 6. Cached compilation reduces a 2-3-minute combined phase to a 35-50-second combined phase.
Framework initialization (Python interpreter startup and PyTorch import) adds tens of seconds of fixed overhead that cannot be reduced through configuration.
The six layers compound. Platform fixes (layers 1-3) eliminate 4-8 minutes of overhead: pre-compiled drivers replace 2-3 minutes of runtime compilation, parallel pull reduces image transfer time from 2-4 minutes to 30-60 seconds, and Karpenter keeps node provisioning to its hardware minimum. Configuration changes (layers 4-5) cut the remaining model startup by 80-93%. Engine initialization (layer 6) drops from 60-120 seconds to 30-45 seconds once the compile cache is warm. Together, cold-node TTFTS drops from 8-15 minutes to approximately 5 minutes.
| Configuration | First pod | Subsequent pod (warm node) |
| Baseline (no tuning) | 82s | 82s |
| + S3 chunk tuning | 65s | 65s |
| + torch.compile cache | 65s | 16s |
| Improvement | -21% | -80% |
| Configuration | First pod | Subsequent pod (warm node) |
| Baseline (no tuning) | 457s | 457s |
| + S3 chunk tuning | 59s | 59s |
| + torch.compile cache | 59s | 32s |
| Improvement | -87% | -93% |
The warm-node subsequent pod number is what matters most for production. It’s what you pay on every pod restart. The 80-93% reduction is consistent across instance types because the optimizations target software bottlenecks (calling patterns, redundant compilation), not hardware limits.
Why does any of this matter? Because GPU nodes are expensive and inference traffic is bursty.
A single p5.48xlarge costs $55/hour on-demand. Even G-family instances commonly used for inference cost $10-20/hour. Every minute of cold start is GPU time you’re paying for but not using. If your autoscaler needs 8+ minutes to bring up new capacity, you must over-provision (burn money on idle GPUs) or accept latency spikes during traffic surges.
“Every minute of cold start is GPU time you’re paying for but not using.”
When model startup drops to 16-32 seconds on warm nodes, the calculus changes. You can scale more aggressively, keep fewer buffer nodes, and respond to traffic spikes without multi-minute startup delays.
For the complete configuration guide, including environment variables, YAML manifests, and instance-specific recommendations, see “Accelerate model loading on Amazon EKS” in the Amazon EKS User Guide.
The post Cut GPU inference cold start from 8 minutes to less than a minute appeared first on The New Stack.
Nvidia plans to acquire Hugging Face for about $12.9 billion, securing the central platform for open AI models. More than 18 million developers and 200,000 companies use the hub. CEO Jensen Huang promises to keep the platform open and hardware-neutral, but the deal also hands him a powerful distribution channel for compute.
The article Nvidia buys the front door to open AI as closed labs increasingly design their own silicon appeared first on The Decoder.
The OpenAI agents involved in last month’s incursion into Hugging Face were trained so heavily on winning a competition that they pursued a relentless campaign to cheat, a new report documented. In the process, and without authorization, they created an improvised message board to hatch a plan that ultimately landed them squarely inside the latter company’s network.
Over the course of May and June, OpenAI gave the agents what the company described as “impossible tasks” to complete on the benchmarking framework ExploitGym. The internal test was designed to test how the agents would respond. To get a full understanding of the agent capabilities, company engineers disabled safety guardrails that normally are in place to prevent the sort of hacks that eventually hit Hugging Face and one other undisclosed organization. The stymied agents’ training made them so focused on winning that they performed tasks they were never explicitly instructed to follow.
The first step was creating a message board that allowed the agents to pass notes to each other. OpenAI hadn’t provided any such platform, so the agents repurposed a platform called Artifactory, which OpenAI was using in internal testing of several unreleased hacking agents. OpenAI was using Artifactory as one of the measures to prevent the agents from egressing its isolated sandboxes and accessing the Internet, while at the same time simulating a real-world hacking environment.
© Getty Images
Nvidia is buying open-source AI platform Hugging Face for $12.9 billion, about 80 times its $150 million annual revenue. The deal fits Nvidia's push to invest billions in open AI models while closed providers like OpenAI and Anthropic move away from Nvidia hardware.
The article Nvidia snaps up Hugging Face for $12.9 billion as closed AI labs pull away appeared first on The Decoder.
Most organizations that know they should sign their images still don’t. Not because they disagree, but because the path to doing it well has been too long. The result is a delivery pipeline built on trust that nobody can verify.
Unsigned container images create an open door for attackers at every stage of the delivery pipeline. Malicious images masquerade as legitimate packages, waiting to be pulled by an unsuspecting team. Compromised CI/CD pipelines silently inject tampered artifacts into production builds with no cryptographic evidence of modification. Stolen credentials let a bad actor impersonate a trusted publisher. Even within a single organization, inconsistent practice means some teams sign while others skip the step entirely, leaving gaps in the chain of trust that nobody has mapped. Compounding all of it is base image inheritance. Every container image inherits the security posture of its parent, so one compromised base image can propagate across dozens of downstream services before anyone notices.
“Scanning is fundamentally reactive. One tells you what is inside. The other tells you whether you can trust it.”
Scanning is fundamentally reactive. It answers, “what vulnerabilities exist in this image?” It cannot answer the question that matters more as artifacts get harder to inspect: “who built this, and has it been modified since it left the build system?” That is the domain of cryptographic signing, which provides proactive provenance. The two are complementary, not interchangeable. One tells you what is inside. The other tells you whether you can trust it.
The workloads have changed faster than the tooling. Model weights, training datasets, inference runtimes, and agent tooling increasingly ship as OCI artifacts. A pickled PyTorch checkpoint itself has no CVE to match against. Safer serialization formats like .safetensors remove the code execution path, but they say nothing about who produced the weights or whether they’re the ones you meant to load. There is no vulnerability database for a set of trained weights, and the CVE and SCA based scanning that registries run has nothing to compare them to.
This is not theoretical. In February 2024, JFrog researchers found a malicious PyTorch model on Hugging Face that opened a reverse shell the moment it loaded, abusing pickle’s __reduce__ hook to execute arbitrary code on torch.load(). Their analysis surfaced roughly 100 models on the hub carrying genuinely malicious payloads. No CVE fired, because there was nothing for a CVE to describe. The malice lived in the serialized weights. Model-specific scanning has since appeared to close that gap. Hugging Face runs ClamAV plus a pickle import scan on every file pushed to the Hub, statically disassembling the pickle’s opcode stream to flag dangerous imports. While they help, they are also already being evaded. In February 2025, ReversingLabs described nullifAI, two models that slipped past picklescan by compressing with 7z instead of ZIP and by corrupting the pickle stream immediately after the payload ran, so static analysis errored out on a file whose reverse shell would have already run. Hugging Face removed the models inside 24 hours and patched picklescan. That is the shape of the problem. Pattern matching scanners are a line that keeps moving, and each one answers whether a file resembles something known to be bad. None of them answers where the file came from.
“A tampered application image defaces a page. A tampered AI model artifact corrupts predictions at scale.”
AI is widening the attack surface in the same motion. Coding assistants suggest dependencies that never pass a human threat model, and that code gets containerized and shipped faster than review can keep up. The blast radius changed too. A tampered application image defaces a page. A tampered AI model artifact corrupts predictions at scale, poisons recommendations served to millions, or in the agentic case takes actions in production: API calls, tool invocations, spend. And when you consume a pre-trained model, you inherit every upstream decision about its training data and its security with zero visibility into any of them. Provenance stopped being a question about your application code. It became a question about the model, the agent, and the tooling that carries them.
But signing is not a checkbox. It is a chain. It only works if every link holds.
Operating the registry at the scale of Amazon ECR has taught us something that shaped how we think about supply chain security. Most teams don’t verify images. They verify addresses. An admission policy allows images from your registry account, push credentials belong to the pipeline rather than to people, and a scanner blocks critical CVEs. That stops a lot of attacks. What it can’t do is tell a good image from a bad one once it’s inside the boundary, because registry provenance is a claim about location, not origin. Anything that can write to the repository produces an image that looks legitimate: a leaked CI token, a misconfigured cross-account role, a compromised build step. Digest pinning tells you that you got the bytes you asked for, not that those were the right bytes to ask for.
Every container image passes through a registry before it runs. It is the last system in the path that sees every artifact, knows who pushed it, and controls who can pull it. It already holds identity context, already enforces access policy, and already stores the metadata that describes what an image contains. The hard part of image signing is doing it consistently across every team and every pipeline without slowing anyone down. The registry is the only layer that can make it invisible.
“The hard part of image signing is doing it consistently across every team. The registry is the only layer that can make it invisible.”
Signing does not make forgery impossible. An attacker who fully compromises a trusted signing identity, stealing both the credential and the permission to sign, can produce a validly signed malicious image that passes verification. What signing does is shrink the attack surface. Without it, tampering anywhere in the path works, because nothing downstream checks. With signing and enforcement, none of it works unless the attacker compromises one narrowly scoped signer, and that rogue signature is an auditable event tied to an identity instead of an anonymous overwrite. Revoke the identity and the whole fleet stops trusting it in one change. Signing turns an invisible, unbounded problem into a scoped, attributable, revocable one.
Signing is a three-step process:
Sign: Generate a signature at build or push time, binding the image digest to a verifiable identity. The hard question is custody: who holds the private key, and how is it rotated and protected?
Verify: At pull time, and critically before the workload is admitted, check the signature against a trust policy which is a declared list of the identities you trust to have signed what you are about to run.
Enforce: A Kubernetes admission controller like Kyverno blocks any image not signed by a trusted identity from ever running. Signing without enforcement changes nothing.
Enabling signing comes with operational cost. Engineers had to install and configure client-side tooling like Notation CLI or Cosign, then own their signing keys, certificates, rotation schedules, and revocation lists, then build custom automation to wire signing into every pipeline. Across an enterprise with thousands of uniquely configured pipelines, that rollout took weeks to months. What we wanted to know was whether the registry itself could absorb the cost, so that signing could become a property of pushing an image rather than a project each team takes on. The answer to that question became Amazon ECR Managed Signing.
The mechanics are deliberately boring, which took some doing. You create a registry level signing configuration with up to ten rules, each pairing a signing profile with repository filters, and every matching push gets signed from then on.
Managed Signing answers the custody question by not giving you the keys. You configure a signing profile in AWS Signer, which pins the signing algorithm, a validity period, and the identity that appears in the signature. Signer keeps the certificate and the private key. This means no signing key ever sits in a repo, a runner, or a build log. Validity defaults to 135 months, so signatures won’t expire on you. Revocation is what you’ll actually use when you find out a build was compromised.
Then what gets signed, which is narrower than people assume. Signer signs a small Notary payload whose targetArtifact describes the image manifest: media type, digest, size. Not the image bytes directly. Because the signed material is content addressed, verification becomes a statement about exact bytes. The signature itself lands in the same repository as a detached OCI artifact, typed application/vnd.cncf.notary.signature, with a subject descriptor pointing at the image manifest digest. One image can carry signatures from several profiles as your trust requirements change.
Signing happens asynchronously, which keeps Signer off the push path. A synchronous call would turn an availability dip or a throttle into a failed docker push for a developer, and it would put signing latency in front of every pipeline. The push commits first, and ECR calls SignPayload after.
Verification and enforcement happen downstream, and the trust policy is where the whole design becomes legible. Your cluster operator writes it and imports it with notation policy import. It’s a short reviewable file:
Login
{
"version": "1.0",
"trustPolicies": [
{
"name": "aws-signer-tp",
"registryScopes": ["*"],
"signatureVerification": { "level": "strict" },
"trustStores": ["signingAuthority:aws-signer-ts"],
"trustedIdentities": [
"arn:aws:signer:us-east-1:111122223333:/signing-profiles/platform_images"
]
}
]
}
That policy says a workload runs only if it carries a signature chaining to the AWS Signer root and produced by that specific profile. Admission does the work in order: resolve the reference to a digest, fetch the signature via OCI Referrers API, validate the envelope against its embedded certificate chain, walk that chain to the root in the trust store, check the signing identity against trustedIdentities, and check revocation. Revoking a profile makes verification fail wherever that profile is trusted. New admissions stop immediately and running pods pick it up when they’re next rescheduled. On EKS you get there with Gatekeeper and Ratify, or with Kyverno. Both paths use the AWS Signer plugin. Every link is checkable by the cluster itself, from the artifact plus a root certificate without asking the verifier to trust the registry it pulled from, or the pipeline that pushed.
Vulnerability scanning answers a question that mattered in the application era: what is broken inside this image? The AI era asks a harder one that scanning was never built to answer. Can you prove where this came from, and that no one touched it?
The cryptography was never the hard part. Making it the path of least resistance was. Sign, verify, and enforce, and let the registry carry the tax so your teams don’t have to.
To explore what’s referenced here, see Amazon ECR managed signing and signature verification on Amazon EKS.
The post Your container images are unsigned. In the AI era, that’s a ticking time bomb. appeared first on The New Stack.
