Most organizations that know they should sign their images still don’t. Not because they disagree, but because the path to doing it well has been too long. The result is a delivery pipeline built on trust that nobody can verify.
Unsigned container images create an open door for attackers at every stage of the delivery pipeline. Malicious images masquerade as legitimate packages, waiting to be pulled by an unsuspecting team. Compromised CI/CD pipelines silently inject tampered artifacts into production builds with no cryptographic evidence of modification. Stolen credentials let a bad actor impersonate a trusted publisher. Even within a single organization, inconsistent practice means some teams sign while others skip the step entirely, leaving gaps in the chain of trust that nobody has mapped. Compounding all of it is base image inheritance. Every container image inherits the security posture of its parent, so one compromised base image can propagate across dozens of downstream services before anyone notices.
“Scanning is fundamentally reactive. One tells you what is inside. The other tells you whether you can trust it.”
Scanning is fundamentally reactive. It answers, “what vulnerabilities exist in this image?” It cannot answer the question that matters more as artifacts get harder to inspect: “who built this, and has it been modified since it left the build system?” That is the domain of cryptographic signing, which provides proactive provenance. The two are complementary, not interchangeable. One tells you what is inside. The other tells you whether you can trust it.
The workloads have changed faster than the tooling. Model weights, training datasets, inference runtimes, and agent tooling increasingly ship as OCI artifacts. A pickled PyTorch checkpoint itself has no CVE to match against. Safer serialization formats like .safetensors remove the code execution path, but they say nothing about who produced the weights or whether they’re the ones you meant to load. There is no vulnerability database for a set of trained weights, and the CVE and SCA based scanning that registries run has nothing to compare them to.
This is not theoretical. In February 2024, JFrog researchers found a malicious PyTorch model on Hugging Face that opened a reverse shell the moment it loaded, abusing pickle’s __reduce__ hook to execute arbitrary code on torch.load(). Their analysis surfaced roughly 100 models on the hub carrying genuinely malicious payloads. No CVE fired, because there was nothing for a CVE to describe. The malice lived in the serialized weights. Model-specific scanning has since appeared to close that gap. Hugging Face runs ClamAV plus a pickle import scan on every file pushed to the Hub, statically disassembling the pickle’s opcode stream to flag dangerous imports. While they help, they are also already being evaded. In February 2025, ReversingLabs described nullifAI, two models that slipped past picklescan by compressing with 7z instead of ZIP and by corrupting the pickle stream immediately after the payload ran, so static analysis errored out on a file whose reverse shell would have already run. Hugging Face removed the models inside 24 hours and patched picklescan. That is the shape of the problem. Pattern matching scanners are a line that keeps moving, and each one answers whether a file resembles something known to be bad. None of them answers where the file came from.
“A tampered application image defaces a page. A tampered AI model artifact corrupts predictions at scale.”
AI is widening the attack surface in the same motion. Coding assistants suggest dependencies that never pass a human threat model, and that code gets containerized and shipped faster than review can keep up. The blast radius changed too. A tampered application image defaces a page. A tampered AI model artifact corrupts predictions at scale, poisons recommendations served to millions, or in the agentic case takes actions in production: API calls, tool invocations, spend. And when you consume a pre-trained model, you inherit every upstream decision about its training data and its security with zero visibility into any of them. Provenance stopped being a question about your application code. It became a question about the model, the agent, and the tooling that carries them.
But signing is not a checkbox. It is a chain. It only works if every link holds.
Operating the registry at the scale of Amazon ECR has taught us something that shaped how we think about supply chain security. Most teams don’t verify images. They verify addresses. An admission policy allows images from your registry account, push credentials belong to the pipeline rather than to people, and a scanner blocks critical CVEs. That stops a lot of attacks. What it can’t do is tell a good image from a bad one once it’s inside the boundary, because registry provenance is a claim about location, not origin. Anything that can write to the repository produces an image that looks legitimate: a leaked CI token, a misconfigured cross-account role, a compromised build step. Digest pinning tells you that you got the bytes you asked for, not that those were the right bytes to ask for.
Every container image passes through a registry before it runs. It is the last system in the path that sees every artifact, knows who pushed it, and controls who can pull it. It already holds identity context, already enforces access policy, and already stores the metadata that describes what an image contains. The hard part of image signing is doing it consistently across every team and every pipeline without slowing anyone down. The registry is the only layer that can make it invisible.
“The hard part of image signing is doing it consistently across every team. The registry is the only layer that can make it invisible.”
Signing does not make forgery impossible. An attacker who fully compromises a trusted signing identity, stealing both the credential and the permission to sign, can produce a validly signed malicious image that passes verification. What signing does is shrink the attack surface. Without it, tampering anywhere in the path works, because nothing downstream checks. With signing and enforcement, none of it works unless the attacker compromises one narrowly scoped signer, and that rogue signature is an auditable event tied to an identity instead of an anonymous overwrite. Revoke the identity and the whole fleet stops trusting it in one change. Signing turns an invisible, unbounded problem into a scoped, attributable, revocable one.
Signing is a three-step process:
Sign: Generate a signature at build or push time, binding the image digest to a verifiable identity. The hard question is custody: who holds the private key, and how is it rotated and protected?
Verify: At pull time, and critically before the workload is admitted, check the signature against a trust policy which is a declared list of the identities you trust to have signed what you are about to run.
Enforce: A Kubernetes admission controller like Kyverno blocks any image not signed by a trusted identity from ever running. Signing without enforcement changes nothing.
Enabling signing comes with operational cost. Engineers had to install and configure client-side tooling like Notation CLI or Cosign, then own their signing keys, certificates, rotation schedules, and revocation lists, then build custom automation to wire signing into every pipeline. Across an enterprise with thousands of uniquely configured pipelines, that rollout took weeks to months. What we wanted to know was whether the registry itself could absorb the cost, so that signing could become a property of pushing an image rather than a project each team takes on. The answer to that question became Amazon ECR Managed Signing.
The mechanics are deliberately boring, which took some doing. You create a registry level signing configuration with up to ten rules, each pairing a signing profile with repository filters, and every matching push gets signed from then on.
Managed Signing answers the custody question by not giving you the keys. You configure a signing profile in AWS Signer, which pins the signing algorithm, a validity period, and the identity that appears in the signature. Signer keeps the certificate and the private key. This means no signing key ever sits in a repo, a runner, or a build log. Validity defaults to 135 months, so signatures won’t expire on you. Revocation is what you’ll actually use when you find out a build was compromised.
Then what gets signed, which is narrower than people assume. Signer signs a small Notary payload whose targetArtifact describes the image manifest: media type, digest, size. Not the image bytes directly. Because the signed material is content addressed, verification becomes a statement about exact bytes. The signature itself lands in the same repository as a detached OCI artifact, typed application/vnd.cncf.notary.signature, with a subject descriptor pointing at the image manifest digest. One image can carry signatures from several profiles as your trust requirements change.
Signing happens asynchronously, which keeps Signer off the push path. A synchronous call would turn an availability dip or a throttle into a failed docker push for a developer, and it would put signing latency in front of every pipeline. The push commits first, and ECR calls SignPayload after.
Verification and enforcement happen downstream, and the trust policy is where the whole design becomes legible. Your cluster operator writes it and imports it with notation policy import. It’s a short reviewable file:
Login
{
"version": "1.0",
"trustPolicies": [
{
"name": "aws-signer-tp",
"registryScopes": ["*"],
"signatureVerification": { "level": "strict" },
"trustStores": ["signingAuthority:aws-signer-ts"],
"trustedIdentities": [
"arn:aws:signer:us-east-1:111122223333:/signing-profiles/platform_images"
]
}
]
}
That policy says a workload runs only if it carries a signature chaining to the AWS Signer root and produced by that specific profile. Admission does the work in order: resolve the reference to a digest, fetch the signature via OCI Referrers API, validate the envelope against its embedded certificate chain, walk that chain to the root in the trust store, check the signing identity against trustedIdentities, and check revocation. Revoking a profile makes verification fail wherever that profile is trusted. New admissions stop immediately and running pods pick it up when they’re next rescheduled. On EKS you get there with Gatekeeper and Ratify, or with Kyverno. Both paths use the AWS Signer plugin. Every link is checkable by the cluster itself, from the artifact plus a root certificate without asking the verifier to trust the registry it pulled from, or the pipeline that pushed.
Vulnerability scanning answers a question that mattered in the application era: what is broken inside this image? The AI era asks a harder one that scanning was never built to answer. Can you prove where this came from, and that no one touched it?
The cryptography was never the hard part. Making it the path of least resistance was. Sign, verify, and enforce, and let the registry carry the tax so your teams don’t have to.
To explore what’s referenced here, see Amazon ECR managed signing and signature verification on Amazon EKS.
The post Your container images are unsigned. In the AI era, that’s a ticking time bomb. appeared first on The New Stack.
When the image is the bottleneck: Machine learning changed what a container image looks like. A typical application ships in a few hundred MB and starts in seconds. A modern ML inference image carries a deep-learning framework, the CUDA stack, and sometimes the model weights. These images roughly reach 20 to 30 GB, with some even higher. On the GPU and accelerated instances these run on, pulling one of those images takes several minutes before the application can serve its first request: minutes during which provisioned accelerators are ready to process real work but waiting for images to be pulled.
We hit this bottleneck on a production ML platform running on Amazon EKS. The team needed pods ready within two minutes, but image pull alone consumed several minutes. Each pod pulled a roughly 30 GB container image on top of loading model data from a shared filesystem. The images were rebuilt on a regular cadence, so worker nodes faced cold pulls with no usable local cache. While the image was pulled, accelerators sat idle, autoscaling lagged demand, and request queues built up.
The natural first suspect for these large image pull times was the network or the registry. After all, 30 GB is a lot of data. However, profiling the image pull path showed neither was the bottleneck on accelerated instances with 100 to 400 Gbps of network bandwidth available. The real constraint was how the software used the hardware already available.
“The real constraint was how the software used the hardware already available.”
By rethinking the pull pipeline to leverage the network bandwidth, storage throughput, and compute these instances already had, we got those multi-minute pulls down to seconds. The improvements are available by default on EKS Auto Mode today, and we contributed the core changes upstream to containerd and the SOCI snapshotter. This is the story of how we dove into the internals of the image pull path, identified where time was being lost, and rebuilt those stages. It starts where we started: understanding what a large container image looks like and how it gets onto a node.
A container image is not one file. It is a stack of layers plus a small JSON manifest listing them. Each layer is a tar archive of part of the filesystem (the base OS in one, CUDA libraries in another, your code in a third), gzip-compressed. For each layer, the manifest records a digest, a SHA-256 hash of the compressed bytes, so the node can prove it received exactly what was published. When the container runs, these layers are stacked and mounted together into a single unified filesystem.
Here is what real ML images look like when measured directly from their registry manifests:
| Image | Compressed Size | Layers | Largest Layer |
| AWS DJL LMI 21.0 Inference (cu129) | 16.5 GB | 29 | 9.5 GB |
| AWS PyTorch Training NeuronX 2.7.0 | 12.8 GB | 21 | 3.9 GB |
| AWS SageMaker Distribution 4.2.1 GPU | 10.5 GB | 28 | 9.4 GB |
Notice that layers within an image are not roughly equal in size. A single layer can account for more than half the total image, with individual layers often reaching 9 GB or larger, while the remaining layers are comparatively small. This size disparity has direct consequences for how long a pull takes, as the next sections explain.
Getting each of these layers onto a node involves six stages, and traditionally containerd, the industry-standard container runtime, performed most of these operations sequentially.
For each layer, containerd performs six operations in sequence. The first three are the download phase: fetch the compressed bytes from the registry over a single HTTP connection, verify the bytes by computing their SHA-256 and comparing it against the manifest digest, and write the compressed blob to local disk.
The next three are the unpack phase: decompress the gzip archive, verify the decompressed content by computing a second SHA-256, and extract the files into the local snapshot directory managed by a containerd snapshotter, the component that stores and serves the on-disk representation of each layer. By default, containerd downloads up to three layers in parallel, but each layer uses a single connection, and unpacking remains strictly sequential across layers.
Two details are worth noting. Both SHA-256 checks mentioned above are mandated by the Open Container Initiative (OCI) specification, and on a multi-gigabyte layer, each of these hashes requires real work. The first is computed over the compressed bytes and proves the download was not corrupted or tampered with.
The second is computed over the decompressed content. It gives the runtime a stable fingerprint of the layer’s actual filesystem data, which is what allows it to recognize shared layers across images and avoid redundant unpacking. Decompression is another deceptively expensive operation on large layers: gzip often triples a layer’s size on expansion, and because each block depends on the previous one, it runs on a single core. At the same time, the rest of the instance sits idle.
Downloading layers can be parallelized, but the unpack sequence runs layer by layer. Layer two cannot begin until layer one finishes all six stages. The result is that at any given moment during a pull, the node is bottlenecked on only one resource: network bandwidth during download, CPU during decompression and hash verification, or disk throughput during extraction. The other resources sit idle, waiting their turn in the pipeline.
“Because layers are not equally sized, the single largest layer becomes the long pole in the pipeline.”
Because layers are not equally sized, the single largest layer becomes the long pole in the pipeline. A 10 GB layer that takes a minute to decompress on one core holds up the entire image, even if the other layers finish in seconds. Total pull time is effectively bounded by that one dominant layer moving through all six stages.
There are well-known approaches to reducing image pull times. Each works around the bottleneck differently, whether by altering images, caching them, or relying on assistance from other parts of the stack.
Image size reduction: The most direct approach is to make images smaller through multi-stage builds, distroless base images, and stripping unused packages. For ML workloads, this hits hard limits. The GPU software stack alone (PyTorch, cuDNN, CUDA) imposes a compressed floor of roughly 3 to 4 GB that no build optimization can remove. Beyond that, ML images are assembled across organizational boundaries: a platform team provides the OS and drivers, a frameworks team adds deep-learning libraries, and researchers contribute application code and model weights. No single team controls the final artifact; model weights frequently end up baked in because serving frameworks expect local paths, and multi-stage builds yield only single-digit percentage savings on images whose bulk is irreducible.
Image caching (pre-pulling): Cache images on nodes by snapshotting image content into a volume and mounting it at provisioning time, so subsequent launches skip the pull entirely. This works for stable images but adds a dedicated pipeline stage for each image version. For ML workloads, the challenge goes beyond frequent rebuilds: a common pattern is a single pod consuming an entire node, so nodes scale in and out with each scheduling decision and every new node faces the full cold pull. This limits pre-caching to workloads with long-lived, static node pools.
“The GPU software stack alone (PyTorch, cuDNN, CUDA) imposes a compressed floor of roughly 3 to 4 GB that no build optimization can remove.”
Registry-side optimization: Some approaches serve image content remotely rather than pulling it to the node. Alibaba’s DADI, for example, presents container images as remote block devices that the node mounts on demand without a discrete pull step. This eliminates startup latency for workloads that access only a fraction of their image. Still, the container depends on the network throughout its lifetime and requires purpose-built serving infrastructure that may not be portable across providers.
Lazy loading: Instead of pulling the whole image up front, start the container immediately and fetch file content on demand. Projects in this space include eStargz and Nydus (which require converting the image to a new format) and AWS’s SOCI (Seekable OCI), which adds a seekable index alongside the unmodified image. These techniques work well when containers touch only a fraction of their data at startup. Still, ML images densely access the framework, CUDA libraries, and model weights before serving the first request, so nearly all the data ends up being fetched anyway. For these workloads, the pull pipeline itself needs to be faster.
Peer-to-peer distribution within the cluster: Tools like Dragonfly (a CNCF graduated project) and Spegel turn nodes that already have an image into seeders for nodes that need it, reducing registry egress and accelerating rolling deployments. For ML workloads, though, the first node still faces the full cold pull, and when images are rebuilt frequently, no node has the new version cached yet. P2P distribution complements rather than replaces improvements to the pull pipeline itself.
Out of the six stages in the pull pipeline, we focused on two: downloading the layer blob from the registry, and unpacking the layers on the node. These are where the most time is spent and where the serialization cost is highest.
Since SOCI was already an open-source containerd snapshotter plugin with the plumbing to intercept and customize the pull path, it served as a staging ground where we could develop and validate these changes before contributing them upstream to containerd.
containerd traditionally uses a single HTTP connection to download each layer. We identified that splitting a layer into fixed-size chunks and fetching them concurrently over separate connections using HTTP range requests was significantly faster. At the same time, the containerd community also independently added parallel chunked download support in containerd 2.1, and we built on the same principle in the SOCI snapshotter.
However, there is one significant difference that allows us to keep the runtime’s memory footprint constant regardless of image size. The difference is where chunks live between arrival and final assembly. Our implementation writes each chunk directly to the local disk the moment it arrives, while containerd holds it in memory as the layer is assembled. This means the runtime’s memory stays flat whether you are pulling a 1 GB layer or a 15 GB layer, which matters on GPU nodes where system memory is shared with model weights and CUDA contexts.
Once download completes in seconds rather than minutes, the compressed layer hash that previously hid behind it becomes visible. Because the layer is now a complete file on disk rather than a stream consumed once, integrity verification and unpacking can proceed at the same time.
After downloading, containerd decompresses and extracts each layer one at a time. This sequencing exists because in some filesystem backends, a later layer can overwrite files from an earlier one, so the order matters. The overlay snapshotter, which is the default on EKS and most Kubernetes clusters, sidesteps this constraint by keeping each layer in its own separate directory. The kernel mounts them together into one unified view only when the container starts. Because each layer extracts to an independent directory, unpacking one layer does not need to depend on another to finish.
With that established, we built an unpack path that decompresses and extracts all layers concurrently. The snapshotter detects whether the backend actually requires ordering and falls back to sequential if it does. For images with multiple large layers, total unpack time goes from the sum of all layers to roughly the time of the single largest one. We contributed this parallel unpack capability upstream to containerd v2.2, so the improvement is available to the broader community.
With chunked parallel download, a large layer that previously took over a minute on a single connection finishes in single-digit seconds. When you then unpack all layers concurrently rather than sequentially, the full pipeline for a large ML image compresses from several minutes to well under a minute on instances with fast local NVMe storage. The machine spends its available compute and bandwidth actively pulling rather than waiting on serial stages. On larger images and instances with faster storage, the gains are more pronounced because the gap between available hardware capacity and what a single connection can use is wider.
The two changes above address download and layer sequencing, but decompression of a single large layer remains serial. On a 64 vCPU machine, decompressing a layer that expands to 18 GB means a lot of compute sitting idle. Two opportunities stand out:
Parallel decompression within a single layer. Libraries like rapidgzip can locate block boundaries and inflate blocks across cores in parallel.
Parallelizable integrity verification. Today, the layer hash requires a single sequential read over the entire compressed blob after all bytes have landed. A tree-structured hash like BLAKE3 would allow computing the layer digest from independently hashed chunks so that verification could run in parallel with download rather than as a separate pass afterward.
Addressing these opportunities will squeeze out the last remaining serial stages in the pull path, bringing total pull time closer to what the raw hardware is capable of delivering.
This image pull optimization is enabled by default on G/P/Trn instances with EKS Auto Mode. You can also benefit from this on other node types through one of two mechanisms:
EKS AL2023 and Bottlerocket AMIs that ship with containerd 2.2 do not enable this feature by default, but you can set the containerd config explicitly as shown below:
AL2023: add the containerd config through the nodeadm NodeConfig in user data:
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
containerd:
config: |
[plugins.'io.containerd.transfer.v1.local']
max_concurrent_downloads = 20
concurrent_layer_fetch_buffer = 16777216
max_concurrent_unpacks = 5
Bottlerocket (K8s 1.36): Bottlerocket generates the same containerd config from its settings API, so set the equivalent keys in user data:
[settings.container-runtime] max-concurrent-downloads = 20 concurrent-download-chunk-size = 16777216 max-concurrent-unpacks = 5
SOCI snapshotter is bundled in the optimized EKS AMIs (AL2023 and Bottlerocket).
Bottlerocket: enable SOCI through EC2 user data:
[settings.container-runtime] snapshotter = "soci" [settings.container-runtime-plugins.soci-snapshotter.parallel-pull-unpack] max-concurrent-downloads = 20 concurrent-download-chunk-size = "16mb" max-concurrent-unpacks-per-image = 5
Tune chunk size and concurrency under [settings.container-runtime-plugins.soci-snapshotter.parallel-pull-unpack]; the right values depend on your instance type and images.
AL2023: enable SOCI through the nodeadm FastImagePull feature gate, which switches image pulls to SOCI’s parallel-pull-unpack mode, and you can override the SOCI tuning parameters through user-data:
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
featureGates:
FastImagePull: true
The image pull problem was never really about the registry, and it was never about needing faster hardware. The network, the storage, and the CPU were always there. When containerd’s pull pipeline took shape, images were measured in hundreds of megabytes, and the sequential approach served that world well.
“The image pull problem was never really about the registry, and it was never about needing faster hardware.”
As AI and ML workloads pushed images past 20 GB, the gap between available hardware throughput and what the pull path was utilizing became impossible to ignore. For workloads running on the overlay snapshotter with high-bandwidth instances and fast local storage, parallelizing download and unpack closes most of that gap today. Other snapshotters and storage backends may have different constraints, and opportunities like parallel decompression and parallelizable integrity verification remain open.
We have been contributing these changes upstream because this is where they belong: in the runtime itself, available to everyone by default rather than locked behind additional software. Some of that work has already landed in containerd 2.2, and more is in progress.
If you are working on container runtimes, image formats, or compression tooling and any of the open problems described here interest you, we would welcome collaboration. The faster we can collectively close the remaining serial stages, the sooner multi-gigabyte images stop being a deployment bottleneck for the entire ecosystem.
The post Pulling multi-gigabyte container images in seconds on Amazon EKS appeared first on The New Stack.
Sophia Space and Caltech want to fold the bulky parts of a space-based data center—solar cells and radiators—into all-in-one tiles with chips.
Every time you ask ChatGPT a question, computer chips in a massive data center whirl into action. In the blink of an eye, they ping back answers. Behind the scenes, though, AI data centers consume enormous amounts of electricity, heat, and water.
The AI boom is impacting communities. After welcoming 37 data centers, residents in Virginia’s Henrico County were hit with skyrocketing electrical bills. Schools and government buildings were asked to turn off lights, shut down computers, and avoid using space heaters to ease strain on the power grid and keep costs down.
Henrico isn’t alone. A growing backlash is prompting many states to consider legislation curbing new facilities. “No data center” signs have sprouted on lawns and alongside roads. Yet as AI demand continues to surge, so does the need for more computing power.
This has top AI companies looking skyward. Instead of routing requests to terrestrial data centers, future queries could be handled by thousands of solar-powered satellites orbiting above. The results would then be beamed back, with users none the wiser.
But there’s a major hurdle: heat.
Space’s frigid vacuum may seem like the perfect place to cool chips, but it’s not that simple. Lacking air and water to carry heat away, orbital data centers would have to use thermal radiation. Here, heat is converted into infrared energy and radiated into space, often requiring bulky hardware that adds weight, cost, and complexity.
With these challenges in mind, California Institute of Technology and Sophia Space, a California startup developing orbital computing, recently unveiled a patent for a chip cooling system designed to radiate heat into deep space. Called Sophia TILE, thousands of these chips could be linked to form large orbital data centers or organized into smaller, distributed clusters.
Powered by abundant sunlight, the chips could operate continuously without eating up Earth’s resources. The team hopes to test their vision by 2030.
“This patent reflects a different way of thinking about computer infrastructure in space,” said Leon Alkalai, founder and chief technology officer at Sophia Space, in a press release. “Instead of beaming down energy to Earth from orbit, we decided to consider putting computing in space and beam[ing] down data.”
The project joins a growing international push towards orbital computing. ADA Space, working with Zhejiang Lab, has already launched satellites for its Three-Body Computing Constellation and plans to expand into a much larger network. Meanwhile, US companies including SpaceX, Starcloud, and Blue Origin are seeking regulatory approval for constellations that could eventually grow to include up to a million AI-capable satellites.
Without doubt, the race is on.
Orbital data centers would consist of high-performance computer chips housed in protective enclosures designed to withstand the harsh conditions of space. In orbit, they would collect uninterrupted solar power. In contrast, solar panels on Earth require batteries to store energy for use after sunset.
Solar power in space is hardly new. The International Space Station, satellites, and other spacecraft have long relied on solar panels. More recently, engineers have developed flexible, lightweight designs such as NASA’s Roll-Out Solar Arrays, which launch tightly rolled and unfurl in orbit.
AI, however, demands far more power. One long-standing idea for harvesting continuous solar power suggests we collect solar energy in space and beam it down to Earth. But that approach doesn’t completely appease the growing ire against data centers. They’d still consume energy on the ground and take up land and other resources. A newer idea flips the question. Rather than delivering energy to computers, why not bring computers nearer to the energy source?
The argument in favor of sending data centers skyward is growing stronger. A recent Gallup poll found roughly 70 percent of Americans oppose data centers in their backyard, while experts agree that meeting AI’s future energy demands on Earth alone will become increasingly unsustainable.
But while power is abundant in space, heat is the main problem. Without air or water to carry heat away, computers in space must rely on thermal radiation. That means adding large, heavy radiators to an already bulky, solar-powered setup. In space, weight is money, and scaling orbital data centers will take a lot of it (to put it mildly).
TILE tackles the cooling problem with a specialized material that converts heat into infrared radiation. The concept may seem alien, but everything warmer than absolute zero cools this way. Our bodies, stovetops, and car engines all shed heat as invisible infrared light.
Each TILE combines solar cells, thermal insulation, processors, memory, and optical communication hardware into a single module. Beneath the electronics sits a custom heat-spreading layer that prevents dangerous hot spots. Like placing a scorching pan onto a baking sheet, it distributes heat over a much larger surface before channeling it to the radiator.
The modules are designed to work together. Thousands of TILES could link into a giant computing mosaic, each acting as a mini computer connected to its neighbors. Like a modern power grid, the distributed architecture improves reliability—if one TILE fails, others can jump in—while simplifying power distribution and thermal management.
The modular design also solves a practical challenge: Rockets don’t have much cargo space. Similar to NASA’s Roll-Out Solar Arrays, a TILE-based data center could launch in a compact configuration before unfolding into a large, flat computing platform in orbit.
Looking further ahead, the team envisions launching multiple interconnected arrays in succession, like strings of pearls. Each could function as an independent data center that exchanges data with others, effectively extending cloud computing into orbit.
Sophia Space is targeting a demonstration mission in late 2027. By 2030, the team estimates an array of 2,000 TILEs could deliver up to a megawatt of dedicated computing power. To put that in perspective, a single ground-based data center can deliver hundreds of megawatts of computing power, and future data centers will stretch that number into the thousands.
There are challenges beyond the purely technical. Earth orbit is crowded with active spacecraft and debris, raising the risk of collisions. SpaceX’s Starlink satellites, for example, perform frequent collision-avoidance maneuvers after a close call in 2019. The breakup of a Chinese Long March rocket in 2024 threatened an estimated 1,000 satellites. Large constellations of data centers—SpaceX has plans for up to a million in low Earth orbit—would add even more traffic.
Beyond collisions, astronomers are worried that expanding satellite numbers could hinder our ability to study the universe by interfering with telescope observations and radio astronomy.
For now, orbital data centers are unlikely to replace their terrestrial counterparts. Instead, they’re more likely to complement them, processing data collected by spacecraft and beaming only the results back to Earth. Although the field is ridden with hype and controversy, there’s also promise and momentum is clearly building.
“It’s just kind of exploding,” Sergio Pellegrino, a Caltech engineer who collaborates with Sophia Space, told The New York Times. “We need to become more comfortable with space doing things for us.”
The post Heat Is an Orbital Data Center’s Greatest Foe. These Tiles Dump It at the Source. appeared first on SingularityHub.
An implant, smaller than a grain of rice, pairs with camera-mounted glasses to communicate visual information to the retina.
Age-related vision loss affects millions of people, and so far, there has been no way to reverse the damage. A newly approved retinal implant could change that by allowing some people with severe vision loss to regain functional sight.
More than five million people worldwide suffer from geographic atrophy, the late stage of the progressive eye condition dry age-related macular degeneration. The disease destroys the photoreceptors at the center of the retina, known as the macula, which is responsible for the sharp central vision required to read or recognize faces.
In the US, treatment options are limited to two drugs that can be injected into the eye to slow the disease’s progression. But neither can undo the damage. That could be about to change. California neurotech startup Science Corporation recently won European approval for a retinal implant designed to treat the condition.
“For decades, losing central vision to this disease meant losing the ability to read, recognize faces, and ultimately losing independence. There was no viable treatment. Now there is,” Max Hodak, Science’s CEO and co-founder, said in a press release.
The company’s PRIMA system combines an implant smaller than a grain of rice installed underneath the patient’s macula with a pair of camera-mounted glasses that translate incoming visual information into near-infrared light that is then beamed to the retina. The eye can’t detect this wavelength, so the device doesn’t interfere with any natural sight that remains.
The chip, which works on similar principles to a solar panel, converts the incoming light into electrical pulses that stimulate retinal neurons called bipolar cells. These are downstream of the rod and cone photoreceptor cells damaged by macular degeneration and normally spared by the disease.
In a clinical trial involving 38 patients across five countries, which was published in the New England Journal of Medicine last year, the company and its collaborators showed participants gained an average of 25.5 letters—more than five lines—on a standard eye chart after having the device fitted.
And now the device has received a CE mark from the European Union making it possible to sell in 30 European countries. The company says the first commercial implants are expected to be fitted in Germany within weeks, with Italy, the Netherlands, and the UK to follow. In the US, PRIMA holds Breakthrough and Humanitarian Use Device designations from the FDA, but the company is confident it will gain full approval in the near future.
The device is a long way from restoring normal vision. The images it produces are black and white and the field of vision is extremely narrow. Hodak described the experience to the Financial Times as “kind of like looking through a straw in the center of their vision,” though he added that they see a pathway to color vision and higher acuity.
While the implantation procedure is fairly simple, it takes months of training to unlock the device’s full potential. Nonetheless, Hodak told STAT that the company expects to install 20 to 40 devices this year and 200 globally by the end of next if they get US approval in early 2027.
The approval is welcome news for the wider neurotech industry, which has absorbed billions of dollars of investment in recent years with little to show in terms of return.
“Science is showing that brain-computer interface companies have a path to real revenue now,” Jacob Robinson, founder of startup Motif Neuroscience, told STAT. “These companies aren’t all just making a bet on a market that is 10 to 15 years away.”
Hodak told the Financial Times hehopes sales from PRIMA will bankroll Science’s more ambitious work on “biohybrid” interfaces, which use genetically engineered living neurons to connect to the brain rather than metallic wires. “This is the financial backbone,” he said. “This is the thing that pays for the rest.”
Other companies are hot on Science’s heels. Neuralink, which Hodak co-founded with Elon Musk before leaving to start Science, is also working on a vision implant called Blindsight, which is due to enter human trials this year.
While the field remains a long way from the sci-fi vision of seamless two-way communication between humans and machines, this approval is growing evidence the neurotech industry is starting to move out of the lab and into the real world.
The post Europe Approves Bionic Eye to Restore Vision Lost to Blindness appeared first on SingularityHub.
How one open-source ecosystem made state-of-the-art AI accessible
The post The Python Ecosystem That Changed AI Development appeared first on Towards Data Science.
Last week’s unprecedented security event in which two OpenAI security hacking models trespassed into the network of fellow AI company Hugging Face was enabled by exploiting one or more zero-day vulnerabilities in Artifactory, JFrog, the product’s developer, said Monday.
In an incident mimicking a dystopian sci-fi novel, two OpenAI models broke out of the restricted environment meant to keep them from accessing the Internet during an internal test, the AI company revealed last week. The models went on to breach Hugging Face’s network and steal confidential information and credentials. OpenAI said its agent achieved the feat by exploiting a previously unknown vulnerability. The company called the event “unprecedented,” and outsiders largely agreed.
OpenAI said the models exploited multiple attack vectors, including stolen credentials and zero-days, to gain remote code execution capabilities, but until now, the vulnerable software was unknown. JFrog’s Monday disclosure said the product was a self-managed instance Artifactory, a repository management system that secures and streamlines customers’ software development operations. JFrog says Artifactory is used by more than 7,500 developer Teams, 80 percent of which work for Fortune 100 companies.
© Aurich Lawson
Cambridge University researchers say launch costs fell from $87,000 to $3,868 per kilogram between 1960 and 2025—or roughly 96%—and could hit $273 by 2040.
Rapidly falling launch costs are making space more accessible than ever. But new research suggests the economics are improving even faster than most people realize, potentially opening the door to entirely new industries beyond Earth.
For most of the space age, the cost of getting material into space was so vast that only the most well-heeled governments and corporations could participate. In 1960, getting a kilogram of payload into orbit would have cost you more than $87,000 (in 2024 US dollars).
But according to researchers at the University of Cambridge, that figure had collapsed 96 percent to $3,868 by 2025. The team’s modeling suggests this trend will continue apace for at least the next few decades, with prices forecast to hit just $1,569 by 2030 and as little as $273 by 2040.
The rapid decline in prices is thanks to a well-established economic principle known as Wright’s Law, which holds that technologies get predictably cheaper as cumulative production grows. The Cambridge team says the trends seen in launch costs could soon make a host of possibilities previously confined to science fiction commercially viable, including orbital solar power, asteroid mining, and space-based manufacturing.
“Space is no longer a science-fiction fantasy or a purely scientific pursuit, it is becoming a marketplace,” Alessio Terzi, who led the study, said in a press release. “Rapidly falling launch costs could open the way to space colonization and commercial activity far beyond low Earth orbit.”
To conduct their study, published in PNAS Nexus,the researchers assembled a massive dataset of rocket launches covering over 4,400 flights by more than 330 different rocket designs from 1960 to 2025. For each launch, they estimated the “unit flyaway cost,” or the total cost to manufacture, maintain, and launch the vehicles, excluding research and development investments.
They then checked how this data stacked up against Wright’s Law, which predicts that every time production volumes double the cost should fall by a fixed percentage. This is known as a technology’s “learning curve” as the reduction in costs is attributed to an industry getting better at producing the technology with experience.
The researchers found space launches obey the law almost perfectly, with every doubling of payload sent to orbit shaving 21.2 percent off the average cost per kilogram. More importantly, this represents a particularly steep learning curve compared to previous technologies.
Solar panels are often held up as the poster boy for learning curves, with prices falling 99.8 percent between 1975 and 2023. But while solar power’s total price reduction is higher than that achieved by launch vehicles, the technology got there by scaling deployment far more. When accounting for total production, solar’s learning curve lags launch costs at 20.2 percent.
The researchers also compared launch costs to another revolution in transport. Steamships transformed our ability to ship goods like wheat and cotton around the world in the 19th century. They found that steamship costs only fell 15.5 percent with each doubling of cargo.
“The cost of space launch technology is now falling faster than during one of history’s greatest transport revolutions,” said Terzi. “Steamships cut costs through explosive growth in global trade. Space technology, by contrast, has achieved even steeper declines at a far smaller scale. This suggests there is plenty of scope for further cost reductions and the industry may now be on the cusp of a comparable economic boom.”
There are, of course, caveats. The researchers note that the industry’s progress is inextricably tied to the fate of a single company. SpaceX already accounts for roughly 80 percent of payload reaching orbit. If the company successfully scales up its reusable, heavy-lift Starship vehicle it could massively reduce costs.
But a company with a stranglehold on the global launch market may be tempted to take advantage of its monopolistic position. This may also push foreign governments and companies away from relying on SpaceX even if it’s the cheapest option.
There’s also the danger that as costs fall and launching material into space becomes more accessible, low Earth orbit could quickly become clogged with debris that makes it increasingly difficult to reach orbit safely.
If these challenges can be sidestepped, the implications of such rapidly falling costs could be profound. The researchers suggest that everything from zero-gravity research and orbital tourism to factories churning out fiber-optic cables and 3D-bioprinted organs could become financially viable.
The post Spaceflight Nears Its Steamship Era appeared first on SingularityHub.
When you run Kubernetes at the scale we do on Amazon EKS, nodes break constantly. GPUs fall off the PCIe bus. Container runtimes wedge. Network interfaces disappear. Across tens of thousands of clusters, “rare” hardware failures happen multiple times a day, somewhere in the fleet.
For years, everyone responded the same way: an operator wakes up, reads a dashboard, SSHes into the node, cordons it, drains it, terminates the instance, and waits for a replacement. Every step is human-paced. Every step is toil. And if the failure lands at 3 a.m. on a weekend, the workload sits degraded for hours before anyone looks.
We built the EKS Node Monitoring Agent to help close that gap, which we open-sourced in April earlier this year. It detects node failures and writes Kubernetes NodeConditions that signal the problem to Karpenter, which then automatically replaces the node if required. The agent is one piece of a larger system. To understand where it fits, you need to understand what manages the nodes it monitors.
“Across tens of thousands of clusters, ‘rare’ hardware failures happen multiple times a day, somewhere in the fleet.”
AWS launched Amazon EKS Auto Mode that fully automates Kubernetes cluster infrastructure: compute provisioning, scaling, networking, storage, OS patching, and security hardening, so teams focus on applications, not cluster operations. It dynamically selects optimal EC2 instances (including GPU instances like P5, P6, and G6 families), scales based on workload demand, consolidates underutilized nodes, and keeps the operating system patched. EKS Auto Mode ships with automatic node repair as the default behavior: detection, severity classification, and Karpenter-driven node replacement all run out of the box with no add-on to install, no controller to configure, and no repair policy to write.
This is the story of how we built automatic node repair, the design decisions that shaped the system, and the hard lessons that came from operating it at GPU scale.
After operating this across thousands of clusters, the lessons compress into a short list. These are not unique to our system. The same patterns show up in NPD, NVSentinel, AKS Periscope, GKE’s auto-repair, and anyone building a custom node controller. They are folklore that should be a checklist. The open-source repo reflects each of these lessons in code, from the reason-code stability guarantees in the API to the jitter implementation that solved GPU workload interference.
“Your reason codes are an API contract. Additions are features. Renames are breaking changes.”
Every node health agent in the Kubernetes ecosystem performs the same translation. Node Problem Detector (NPD), NVSentinel, GKE’s auto-repair, AKS’s Linux Extension, and the EKS Node Monitoring Agent all take noisy, low-level signals from a machine and translate them into a set of Kubernetes primitives: NodeCondition, Event, sometimes a CRD. The translation looks simple. It isn’t.
The output is a NodeCondition, which is just a type, a status (True/False/Unknown), a reason code, and a message. Four fields. But that surface area hides decisions that determine whether a repair action helps or hurts.
Reason codes are a public API. We learned this the hard way. In version 1.6.2, we changed NvidiaDeviceCountMismatch from Warning severity to Fatal. The technical reasoning was sound: once a GPU drops off the PCIe bus, it doesn’t come back without a node reboot or replacement. Leaving it as Warning meant GPU workloads kept getting scheduled onto degraded nodes, wasting expensive accelerator capacity. So we shipped the fix. Downstream automation broke. Customers had repair configurations keyed on the old severity. Dashboards that filtered on Warning stopped showing the fault. Automation that only acted on Fatal suddenly started draining nodes it hadn’t touched before. Dashboards that filtered on Warning stopped showing the fault. Automation that only acted on Fatal suddenly started draining nodes it hadn’t touched before. From that point, we treat every reason code addition as feature work and every rename or severity change as a breaking change.
“Absent” must not equal “healthy.” When we shipped per-monitor configurability in v1.6.0, we had to make a choice. A disabled monitor needs to produce some output (or no output). The three options: write True (your auto-repair now thinks the node is healthy because you’re not watching), write Unknown (ambiguous, might trigger repair depending on downstream logic), or omit the condition entirely. Only the third is safe.
This seems obvious in retrospect, but consider that NPD achieves the same result through a completely different mechanism: compile-time disable via build tags. NVSentinel delegates it to operator-authored CEL rules. The upstream Kubernetes spec defines what Unknown means, but if your repair automation treats Unknown as actionable, you will lose nodes for no reason. We chose to emit nothing when a monitor is off, and documented it as a hard contract.
Detection latency is bounded by the source, not by the agent. We originally told customers, “We detect kernel panics within 30 seconds.” This was wrong. Our agent’s detection time was under 30 seconds. But the kernel panic shows up in journald, and journald’s flush cadence is the actual bottleneck. If journald takes 45 seconds to write the line, our 30-second claim was incomplete.
For GPU faults, the picture is more nuanced because we use two detection paths with very different latency characteristics. The critical faults (double-bit ECC errors, XID errors, NVLink failures, page retirements, thermal and power violations) go through DCGM’s push-based policy violation channel. DCGM notifies our agent the moment it detects the violation; there is no polling interval. Detection of these faults is near-instant (sub-second in practice). A separate path uses a 5-minute field-value window to monitor NVSwitch fabric health, Fabric Manager status, and clock-throttle reasons. That window is the floor for those specific detections, but it does not apply to the critical GPU faults that trigger automatic repair. The lesson: the customer-facing SLO must include source-of-truth latency, and different signal paths within the same subsystem can have radically different floors.
The kubelet already reports DiskPressure, MemoryPressure, and PIDPressure. NMA complements those with five additional conditions covering domains the kubelet does not monitor: kernel health, container runtime, networking, storage, and accelerated hardware. Every detection carries one of two severities, and severity is the switch that decides whether the repair cycle fires.
Condition severity is a terminal fault. It flips the matching condition to False and makes the node eligible for automatic repair. GPU device-count mismatches, critical XID and double-bit ECC errors, NVLink and NVSwitch fabric failures, a missing Fabric Manager, and Neuron DMA and HBM uncorrectable errors. On the networking and runtime side: VPC CNI process down, IPAMD unable to reach the API server, fork failures due to PID exhaustion, and pods wedged, terminating behind a broken container runtime. These are faults that won’t recover on their own. On GPU nodes, a single degraded accelerator can corrupt training checkpoints or waste thousands of dollars in compute per hour.
Event severity is informational. It posts a Kubernetes event, the NodeCondition remains True, and operators get visibility without disruption. Bandwidth ceilings, connection-tracking limits, Amazon Elastic Block Store (Amazon EBS) IOPS throttling, I/O delays, filesystem fragmentation, clock drift, liveness and readiness probe failures, kube-proxy anomalies, GPU thermal and power warnings, PCIe link degradation, and page-retirement thresholds. These signal trouble building before it turns terminal.
Getting severity wrong in either direction is expensive. Too aggressive, and you terminate healthy nodes and needlessly displace workloads. Too conservative, and degraded nodes serve traffic for hours while a GPU with a failing memory bank corrupts training checkpoints. The classification principle: if the failure is deterministic and infrastructure-owned (hardware broke, firmware crashed, a physical link went down), it triggers replacement. If the signal could be application-induced or transient, it stays informational. You never want to terminate a healthy node because a misbehaving pod saturated a resource.
“Getting severity wrong in either direction is expensive. Too aggressive, and you terminate healthy nodes. Too conservative, and degraded nodes serve traffic for hours.”
DiskPressure, MemoryPressure, and PIDPressure are the canonical examples. Every major auto-repair system (GKE, AKS, NPD) has independently converged on the same answer: don’t touch them. These are workload-driven conditions, not node-level faults. Replacing the node just moves the misbehaving workload to a fresh machine, where it will eat memory again. The correct response is kubelet-level pod eviction, not node replacement. If you’re building a node-health system, draw this boundary early and document it publicly.
The hardest lesson came from a customer running large-scale distributed GPU training. Their workload used NCCL collectives across hundreds of GPU nodes, where every node in a communication group must complete its step before any can proceed. One slow node makes every node wait.
They found that NMA itself was causing periodic slowdowns. The agent’s monitors all ran on independent goroutines, and when their polling intervals aligned, dozens of goroutines would wake simultaneously and burst onto many CPU cores at once. On a general-purpose web service, this would be invisible. In a distributed training job, microseconds of jitter on one node can cascade across the entire GPU cluster, causing measurable throughput loss.
The customer disabled NMA entirely and saw an immediate improvement. That was the worst possible outcome for us: a health agent that interferes with the workload it exists to protect is worse than no agent at all.
The fix was straightforward once we understood the problem. We added a startup jitter to every monitor’s polling interval. Each goroutine delays its first tick by a random offset (up to 20% of its base interval), staggering the wake times so they don’t align on boot. We cached system calls that hit /proc on every poll. We consolidated handlers that shared an interval into a single sequential work queue, reducing the goroutine count for monitors that didn’t need their own thread. The result was an agent whose CPU profile is flat and predictable rather than bursty.
The lesson generalized: if your health agent runs on the same host as the workload, its resource consumption pattern matters as much as its resource consumption total. A process that uses 0.5% CPU spread evenly is invisible. A process that uses 0.5% CPU in concentrated bursts can disrupt latency-sensitive distributed GPU workloads in ways that show up as lost training time rather than a CPU alarm.
This is why per-monitor configurability matters. Not every monitor is relevant to every workload. A dedicated GPU training cluster with one pod per node and no pod churn doesn’t need IPAMD monitoring or environment scanning. We shipped the ability to disable individual monitors so customers can keep the health coverage they need without paying the overhead of coverage they don’t.
Karpenter is the compute controller that provisions and scales EKS Auto Mode nodes. It already owns the lifecycle of every node it launched, and consuming our NodeConditions for repair is a natural extension of that ownership. There’s no separate repair backend, no sidecar controller, no webhook chain. The same system that created the node is the one that replaces it.
Karpenter’s AWS cloud provider declares repair policies: each one pairs a condition type with a status that means “replace this node.” The policies include toleration windows that prevent reacting to transient blips:
The flow:
The replacement node comes up with a fresh agent monitoring it from boot. No operator in the path. In our testing, the full cycle from fault injection to replacement node running workloads took under 12 minutes. Detection landed in under a second (critical GPU faults use DCGM’s push-based policy channel, not polling). Then 10 minutes of toleration, and roughly 90 seconds for the replacement to launch and register.
Auto-repair handles the common case: broken node gets replaced, workload keeps running. But “why did that node fail?” is a different question, and one we initially tried to answer inside the detection path. That was a mistake.
Detection answers “is this node healthy?” It runs continuously with minimal overhead, and it needs to be fast: a condition flip that takes 5 minutes to produce is 5 minutes of degraded workload. Diagnosis answers “what went wrong?” It needs to collect detailed artifacts: full journald output, containerd state, network configuration, dmesg, GPU driver logs. In our testing, that collection completes in about 7 seconds and produces a compressed log bundle. Baking it into the detection hot path would have slowed down the thing customers care most about: how fast the system reacts.
We built them as separate concerns sharing an agent binary. The NodeDiagnostic CRD lets you request a full log bundle from any node through kubectl, without SSH. On EKS Auto Mode, where nodes are Amazon Elastic Compute Cloud (Amazon EC2) managed instances with no shell access by design, this is the only way to investigate after a GPU failure or any other node-level fault.
The experience is one command:
kubectl ekslogs <node-name>
The plugin creates a NodeDiagnostic resource. The agent on the target node detects it via a watch, collects system state into a compressed tarball, and stores it temporarily (available for 10 minutes). The plugin then downloads it through the kubelet’s Node Log Query API (KEP-2258, GA in Kubernetes 1.36). No SSH, no security groups, no key pairs.
This separation means detection doesn’t slow down to collect evidence, diagnosis doesn’t need to be always-on (saving node resources), and you can diagnose a node that auto-repair has already flagged but hasn’t yet terminated. The 10-minute window for accelerated hardware faults gives you exactly enough time to grab the logs before the node is gone. If you’re interested in further improvements, engage with us on EKS public roadmap.
On EKS Auto Mode, all of this is on by default. Auto Mode fully manages your cluster infrastructure (compute, networking, storage, patching, and security hardening) so you focus on applications, not cluster operations. The agent runs as a systemd service in the node image (not a DaemonSet you manage), Karpenter consumes its conditions as part of the compute lifecycle it already owns, and kubectl ekslogs gives you diagnostic access without SSH. There is nothing to install, configure, or operate. For GPU workloads, this means your expensive accelerator nodes are automatically monitored, classified, and replaced without any operator intervention.
On managed node groups or self-managed Karpenter, you can assemble the same loop: install the Node Monitoring Agent as an EKS add-on and opt each node group into auto-repair. The architecture is the same, just not pre-assembled.
The EKS Node Monitoring Agent is Apache 2.0 open source at github.com/aws/eks-node-monitoring-agent.
The failure modes we hit when running it at scale, and the fixes that come out of them, flow back to anyone using it. If you’re building a node-health system or running ours and hitting an edge case, come build with us!
The post Self-healing GPU nodes in Kubernetes: What we learned building the EKS node monitoring agent appeared first on The New Stack.
