NVIDIA CUDA Container Tags Can Disappear: StableBuild Now Supports NVIDIA NGC and GitHub Container Registry
You pin an NVIDIA CUDA image all the way down to the patch release, image flavor, and operating system:
nvidia/cuda:11.2.1-base-ubuntu20.04
It looks safe.
It is not the latest. It is not just CUDA 11. It says exactly which CUDA release you want, which image flavor you want, and which Ubuntu release should sit underneath it.
Then one day you run the same pull again:
$ docker pull nvidia/cuda:11.2.1-base-ubuntu20.04
Error response from daemon: manifest for nvidia/cuda:11.2.1-base-ubuntu20.04 not found:
manifest unknown: manifest unknown
Nothing is wrong with the Dockerfile.
The image tag is gone.
That example is not hypothetical. The tag was used in older public GPU workflows and documentation, and a pull attempt in August 2026 returned the error above. NVIDIA has an official support policy explaining why this can happen.
For teams maintaining AI, machine-learning, scientific-computing, or other GPU-heavy software, that turns a normal rebuild into an emergency dependency migration.
Your code did not change.
NVIDIA’s container lifecycle did.
NVIDIA CUDA container tags are designed to have a lifetime
NVIDIA’s CUDA container documentation says it directly: “CUDA image container tags have a lifetime.”
Under NVIDIA’s support policy, CUDA image sets can reach end-of-life when the driver they shipped with reaches EOL, and only the latest point release for a toolkit version remains supported. Under NVIDIA's support policy, CUDA image tags are deleted six months after the last supported driver reaches end-of-life, or after a newer update release ships for the same CUDA version.
The policy also says NVIDIA supports two major CUDA versions at a time. Its own example explains that CUDA 11 is marked EOL after CUDA 13 is released. That doesn't give every CUDA 11 tag a universal public deletion date, but it makes the larger point clear: older CUDA images aren't intended to remain available forever.
From NVIDIA’s side, there are reasonable reasons for this. Old images can contain unpatched vulnerabilities, and maintaining years of large, rarely used container layers has a real infrastructure cost.
From your build’s side, none of that changes the immediate result.
If a build still depends on a removed image, the build fails.
StableBuild wrote about the same problem previously in “Getting Repository does not exist for NVIDIA CUDA containers?”. The new release goes a step further: StableBuild’s Docker mirror now supports NVIDIA NGC and GitHub Container Registry in addition to Docker Hub.
A very specific tag can still disappear.
The `11.2.1-base-ubuntu20.04` example is useful because it does not look sloppy.
• 11.2.1 specifies the CUDA release.
• base specifies the image flavor.
• ubuntu20.04 specifies the operating-system base.
That is much better than building on `latest`.
But specificity and permanence are different things.
A tag can identify exactly what you want while it exists. It does not force the registry owner to keep serving that artifact forever.
Some developers have responded by making their own copies. One community CUDA mirror on Docker Hub describes its reason for existing with a single sentence: “Just to make sure NVIDIA does not remove the image needed for qlora finetuning.”
That is funny because it is also a real continuity strategy people are resorting to: hoard the image before somebody upstream cleans it up.
If your disaster-recovery plan is “hopefully somebody mirrored it,” you probably want a more deliberate answer.
Quick definitions: CUDA, NGC, NVCR and the Container Toolkit
A few NVIDIA names in this area sound interchangeable until you have to debug a broken GPU build.
CUDA is NVIDIA’s parallel-computing platform for using GPUs for general-purpose computing. The NVIDIA CUDA toolkit includes the runtime, libraries, compiler, and development tools used to build GPU-accelerated applications.
An NVIDIA container packages some of that environment into a container image, so you don't have to assemble the full CUDA stack from scratch every time. NVIDIA publishes base, runtime, and development image variants for different CUDA and operating-system versions.
NVIDIA NGC is NVIDIA’s catalog and registry ecosystem for GPU and AI software. The container-registry hostname is `nvcr.io`.
The NVIDIA Container Toolkit is different. It is the host-side runtime tooling that lets Docker, containerd, and other runtimes expose NVIDIA GPUs to a container. It helps the container use the GPU; it does not preserve the container image in a registry.
That distinction matters. StableBuild is addressing the image-availability side of the problem, not replacing NVIDIA’s GPU runtime stack.
NVCR vs. GHCR: they are not really competing choices
At first glance, “nvcr.io vs. ghcr.io” sounds like an either/or comparison, but most engineering teams shouldn't think of it that way.
`nvcr.io` is the registry endpoint for NVIDIA NGC. GitHub Container Registry, available at `ghcr.io`, is GitHub’s general-purpose OCI container registry.
A single AI build can easily depend on both, plus Docker Hub, PyPI, Ubuntu repositories, and a few files fetched directly from URLs.
That is the real operational problem: one build can inherit several different upstream lifecycles.
Registry
What it is
What your build depends on
NVIDIA NGC / nvcr.io
NVIDIA’s GPU and AI container registry
NVIDIA continuing to retain and serve the image
GitHub Container Registry / ghcr.io
GitHub-hosted Docker/OCI images
The publisher retaining the image and GitHub continuing to serve it
Docker Hub
General public container registry; also hosts nvidia/cuda
Publisher lifecycle, registry availability, and rate limits
Every additional registry is another upstream lifecycle your build inherits.
A tag can move, an image can disappear, credentials can expire, or the service can simply be unavailable when CI needs it.
The goal is not to stop using public registries. They are excellent distribution systems. The goal is to stop treating them as your permanent historical archive.
Pinning a tag is not the same as preserving an image.
Three related ideas here are easy to collapse into one.
1. A tag gives you a human-readable reference.
A tag such as `11.2.1-base-ubuntu20.04` is specific and understandable. It is still a registry-managed reference, and the registry can remove it.
2. A digest gives you an exact identity.
GitHub’s own container-registry documentation recommends pulling by SHA digest when you need to guarantee that you are requesting the same image content.
That is good practice.
But a digest answers “which image?” It does not, by itself, answer “will the registry still serve that image three years from now?”
StableBuild’s Docker mirror documentation makes the same distinction: pinning to a hash protects identity, but upstream deletion can still break the build if the image itself is gone.
3. Retention gives you continued availability.
For a long-lived build, you want both identity and retention: the exact image, and a copy that is still there when you need it.
That is the gap an immutable pull-through cache is designed to close.
StableBuild now supports NVIDIA NGC and GitHub Container Registry.
StableBuild’s Docker mirror now acts as an immutable pull-through cache for Docker Hub, GitHub Container Registry, and NVIDIA NGC.
The first time StableBuild pulls an image, it retrieves the upstream image and stores it. Future pulls of that cached tag return the stored image instead of asking the public registry to recreate your historical build environment.
In practical terms:
• If NVIDIA later removes an EOL CUDA tag that StableBuild has already captured, your build can continue using the preserved image.
• If a mutable upstream tag later points somewhere else, the cached StableBuild tag does not silently follow it.
• If your build pulls public images from both NGC and GHCR, you can apply the same immutable-mirror model across both registries instead of maintaining separate ad-hoc caches.
This is not a replacement for NGC or GHCR. StableBuild sits between your build and those upstream registries and preserves the image your build actually used.
Timing matters: preserve the image before you discover the upstream copy is gone.
What changes in a Dockerfile?
The basic change is intentionally boring: instead of pulling the image directly from the upstream registry, prepend your StableBuild Docker mirror domain to the original image path.
For example, pulling directly from NVIDIA NGC may look like:
FROM nvcr.io/nvidia/cuda:12.3.1-runtime-ubuntu22.04
Through StableBuild, it becomes:
FROM your-domain.dockermirror.stablebuild.com/nvcr.io/nvidia/cuda:12.3.1-runtime-ubuntu22.04
The same pattern applies to GitHub Container Registry.
Direct from GHCR:
# Direct upstream
FROM ghcr.io/astral-sh/uv:0.5.11
# Through StableBuild
FROM your-domain.dockermirror.stablebuild.com/ghcr.io/astral-sh/uv:0.5.11
In other words, you keep the original registry path and simply place your StableBuild Docker mirror domain in front of it.
Use the exact StableBuild mirror domain assigned to your account rather than copying your-domain literally.
This isn't a new build system. Your Dockerfile still asks for the image it needs. StableBuild changes who has to remember it.
Why this matters even more for AI and ML builds
GPU environments tend to age differently from ordinary application containers.
A model-training or inference project might depend on a particular CUDA release, cuDNN version, PyTorch build, driver compatibility window, Python environment, and operating-system base. Updating one piece can force updates elsewhere.
That makes “just move to the newest CUDA image” a much bigger request than it sounds.
Sometimes you are rebuilding an old research environment. Sometimes you need to reproduce a training run. Sometimes a production model has not changed in two years, and there is no business reason to migrate its entire GPU stack during an unrelated deployment.
The model may still exist.
The environment required to build or run it may not.
This is also why developers end up making personal CUDA mirrors for QLoRA and other older workflows. They are not necessarily trying to stay on old software forever. They are trying to keep a known environment rebuildable long enough to update it deliberately.
Container retention is only one part of full reproducibility. Model weights, Python wheels, apt packages, external downloads, build tools, data, and nondeterministic training behavior can all matter too. StableBuild covers several of those dependency surfaces, and its broader approach is explained in Reproducible Builds When Dependencies Disappear.
Preserving an old CUDA image does not mean you should run it forever.
There is an important security caveat here.
NVIDIA has a legitimate reason to EOL old images: unsupported containers can contain known vulnerabilities and outdated components.
StableBuild preserving an image should not be read as a recommendation to keep vulnerable software in production indefinitely.
Preservation gives you control over the migration.
You can rebuild the historical environment, investigate a production issue, reproduce an older result, test the newer CUDA stack, and then move when your application is ready rather than when an upstream deletion forces you to.
Reproducibility and patching are not opposites. A mature workflow wants both: a known historical environment and a controlled path to a supported one.
A few questions developers are likely to ask
Does NVIDIA really delete CUDA container tags?
Yes. NVIDIA’s published CUDA Container Support Policy explicitly says EOL tags are deleted from Docker Hub and NVIDIA NGC after the support window. NVIDIA also warns that tag deletion can break standard container pulls because the underlying image layers are no longer available.
Is CUDA 11 going away?
NVIDIA’s policy says it supports two major CUDA versions at a time and gives CUDA 11 being marked EOL after CUDA 13 as its own example. Exact deletion timing can vary by image set and tag, so there isn't a single public “all CUDA 11 disappears on this date” deadline. But older CUDA 11 tags are already a real availability risk, and the `11.2.1-base-ubuntu20.04` example shows why.
Is pulling by digest enough?
It solves one problem: it tells the registry exactly which image content you want.
It doesn't guarantee the registry will retain and serve that content forever. Identity is not the same thing as availability.
What is the difference between nvcr.io and ghcr.io?
`nvcr.io` is NVIDIA’s NGC container-registry endpoint. `ghcr.io` is GitHub Container Registry. They host different image ecosystems, and the same build can depend on both. StableBuild now supports immutable pull-through caching for both.
Do I still need the NVIDIA Container Toolkit?
Yes, when your host needs to expose NVIDIA GPUs to Docker or another supported runtime. StableBuild changes where the container image is retrieved and preserved; it does not replace the NVIDIA runtime components that make the GPU available inside the container.
Can StableBuild recover an image after it has already disappeared?
StableBuild can keep serving images it has already captured. It cannot guarantee recovery of an upstream artifact that disappeared before StableBuild ever saw it. That is why the best time to pin a dependency is while it's still available.
Pin it before you need it.
If you have an NVIDIA CUDA, NGC, or GHCR image that a real build depends on today, do not wait for a production rebuild to discover whether the upstream tag still exists.
Pull it through StableBuild while it is available.
Then, when NVIDIA retires an old CUDA image, a GitHub package is removed, or another upstream registry changes something you were depending on, you aren't forced to rebuild your environment on somebody else’s schedule.
Your AI build should change because you changed it, not because somebody else ran a cleanup job.
StableBuild’s Community tier is free, so you can start by preserving one container your build actually depends on and see how the workflow fits.
