Docker GPU Troubleshooting

Fix the most common container GPU failures by checking runtime registration, driver visibility, permissions, and CUDA compatibility.

By Hostnot GPU · 9/6/2026

Diagnose container GPU failures from the host inward: allocation, host driver, container engine, GPU injection, then framework. Rebuilding an application image cannot repair a GPU that the host driver cannot reach. Keep the failing container's image digest and first error before replacing it.

Confirm which host runs Docker

Run these checks where you have authorized access. If the rental exposes only an application container, ask the operator for host-side checks instead of assuming that container is the host.

docker context show
docker version
docker info --format '{{json .Runtimes}}'
nvidia-smi -L

A remote Docker context can point to another machine. A working local nvidia-smi does not prove that remote daemon has GPUs. Permission denied on the Docker socket is a daemon-access issue; do not make the socket world-writable. Membership in the Docker group is security-sensitive and should follow host policy.

Separate injection from application execution

Set CUDA_IMAGE to an approved official NVIDIA CUDA image digest compatible with the host; select an allocated UUID. Run a short visibility check before testing the application image:

: "${CUDA_IMAGE:?Set the approved CUDA image digest}"
: "${GPU_UUID:?Set an allocated GPU UUID}"
docker run --rm --gpus "device=$GPU_UUID" "$CUDA_IMAGE" nvidia-smi

Visibility of one selected GPU is success even if the host has more. A missing nvidia-smi executable inside a minimal application image is not definitive proof that CUDA is unavailable: images and injected driver capabilities differ. The utility uses NVML; run a small tensor operation in the actual framework image to establish compute access.

Use the exact error to choose the next check

Could not select device driver with capabilities [[gpu]]: verify the NVIDIA toolkit is installed and configured on the daemon host. Use the setup procedure; do not fix it with privileged mode.
Unsatisfied condition: cuda>=…: the image's runtime constraint exceeds driver support. Select a compatible pinned image or request an owner-managed driver update.
Failed to initialize NVML: driver/library version mismatch: inspect host package activation and compare host/container results.
Failed to initialize NVML: Unknown Error after a host management change: NVIDIA documents a systemd/cgroup case where containers lose GPU access. Preserve the timeline and ask the owner to apply the documented recovery; do not globally disable cgroups.
PyTorch reports no CUDA, but the utility works: inspect the wheel's CUDA build, CUDA_VISIBLE_DEVICES, and driver capabilities. A CPU-only wheel remains CPU-only inside a GPU-enabled container.
CUDA out of memory: injection may already be correct. Reduce the application's context, batch, or concurrency; inspect competing processes within your authorized scope.

Inspect a container without dumping secrets

Set CONTAINER to the failing container's name or ID. Read selected fields rather than printing full inspect output, which can contain environment secrets.

: "${CONTAINER:?Set the failing container name or ID}"
docker inspect --format '{{json .HostConfig.DeviceRequests}}' "$CONTAINER"
docker inspect --format 'OOMKilled={{.State.OOMKilled}} ExitCode={{.State.ExitCode}}' "$CONTAINER"

OOMKilled=true generally concerns host/cgroup memory termination, not a CUDA VRAM exception. DataLoader shared-memory failures require checking /dev/shm and worker count. Set a justified container shared-memory allocation if needed rather than defaulting to unrestricted host IPC. Docker CPU/RAM limits and GPU VRAM limits are different controls.

Close the incident with a representative test

Record host driver, toolkit version, engine context, image digest, selected GPU, and the first failing operation. After one correction, rerun the minimal test and the same application input. Avoid broad prune commands that can remove others' images or data. Bound image downloads and repeated restarts, save outputs, and terminate unused rentals through instance controls. Keep tokens and private mounts out of support logs.