Docker GPU Setup: NVIDIA Container Runtime
Install and verify GPU access in Docker while keeping host drivers, images, permissions, and secrets under control.
By Hostnot GPU ·
This guide targets Docker Engine on a self-managed Linux NVIDIA host. A managed GPU image may already provide the entire stack, or may give you a container rather than permission to administer Docker. Establish that boundary first. Windows/WSL and rootless Docker require their own documented configuration paths.
Prove the prerequisites
You need a supported OS, a working host driver, Docker daemon access, sufficient disk for images, and permission to run the selected GPU workload. Check these before installing anything:
nvidia-smi -L
nvidia-smi --query-gpu=name,driver_version --format=csv
docker version
docker context show
docker info --format '{{json .Runtimes}}'
nvidia-ctk --versionA Docker client version alone does not establish daemon access. Check the server section and active context: a remote Docker daemon uses its own GPUs, not your laptop's. A missing nvidia-ctk suggests the toolkit is absent from this host, but first confirm you are not inside an already provisioned container.
Install through the supported host workflow
- Read NVIDIA's toolkit platform support and distribution-specific installation page. Use its signed stable package repository, selecting an approved toolkit version and matching companion package versions. Avoid mixing distribution packages, experimental repositories, and unrelated install scripts.
- For an owned host, back up existing Docker configuration, drain affected workloads, and confirm a recovery path.
- After toolkit installation, the standard rootful Docker configuration command below updates
/etc/docker/daemon.json. Review the resulting configuration. Apply the documented Docker restart only in the host's approved maintenance window; it can interrupt unrelated containers.
# Host administrator only, after installation and maintenance approval:
sudo nvidia-ctk runtime configure --runtime=dockerRootless Docker uses a user configuration file and user daemon; do not apply rootful settings or global cgroup changes blindly. On a managed instance, ask for the supported runtime configuration instead of replacing provider settings. No kernel driver belongs in the application container.
Run a bounded device-visibility test
Select an official NVIDIA CUDA image compatible with the host and pin its reviewed digest. Set CUDA_IMAGE to that full image reference and GPU_UUID to one allocated GPU from nvidia-smi -L. The variables below deliberately have no invented digest. Downloading an uncached image consumes network, disk, and rented time.
: "${CUDA_IMAGE:?Set a reviewed nvidia/cuda image reference with digest}"
: "${GPU_UUID:?Set the UUID of one GPU allocated to this workload}"
docker run --rm --gpus "device=$GPU_UUID" "$CUDA_IMAGE" nvidia-smiExpect the selected device, not necessarily every GPU visible to the host. This tests driver utility access through the container, not CUDA arithmetic. Next run the small tensor check inside your actual framework image. Capture the image digest and application versions after it passes.
Constrain the real workload
Use an explicit GPU selection, a non-root application user where supported, bounded CPU/RAM, and only required mounts. Mount input data read-only and outputs into a deliberate directory. Do not mount the Docker socket or use --privileged to solve a GPU visibility problem: Docker daemon access already conveys substantial host authority. Publish development ports to loopback and use SSH forwarding. Keep model-access tokens out of image layers and logged command lines.
Docker host-memory limits do not cap GPU VRAM; control model context, batch, and concurrency separately. --rm removes the exited container, not model caches, images, or the paid instance. Preserve outputs and terminate unneeded compute via instance controls. If the sample fails, use the layer-by-layer Docker diagnosis before rebuilding an image.