GPU Driver Crash Recovery

A safe recovery sequence for driver failures, Xid errors, lost devices, and mismatched CUDA environments.

By Hostnot GPU · 9/6/2026

On Linux, a CUDA error, an NVIDIA Xid event, and an OS crash are different observations. Preserve the first failure and its scope before rebooting. A driver restart can erase useful volatile evidence, interrupt other jobs, and leave you without a reproduction.

Stabilize and collect a short timeline

Stop submitting new work to the affected allocation. Save application logs and checkpoints if the machine remains responsive. Record UTC/timezone, GPU allocation, model or dataset revision, image digest, last successful step, and recent changes. Run the following only with authorized host visibility:

date -u
uname -r
nvidia-smi -L
nvidia-smi
journalctl -k -b --since '-15 minutes' --no-pager

Kernel logs may require elevated permission or may be inaccessible inside a container. Ask the host owner for a bounded extract rather than changing log-access policy. Look for NVIDIA/NVRM/Xid events, PCIe faults, and host out-of-memory messages near the failure; redact unrelated workload details before sharing. If nvidia-smi hangs, interrupt it and report that behavior rather than creating many simultaneous probes.

Classify the failure

CUDA out of memory with a healthy driver: reduce memory demand or eliminate retained tensors. This usually needs application changes, not a driver reinstall.
Illegal memory access / device-side assert in one program: stop that process. Its CUDA context may no longer be usable. Reproduce the smallest failing input in a fresh process and inspect custom kernels, indices, and tensor shapes.
NVML driver/library mismatch: compare the loaded host module with installed user-space packages, especially after maintenance.
GPU lost or repeated Xid events across programs: preserve the exact Xid, associated text, GPU identifier, driver branch, and time. Consult NVIDIA's current Xid catalog; the code is evidence, not a standalone root-cause verdict.
Whole host unreachable or kernel panic: use the provider's supported recovery/support route rather than SSH-level fixes.

Debug application faults in isolation

CUDA work is asynchronous, so the reported failing line can be later than the actual faulty launch. On a small, non-production reproduction, CUDA_LAUNCH_BLOCKING=1 python repro.py makes launches synchronous and can improve error attribution. It slows execution and changes timing; do not use it for benchmark results or leave it enabled in production. Compute Sanitizer can help a developer investigate custom CUDA code, but instrumented runs can be expensive.

Recover at the narrowest safe boundary

If only the application failed and host health is normal, try one fresh process after preserving evidence. If the GPU requires recovery, let the host owner drain users and follow the documented action for that device and error. A GPU reset may affect peer-connected devices, may be unavailable in virtualized environments, and is not guaranteed to recover hardware. Do not run blanket reset, module-unload, or driver-reinstall commands on a managed host.

Following an approved restart or replacement, verify device enumeration, a minimal tensor operation, and the original small reproduction before resuming a long job. Repeated failures with a known-good workload warrant operator investigation. Repeatedly restarting the same failing input can waste compute and destroy evidence.

Prevent a costly recurrence

Pin validated packages and image digests, keep restart counts bounded, and checkpoint so retries resume from a known state. Check whether the failed job produced partial outputs before resubmitting. A stalled process or SSH disconnect does not imply the instance stopped billing. Save diagnostics securely and review instance lifecycle. Use planned driver maintenance only when the compatibility evidence supports it.