GPU Troubleshooting: Blue Screen and Crash Recovery
A safe incident checklist for host crashes, kernel faults, workload failures, and preserving diagnostics.
By Hostnot GPU ·
A blue screen is a Windows system stop, not a synonym for a failed Linux CUDA process. First establish whether the crash happened on your Windows workstation, a Windows guest you administer, or the remote Linux GPU machine. This article does not imply that Windows GPU instances are available in the platform catalog.
Identify the failing system
If your local Windows PC blue-screens while an SSH job runs remotely, the remote job may still be running and accruing usage. Check its state from a separate trusted device before launching a duplicate. If only a Python process reports CUDA out of memory while the OS remains responsive, use application memory diagnosis instead. For a Linux kernel panic or inaccessible GPU, follow the Linux driver incident guide.
Preserve evidence after recovery
- Record the stop-code name, numeric code, time and timezone, and whether it occurred at boot, under load, or after sleep/resume.
- Note the OS build, GPU model, driver version, recent updates, and the last workload or display change.
- In Event Viewer, inspect Windows Logs → System around the event. Reliability Monitor (
perfmon /rel) provides a useful timeline of crashes and installations. - If Windows was configured to retain crash dumps, preserve the relevant dump securely. A dump can contain credentials or customer data; send it only through an approved private diagnostic channel.
The following PowerShell query reads recent selected System events on the Windows machine. It does not change driver settings.
Get-WinEvent -FilterHashtable @{LogName='System'; StartTime=(Get-Date).AddHours(-2)} |
Where-Object { $_.Id -in 41, 1001, 4101 } |
Select-Object TimeCreated, Id, ProviderName, MessageEvent IDs must be interpreted with their provider and message. A Kernel-Power 41 event establishes an unexpected shutdown, not a failing power supply. Missing events or dumps do not prove a healthy GPU; collection may have been disabled or the crash may have prevented logging.
Understand a TDR failure
Microsoft defines VIDEO_TDR_FAILURE (0x116) as a failed attempt to reset the display driver and recover from a timeout. Software, a long-running GPU workload, thermal/power conditions, or faulty hardware can contribute. Seeing nvlddmkm.sys in a report identifies a driver involved in the failure, not necessarily its root cause.
An administrator analyzing a saved dump in WinDbg can run !analyze -v and inspect implicated modules. Do not increase TdrDelay or disable TDR as a general fix: Microsoft's TDR registry controls are for targeted driver development/testing and can turn a recoverable timeout into a longer hang.
Recover with one controlled change
Pause retry loops and save recoverable outputs. If a specific driver update immediately preceded the failure, use the OEM-supported rollback or recovery procedure with console/recovery access available. Otherwise compare the supported driver and Windows combination, remove a newly introduced application/extension from the reproduction, and test a small bounded workload. For an owned physical PC, have qualified personnel check cooling and power according to the manufacturer's instructions. In managed infrastructure, escalate physical or hypervisor issues to the operator.
Repeated stops under unrelated workloads, boot loops, or signs of physical failure warrant stopping tests and obtaining support rather than running stress loops. Verify normal boot, stable device enumeration, and a short application test before resuming long jobs. Keep a timeline of changes and rollback results.
Protect work and control costs
Checkpoint jobs to verified storage, reconcile whether a previous run completed, and avoid automatic resubmission of expensive work. Closing a crashed client does not terminate the remote allocation. Check instance status and termination after recovering your outputs.