Appendix B: the troubleshooting playbook, from missing GPUs to NCCL hangs and Xid codes

When a lower layer breaks it rarely speaks; the layer above fails and reports it in its own words, so the error says who noticed, not what broke. Check each layer from the bottom. A classic case: an update replaced the driver on disk while the old module was still loaded, and a reboot fixes it.

In Daveth Pro (AI Engineering)

What this page answers

Why does nvidia-smi fail after an automatic driver update?

The update replaced the driver files on disk, but the old module was still loaded in the running kernel. New programs load the new library, which refuses to talk to the old module. One reboot loads the new one. Read it in “Three reinstalls later”

How do I find which layer of the GPU stack is broken?

The error message tells you who noticed. Check each layer starting from the bottom, the GPU on the PCIe bus, then the kernel module, the driver libraries, any container and PyTorch; the first check that fails is the layer to fix. Read it in “Noticed here, broken there”

Three reinstalls later

Tuesday morning. Yesterday this machine trained fine. Today PyTorch says there's no GPU. You reinstall PyTorch, then reinstall it with a different CUDA build, then once more from scratch. Nothing changes. Then, finally, you run the GPU's own tool:

$ python -c "import torch; print(torch.cuda.is_available())"
UserWarning: CUDA initialization: ... Error 803: system has unsupported display driver / cuda driver combination
False
$ history | grep -c "pip install.*torch"
3
$ nvidia-smi
Failed to initialize NVML: Driver/library version mismatch
NVML library version: 580.95
$ head -1 /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  580.82.07  ...

The messages are real ones (shortened). The version numbers are an example.

The library on disk says 580.95. The driver loaded in the kernel says 580.82. What fixes this?

  • A fourth reinstall, with a PyTorch built for an older CUDA
  • Reboot, so the kernel loads the new driver that's already on disk
  • Downgrade the driver back to 580.82

Answer. Overnight, automatic updates installed a new driver. The update replaced the driver's files on disk, but the old driver was still loaded in the running kernel, and a kernel module stays in memory until it's unloaded or the machine restarts. Now every new program loads the new 580.95 library, which asks the old 580.82 module for the GPU, and the two refuse to talk. One sudo reboot loads the new module, and both halves match again.

Look at where the error appeared: in Python. Look at where the problem was: two layers lower, in the driver. Every one of the three reinstalls fixed a layer that was never broken. That's the most common way to lose a morning on a GPU machine, and the reason this playbook starts with a method rather than a list.

Noticed here, broken there

Your training code sits on a tall stack: PyTorch, then maybe a container, then the driver's libraries, then the driver's kernel module, then the GPU itself on the PCIe bus. When a lower layer breaks, it usually doesn't say anything. The layer above tries to use it, fails, and reports the failure in its own words. So the error message tells you who noticed, and the real question is how far down to look.

The answer is the same five-step ladder from Down the Lab: check each layer, starting from the bottom. The first check that fails is the layer to fix. Here's the ladder with the hook's results filled in:

Drawing: Layer, Check it with, In the hook, 5 · Your code and PyTorch, torch.cuda.is_available(), fails, 4 · Container, if you use one, nvidia-smi inside it, no container, 3 · Driver libraries, nvidia-smi, 2 · Driver kernel module, lsmod | grep nvidia, passes, 1 · GPU on the PCIe bus, lspci | grep -i nvidia, Noticed here:, PyTorch prints False, Broken here:, library 580.95, module 580.82, Read fro

Two habits come out of this picture. First, before you change anything, spend two minutes running the checks from the bottom up. They're fast and they only look, so they can't make things worse. Second, when a reinstall “didn't help”, that's evidence: the layer you reinstalled is probably fine, so look lower. Every playbook below is this ladder, shaped around one particular error.

Paste the error

Error messages are long, but usually one short phrase identifies the problem: version mismatch, GLIBCXX_3.4.32, Bus error, Watchdog caught. Paste the error you're seeing (the last few lines of the traceback are enough), and this box finds the phrase, tells you which layer it usually points to, and opens the right playbook.

A bare Killed means the system's memory ran out and the kernel stopped the program, a different layer with a different fix. The words “out of memory” appear in both, which is exactly why reading the precise phrase matters.

The playbooks

Each step gives you a command to run and asks what you saw. Your answer picks the next step, and the walk ends at a cause and a fix, usually within three or four steps.

Drawing: These cover the common causes, in the order worth checking. Rare causes are left out on purpose: if a walk ends somewhere that doesn't fit, collect the evidence (the full error, dmesg, the versions) and go to the chapter's lesson.

Walk the CUDA playbook twice with different answers and look at where it ends: the hardware, the kernel, the driver, the container, the Python package, an environment variable. Six different layers, one error message. That's the method in action. And notice the order: the checks always go from the cheapest and lowest layer upward, because a hardware or driver problem makes every later check meaningless.

Why version pairs drift apart

A surprising number of the errors on this page are the same mechanism wearing different clothes: two pieces that must match, loaded at different times. The NVIDIA driver is the clearest case. It comes in two halves: a kernel module, loaded into memory when the machine starts, and libraries on disk, loaded fresh by every new program. An update changes the files on disk straight away, but the module in memory stays the old one until the next restart. Here's the hook's week as a timeline:

Drawing: Loaded in memory, Files on disk, kernel module 580.82, 580.95, libraries 580.82, libraries 580.95, new program: library and module disagree, Mon 08:00 boot, Tue 03:10 update, Tue 09:00 you start work, reboot, From 03:10 until the reboot, the two halves disagree, Times are an example. On Ubuntu, unattended upgrades can install driver updates overnight unless the driver is held with apt-mar

Between the update and the reboot, the machine is in a split state: the top lane still says 580.82, the bottom lane already says 580.95. Once you see that shape, you'll spot it everywhere. Each pair below has one piece fixed early and one piece that can change later:

The pairFixed whenChanges whenThe error
Driver module and driver librariesBootA driver updateDriver/library version mismatch
Kernel and the driver module built for itDKMS buildA kernel updatecouldn't communicate with the NVIDIA driver
A program and the C++ library it was built withCompile time, on someone's machineNever, but your OS may be olderGLIBCXX_3.4.32 not found
PyTorch's CUDA build and the driverpip installA driver change, or a new PyTorchdriver on your system is too old
An image's libraries and the host driverdocker buildA host driver updateMismatch, but only inside the container

The fix in every row is to make the two match, and the prevention is to control when the “changes later” half changes: hold the driver package, keep kernel headers installed so DKMS rebuilds automatically, pin PyTorch versions, and never copy driver libraries into an image (Down the Image). Down the Library covers the GLIBCXX row in depth.

Predict. A training job started on Monday is still running on Tuesday morning, after the update. nvidia-smi now fails with the mismatch. Is the running job about to crash?

Answer. Usually not. The job loaded the 580.82 libraries when it started, and they're still in its memory, talking to the 580.82 module. It keeps training. Only programs started after the update fail, which is why the error seems to appear out of nowhere.

The trap comes later: if the job crashes or you stop it, it can't start again until you reboot. So let it finish (or reach a checkpoint), stop it cleanly, and reboot then. On a shared machine, that means draining it first.