GPU · basic
Right after your config-management system pushed a new NVIDIA driver package to a live node, users report CUDA failures and you see the output below. What is the correct fix?
$ nvidia-smi
Failed to initialize NVML: Driver/library version mismatch
NVML library version: 550.54
The options
- AReinstall the CUDA toolkit so its runtime libraries are rebuilt against the currently loaded driver, since NVML ships with the toolkit
- BReboot the node (or unload and reload the NVIDIA kernel modules) so the running kernel module matches the new userspace libraries
- CThe GPU has fallen off the bus; power-cycle the chassis and reseat the card, since NVML cannot version-check a device that is gone
- DBlacklist the nouveau driver and regenerate the initramfs, since nouveau reclaimed the devices when the package was replaced
The answer
B. Reboot the node (or unload and reload the NVIDIA kernel modules) so the running kernel module matches the new userspace libraries
Why
The package update replaced the userspace NVML library (550.54) on disk, but the old driver kernel module is still loaded in memory, so the two disagree. Rebooting, or stopping all GPU clients and doing rmmod/modprobe of the nvidia modules, brings the loaded module in line with the libraries. The CUDA toolkit is unrelated to NVML versioning, a fallen-off-the-bus GPU produces XID 79 and missing devices rather than this message, and nouveau conflicts prevent the NVIDIA module from loading at all instead of causing a version mismatch.
More GPU questions
This is 1 of 10 free questions. The full bank is 150 questions and 10 incident labs against a simulated 4-node HGX cluster you can break and repair — €7.99. All free questions · Field notes
← Back to ClusterDrill