Recovering GPU passthrough after a kernel upgrade broke DKMS

OpenMediaVault runs as a virtual machine on a Proxmox host. It provides the mdadm RAID1 storage and SMB shares for the home network and is also the main Docker host, running DNS, reverse proxy, monitoring and an Ollama service that uses an NVIDIA RTX A2000 passed through from the host.

Because storage, containers and the GPU all live in the same VM, an upgrade problem there has a wide blast radius. Anything that destabilises the guest affects every service on the network.

A routine package upgrade of the VM pulled in a newer Linux kernel. The NVIDIA driver is installed as a DKMS module, which means it has to be rebuilt against every new kernel at install time. On this upgrade, that rebuild did not succeed.

• —After the reboot the GPU was no longer usable inside the guest.
• —GPU-accelerated Ollama workloads stopped.
• —The Docker NVIDIA runtime could not access the device.
• —Storage and the other containers appeared to be running, but their state had not yet been verified.

The temptation with driver problems is to reinstall components until something works. The aim here was to establish which layer had failed before changing anything.

• —Reviewed the system logs and the DKMS build logs.
• —Checked the installed kernel version against the kernel the driver module had last been built for.
• —Ran nvidia-smi to confirm whether the driver could see the GPU at all.
• —Verified the Docker NVIDIA runtime separately from the host, so a container-level problem would not be mistaken for a host-level one.
• —Confirmed that the Proxmox PCI passthrough configuration was unchanged and the device was still presented to the VM.

The passthrough was intact. The DKMS build of the NVIDIA module had failed against the newer kernel, so the driver never loaded and nothing above it, from the container runtime to the model, could work. It was a kernel and driver compatibility problem, not a hardware or virtualisation problem.

• —Booted the VM on the last known-good kernel, where the existing module built and loaded.
• —Pinned that kernel so an automatic upgrade could not reintroduce the fault.
• —Confirmed the NVIDIA 550 driver loaded and that nvidia-smi reported the RTX A2000.
• —Made no further changes until a stable recovery point had been confirmed.

• —Tested the GPU from the host operating system.
• —Tested from inside the Ollama container and confirmed models loaded into GPU memory.
• —Checked the other Docker services individually.
• —Checked the storage volumes and SMB access.

The system was not declared recovered until storage, containers, network access and the GPU workload had each been tested. The people relying on the hosted services were kept informed of the outage, its scope and when service was restored.

• —Kernel and driver compatibility is checked before an upgrade, not after.
• —A known-good kernel is retained rather than letting the package manager remove it.
• —Working kernel and driver versions are recorded.
• —Critical services are validated stage by stage after any change.
• —No unrelated changes are made until a stable recovery point is confirmed.

The broader lesson is that an upgrade on a system with kernel-dependent drivers and passed-through hardware cannot be treated as a single package-management step.