How to remove nvidia driver from ubuntu 16.04 kernel client and kernel module, or get kernel clent & module to use the same nvidia driver?

Question

After viewing /var/log/syslog and /var/log/kern.log I found the following kernel issue:

NVRM: API mistmatch: the client has version 375.26, but
NVRM: this kernel module has the version 375.39. Please 
NVRM: make sure that kernel module and all NVIDIA driver
NVRM: components have the same version.

I found Nvidia driver updated - mixed versions of 304.88 and 319.32 describing the same problem. To resolve my issue, I ran commands:

dpkg -l | grep nvidia
sudo apt-get autoremove nvidia375
sudo apt-get --purge remove nvidia*
dpkg -l | grep nvidia

The output from the last command confirmed no nvidia packages are installed. Thereafter, I ran:

sudo update-grub
sudo update-initramfs -u
sudo reboot

However, I am still experiencing the same kernel issue. Upon bootup, I am stuck in low graphics mode setup and kernel and /var/log/syslog and /var/log/kern.log still complaining client has Nvidia driver 375.26 and kernel module has Nvidia driver 375.39.

My current kernel is 4.4.0-64-generic.

Update1: To see the nvidia modules read in by the kernel, I ran this command.

$ lsmod | grep nvidia
module            size    used by
nvidia_drm        53248   0
nvidia_modeset   790528   1 nvidia_drm
nvidia         12144640   1 nvidia_modeset
drm_helper       155648   2 i915_bpo,nvidia_drm
drm              364544   4 i915_bpo,drm_helper,nvidia_drm

Also found out using modinfo module_name that these modules are located in folder /lib/modules/4.4.0-64-generic/updates/dkms/ and their filenames are nvidia-drm.ko, nvidia-modeset.ko and nvidia.ko, and their version is 375.39.

2nd Question: Why do these modules still exist even after I have purged all nvidia packages? Should I remove them? If so, how?

Solution

I was able to resolve my issue by performing the following steps:

1. First reinstall the nvidia-418 repository graphic driver.
   sudo apt-get install nvidia-418

During the installation, I noticed that nvidia-418.26 driver was being installed.

2. I moved the entire /lib/modules/4.18.0-17-generic/updates/dkms folder to my home directory backup folder. This was to make sure all old .ko files were removed.
   sudo mv /lib/modules/4.18.0-17-generic/updates/dkms/nvidia* /home//backup

3. I regenerated the .ko files for the installed driver using
   sudo dpkg-reconfigure nvidia-418

   It created a new dkms folder with the relevant nvidia kernel modules. I also checked their version using command:

   sudo modinfo /lib/modules/4.18.0-17-generic/updates/dkms/nvidia.ko
   sudo modinfo /lib/modules/4.18.0-17-generic/updates/dkms/nvidia_drm.ko
   sudo modinfo /lib/modules/4.18.0-17-generic/updates/dkms/nvidia_modeset.ko
   sudo modinfo /lib/modules/4.18.0-17-generic/updates/dkms/nvidia_uvm.ko
   

   Their output showed that they were for version 418.26. Great, this confirmed that the kernel client and kernel modules were referring to the same NVidia driver version.

4. (Optional but recommended to ensure apt-get works correctly) Uninstall and reinstall the driver sudo apt-get remove "nvidia" sudo apt-get install nvidia-driver-418

5. I rebooted the system with
   sudo reboot

Adapted from https://devtalk.nvidia.com/default/topic/525877/linux/api-mismatch-means-ubuntu-can-39-t-boot-i-can-39-t-fix-i-please-help-/1

Comments

[youri]

This fixed my 17.10 upgrade with nvidia-384

[lamhoangtung]

Thanks, just remove that dkms folder fixed my same problem