Working Around Occasional Transcoding Issues with Nvidia Driver 450

If you are planning on transcoding using a Nvidia GPU, you will need to install a Nvidia driver. At the moment, certain driver versions such as 450 are known to cause issues with transcoding sometimes. In these scenarios, an orchestrator can end up with many “stuck” transcode sessions. The GPU starts to transcode a video segment, but never returns a result and the node thinks that the transcode session is actively transcoding segments when it is not actually being utilized. A signal of stuck transcode sessions if you see many “OrchestratorCapped” errors in the historical test results for your orchestrator on the leaderboard, but do not see any transcoding logs for your node.

The current recommendation to workaround these issues is to use the following driver versions which should not be impacted by these issues:

If additional testing indicates that other driver versions are not impacted by these issues as well, this recommendation can be updated to include those versions.

The following instructions are for Linux users on Ubuntu 18.04.

Uninstall Existing Drivers

If you already have an existing Nvidia driver installed, you will need to uninstall it.

Install Driver 440

apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
add-apt-repository "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /"
apt install nvidia-driver-440

If you encounter an error during installation, you may need to fix missing or broken packages.

Thanks to @chris-chainflow for sharing the links for uninstalling existing Nvidia drivers and fixing missing or broken packages.

Adding to the topic, here’s my installation method for ubuntu 20.04 and driver 460.39 (basically the instructions of the nvidia unlimited patch github - which I can’t link here). The 460.39 driver seem to work well for transcoding.

Make sure you have kernel headers and compiler installed before running Nvidia driver installer. Kernel headers and compiler are required to build nvidia kernel module. Recommended way to do this is to install dkms package, if it is available in your distro. This way dkms package will pull all required dependencies to allow building kernel modules and kernel module builds will be automated in a reliable fashion.

→ Even though I had the dkms package, my kernel headers were missing… apparently because I did “apt-get dist-upgrade”. In order to fix that, I ran sudo apt-get install linux-headers-$(uname -r)

Then I followed those instructions but replaced the link with the 460.39 driver version:

mkdir /opt/nvidia && cd /opt/nvidia
wget https://international.download.nvidia.com/XFree86/Linux-x86_64/430.50/NVIDIA-Linux-x86_64-430.50.run
chmod +x ./NVIDIA-Linux-x86_64-430.50.run
./NVIDIA-Linux-x86_64-430.50.run

1 Like