Guide: Transcoder monitoring with Prometheus/Grafana

Hey all, here’s a guide on how to setup prometheus/grafana monitoring for your GPU transcoder.

The dashboards and the Nvidia exporter are heavily inspired/copied from the “official” Livepeer Monitoring - so many thanks to the original creator! You might want to try to run their docker solution first. I was looking for a more modular approach since I was already running Grafana/Prometheus for other stuff. Also a thank you to @yondon for answering my questions on Discord :slight_smile:

I hope this guide works for you - please comment if you have any questions/problems. Let’s get started:

  1. Prometheus:
  • Go to Download | Prometheus and get the latest version (with wget)
  • Extract with tar -xzf prometheus-...
  • Inside the prometheus folder, edit the prometheus.yml file according to this example
  • You can create a systemd or simply run prometheus in a separate terminal (e.g. using screen)
  1. Nvidia-smi exporter:
  • You need to have golang installed
  • Download nvidia_exporter.go
  • You can create a systemd or simply run this file in a separate terminal (e.g. with screen) with go run nividia_exporter.go
  1. Grafana
  • Install Grafana according to here (latest OSS): Install on Debian/Ubuntu | Grafana Labs (you might need to install the gpg-agent if not already installed)
  • Grafana already comes with a systemd service that you can start with sudo systemctl start grafana-server
  • Now you should have access to “your-IP”:3000 in your browser (might need to open the port in your server’s firewall settings). The default user and password is “admin”, change accordingly.
  • Once logged in, first add Prometheus under “configuration → data sources → add data source”. The URL is http://localhost:9090
  • Next go to “Dashboards → Manage → Import” and paste the json of the dashboards that you want: Dashboards
  1. Run your Livepeer node with the -monitor flag

3 Likes

Thank you for this :pray:

1 Like

Got mine up and running - thanks a lot @vires-in-numeris!

1 Like