Hey all, here’s a guide on how to setup prometheus/grafana monitoring for your GPU transcoder.
The dashboards and the Nvidia exporter are heavily inspired/copied from the “official” Livepeer Monitoring - so many thanks to the original creator! You might want to try to run their docker solution first. I was looking for a more modular approach since I was already running Grafana/Prometheus for other stuff. Also a thank you to @yondon for answering my questions on Discord
I hope this guide works for you - please comment if you have any questions/problems. Let’s get started:
- Go to Download | Prometheus and get the latest version (with wget)
- Extract with
tar -xzf prometheus-...
- Inside the prometheus folder, edit the prometheus.yml file according to this example
- You can create a systemd or simply run prometheus in a separate terminal (e.g. using screen)
- Nvidia-smi exporter:
- You need to have golang installed
- Download nvidia_exporter.go
- You can create a systemd or simply run this file in a separate terminal (e.g. with screen) with
go run nividia_exporter.go
- Install Grafana according to here (latest OSS): Install on Debian/Ubuntu | Grafana Labs (you might need to install the gpg-agent if not already installed)
- Grafana already comes with a systemd service that you can start with
sudo systemctl start grafana-server
- Now you should have access to “your-IP”:3000 in your browser (might need to open the port in your server’s firewall settings). The default user and password is “admin”, change accordingly.
- Once logged in, first add Prometheus under “configuration → data sources → add data source”. The URL is http://localhost:9090
- Next go to “Dashboards → Manage → Import” and paste the json of the dashboards that you want: Dashboards
- Run your Livepeer node with the