# Guide: Transcoder monitoring with Prometheus/Grafana **URL:** \ **Category:** Transcoders\ **Created:** [February 5, 2021, 2:14pm UTC](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225 "2021-02-05T14:14:30Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![vires-in-numeris](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.livepeer.org/vires-in-numeris/32/731_2.png) [@vires-in-numeris](https://forum.livepeer.org/u/vires-in-numeris)\ **Post date:** [February 5, 2021, 2:14pm UTC](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225/1 "2021-02-05T14:14:30Z")
Hey all, here’s a guide on how to setup prometheus/grafana monitoring for your GPU transcoder. The dashboards and the Nvidia exporter are heavily inspired/copied from the “official” [Livepeer Monitoring](https://github.com/livepeer/docker-livepeer/tree/master/monitoring) - so many thanks to the original creator! You might want to try to run their docker solution first. I was looking for a more modular approach since I was already running Grafana/Prometheus for other stuff. Also a thank you to @yondon for answering my questions on Discord 🙂 I hope this guide works for you - please comment if you have any questions/problems. Let’s get started: 1. Prometheus: - Go to [Download | Prometheus](https://prometheus.io/download/) and get the latest version (with wget) - Extract with `tar -xzf prometheus-...` - Inside the prometheus folder, edit the prometheus.yml file according to [this example](https://github.com/0xVires/livepeer-grafana-monitoring/blob/main/prometheus.yml) - You can create a systemd or simply run prometheus in a separate terminal (e.g. using screen) 1. Nvidia-smi exporter: - You need to have golang installed - Download [nvidia\_exporter.go](https://raw.githubusercontent.com/0xVires/livepeer-grafana-monitoring/main/nvidia_exporter.go) - You can create a systemd or simply run this file in a separate terminal (e.g. with screen) with `go run nividia_exporter.go` 1. Grafana - Install Grafana according to here (latest OSS): [Install on Debian/Ubuntu | Grafana Labs](https://grafana.com/docs/grafana/latest/installation/debian/) (you might need to install the gpg-agent if not already installed) - Grafana already comes with a systemd service that you can start with `sudo systemctl start grafana-server` - Now you should have access to “your-IP”:3000 in your browser (might need to open the port in your server’s firewall settings). The default user and password is “admin”, change accordingly. - Once logged in, first add Prometheus under “configuration → data sources → add data source”. The URL is [http://localhost:9090](http://localhost:9090) - Next go to “Dashboards → Manage → Import” and paste the json of the dashboards that you want: [Dashboards](https://github.com/0xVires/livepeer-grafana-monitoring/tree/main/Dashboards) 1. Run your Livepeer node with the `-monitor` flag ![image](https://canada1.discourse-cdn.com/flex030/uploads/livepeer/original/1X/1b4d9f4f8fa0400fc2c6e87dcc84723135922869.png) ![image](https://canada1.discourse-cdn.com/flex030/uploads/livepeer/original/1X/d773d5cfc325f90fea1043e227cf7f0348126f06.png) ![image](https://canada1.discourse-cdn.com/flex030/uploads/livepeer/original/1X/56a4a4eb5cee4c813b8e9cdd47602dcd444c011e.jpeg) ---
**Author:** ![chris-chainflow](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.livepeer.org/chris-chainflow/32/704_2.png) [@chris-chainflow](https://forum.livepeer.org/u/chris-chainflow)\ **Post date:** [February 5, 2021, 7:15pm UTC](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225/2 "2021-02-05T19:15:48Z")
Thank you for this 🙏 ---
**Author:** ![ChuckChain](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.livepeer.org/chuckchain/32/1678_2.png) [@ChuckChain](https://forum.livepeer.org/u/ChuckChain)\ **Post date:** [February 6, 2021, 9:27pm UTC](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225/3 "2021-02-06T21:27:27Z")
Got mine up and running - thanks a lot @vires-in-numeris! ---
**Author:** ![Titan-Node](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.livepeer.org/titan-node/32/1460_2.png) [@Titan-Node](https://forum.livepeer.org/u/Titan-Node)\ **Post date:** [May 29, 2021, 2:05am UTC](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225/5 "2021-05-29T02:05:28Z")
Got mine up and running. Thank you!! ---
**Author:** ![Wiser](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.livepeer.org/wiser/32/1123_2.png) [@Wiser](https://forum.livepeer.org/u/Wiser)\ **Post date:** [May 29, 2021, 3:37pm UTC](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225/6 "2021-05-29T15:37:28Z")
Hi and thanks for this. It’s possible to monitoring if server Linux is only orchestrator and transcoder on 2 remote Windows 10 PC ? ---
**Author:** ![Titan-Node](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.livepeer.org/titan-node/32/1460_2.png) [@Titan-Node](https://forum.livepeer.org/u/Titan-Node)\ **Post date:** [May 29, 2021, 5:53pm UTC](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225/7 "2021-05-29T17:53:17Z")
> [@vires-in-numeris](#): > > go run nividia\_exporter.go Most likely yes but I would assume this needs some custom configuration ---
**Author:** ![Wiser](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.livepeer.org/wiser/32/1123_2.png) [@Wiser](https://forum.livepeer.org/u/Wiser)\ **Post date:** [May 29, 2021, 6:41pm UTC](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225/8 "2021-05-29T18:41:55Z")
Ok i ll try anyway but i think it’s far beyond my knowledge 😁 but who knows… Thanks ---
**Author:** ![jjassonn69](https://avatars.discourse-cdn.com/v4/letter/j/838e76/32.png) [@jjassonn69](https://forum.livepeer.org/u/jjassonn69)\ **Post date:** [June 9, 2021, 9:30pm UTC](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225/10 "2021-06-09T21:30:19Z")
hei, I tried to set it all up. The nvidia exporter works but the livepeer metrics doesnt. if I check the localhost:7935/metrics I cant see anything there but if i use 127.0.0.1:7935 I can see the metrics. How do I change the settings so that prometheus sees the metrics from 127.0.0.1:7935 instead of localhost which when i open in browser resolved into jason-pc:7935 which is my pc name. help… ---
**Author:** ![vires-in-numeris](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.livepeer.org/vires-in-numeris/32/731_2.png) [@vires-in-numeris](https://forum.livepeer.org/u/vires-in-numeris)\ **Post date:** [June 9, 2021, 10:59pm UTC](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225/11 "2021-06-09T22:59:02Z")
You can customize the target in the prometheus.yml file. Just change the “livepeer” one from “localhost:7935” to “127.0.0.1:7935”. ---
**Author:** ![jjassonn69](https://avatars.discourse-cdn.com/v4/letter/j/838e76/32.png) [@jjassonn69](https://forum.livepeer.org/u/jjassonn69)\ **Post date:** [June 9, 2021, 11:22pm UTC](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225/12 "2021-06-09T23:22:13Z")
I did that but I am not getting and livepeer data on prometheus. I have only estimated gas in the graphana and everything else is no data. ---
**Author:** ![vires-in-numeris](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.livepeer.org/vires-in-numeris/32/731_2.png) [@vires-in-numeris](https://forum.livepeer.org/u/vires-in-numeris)\ **Post date:** [September 3, 2021, 9:44am UTC](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225/13 "2021-09-03T09:44:14Z")
Since Grafana showed that I’ve received a winning ticket but my Orchestrator didn’t automatically redeem it, I’ve decided to set up an alert in Grafana that notifies me about a winning ticket. Here’s how you do it: 1. Set up a notification channel by going to “Alerting” → “Notification channels” on the left menu bar. Click on “New channel” 2. The next step depends on the channel you want, here’s an instruction on how you do it for Telegram: [https://gist.github.com/dl6nm/c312acbc6fddf1a56d749e045f040ca3](https://gist.github.com/dl6nm/c312acbc6fddf1a56d749e045f040ca3) 3. Create/edit the “Winning Tickets” time series panel. The query should be this: `sum(livepeer_winning_tickets_recv OR on() vector(0))`. The sum and vector(0) is that prometheus returns 0 instead of no data when there is no winning ticket. 4. Switch from query to the “Alert” tab. Set the conditions equal to `WHEN diff() OF query(A, 15m, now) IS ABOVE 0`, edit the “Send to” to add your notification channel that you added in the first step, adjust the message to your liking and that’s it 🙂 ---
**Author:** ![Strykar](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.livepeer.org/strykar/32/758_2.png) [@Strykar](https://forum.livepeer.org/u/Strykar)\ **Post date:** [September 17, 2021, 11:41am UTC](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225/14 "2021-09-17T11:41:45Z")
Amazing, thanks for this! 🙏 Would you consider adding avg GPU power draw to the Nvidia dashboard? I have an Nvidia made P400 and should be receiving a PNY P400 V2 revision (which has lower power consumption) and it would be nice to plot the difference in Grafana. It would also help to create another plot (based on the user’s actual currency) to see how much electricity the transcoding costs. Something like `nvidia-smi --query-gpu=power.draw --format=csv`? Just wanted to also add that you can use the free hosted instance of Grafana’s own cloud instead of installing it locally. I’ve installed prometheus to listen on my LAN IP, setup port forwarding on the router and Basic Auth for prometheus and it works well. ---
**Author:** ![vires-in-numeris](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.livepeer.org/vires-in-numeris/32/731_2.png) [@vires-in-numeris](https://forum.livepeer.org/u/vires-in-numeris)\ **Post date:** [September 18, 2021, 9:44pm UTC](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225/15 "2021-09-18T21:44:04Z")
Thank you! To add the power, you’d have to adjust the nvidia\_exporter.go script and add this to the metrics function. Shouldn’t be to hard I think, have you tried it already? ---
**Author:** ![Strykar](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.livepeer.org/strykar/32/758_2.png) [@Strykar](https://forum.livepeer.org/u/Strykar)\ **Post date:** [September 18, 2021, 10:41pm UTC](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225/16 "2021-09-18T22:41:16Z")
I did try, but do not know Go and gave up after a few failed attempts and asked here. Appears there’s more to it than simply adding `power.draw` to the list in `--query-gpu=name,index,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used`. Think I need to add it to the metric list too, unsure… ---
**Author:** ![Titan-Node](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.livepeer.org/titan-node/32/1460_2.png) [@Titan-Node](https://forum.livepeer.org/u/Titan-Node)\ **Post date:** [September 20, 2021, 11:03pm UTC](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225/17 "2021-09-20T23:03:28Z")
Would there be a way to include the Encoding and Decoding resource to the list of metrics? I assume this would have to be done in the nvidia\_exporter.go script but I’m not sure how to do that. Any suggestions would be great 🙂 ---
**Author:** ![elitmik](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.livepeer.org/elitmik/32/1319_2.png) [@elitmik](https://forum.livepeer.org/u/elitmik)\ **Post date:** [October 25, 2021, 3:18am UTC](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225/18 "2021-10-25T03:18:58Z")
How can this be configured to support a remote transcoder configuration? Does Prometheus need to be running on the Orchestrator box specifically? ---
**Author:** ![vires-in-numeris](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.livepeer.org/vires-in-numeris/32/731_2.png) [@vires-in-numeris](https://forum.livepeer.org/u/vires-in-numeris)\ **Post date:** [October 25, 2021, 8:40am UTC](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225/19 "2021-10-25T08:40:46Z")
Yes, the default setting/setup requires that prometheus is run on the Orchestrator box. I’m also running grafana on the Orchestrator - those two processes don’t require that much resources and it’s the most simple setup. ---
**Author:** ![pushbutton\_snowbank](https://avatars.discourse-cdn.com/v4/letter/p/3ec8ea/32.png) [@pushbutton\_snowbank](https://forum.livepeer.org/u/pushbutton_snowbank)\ **Post date:** [October 25, 2021, 4:21pm UTC](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225/20 "2021-10-25T16:21:47Z")
This is awesome! Thanks for recording your process 😉 ---
**Author:** ![musicmank545](https://avatars.discourse-cdn.com/v4/letter/m/34f0e0/32.png) [@musicmank545](https://forum.livepeer.org/u/musicmank545)\ **Post date:** [October 27, 2021, 3:25pm UTC](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225/21 "2021-10-27T15:25:35Z")
It is possible to setup remote monitoring, that’s how I do it. You just need to ensure port 7935 (or whatever port you set as the cliAddr) is open and set the listening address to zeros so it accepts requests from all IPs: -cliAddr 0.0.0.0:7935. In the prometheus.yml config, instead of ‘localhost:7935’, set the targets to :7935. I don’t think it’s safe to just have the cli port open to the internet so I use a firewall to only accept traffic from my monitoring node. ---
**Author:** ![elitmik](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.livepeer.org/elitmik/32/1319_2.png) [@elitmik](https://forum.livepeer.org/u/elitmik)\ **Post date:** [December 3, 2021, 1:38am UTC](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225/22 "2021-12-03T01:38:44Z")
I’m trying to make sure that this is working correctly. Does it need to be done using a certain one of the dashboards or does it work with all of them? I tried to evaluate the query and I see this error. I do not see the variable “livepeer\_winning\_tickets\_recv” anywhere on the metrics browser. ![image](https://canada1.discourse-cdn.com/flex030/uploads/livepeer/original/1X/33c3a01ad6b179e23452f7eeb65c8950fd5b4078.png) [Next page](https://forum.livepeer.org/t/guide-transcoder-monitoring-with-prometheus-grafana/1225.md?page=2)