Hmm are you running it on your Orchestrator? Most metrics are not supported if you’re running it on a Transcoder
Here’s how you can add a table to your grafana that shows the latest winning tickets:
Edit: You’ll need to have a winner in the selected “Relative time”, otherwise you won’t see anything. You also need to increase the prometheus storage rentention to something like 1y (default is only 14 days, you can do this by adding --storage.tsdb.retention.time=1y to your startup command/systemd file).
First make sure that you select “Table” as your visualization. Then in the query tab, click on the query options and adjust the numbers according to the screenshot.
The query is:
min((max_over_time(livepeer_winning_tickets_recv[10m]) or livepeer_ticket_value_recv*0) - (max_over_time(livepeer_winning_tickets_recv[10m] offset 10m) or livepeer_ticket_value_recv*0)) > 0
Then click on the “Overrides” tab on the right side, select your query and switch to “Hide in table”.
Nerd mode: The part with the or livepeer_ticket_value_recv*0 might seem weird but it’s needed since the winning_tickets value is not initialized until you get your first winner - so the query without that wouldn’t work for the first winner after every restart. To counter that we use another value that’s initialized faster and set it to 0. The max_over_time query will then take whichever value is bigger.
If you want a bigger “Relative time” aka history, you need to adjust some other values in order to not run into query limits. Here’s an example for 90 days:
2 Likes
Hey! Is there a way to have 2 machines use the same Grafana data? For example I have my main Orchestrator on 1 machine and a backup O on a second machine, howler they both use their own Prometheus instance which is a bit inconvenient.
hey guys,
thanks OP for the first post/guide. I just want to say that im running everything on windows and can be done using nvidia_gpu_exporter (GitHub - utkuozdemir/nvidia_gpu_exporter: Nvidia GPU exporter for prometheus using nvidia-smi binary)
@Authority_Null as far as i know, grafana doesn’t store data (like kibana) it just display it. So you need one prometheus instance as a source. you could try prometheus agent mode
2 Likes
Thank you! I brought this us on the weekly water cooler chat and it was explained to me that we can open port 9090 and have Grafana point to the correct Prometheus instances. This would require some router firewalling to be secure but it seems easy enough to configure.
What is the nvida_exporter used for?
The nvidia_exporter allows you to see detailed stats for your GPU