Who is this guide for?
This guide is for Orchestrators who are looking to dramatically improve operational efficiency, specifically those running a multi-node setup.
Massive thanks to @MikeZupper for spending countless hours sharing his knowledge with the community. This guide is thanks to him as well as @papa_bear who helped document and test everything.
This guide will use Ubuntu and Portainer version 2.13 but steps will be similar on other operating systems and versions of Portainer.
A few benefits:
-
A central management point for all livepeer nodes.
-
Tons of flexibility for scaling and custom-tailoring your operation to fit your needs.
-
Quick livepeer updates/deployment.
-
Although this guide will only cover the basics, advanced configurations of this method can allow for the complete disregard of SSH.
Please feel free to provide critique/feedback as I’m sure there are many different ways to get everything up and running.
You’ll need:
-
At-least 2 machines. One being your O/T and another for Portainer/Traefik. A $5/mo shared CPU server from Linode is perfectly acceptable.
-
Basic knowledge of Docker and livepeer.
-
A Portainer business license. Sign up here for 5 free nodes.
-
DNS (need to be able to create some sub-domains).
-
A stress ball or punching bag.
-
A general understanding of proxies/Traefik is recommended but not required.
Step 1: Installing Docker and Docker Compose
On the Portainer server, install Docker. We’ll be using the steps listed in Mike Zuppers Docker install guide, however, if you run into any issues you can also use Docker’s official installation steps.
Remove older version of docker:
sudo apt-get remove docker docker-engine docker.io containerd runc
Install the Docker repo for Ubuntu:
apt-get update
apt-get install ca-certificates curl wget net-tools net-tools sysstat gnupg lsb-release
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
Once the repo is installed… update the repo, install docker, enable docker to autostart on boot:
apt-get update
apt-get install docker-ce docker-ce-cli containerd.io -y
systemctl --now enable docker
Install Docker Compose and invoke necessary permissions:
sudo curl -L https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m) -o /usr/bin/docker-compose
sudo chmod 755 /usr/bin/docker-compose
Step 2: Portainer and Traefik
Create a docker-compose.yml
sudo nano docker-compose.yml
Paste the provided traefik + portainer stack
version: '3'
services:
traefik:
image: "traefik:latest"
container_name: "traefik"
restart: unless-stopped
ports:
- "80:80"
- "443:443"
# (Optional) Expose Dashboard
#- "8080:8080" # Don't do this in production!
volumes:
- traefik:/etc/traefik
- /var/run/docker.sock:/var/run/docker.sock:ro
- traefik:/var/log/traefik/
portainer:
image: portainer/portainer-ee:latest
container_name: portainer
restart: unless-stopped
depends_on:
- traefik
#security_opt:
# - no-new-privileges:true
# ports:
# - 8000:8000
# - 9000:9000
volumes:
- /etc/localtime:/etc/localtime:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- portainer_data:/data:rw
labels:
- "traefik.enable=true"
- "traefik.http.routers.portainer.entrypoints=websecure"
- "traefik.http.routers.portainer.rule=Host(`portainer.example.com`)"
- "traefik.http.routers.portainer.service=portainer"
- "traefik.http.services.portainer.loadbalancer.server.port=9000"
- "traefik.http.routers.portainer.tls.certresolver=production"
- "traefik.http.routers.portainer_edge.rule=Host(`edge.example.com`)"
- "traefik.http.routers.portainer_edge.entrypoints=websecure"
- "traefik.http.routers.portainer_edge.service=portainer_edge"
- "traefik.http.services.portainer_edge.loadbalancer.server.port=8000"
- "traefik.http.routers.portainer_edge.tls.certresolver=production"
networks:
default:
name: portainer
external: true
volumes:
portainer_data:
external: true
traefik:
external: true
Looking at the config, we can see that in the label sections, 2 domains are specified:
portainer.example.com
edge.example.com
Go to whichever DNS provider you use and create 2 new records.
Name the first one
portainer.<your domain>.com
Name the second one
edge.<your domain>.com
Once created, replace the example domains in the docker-compose.yml with your own.
Save the docker-compose.yml file and exit.
Before we launch the file, we need to create the volumes and network specified in the docker-compose.yml as well as verify that they’ve been created:
sudo docker network create portainer
Verify:
sudo docker network list
sudo docker volume create portainer_data
sudo docker volume create traefik
Verify:
sudo docker volume list
We also need to create a config file for Traefik.
Go to the Traefik volume we just created:
cd /var/lib/docker/volumes/traefik/_data/
Create a traefik.yml:
sudo nano traefik.yml
Paste the provided Traefik config:
global:
checkNewVersion: true
sendAnonymousUsage: false # true by default
# (Optional) Log information
# ---
log:
level: ERROR # DEBUG, INFO, WARNING, ERROR, CRITICAL
format: common # common, json, logfmt
filePath: /var/log/traefik/traefik.log
# (Optional) Accesslog
# ---
accesslog:
format: common # common, json, logfmt
filePath: /var/log/traefik/access.log
# (Optional) Enable API and Dashboard
# ---
api:
dashboard: true # true by default
insecure: true # Don't do this in production!
# Entry Points configuration
# ---
entryPoints:
web:
address: :80
# (Optional) Redirect to HTTPS
# ---
http:
redirections:
entryPoint:
to: websecure
scheme: https
websecure:
address: :443
# Configure your CertificateResolver here...
# ---
certificatesResolvers:
staging:
acme:
email: examplel@domain.com
storage: /etc/traefik/certs/acme-staging.json
caServer: "https://acme-staging-v02.api.letsencrypt.org/directory"
httpChallenge:
entryPoint: web
#
production:
acme:
email: example@domain.com
storage: /etc/traefik/certs/acme.json
caServer: "https://acme-v02.api.letsencrypt.org/directory"
httpChallenge:
entryPoint: web
# (Optional) Overwrite Default Certificates
# tls:
# stores:
# default:
# defaultCertificate:
# certFile: /etc/traefik/certs/cert.pem
# keyFile: /etc/traefik/certs/cert-key.pem
# (Optional) Disable TLS version 1.0 and 1.1
# options:
# default:
# minVersion: VersionTLS12
providers:
docker:
exposedByDefault: false # Default is true
file:
# watch for dynamic configuration changes
directory: /etc/traefik
watch: true
Scroll down until you find the certificatesResolvers section and change the 2 email fields to a working email that you have access to.
Save the file and exit.
Create a certs folder for traefik:
sudo mkdir /var/lib/docker/volumes/traefik/_data/certs
sudo chmod 600 /var/lib/docker/volumes/traefik/_data/certs
Navigate back to the directory where we created the docker-compose file and launch it:
sudo docker-compose up
Both Traefik and Portainer will start in the foreground.
Make sure both Traefik and Portainer start-up, then cancel with ctrl + c and instead run it in the background.
sudo docker-compose up -d
Step 3: Portainer UI
Reminder: Newer versions of Portainer (like 2.14) have made adjustments to the UI, so the screenshots may not match up perfectly.
Now that everything is running we can log into our Portainer UI.
Go to your Portainer URL:
portainer.<your domain>.com
Create a strong password, enter your license key and log in.
Click get started
Click on local environment
You’re now accessing the machine where Portainer/Traefik is installed.
Before we can start connecting to O/T machines we need to create some templates.
Click Templates and Custom Template
Click the Add Custom Template button.
The first template we’ll create will be for combined O/T’s.
Give the template a title and description and paste the example config in the Web editor section:
version: '3'
services:
combined-orchestrator:
image: livepeer/go-livepeer:0.5.29
hostname: "combined-orchestrator"
container_name: "livepeer-combined"
runtime: nvidia
restart: unless-stopped
ports:
- 8935:8935
volumes:
- lpdata:/root/.lpData
command: '-config /root/.lpData/lporch.conf'
volumes:
lpdata:
external: true
networks:
default:
name: livepeer
external: true
As you can see, we’ve specified a livepeer version, container name, volume, and network. You can simply change the livepeer version to install whichever one you’d like.
You can also change the container name but keep note of this as you’ll need to specify the container name in your livepeer/Prometheus config files in a later step.
Once you’re happy with the template, click Create custom template at the bottom of the page.
Let’s repeat these steps for a standalone Orchestrator node and Prometheus.
Simply go back to Custom Templates and create 2 more.
Standalone Orchestrator:
version: '3.8'
services:
combined-orchestrator:
image: livepeer/go-livepeer:0.5.32
hostname: "standalone-orchestrator"
container_name: "livepeer-standalone"
restart: unless-stopped
ports:
- 8935:8935
volumes:
- lpdata:/root/.lpData
command: '-config /root/.lpData/lporch.conf'
volumes:
lpdata:
external: true
networks:
default:
name: livepeer
external: true
Prometheus starter template:
services:
prometheus:
image: prom/prometheus:latest
container_name: monitoring_prometheus
restart: unless-stopped
volumes:
- prometheus:/prometheus
- prometheus:/etc/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus/data'
- '--storage.tsdb.retention.time=1y'
- '--web.console.libraries=/usr/share/prometheus/console_libraries'
- '--web.console.templates=/usr/share/prometheus/consoles'
ports:
- 9090:9090
node-exporter:
image: prom/node-exporter:latest
container_name: node-exporter
restart: unless-stopped
volumes:
- prometheus:/host/proc:ro
- prometheus:/host/sys:ro
- prometheus:/rootfs:ro
command:
- '--path.procfs=/host/proc'
- '--path.rootfs=/rootfs'
- '--path.sysfs=/host/sys'
- '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
volumes:
prometheus:
external: true
networks:
default:
name: livepeer
external: true
Once you have your templates all set up we can start connecting nodes and installing livepeer + Prometheus.
Step 4: Edge agents
The way we connect to our various servers is by adding edge agent environments.
Go to the Environments page and click Add environment:
Select Edge Agent, enter a name and click Add Environment again.
On the next screen you’ll see a Docker run command. In a general install, we’d just copy this to a server with Docker installed and run it as is, however, because we’re using Traefik as a proxy we need to make some modifications.
Don’t exit this page.
Copy the entire Docker run command and paste it in a text editor of your choice. We’re going to be replacing the EDGE_KEY .
Cut/copy the edge key. Do not cut/copy the = or \.
Go to https://www.base64decode.org/.
Click on Decode.
Paste the edge key you just copied into the top field and click Decode.
In the results section you should see something like this:
https://portainer.domain.com/%7Cportainer.<domain>.com:8000|d9:7c:00:99:3f:7a:5a:2f:c8:ef:4f:7e:94:00:b4:71|24
Let’s change a couple of values.
Replace the second portainer.<your domain>.com with https://edge.<your domain>.com and remove port 8000.
Copy the edited string which should now look like this:
https://portainer.<your domain>.com|https://edge.<your domain>.com|d9:7c:00:99:3f:7a:5a:2f:c8:ef:4f:7e:94:00:b4:71|32
Go to Encode, paste the edited string into the top field, make sure URL-safe encoding is checked and click Encode.
Copy the encoded result and paste it in-place of the edge key previously included in the Docker run command which we pasted into a text editor.
Before we can run the Docker command on our livepeer node, we need to prep it by installing Docker.
Access the livepeer server you’ll be linking to Portainer and install Docker the same way we did on the Portainer server. We don’t need to install docker-compose here.
Remove older version of docker:
sudo apt-get remove docker docker-engine docker.io containerd runc
Install the Docker repo for Ubuntu:
apt-get update
apt-get install ca-certificates curl wget net-tools net-tools sysstat gnupg lsb-release
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
Once the repo is installed… update the repo, install docker, enable docker to autostart on boot:
apt-get update
apt-get install docker-ce docker-ce-cli containerd.io -y
systemctl --now enable docker
If this is a combined O/T, we’ll need to install Nvidia drivers and patch them as well as installing Nvidia runtime for docker.
Driver Install:
apt-get install -y nvidia-driver-510
# Optional Step
systemctl set-default multi-user
Patch:
wget https://raw.githubusercontent.com/keylase/nvidia-patch/master/patch.sh
chmod a+x ./patch.sh
./patch.sh
Disable NVIDIA auto updates:
apt-mark hold nvidia-driver-510
apt-mark hold nvidia-cuda-dev
Install NVIDIA Container Runtime:
distribution=$(. /etc/os-release;echo $ID$VERSION_ID) && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
apt-get update
apt-get install -y nvidia-docker2
systemctl restart docker
docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi
verfy:
docker -v
- output Docker version 20.10.14, build a224086
docker info
- Look for the Runtimes section and ensure you see the nvidia entry listed
- Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux nvidia .
nvidia-smi
The NVIDIA-SMI output should contain:
- NVIDIA-SMI 510.60.02
- Driver Version: 510.60.02
- CUDA Version: 11.6
docker run --gpus all nvidia/cuda:11.0-base
- should have the same output as nvidia-smi
- if this fails, please review your NVIDIA / Docker setup.
Once you’ve confirmed that everything is installed correctly you can paste and run the entire docker run command (the one we edited and have ready in a text editor).
Now, back to where we left off in the Portainer UI.
In the Configuration section, enter a name and the public IP or domain of your livepeer server and click Update environment. The Update environment button may be called something else in newer versions of Portainer. Look for the blue button 
Note: Adding your public IP or domain is optional and the edge agent environment should connect without it.
In Portainer 2.14+, you won’t see the Public IP field as shown in the screenshot. In order to add a Public IP/Domain, you’ll need to navigate to Environments (left-hand side menu) after you deploy it and click on the new environment.
Congrats, you just added a livepeer node to Portainer!
Now click on Home and select your new edge environment:
It may error out a couple of times before it connects. If you cannot connect at all, drop a comment below and we’ll troubleshoot.
Once connected we need to add a network and volume to the environment.
Click Networks, Add network, enter “livepeer” as the network name and click Create the network.
This is exactly the same as what we did on the Portainer server, only now we don’t have to touch the CLI.
Click Volumes and add 2 new volumes. Name 1 lpdata and the second one prometheus:
The lpdata volume is where you upload your keystore, password file (if using one) and your livepeer config.
Want to learn how to secure your Orchestrator keystore? There’s a guide for that
Note: The CliAddr flag value needs to be replaced with <container name>:7935.
Also, add these flags which point to the new volume:
ethKeystorePath /root/.lpData/
datadir /root/.lpData/
ethPassword /root/.lpData/password.txt
If you changed the volume location in the template, make sure the paths match.
The prometheus volume is where you upload your prometheus.yml and data folder.
Note: The same rule applies here. Instead of IP addresses for you target fields of your prometheus.yml, enter the container name followed by the port.
Step 5: deployment
Click Stacks, Add stack, Custom template and choose your livepeer template:
In the Web editor you can make adjustments that won’t effect your main template. For example, you can change the livepeer version or container name. Just remember, whatever you change the container name to needs to be mirrored in your livepeer.conf and prometheus.yml.
Once you’ve made your changes click Deploy the stack, then click Stacks and click on your newly deployed stack:
You should now see the container up and marked as running. Click the logs icon to get a live view:
Any errors so far? Leave a comment.
At this point we can simply repeat the deployment steps for Prometheus.
If you already have a data folder for Prometheus, go to the livepeer server and move the data folder to the Prometheus volume we created earlier.
sudo mv <path to existing data folder> /var/lib/docker/volumes/prometheus/_data/
If you don’t have an existing data folder, simply go back to Portainer, click on volumes, then click the Prometheus volume and inside create a directory and call it data.
Regardless of which option you choose, you need to go back to the livepeer server and invoke the necessary permissions so Prometheus can read and write from/to the data folder:
sudo chmod 777 -Rf <path to data folder>
If you didn’t make any custom modifications to the volume paths, the command will look like this:
sudo chmod 777 -Rf /var/lib/docker/volumes/prometheus/_data/data/*
You can now start your Prometheus stack/container. Don’t forget that you can view container logs at any time via the Portainer UI. If you’d like to view the logs from your terminal you can do so by typing:
sudo docker logs <container name> -f -n 10
Aaaaand we’re done!
Want to add more servers? Just repeat from step 4.
That just about wraps up the guide. I know it was a long one, but this is a great method for those who are looking for alternative ways to run their nodes in production.
This was really just a basic overview of how to get started with a livepeer + Docker + Portainer combo. The configuration options are pretty much endless and I’m still learning what all the buttons do myself. If you find more interesting ways to make use of Portainer’s functionality for livepeer, please add them to this thread.
