Guide: Livepeer with Docker and Portainer for operating at scale

Who is this guide for?

This guide is for Orchestrators who are looking to dramatically improve operational efficiency, specifically those running a multi-node setup.

Massive thanks to @MikeZupper for spending countless hours sharing his knowledge with the community. This guide is thanks to him as well as @papa_bear who helped document and test everything.

This guide will use Ubuntu and Portainer version 2.13 but steps will be similar on other operating systems and versions of Portainer.

A few benefits:

  • A central management point for all livepeer nodes.

  • Tons of flexibility for scaling and custom-tailoring your operation to fit your needs.

  • Quick livepeer updates/deployment.

  • Although this guide will only cover the basics, advanced configurations of this method can allow for the complete disregard of SSH.

Please feel free to provide critique/feedback as I’m sure there are many different ways to get everything up and running.

You’ll need:

  • At-least 2 machines. One being your O/T and another for Portainer/Traefik. A $5/mo shared CPU server from Linode is perfectly acceptable.

  • Basic knowledge of Docker and livepeer.

  • A Portainer business license. Sign up here for 5 free nodes.

  • DNS (need to be able to create some sub-domains).

  • A stress ball or punching bag.

  • A general understanding of proxies/Traefik is recommended but not required.

Step 1: Installing Docker and Docker Compose

On the Portainer server, install Docker. We’ll be using the steps listed in Mike Zuppers Docker install guide, however, if you run into any issues you can also use Docker’s official installation steps.

Remove older version of docker:

sudo apt-get remove docker docker-engine docker.io containerd runc

Install the Docker repo for Ubuntu:

apt-get update

apt-get install ca-certificates curl wget net-tools net-tools sysstat gnupg lsb-release

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Once the repo is installed… update the repo, install docker, enable docker to autostart on boot:

apt-get update

apt-get install docker-ce docker-ce-cli containerd.io -y

systemctl --now enable docker

Install Docker Compose and invoke necessary permissions:

sudo curl -L https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m) -o /usr/bin/docker-compose
sudo chmod 755 /usr/bin/docker-compose

Step 2: Portainer and Traefik

Create a docker-compose.yml

sudo nano docker-compose.yml

Paste the provided traefik + portainer stack

version: '3'


services:
  traefik:
    image: "traefik:latest"
    container_name: "traefik"
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      # (Optional) Expose Dashboard
      #- "8080:8080"  # Don't do this in production!
    volumes:
      - traefik:/etc/traefik
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - traefik:/var/log/traefik/

  portainer:
    image: portainer/portainer-ee:latest
    container_name: portainer
    restart: unless-stopped
    depends_on:
      - traefik
    #security_opt:
     # - no-new-privileges:true
   # ports:
     # - 8000:8000
     # - 9000:9000
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - portainer_data:/data:rw
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.portainer.entrypoints=websecure"
      - "traefik.http.routers.portainer.rule=Host(`portainer.example.com`)"
      - "traefik.http.routers.portainer.service=portainer"
      - "traefik.http.services.portainer.loadbalancer.server.port=9000"
      - "traefik.http.routers.portainer.tls.certresolver=production"


      - "traefik.http.routers.portainer_edge.rule=Host(`edge.example.com`)"
      - "traefik.http.routers.portainer_edge.entrypoints=websecure"
      - "traefik.http.routers.portainer_edge.service=portainer_edge"
      - "traefik.http.services.portainer_edge.loadbalancer.server.port=8000"
      - "traefik.http.routers.portainer_edge.tls.certresolver=production"

networks:
  default:
    name: portainer
    external: true

volumes:
  portainer_data:
    external: true
  traefik:
    external: true

Looking at the config, we can see that in the label sections, 2 domains are specified:

portainer.example.com
edge.example.com

Go to whichever DNS provider you use and create 2 new records.
Name the first one
portainer.<your domain>.com
Name the second one
edge.<your domain>.com

Once created, replace the example domains in the docker-compose.yml with your own.

Save the docker-compose.yml file and exit.

Before we launch the file, we need to create the volumes and network specified in the docker-compose.yml as well as verify that they’ve been created:

sudo docker network create portainer 

Verify:

sudo docker network list
sudo docker volume create portainer_data
sudo docker volume create traefik

Verify:

sudo docker volume list

We also need to create a config file for Traefik.

Go to the Traefik volume we just created:

cd /var/lib/docker/volumes/traefik/_data/

Create a traefik.yml:

sudo nano traefik.yml

Paste the provided Traefik config:

global:
  checkNewVersion: true
  sendAnonymousUsage: false  # true by default

# (Optional) Log information
# ---
log:
  level: ERROR  # DEBUG, INFO, WARNING, ERROR, CRITICAL
  format: common  # common, json, logfmt
  filePath: /var/log/traefik/traefik.log

# (Optional) Accesslog
# ---
accesslog:
  format: common  # common, json, logfmt
  filePath: /var/log/traefik/access.log

# (Optional) Enable API and Dashboard
# ---
api:
  dashboard: true  # true by default
  insecure: true  # Don't do this in production!

# Entry Points configuration
# ---
entryPoints:
  web:
    address: :80
    # (Optional) Redirect to HTTPS
    # ---
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https

  websecure:
    address: :443

# Configure your CertificateResolver here...
# ---
certificatesResolvers:
   staging:
     acme:
       email: examplel@domain.com
       storage: /etc/traefik/certs/acme-staging.json
       caServer: "https://acme-staging-v02.api.letsencrypt.org/directory"
       httpChallenge:
         entryPoint: web
#
   production:
     acme:
       email: example@domain.com
       storage: /etc/traefik/certs/acme.json
       caServer: "https://acme-v02.api.letsencrypt.org/directory"
       httpChallenge:
         entryPoint: web

# (Optional) Overwrite Default Certificates
#   tls:
#     stores:
#       default:
#         defaultCertificate:
#           certFile: /etc/traefik/certs/cert.pem
#           keyFile: /etc/traefik/certs/cert-key.pem
# (Optional) Disable TLS version 1.0 and 1.1
#   options:
#      default:
#         minVersion: VersionTLS12

providers:
  docker:
    exposedByDefault: false  # Default is true
  file:
    # watch for dynamic configuration changes
    directory: /etc/traefik
    watch: true

Scroll down until you find the certificatesResolvers section and change the 2 email fields to a working email that you have access to.

Save the file and exit.

Create a certs folder for traefik:

sudo mkdir /var/lib/docker/volumes/traefik/_data/certs
sudo chmod 600 /var/lib/docker/volumes/traefik/_data/certs

Navigate back to the directory where we created the docker-compose file and launch it:

sudo docker-compose up

Both Traefik and Portainer will start in the foreground.
Make sure both Traefik and Portainer start-up, then cancel with ctrl + c and instead run it in the background.

sudo docker-compose up -d

Step 3: Portainer UI

Reminder: Newer versions of Portainer (like 2.14) have made adjustments to the UI, so the screenshots may not match up perfectly.

Now that everything is running we can log into our Portainer UI.
Go to your Portainer URL:

portainer.<your domain>.com

Create a strong password, enter your license key and log in.

Click get started

Click on local environment

You’re now accessing the machine where Portainer/Traefik is installed.

Before we can start connecting to O/T machines we need to create some templates.
Click Templates and Custom Template

Click the Add Custom Template button.

The first template we’ll create will be for combined O/T’s.
Give the template a title and description and paste the example config in the Web editor section:

version: '3'

services:
  combined-orchestrator:
    image: livepeer/go-livepeer:0.5.29
    hostname: "combined-orchestrator"
    container_name: "livepeer-combined"
    runtime: nvidia
    restart: unless-stopped
    ports:
       - 8935:8935
    volumes:
      - lpdata:/root/.lpData
    command: '-config /root/.lpData/lporch.conf'
volumes:
  lpdata:
    external: true
networks:
  default:
    name: livepeer
    external: true

As you can see, we’ve specified a livepeer version, container name, volume, and network. You can simply change the livepeer version to install whichever one you’d like.

You can also change the container name but keep note of this as you’ll need to specify the container name in your livepeer/Prometheus config files in a later step.

Once you’re happy with the template, click Create custom template at the bottom of the page.
Let’s repeat these steps for a standalone Orchestrator node and Prometheus.

Simply go back to Custom Templates and create 2 more.

Standalone Orchestrator:

version: '3.8'

services:
  combined-orchestrator:
    image: livepeer/go-livepeer:0.5.32
    hostname: "standalone-orchestrator"
    container_name: "livepeer-standalone"
    restart: unless-stopped
    ports:

      - 8935:8935
    volumes:
      - lpdata:/root/.lpData
    command: '-config /root/.lpData/lporch.conf'
    
volumes:
  lpdata:
    external: true
networks:
  default:
    name: livepeer
    external: true

Prometheus starter template:

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: monitoring_prometheus

    restart: unless-stopped
    volumes:
      - prometheus:/prometheus
      - prometheus:/etc/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml' 
      - '--storage.tsdb.path=/prometheus/data'
      - '--storage.tsdb.retention.time=1y'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    ports:
     - 9090:9090

      
  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    volumes:
      - prometheus:/host/proc:ro
      - prometheus:/host/sys:ro
      - prometheus:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
      
volumes:
  prometheus:
    external: true
networks:
  default:
    name: livepeer
    external: true

Once you have your templates all set up we can start connecting nodes and installing livepeer + Prometheus.

Step 4: Edge agents

The way we connect to our various servers is by adding edge agent environments.

Go to the Environments page and click Add environment:

Select Edge Agent, enter a name and click Add Environment again.

On the next screen you’ll see a Docker run command. In a general install, we’d just copy this to a server with Docker installed and run it as is, however, because we’re using Traefik as a proxy we need to make some modifications.

Don’t exit this page.

Copy the entire Docker run command and paste it in a text editor of your choice. We’re going to be replacing the EDGE_KEY .

Cut/copy the edge key. Do not cut/copy the = or \.

Go to https://www.base64decode.org/.
Click on Decode.
Paste the edge key you just copied into the top field and click Decode.
In the results section you should see something like this:

https://portainer.domain.com/%7Cportainer.<domain>.com:8000|d9:7c:00:99:3f:7a:5a:2f:c8:ef:4f:7e:94:00:b4:71|24

Let’s change a couple of values.
Replace the second portainer.<your domain>.com with https://edge.<your domain>.com and remove port 8000.

Copy the edited string which should now look like this:

https://portainer.<your domain>.com|https://edge.<your domain>.com|d9:7c:00:99:3f:7a:5a:2f:c8:ef:4f:7e:94:00:b4:71|32

Go to Encode, paste the edited string into the top field, make sure URL-safe encoding is checked and click Encode.

Copy the encoded result and paste it in-place of the edge key previously included in the Docker run command which we pasted into a text editor.

Before we can run the Docker command on our livepeer node, we need to prep it by installing Docker.

Access the livepeer server you’ll be linking to Portainer and install Docker the same way we did on the Portainer server. We don’t need to install docker-compose here.

Remove older version of docker:

sudo apt-get remove docker docker-engine docker.io containerd runc

Install the Docker repo for Ubuntu:

apt-get update

apt-get install ca-certificates curl wget net-tools net-tools sysstat gnupg lsb-release

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Once the repo is installed… update the repo, install docker, enable docker to autostart on boot:

apt-get update

apt-get install docker-ce docker-ce-cli containerd.io -y

systemctl --now enable docker

If this is a combined O/T, we’ll need to install Nvidia drivers and patch them as well as installing Nvidia runtime for docker.

Driver Install:

apt-get install -y nvidia-driver-510

# Optional Step
systemctl set-default multi-user

Patch:

wget https://raw.githubusercontent.com/keylase/nvidia-patch/master/patch.sh
chmod a+x ./patch.sh
./patch.sh

Disable NVIDIA auto updates:

apt-mark hold nvidia-driver-510      
apt-mark hold nvidia-cuda-dev

Install NVIDIA Container Runtime:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)       && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg       && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list |             sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' |             sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
apt-get update
apt-get install -y nvidia-docker2
systemctl restart docker
docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi

verfy:

docker -v

  • output Docker version 20.10.14, build a224086

docker info

  • Look for the Runtimes section and ensure you see the nvidia entry listed
  • Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux nvidia .

nvidia-smi

The NVIDIA-SMI output should contain:

  • NVIDIA-SMI 510.60.02
  • Driver Version: 510.60.02
  • CUDA Version: 11.6

docker run --gpus all nvidia/cuda:11.0-base

  • should have the same output as nvidia-smi
  • if this fails, please review your NVIDIA / Docker setup.

Once you’ve confirmed that everything is installed correctly you can paste and run the entire docker run command (the one we edited and have ready in a text editor).

Now, back to where we left off in the Portainer UI.

In the Configuration section, enter a name and the public IP or domain of your livepeer server and click Update environment. The Update environment button may be called something else in newer versions of Portainer. Look for the blue button :wink:

Note: Adding your public IP or domain is optional and the edge agent environment should connect without it.

In Portainer 2.14+, you won’t see the Public IP field as shown in the screenshot. In order to add a Public IP/Domain, you’ll need to navigate to Environments (left-hand side menu) after you deploy it and click on the new environment.

portainer_edge5

Congrats, you just added a livepeer node to Portainer!

Now click on Home and select your new edge environment:

It may error out a couple of times before it connects. If you cannot connect at all, drop a comment below and we’ll troubleshoot.

Once connected we need to add a network and volume to the environment.

Click Networks, Add network, enter “livepeer” as the network name and click Create the network.
This is exactly the same as what we did on the Portainer server, only now we don’t have to touch the CLI.

Click Volumes and add 2 new volumes. Name 1 lpdata and the second one prometheus:

portainer_volumes

The lpdata volume is where you upload your keystore, password file (if using one) and your livepeer config.

Want to learn how to secure your Orchestrator keystore? There’s a guide for that :wink:

Note: The CliAddr flag value needs to be replaced with <container name>:7935.
Also, add these flags which point to the new volume:

ethKeystorePath /root/.lpData/
datadir /root/.lpData/
ethPassword /root/.lpData/password.txt

If you changed the volume location in the template, make sure the paths match.

The prometheus volume is where you upload your prometheus.yml and data folder.

Note: The same rule applies here. Instead of IP addresses for you target fields of your prometheus.yml, enter the container name followed by the port.

Step 5: deployment

Click Stacks, Add stack, Custom template and choose your livepeer template:

In the Web editor you can make adjustments that won’t effect your main template. For example, you can change the livepeer version or container name. Just remember, whatever you change the container name to needs to be mirrored in your livepeer.conf and prometheus.yml.

Once you’ve made your changes click Deploy the stack, then click Stacks and click on your newly deployed stack:

You should now see the container up and marked as running. Click the logs icon to get a live view:

portainer_logs

Any errors so far? Leave a comment.

At this point we can simply repeat the deployment steps for Prometheus.

If you already have a data folder for Prometheus, go to the livepeer server and move the data folder to the Prometheus volume we created earlier.

sudo mv <path to existing data folder> /var/lib/docker/volumes/prometheus/_data/

If you don’t have an existing data folder, simply go back to Portainer, click on volumes, then click the Prometheus volume and inside create a directory and call it data.

Regardless of which option you choose, you need to go back to the livepeer server and invoke the necessary permissions so Prometheus can read and write from/to the data folder:

sudo chmod 777 -Rf <path to data folder> 

If you didn’t make any custom modifications to the volume paths, the command will look like this:

sudo chmod 777 -Rf /var/lib/docker/volumes/prometheus/_data/data/*

You can now start your Prometheus stack/container. Don’t forget that you can view container logs at any time via the Portainer UI. If you’d like to view the logs from your terminal you can do so by typing:

sudo docker logs <container name> -f -n 10

Aaaaand we’re done!

Want to add more servers? Just repeat from step 4.

That just about wraps up the guide. I know it was a long one, but this is a great method for those who are looking for alternative ways to run their nodes in production.

This was really just a basic overview of how to get started with a livepeer + Docker + Portainer combo. The configuration options are pretty much endless and I’m still learning what all the buttons do myself. If you find more interesting ways to make use of Portainer’s functionality for livepeer, please add them to this thread.

5 Likes

This is awesome great write up! :clap:

1 Like

Great job ! Thanks a lot :pray:

1 Like

Thanks @Pon for running through the guide. I’ve updated a few sections but can confirm it works :slight_smile:

Cant wait to implement this in the future

1 Like

It’s important to note that this guide was written for Portainer 2.13. If you’re using a newer version, the UI may look different but steps should be similar.

*I’ve added this to the guide.

Complete video walkthrough now available: