Hey folks!
I wanted to quickly put this Livepeer-specific guide together on using systemd to manage your Livepeer services.
Community Contributions
- @Strykar for additions to security, systemd usage, and resources
Prerequisites
-
You’ve followed @vires-in-numeris ’ original monitoring guide: Guide: Transcoder monitoring with Prometheus/Grafana
-
You’re running some flavor of Linux
Resources
- systemd for Administrators, Part IX (compliments of @Strykar )
- systemd service sandboxign and security hardening 101 (compliments of @Strykar )
- Strykar’s ArchLinux Package (compliments of @Strykar )
What to expect
Step-by-step guide on moving your existing services to be managed by systemd. At the end of this tutorial, you’ll be able to start/stop/restart all of your Livepeer-dependent services with a single line.
What not to expect
“Fluff” explaining how systemd works and why certain configurations are chosen. This is left up to the reader.
"not a financial advisor"-esque Disclaimer
The contents of this guide are given as-is. The reader is responsible for validating security and any configurations that best suit them.
Guide:
- Gather some info
Before continuing, write down the following user-specific information: path to prometheus, path to nvidia_exports.go, current user (runwhoami), current group (runid -gn) - Create a Livepeer script named
livepeer.sh
This script will contain the livepeer command you usually use to run your transcoder/orchestrator. An example below:
#! /bin/bash
ETH_URL="https://mainnet.infura.io/v3/..."
ETH_ACCT_ADDR=""
SERVICE_ADDR="ORCHESTRATOR_ADDRESS:8935"
PRICE_PER_UNIT=900
MAX_SESSIONS=60
/usr/local/bin/livepeer \
-network mainnet \
-ethUrl $ETH_URL \
-ethAcctAddr $ETH_ACCT_ADDR \
-ethPassword /PATH_TO_ETH_PASSWORD/eth.pwd \
-orchestrator \
-transcoder \
-pricePerUnit $PRICE_PER_UNIT \
-serviceAddr $SERVICE_ADDR \
-monitor \
-reward=false \
-autoAdjustPrice=true \
-maxSessions=$MAX_SESSIONS
- Create Livepeer service
systemctl edit livepeer.servicewith the following contents (be sure to update YOUR_USER, YOUR_GROUP, and PATH_TO_LIVEPEER_SCRIPT accordingly)
[Unit]
Description="Livepeer transcoder service."
After=network.target
PartOf=transcoder.target
[Service]
User=YOUR_USER
Group=YOUR_GROUP
Environment=LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8
ExecStart=/PATH_TO_LIVEPEER_SCRIPT/livepeer.sh
Restart=on-failure
RestartSec=5s
ProtectHome=yes
ProtectClock=yes
PrivateDevices=yes
ProtectHostname=yes
NoNewPrivileges=yes
ProtectSystem=strict
RestrictRealtime=yes
RestrictSUIDSGID=yes
ProtectKernelLogs=yes
RestrictNamespaces=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ProtectKernelTunables=yes
SystemCallErrorNumber=EPERM
SystemCallArchitectures=native
SystemCallFilter=@system-service
[Install]
WantedBy=multi-user.target
- Create Prometheus service
systemctl edit prometheus.servicewith the following contents (be sure to update YOUR_USER, YOUR_GROUP, and PATH_TO_PROMETHEUS_DIRECTORY accordingly)
[Unit]
Description="Prometheus service used to collect GPU, Livepeer, and System metrics."
After=network.target
PartOf=transcoder.target
[Service]
User=YOUR_USER
Group=YOUR_GROUP
Environment=LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8
WorkingDirectory=/PATH_TO_PROMETHEUS_DIRECTORY/prometheus/
ExecStart=/PATH_TO_PROMETHEUS_DIRECTORY/prometheus/prometheus
Restart=on-failure
RestartSec=5s
ProtectHome=yes
ProtectClock=yes
PrivateDevices=yes
ProtectHostname=yes
NoNewPrivileges=yes
ProtectSystem=strict
RestrictRealtime=yes
RestrictSUIDSGID=yes
ProtectKernelLogs=yes
RestrictNamespaces=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ProtectKernelTunables=yes
SystemCallErrorNumber=EPERM
SystemCallArchitectures=native
SystemCallFilter=@system-service
[Install]
WantedBy=multi-user.target
- Create Nvidia Exporter service
systemctl edit nvidia-exporter.servicewith the following contents (be sure to update YOUR_USER, YOUR_GROUP, and PATH_TO_NVIDIA_EXPORTER_DIRECTORY accordingly)
[Unit]
Description="Go program that ships Nvidia metrics to prometheus."
After=network.target
PartOf=transcoder.target
[Service]
User=YOUR_USER
Group=YOUR_GROUP
Environment=LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8
ExecStart=/usr/bin/go run /PATH_TO_NVIDIA_EXPORTS_DIRECTORY/nvidia_exports.go
Restart=on-failure
RestartSec=5s
ProtectHome=yes
ProtectClock=yes
PrivateDevices=yes
ProtectHostname=yes
NoNewPrivileges=yes
ProtectSystem=strict
RestrictRealtime=yes
RestrictSUIDSGID=yes
ProtectKernelLogs=yes
RestrictNamespaces=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ProtectKernelTunables=yes
SystemCallErrorNumber=EPERM
SystemCallArchitectures=native
SystemCallFilter=@system-service
[Install]
WantedBy=multi-user.target
- Create Transcoder target
systemctl edit transcoder.targetwith the following contents. This is responsible for grouping all of our services together for management through a single entry point.
[Unit]
After=network.target
Wants=livepeer.service nvidia-metrics.service prometheus.service grafana-server.service
[Install]
WantedBy=multi-user.target
- Start your services!
sudo systemctl start transcoder.targetThis will start Livepeer, Nvidia Exporter, Prometheus, and Grafana simultaneously.
Using systemd
-
Start an individual service:
sudo systemctl start livepeer.service -
Stop an individual service:
sudo systemctl stop livepeer.service -
Restart an individual service:
sudo systemctl restart livepeer.service -
Status of an individual service:
sudo systemctl status livepeer.service -
Start all services:
sudo systemctl start transcoder.target -
Stop all services:
sudo systemctl stop transcoder.target -
Restart all services:
sudo systemctl restart transcoder.target -
List all services in target and their status:
sudo systemctl list-dependencies transcoder.target -
View logs of an individual service:
journalctl -u livepeer.service -
Tail logs of an individual service:
journalctl -u livepeer.service -f
Optional Next steps
- Create a Livepeer user/group
Create a livepeer user/group that is used to run each service (replace YOUR_USER and YOUR_GROUP with “livepeer”)
sudo useradd --no-create-home --shell /bin/false livepeer
You can really go as far as you want with this. I use a livepeer user and store all service-related scripts in that user’s home directory. Obviously, you’d remove --no-create-home from the above command if you’d like. However, this is beyond the scope of the guide.
- Update the configuration of your systemd services!
For more information on what you can change, see the systemd docs systemd
