Guide: Using systemd to manage your Livepeer services

payton · December 20, 2021, 6:29pm

Hey folks!

I wanted to quickly put this Livepeer-specific guide together on using systemd to manage your Livepeer services.

Community Contributions

@Strykar for additions to security, systemd usage, and resources

Prerequisites

You’ve followed @vires-in-numeris ’ original monitoring guide: Guide: Transcoder monitoring with Prometheus/Grafana
You’re running some flavor of Linux

Resources

systemd for Administrators, Part IX (compliments of @Strykar )
systemd service sandboxign and security hardening 101 (compliments of @Strykar )
Strykar’s ArchLinux Package (compliments of @Strykar )

What to expect
Step-by-step guide on moving your existing services to be managed by systemd. At the end of this tutorial, you’ll be able to start/stop/restart all of your Livepeer-dependent services with a single line.

What not to expect
“Fluff” explaining how systemd works and why certain configurations are chosen. This is left up to the reader.

"not a financial advisor"-esque Disclaimer
The contents of this guide are given as-is. The reader is responsible for validating security and any configurations that best suit them.

Guide:

Gather some info
Before continuing, write down the following user-specific information: path to prometheus, path to nvidia_exports.go, current user (run whoami), current group (run id -gn)
Create a Livepeer script named livepeer.sh
This script will contain the livepeer command you usually use to run your transcoder/orchestrator. An example below:

#! /bin/bash

ETH_URL="https://mainnet.infura.io/v3/..."
ETH_ACCT_ADDR=""
SERVICE_ADDR="ORCHESTRATOR_ADDRESS:8935"
PRICE_PER_UNIT=900
MAX_SESSIONS=60

/usr/local/bin/livepeer \
    -network mainnet \
    -ethUrl $ETH_URL \
    -ethAcctAddr $ETH_ACCT_ADDR \
    -ethPassword /PATH_TO_ETH_PASSWORD/eth.pwd \
    -orchestrator \
    -transcoder \
    -pricePerUnit $PRICE_PER_UNIT \
    -serviceAddr $SERVICE_ADDR \
    -monitor \
    -reward=false \
    -autoAdjustPrice=true \
    -maxSessions=$MAX_SESSIONS

Create Livepeer service systemctl edit livepeer.service with the following contents (be sure to update YOUR_USER, YOUR_GROUP, and PATH_TO_LIVEPEER_SCRIPT accordingly)

[Unit]
Description="Livepeer transcoder service."
After=network.target
PartOf=transcoder.target

[Service]
User=YOUR_USER
Group=YOUR_GROUP
Environment=LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8
ExecStart=/PATH_TO_LIVEPEER_SCRIPT/livepeer.sh
Restart=on-failure
RestartSec=5s
ProtectHome=yes
ProtectClock=yes
PrivateDevices=yes
ProtectHostname=yes
NoNewPrivileges=yes
ProtectSystem=strict
RestrictRealtime=yes
RestrictSUIDSGID=yes
ProtectKernelLogs=yes
RestrictNamespaces=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ProtectKernelTunables=yes
SystemCallErrorNumber=EPERM
SystemCallArchitectures=native
SystemCallFilter=@system-service

[Install]
WantedBy=multi-user.target

Create Prometheus service systemctl edit prometheus.service with the following contents (be sure to update YOUR_USER, YOUR_GROUP, and PATH_TO_PROMETHEUS_DIRECTORY accordingly)

[Unit]
Description="Prometheus service used to collect GPU, Livepeer, and System metrics."
After=network.target
PartOf=transcoder.target

[Service]
User=YOUR_USER
Group=YOUR_GROUP
Environment=LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8
WorkingDirectory=/PATH_TO_PROMETHEUS_DIRECTORY/prometheus/
ExecStart=/PATH_TO_PROMETHEUS_DIRECTORY/prometheus/prometheus
Restart=on-failure
RestartSec=5s
ProtectHome=yes
ProtectClock=yes
PrivateDevices=yes
ProtectHostname=yes
NoNewPrivileges=yes
ProtectSystem=strict
RestrictRealtime=yes
RestrictSUIDSGID=yes
ProtectKernelLogs=yes
RestrictNamespaces=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ProtectKernelTunables=yes
SystemCallErrorNumber=EPERM
SystemCallArchitectures=native
SystemCallFilter=@system-service

[Install]
WantedBy=multi-user.target

Create Nvidia Exporter service systemctl edit nvidia-exporter.service with the following contents (be sure to update YOUR_USER, YOUR_GROUP, and PATH_TO_NVIDIA_EXPORTER_DIRECTORY accordingly)

[Unit]
Description="Go program that ships Nvidia metrics to prometheus."
After=network.target
PartOf=transcoder.target

[Service]
User=YOUR_USER
Group=YOUR_GROUP
Environment=LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8
ExecStart=/usr/bin/go run /PATH_TO_NVIDIA_EXPORTS_DIRECTORY/nvidia_exports.go
Restart=on-failure
RestartSec=5s
ProtectHome=yes
ProtectClock=yes
PrivateDevices=yes
ProtectHostname=yes
NoNewPrivileges=yes
ProtectSystem=strict
RestrictRealtime=yes
RestrictSUIDSGID=yes
ProtectKernelLogs=yes
RestrictNamespaces=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ProtectKernelTunables=yes
SystemCallErrorNumber=EPERM
SystemCallArchitectures=native
SystemCallFilter=@system-service

[Install]
WantedBy=multi-user.target

Create Transcoder target systemctl edit transcoder.target with the following contents. This is responsible for grouping all of our services together for management through a single entry point.

[Unit]
After=network.target
Wants=livepeer.service nvidia-metrics.service prometheus.service grafana-server.service

[Install]
WantedBy=multi-user.target

Start your services! sudo systemctl start transcoder.target This will start Livepeer, Nvidia Exporter, Prometheus, and Grafana simultaneously.

Using systemd

Start an individual service: sudo systemctl start livepeer.service
Stop an individual service: sudo systemctl stop livepeer.service
Restart an individual service: sudo systemctl restart livepeer.service
Status of an individual service: sudo systemctl status livepeer.service
Start all services: sudo systemctl start transcoder.target
Stop all services: sudo systemctl stop transcoder.target
Restart all services: sudo systemctl restart transcoder.target
List all services in target and their status: sudo systemctl list-dependencies transcoder.target
View logs of an individual service: journalctl -u livepeer.service
Tail logs of an individual service: journalctl -u livepeer.service -f

Optional Next steps

Create a Livepeer user/group

Create a livepeer user/group that is used to run each service (replace YOUR_USER and YOUR_GROUP with “livepeer”)

sudo useradd --no-create-home --shell /bin/false livepeer

You can really go as far as you want with this. I use a livepeer user and store all service-related scripts in that user’s home directory. Obviously, you’d remove --no-create-home from the above command if you’d like. However, this is beyond the scope of the guide.

Update the configuration of your systemd services!

For more information on what you can change, see the systemd docs systemd

Strykar · December 20, 2021, 8:36pm

Good stuff, here is some feedback, none of which is criticism, merely suggestions to do things the Linux way instead.

Most of which comprises ~30 years of experience condensed into things like the Debian policy manual and other distro’s packaging guidelines. See systemd for Administrators, Part IX

Systemd units that reside in /etc/systemd/system/ or /usr/lib/systemd/system/ should never be edited. This is why systemd’s systemctl edit livepeer.service feature creates a drop-in file with changes the administrator makes.
This reduces package maintainer burden and eases system maintenance when upgrading the package.

Inserting variables into the systemd unit file forces the user to edit them, instead, it’s recommended to flesh them out into an external config file using systemd’s built-in $ARGS variable.

Even better, is if the binary itself is capable of reading from a config file, usually residing in something like /etc/livepeer/livepeer.conf which should be available in the next livepeer release.
This reduces clutter in the systemd unit, since ExecStart can now just be =/usr/bin/livepeer --config /etc/livepeer.conf

The proper solution is to create a package for the distros used by O’s, mostly Ubuntu/Debian IIRC.
This takes care of creating the appropriate users / groups, prevents UID / GID mismatches and generally eases adoption.

I have created a package for Arch Linux, which I’m aware is not the distro of choice for most users, and I was going to create packages for Debian / Ubuntu but since @Thulinma mentioned Mist’s entire team uses Arch linux for development, I suggested someone from his team take over the Arch package and also create packages for other distros like Ubuntu, which he seemed ok with so hopefully we can see signed package for Ubuntu soon™.

Last, but definitely not the least, is improving security, see systemd service sandboxing and security hardening 101. TLDR; don’t introduce user editable variables and you at least want these security restrictions in your systemd unit:

ProtectHome=yes
ProtectClock=yes
PrivateDevices=yes
ProtectHostname=yes
NoNewPrivileges=yes
ProtectSystem=strict
RestrictRealtime=yes
RestrictSUIDSGID=yes
ProtectKernelLogs=yes
RestrictNamespaces=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ProtectKernelTunables=yes
SystemCallErrorNumber=EPERM
ReadOnlyPaths=/etc/go-livepeer
SystemCallArchitectures=native
SystemCallFilter=@system-service
ReadWritePaths=/var/cache/livepeer /var/lib/livepeer

My 2c

payton · December 20, 2021, 9:25pm

Thank you! All super valuable feedback and I appreciate you taking the time to compose a response. I’m definitely interested in modifying this guide as more expertise is added.

Adding both of these to a resources section.

Love it. The intent was to make this guide very bare bones but these are good call outs. Added.

Indeed - I’m looking forward to that. For those who are reading, this is the change we are referring to cmd: Add config file support (#2137) · livepeer/go-livepeer@832166f · GitHub

Ah, thanks for the link. I was trying to find this actually to include in the next steps section!

Again, thank you!

Topic		Replies	Views
Running Livepeer as a systemd service, almost working script, need to figure out one thing	0	816	August 4, 2018
Guide: Livepeer with Docker and Portainer for operating at scale Transcoders	6	1872	August 12, 2022
Setting up a Livepeer Transcoder Node (Beginner's Guide) Transcoders	16	13995	May 27, 2021
Transcoder Tip - Geth Light Client Transcoders	9	2893	October 23, 2018
Setup Livepeer Node on top of ETH-Miner (NVIDIA) using Docker? Help: Getting Started	2	1458	September 16, 2019

Guide: Using systemd to manage your Livepeer services

Related Topics