Enabling Fail2ban for Livepeer

Fail2ban scans log files (e.g. /var/log/apache/error_log) and bans IPs that show the malicious signs – too many password failures, seeking for exploits, etc. Generally Fail2Ban is then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action (e.g. sending an email) could also be configured. Out of the box Fail2Ban comes with filters for various services (apache, courier, ssh, etc).

In this post we configure Fail2ban to find livepeer log lines that match Tue 2022-12-06 04:14:08 IST livepeer[390680]: 2022/12/06 04:14:08 http: TLS handshake error from 172.68.6.145:28576: EOF and ban the offending IPs 99.99% of which are bots running port scans which fail the TLS initiation and thus SSL posts an EOF error. We assume livepeer is running with verbosity set to 6, (-v 6).

There are a few files we need to create and populate:

$ cat /etc/fail2ban/jail.local 
[DEFAULT]
banaction = nftables
banaction_allports = nftables[type=allports]

$ cat /etc/fail2ban/fail2ban.local 
[Definition]
logtarget = /var/log/fail2ban.log

$ sudo touch /var/log/fail2ban.log

$ $ cat /etc/fail2ban/fail2ban.local 
[Definition]
logtarget = /var/log/fail2ban.log

Now we create the fail2ban jail configuration:

$ cat /etc/fail2ban/jail.d/livepeer.conf 
[livepeer]
enabled = true
filter = livepeer
failregex = ^\s*\S+ livepeer\[\d+\]: (?:\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} )?http: TLS handshake error from <ADDR>:\d+: EOF$
datepattern = ^{DATE}
port = 8935
protocol = all
action = nftables-multiport[name=LPTPortScan, port="8935,3000"]
backend = systemd[journalflags=1]
journalmatch = _SYSTEMD_CGROUP=/user.slice/user-1000.slice/user@1000.service/app.slice/livepeer.service + _COMM=livepeer
banTime  = 864000
findtime = 1800
maxRetry = 3
ignoreip  = 127.0.0.1/8 ::1 10.0.0.1/24 192.168.1.1/24 182.70.116.80/32 172.105.60.167/32 89.187.177.134/32 89.187.177.138/32 89.187.177.196/32 89.187.178.130/32 89.187.179.35/32 195.181.169.69/32 89.187.185.130/32 89.187.185.153/32 89.187.185.251/32 143.244.50.113/32 89.187.181.249/32 143.244.61.193/32 143.244.61.205/32 212.102.58.242/32 143.244.33.78/32 143.244.33.79/32 143.244.33.95/32 89.187.188.237/32 89.187.188.246/32 212.102.38.92/32 84.17.50.98/32 84.17.50.99/32 89.187.88.237/32 185.59.221.179/32 89.187.169.65/32 138.199.36.76/32 185.102.219.91/32 185.102.219.180/32 195.181.174.39/32 195.181.174.186/32 138.199.4.163/32

Now we create the fail2ban filter configuration:

$ cat /etc/fail2ban/filter.d/livepeer.conf 
# Fail2Ban filter for livepeer (https://github.com/livepeer/go-livepeer)
#
# Sample journalctl logs:
# Nov 30 21:36:39 graf livepeer[3014]: 2022/11/30 21:36:39 http: TLS handshake error from 162.158.78.45:47024: EOF
# Nov 30 21:36:39 graf.sun.in livepeer[3014]: 2022/11/30 21:36:39 http: TLS handshake error from 162.158.78.45:47024: EOF
# Fri 2022-12-02 01:34:52 IST livepeer[312413]: 2022/12/02 01:34:52 http: TLS handshake error from 162.158.162.235:45566: EOF
#
# NOTE: The regex below is ONLY intended to work with livepeer Orchestrator log verbosity set to 6 (-v 6).
# See https://docs.livepeer.org/video-miners/reference/configuration#onchain
#
# More information: https://github.com/fail2ban/fail2ban/issues/3423

[DEFAULT]

[Definition]

failregex = ^\s*\S+ livepeer\[\d+\]: (?:\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} )?http: TLS handshake error from <ADDR>:\d+: EOF$
datepattern = ^{DATE}
port = 8935
protocol = all
# for log monitoring:
#backend = auto
#logpath = /var/log/livepeer.log
# for journal monitoring:
backend = systemd
journalmatch = _SYSTEMD_UNIT=livepeer.service + _COMM=livepeer
enabled = true
# If livepeer runs as a systemd user service, set this instead:
#backend = systemd[journalflags=1]
#journalmatch = _SYSTEMD_UNIT=livepeer.service _SYSTEMD_CGROUP=/user.slice/user-1000.slice/user@1000.service/app.slice/livepeer.service + _COMM=livepeer
# See - https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html

# DEV Notes:
#
# If you wish to match any kind of TLS error rather (not EOF only), change failregex like:
#
# - failregex = ^\s*\S+ livepeer\[\d+\]: (?:\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} )?http: TLS handshake error from <ADDR>:\d+: EOF$
# + failregex = ^\s*\S+ livepeer\[\d+\]: (?:\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} )?http: TLS handshake error from <ADDR>:\d+:\s
# Note that the \s at end (or some other word boundary or anchor) must be mandatory, because otherwise it can mistakenly match or
# vice versa ignore the part :\d+: as a numeric part inside of IPv6 address (IPv6 addresses have dynamic length due to :: in the middle of address).
#
# Authors:
# Sergey G. Brester (sebres)
# Strykar (Sundara.eth)

With these few files created, start or restart the fail2ban service and view the status of the jails by sudo fail2ban-client status livepeer:

$ sudo fail2ban-client status livepeer
Status for the jail: livepeer
|- Filter
|  |- Currently failed:	0
|  |- Total failed:	0
|  `- Journal matches:	_SYSTEMD_CGROUP=/user.slice/user-1000.slice/user@1000.service/app.slice/livepeer.service + _COMM=livepeer
`- Actions
   |- Currently banned:	10
   |- Total banned:	10
   `- Banned IP list:	104.156.155.31 108.162.210.230 108.162.210.231 108.162.241.152 108.162.241.153 162.142.125.210 162.158.78.76 162.158.78.77 23.225.180.203 45.33.50.110

To unban an IP address 192.168.72.186 which is in the jail livepeer the command would be:
$ sudo fail2ban-client set livepeer unbanip 192.168.72.186

We can enable recidive jails for repeat offenders.
Post any Qs here or on Discord and I’d be happy to help.

6 Likes

Thanks for sharing this. One interesting opportunity here is for the resulting jail config files could be made publicly available over network for O’s to optionally opt-in to.

There could be lists for spam, attackers, certain content types, etc. Then node operators could optionally choose to not provide compute to categories of their choosing. I know moderation is always a sticky subject, but this creates the building blocks for node operators to feel in control of what they contribute to vs what they don’t, and other node operators could always choose not to opt into certain lists.

This is an interesting idea.

But how effective would this be?

Surely attackers would just switch IPs and continue attacking, although I assume the whole point of Fail2Ban to try prevent this?

But I like the idea of using a ban list for content that an O would prefer not to support. But would it be easier to just ban particular broadcaster addresses?

And for Livepeer Studio they all come from the same source anyway so Fail2Ban wouldn’t work for a hosted gateway.

Just my 2 cents, haven’t put much thought into this but I really like the security aspect of what Strykar has built so far.

I created a solution that let’s O’s share IP blacklists using vallumd, MQTT and Fail2ban.
This has been working well for me for over a month now, Marco offered to help test but I have not heard back from him yet.

O’s may choose MQTT topics like SSHD to share IP blacklists for SSH, not just livepeer.
Broadcaster IPs / Wallet addresses could also be shared as a separate topic that O’s could selective opt-in if they choose.

I’m a little unclear what path to chart regarding “resulting jail config files could be made publicly available over network for O’s to optionally opt-in to”. After discussing this with some folks, it appears this would require some Go integration into Livepeer.
Is this what you had in mind?

Great question!
The answer is exactly as effective as Fail2ban protecting SSH today.

Yes, that was the thinking. Go integration into go-livepeer would enable both the IP filtering and the broadcaster address filtering.

For what it’s worth, I think practical first steps, such as via quick scripts outside of direct integration, are a good idea, so I wouldn’t say it’s an urgent priority to press forward on the above.