Fail2ban scans log files (e.g. /var/log/apache/error_log) and bans IPs that show the malicious signs – too many password failures, seeking for exploits, etc. Generally Fail2Ban is then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action (e.g. sending an email) could also be configured. Out of the box Fail2Ban comes with filters for various services (apache, courier, ssh, etc).
In this post we configure Fail2ban to find livepeer log lines that match Tue 2022-12-06 04:14:08 IST livepeer[390680]: 2022/12/06 04:14:08 http: TLS handshake error from 172.68.6.145:28576: EOF
and ban the offending IPs 99.99% of which are bots running port scans which fail the TLS initiation and thus SSL posts an EOF error. We assume livepeer is running with verbosity set to 6, (-v 6
).
There are a few files we need to create and populate:
$ cat /etc/fail2ban/jail.local
[DEFAULT]
banaction = nftables
banaction_allports = nftables[type=allports]
$ cat /etc/fail2ban/fail2ban.local
[Definition]
logtarget = /var/log/fail2ban.log
$ sudo touch /var/log/fail2ban.log
$ $ cat /etc/fail2ban/fail2ban.local
[Definition]
logtarget = /var/log/fail2ban.log
Now we create the fail2ban jail configuration:
$ cat /etc/fail2ban/jail.d/livepeer.conf
[livepeer]
enabled = true
filter = livepeer
failregex = ^\s*\S+ livepeer\[\d+\]: (?:\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} )?http: TLS handshake error from <ADDR>:\d+: EOF$
datepattern = ^{DATE}
port = 8935
protocol = all
action = nftables-multiport[name=LPTPortScan, port="8935,3000"]
backend = systemd[journalflags=1]
journalmatch = _SYSTEMD_CGROUP=/user.slice/user-1000.slice/user@1000.service/app.slice/livepeer.service + _COMM=livepeer
banTime = 864000
findtime = 1800
maxRetry = 3
ignoreip = 127.0.0.1/8 ::1 10.0.0.1/24 192.168.1.1/24 182.70.116.80/32 172.105.60.167/32 89.187.177.134/32 89.187.177.138/32 89.187.177.196/32 89.187.178.130/32 89.187.179.35/32 195.181.169.69/32 89.187.185.130/32 89.187.185.153/32 89.187.185.251/32 143.244.50.113/32 89.187.181.249/32 143.244.61.193/32 143.244.61.205/32 212.102.58.242/32 143.244.33.78/32 143.244.33.79/32 143.244.33.95/32 89.187.188.237/32 89.187.188.246/32 212.102.38.92/32 84.17.50.98/32 84.17.50.99/32 89.187.88.237/32 185.59.221.179/32 89.187.169.65/32 138.199.36.76/32 185.102.219.91/32 185.102.219.180/32 195.181.174.39/32 195.181.174.186/32 138.199.4.163/32
Now we create the fail2ban filter configuration:
$ cat /etc/fail2ban/filter.d/livepeer.conf
# Fail2Ban filter for livepeer (https://github.com/livepeer/go-livepeer)
#
# Sample journalctl logs:
# Nov 30 21:36:39 graf livepeer[3014]: 2022/11/30 21:36:39 http: TLS handshake error from 162.158.78.45:47024: EOF
# Nov 30 21:36:39 graf.sun.in livepeer[3014]: 2022/11/30 21:36:39 http: TLS handshake error from 162.158.78.45:47024: EOF
# Fri 2022-12-02 01:34:52 IST livepeer[312413]: 2022/12/02 01:34:52 http: TLS handshake error from 162.158.162.235:45566: EOF
#
# NOTE: The regex below is ONLY intended to work with livepeer Orchestrator log verbosity set to 6 (-v 6).
# See https://docs.livepeer.org/video-miners/reference/configuration#onchain
#
# More information: https://github.com/fail2ban/fail2ban/issues/3423
[DEFAULT]
[Definition]
failregex = ^\s*\S+ livepeer\[\d+\]: (?:\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} )?http: TLS handshake error from <ADDR>:\d+: EOF$
datepattern = ^{DATE}
port = 8935
protocol = all
# for log monitoring:
#backend = auto
#logpath = /var/log/livepeer.log
# for journal monitoring:
backend = systemd
journalmatch = _SYSTEMD_UNIT=livepeer.service + _COMM=livepeer
enabled = true
# If livepeer runs as a systemd user service, set this instead:
#backend = systemd[journalflags=1]
#journalmatch = _SYSTEMD_UNIT=livepeer.service _SYSTEMD_CGROUP=/user.slice/user-1000.slice/user@1000.service/app.slice/livepeer.service + _COMM=livepeer
# See - https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html
# DEV Notes:
#
# If you wish to match any kind of TLS error rather (not EOF only), change failregex like:
#
# - failregex = ^\s*\S+ livepeer\[\d+\]: (?:\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} )?http: TLS handshake error from <ADDR>:\d+: EOF$
# + failregex = ^\s*\S+ livepeer\[\d+\]: (?:\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} )?http: TLS handshake error from <ADDR>:\d+:\s
# Note that the \s at end (or some other word boundary or anchor) must be mandatory, because otherwise it can mistakenly match or
# vice versa ignore the part :\d+: as a numeric part inside of IPv6 address (IPv6 addresses have dynamic length due to :: in the middle of address).
#
# Authors:
# Sergey G. Brester (sebres)
# Strykar (Sundara.eth)
With these few files created, start or restart the fail2ban service and view the status of the jails by sudo fail2ban-client status livepeer
:
$ sudo fail2ban-client status livepeer
Status for the jail: livepeer
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- Journal matches: _SYSTEMD_CGROUP=/user.slice/user-1000.slice/user@1000.service/app.slice/livepeer.service + _COMM=livepeer
`- Actions
|- Currently banned: 10
|- Total banned: 10
`- Banned IP list: 104.156.155.31 108.162.210.230 108.162.210.231 108.162.241.152 108.162.241.153 162.142.125.210 162.158.78.76 162.158.78.77 23.225.180.203 45.33.50.110
To unban an IP address 192.168.72.186
which is in the jail livepeer
the command would be:
$ sudo fail2ban-client set livepeer unbanip 192.168.72.186
We can enable recidive
jails for repeat offenders.
Post any Qs here or on Discord and I’d be happy to help.