CentOS, FreeBSD: Secure SSH with fail2ban and blacklistd with MFA (Google Authenticator)

Kliment Andreev — Fri, 27 Mar 2020 18:57:52 +0000

I have some servers that are exposed to the Internet on port 22 and I see a lot of brute force attacks coming. The easiest and best way to protect is to block port 22 for everyone except a handful of IPs or safe subnets. But sometimes, these servers need to be available for some other people who work from who knows where, so you can’t just restrict IPs anymore. In this case, the best practice is to enforce MFA (multi factor authentication). But, this won’t work if you have some service accounts that need to log to the server. You can’t MFA a service account.To balance between security and availability, you can also install some tools that will block the repeated offenders, IPs that crawl the network and try to login to your network over SSH. In this post I’ll describe how to use fail2ban for CentOS and blacklistd for FreeBSD. fail2ban for FreeBSD is too complex to setup while blacklistd is a native tool and does the same job.

CentOS

I have a CentOS box with both SELinux and firewalld enabled. I am not using keys to log to the server, I have a generic user that uses password and the root is NOT allowed to log to the server using ssh (PermitRootLogin no in /etc/ssh/sshd_config).

Log with SSH keys (optional)

If you want to have some users that need to log with SSH keys, you have to do the following. Log as that user and run ssh-keygen and hit enter for the defaults.

ssh-keygen -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/home/test/.ssh/id_rsa):
Created directory '/home/test/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/test/.ssh/id_rsa.
Your public key has been saved in /home/test/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:CmYzEZq7u4IilphSEC/cdTFKQY2+fjQgkw8acCXZjIo test@localhost.localdomain
The key's randomart image is:
+---[RSA 4096]----+
|  .*+++o.        |
|o ++++.o.        |
|+=+ =..          |
|Eo.B +           |
| o+ @ o S        |
| ..+ * +         |
|ooo . o .        |
|Oo . . .         |
|=.o.  .          |
+----[SHA256]-----+

You have two files created for the test user, a private (id_rsa) and a public (id_rsa.pub) file.
You need the private file in order to use it with putty. But putty uses it’s own PPK file, so you have to convert that with puttygen. You can use the Windows version or you can install putty and convert it. NOTE: putty is a GUI program so don’t install it on a server because it will install a bunch of GUI libraries.
Install epel-release repo.

yum install -y epel-release

…and then install putty.

yum install -y putty

Log as the user and convert the file from PEM to PPK.

puttygen .ssh/id_rsa -o private.ppk -O private

Your key will be in the home directory as private.ppk. Use this key to log to your server. But, you have to authorize your public key first. Logged as test user do:

cat .ssh/id_rsa.pub >> .ssh/authorized_keys
chmod 0600 .ssh/authorized_keys

Now, you can log as test user with a key using putty. This will affect only the test user to log with a key. The other users will continue logging as they used to do before. Mind that at this point you can log as a test user using a key and the password. If you want to completely disable password logins make sure that these settings are configured under /etc/ssh/sshd_config.

PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes

Restart sshd (systemctl restart sshd) anytime you make a change in that file.

Google Authenticator

In order to enable MFA, we’ll have to install Google Authenticator on the server and a client on a phone or a desktop. I use authy for that. You can use any MFA client you want on your phone. I will also use key only logins and no passwords so go ahead and change those 3 lines above in sshd_config. Also, make sure that you can login with at least one user that can sudo or you’ll be locked. The only way to get in will be thru the console which does not use ssh so it’s not affected.
To install Google Authenticator, you need the epel repo. Skip this step if you already completed any of the previous steps.

yum install -y epel-release

Then install Google Authenticator.

yum -y install google-authenticator

Now, you can go two ways. Mandate that all accounts use MFA or only certain ones. It depends on your server usage case. If you have some automated scripts that use SSH to do some work, the first option is not viable. For each account that you want to use MFA, you have to configure the authenticator. I’ll use my test account. Logged as the test user, type google-authenticator. I answered y on all of the questions.

google-authenticator
Do you want authentication tokens to be time-based (y/n) y
Warning: pasting the following URL into your browser exposes the OTP secret to Google:
https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/test@localhost.l....

Your new secret key is: HF52123FWAGRKLGX5EJ4567CLE
Your verification code is 880933
Your emergency scratch codes are:
94310385
95472197
98736692
31351434
83561212

Do you want me to update your "/home/test/.google_authenticator" file? (y/n) y

Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n) y

By default, a new token is generated every 30 seconds by the mobile app.
In order to compensate for possible time-skew between the client and the server,
we allow an extra token before and after the current time. This allows for a
time skew of up to 30 seconds between authentication server and client. If you
experience problems with poor time synchronization, you can increase the window
from its default size of 3 permitted codes (one previous code, the current
code, the next code) to 17 permitted codes (the 8 previous codes, the current
code, and the 8 next codes). This will permit for a time skew of up to 4 minutes
between client and server.
Do you want to do so? (y/n) y

If the computer that you are logging into isn't hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting? (y/n) y

In a console session you’ll see a barcode but it’s garbled. Instead scroll up and you’ll see a URL (line 4). Go to that URL and scan the barcode with your MFA app.
Open /etc/ssh/sshd_config, scroll all the way down and add this line as the last line.

AuthenticationMethods publickey,password publickey,keyboard-interactive

Then change this line to yes.

ChallengeResponseAuthentication yes

Edit /etc/pam.d/sshd and after these two lines, add the auth line.

# Used with polkit to reauthorize users in remote sessions
-session   optional     pam_reauthorize.so prepare
auth required pam_google_authenticator.so nullok

nullok means that you are allowing some accounts to log as usual, without MFA. If you remove nullok, all accounts will be required to use MFA.
Also, comment this line like this.

#auth       substack     password-auth

Restart the sshd daemon now.

systemctl restart sshd

fail2ban

In order to install fail2ban you have to install the epel-release first. Skip this step if you followed the previous step because you already installed epel repo.

yum -y install epel-release

Then install fail2ban.

yum -y install fail2ban

Make sure it starts on boot.

systemctl enable fail2ban

The config files for fail2ban are under /etc/fail2ban, but you shouldn’t change any of the existing files. Instead create a new file called sshd.conf under /etc/fail2ban/jail.d directory. Here is mine.

[sshd]
enabled = true

# "ignoreip" can be a list of IP addresses, CIDR masks or DNS hosts. Fail2ban
# will not ban a host which matches an address in this list. Several addresses
# can be defined using space (and/or comma) separator.
ignoreip = 127.0.0.1/8

# "bantime" is the number of seconds that a host is banned.
# add m for minutes or hr for hours
bantime  = 10m

# A host is banned if it has generated "maxretry" during the last "findtime"
# seconds.
findtime = 300

# "maxretry" is the number of failures before a host get banned.
maxretry = 3

It is very self explanatory with the included comments. In my example, I’ll ban an IP for 10 minutes if I see that IP try and fail 3 times to login in 300 seconds (5 mins). You can start/restart the service now.

systemctl start fail2ban

The fail2ban log file is under /var/log/fail2ban.log and the SSH logs are under /var/log/secure.
Most of the hits that you’ll see in the /var/log/secure are IPs that will try to log as admin or root, but only once in a hour or two. So, they won’t be caught with these settings. You should add all of your subnets under ignoreip setting and change the maxretry to 1 and bantime to 8760h (1 year). It will ultimately block the IP for a year for anyone that makes any unauthorized attempt. It’s up to you what settings you use.
Here is an example in the /var/log/secure of an IP (3.92.82.165) trying to log as root.

Mar 21 10:26:13 localhost sshd[13207]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Mar 21 10:26:15 localhost sshd[13207]: Failed password for root from 3.92.82.165 port 60362 ssh2
Mar 21 10:26:16 localhost unix_chkpwd[13211]: password check failed for user (root)
Mar 21 10:26:16 localhost sshd[13207]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Mar 21 10:26:18 localhost sshd[13207]: Failed password for root from 3.92.82.165 port 60362 ssh2
Mar 21 10:26:18 localhost sshd[13207]: Connection closed by 3.92.82.165 port 60362 [preauth]
Mar 21 10:26:18 localhost sshd[13207]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=ec2-3-92-82-165.compute-1.amazonaws.com  user=root

…and the IP got banned. You can see that from /var/log/fail2ban.

2020-03-21 10:26:12,259 fail2ban.filter         [13131]: INFO    [sshd] Found 3.92.82.165 - 2020-03-21 10:26:12
2020-03-21 10:26:15,382 fail2ban.filter         [13131]: INFO    [sshd] Found 3.92.82.165 - 2020-03-21 10:26:15
2020-03-21 10:26:18,887 fail2ban.filter         [13131]: INFO    [sshd] Found 3.92.82.165 - 2020-03-21 10:26:18
2020-03-21 10:26:19,286 fail2ban.actions        [13131]: NOTICE  [sshd] Ban 3.92.82.165

You can also see the banned IPs using this command. sshd is the name of the jail. If you scroll up, you’ll see that I named my jail like that [sshd] in the conf file above.

fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 1
|  |- Total failed:     8
|  `- Journal matches:  _SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
   |- Currently banned: 2
   |- Total banned:     2
   `- Banned IP list:   3.81.85.240 3.92.82.165

But mistakes happen. If someone gets banned, you can unban the IP with this command, where sshd is the jail name and the IP you want unbanned.

fail2ban-client set sshd unbanip 3.92.82.165
1

You should get 1 as an output. Means OK, True. If you get 0, it means the IP wasn’t there. If you can’t see all banned IPs, do:

zgrep 'Ban' /var/log/fail2ban.log*

FreeBSD

I have a FreeBSD box with no firewall enabled for now. For the blacklistd part, I’ll use pf. If you use ipfirewall or ipfilter, then you have to find another guide. I am not using keys to log to the server, I have a generic user that uses password and the root is NOT allowed to log to the server using ssh (PermitRootLogin no in /etc/ssh/sshd_config).

Log with SSH keys (optional)

If you want to have some users that need to log with SSH keys, you have to do the following. Log as that user and run ssh-keygen and hit enter for the defaults.

ssh-keygen -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/home/test/.ssh/id_rsa):
Created directory '/home/test/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/test/.ssh/id_rsa.
Your public key has been saved in /home/test/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:rT2m3a/KrC/jk1lxHUPMwuXM5g/5LpjkurrC29EO1z8 test@bsd.andreev.local
The key's randomart image is:
+---[RSA 4096]----+
|                 |
|                 |
|           .     |
|        ..o o    |
|     . =S..+     |
|.   . + B+.      |
|o. o + @+.+o     |
|..o * o.OE+o     |
| ..o.+o.+@@.oo.  |
+----[SHA256]-----+

You have two files created for the test user, a private (id_rsa) and a public (id_rsa.pub) file.
You need the private file in order to use it with putty. But putty uses it’s own PPK file, so you have to convert that with puttygen. You can use the Windows version or you can install putty and convert it.

pkg install putty-nogtk

Log as the user and convert the file from PEM to PPK.

puttygen .ssh/id_rsa -o private.ppk -O private

Your key will be in the home directory as private.ppk. Use this key to log to your server. But, you have to authorize your public key first. Logged as test user do:

cat .ssh/id_rsa.pub >> .ssh/authorized_keys
chmod 0600 .ssh/authorized_keys

PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes

Restart sshd (service sshd restart) anytime you make a change in that file.

Google Authenticator

pkg install pam_google_authenticator

$ google-authenticator
Do you want authentication tokens to be time-based (y/n) y
Warning: pasting the following URL into your browser exposes the OTP secret to Google:
https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/test@bsd.andreev.local%3Fsecret%3D24345SOFKCTVX44YXYZVYOQKCA%26issuer%3Dbsd.andreev.local
Your new secret key is: 24HJ3SOFKCTVZ33YVBKVYOKEDB
Enter code from app (-1 to skip): -1
Code confirmation skipped
Your emergency scratch codes are:
34344229
32681543
80840666
32593123
29994070

Do you want me to update your "/home/test/.google_authenticator" file? (y/n) y

In a console session you’ll see a barcode but it’s garbled. Instead scroll up and you’ll see a URL (line 4). Go to that URL and scan the barcode with your MFA app.
Now, log as root or sudo to open /etc/ssh/sshd_config file. Change this line to yes.

ChallengeResponseAuthentication yes

Scroll all the way down and add this line as the last line.

AuthenticationMethods publickey,password publickey,keyboard-interactive

Edit /etc/pam.d/sshd and after these two lines, add the last auth line but commend the line above so it looks like this.

#auth           sufficient      pam_ssh.so              no_warn try_first_pass
#auth            required        pam_unix.so             no_warn try_first_pass
auth            required        pam_google_authenticator.so nullok

nullok means that you are allowing some accounts to log as usual, without MFA. If you remove nullok, all accounts will be required to use MFA. Restart the sshd daemon now.

service sshd restart

At this point, you’ll be able to login only with a public key and an MFA for the users that have google-authenticator and a public key configured. The password users won’t be able to log in. The console logins are not affected. So, if you have some users without a public key and google-authneticator, they’ll still be able to login with a password from a console.
If you don’t want to use public key and prefer passwords and MFA, then remove the last line from sshd_config AuthenticationMethods and uncomment pam_unix.so in /etc/pam.d/sshd above google_authenticator line. Restart sshd.

blacklistd

fail2ban is available for FreeBSD, but it takes some time to configure it right with the proper firewalls. Instead, I use blacklistd which comes preinstalled. The only prerequisite is a firewall and I’ll be using pf for that. So, first let’s enable pf and blacklistd by adding these lines in /etc/rc.conf.

pf_enable="YES"
pflog_enable="YES"
blacklistd_enable="YES"
blacklistd_flags="-r"

Let’s create a very simple pf config file /etc/pf.conf.

ext_if = "vmx0"
set skip on lo0
anchor "blacklistd/*" in on $ext_if
table  persist file "/etc/blocked_subnets"
block all
block in log quick on $ext_if from  to any
block out log quick on $ext_if from any to 
pass in on $ext_if proto tcp from any to any port {ssh} keep state
pass out quick on $ext_if keep state

Make sure you replace the name of your network card under line #1. You can get the name from ifconfig output. Also, if you want to block certain subnets by default, you can list them under /etc/blocked_subnets file. The subnet(s) should be in CIDR format. For example, if you want to block certain countries, you can look at this website. BTW, the subnets blocking is part of pf not blacklistd. You can reboot now to make sure everything comes back OK and you are able to login. But before doing that, edit the sshd daemon and let it know that we’ll use blacklistd. Find UseBlacklist in /etc/ssh/sshd_conf and change it to:

UseBlacklist yes

The reboot will start pf and blacklistd. The blacklistd conf file is /etc/blacklistd.conf. You can read more about blacklistd here. For example, if I want to whitelist certain subnets from being blacklisted, I’ll add them under the [remote] section. In my case, I am whitelisting my home subnet and an external subnet.

[remote]
172.16.1.0/24:ssh      *       *       *               =       *       *
154.21.3.0/24:ssh      *       *       *               =       *       *

You can see your SSH activity in /var/log/auth.log. To see who is blocked, run:

blacklistctl dump -br
        address/ma:port id      nfail   remaining time
  213.74.176.36/32:22   OK      8/3     23h58m57s

This IP will be blocked for 24hrs by default. You can change that under the disable column in /etc/blacklistd.conf.
You can also see the blocked IPs by looking at the pf blacklistd table.

pfctl -a blacklistd/22 -t port22 -T show
   213.74.176.36

And if you want to remove an IP or a subnet do:

pfctl -a blacklistd/22 -t port22 -T delete 213.74.176.36/32
1/1 addresses deleted.

fail2ban – Blog of Kliment Andreev – A place so I won't forget things

CentOS, FreeBSD: Secure SSH with fail2ban and blacklistd with MFA (Google Authenticator)

CentOS

Log with SSH keys (optional)

Google Authenticator

fail2ban

FreeBSD

Log with SSH keys (optional)

Google Authenticator

blacklistd