I have some servers that are exposed to the Internet on port 22 and I see a lot of brute force attacks coming. The easiest and best way to protect is to block port 22 for everyone except a handful of IPs or safe subnets. But sometimes, these servers need to be available for some other people who work from who knows where, so you can’t just restrict IPs anymore. In this case, the best practice is to enforce MFA (multi factor authentication). But, this won’t work if you have some service accounts that need to log to the server. You can’t MFA a service account.To balance between security and availability, you can also install some tools that will block the repeated offenders, IPs that crawl the network and try to login to your network over SSH. In this post I’ll describe how to use fail2ban for CentOS and blacklistd for FreeBSD. fail2ban for FreeBSD is too complex to setup while blacklistd is a native tool and does the same job.
Table of Contents
CentOS
I have a CentOS box with both SELinux and firewalld enabled. I am not using keys to log to the server, I have a generic user that uses password and the root is NOT allowed to log to the server using ssh (PermitRootLogin no in /etc/ssh/sshd_config).
Log with SSH keys (optional)
If you want to have some users that need to log with SSH keys, you have to do the following. Log as that user and run ssh-keygen and hit enter for the defaults.
ssh-keygen -b 4096 Generating public/private rsa key pair. Enter file in which to save the key (/home/test/.ssh/id_rsa): Created directory '/home/test/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/test/.ssh/id_rsa. Your public key has been saved in /home/test/.ssh/id_rsa.pub. The key fingerprint is: SHA256:CmYzEZq7u4IilphSEC/cdTFKQY2+fjQgkw8acCXZjIo [email protected] The key's randomart image is: +---[RSA 4096]----+ | .*+++o. | |o ++++.o. | |+=+ =.. | |Eo.B + | | o+ @ o S | | ..+ * + | |ooo . o . | |Oo . . . | |=.o. . | +----[SHA256]-----+
You have two files created for the test user, a private (id_rsa) and a public (id_rsa.pub) file.
You need the private file in order to use it with putty. But putty uses it’s own PPK file, so you have to convert that with puttygen. You can use the Windows version or you can install putty and convert it. NOTE: putty is a GUI program so don’t install it on a server because it will install a bunch of GUI libraries.
Install epel-release repo.
yum install -y epel-release
…and then install putty.
yum install -y putty
Log as the user and convert the file from PEM to PPK.
puttygen .ssh/id_rsa -o private.ppk -O private
Your key will be in the home directory as private.ppk. Use this key to log to your server. But, you have to authorize your public key first. Logged as test user do:
cat .ssh/id_rsa.pub >> .ssh/authorized_keys chmod 0600 .ssh/authorized_keys
Now, you can log as test user with a key using putty. This will affect only the test user to log with a key. The other users will continue logging as they used to do before. Mind that at this point you can log as a test user using a key and the password. If you want to completely disable password logins make sure that these settings are configured under /etc/ssh/sshd_config.
PasswordAuthentication no ChallengeResponseAuthentication no UsePAM yes
Restart sshd (systemctl restart sshd) anytime you make a change in that file.
Google Authenticator
In order to enable MFA, we’ll have to install Google Authenticator on the server and a client on a phone or a desktop. I use authy for that. You can use any MFA client you want on your phone. I will also use key only logins and no passwords so go ahead and change those 3 lines above in sshd_config. Also, make sure that you can login with at least one user that can sudo or you’ll be locked. The only way to get in will be thru the console which does not use ssh so it’s not affected.
To install Google Authenticator, you need the epel repo. Skip this step if you already completed any of the previous steps.
yum install -y epel-release
Then install Google Authenticator.
yum -y install google-authenticator
Now, you can go two ways. Mandate that all accounts use MFA or only certain ones. It depends on your server usage case. If you have some automated scripts that use SSH to do some work, the first option is not viable. For each account that you want to use MFA, you have to configure the authenticator. I’ll use my test account. Logged as the test user, type google-authenticator. I answered y on all of the questions.
google-authenticator Do you want authentication tokens to be time-based (y/n) y Warning: pasting the following URL into your browser exposes the OTP secret to Google: https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/[email protected].... Your new secret key is: HF52123FWAGRKLGX5EJ4567CLE Your verification code is 880933 Your emergency scratch codes are: 94310385 95472197 98736692 31351434 83561212 Do you want me to update your "/home/test/.google_authenticator" file? (y/n) y Do you want to disallow multiple uses of the same authentication token? This restricts you to one login about every 30s, but it increases your chances to notice or even prevent man-in-the-middle attacks (y/n) y By default, a new token is generated every 30 seconds by the mobile app. In order to compensate for possible time-skew between the client and the server, we allow an extra token before and after the current time. This allows for a time skew of up to 30 seconds between authentication server and client. If you experience problems with poor time synchronization, you can increase the window from its default size of 3 permitted codes (one previous code, the current code, the next code) to 17 permitted codes (the 8 previous codes, the current code, and the 8 next codes). This will permit for a time skew of up to 4 minutes between client and server. Do you want to do so? (y/n) y If the computer that you are logging into isn't hardened against brute-force login attempts, you can enable rate-limiting for the authentication module. By default, this limits attackers to no more than 3 login attempts every 30s. Do you want to enable rate-limiting? (y/n) y
In a console session you’ll see a barcode but it’s garbled. Instead scroll up and you’ll see a URL (line 4). Go to that URL and scan the barcode with your MFA app.
Open /etc/ssh/sshd_config, scroll all the way down and add this line as the last line.
AuthenticationMethods publickey,password publickey,keyboard-interactive
Then change this line to yes.
ChallengeResponseAuthentication yes
Edit /etc/pam.d/sshd and after these two lines, add the auth line.
# Used with polkit to reauthorize users in remote sessions -session optional pam_reauthorize.so prepare auth required pam_google_authenticator.so nullok
nullok means that you are allowing some accounts to log as usual, without MFA. If you remove nullok, all accounts will be required to use MFA.
Also, comment this line like this.
#auth substack password-auth
Restart the sshd daemon now.
systemctl restart sshd
At this point, you’ll be able to login only with a public key and an MFA for the users that have google-authenticator and a public key configured. The password users won’t be able to log in. The console logins are not affected. So, if you have some users without a public key and google-authneticator, they’ll still be able to login with a password from a console.
fail2ban
In order to install fail2ban you have to install the epel-release first. Skip this step if you followed the previous step because you already installed epel repo.
yum -y install epel-release
Then install fail2ban.
yum -y install fail2ban
Make sure it starts on boot.
systemctl enable fail2ban
The config files for fail2ban are under /etc/fail2ban, but you shouldn’t change any of the existing files. Instead create a new file called sshd.conf under /etc/fail2ban/jail.d directory. Here is mine.
[sshd] enabled = true # "ignoreip" can be a list of IP addresses, CIDR masks or DNS hosts. Fail2ban # will not ban a host which matches an address in this list. Several addresses # can be defined using space (and/or comma) separator. ignoreip = 127.0.0.1/8 # "bantime" is the number of seconds that a host is banned. # add m for minutes or hr for hours bantime = 10m # A host is banned if it has generated "maxretry" during the last "findtime" # seconds. findtime = 300 # "maxretry" is the number of failures before a host get banned. maxretry = 3
It is very self explanatory with the included comments. In my example, I’ll ban an IP for 10 minutes if I see that IP try and fail 3 times to login in 300 seconds (5 mins). You can start/restart the service now.
systemctl start fail2ban
The fail2ban log file is under /var/log/fail2ban.log and the SSH logs are under /var/log/secure.
Most of the hits that you’ll see in the /var/log/secure are IPs that will try to log as admin or root, but only once in a hour or two. So, they won’t be caught with these settings. You should add all of your subnets under ignoreip setting and change the maxretry to 1 and bantime to 8760h (1 year). It will ultimately block the IP for a year for anyone that makes any unauthorized attempt. It’s up to you what settings you use.
Here is an example in the /var/log/secure of an IP (3.92.82.165) trying to log as root.
Mar 21 10:26:13 localhost sshd[13207]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root" Mar 21 10:26:15 localhost sshd[13207]: Failed password for root from 3.92.82.165 port 60362 ssh2 Mar 21 10:26:16 localhost unix_chkpwd[13211]: password check failed for user (root) Mar 21 10:26:16 localhost sshd[13207]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root" Mar 21 10:26:18 localhost sshd[13207]: Failed password for root from 3.92.82.165 port 60362 ssh2 Mar 21 10:26:18 localhost sshd[13207]: Connection closed by 3.92.82.165 port 60362 [preauth] Mar 21 10:26:18 localhost sshd[13207]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=ec2-3-92-82-165.compute-1.amazonaws.com user=root
…and the IP got banned. You can see that from /var/log/fail2ban.
2020-03-21 10:26:12,259 fail2ban.filter [13131]: INFO [sshd] Found 3.92.82.165 - 2020-03-21 10:26:12 2020-03-21 10:26:15,382 fail2ban.filter [13131]: INFO [sshd] Found 3.92.82.165 - 2020-03-21 10:26:15 2020-03-21 10:26:18,887 fail2ban.filter [13131]: INFO [sshd] Found 3.92.82.165 - 2020-03-21 10:26:18 2020-03-21 10:26:19,286 fail2ban.actions [13131]: NOTICE [sshd] Ban 3.92.82.165
You can also see the banned IPs using this command. sshd is the name of the jail. If you scroll up, you’ll see that I named my jail like that [sshd] in the conf file above.
fail2ban-client status sshd Status for the jail: sshd |- Filter | |- Currently failed: 1 | |- Total failed: 8 | `- Journal matches: _SYSTEMD_UNIT=sshd.service + _COMM=sshd `- Actions |- Currently banned: 2 |- Total banned: 2 `- Banned IP list: 3.81.85.240 3.92.82.165
But mistakes happen. If someone gets banned, you can unban the IP with this command, where sshd is the jail name and the IP you want unbanned.
fail2ban-client set sshd unbanip 3.92.82.165 1
You should get 1 as an output. Means OK, True. If you get 0, it means the IP wasn’t there. If you can’t see all banned IPs, do:
zgrep 'Ban' /var/log/fail2ban.log*
FreeBSD
I have a FreeBSD box with no firewall enabled for now. For the blacklistd part, I’ll use pf. If you use ipfirewall or ipfilter, then you have to find another guide. I am not using keys to log to the server, I have a generic user that uses password and the root is NOT allowed to log to the server using ssh (PermitRootLogin no in /etc/ssh/sshd_config).
Log with SSH keys (optional)
If you want to have some users that need to log with SSH keys, you have to do the following. Log as that user and run ssh-keygen and hit enter for the defaults.
ssh-keygen -b 4096 Generating public/private rsa key pair. Enter file in which to save the key (/home/test/.ssh/id_rsa): Created directory '/home/test/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/test/.ssh/id_rsa. Your public key has been saved in /home/test/.ssh/id_rsa.pub. The key fingerprint is: SHA256:rT2m3a/KrC/jk1lxHUPMwuXM5g/5LpjkurrC29EO1z8 [email protected] The key's randomart image is: +---[RSA 4096]----+ | | | | | . | | ..o o | | . =S..+ | |. . + B+. | |o. o + @+.+o | |..o * o.OE+o | | ..o.+o.+@@.oo. | +----[SHA256]-----+
You have two files created for the test user, a private (id_rsa) and a public (id_rsa.pub) file.
You need the private file in order to use it with putty. But putty uses it’s own PPK file, so you have to convert that with puttygen. You can use the Windows version or you can install putty and convert it.
pkg install putty-nogtk
Log as the user and convert the file from PEM to PPK.
puttygen .ssh/id_rsa -o private.ppk -O private
Your key will be in the home directory as private.ppk. Use this key to log to your server. But, you have to authorize your public key first. Logged as test user do:
cat .ssh/id_rsa.pub >> .ssh/authorized_keys chmod 0600 .ssh/authorized_keys
Now, you can log as test user with a key using putty. This will affect only the test user to log with a key. The other users will continue logging as they used to do before. Mind that at this point you can log as a test user using a key and the password. If you want to completely disable password logins make sure that these settings are configured under /etc/ssh/sshd_config.
PasswordAuthentication no ChallengeResponseAuthentication no UsePAM yes
Restart sshd (service sshd restart) anytime you make a change in that file.
Google Authenticator
In order to enable MFA, we’ll have to install Google Authenticator on the server and a client on a phone or a desktop. I use authy for that. You can use any MFA client you want on your phone. I will also use key only logins and no passwords so go ahead and change those 3 lines above in sshd_config. Also, make sure that you can login with at least one user that can sudo or you’ll be locked. The only way to get in will be thru the console which does not use ssh so it’s not affected. To install Google Authenticator type:
pkg install pam_google_authenticator
Now, you can go two ways. Mandate that all accounts use MFA or only certain ones. It depends on your server usage case. If you have some automated scripts that use SSH to do some work, the first option is not viable. For each account that you want to use MFA, you have to configure the authenticator. I’ll use my test account. Logged as the test user, type google-authenticator. I answered y on all of the questions and -1 to skip.
$ google-authenticator Do you want authentication tokens to be time-based (y/n) y Warning: pasting the following URL into your browser exposes the OTP secret to Google: https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/[email protected]%3Fsecret%3D24345SOFKCTVX44YXYZVYOQKCA%26issuer%3Dbsd.andreev.local Your new secret key is: 24HJ3SOFKCTVZ33YVBKVYOKEDB Enter code from app (-1 to skip): -1 Code confirmation skipped Your emergency scratch codes are: 34344229 32681543 80840666 32593123 29994070 Do you want me to update your "/home/test/.google_authenticator" file? (y/n) y Do you want to disallow multiple uses of the same authentication token? This restricts you to one login about every 30s, but it increases your chances to notice or even prevent man-in-the-middle attacks (y/n) y By default, a new token is generated every 30 seconds by the mobile app. In order to compensate for possible time-skew between the client and the server, we allow an extra token before and after the current time. This allows for a time skew of up to 30 seconds between authentication server and client. If you experience problems with poor time synchronization, you can increase the window from its default size of 3 permitted codes (one previous code, the current code, the next code) to 17 permitted codes (the 8 previous codes, the current code, and the 8 next codes). This will permit for a time skew of up to 4 minutes between client and server. Do you want to do so? (y/n) y If the computer that you are logging into isn't hardened against brute-force login attempts, you can enable rate-limiting for the authentication module. By default, this limits attackers to no more than 3 login attempts every 30s. Do you want to enable rate-limiting? (y/n) y
In a console session you’ll see a barcode but it’s garbled. Instead scroll up and you’ll see a URL (line 4). Go to that URL and scan the barcode with your MFA app.
Now, log as root or sudo to open /etc/ssh/sshd_config file. Change this line to yes.
ChallengeResponseAuthentication yes
Scroll all the way down and add this line as the last line.
AuthenticationMethods publickey,password publickey,keyboard-interactive
Edit /etc/pam.d/sshd and after these two lines, add the last auth line but commend the line above so it looks like this.
#auth sufficient pam_ssh.so no_warn try_first_pass #auth required pam_unix.so no_warn try_first_pass auth required pam_google_authenticator.so nullok
nullok means that you are allowing some accounts to log as usual, without MFA. If you remove nullok, all accounts will be required to use MFA. Restart the sshd daemon now.
service sshd restart
At this point, you’ll be able to login only with a public key and an MFA for the users that have google-authenticator and a public key configured. The password users won’t be able to log in. The console logins are not affected. So, if you have some users without a public key and google-authneticator, they’ll still be able to login with a password from a console.
If you don’t want to use public key and prefer passwords and MFA, then remove the last line from sshd_config AuthenticationMethods and uncomment pam_unix.so in /etc/pam.d/sshd above google_authenticator line. Restart sshd.
blacklistd
fail2ban is available for FreeBSD, but it takes some time to configure it right with the proper firewalls. Instead, I use blacklistd which comes preinstalled. The only prerequisite is a firewall and I’ll be using pf for that. So, first let’s enable pf and blacklistd by adding these lines in /etc/rc.conf.
pf_enable="YES" pflog_enable="YES" blacklistd_enable="YES" blacklistd_flags="-r"
Let’s create a very simple pf config file /etc/pf.conf.
ext_if = "vmx0"
set skip on lo0
anchor "blacklistd/*" in on $ext_if
table <blocked_subnets> persist file "/etc/blocked_subnets"
block all
block in log quick on $ext_if from <blocked_subnets> to any
block out log quick on $ext_if from any to <blocked_subnets>
pass in on $ext_if proto tcp from any to any port {ssh} keep state
pass out quick on $ext_if keep state
Make sure you replace the name of your network card under line #1. You can get the name from ifconfig output. Also, if you want to block certain subnets by default, you can list them under /etc/blocked_subnets file. The subnet(s) should be in CIDR format. For example, if you want to block certain countries, you can look at this website. BTW, the subnets blocking is part of pf not blacklistd. You can reboot now to make sure everything comes back OK and you are able to login. But before doing that, edit the sshd daemon and let it know that we’ll use blacklistd. Find UseBlacklist in /etc/ssh/sshd_conf and change it to:
UseBlacklist yes
The reboot will start pf and blacklistd. The blacklistd conf file is /etc/blacklistd.conf. You can read more about blacklistd here. For example, if I want to whitelist certain subnets from being blacklisted, I’ll add them under the [remote] section. In my case, I am whitelisting my home subnet and an external subnet.
[remote] 172.16.1.0/24:ssh * * * = * * 154.21.3.0/24:ssh * * * = * *
You can see your SSH activity in /var/log/auth.log. To see who is blocked, run:
blacklistctl dump -br
address/ma:port id nfail remaining time
213.74.176.36/32:22 OK 8/3 23h58m57s
This IP will be blocked for 24hrs by default. You can change that under the disable column in /etc/blacklistd.conf.
You can also see the blocked IPs by looking at the pf blacklistd table.
pfctl -a blacklistd/22 -t port22 -T show 213.74.176.36
And if you want to remove an IP or a subnet do:
pfctl -a blacklistd/22 -t port22 -T delete 213.74.176.36/32 1/1 addresses deleted.
