CentOS – Blog of Kliment Andreev – A place so I won't forget things

Ansible: Quick Start Guide for FreeBSD, CentOS and Ubuntu

Kliment Andreev — Mon, 08 Feb 2021 17:04:37 +0000

In this post/howto, I’ll explain how to install Ansible as control and managed node on FreeBSD 12, CentOS 8 and Ubuntu 18. Then, I’ll explain how to create SSH keys so the nodes can communicate and some basic tasks. Then, I’ll show you how to create a playbook to install the latest updates and also install an Apache server with the default settings. Finally, I’ll show an example of how to use variables.

Control and managed nodes

A control node is where you execute all of your Ansible commands and eventually keep your playbooks, configs, inventory etc. It’s pretty much your workstation. A managed node is where the actual playbooks are executed. These are the main concepts and the terminology.
In my case, I have 4 VMs/instances. The main one which is the control node and 3 managed nodes.

CentOS 8

If you want to install Ansible on the control node, you have to install Python 3.x first.

sudo dnf install python3

This will also install pip. Type python3 to test, CTRL-D to exit and then install ansible.

sudo pip3 install ansible

Type ansible to test.

Ubuntu

Ubuntu comes with python installed, but not with pip. Install pip with:

sudo apt install python3-pip

Then install ansible with:

sudo pip3 install ansible

Type ansible to test.

FreeBSD

FreeBSD comes with python installed but pip is not. Check the version and then install the same pip version.

ls -l /usr/local/bin/python*

If your output is for example python37, install the same pip version.

pkg install py37-pip

Then install ansible.

pip install ansible

Type ansible to test.

SSH keys

While Ansible can use standard *nix username/password authentication, it’s recommended that you use SSH keys to communicate from control node to the managed nodes. For that, you’ll have to create your SSH keys. Let’s say you have an account on your control node and the username is admin. You also want to use the user ansible on the managed nodes. It really doesn’t matter what usernames you are going to choose. You can always override the keys to use, but in this case, I’ll create a key on the control node and send it to all managed nodes.
On the control node, regardless of your OS, do:

ssh-keygen -b 4096

This will create a subfolder .ssh with two files: id_rsa and id_rsa.pub. The former is your private key and the later is your public key.
Copy the key to your managed nodes.

ssh-copy-id ansible@nodeX.andreev.local

This will copy my key for the user admin to the node X under the ansible user. Then test the passwordless connection.

ssh ansible@nodeX.andreev.local

Mind that the use of FQDN (nodex.andreev.local) vs. hostname (nodex) is important. For SSH these two are different. Once you log to the managed node, the node will be added to the list of known hosts in the file .ssh/known_hosts.

Inventory and the config file

Ansible uses the inventory files to execute an action against using the options and parameters specified in the config file. The config file is /etc/ansible/ansible.cfg for CentOS and Ubuntu and /usr/local/etc/ansible/ansible.cfg for FreeBSD. You can also put the inventory file in the same directory and name it as you wish, but you have to specify the inventory as a parameter on the command line or an entry in the config file. In addition, you can have your config file in your current directory or under the .ansible directory in your home folder. ansible.cfg in the current directory has a precedence over .ansible.cfg in the home directory which has a precedence over /etc/ansible/ansible.cfg. Here is how that looks.
Let’s list all the inventory.

ansible --list-hosts all

You’ll get a message that there is no inventory file. Let’s create one, we’ll name it inventory.txt.

[freebsd]
node1.andreev.local
[centos]
node2.andreev.local
[ubuntu]
node3.andreev.local
[bsd]
node1.andreev.local
[linux]
node2.andreev.local
node3.andreev.local

If we specify the inventory file, we’ll get this.

ansible --list-hosts all -i inventory.txt
  hosts (3):
    node1.andreev.local
    node2.andreev.local
    node3.andreev.local

If we create a config file, we can tell ansible where to look for the inventory. Create a file ansible.cfg in the same directory.

[defaults]
inventory=/home//inventory.txt

If you do ansible –list-hosts all now, you’ll get the same result as before, but without specifying the inventory file.
Or something like this.

ansible all -m shell -a "uname -a"
node1.andreev.local | CHANGED | rc=0 >>
FreeBSD node1.andreev.local 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64
node2.andreev.local | CHANGED | rc=0 >>
Linux node2.andreev.local 4.18.0-193.14.2.el8_2.x86_64 #1 SMP Sun Jul 26 03:54:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
node3.andreev.local | CHANGED | rc=0 >>
Linux node3.andreev.local 4.15.0-129-generic #132-Ubuntu SMP Thu Dec 10 14:02:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

By default, ansible runs on the managed node with the currently logged user that executes the playbook on the control node. If you get an error saying that the previous command cannot connect to the host, you have to specify the same user that you used when you test the connection with ssh -user-@managednode. So, edit ansible.cfg and add this line.

remote_user=

We can target only the group linux which consists of Linux hosts only in the inventory file.

ansible linux -m shell -a "date"
node3.andreev.local | CHANGED | rc=0 >>
Fri Jan  8 15:23:40 UTC 2021
node2.andreev.local | CHANGED | rc=0 >>
Fri Jan  8 10:23:40 EST 2021

Playbooks

The playbooks are the blueprints of the automation tasks. Instead of running the ansible command to execute each task separately, we combine these tasks in a YAML file and execute them sequentially. Here are 3 playbooks that update each of our managed nodes. There is no update module for FreeBSD, so we use the shell command.

# freebsd-update.yml
---
  - hosts: freebsd
    become: yes
    tasks:
      - name: Fetch all packages
        shell: freebsd-update fetch
      - name: Install FreeBSD updates
        shell: freebsd-update install
      - name: Reboot
        reboot:

For CentOS we’ll use yum.

# centos-update.yml
---
  - hosts: centos
    become: yes
    tasks:
      - name: Update all packages
        yum: name=* state=latest
      - name: Reboot
        reboot:

…and for Ubuntu we’ll use apt.

# ubuntu-update.yml
---
  - hosts: ubuntu
    become: yes
    tasks:
      - name: Update all packages
        apt: name=* state=latest
      - name: Reboot
        reboot:

Save these files with a YAML extension and you can execute them with the following command.

ansible-playbook

All of them will probably fail. That’s because your ansible user on the managed nodes is required a password when executing a sudo command. In order to fix that, you’ll have to add a line in the sudoers file. Edit this file using the visudo command.

visudo

…and then add this line right before the @includedir <...> which is the last line in the file.

ansible ALL=(ALL) NOPASSWD:ALL

Where ansible is the user that runs the playbooks on the managed nodes. FreeBSD doesn’t come with sudo preinstalled, so you’ll have to install it first on the managed node.

pkg install sudo

These playbooks will update the OS and the packages for the Linux. For FreeBSD, it will update only the OS. Here is another example of playbooks that will install Apache server in a default configuration and change the ServerName and ServerAdmin lines. We’ll also install PHP and test our server.
If you have a firewall enabled, make sure you open it up first on CentOS. Ubuntu and FreeBSD do not come with the firewall enabled.

firewall-cmd --zone=public --permanent --add-service=http
firewall-cmd --reload

For FreeBSD, the playbook looks like this.

# freebsd-apache.yml
---
  - hosts: freebsd
    become: yes
    tasks:
      - name: Install apache and php
        pkgng:
          name:
            - apache24
            - php74
            - mod_php74
          state:  present
      - name: Start on reboot
        service: name=apache24 enabled=yes
      - name: Copy index.php
        copy:
          src: ../files/index.php
          dest: /usr/local/www/apache24/data
          mode: 0755
      - name: Copy mod_php.conf
        copy:
          src: ../files/mod_php.conf
          dest: /usr/local/etc/apache24/modules.d
          mode: 0755
      - name: Start apache now
        service: name=apache24 state=started

For CentOS, it looks like this.

# centos-apache.yml
---
  - hosts: centos
    become: yes
    tasks:
      - name: Install apache and php
        yum:
          name:
            - httpd
            - php
          state:  present
      - name: Start apache now and on reboot
        service: name=httpd state=started enabled=yes
      - name: Copy index.php
        copy:
          src: ../files/index.php
          dest: /var/www/html
          mode: 0755

…and for Ubuntu it looks like this.

# ubuntu-apache.yml
---
  - hosts: ubuntu
    become: yes
    tasks:
      - name: Install apache and php
        apt:
          name:
            - apache2
            - php
          state:  present
      - name: Start apache now and on reboot
        service: name=apache2 state=started enabled=yes
      - name: Copy index.php
        copy:
          src: ../files/index.php
          dest: /var/www/html
          mode: 0755

You will also need these two files in a directory called files. In my case it’s one level above the directory where I keep my playbooks.
index.php

001_mod-php.conf


    DirectoryIndex index.php index.html
    
        SetHandler application/x-httpd-php
    
    
        SetHandler application/x-httpd-php-source

The index.php file is the standard test file to test the PHP distributions and the 001_mod-php.conf is needed for FreeBSD only. As you can see the playbooks differ quite a bit for these three OSes. Once you deploy the playbooks, you can test the result by going to http://[nodeX]/index.php.
Looks like this.

Service handlers

Service handlers are used only when a change is made on the managed node. For example, we can restart a service only if the config file was changed. If the file is not changed, then there is no need to restart. Here is an example of a service handler. We’ll restart postfix service only if the main.cf file was changed.

# service-handler.yml
---
- hosts: centos
  become: yes
  tasks:
  - name: Configure main.cf
    lineinfile:
      path: /etc/postfix/main.cf
      regexp: ^#mydomain
      line: 'mydomain = example.com'
    notify: restart postfix

  handlers:
  - name: restart postfix
    service: name=postfix state=restarted

Make sure that the name for the handler is the same (lines 11 and 14), so Ansible knows what service handlers is referred.

Variables

When ansible runs a playbook on a managed node, the first task is to gather info about the managed node. The info is a bunch of settings that we can use in our playbooks. For example, if you execute the following command, you can see the IP address, the CPU model, python version etc.

ansible -m setup

We can use these settings and use them as variables if we need them. For example, this playbook displays the hostname and the IP.

# showip.yml
---
  - hosts: freebsd
    become: yes
    tasks:
      - name: Show the IP address
        debug:
          msg: "The hostname is {{ inventory_hostname}}  and the IP is {{ansible_default_ipv4.address }}"

And if you run the playbook, you’ll see something like this.

TASK [Show the IP address] *********************************************************************************************
ok: [node1.andreev.local] => {
    "msg": "The hostname is node1.andreev.local  and the IP is 192.168.1.211"
}

Here is another example of using variables. In this case, we’ll specify a file and change the ownership and the mode.

# owner.yml
---
  - hosts: centos
    become: yes
    vars:
      filename: "/var/www/html/index.php"
    tasks:
      - name: Change the owner of the file
        file:
          path: "{{ filename }}"
          owner: apache
          group: apache
          mode: '0755'

In case we want to assign an output to a variable, we’ll have to use the keyword register. Here is an example of how to get the output from a command and print it on the screen with the keyword debug.

# variables.yml
---
  - hosts: freebsd

    tasks:
    - name: Get the uptime manually
      command: uptime
      register: var_uptime

    - name: Print the uptime
      debug:
        msg: The uptime is "{{ var_uptime }}"

Roles

Roles let you automatically load related vars_files, tasks, handlers, and other Ansible artifacts based on a known file structure. Once you group your content in roles, you can easily reuse them and share them with other users. The idea is to separate the tasks, handlers and vars in different files. Let’s see this playbook for example. It changes a line in main.cf file, restarts postfix and copies a file under the postfix main directory.

# roles.yml
---
  - hosts: centos
    become: yes

    vars:
      filevd: "/etc/postfix/virtual_domains"
      cfgpostfix: "/etc/postfix/main.cf"


    tasks:
    - name: Configure main.cf
      lineinfile:
        path: "{{ cfgpostfix }}"
        regexp: ^#mydomain
        line: 'mydomain = example.com'
      notify: restart postfix
    - name: Copy virtual_domains
      copy:
        src: ../files/virtual_domains
        dest: "{{ filevd }}"
        mode: 0755

    handlers:
    - name: restart postfix
      service: name=postfix state=restarted

We can rewrite this file by separating the tasks, variables, files and handlers. Run this command.

ansible-galaxy role init postfix

If you look at the file/directory structure of the newly created directory postfix, it looks like this.

tree postfix
postfix
├── defaults
│   └── main.yml
├── files
├── handlers
│   └── main.yml
├── meta
│   └── main.yml
├── README.md
├── tasks
│   └── main.yml
├── templates
├── tests
│   ├── inventory
│   └── test.yml
└── vars
    └── main.yml

Create a file virtual_domains under the files directory.

echo "mydomain.com" > postfix/files/virtual_domains

Move the config for vars, handlers and tasks in the separate main.yml files. For example, this is how my files look like.
postfix/vars/main.yml

---
# vars file for postfix

  filevd: "/etc/postfix/virtual_domains"
  cfgpostfix: "/etc/postfix/main.cf"

As you can see the keyword vars: does not exists. Ansible knows that this file is for vars so there is no need to enter the vars keyboard.
postfix/tasks/main.yml

---
# tasks file for postfix

  - name: Configure main.cf
    lineinfile:
      path: "{{ cfgpostfix }}"
      regexp: ^#mydomain
      line: 'mydomain = example.com'
    notify: restart postfix
  - name: Copy virtual_domains
    copy:
      src: ../files/virtual_domains
      dest: "{{ filevd }}"
      mode: 0755

postfix/handlers/main.yml

---
# handlers file for postfix

  - name: restart postfix
    service: name=postfix state=restarted

Finally, create a file called something.yml that will be your main file. This file has to be outside the postfix directory structure.
In my case it looks like this.
something.yml

# something.yml
---
  - hosts: centos
    become: yes
    roles:
      - postfix

Now, if you execute this playbook, ansible will automatically execute the rest of the dependant playbooks as well.

ansible-playbook something.yml

Error handling

Sometimes we want certain changes to be ignored. Sometimes, we know the behavior of certain commands and we know that they might return non-zero code and we want that ignored. For example, consider this part of a playbook.

- hosts: centos
  tasks: 
  - name: Type something that will fail
    command: thiscommanddoesntexist
    ignore_errors: yes

  - name: Run command remotely
    command: /usr/local/bin/somecommand
    register: cmd_result
    changed_when: cmd_result == 2

Ansible would report a task as changed as long as the command (or) script gives zero return code.
In the first part, we know that the task will fail, but we decide to ignore it using the keyword ignore_errors. No matter what the command returns, ignore_errors: yes will never report to ansible that the command failed.
In the second command we can ignore the error based on the output of the command. For example, if the output is 2, the the error will be ignored. If cmd_result is not equal to 2, the task will be marked as changed.
So whenever this condition is true, the task will be marked as changed.

Ansible Vault

Ansible Vault encrypts variables and files so you can protect sensitive content such as passwords or keys rather than leaving it visible as plaintext in playbooks or roles.
First, you have to create a vaulted file where we’ll store the passwords. When you run this command it will ask you to create a password and then an empty file will show up.

ansible-vault create secrets.yml

Add some passwords there and save the file.

mysql_pwd: "DifficultPassword"
ht_pwd: "PasswordXYZ"

If you look at the file now, you’ll see that it’s encrypted and you can’t see the passwords anymore.
If you want to edit the file do ansible-vault edit secrets.yaml and enter the vault password.
Create a small playbook that displays the password.

# centos-vault.yml
---
- hosts: centos
  vars_files:
    - secrets.yml

  tasks:
  - name: Show mysql pwd
    debug:
      msg: "{{ mysql_pwd }}"

If you run the playbook now, ansible will throw an error saying ERROR! Attempting to decrypt but no vault secrets found. You have to specify the parameter –ask-vault-pass and enter the vault password when prompted.

Prompts

In case you need to pause the playbook execution and ask the user for some input such as confirmation or password, use prompts.
If you want the output to echo, use private: no, otherwise what you type won’t show up on the screen. Here is an example of a playbook that asks you to confirm if a file needs to be copied to the node. If you type yes and hit enter, the file will be copied, otherwise it won’t.

# centos-prompt.yml
---
- hosts: centos
  become: yes

  vars_prompt:
    name: upload
    private: no
    prompt: "Do you want to upload xyz.txt?"

  tasks:
  - name: Upload xyz.txt
    copy:
      src: xyz.txt
      dest: /var/log
    when: upload == "yes"

Useful options

ansible-playbook  --syntax-check # checks the syntax of the playbook
ansible-playbook  --check # does a dry run, reports the will-be changes, but the playbook is not executed
ansible-playbook  --step # ask to confirm each-step

CentOS, FreeBSD: Secure SSH with fail2ban and blacklistd with MFA (Google Authenticator)

Kliment Andreev — Fri, 27 Mar 2020 18:57:52 +0000

I have some servers that are exposed to the Internet on port 22 and I see a lot of brute force attacks coming. The easiest and best way to protect is to block port 22 for everyone except a handful of IPs or safe subnets. But sometimes, these servers need to be available for some other people who work from who knows where, so you can’t just restrict IPs anymore. In this case, the best practice is to enforce MFA (multi factor authentication). But, this won’t work if you have some service accounts that need to log to the server. You can’t MFA a service account.To balance between security and availability, you can also install some tools that will block the repeated offenders, IPs that crawl the network and try to login to your network over SSH. In this post I’ll describe how to use fail2ban for CentOS and blacklistd for FreeBSD. fail2ban for FreeBSD is too complex to setup while blacklistd is a native tool and does the same job.

CentOS

I have a CentOS box with both SELinux and firewalld enabled. I am not using keys to log to the server, I have a generic user that uses password and the root is NOT allowed to log to the server using ssh (PermitRootLogin no in /etc/ssh/sshd_config).

Log with SSH keys (optional)

If you want to have some users that need to log with SSH keys, you have to do the following. Log as that user and run ssh-keygen and hit enter for the defaults.

ssh-keygen -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/home/test/.ssh/id_rsa):
Created directory '/home/test/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/test/.ssh/id_rsa.
Your public key has been saved in /home/test/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:CmYzEZq7u4IilphSEC/cdTFKQY2+fjQgkw8acCXZjIo test@localhost.localdomain
The key's randomart image is:
+---[RSA 4096]----+
|  .*+++o.        |
|o ++++.o.        |
|+=+ =..          |
|Eo.B +           |
| o+ @ o S        |
| ..+ * +         |
|ooo . o .        |
|Oo . . .         |
|=.o.  .          |
+----[SHA256]-----+

You have two files created for the test user, a private (id_rsa) and a public (id_rsa.pub) file.
You need the private file in order to use it with putty. But putty uses it’s own PPK file, so you have to convert that with puttygen. You can use the Windows version or you can install putty and convert it. NOTE: putty is a GUI program so don’t install it on a server because it will install a bunch of GUI libraries.
Install epel-release repo.

yum install -y epel-release

…and then install putty.

yum install -y putty

Log as the user and convert the file from PEM to PPK.

puttygen .ssh/id_rsa -o private.ppk -O private

Your key will be in the home directory as private.ppk. Use this key to log to your server. But, you have to authorize your public key first. Logged as test user do:

cat .ssh/id_rsa.pub >> .ssh/authorized_keys
chmod 0600 .ssh/authorized_keys

Now, you can log as test user with a key using putty. This will affect only the test user to log with a key. The other users will continue logging as they used to do before. Mind that at this point you can log as a test user using a key and the password. If you want to completely disable password logins make sure that these settings are configured under /etc/ssh/sshd_config.

PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes

Restart sshd (systemctl restart sshd) anytime you make a change in that file.

Google Authenticator

In order to enable MFA, we’ll have to install Google Authenticator on the server and a client on a phone or a desktop. I use authy for that. You can use any MFA client you want on your phone. I will also use key only logins and no passwords so go ahead and change those 3 lines above in sshd_config. Also, make sure that you can login with at least one user that can sudo or you’ll be locked. The only way to get in will be thru the console which does not use ssh so it’s not affected.
To install Google Authenticator, you need the epel repo. Skip this step if you already completed any of the previous steps.

yum install -y epel-release

Then install Google Authenticator.

yum -y install google-authenticator

Now, you can go two ways. Mandate that all accounts use MFA or only certain ones. It depends on your server usage case. If you have some automated scripts that use SSH to do some work, the first option is not viable. For each account that you want to use MFA, you have to configure the authenticator. I’ll use my test account. Logged as the test user, type google-authenticator. I answered y on all of the questions.

google-authenticator
Do you want authentication tokens to be time-based (y/n) y
Warning: pasting the following URL into your browser exposes the OTP secret to Google:
https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/test@localhost.l....

Your new secret key is: HF52123FWAGRKLGX5EJ4567CLE
Your verification code is 880933
Your emergency scratch codes are:
94310385
95472197
98736692
31351434
83561212

Do you want me to update your "/home/test/.google_authenticator" file? (y/n) y

Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n) y

By default, a new token is generated every 30 seconds by the mobile app.
In order to compensate for possible time-skew between the client and the server,
we allow an extra token before and after the current time. This allows for a
time skew of up to 30 seconds between authentication server and client. If you
experience problems with poor time synchronization, you can increase the window
from its default size of 3 permitted codes (one previous code, the current
code, the next code) to 17 permitted codes (the 8 previous codes, the current
code, and the 8 next codes). This will permit for a time skew of up to 4 minutes
between client and server.
Do you want to do so? (y/n) y

If the computer that you are logging into isn't hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting? (y/n) y

In a console session you’ll see a barcode but it’s garbled. Instead scroll up and you’ll see a URL (line 4). Go to that URL and scan the barcode with your MFA app.
Open /etc/ssh/sshd_config, scroll all the way down and add this line as the last line.

AuthenticationMethods publickey,password publickey,keyboard-interactive

Then change this line to yes.

ChallengeResponseAuthentication yes

Edit /etc/pam.d/sshd and after these two lines, add the auth line.

# Used with polkit to reauthorize users in remote sessions
-session   optional     pam_reauthorize.so prepare
auth required pam_google_authenticator.so nullok

nullok means that you are allowing some accounts to log as usual, without MFA. If you remove nullok, all accounts will be required to use MFA.
Also, comment this line like this.

#auth       substack     password-auth

Restart the sshd daemon now.

systemctl restart sshd

fail2ban

In order to install fail2ban you have to install the epel-release first. Skip this step if you followed the previous step because you already installed epel repo.

yum -y install epel-release

Then install fail2ban.

yum -y install fail2ban

Make sure it starts on boot.

systemctl enable fail2ban

The config files for fail2ban are under /etc/fail2ban, but you shouldn’t change any of the existing files. Instead create a new file called sshd.conf under /etc/fail2ban/jail.d directory. Here is mine.

[sshd]
enabled = true

# "ignoreip" can be a list of IP addresses, CIDR masks or DNS hosts. Fail2ban
# will not ban a host which matches an address in this list. Several addresses
# can be defined using space (and/or comma) separator.
ignoreip = 127.0.0.1/8

# "bantime" is the number of seconds that a host is banned.
# add m for minutes or hr for hours
bantime  = 10m

# A host is banned if it has generated "maxretry" during the last "findtime"
# seconds.
findtime = 300

# "maxretry" is the number of failures before a host get banned.
maxretry = 3

It is very self explanatory with the included comments. In my example, I’ll ban an IP for 10 minutes if I see that IP try and fail 3 times to login in 300 seconds (5 mins). You can start/restart the service now.

systemctl start fail2ban

The fail2ban log file is under /var/log/fail2ban.log and the SSH logs are under /var/log/secure.
Most of the hits that you’ll see in the /var/log/secure are IPs that will try to log as admin or root, but only once in a hour or two. So, they won’t be caught with these settings. You should add all of your subnets under ignoreip setting and change the maxretry to 1 and bantime to 8760h (1 year). It will ultimately block the IP for a year for anyone that makes any unauthorized attempt. It’s up to you what settings you use.
Here is an example in the /var/log/secure of an IP (3.92.82.165) trying to log as root.

Mar 21 10:26:13 localhost sshd[13207]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Mar 21 10:26:15 localhost sshd[13207]: Failed password for root from 3.92.82.165 port 60362 ssh2
Mar 21 10:26:16 localhost unix_chkpwd[13211]: password check failed for user (root)
Mar 21 10:26:16 localhost sshd[13207]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Mar 21 10:26:18 localhost sshd[13207]: Failed password for root from 3.92.82.165 port 60362 ssh2
Mar 21 10:26:18 localhost sshd[13207]: Connection closed by 3.92.82.165 port 60362 [preauth]
Mar 21 10:26:18 localhost sshd[13207]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=ec2-3-92-82-165.compute-1.amazonaws.com  user=root

…and the IP got banned. You can see that from /var/log/fail2ban.

2020-03-21 10:26:12,259 fail2ban.filter         [13131]: INFO    [sshd] Found 3.92.82.165 - 2020-03-21 10:26:12
2020-03-21 10:26:15,382 fail2ban.filter         [13131]: INFO    [sshd] Found 3.92.82.165 - 2020-03-21 10:26:15
2020-03-21 10:26:18,887 fail2ban.filter         [13131]: INFO    [sshd] Found 3.92.82.165 - 2020-03-21 10:26:18
2020-03-21 10:26:19,286 fail2ban.actions        [13131]: NOTICE  [sshd] Ban 3.92.82.165

You can also see the banned IPs using this command. sshd is the name of the jail. If you scroll up, you’ll see that I named my jail like that [sshd] in the conf file above.

fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 1
|  |- Total failed:     8
|  `- Journal matches:  _SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
   |- Currently banned: 2
   |- Total banned:     2
   `- Banned IP list:   3.81.85.240 3.92.82.165

But mistakes happen. If someone gets banned, you can unban the IP with this command, where sshd is the jail name and the IP you want unbanned.

fail2ban-client set sshd unbanip 3.92.82.165
1

You should get 1 as an output. Means OK, True. If you get 0, it means the IP wasn’t there. If you can’t see all banned IPs, do:

zgrep 'Ban' /var/log/fail2ban.log*

FreeBSD

I have a FreeBSD box with no firewall enabled for now. For the blacklistd part, I’ll use pf. If you use ipfirewall or ipfilter, then you have to find another guide. I am not using keys to log to the server, I have a generic user that uses password and the root is NOT allowed to log to the server using ssh (PermitRootLogin no in /etc/ssh/sshd_config).

Log with SSH keys (optional)

If you want to have some users that need to log with SSH keys, you have to do the following. Log as that user and run ssh-keygen and hit enter for the defaults.

ssh-keygen -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/home/test/.ssh/id_rsa):
Created directory '/home/test/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/test/.ssh/id_rsa.
Your public key has been saved in /home/test/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:rT2m3a/KrC/jk1lxHUPMwuXM5g/5LpjkurrC29EO1z8 test@bsd.andreev.local
The key's randomart image is:
+---[RSA 4096]----+
|                 |
|                 |
|           .     |
|        ..o o    |
|     . =S..+     |
|.   . + B+.      |
|o. o + @+.+o     |
|..o * o.OE+o     |
| ..o.+o.+@@.oo.  |
+----[SHA256]-----+

You have two files created for the test user, a private (id_rsa) and a public (id_rsa.pub) file.
You need the private file in order to use it with putty. But putty uses it’s own PPK file, so you have to convert that with puttygen. You can use the Windows version or you can install putty and convert it.

pkg install putty-nogtk

Log as the user and convert the file from PEM to PPK.

puttygen .ssh/id_rsa -o private.ppk -O private

Your key will be in the home directory as private.ppk. Use this key to log to your server. But, you have to authorize your public key first. Logged as test user do:

cat .ssh/id_rsa.pub >> .ssh/authorized_keys
chmod 0600 .ssh/authorized_keys

PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes

Restart sshd (service sshd restart) anytime you make a change in that file.

Google Authenticator

pkg install pam_google_authenticator

$ google-authenticator
Do you want authentication tokens to be time-based (y/n) y
Warning: pasting the following URL into your browser exposes the OTP secret to Google:
https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/test@bsd.andreev.local%3Fsecret%3D24345SOFKCTVX44YXYZVYOQKCA%26issuer%3Dbsd.andreev.local
Your new secret key is: 24HJ3SOFKCTVZ33YVBKVYOKEDB
Enter code from app (-1 to skip): -1
Code confirmation skipped
Your emergency scratch codes are:
34344229
32681543
80840666
32593123
29994070

Do you want me to update your "/home/test/.google_authenticator" file? (y/n) y

In a console session you’ll see a barcode but it’s garbled. Instead scroll up and you’ll see a URL (line 4). Go to that URL and scan the barcode with your MFA app.
Now, log as root or sudo to open /etc/ssh/sshd_config file. Change this line to yes.

ChallengeResponseAuthentication yes

Scroll all the way down and add this line as the last line.

AuthenticationMethods publickey,password publickey,keyboard-interactive

Edit /etc/pam.d/sshd and after these two lines, add the last auth line but commend the line above so it looks like this.

#auth           sufficient      pam_ssh.so              no_warn try_first_pass
#auth            required        pam_unix.so             no_warn try_first_pass
auth            required        pam_google_authenticator.so nullok

nullok means that you are allowing some accounts to log as usual, without MFA. If you remove nullok, all accounts will be required to use MFA. Restart the sshd daemon now.

service sshd restart

At this point, you’ll be able to login only with a public key and an MFA for the users that have google-authenticator and a public key configured. The password users won’t be able to log in. The console logins are not affected. So, if you have some users without a public key and google-authneticator, they’ll still be able to login with a password from a console.
If you don’t want to use public key and prefer passwords and MFA, then remove the last line from sshd_config AuthenticationMethods and uncomment pam_unix.so in /etc/pam.d/sshd above google_authenticator line. Restart sshd.

blacklistd

fail2ban is available for FreeBSD, but it takes some time to configure it right with the proper firewalls. Instead, I use blacklistd which comes preinstalled. The only prerequisite is a firewall and I’ll be using pf for that. So, first let’s enable pf and blacklistd by adding these lines in /etc/rc.conf.

pf_enable="YES"
pflog_enable="YES"
blacklistd_enable="YES"
blacklistd_flags="-r"

Let’s create a very simple pf config file /etc/pf.conf.

ext_if = "vmx0"
set skip on lo0
anchor "blacklistd/*" in on $ext_if
table  persist file "/etc/blocked_subnets"
block all
block in log quick on $ext_if from  to any
block out log quick on $ext_if from any to 
pass in on $ext_if proto tcp from any to any port {ssh} keep state
pass out quick on $ext_if keep state

Make sure you replace the name of your network card under line #1. You can get the name from ifconfig output. Also, if you want to block certain subnets by default, you can list them under /etc/blocked_subnets file. The subnet(s) should be in CIDR format. For example, if you want to block certain countries, you can look at this website. BTW, the subnets blocking is part of pf not blacklistd. You can reboot now to make sure everything comes back OK and you are able to login. But before doing that, edit the sshd daemon and let it know that we’ll use blacklistd. Find UseBlacklist in /etc/ssh/sshd_conf and change it to:

UseBlacklist yes

The reboot will start pf and blacklistd. The blacklistd conf file is /etc/blacklistd.conf. You can read more about blacklistd here. For example, if I want to whitelist certain subnets from being blacklisted, I’ll add them under the [remote] section. In my case, I am whitelisting my home subnet and an external subnet.

[remote]
172.16.1.0/24:ssh      *       *       *               =       *       *
154.21.3.0/24:ssh      *       *       *               =       *       *

You can see your SSH activity in /var/log/auth.log. To see who is blocked, run:

blacklistctl dump -br
        address/ma:port id      nfail   remaining time
  213.74.176.36/32:22   OK      8/3     23h58m57s

This IP will be blocked for 24hrs by default. You can change that under the disable column in /etc/blacklistd.conf.
You can also see the blocked IPs by looking at the pf blacklistd table.

pfctl -a blacklistd/22 -t port22 -T show
   213.74.176.36

And if you want to remove an IP or a subnet do:

pfctl -a blacklistd/22 -t port22 -T delete 213.74.176.36/32
1/1 addresses deleted.

CentOS: Create disposable e-mail addresses using postfix, dovecot and squirrelmail

Kliment Andreev — Tue, 14 Jan 2020 19:51:37 +0000

There are many sites that offer this functionality (temp-mail.org, guerrillamail.org, throwawaymail.com), but if you want to build your own solution from scratch, I’ll show how I managed to accomplish this. I have a CentOS 7 server running in a cloud with a public IP and a valid domain name. You won’t need any SSL certificates, but you can add them if you want. This is not a complete guide on how to create a valid postfix and dovecot e-mail solution. If you are looking for that, stop and look further. The solution described here uses just the basic functionalities to send and receive e-mails. I’ll create some small scripts to manage the temporary e-mail addresses and you can use squirrelmail to check your e-mails. Actually, if you want you can skip that part and use some other e-mail POP3/IMAP e-mail client to check your e-mails.

Valid domain and MX records

You will need a valid domain for this. The scripts cover multiple virtual domains, so if you have many domains, you can use them as well. In the DNS for your domain make sure you have a valid MX record that points to your server. I use Route 53 and this is how my records look like.

Make sure you have the MX record set properly. Go to this site and type your domain name. I’ll use my domain called cloudranger.live. This is how it looks like.

Don’t worry about the DMARC error, we won’t use that. As you can see from both screenshots above, the IPs from the DNS and the tool match, which means your e-mail server will be properly configured on the Internet to receive e-mails.

Firewall and SELinux

If you use firewall, make sure you have these ports opened: 25 for SMTP, 143 for IMAP or 110 for POP3 and 80 for squirrelmail/Apache.
SELinux can stay as is, but if you use squirrelmail, you’ll have to disable it in /etc/selinux/config. Or temporarily, do:

setenforce 0

postfix Part I

We’ll use postfix as MTA. It comes preinstalled on CentOS, but it runs on the localhost interface only, so we have to make some changes. The postfix daemon is running under the postfix user, but later we’ll see that because this users UID is lower than 1000, we can’t use it with dovecot. Actually, we can, but it’s not recommended. We’ll create a user called vpostfix.

groupadd vpostfix && useradd vpostfix -g vpostfix -s /sbin/nologin -c "Virtual postfix user" -d /var/empty

This will create a user and a group called vpostfix. Let’s get the UID and GID.

grep vpostfix /etc/passwd && grep vpostfix /etc/group
vpostfix:x:1002:1002:Virtual postfix user:/var/empty:/sbin/nologin
vpostfix:x:1002:

The UID and the GID are 1002. Write down this number. Now, edit the main configuration file for postfix which is /etc/postfix/main.cf. Make a copy first for the config file in case something goes wrong.

cd /etc/postfix
cp main.cf main.cf.ORIG

Then edit main.cf and search for inet_interfaces. IMPORTANT! Uncomment inet_interfaces = all and add a comment for inet_interfaces = localhost. If you don’t put comment in front of …localhost it will override the …all.

inet_interfaces = all
#inet_interfaces = $myhostname
#inet_interfaces = $myhostname, localhost
#inet_interfaces = localhost

Find and uncomment this line.

#home_mailbox = Maildir/

Then add these lines at the end of the file.

# Virtual domains, users, and aliases
virtual_mailbox_domains = mysql:/etc/postfix/virtual_domains.cf
virtual_mailbox_maps = mysql:/etc/postfix/virtual_users.cf
virtual_mailbox_base = /var/mail/virtual_domains
# Make sure you replace these UID:GID numbers
virtual_minimum_uid = 1002
virtual_uid_maps = static:1002
virtual_gid_maps = static:1002
virtual_transport = lmtp:unix:private/dovecot

So, we’ll use virtual domains that will be defined in a MySQL database, our temp/disposable e-mail addresses will be also stored in a MySQL database and we’ll have a directory /var/mail/virtual_domains where we’ll get the e-mails in a Maildir format. Make sure you replace those numbers that you got for the postfix user. The last line is used to override the actual postfix delivery and use dovecot for that.
Let’s create the directory where e-mails will be stored.

mkdir /var/mail/virtual_domains
chown -R vpostfix:vpostfix /var/mail/virtual_domains

Leave postfix as is for now and let’s proceed with MySQL database.

MySQL

I prefer to use MariaDB, so we’ll install that first.

yum -y install mariadb-server

Enable the daemon on boot and start it.

systemctl enable mariadb
systemctl start mariadb

Do an initial config.

mysql_secure_installation

Pretty much, accept the defaults with ENTER and just enter the new root database password.
Now, we have to create the database and two tables for the virtual domains and the virtual users as specified in lines 2 and 3 in the main.cf file above. Copy and paste these lines but change the password in line 3. That is your password for the postfix database user. This DB user has nothing to do with postfix account in /etc/passwd that runs postfix. You can change the name if you want, but you have to remember it later when we do the scripts.

cat << 'EOF' > /tmp/temp.sql
CREATE DATABASE db_postfix;
GRANT ALL PRIVILEGES ON db_postfix.* TO "postfix"@"localhost" IDENTIFIED BY "your_password";

use db_postfix;

CREATE TABLE virtual_domains (
  id INT(11) NOT NULL AUTO_INCREMENT,
  name VARCHAR(50) NOT NULL,
  PRIMARY KEY (id),
  UNIQUE(name)
);

CREATE TABLE virtual_users (
  id int(11) NOT NULL AUTO_INCREMENT,
  domain_id INT(11) NOT NULL,
  password VARCHAR(106) NOT NULL,
  email VARCHAR(100) NOT NULL,
  PRIMARY KEY (id),
  UNIQUE KEY email (email),
  FOREIGN KEY (domain_id) REFERENCES virtual_domains(id) ON DELETE CASCADE
) DEFAULT CHARSET=utf8;
EOF

Run the script.

mysql -u root -p < /tmp/temp.sql

You can check the tables. Log to the database as the postfix with the password you specified in line 3.

mysql -u postfix -p

Then switch to our new database db_postfix defined in line 2.

use db_postfix;

Show the tables.

show tables;
+----------------------+
| Tables_in_db_postfix |
+----------------------+
| virtual_domains      |
| virtual_users        |
+----------------------+
2 rows in set (0.00 sec)

At this point, we are good with the database. Now we have to create the scripts that will manage the domains and users. You can (of course) use mysql commands to create domains and users, but I’ve made some scripts that will make this easier.

Scripts

These scripts are the backbone of how you’ll interact when creating domains and users. It simplifies everything for you. So, we’ll have a couple of scripts/commands and this is their syntax. I’ll put them under /usr/local/bin so if you run these commands as root, this directory is not in the path. But under CentOS, if you have any non-root user, /usr/local/bin is already in the $PATH. So, do echo $PATH and see if you can execute them without specifying the full path.
There are 8 scripts in total. If you run them without any parameters, they’ll give you the syntax. This is a brief description.

Here is the kick. These scripts call the mysql utility and you have to provide a password for every interaction with the database. You can easily modify them and specify -p parameter in the script and type the password there, but that’s not recommended. So, you have to either specify the postfix DB user password all the time… or, my suggestion is to export the postfix password in a variable that only lasts for the duration of your session.
So, before running any of the scripts, I recommend that you export your password like this. This is the password for the postfix DB user, not the root DB user.

export MYSQL_PWD=your_password

Once you do this, you can run any of the scripts without any prompts. Copy and paste the following scripts.
IMPORTANT! If you modified the DB name or any table name from my script (/tmp/temp.sql), you have to modify these scripts as well.
==================================================
add_domain

cat << 'EOF' > /usr/local/bin/add_domain
#!/bin/bash
usage () {
    echo "Usage: add_domain "
}

if [ $# -ne 1 ] ; then
    usage
    exit 1
fi

if [ -z "$MYSQL_PWD" ] ; then
    OPTION="-p"
else
    OPTION=""
fi

/usr/bin/mysql -u postfix db_postfix $OPTION -e "INSERT INTO db_postfix.virtual_domains (name) VALUES ('$1');"
EOF

add_user

cat << 'EOF' > /usr/local/bin/add_user
#!/bin/bash
if [ $# -ne 2 ] ; then
    echo "Usage: add_user  "
    exit 1
fi

if [ -z "$MYSQL_PWD" ] ; then
    OPTION="-p"
else
    OPTION=""
fi
regex="^[a-z0-9!#\$%&'*+/=?^_\`{|}~-]+(\.[a-z0-9!#$%&'*+/=?^_\`{|}~-]+)*@([a-z0-9]([a-z0-9-]*[a-z0-9])?\.)+[a-z0-9]([a-z0-9-]*[a-z0-9])?\$"

if [[ $1 =~ $regex ]] ; then
    DOMAIN=`echo $1 | cut -d'@' -f2`
else
    echo "Not a valid e-mail address."
    exit 1
fi
ID=$(/usr/bin/mysql -u postfix db_postfix -s $OPTION -N -e "SELECT id from db_postfix.virtual_domains WHERE NAME='$DOMAIN';")
if [ -z "$ID" ] ; then
    echo "The domain $DOMAIN is missing. Use add_domain $DOMAIN first."
else
   /usr/bin/mysql -u postfix db_postfix $OPTION -e "INSERT INTO db_postfix.virtual_users (domain_id, email, password) VALUES ($ID, '$1', ENCRYPT('$2', CONCAT('\$6\$', SUBSTRING(SHA(RAND()), -16))));"
fi
EOF

ls_domains

cat << 'EOF' > /usr/local/bin/ls_domains
#!/bin/bash
if [ -z "$MYSQL_PWD" ] ; then
    OPTION="-p"
else
    OPTION=""
fi
/usr/bin/mysql -u postfix db_postfix $OPTION -e "SELECT * FROM virtual_domains;"
EOF

ls_users

cat << 'EOF' > /usr/local/bin/ls_users
#!/bin/bash

if [ -z "$MYSQL_PWD" ] ; then
    OPTION="-p"
else
    OPTION=""
fi

/usr/bin/mysql -u postfix db_postfix $OPTION -e "SELECT * FROM virtual_users;"

EOF

rmall_domains

cat << 'EOF' > /usr/local/bin/rmall_domains
#!/bin/bash
if [ -z "$MYSQL_PWD" ] ; then
    OPTION="-p"
else
    OPTION=""
fi

while true; do
    read -p "All domain records will be deleted. Please confirm [yn] " yn
    case $yn in
        [Yy]* ) /usr/bin/mysql -u postfix db_postfix $OPTION -e "DELETE FROM db_postfix.virtual_domains"; break;;
        [Nn]* ) exit;;
        * ) echo "Answer yes(y) or no(n).";;
    esac
done
EOF

rmall_users

cat << 'EOF' > /usr/local/bin/rmall_users
#!/bin/bash
if [ -z "$MYSQL_PWD" ] ; then
    OPTION="-p"
else
    OPTION=""
fi

while true; do
    read -p "All user records will be deleted. Please confirm [yn] " yn
    case $yn in
        [Yy]* ) /usr/bin/mysql -u postfix db_postfix $OPTION -e "DELETE FROM db_postfix.virtual_users"; break;;
        [Nn]* ) exit;;
        * ) echo "Answer yes(y) or no(n).";;
    esac
done

EOF

rm_domain

cat << 'EOF' > /usr/local/bin/rm_domain
#!/bin/bash
if [ $# -ne 1 ]
  then
    echo "Usage: rm_domain "
    exit
fi
if [ -z "$MYSQL_PWD" ] ; then
    OPTION="-p"
else
    OPTION=""
fi

/usr/bin/mysql -u postfix db_postfix $OPTION -e "DELETE FROM db_postfix.virtual_domains WHERE name = '$1';"

EOF

rm_user

cat << 'EOF' > /usr/local/bin/rm_user
#!/bin/bash
if [ $# -ne 1 ]
  then
    echo "Usage: rm_user "
    exit
fi
if [ -z "$MYSQL_PWD" ] ; then
    OPTION="-p"
else
    OPTION=""
fi

/usr/bin/mysql -u postfix db_postfix $OPTION -e "DELETE FROM db_postfix.virtual_users WHERE email = '$1';"
EOF

==================================================
Make the scripts executable.

cd /usr/local/bin
chmod +x add_domain ls_domains rmall_domains rm_domain add_user ls_users rmall_users rm_user

Now that we have the scripts let’s create our first domain and first disposable e-mail address. It’s very easy.
Log as regular user or add /usr/local/bin in the PATH if you run these as root.

export MYSQL_PWD=pwd_for_postfix_user
add_domain cloudranger.live
add_user klimenta@cloudranger.live some_pwd

Check what you did.

ls_domains
+----+------------------+
| id | name             |
+----+------------------+
|  1 | cloudranger.live |
+----+------------------+
ls_users
+----+-----------+----------------------------------------------------------+---------------------------+
| id | domain_id | password                                                 | email                     |
+----+-----------+----------------------------------------------------------+---------------------------+
|  1 |         1 | $6$4asdfasdfadsfasdfasdfadsfasdfadsfasdfasdfadsfa3qw8sF/ | klimenta@cloudranger.live |
+----+-----------+----------------------------------------------------------+---------------------------+

The password is hashed. Use the other commands if you want to remove a domain, user, all domains or all users.

postfix Part II

Now that we have MySQL ready, you should create two files that will tell postfix how to tie with MySQL. If you look at the postfix main.cf file, lines 2 and 3 specify exactly that info. Here are the files. Just copy and paste these, but make sure you change the password in line 2. Log as root or sudo.

cat << 'EOF' > /etc/postfix/virtual_domains.cf
user = postfix
password = password_for_db_user_postfix
hosts = 127.0.0.1
dbname = db_postfix
query = SELECT 1 from virtual_domains WHERE name='%s'
EOF

And the same file for the e-mails.

cat << 'EOF' > /etc/postfix/virtual_users.cf
user = postfix
password = password_for_db_user_postfix
hosts = 127.0.0.1
dbname = db_postfix
query = SELECT 1 from virtual_users WHERE email='%s'
EOF

dovecot

We’ll use dovecot so we can get the received e-mails as POP3 or IMAP. I’ll enable both protocols but I’ll use IMAP later with the squirrelmail. We need to install it first and enable it to boot.

yum -y install dovecot dovecot-mysql
systemctl enable dovecot

Now, we’ll configure it. dovecot has one main config file which /etc/dovecot/dovecot.conf and separate config files for each functionality under /etc/dovecot/conf.d directory. Make a copy of both.

cd /etc/dovecot
cp dovecot.conf dovecot.conf.ORIG
cp -R conf.d conf.d.ORIG

Edit /etc/dovecot/dovecot.conf and uncomment the following line.

protocols = imap pop3 lmtp

Then, copy the template dovecot-sql file. You might have to check the directory where this file is, because the dovecot version number is part of the name.

cd /etc/dovecot
cp /usr/share/doc/dovecot-2.2.36/example-config/dovecot-sql.conf.ext .

Edit this file and search for these directives, uncomment them and configure them properly.

#driver =

…should become

driver = mysql

Then…

#connect =

…should become

connect = host=127.0.0.1 dbname=db_postfix user=postfix password=pwd_for_db_postfix_user

Again…

#default_pass_scheme = MD5

…should become

default_pass_scheme = SHA512-CRYPT

Scroll down and finally change this.

#password_query = \
#  SELECT username, domain, password \
#  FROM users WHERE username = '%n' AND domain = '%d'

…to this.

password_query = \
  SELECT email as user, password \
  FROM virtual_users WHERE email = '%u'

But it’s not over. We have to make some changes in the other files as well.
Edit each of these files and make these changes under /etc/dovecot/conf.d.
====================================================
10-master.conf
Look for this snippet and make sure…

service lmtp {
  unix_listener lmtp {
    #mode = 0666
  }

…it’s like this. The first occurrence for user/group is postfix, the rest are vpostfix. It’s not a typo.

service lmtp {
  unix_listener /var/spool/postfix/private/dovecot {
    #mode = 0666
    mode = 0600
    user = postfix
    group = postfix
  }

Same further down.

  unix_listener auth-userdb {
    #mode = 0666
    #user =
    #group =
  }

  # Postfix smtp-auth
  #unix_listener /var/spool/postfix/private/auth {
  #  mode = 0666
  #}

Should look like this.

  unix_listener auth-userdb {
    mode = 0600
    user = vpostfix
    group =  vpostfix
  }

  # Postfix smtp-auth
  unix_listener /var/spool/postfix/private/auth {
    mode = 0666
    user = vpostfix
    group = vpostfix
  }

10-auth.conf
Uncomment these 3 lines.

disable_plaintext_auth = yes
auth_mechanisms = plain
!include auth-sql.conf.ext

Then modify auth_mechanisms = plain and add login so it looks like this.

auth_mechanisms = plain login

auth-sql.conf.ext
Comment these lines.

userdb {
  driver = sql
  args = /etc/dovecot/dovecot-sql.conf.ext
}

So it’s like this.

#userdb {
#  driver = sql
#  args = /etc/dovecot/dovecot-sql.conf.ext
#}

…and all the way at the bottom.

#userdb {
  #driver = static
  #args = uid=vmail gid=vmail home=/var/vmail/%u
#}

should be…Don’t just uncomment. Change the args line.

userdb {
  driver = static
  args = uid=vpostfix gid=vpostfix home=/var/mail/virtual_domains/%d/%n
}

10-logging.conf
Uncomment the log_path line and specify a log file for dovecot.

log_path = /var/log/dovecot.log

10-mail.conf
Uncomment the mail_location and specify where the e-mails will be stored. %d/%n means the domain name and the login name, so the path will be something like /var/mail/virtual_domains/cloudranger.live/klimenta. dovecot will take care of the directory creation, so don’t worry about that.

mail_location = maildir:/var/mail/virtual_domains/%d/%n

====================================================

Test e-mail delivery

At this point we should be able to test our solution. Restart both daemons, so all the changes take effect.

systemctl restart mariadb
systemctl restart postfix
systemctl restart dovecot

Check the logs for both postfix and dovecot. They should look similar to this.

tail /var/log/maillog
Jan 13 16:21:08 ip-172-31-72-148 postfix/postfix-script[27817]: stopping the Postfix mail system
Jan 13 16:21:08 ip-172-31-72-148 postfix/master[2774]: terminating on signal 15
Jan 13 16:21:08 ip-172-31-72-148 postfix/postfix-script[27897]: starting the Postfix mail system
Jan 13 16:21:08 ip-172-31-72-148 postfix/master[27899]: daemon started -- version 2.10.1, configuration /etc/postfix

…and

tail /var/log/dovecot.log
Jan 13 16:21:14 master: Info: Dovecot v2.2.36 (1f10bfa63) starting up for imap, pop3, lmtp (core dumps disabled)

From another computer send an e-mail to your disposable e-mail account. You’ll see that it’s recorded in the postfix log.
This is my /var/log/maillog.

Jan 13 23:34:26 ip-172-31-86-23 postfix/smtpd[23108]: connect from something.domain.com[114.102.113.201]
Jan 13 23:34:26 ip-172-31-86-23 postfix/smtpd[23108]: ACB334065F8: client=something.domain.com[114.102.113.201]
Jan 13 23:34:26 ip-172-31-86-23 postfix/cleanup[23116]: ACB334065F8: message-id=<000701d5ca69$fe8e8eb0$fbabac10$@heythere.com>
Jan 13 23:34:26 ip-172-31-86-23 postfix/smtpd[23108]: disconnect from something.domain.com[114.102.113.201]
Jan 13 23:34:26 ip-172-31-86-23 postfix/qmgr[23052]: ACB334065F8: from=, size=2814, nrcpt=1 (queue active)
Jan 13 23:34:26 ip-172-31-86-23 postfix/lmtp[23117]: ACB334065F8: to=, relay=ip-172-31-86-23.ec2.internal[private/dovecot], delay=0.09, delays=0.02/0.01/0.01/0.04, dsn=2.0.0, status=sent (250 2.0.0  gDpGLIL+HF5OWgAAbmziKA Saved)
Jan 13 23:34:26 ip-172-31-86-23 postfix/qmgr[23052]: ACB334065F8: removed

Look at line 6 (scroll to the right). SMTP code 250 means all went well.
And if you go to /var/mail/virtual_domains/your_domain/your_email, you’ll see a bunch of folders created. The e-mail that just arrived is under new directory. You can cat the file and see your e-mail.

pwd
/var/mail/virtual_domains/cloudranger.live/klimenta/new
ls -l
total 4
-rw-------. 1 vpostfix vpostfix 2993 Jan 13 23:34 1578958466.M778127P23118.ip-172-31-86-23.ec2.internal,S=2993,W=3076

At this point, you can use some IMAP/POP3 client to get the e-mail (see notes at the end). Specify the hostname of the server as your mail server, if asked for port, put 110 for POP and 143 for IMAP and your disposable e-mail/password as username and password. In my case, I don’t want to deal with constant e-mail client configuration for these temp e-mails, so I installed squirrelmail. You can use Roundcube Mail if you like sleeker interface, but there is more configuration involved.

squirrelmail (optional)

In order to install squirrelmail, you have to install epel-release repo. It also installs Apache and PHP for you. Bear this in mind and if you are not comfortable, stop.

yum -y install epel-release

Then install squirrelmail.

yum -y install squirrelmail

Enable Apache on boot.

systemctl enable httpd

Go to /etc/httpd/conf.d and replace the squirrelmail.conf file. Keep the original file. Just copy and paste.

cd /etc/httpd/conf.d
mv squirrelmail.conf squirrelmail.conf.ORIG
cat << 'EOF' > /etc/httpd/conf.d/squirrelmail.conf
#
# SquirrelMail is a webmail package written in PHP.
#

Alias /webmail /usr/share/squirrelmail


        Options None
        AllowOverride none
        Require all granted

EOF

Let’s configure it.

cd /usr/share/squirrelmail/config/
./conf.pl

Go to #2, then #3. Switch to #2 (SMTP). Go back with R and then go to option D and type dovecot. Finally type S to Save data. Q to quit.
Start Apache.

systemctl start httpd

…and go to your server’s IP or hostname if you have it in DNS as http://ip_or_hostname/webmail. Log with the disposable e-mail and you’ll see your mailbox.

Note about POP3/IMAP clients

If you want to access your mailbox from a GUI client from another computer, you won’t be able to do so because of the dovecot configuration. That’s because dovecot considers localhost connections secure, but any other subnet rather than that is considered unsecure.
In order to make this work, you’ll have to make changes. But mind that if you make these changes, your password will be sent as a plain text across the network. If you are fine with this, proceed.
Change this line to = no in /etc/dovecot/conf.d/10-auth.conf

disable_plaintext_auth = yes

Change ssl = required in /etc/dovecot/conf.d/10-ssl.conf to ssl = no.

ssl = no

Restart dovecot and you should be able to access your mailbox remotely.

AWS: squid transparent proxy for DNS filtering/whitelisting

Kliment Andreev — Sun, 17 Nov 2019 22:38:01 +0000

If you have some instances running in AWS, most likely they are in the private subnet and to access Internet they use the Internet Gateway. By default, they can access everything and that’s a big no-no. You don’t want to be an open SMTP relay or a Bitcoin node if your instance is compromised. But the Internet Gateway can’t do the filtering for you. You are able to access everything. Granted, you can modify the rules in the Network ACL, but you can only deny access to IPs, not URLs. The problem with this scenario is that many update sites have IPs that change from time to time or they use CDN networks with a tons of IPs behind. If you whitelist these IPs, then technically you are whitelisting every single site that’s behind the CDN network such as akamai.
One of the solutions is to use squid. squid can act as a cache server and transparent proxy. Transparent proxy means that the instances that are accessing the Internet are not aware that they are behind a proxy, you don’t have to specify any settings on the instance that’s accessing the Internet. A transparent proxy is much more easier for the end user/client, they don’t have to configure anything on their end.
I’ll show you how to configure a squid transparent proxy that sits in a public subnet in AWS and the instances that are in the private subnets will use squid to access Internet. I’ll implement whitelisting ACL which means that only allowed sites will be reachable. The rest is blocked. This solution is not for your home/lab network where you want to access everything. With this solution most sites will not show properly unless you whitelist all the dependencies. For example, some sites have their CSS scripts in a separate domain, so if you whitelist amazon.com, you’ll be able to access it, but it will look garbled because their CSS is on awsstatic.com. So, you have to whitelist this site. I’ll explain how to find that as well if you encounter these problems.
Before I begin, these are the pre-requisites and the expectations. We’ll use CentOS 7 (with SELinux disabled) as a squid server that runs in a public subnet and the instances that want to access the Internet are in a private subnet. squid constantly evolves, so whatever works now might not work tomorrow, because of the nature of how we access SSL sites (related to TLS 1.3). I’ll use CentOS 7 and squid 4.9. If you use a different version, this might not work. Also, subsequent updates might break this solution. If you want an Enterprise solution, you might want to look somewhere else.
Here is my architectural diagram. I’ll explain every detail. If you have your VPCs, subnets, routes ready, you might want to skip this, but pay attention on the routing.

VPC, subnets, public route and Internet Gateway

I’ll create a new VPC called vpcSquid and I’ll carve that VPC with two subnets, one private and one public. Each will accommodate 256 hosts.

Let’s create the two subnets. Go to the VPC menu and from the Subnets submenu click on Create subnet.
The first one is the public subnet where squid server will reside.

Create the private subnet.

Now go to the Route tables menu and you’ll see that you have a route there already. This route was created when you created the VPC. It refers to the public subnet, but it’s not associated yet.

Click one the Subnet associations tab and then click on Edit subnet assications button. Select the public subnet and click Save.

Before we create a route to the Internet for the public subnet, we need an Internet Gateway. Click on the Internet Gateways menu on the left and click Create Internet Gateway. I named mine gwSquid. Select the newly created Internet Gateway and then click on the Actions button and click on Attach to VPC. Assign the VPC that we created.

Now, go back to the Route tables, then select the public route again and click on the Routes tab. Click on Edit routes button, then click on Add route.
Add a route to the Internet (0.0.0.0/0) over the Internet gateway that we just created. Click on Save routes.

squid instance

Let’s create the instance now. Choose CentOS 7 image, deploy it in the VPC that we just created. Make sure it’s in the public subnet and that you have a public IP.

For storage, choose whatever you like. If you want to use squid as a cache server, then choose more than the default 8GB. For example, if you choose 20GB disk, you can also restrict squid to keep only 12GB of cache for example. Your choice.
Assign a security group that allows port 22 (SSH). In my case I allowed access from everywhere (0.0.0.0/0), but it’s wisely to choose only certain IPs or subnets. This is a test instance, so I’ll proceed with 0.0.0.0/0.

Finally choose your key and launch the instance. While the instance spins up, go back to the VPC | Route tables menu.

Private route

Click on Create Route table button and enter the name of the private route and associate it the the VPC. Click the Create button.

Select the private route and click on the Subnet Associations tab. Click Edit subnet associations button and select the private subnet.

Then click on the Routes tab and click on Edit routes. Click Add route. For Destination type 0.0.0.0/0, for the target select Instance and select the instance that we just created. Click Save routes button. What we just did is we told AWS to route the Internet request in the private subnet through our squid instance. So, the Internet request will go to the instance first and then through the Internet gateway.

Source/Destination Check

At this point, the only thing remaining is to go to the instance and change the Source/Destination Check setting. Select the instance, click the Actions button, then select Networking and then Select Source/Dest. Check. Click on Yes, Disable.

Install squid from source

SSH to the instance using centos as user and your key as password. Type sudo su to get the root prompt. CentOS comes with squid 3.5, but we’ll install squid 4.9 from the source.
First, install the pre-requisites.

yum install -y perl gcc-c++ autoconf automake make wget
yum install -y libxml2-devel libecap-devel openssl-devel openldap-devel pam-devel libdb-devel

Then, create a user that will run the squid daemon.

adduser squid -r -s /sbin/nologin -m

Go to the /tmp folder, download the squid tarball, extract it and go to that directory.

cd /tmp
wget http://www.squid-cache.org/Versions/v4/squid-4.9.tar.gz
tar xzvf squid-4.9.tar.gz
cd squid-4.9

Configure the squid with all helpers. Click the plus sign to expand the source.

./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu \
--program-prefix= --prefix=/usr --exec-prefix=/usr \
--bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc \
--datadir=/usr/share --includedir=/usr/include \
--libdir=/usr/lib64 --libexecdir=/usr/libexec \
--sharedstatedir=/var/lib --mandir=/usr/share/man \
--infodir=/usr/share/info \
--exec_prefix=/usr --libexecdir=/usr/lib64/squid \
--localstatedir=/var --datadir=/usr/share/squid \
--sysconfdir=/etc/squid --with-logdir=/var/log/squid \
--with-pidfile=/var/run/squid.pid --enable-auth \
--enable-auth-basic=DB,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB,getpwnam,fake \
--enable-auth-ntlm=fake --enable-auth-digest=file,LDAP,eDirectory \
--enable-auth-negotiate=kerberos,wrapper \
--enable-external-acl-helpers=file_userip,LDAP_group,time_quota,delayer,SQL_session,session,unix_group,wbinfo_group,kerberos_ldap_group \
--enable-cache-digests --enable-cachemgr-hostname=localhost \
--enable-delay-pools --enable-epoll --enable-icap-client --enable-ident-lookups \
--enable-linux-netfilter --enable-removal-policies=heap,lru \
--enable-snmp --enable-storeio=aufs,diskd,rock,ufs --enable-wccpv2 \
--enable-esi --enable-ecap --with-aio \
--enable-security-cert-generators --enable-security-cert-validators \
--enable-icmp --with-filedescriptors=16384 --disable-arch-native --without-nettle \
--disable-dependency-tracking --enable-eui --enable-follow-x-forwarded-for \
--with-default-user=squid --enable-ssl-crtd --with-included-ltdl \
--with-dl --with-openssl --with-pthreads --disable-arch-native

Compile and install. It takes about 20 mins on 2 CPU, 2GB RAM t3 instance.

make && make install

While the installation runs, open up a new session and sudo su again as root. We have to create the systemd startup scripts. Just copy and paste these three commands. It will create three files.

cat << 'EOF' > /usr/lib/systemd/system/squid.service
[Unit]
Description=Squid caching proxy
After=syslog.target network.target nss-lookup.target

[Service]
Type=forking
LimitNOFILE=16384
EnvironmentFile=/etc/sysconfig/squid
ExecStartPre=/usr/libexec/squid/cache_swap.sh
ExecStart=/usr/sbin/squid $SQUID_OPTS -f $SQUID_CONF
ExecReload=/usr/sbin/squid $SQUID_OPTS -k reconfigure -f $SQUID_CONF
ExecStop=/usr/sbin/squid -k shutdown -f $SQUID_CONF
TimeoutSec=0

[Install]
WantedBy=multi-user.target
EOF

cat << 'EOF' > /etc/sysconfig/squid
# default squid options
SQUID_OPTS=""

# Time to wait for Squid to shut down when asked. Should not be necessary
# most of the time.
SQUID_SHUTDOWN_TIMEOUT=100

# default squid conf file
SQUID_CONF="/etc/squid/squid.conf"
EOF

mkdir /usr/libexec/squid
cat << 'EOF' > /usr/libexec/squid/cache_swap.sh
#!/bin/bash
if [ -f /etc/sysconfig/squid ]; then
        . /etc/sysconfig/squid
fi

SQUID_CONF=${SQUID_CONF:-"/etc/squid/squid.conf"}

CACHE_SWAP=`sed -e 's/#.*//g' $SQUID_CONF | \
        grep cache_dir | awk '{ print $3 }'`

for adir in $CACHE_SWAP; do
        if [ ! -d $adir/00 ]; then
                echo -n "init_cache_dir $adir... "
                squid -N -z -F -f $SQUID_CONF >> /var/log/squid/squid.out 2>&1
        fi
done
EOF
chmod +x /usr/libexec/squid/cache_swap.sh

Once the compilation and installation ends, execute these lines to allow the squid user to access the log directory and enable the auto-start on boot. Also, make sure SELinux is disabled and change /etc/selinux/config file to SELINUX=permissive or SELINUX=disabled.

chown -R squid:squid /var/log/squid
systemctl enable squid
setenforce 0

Feel free to start squid, but it will run with the default config and in non-transparent proxy mode.

# Optional
systemctl start squid
systemctl status squid -l
tail /var/log/squid/cache.log

If you like you can test. Log to an instance in a private subnet and set the proxy. Change the IP below to the IP of your squid server and make sure the squid server has a security group that allows port 3128 from 10.10.51.0/24 subnet.

# Optional, do this on the client instance
export http_proxy=10.10.50.235:3128
export https_proxy=10.10.50.235:3128

If you do curl -v https://wordpress.org you’ll get their page back in HTML/CSS.

unset http_proxy
unset https_proxy

If you reset the proxy settings to nothing and if you do culr again, you’ll see that it will time-out. Even with the proxy set, you can’t use yum because yum uses proxy settings specified in /etc/yum.conf. That’s why a transparent proxy is much more superior. You don’t have to worry about configuring the proxy on the instances.
NOTE: Make sure you unset the proxy settings if you already exported them.
Now that we have pretty-much everything ready, let’s create the config file. It’s using SSLBump peek and splice config. For more info click here.
Just copy and paste the snippet below. It will create the config for you.

cat << 'EOF' > /etc/squid/squid.conf
cache deny all
# HTTP
http_port 3128
http_port 3129 intercept
acl http-whitelist dstdomain "/etc/squid/whitelist.txt"
http_access allow http-whitelist

# HTTPS
https_port 3130 cert=/etc/squid/ssl/squid.pem ssl-bump intercept
acl sslport port 443
http_access allow sslport
acl https-whitelist ssl::server_name "/etc/squid/whitelist.txt"
acl step1 at_step SslBump1
acl step2 at_step SslBump2
acl step3 at_step SslBump3
ssl_bump peek step1 all
ssl_bump peek step2 https-whitelist
ssl_bump splice step3 https-whitelist
ssl_bump terminate step2 all
http_access deny all
EOF

And for the whitelist file, create a file with your allowed domains. Everything else will be blocked. You can put IPs here as well.

cat << 'EOF' > /etc/squid/whitelist.txt
.amazon.com
.amazonaws.com
.wordpress.org
.api.google.com
EOF

If squid is running, stop it with systemctl stop squid and execute these lines to create an internal self-signed certificate. More info on why this is needed, in the link that I’ve presented above.

mkdir /etc/squid/ssl
cd /etc/squid/ssl
openssl genrsa -out squid.key 4096
openssl req -new -key squid.key -out squid.csr -subj "/C=US/ST=NJ/L=squid/O=squid/CN=squid"
openssl x509 -req -days 3650 -in squid.csr -signkey squid.key -out squid.crt
cat squid.key squid.crt >> squid.pem

Initialize the SSL database.

/usr/lib64/squid/security_file_certgen -c -s /var/cache/squid/ssl_db -M 4MB

…otherwise you’ll receive these errors in /var/log/squid/cache.log.

2019/11/17 15:44:19 kid1| WARNING: /usr/lib64/squid/security_file_certgen -s /var/cache/squid/ssl_db -M 4MB #Hlpr1 exited
2019/11/17 15:44:19 kid1| Too few /usr/lib64/squid/security_file_certgen -s /var/cache/squid/ssl_db -M 4MB processes are running (need 1/32)

Change the permission of this file…

chmod 4755 /usr/lib64/squid/pinger

…otherwise you’ll receive these errors in /var/log/squid/cache.log.

2019/11/17 15:38:47| pinger: Initialising ICMP pinger ...
2019/11/17 15:38:47| Open  icmp_sock: (1) Operation not permitted
2019/11/17 15:38:47| pinger: Unable to start ICMP pinger.

If you start squid now, you’ll get something like this in /var/log/squid/cache.log. If you still get errors like the ones above (pinger: Unable to start ICMP pinger), reboot one more time.

2019/11/17 15:49:17| Created PID file (/var/run/squid.pid)
2019/11/17 15:49:17 kid1| Current Directory is /
2019/11/17 15:49:17 kid1| Starting Squid Cache version 4.9 for x86_64-redhat-linux-gnu...
2019/11/17 15:49:17 kid1| Service Name: squid
2019/11/17 15:49:17 kid1| Process ID 3705
2019/11/17 15:49:17 kid1| Process Roles: worker
2019/11/17 15:49:17 kid1| With 16384 file descriptors available
2019/11/17 15:49:17 kid1| Initializing IP Cache...
2019/11/17 15:49:17 kid1| DNS Socket created at [::], FD 5
2019/11/17 15:49:17 kid1| DNS Socket created at 0.0.0.0, FD 9
2019/11/17 15:49:17 kid1| Adding domain ec2.internal from /etc/resolv.conf
2019/11/17 15:49:17 kid1| Adding nameserver 10.10.50.2 from /etc/resolv.conf
2019/11/17 15:49:17 kid1| helperOpenServers: Starting 5/32 'security_file_certgen' processes
2019/11/17 15:49:17 kid1| Logfile: opening log daemon:/var/log/squid/access.log
2019/11/17 15:49:17 kid1| Logfile Daemon: opening log /var/log/squid/access.log
2019/11/17 15:49:17 kid1| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
2019/11/17 15:49:17 kid1| Store logging disabled
2019/11/17 15:49:17 kid1| Swap maxSize 0 + 262144 KB, estimated 20164 objects
2019/11/17 15:49:17 kid1| Target number of buckets: 1008
2019/11/17 15:49:17 kid1| Using 8192 Store buckets
2019/11/17 15:49:17 kid1| Max Mem  size: 262144 KB
2019/11/17 15:49:17 kid1| Max Swap size: 0 KB
2019/11/17 15:49:17 kid1| Using Least Load store dir selection
2019/11/17 15:49:17 kid1| Current Directory is /
2019/11/17 15:49:17 kid1| Finished loading MIME types and icons.
2019/11/17 15:49:17 kid1| HTCP Disabled.
2019/11/17 15:49:17 kid1| Pinger socket opened on FD 26
2019/11/17 15:49:17 kid1| Squid plugin modules loaded: 0
2019/11/17 15:49:17 kid1| Adaptation support is off.
2019/11/17 15:49:17 kid1| Accepting HTTP Socket connections at local=[::]:3128 remote=[::] FD 22 flags=9
2019/11/17 15:49:17 kid1| Accepting NAT intercepted HTTP Socket connections at local=[::]:3129 remote=[::] FD 23 flags=41
2019/11/17 15:49:17 kid1| Accepting NAT intercepted SSL bumped HTTPS Socket connections at local=[::]:3130 remote=[::] FD 24 flags=41
2019/11/17 15:49:17| pinger: Initialising ICMP pinger ...
2019/11/17 15:49:17| pinger: ICMP socket opened.
2019/11/17 15:49:17| pinger: ICMPv6 socket opened
2019/11/17 15:49:18 kid1| storeLateRelease: released 0 objects

If you see my squid.conf, you’ll see that I am not using squid as cache server (cache deny all). If you want to cache your responses, replace that line with this one. 100 means 100MB. See here for more explanation.

cache_dir ufs /var/cache/squid 100 16 256

In order to run in transparent proxy mode, we have to make some firewall changes and redirect the traffic on ports 80 and 443 to squid which will listen on 3129 and 3130 (for https). Do this on the squid server. You don’t have to do anything on the clients.

iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 3129
iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-port 3130

If we reboot the server, the iptables changes will be lost, so we have to save them.

yum -y install iptables-services
systemctl enable iptables
systemctl start iptables
service iptables save

Reboot to make sure everything comes back. Check the redirects. Lines 4 and 5 should be as below.

iptables --table nat --list
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:http redir ports 3129
REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:https redir ports 3130

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  anywhere             anywhere

Test

In order to test, we’ll build a small Linux instance in the private subnet and test from there. Do not assign a public IP to this instance. If you don’t have a VPN or Direct Connect to your private subnets, you’ll have to use the squid server to access the instance. Copy the key (PEM) file to the squid instance and SSH to the private instance using ssh -i key.pem internal_IP_of_test_instance. You have to chmod 600 key.pem first. The key file is the one that you use to connect to the squid instance. That’s in my case. If you have multiple key files, then copy the key file that you used when you created the instance in the private subnet. Assign a security group that allows SSH from the squid server instance, or use the same one that I used when created the squid server which allows access from everywhere on port 22. You have to rectify this later. Don’t allow access from everywhere in any circumstances. This is just for test.
Make sure that you have a new security group created for the squid server instance. This security group should allow ports 80 and 443 from the private subnet only (10.10.51.0/24). Attach this security group to the squid server so it allows HTTP/HTTPS ports from the client instances.

On the client instance, make sure you have unset http_proxy and https_proxy variables. If you try to access a site that’s not listed in the whitelist.txt, you’ll get an Access Denied page from squid or sometimes you’ll get some SSL errors. For the sites that are listed, you might get them to show properly, but sometimes they’ll be garbled. If you look at the source of the page, you’ll see what URLs is the main site trying to access. Most likely there are some JavaScripts hosted on different domains. You’ll have to whitelist these as well if you want proper rendering. But sometimes, you might want to connect to an update server and you won’t be able to see the HTML rendering, e.g. CentOS update sites. In those cases Fiddler might be able to help.

AWS, CentOS: Create your own radio station and deploy it on Alexa (optional)

Kliment Andreev — Sun, 06 Oct 2019 12:55:24 +0000

In this post I’ll explain how to run your own Internet radio station. On top of that, I’ll show you how to build an Alexa skill to deploy your station in the cloud, so anyone with Alexa can say: “Alexa, play <whatever_station>” and hear your Internet radio station. There are many tutorials that describe this, but most of them require that you pay for some radio station hosting. If that’s easier for you, then you can skip this tutorial. If you want to have your own radio on the Internet (build from scratch) and deploy it or not on Alexa, than this guide is for you. There are some prerequisites though. You’ll need a CentOS 7 instance with a public IP and if you decide to publish it on Alexa, then you’ll need valid SSL certificate.
IMPORTANT: If your want your station to be public on Alexa, then you also need license for the music. But no worries, if you just want your music played over Alexa at home (and not public), you can still do it.
Also, the ices2 client can’t stream MP3 files, so you have to convert them to OGG. If you want to stream MP3s directly, check the earlier version of ices. The configuration file is almost identical. Ogg-Vorbis is perfectly fine and I’ll show you how to convert your library from MP3 to OGG.
So, there are two parts of this tutorial. Build your own station first and then the optional Alexa part. Let’s start.

CentOS 7

Make sure you have a CentOS 7 instance ready in the cloud with a public IP. NATed instances are OK as long as you can control the ports. In addition, make sure SELinux is disabled. Disabling the OS firewall is optional. I assume that the firewall on the instance is on.

icecast

icecast is a streaming server similar to shoutcast. It broadcasts a stream that’s provided by a source. The source can be a microphone, sound card or a file. In order to stream a file, we need a client such as ices, LiquidSoap etc. You are not the client, you are the listener. Now that we have this straighten up, let’s install icecast from the source. Just copy and paste everything below. If you get an error, execute line by line and see where it fails. Fix any issues, if any.

yum -y groupinstall "Development Tools"
yum install -y wget curl-devel libvorbis-devel libxslt-devel libxslt-devel openssl-devel
cd /tmp
wget http://downloads.xiph.org/releases/icecast/icecast-2.4.4.tar.gz
tar xzvf icecast-2.4.4.tar.gz
cd icecast-2.4.4
./configure --prefix=/opt/icecast --with-curl --with-openssl
make
make install

We specified /opt/icecast where icecast will be installed (line 7). You can change that in the configure statement, but you have to adjust everything after. icecast creates a sample configuration file named icecast.xml under /opt/icecast/etc. If you look at the XML file, you’ll see a bunch of settings and some of them have to be changed.
We’ll change the following parameters:

location

admin

hostname

source-password

relay-password

admin-user

admin-password

logdir

user

group

So, let’s change these manually or with this code. Adjust to suit your needs.

cd /opt/icecast/etc
location="New Jersey"
admin=klimenta@iandreev.com
hostname=radio.iandreev.com
sourcepwd=Sup3rStr0ngP455w0rd
relaypwd=Sup3rStr0ngP455w0rd
admin=myusername
password=Sup3rStr0ngP455w0rd
logdir=/opt/icecast/log
user=nobody
group=nobody
sed -i "s/Earth/$location/" icecast.xml
sed -i "s/icemaster@localhost/$admin/" icecast.xml
sed -i "s/localhost/$hostname/" icecast.xml
sed -i "s/hackme/$sourcepwd/" icecast.xml
sed -i "s/hackme/$relaypwd/" icecast.xml
sed -i "s/admin/$admin/" icecast.xml
sed -i "s/hackme/$password/" icecast.xml
sed -i "s|/opt/icecast/var/log/icecast|$logdir|" icecast.xml
sed -i "s/nobody/$user/" icecast.xml
sed -i "s/nogroup/$group/" icecast.xml

Let’s create the log directory and give the icecast daemon permission to write.

cd /opt/icecast
mkdir log
chown -R nobody:nobody log

icecast doesn’t come with a startup systemd script, so we’ll have to take care of that.

cat << EOF > /etc/systemd/system/icecast.service
[Unit]
Description=Icecast Network Audio Streaming Server
After=network.target

[Service]
Type=simple
User=nobody
Group=nobody
ExecStart=/opt/icecast/bin/icecast -c /opt/icecast/etc/icecast.xml
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/kill $MAINPID

[Install]
WantedBy=multi-user.target
EOF

Now you can enable icecast on boot and start it.

systemctl enable icecast
systemctl start icecast

If you run systemctl status icecast, you should see that the daemon is running. Also, if you check the error log (tail /opt/icecast/log/error.log), you’ll see that the daemon is running too. Ignore the SSL warning for now.
icecast runs on port 8000, so if you have a firewall, you’ll have to poke a hole.

firewall-cmd --add-port=8000/tcp --zone=public --permanent
firewall-cmd --reload

From a browser, go to your server’s IP on port 8000, e.g. http://201.202.203.24:8000 and you’ll see something like this.

If you click on Administration, you’ll be prompted to log in. Use the admin-user and admin-password that you’ve changed in icecast.xml.

ices

Now that we have the streaming server ready, let’s install the source client. The source client will provide the stream to the server. We’ll use ices for this.

cd /tmp
wget http://downloads.us.xiph.org/releases/ices/ices-2.0.2.tar.gz
tar xzvf ices-2.0.2.tar.gz
cd ices-2.0.2
yum -y install libshout-devel
./configure --prefix=/opt/ices
make && make install

ices creates some configuration files under /opt/ices/share/ices. We’ll use the ices-playlist.xml template to generate a radio station for us.

cd /opt/ices/share/ices
cp ices-playlist.xml myradio.xml

Now edit myradio.xml and change the following with a script or manually. Make sure you adjust for your needs.

logpath

logfile

name

genre

description

password

source-password in icecast.xml

mount

param name

cd /opt/ices/share/ices
logpath=/opt/ices/log
logfile=myradio.log
name="Radio MyRadio"
genre="Some genre"
descr="Best Radio in the Universe and beyond"
pwd=Sup3rStr0ngP455w0rd
mount=/myradio.ogg
playlist=/opt/ices/playlists/myradio.txt
sed -i "s|/var/log/ices|$logpath|" myradio.xml
sed -i "s/ices.log/$logfile/" myradio.xml
sed -i "s/Example stream name/$name/" myradio.xml
sed -i "s/Example genre/$genre/" myradio.xml
sed -i "s/A short description of your stream/$descr/" myradio.xml
sed -i "s/hackme/$pwd/" myradio.xml
sed -i "s|/example1.ogg|$mount|" myradio.xml
sed -i "s|playlist.txt|$playlist|" myradio.xml

Let’s create two directories. One for the logs and the other for the files where we’ll store playlists. Not the music files – the playlists. The playlists tells ices where to look for your music.

cd /opt/ices
mkdir log playlists
chown -R nobody:nobody log

Create a startup script for ices as well. Look at line 10, that’s where my config file for this radio station is. Can you have multiple radio stations? Absolutely, one icecast server can handle multiple ices clients. Just invoke another ices client with a different config file and create a similar startup script with a different URL, e.g …/myradio.ogg and …/mystation.ogg.

cat << EOF > /etc/systemd/system/ices-myradio.service
[Unit]
Description=Ices Network Audio Streaming Client for MyRadio
After=icecast.service

[Service]
Type=simple
User=nobody
Group=nobody
ExecStart=/opt/ices/bin/ices /opt/ices/share/ices/myradio.xml
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/kill $MAINPID
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target
EOF

At this point you can tell ices where your music is stored. As I mentioned earlier, we’ll have to use Ogg-Vorbis files (.ogg).
So, create a folder where your music files will be stored. I’ll use /music/myradio and put some public royalty free music there.
Courtesy of Patrick de Arteaga. Please give him some credit and visit his site.

mkdir -p /music/myradio
cd /music/myradio
wget -U 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.6) Gecko/20070802 SeaMonkey/1.1.4' https://patrickdearteaga.com/audio/Battleship.ogg
wget -U 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.6) Gecko/20070802 SeaMonkey/1.1.4' https://patrickdearteaga.com/audio/Chiptronical.ogg
wget -U 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.6) Gecko/20070802 SeaMonkey/1.1.4' https://patrickdearteaga.com/audio/Friends.ogg

OK, so we have our music there. When the stream ends, it will start from the first song. Now we have to create the playlist and put it under the /opt/ices/conf directory.
If you remember we specified directive in the myradio.xml file referencing where the playlist is stored. Let’s create the file.

ls -d /music/myradio/*.ogg > /opt/ices/playlists/myradio.txt

Now that we have everything ready, let’s enable ices on boot and start it.

systemctl enable ices-myradio
systemctl start ices-myradio

Check if everything is OK.

systemctl status ices-myradio
tail /opt/ices/log/myradio.log

If everything looks good, you can listen to your stream at http://whateverIP:8000/myradio.ogg.
If you are satisfied with this, you can stop reading. You’ve accomplished enough. If you want to see how to convert MP3 to OGG and how to use a reverse proxy so you can stream over HTTP, then read further.

MP3 to OGG conversion (sox)

If you have a bunch of files that you want to convert to OGG, you can use sox. It’s a very powerful tool for conversion between formats. Let’s install it first.

yum -y install libmad-devel libid3tag-devel lame lame-devel flac flac-devel
cd /tmp
wget https://sourceforge.net/projects/sox/files/sox/14.4.2/sox-14.4.2.tar.gz
tar xvfz sox-14.4.2.tar.gz
cd sox-14.4.2
./configure
make -s && make install

Now you can convert the files from MP3 to OGG and vice versa. Conversion is very simple. Just specify the source and the destination file, e.g. sox file1.mp3 file1.ogg and that’s it. If you want to see the details of the file such as bit-rate, duration, sample etc, use soxi, e.g. soxi file.ogg. If you want more details on sox and it’s usage, please check their website. For soxi, check this link.
If you want to convert your files in bulk, you can use this command that will search for MP3 files in your current directory and subdirectories and convert them to OGG.

find . -type f -name "*.mp3" -exec sh -c 'sox "$1" "${1%.mp3}.ogg"' _ {} \;

nginx as reverse proxy

If you recall, we are using http://somenameorip:8000/myradio.ogg to access our station. This is fine for regular users from their home, but in any corporate environment, anything than port 80 and 443 are blocked. So, we have to stream our music over port 80. I’ll show later how to stream over 443 because it’s a requirement for Alexa. First thing first, let’s install nginx.

yum -y install nginx
systemctl enable nginx

nginx creates a configuration file under /etc/nginx called nginx.conf. Look for this snippet in the configuration file.

        location / {
        }

…and change it to this.

        location /myradio.ogg {
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_pass http://localhost:8000/myradio.ogg;
        }

You can start the reverse proxy now and verify that it’s working.

systemctl start nginx
systemctl status nginx

Because we are proxying over HTTP, we have to open the firewall too and if you want you can close port 8000. It’s not needed anymore.

firewall-cmd --add-port=80/tcp --zone=public --permanent
firewall-cmd --remove-port=8000/tcp --zone=public --permanent
firewall-cmd --reload

Now you can access your station as http://serveriporname/myradio.ogg, so there is no need to specify port 8000.

SSL certificate

If you decide to publish your station on Alexa, you’ll need a valid (not self-signed) certificate. You will also need to reconfigure both nginx and icecast to use SSL. You can’t access your server by IP because you’ll get an invalid certificate warning, so make sure you have your DNS ready to resolve your IP to a valid domain name. Get the certificates ready. You’ll need the private key, the certificate, the intermediate certificate and the root CA certificate. When you purchase a certificate all of these will be available for you (except for the private key).
Let’s create the chain. In my case, I have each certificate in a separate file and I’ll create one certificate for icecast.

mkdir /opt/icecast/ssl
cat star.iandreev.com.key star.iandreev.com.crt star.iandreev.com.inter star.iandreev.com.CA > /opt/icecast/ssl/icecast.pem
chown -R nobody:nobody /opt/icecast/ssl

Then, edit /opt/icecast/etc/icecast.xml and find the following snippet.

Copy these three lines after, so the config looks like this now. We are telling icecast to spawn itself and listen on port 8443 too.

We also have to specify the certificate chain. Find this line in the config and uncomment it. Make sure you uncomment the line, it’s commented by default.

/opt/icecast/ssl/icecast.pem

Now, you have to restart icecast and ices too.

systemctl restart icecast
systemctl restart ices-myradio

And if you check the error.log (yes, error log), you’ll see that icecast is using SSL now.

tail /opt/icecast/log/error.log
[2019-10-03  22:29:23] INFO connection/get_ssl_certificate SSL certificate found at /opt/icecast/ssl/icecast.pem
[2019-10-03  22:29:23] INFO connection/get_ssl_certificate SSL using ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES

Now the nginx part. Edit /etc/nginx/nginx.conf.

        location /myradio.ogg {
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_pass http://localhost:8000/myradio.ogg;
        }

– Copy your private key to /etc/pki/nginx/private/server.key and your public certificate to /etc/pki/nginx/server.crt. If the directories do not exists, create them. (cd /etc/pki && mkdir -p nginx/private). Also, make sure that the certificate (server.crt) is a chain of certificates, not just a simple server certificate. If you do curl -iv https://yoursite.com, the result for the SSL should be clear without any errors or warning. You can reuse the icecast.pem certificate for icecast and copy that file as server.crt under /etc/pki/nginx/. You can even reference the same file in the same directory, but then you have to change the config files and permissions.
– Reload nginx (systemctl reload nginx)
– Open the firewall

firewall-cmd --add-port=443/tcp --zone=public --permanent
firewall-cmd --reload

Now you can access your server with https. If you want to be even more precise, you can redirect the http to https, so you don’t have to specify the protocol in the browser. Edit nginx.conf and add line 5. Reload nginx after (systemctl reload nginx).

server {
    listen       80 default_server;
    listen       [::]:80 default_server;
    server_name  _;
    return 301 https://$host$request_uri;
    root         /usr/share/nginx/html;

Here is my /etc/nginx/nginx.conf file.

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _;
        return 301 https://$host$request_uri;
        root         /usr/share/nginx/html;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_pass http://localhost:8000/myradio.ogg;
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }

# Settings for a TLS enabled server.

    server {
        listen       443 ssl http2 default_server;
        listen       [::]:443 ssl http2 default_server;
        server_name  _;
        root         /usr/share/nginx/html;

        ssl_certificate "/etc/pki/nginx/server.crt";
        ssl_certificate_key "/etc/pki/nginx/private/server.key";
        ssl_session_cache shared:SSL:1m;
        ssl_session_timeout  10m;
        ssl_ciphers HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers on;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location /myradio.ogg {
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_pass http://localhost:8000/myradio.ogg;
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }

}

Alexa

Now comes the fun part. It’s much easier than you think thanks to Andy Moore. He has a great step-by-step tutorial for this. I’ll quickly go over what’s needed.
You’ll need your:

Fun fact. I named my station Radio Andreev and whatever I said, Alexa couldn’t understand. It was looking for Radio Andrea all the time. So, do yourself a favor and name your station something that Alexa can understand. I named my station Radio Tincup, my website is https://radio.iandreev.com and the stream is at https://radio.iandreev.com/myradio.ogg. Generate the JSON from the site above or get this code and change your values. Click the (+) sign to expand.

{
  "interactionModel": {
    "languageModel": {
      "invocationName": "radio tincup",
      "intents": [
        {
          "name": "AMAZON.CancelIntent",
          "samples": [
            "Cancel"
          ]
        },
        {
          "name": "AMAZON.HelpIntent",
          "samples": [
            "Help"
          ]
        },
        {
          "name": "AMAZON.StopIntent",
          "samples": [
            "Stop"
          ]
        },
        {
          "name": "Play",
          "samples": [
            "Ask radio tincup to play radio tincup",
            "Start radio tincup",
            "Ask radio tincup to play",
            "Play radio tincup"
          ]
        },
        {
          "name": "AMAZON.NavigateHomeIntent",
          "samples": []
        },
        {
          "name": "AMAZON.PauseIntent",
          "samples": []
        },
        {
          "name": "AMAZON.ResumeIntent",
          "samples": []
        }
      ],
      "types": []
    }
  }
}

And this is your JavaScript.

const Alexa = require('ask-sdk-core');
var https = require('https');

const PlayHandler = {
	canHandle(handlerInput)
	{
		return (
			handlerInput.requestEnvelope.request.type === 'LaunchRequest' ||
			(
				handlerInput.requestEnvelope.request.type === 'IntentRequest' &&
				handlerInput.requestEnvelope.request.intent.name === 'Play'
			) ||
			(
				handlerInput.requestEnvelope.request.type === 'IntentRequest' &&
				handlerInput.requestEnvelope.request.intent.name === 'AMAZON.ResumeIntent'
			)
		);
	},
	handle(handlerInput)
	{
		return handlerInput.responseBuilder
			.addDirective({
				type: 'AudioPlayer.Play',
				playBehavior: 'REPLACE_ALL',
				audioItem:{
					stream:{
						token: '0',
						url: 'https://radio.iandreev.com/myradio.ogg',
						offsetInMilliseconds: 0
					}
				}
			})
			.getResponse();
	}
};

const PauseStopHandler = {
	canHandle(handlerInput)
	{
		return (
				handlerInput.requestEnvelope.request.type === 'IntentRequest' &&
				(
					handlerInput.requestEnvelope.request.intent.name === 'AMAZON.CancelIntent' ||
					handlerInput.requestEnvelope.request.intent.name === 'AMAZON.StopIntent'
				)
			) ||
			(
				handlerInput.requestEnvelope.request.type === 'IntentRequest' &&
				handlerInput.requestEnvelope.request.intent.name === 'AMAZON.PauseIntent'
			);
	},
	handle(handlerInput)
	{
		return handlerInput.responseBuilder
			.addDirective({
				type: 'AudioPlayer.ClearQueue',
				clearBehavior: 'CLEAR_ALL'
			})
			.getResponse();
	}
};

const HelpIntentHandler = {
	canHandle(handlerInput)
	{
		return handlerInput.requestEnvelope.request.type === 'IntentRequest' &&
			handlerInput.requestEnvelope.request.intent.name === 'AMAZON.HelpIntent';
	},
	handle(handlerInput)
	{
		const speechText = 'You can say Play, Stop or Resume. For more information please visit radio dot iandreev dot com';
		return handlerInput.responseBuilder
			.speak(speechText)
			.getResponse();
	}
};

const SessionEndedRequestHandler = {
	canHandle(handlerInput)
	{
		return handlerInput.requestEnvelope.request.type === 'SessionEndedRequest';
	},
	handle(handlerInput)
	{
		return handlerInput.responseBuilder.getResponse();
	}
};

const IntentReflectorHandler = {
	canHandle(handlerInput)
	{
		return handlerInput.requestEnvelope.request.type === 'IntentRequest';
	},
	handle(handlerInput)
	{
		const intentName = handlerInput.requestEnvelope.request.intent.name;
		const speechText = 'NO INTENT HELP TEXT';
		return handlerInput.responseBuilder
			.speak(speechText)
			.getResponse();
	}
};

const ErrorHandler = {
	canHandle()
	{
		return true;
	},
	handle(handlerInput, error)
	{
		const speechText = `Sorry, I could not understand what you said. Please try again.`;
		return handlerInput.responseBuilder
			.speak(speechText)
			.reprompt(speechText)
			.getResponse();
	}
};

exports.handler = Alexa.SkillBuilders.custom()
	.addRequestHandlers(
		PlayHandler,
		PauseStopHandler,
		HelpIntentHandler,
		SessionEndedRequestHandler,
		IntentReflectorHandler)
	.addErrorHandlers(
		ErrorHandler)
	.lambda();

Go to https://developer.amazon.com and log in (or register). Make sure you use the same account in the developer console with the same account that you have your Alexa configured. Once logged in, click on Alexa from the menu and then Alexa Skills Kit.

Click Create Skill. Enter the name of your station and the language.

Choose a Custom model.

Choose Alexa-Hosted.

Scroll up and in the top right corner click Create skill. After 30-40 seconds, you’ll be presented with a different screen. On the left side find the Interfaces menu, then click on it. Then enable the Audio Player on the right.

Scroll up and click on Save Interfaces first, then Build Model.

Click on the JSON Editor on the left, right above Interfaces. Once the editor shows up, delete everything there and paste the JSON file.

Click Save Model then Build Model from the top.

Click on Code from the top and paste the JavaScript in the editor.

Click Save then Deploy from the upper right.

Click on Test, allow access to the microphone and change Skill testing from Off to Development.

Now, with the mouse hit the microphone icon, hold it there, say Alexa Play Radio Tincup and release the button.

Here is what I was talking to. Alexa can’t understand me…or maybe my mic is bad. If Alexa doesn’t understand, just type “Alexa play Radio Tincup” in the text field.

You won’t hear Alexa playing and you won’t hear any music from the computer, but you’ll see a JSON response with the valid URL of your OGG stream.

At this point you are all set. You can play your music using Alexa. For most people this would be the end. But if you really like to publish your station to be public, then proceed. In the console, click on Distribution.

You’ll see a bunch of questions regarding the nature of your station, are you collecting customer information, are you advertising and so on. Just follow the process and hopefully, you’ll get your station published.

Cleanup

You might want to delete the tarballs for the icecast, ices and sox that you’ve downloaded.

rm -rf /tmp/ice*
rm -rf /tmp/sox*

You’ll also want to make sure that the firewall allows only port 80 and 443. Ports 8000 and 8443 can be, but they don’t have to be exposed to the Internet.

firewall-cmd --remove-port=8000/tcp --zone=public --permanent
firewall-cmd --remove-port=8443/tcp --zone=public --permanent
firewall-cmd --reload

FreeBSD, CentOS: Install Jenkins behind a nginx reverse proxy using SSL certificates

Kliment Andreev — Sat, 20 Jul 2019 13:06:24 +0000

In this post, I’ll explain how to install Jenkins on FreeBSD and CentOS. Jenkins runs on port 8080 (8180 in FreeBSD), so sometimes it’s not possible to access these ports because of corporate firewalls. We’ll put Jenkins behind nginx that will act as a reverse proxy. We’ll use port 80 (HTTP) and 443 (HTTPS) if you want to have SSL certificates. The installation is very simple.

CentOS

The Jenkins package is not part of the default CentOS repository, so we have to download that one first, install Jenkins, enable it to start on boot and then start it. You can skip the first line if you have wget installed. By default, it is not installed.

yum -y install wget
wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key
yum -y install jenkins java
systemctl enable jenkins
systemctl start jenkins

If you have a firewall enabled, you have to open the port for 8080.

firewall-cmd --permanent --zone=public --add-port=8080/tcp
firewall-cmd --reload

Now, go to http://[ip]:8080 and replace the with the IP of your server and you should see this greeting page.

Cat out that file, copy & paste the value in the browser and click Continue.

cat /var/lib/jenkins/secrets/initialAdminPassword
7f09cb85a4b843bdab66c768020a7c6e

And that’s it. Follow the prompts to install plugins, provide admin password etc…

nginx as reverse proxy

In order to access the server with out the need to specify the port 8080 and use a DNS name, we’ll have to use nginx that will listen on port 80 or 443 and redirect the traffic to 8080. Let’s install and configure it. In my case, I’ll access the server as jenkins.domain.com.

yum -y install epel-release
yum -y install nginx
systemctl enable nginx

Edit /etc/nginx/nginx.conf and delete everything after this line around line 36. The include line should stay and the last line “}” should stay as well. So, the last lines should look like this.

       include /etc/nginx/conf.d/*.conf;
}

Go to /etc/nginx/conf.d folder and create a new file called jenkins.conf.
This is how my jenkins.conf file looks like. Change the domain in the highlighted line.

server {
    listen 80;
    server_name jenkins.domain.com;

    location / {
        proxy_set_header   X-Forwarded-For $remote_addr;
        proxy_set_header   Host $http_host;
        proxy_pass         "http://127.0.0.1:8080";
    }
}

Don’t forget to open port 80 in your firewall.

firewall-cmd --permanent --zone=public --add-port=80/tcp
firewall-cmd --reload

If you have SELinux, you’ll have to allow HTTP traffic.

setsebool -P httpd_can_network_connect 1

And restart nginx for changes to take effect.

systemctl restart nginx

Now, you can access the server as jenkins.domain.com.
For HTTPS traffic, the configuration is different. You’ll need certificates specified in lines 15 and 16 (under /etc/nginx) and the domain changed in 7,13 and 31. Here is the config (/etc/nginx/conf.d/jenkins.conf) in order to access jenkins over SSL.

upstream jenkins {
  server 127.0.0.1:8080 fail_timeout=0;
}

server {
  listen 80;
  server_name jenkins.domain.com;
  return 301 https://$host$request_uri;
}

server {
  listen 443 ssl;
  server_name jenkins.domain.com;

  ssl_certificate public_cert.pem;
  ssl_certificate_key private_cert.key;
  client_max_body_size 4M;

  location / {
    proxy_set_header        Host $host:$server_port;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto $scheme;
    proxy_redirect http:// https://;
    proxy_pass              http://jenkins;
    # Required for new HTTP-based CLI
    proxy_http_version 1.1;
    proxy_request_buffering off;
    proxy_buffering off; # Required for HTTP-based CLI to work over SSL
    # workaround for https://issues.jenkins-ci.org/browse/JENKINS-45651
    add_header 'X-SSH-Endpoint' 'jenkins.domain.com:50022' always;
  }
}

Make sure you open port 443 on the firewall as we did for port 80 and change SELinux with setsebool.

firewall-cmd --permanent --zone=public --add-port=443/tcp
firewall-cmd --reload

SELinux change.

setsebool -P httpd_can_network_connect 1

You should also restart nginx for the changes to take effect.

systemctl restart nginx

That’s it. You should be able to access Jenkins as jenkins.domain.com over https.

FreeBSD

The installation on FreeBSD is a little bit different. You can install Jenkins from the packages. It will install JDK as well.

pkg install jenkins

The OpenJDK installation depends on these “memory filesystems”. Mount them now.

mount -t fdescfs fdesc /dev/fd
mount -t procfs proc /proc

…and make sure they are mounted on boot.

cat <> /etc/fstab
fdesc   /dev/fd         fdescfs         rw      0       0
proc    /proc           procfs          rw      0       0
EOF

We need to make sure jenkins starts on boot.

sysrc jenkins_enable=YES

Now, we can start jenkins.

service jenkins start

Go to http://[IP]:8180/jenkins and you should see the welcome screen. Replace the [IP] with the IP address of your server. You should see this screen.

Cat out that file, copy & paste the value in the browser and click Continue.

cat /usr/local/jenkins/secrets/initialAdminPassword
c33a584df234433d9a88e19d8e14c289

And that’s it. Follow the prompts to install plugins, provide admin password etc…The jenkins installation on FreeBSD is accessed with a suffix /jenkins. If you try to access it without the suffix, you’ll get this error.

HTTP ERROR 404
Problem accessing /. Reason:
    Not Found

It’s much easier to do the redirects with nginx without the suffix, so we’ll change it.
Edit this file /usr/local/etc/rc.d/jenkins and around line 54 find this.

: ${jenkins_args="--webroot=${jenkins_home}/war --httpPort=8180 --prefix=/jenkins"}

Change it so it looks like this.

: ${jenkins_args="--webroot=${jenkins_home}/war --httpPort=8180 --prefix=/"}

Restart Jenkins and you’ll be able to access it as http://[IP]:8180.

service jenkins restart

nginx as reverse proxy

In order to access the server with out the need to specify the port 8180 and use a DNS name, we’ll have to use nginx that will listen on port 80 or 443 and redirect the traffic to 8180. Let’s install and configure nginx. In my case, I’ll access the server as jenkins.domain.com.

pkg install nginx

Run on boot.

sysrc nginx_enable=YES

Edit /usr/local/etc/nginx/nginx.conf and delete everything after this line around line 39 after #gzip on directive. Add the include line, so the last three lines look like this.

    #gzip  on;
    include /usr/local/etc/nginx/conf.d/*.conf;
}

Create a new conf.d folder mkdir /usr/local/etc/nginx/conf.d and create a new file called jenkins.conf.
This is how my jenkins.conf file looks like. Change the domain in the highlighted line.

server {
    listen 80;
    server_name jenkins.domain.com;

    location / {
        proxy_set_header   X-Forwarded-For $remote_addr;
        proxy_set_header   Host $http_host;
        proxy_pass         "http://127.0.0.1:8180";
    }
}

Restart Jenkins and you should be able to access the site now with http://jenkins.domain.com.
For HTTPS traffic, the configuration is different. You’ll need certificates specified in lines 15 and 16 (under /usr/local/etc/nginx) and the domain changed in 7,13 and 31. Here is the config (/usr/local/etc/nginx/conf.d/jenkins.conf) in order to access jenkins over SSL.

upstream jenkins {
  server 127.0.0.1:8180 fail_timeout=0;
}

server {
  listen 80;
  server_name jenkins.domain.com;
  return 301 https://$host$request_uri;
}

server {
  listen 443 ssl;
  server_name jenkins.domain.com;

  ssl_certificate public_cert.pem;
  ssl_certificate_key private_cert.key;
  client_max_body_size 4M;

  location / {
    proxy_set_header        Host $host:$server_port;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto $scheme;
    proxy_redirect http:// https://;
    proxy_pass              http://jenkins;
    # Required for new HTTP-based CLI
    proxy_http_version 1.1;
    proxy_request_buffering off;
    proxy_buffering off; # Required for HTTP-based CLI to work over SSL
    # workaround for https://issues.jenkins-ci.org/browse/JENKINS-45651
    add_header 'X-SSH-Endpoint' 'jenkins.domain.com:50022' always;
  }
}

You should also restart nginx for the changes to take effect.

service nginx restart

That’s it. You should be able to access Jenkins as jenkins.domain.com over https.

Alibaba Cloud, CentOS: Build WordPress site using NFS, RDS and Load Balancers

Kliment Andreev — Mon, 15 Apr 2019 16:50:40 +0000

I got an invitation to attend an event in NYC for the Alibaba cloud, so I decided to play a little bit with their cloud offering. I was actually surprised to see that they are doing a really good job. While many features are still not there, the ease and simplicity of Alibaba cloud is a plus. If you are familiar with AWS, it won’t take you more than 5 minutes to feel comfortable. The interface, although different is very intuitive. You can get a free trial (a credit card and phone verification are needed) and start playing. Visit this link to start. Once you open an account and verify it, you’ll be presented with a console.
I’ve decided to demonstrate how to start in Alibaba Cloud by building a WordPress site running on two instances behind a load balancer. The database will be a MySQL RDS offering from Alibaba. They still don’t offer NFS filesystem, so I had to create an instance for that. First thing first.

VPC

The ubiquitous VPCs from AWS are named the same in Alibaba and they serve the same purpose. From the top menu choose a region, from the left click on Virtual Private Cloud and then click on Create VPC.

I’ll name my VPC, TestVPC and the CIDR (size of the VPC in terms of available IPs and subnets) is going to be 192.168.0.0/16 (which is default).

If you scroll down, you’ll see that you’ll also have to create a VSwitch. A VSwitch is nothing by a subnet or availability zone (AZ) in AWS terms. My first one will be called VSwitchA in Virginia Zone A and I’ll carve a smaller subnet 192.168.1.0/24 for that.

Let’s create another one, but this time in a different zone. The subnets can’t overlap. On the left side, you’ll see the menu for VSwitches, so click there and then click on Create VSwitch.

Do pretty much the same as previously. Choose the right VPC this time, name the VSwitch, choose the second zone and choose a different subnet. In my case it’s 192.168.2.0/24.

Security Groups

We’ll need several Security Groups. From the top left menu, click on Security Groups and the click on Create Security Group from the top right.

For the Load Balancer, we’ll need a security group that will allow port 80 from everywhere (0.0.0.0/0). For the two instances that will run Apache and PHP, we’ll allow port 22 only from specific addresses and port 80 allowed from the load balancer only. We don’t want the actual instances to be exposed on port 80 on the Internet. For the RDS instance, we’ll create a security group that allows port 3306 (MySQL) from the two instances only. Finally, for the NFS server, we’ll also allow access from the two instances only on port 2049 (TCP).
Here is an example of what I did for the first security group. I choose a custom template, named it sgLoadBalancer, added a description and chose my TestVPC.

You can’t create the rules like you do in AWS, but once you create the security group, you’ll see a pop-up reminding you to create the rules. Click on Create Rules Now.

This is how my rule for HTTP traffic looks like.

Name the second security group sgInstances, but before you create the group copy the security group ID for the first group that you created.

When you add the rules for the 2nd security group for the instances, make sure that you change the Authorization Type parameter to Security Group instead of IPv4 CID Block. This way you can specify that another security group has access to a specific port, not an IP. We don’t know the IP of the load balancer and it probably changes from time to time.

You have to create another rule for the instances. This time for the port 22. I chose to restrict port 22 to my home IP only. This is how those rules actually look like once created.

For the next two security group, copy the security group ID of the newly created group sgInstances.
Create two new security groups, sgRDS and sgNFS. For sgRDS allow port 3306 from the security group ID of sgInstances and for sgNFS allow port 2049 for the same security group ID of sgInstances.
At the end you should have 4 security groups: sgLoadBalancer, sgInstances, sgRDS and sgNFS. These are the rules.

And these security groups should be assigned to these resources.

SSH Keys

Before we start provisioning the servers, let’s create a SSH key pair so we can access them. The menu for the creation of SSH keys is bellow Security Groups.

Click on Create SSH Key Pair in the top right corner. You can import or create your own key pair. I chose to create the key pair.

As a result you’ll get a certificate file (.pem). Keep this file somewhere safe. If you use putty to connect to your instances, you’ll have to convert this file to ppk using puttygen.

NFS instance

From the menu, go to Instances and then from the top right, click Create Instance.

I am going with Pay-As-You-Go model and Random zone. Choose your instance type. The cheapest are under Entry-Level (Shared). I went with 1 CPU and 1GB RAM and 20GB disk on CentOS 7.6 x-64. Looks like 20GB is minimum, I couldn’t choose anything below 20GB for the disk. Click Next: Networking when you are done here.

On the Networking screen, I have these settings. The defaults for the VPC and the zone, click to Assign public IP and click to select a security group. For some reason, I was able to select only one security group, so select sgInstances as a security group.

If you click next, you’ll see the System Configuration page. Specify the key that you’ve selected, name the instance so it shows up properly in the console, add a description if you want and enter the hostname (optional). From here, you can accept the default and next-through all the way to the end. Alibaba will create the instance for you in 1-3 minutes.

After the instance is create, go back to the Security Groups menu and find the sgNFS group. Click on Manage Instances.

From the top right, click on Add Instance and add the nfs instance. This way, we’ll have both sgInstances and sgNFS securty groups attached to the nfs server.
Go back to your instances and you’ll see the instance is up and running. SSH to its public IP and specify the SSH key to log in. Your username is root and the ppk file is your password.

Once you are in, you’ll be greeted with the Alibaba banner.

Now you are ready to install the NFS filesystem that will be used as a shared disk for both WordPress instances. I already have a small tutorial for this, so go to this link and do the server part. You can skip the part for the firewall (last two commands). The firewall is not enabled by default on Alibaba CentOS. SELinux is also disabled. Once you are done, we have to download WordPress and extract it under the nfs share. Then, we we build the webservers we’ll mount this directory and point our webserver there.

cd /nfs
wget https://www.wordpress.org/latest.tar.gz
tar xzvf latest.tar.gz --strip 1

RDS

We’ll use the PaaS offering from Alibaba for our MySQL database. From the menu on the left, click AsparaDB for RDS.

Click Create Instance in the top right corner. This is what I chose for my database. Make sure Pay-As-You-Go is selected and then choose the RDS instance in the same region as your other instances. The engine is MySQL 5.7 with no HA. I chose a small 1 CPU/1GB DB instance with 20GB storage.

Click Buy Now and then Pay Now when you are done. If you go to the console, you won’t see your database. Just wait for a minute or two and it will show up with a status of Creating. After 3-5 mins, once the status changes to Running, click on the database. Let’s assign a security group first. Click on Basic Infomation on the left and then click on Set Whitelist.

Look at the bottom right, where it says Add to Security Group. The names won’t show up, so get your security group ID for sgRDS from the Security Groups menu first. Unlike AWS, you can assign only one security group per DB instance. If you get an error that this operation is not supported, do not assign a security group. Just whitelist the VPC subnet. Click Add a Whitelist Group and add the VPC subnet 192.168.0.0/16.

Now, go to the Accounts menu on the left and click Create Account on the right. I’ve created a superuser called dbmaster.

Again, go to Databases menu on the left and click Create Database on the right. Type the name of the database dbwordpress and click to create a new account. You’ll be back at the accounts menu. This time create a user called wordpress and make it a standard user. It’s kind of kludgy, but now you have to go back to the Databases menu, click Create Database again, fill the name dbwordpress again, but this time you can choose the account that you just created.

Finally, go to the Basic Information menu and choose the menu next to Intranet address. Here you have to create a whitelist group of IPs that can access the database. By default, only 127.0.0.1 is allowed. We have to enter our VPC CIDR.

Once you do that, you’ll see your RDS hostname.

WordPress instances

We’ll create two instances, install Apache + PHP, mount the NFS system and configure the web server. First, create two instances, each in a different zone. Use the section above for the NFS server. Name the servers web1 and web2 and assign the sgInstances security group to both of them. Make sure you choose Pay-As-You-Go model and assign public IPs so you can log in. And choose the SSH key pair that we created. Log to both of the instances.
First, mount the NFS system. Replace the IP with the IP of your NFS server in the mount command.

yum -y install nfs-utils
mount 192.168.1.91:/nfs /var/www/html
echo "192.168.1.91:/nfs       /var/www/html    nfs     defaults 0 0" >> /etc/fstab
chown -R apache:apache /var/www/html

Then, check if you can reach the RDS MySQL server. You should receive some garbage.

yum -y install telnet
telnet rm-2evrh4h3aq9tk800z.mysql.rds.aliyuncs.com 3306
Trying 172.16.1.41...
Connected to rm-2evrh4h3aq9tk800z.mysql.rds.aliyuncs.com.
Escape character is '^]'.
N
5.7.20-log▒F5;I!]:p]H3R[^mysql_native_password

Second, install Apache and all PHP modules on both servers. I’ll install PHP 5.4 because it’s already in the default repo. You should install PHP 7.x, it’s a little bit more work.

yum -y install httpd php php-common php-mysql php-gd php-xml php-mbstring php-mcrypt
systemctl enable httpd
systemctl start httpd

Now, we have to configure the Apache server. It’s a simple change in /etc/httpd/conf/httpd.conf file. You’ll probably want to use virtual hosts, but in interest of keeping this short, I’ll use a single site config. Find the directive that says #ServerName www.example.com:80 and remove the comment.

sed -i '/ServerName www/s/^#//g' /etc/httpd/conf/httpd.conf

Load Balancer

From the menu on the left, click on Server Load Balancer.

Once the screen shows up, click on Create SLB Instance. Then choose the specs. Make sure it is in the same region with your instances and in the same zones (VSWitches). Name the load balancer too, e.g. lbWordPress.

Once it’s ready, click on Add Backend Servers.

From the list of the available servers, choose web1 and web2 in the Default Server Group.

Click on Next: Set Weight and Port button and click OK to accept the defaults. Go back to the load balancer and this time click Configure Listener (above Add Backend Servers).

Choose HTTP traffic, port 80, click Modify next to Advanced and Enable Session Persistence. Click Next.

On this screen click on Default Server Group, type port 80 for the servers and click Next. You can accept the defaults in the next screen or make a change if you like.

Open a browser and go to your load balancer IP. It might take up a minute for the LB to configure so be patient if your browser is just sitting there.

If you used my username and database names, fill out the config page like this.

…and pretty much that’s it.

CentOS: Upgrade PostgreSQL from 9.2 to 9.6

Kliment Andreev — Tue, 05 Feb 2019 00:12:37 +0000

If you’ve installed BitBucket on a CentOS 7 server and use PostgreSQL as the back-end database, you’ve probably seen this nagging warning that support for PostgreSQL 9.2.24 has been deprecated and will be removed in an upcoming release.

While PostgeSQL 9.2 is quite old and they are already announcing version 12, mind that any of the Atlassian stack can’t support anything more than version 9.6 at the time of this writing.
So, I’ll explain how to upgrade PostgreSQL on CentOS from 9.2.24 to PostgreSQL 9.6. While this post is for CentOS, it might work for other distros too. There are some prerequisites though. First, I assume that you installed PostgreSQL using yum from the default repo. In this case, your database most likely resides under /var/lib/pgsql/data and the binaries are under /usr/bin/psql. If you use the default version, then you probably use systemctl start/stop postgresql to manage the start/stop of the database daemon. If you have something different, the following tutorial will probably work, but you’ll have to adjust some directories.
First thing first, let’s check the version of PostgreSQL that we use.

psql --version
psql (PostgreSQL) 9.2.24

Next, stop any services that use the database, such as BitBucket, Jira or Confluence or whatever you have there.
Log as the postgres user that you use to manage the database. By the default this user is postgres.
Do a full backup and move the backup somewhere because the home directory for the postgres user is where the database home is. You can compress the SQL file first. It’s just a plain text file.

pg_dumpall > backup.sql
mv backup.sql somewhere/

Once done, stop the postgreSQL daemon, but log as root first. You won’t be able to stop the daemon using the postgres user.

systemctl stop postgresql

Go to a temp folder and get the RPM package from the PostgreSQL site. In case the link doesn’t work, go to the website using a browser and see what’s the latest 9.6 version.

cd /tmp
wget https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-centos96-9.6-3.noarch.rpm

Install the newer version.

rpm -ivh pgdg-centos96-9.6-3.noarch.rpm
yum install postgresql96-server

You’ll get some warning that symlinks can’t be created but you can ignore them. The new version installs the binaries under /usr/psql-9.6/bin.
Log as the postgres user and initialize the new database.

/usr/pgsql-9.6/bin/initdb -D /var/lib/pgsql/9.6/data/
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/pgsql/9.6/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    /usr/pgsql-9.6/bin/pg_ctl -D /var/lib/pgsql/9.6/data/ -l logfile start

Ignore the line that says that we can start the server. We have to upgrade the current database first. Make sure that the directories below are the same as yours. If not, make changes.

/usr/pgsql-9.6/bin/pg_upgrade --old-datadir /var/lib/pgsql/data/ --new-datadir /var/lib/pgsql/9.6/data/ --old-bindir /usr/bin/ --new-bindir /usr/pgsql-9.6/bin/
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok

*failure*
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

connection to database failed: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/lib/pgsql/.s.PGSQL.50432"?


could not connect to old postmaster started with the command:
"/usr/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data/" -o "-p 50432 -b  -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directory='/var/lib/pgsql'" start
Failure, exiting

Oh boy. If you check the error message, you’ll get some more info.

tail /var/lib/pgsql/pg_upgrade_server.log
server stopped


command: "/usr/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data/" -o "-p 50432 -b  -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directory='/var/lib/pgsql'" start >> "pg_upgrade_server.log" 2>&1
waiting for server to start....FATAL:  unrecognized configuration parameter "unix_socket_directory"
 stopped waiting
pg_ctl: could not start server
Examine the log output.

The reason being that as of PostgreSQL version 9.3 the config parameter unix_socket_directory has been replaced with unix_socket_directories. There is a workaround thanks to this guy. The link for more info is here.
Pretty much, all you have to do is execute these lines first. Execute these lines as root or sudo user.

mv /usr/bin/pg_ctl{,-orig}
echo '#!/bin/bash' > /usr/bin/pg_ctl
echo '"$0"-orig "${@/unix_socket_directory/unix_socket_directories}"' >> /usr/bin/pg_ctl
chmod +x /usr/bin/pg_ctl

Now you can do the upgrade.

/usr/pgsql-9.6/bin/pg_upgrade --old-datadir /var/lib/pgsql/data/ --new-datadir /var/lib/pgsql/9.6/data/ --old-bindir /usr/bin/ --new-bindir /usr/pgsql-9.6/bin/
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* system OID user data types                ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for roles starting with 'pg_'                      ok
Checking for invalid "line" user columns                    ok
Creating dump of global objects                             ok
Creating dump of database schemas
                                                            ok
Checking for presence of required libraries                 ok
Checking database user is the install user                  ok
Checking for prepared transactions                          ok

If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.

Performing Upgrade
------------------
Analyzing all rows in the new cluster                       ok
Freezing all rows on the new cluster                        ok
Deleting files from new pg_clog                             ok
Copying old pg_clog to new server                           ok
Setting next transaction ID and epoch for new cluster       ok
Deleting files from new pg_multixact/offsets                ok
Setting oldest multixact ID on new cluster                  ok
Resetting WAL archives                                      ok
Setting frozenxid and minmxid counters in new cluster       ok
Restoring global objects in the new cluster                 ok
Restoring database schemas in the new cluster
                                                            ok
Setting minmxid counter in new cluster                      ok
Copying user relation files
                                                            ok
Setting next OID for new cluster                            ok
Sync data directory to disk                                 ok
Creating script to analyze new cluster                      ok
Creating script to delete old cluster                       ok

Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade so,
once you start the new server, consider running:
    ./analyze_new_cluster.sh

Running this script will delete the old cluster's data files:
    ./delete_old_cluster.sh

Start the new database now and check everything. As you can see, the startup script is different. This time we have to specify the version of PostgreSQL. As root do:

systemctl start postgresql-9.6

You can revert the change back.

mv -f /usr/bin/pg_ctl{-orig,}

Log as postgres user and do a check up. It might take some time if your database is large.

./analyze_new_cluster.sh

You can delete the old database if you want.

./delete_old_cluster.sh

If you check the new version, you’ll see that it’s the same.

psql --version
psql (PostgreSQL) 9.2.24

The database is upgraded and running 9.6, it’s just the psql binary that’s the old version. You have to tell the postgres user to use the new binaries. Delete the old binaries first, uninstall the old version, make sure it doesn’t start on boot and enable the new version. Also, if you have any specific configurations, make sure you compare postgresql.conf and pg_hba.conf under /var/lib/pgsql/data and /var/lib/pgsql/9.6/data. Log as root first. Make sure it says 9.2 when you run yum remove.

systemctl disable postgresql
yum remove postgresql

Enable postgreSQL to boot on start.

systemctl enable postgresql-9.6

Add the following line under .bash_profile for the postgres user.

export PATH=$PATH:/usr/pgsql-9.6/bin

If you don’t have a .bash_profile file, check if there is any .bash_profile.rpmsave. Rename this one as .bash_profile and add the above line. It should look like this.

[ -f /etc/profile ] && source /etc/profile
PGDATA=/var/lib/pgsql/9.6/data
export PGDATA
# If you want to customize your settings,
# Use the file below. This is not overridden
# by the RPMS.
[ -f /var/lib/pgsql/.pgsql_profile ] && source /var/lib/pgsql/.pgsql_profile
export PATH=$PATH:/usr/pgsql-9.6/bin

If you log off, log back on as the postgres user or execute source .bash_profile and check the version, you’ll see that you are all set.

psql --version
psql (PostgreSQL) 9.6.11

CentOS: Apache Guacamole – Remote Desktop Gateway

Kliment Andreev — Fri, 28 Dec 2018 23:06:43 +0000

In this post I’ll explain how to install Apache Guacamole from the source on a CentOS server. Guacamole is a clientless remote gateway that supports VNC, RDP, SSH and telnet protocols. The purpose of the remote gateway is to allow you to connect to a device over the gateway, but not directly, in case you have firewall restrictions between your computer and the end device. Let’s say you have a desktop at work and a server with a public IP at home or in the cloud. The server at home/cloud normally listens on port 3389 for RDP or port 22 for SSH. But your corporate firewall doesn’t allow outgoing traffic on these ports so you can’t connect from your work to your servers. Here comes the remote gateway. This server listens on ports 80 and/or 443 and will tunnel the traffic from your work desktop to your home server over 80 or 443. The remote gateway will need access to your end device on the specified ports for RDP or SSH. Most corporate environments allow port 80 and 443 only, that way you can bypass your corporate firewall. I have another post that describes the same exact scenario using Microsoft RDP Gateway, shellinabox and tunneling over putty.

Install Apache Guacamole

Apache Guacamole has some pre-requisites and we need to install them first. Let’s create a work directory where we can place all files necessary.

mkdir /workdir && cd /workdir
yum -y install wget libpng-devel libjpeg-turbo-devel cairo-devel uuid-devel  epel-release maven tomcat
yum -y install freerdp-devel pango-devel libssh2-devel libtelnet-devel libvncserver-devel 
yum -y install pulseaudio-libs-devel openssl-devel libvorbis-devel libwebp-devel terminus-fonts
systemctl enable tomcat

The FFmpeg library is not in the official or EPEL repo, so we need to install it as RPM package.

rpm --import http://li.nux.ro/download/nux/RPM-GPG-KEY-nux.ro
rpm -Uvh http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-5.el7.nux.noarch.rpm
yum -y install ffmpeg-devel

We also need some compiler tools.

yum -y groupinstall "Development Tools"

Guacamole comes in two parts, a server and a client. The client is a Java servlet running on Tomcat, but both are installed on a server. Don’t get confused, the Guacamole client has nothing to do with the end user client. You don’t need to install any clients on your desktop. Just a plain HTML5 browser.
Let’s download the server first.

wget -O server.tar.gz "http://archive.apache.org/dist/guacamole/0.9.14/source/guacamole-server-0.9.14.tar.gz"

Untar the tarball and configure the server before compiling it.

tar xzvf server.tar.gz
cd guacamole-server-0.9.14
./configure --with-init-dir=/etc/init.d

You should see something like this as the output. Make sure it matches. All of the components except winsock should be installed. If not, check the first yum install commands above and see what went wrong.
NOTE: Some newer versions of CentOS 7 now come with freerdp 2.0 as a default package. If you see that freerdp sections says no, it’s because you have freerdp 2.0 installed. Install freerdp1.2 with yum -y install freerdp1.2.

Library status:

     freerdp ............. yes
     pango ............... yes
     libavcodec .......... yes
     libavutil ........... yes
     libssh2 ............. yes
     libssl .............. yes
     libswscale .......... yes
     libtelnet ........... yes
     libVNCServer ........ yes
     libvorbis ........... yes
     libpulse ............ yes
     libwebp ............. yes
     wsock32 ............. no

   Protocol support:

      RDP ....... yes
      SSH ....... yes
      Telnet .... yes
      VNC ....... yes

   Services / tools:

      guacd ...... yes
      guacenc .... yes

   Init scripts: /etc/init.d

Type "make" to compile guacamole-server.

If everything looks good, compile and install.

make
make install
ldconfig

Let’s enable the automatic startup for the Guacamole daemon.

systemctl enable guacd

It’s time to install the client. You can get a pre-compiled WAR library for the client or you can compile it yourself. I’ll do the later.

cd /workdir
wget -O client.tar.gz "http://archive.apache.org/dist/guacamole/0.9.14/source/guacamole-client-0.9.14.tar.gz"
tar xzvf client.tar.gz
cd guacamole-client-0.9.14
mvn package

Copy the WAR file under Tomcat.

cp guacamole/target/guacamole-0.9.14.war /var/lib/tomcat/webapps/guacamole.war

MySQL server

In order for the end users to authenticate to Guacamole, we can choose LDAP authentication, file based authentication in XML file and authentication that’s stored in a SQL database (MySQL, PostgreSQL). While the file based authentication is the easiest to configure, it doesn’t allow configuring different levels of access. LDAP is an overkill, so the best bet is a MySQL server that’s used to store the authentication.
I’ll use MariaDB SQL. Let’s install it and configure it first.

yum -y install mariadb-server
systemctl enable mariadb
systemctl start mariadb

This is the initial configuration. Highlighted lines is what you type, the rest is output.

mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user.  If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] Y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] Y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] Y
 ... Success!

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] Y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] Y
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

We’ll have to create the database now and assign a SQL user with rights to that database. Enter the root SQL password that you’ve previously created and change the database name, user and password to suit your needs.

mysql -u root -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 11
Server version: 5.5.60-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> CREATE DATABASE guacamole_db;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> CREATE USER 'guacamole_user'@'localhost' IDENTIFIED BY 'guacamole_password';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> GRANT SELECT,INSERT,UPDATE,DELETE ON guacamole_db.* TO 'guacamole_user'@'localhost';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> quit
Bye

We also need a JDBC auth plugin for MySQL, so Guacamole can talk to MySQL. You’ll be prompted for the MySQL root password at the end of the cat command.

cd /workdir
wget -O auth.tar.gz "http://archive.apache.org/dist/guacamole/0.9.14/binary/guacamole-auth-jdbc-0.9.14.tar.gz"
tar xzvf auth.tar.gz
cd guacamole-auth-jdbc-0.9.14/mysql/schema/
cat *.sql | mysql -u root -p guacamole_db

We have to copy the JAR archive to a specific location so Guacamole can find it.

cd /workdir
mkdir -p /etc/guacamole/{extensions,lib}
cp guacamole-auth-jdbc-0.9.14/mysql/guacamole-auth-jdbc-mysql-0.9.14.jar /etc/guacamole/extensions

We also need a MySQL connector for Java, so the Java servlets can talk to MySQL.

cd /workdir
wget -O conn.tar.gz https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.13.tar.gz
tar xzvf conn.tar.gz
cp mysql-connector-java-8.0.13/mysql-connector-java-8.0.13.jar /etc/guacamole/lib/

Finally, we have to tell Guacamole that we’ll use MySQL. Create a file called guacamole.properties under /etc/guacamole directory. Make sure you have the correct database name, user and password for MySQL when you created the database.

cat < /etc/guacamole/guacamole.properties
# Hostname and port of guacamole proxy
guacd-hostname: localhost
guacd-port:     4822
# MySQL properties
mysql-hostname: localhost
mysql-port: 3306
mysql-database: guacamole_db
mysql-username: guacamole_user
mysql-password: guacamole_password
mysql-default-max-connections-per-user: 0
mysql-default-max-group-connections-per-user: 0
EOF

Start the application

Now it’s time to start Guacamole.

systemctl start guacd
systemctl start tomcat

By default, Tomcat apps use port 8080, so if you have a firewall, you have to open the port.

firewall-cmd --zone=public --add-port=8080/tcp --permanent
firewall-cmd --reload

If you use SELinux, you have to allow the HTTP traffic.

setsebool -P httpd_can_network_connect 1

At this point you can go and test Guacamole by going to http://guacamole_ip:8080/guacamole. The default admin user and password is guacadmin.
NOTE: I strongly suggest that you change the password or even create a new admin user.
When I created a test connection, everything worked fine, but I was stuck at the remote session and couldn’t go back to Guacamole. Every time I went to http://guacamole_ip:8080/guacamole, the browser opened the session. If this happens to you, make sure you have at least two sessions configured or simply go to http://guacamole_ip:8080/guacamole/#settings/sessions URL.

nginx as reverse proxy

Using nginx as reverse proxy is optional, but initially I said that most corporate environments block everything except port 80 and 443. Guacamole runs on port 8080 which means it will be blocked as well. You can probably change the port from 8080 to 80, but then you have to reconfigure Tomcat and there might be some issues because everything running on ports 1024 and below requires root privileges. So, I’ll do what everyone else does, use a reverse proxy.
The purpose of nginx will be to listen on port 80 or port 443 and forward that traffic locally to Tomcat on 8080.
Let’s install and configure nginx.

yum -y install nginx
systemctl enable nginx
systemctl start nginx

Open /etc/nginx/nginx.conf and remove the section with the server directive. So, this section has to go. Comment it or delete it, then save the file.

Now, create a new nginx config file.

cat < /etc/nginx/conf.d/guacamole.conf
server {
    listen 80;
    server_name localhost;
        location /guacamole/ {
                proxy_pass http://localhost:8080/guacamole/;
                proxy_buffering off;
                proxy_http_version 1.1;
                proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
                proxy_set_header Upgrade \$http_upgrade;
                proxy_set_header Connection \$http_connection;
                access_log off;
        }
}
EOF

Reload the nginx config and you can access your server as http://guacamole_ip/guacamole now.

systemctl restart nginx

Make sure you open the firewall if you have it installed.

firewall-cmd --zone=public --add-port=80/tcp --permanent
firewall-cmd --reload

If you want to go even further and use a specific URL and SSL certificates, then do this. Let’s say you want to access your Guacamole install using https://guac.domain.com. In that case, your guacamole.conf should look like this.

cat < /etc/nginx/conf.d/guacamole.conf
server {
    listen 80;
    server_name guac.domain.com;
        return 301 https://\$host\$request_uri;
}
server {
    listen 443 ssl;
    server_name guac.domain.com;
    ssl_certificate star.domain.com.crt;
    ssl_certificate_key star.domain.com.key;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;

    access_log off;
    location / {
        proxy_pass http://localhost:8080/guacamole/;
        proxy_buffering off;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
        proxy_set_header Upgrade \$http_upgrade;
        proxy_set_header Connection \$http_connection;
    }
}
EOF

Put the certificates under /etc/nginx directory, allow the port 443 on the firewall and restart nginx.

firewall-cmd --zone=public --add-port=443/tcp --permanent
firewall-cmd --reload
systemctl restart nginx

For this to work, you’ll need guac.domain.com to point to your Guacamole server IP in DNS and you’ll need the SSL certificate and private key under /etc/nginx directory.
Finally, when everything looks OK, get rid of the workdir.

rm -Rf /workdir

CentOS: Install an authoritative DNS server (BIND)

Kliment Andreev — Wed, 19 Dec 2018 20:50:09 +0000

In this post I’ll explain how to install and configure BIND DNS server to act as an authoritative server for a public domain in a master/slave configuration. This is not a setup for a server that will act as a DNS server in your local environment and does DNS resolution for your local network. In this post, the DNS server won’t resolve any DNS queries except for the domain that it is authoritative. The domain that I’ll use is cloudwerk.us registered with Go Daddy, so some of the configurations will apply to Go Daddy only. The IPs and keys that I’ll use will be visible. The setup will be gone by the time you read this.
So, I have my master server, ns1.cloudwerk.us (67.205.160.87) and the slave server ns2.cloudwerk.us (104.248.184.142).
I will be using CentOS 7.5 and BIND 9.9.4-72. Before you start, make sure you have your domain purchased and you have 2 servers with public IPs up and running. You can leave the default registrant’s DNS servers for now.

Installation, startup and control

First, I’ll install BIND on both servers. All commands below are executed as root.

yum -y install bind

Then, we have to configure the rndc (Remote Name Daemon Control) utility. This utility allows management of the named (DNS) daemon. Moreover, you can manage remote DNS servers and I’ll explain how. So, let’s create the keys (highlighted line is what you type, the rest is output). DO this on the master server only.

rndc-confgen -a -r /dev/urandom 
wrote key file "/etc/rndc.key"

As you can see, there is a file called rndc.key created under /etc directory. This is the default directory where BIND expects to find its config files. The zone files are under /var.
Now, copy this file to the second server in the same location. You can use scp, copy & paste in a terminal session, it doesn’t matter. What matters is that you have to change the permissions and the owner of the file. BIND runs under the named user, not root. Do this on both servers.

cd /etc
chown root:named rndc.key
chmod 640 rndc.key

Let’s make sure that BIND starts on boot. On both servers, execute:

systemctl enable named

Now, we can start the BIND daemon on both.

systemctl start named

Check the logs.

tail /var/log/messages
Dec 19 16:33:40 master named[1308]: zone 1.0.0.127.in-addr.arpa/IN: loaded serial 0
Dec 19 16:33:40 master named[1308]: zone localhost.localdomain/IN: loaded serial 0
Dec 19 16:33:40 master named[1308]: zone 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa/IN: loaded serial 0
Dec 19 16:33:40 master named[1308]: all zones loaded
Dec 19 16:33:40 master named[1308]: running
Dec 19 16:33:40 master systemd: Started Berkeley Internet Name Domain (DNS).
Dec 19 16:33:40 master systemd: Reached target Host and Network Name Lookups.
Dec 19 16:33:40 master systemd: Starting Host and Network Name Lookups.
Dec 19 16:33:40 master named[1308]: network unreachable resolving './DNSKEY/IN': 2001:500:84::b#53
Dec 19 16:33:40 master named[1308]: network unreachable resolving './NS/IN': 2001:500:84::b#53

We can’t start the server with the rndc utility, but we can stop it if needed with rndc stop or simply systemctl stop named.
To control the slave server from the master, we have to modify the main BIND configuration file, named.conf on the slave server. Also, the keys have to be the same, but we already took care of that part above.
So, in the same /etc directory, edit named.conf and add these lines, right before the options directive. Do this on the slave server, not the master.

include "/etc/rndc.key";

controls {
        inet 127.0.0.1 allow { localhost; } keys { "rndc-key"; };
        inet 104.248.184.142 allow { 67.205.160.87; } keys { "rndc-key"; };
};

If you look at my IPs above, this means that the rndc utility will be accepted from the localhost and from the the IP 67.205.160.87 which is the master server. But, we also have to tell BIND to listen on the public IP. By default, it listens on the localhost only.
Do this on BOTH servers. In named.conf find the options directive and modify the listen-on option so it looks like this. You should put the public IP of each server there, so the line will differ on both servers.

//Master server
listen-on       { 127.0.0.1; 67.205.160.87; };

//Slave server
listen-on       { 127.0.0.1; 104.248.184.142; };

Allow traffic on the firewalls if you have them enabled. Do this on both.

firewall-cmd --add-port=53/tcp --permanent 
firewall-cmd --add-port=53/udp --permanent

On the master node, do this.

firewall-cmd --permanent --zone=public --add-rich-rule=' 
  rule family="ipv4" 
  source address="104.248.184.142/32" 
  port protocol="tcp" port="953" accept'

On the slave node, do this.

firewall-cmd --permanent --zone=public --add-rich-rule=' 
  rule family="ipv4" 
  source address="67.205.160.87/32" 
  port protocol="tcp" port="953" accept'

Reload the rules on both.

firewall-cmd --reload

Restart BIND on both.

systemctl restart named

Now, you can manage them with rndc.

rndc reload
rndc status

From the master server, try to reload the slave server.

rndc -s 104.248.184.142 reload
server reload successful

What we also want to do is tell the OS that from now on, the default DNS server will be our local DNS.
So, edit /etc/resolv.conf and add this line as the first line on both.

nameserver 127.0.0.1

If we try to ping google.com we should get a response or at least an IP resolved. That’s good. From another computer, try to use any of our servers as a DNS. This is what you should get: Query refused. That’s good too, because we don’t want our server to be used as a DNS server for other domains. We want to access the Internet from the server, e.g. to update the server, so we need to be able to use the local server to resolve any domains but only from localhost.

C:\Users\klimenta>nslookup
Default Server:  google-public-dns-a.google.com
Address:  8.8.8.8

> server 67.205.160.87
Default Server:  [67.205.160.87]
Address:  67.205.160.87

> www.google.com
Server:  [67.205.160.87]
Address:  67.205.160.87

*** [67.205.160.87] can't find www.google.com: Query refused

You can allow this server to act as a forwarder by including recursion yes; option under options and allow-access to all IPs (0.0.0.0/0), but that’s not the intended use of an authoritative DNS server.

Zone file

Now that we have the servers up and running, let’s create the zone file. The zone file tells where the records can be found. On the master server only, edit named.conf and add these two lines right after the listen-on option. We are allowing the transfer of the zones to the slave server only

allow-transfer { localhost; 104.248.184.142; };
notify yes;

Go all the way at the bottom and add these line to define the zone file.
NOTE: If you have SELinux enabled, you can’t define your zone anywhere. Make sure you use /var/named for your zone file.).

zone "cloudwerk.us" {
        type master;
        file "/var/named/db.cloudwerk.us";
};

Now, go to the /var/named directory and create the zone file to look like this. The file should be called db.cloudwerk.us. Do this on the master server only. You might want to read the following link if you are not familiar with the syntax of a zone file.

$TTL    604800
@       IN      SOA     ns1.cloudwerk.us.       admin.cloudwerk.us. (
                        2018121901;     Serial
                        3H;             Refresh
                        15M;            Retry
                        2W;             Expiry
                        1D );           Minimum

; name servers - NS records
        IN      NS      ns1.cloudwerk.us.
        IN      NS      ns2.cloudwerk.us.

; name servers - A records
ns1.cloudwerk.us.       IN      A       67.205.160.87
ns2.cloudwerk.us.       IN      A       104.248.184.142

; other servers - A records
www.cloudwerk.us.       IN      A       52.71.63.84

All you have to do now is to reload the new config with rndc reload. At this point we have the master server up and running. Now, we have to configure the slave server. Edit the named.conf file and add these lines at the bottom. Very self-explanatory.

zone "cloudwerk.us" {
        type slave;
        masters { 67.205.160.87; };
        file "/var/named/slaves/db.cloudwerk.us";
};

Restart the named daemon and if you go to /var/named/slaves directory, you’ll see that the zone file replicated from the master. Finally, on both servers you have to allow the queries from the Internet. Edit named.conf and comment this line.

//allow-query     { localhost; };

Reload with rndc reload on both after.
At this point, we have both the master and the slave servers configured, up and running. But no one on the Internet knows that our servers are authoritative servers. You have to make this change at your domain registrant. In my case, I’ve purchased the domain from Go Daddy, so I went to the DNS management panel and added two A records for ns1.cloudwerk.us and ns2.cloudwerk.us. Then, at the bottom of the web control panel, I’ve changed the DNS from xy.domaincontrol.com to custom and added my servers. I’ve encountered some issues, but the on-line chat with the support resolved this in 5 minutes. In essence, if you open up the DNS console, scroll all the way down and under Advanced Features, you’ll see Host names link. Click on this one and create two records, ns1 and ns2 with your IPs. Now, you can switch to Custom Nameserves on the same page.
Leave some time for replication. It might take up to 24 hrs, but in my case it took less than 20 mins.
You should know that you are in charge when these commands return your servers. In Windows:

C:\Users\klimenta>nslookup
Default Server:  google-public-dns-a.google.com
Address:  8.8.8.8

> set query=soa
> cloudwerk.us
Server:  google-public-dns-a.google.com
Address:  8.8.8.8

Non-authoritative answer:
cloudwerk.us
        primary name server = ns1.cloudwerk.us
        responsible mail addr = admin.cloudwerk.us
        serial  = 2018121901
        refresh = 10800 (3 hours)
        retry   = 900 (15 mins)
        expire  = 1209600 (14 days)
        default TTL = 86400 (1 day)
> set query=ns
> cloudwerk.us
Server:  google-public-dns-a.google.com
Address:  8.8.8.8

Non-authoritative answer:
cloudwerk.us    nameserver = ns1.cloudwerk.us
cloudwerk.us    nameserver = ns2.cloudwerk.us

In Linux/BSD if you use dig:

dig +short NS cloudwerk.us
ns2.cloudwerk.us.
ns1.cloudwerk.us.

Or, if you want to do it on-line.
This is what I got.

It says that the slave server is leaking the zone which means anyone can list all the records. In order to prevent that, we have to edit named.conf on the slave server and add this line after listen-on option.

allow-transfer { localhost; 67.205.160.87; };

Do rndc reload after and if you re-execute the test, you’ll see that everything looks OK.

DNSSEC (optional)

I won’t go into details of what is DNSSEC, I’ll just explain how to configure and secure your DNS server. It’s pretty much a PKI infrastructure for securing your records by signing them with a private key. If you need more info, you can find tons of info by googling DNSSEC.
On the master server, add this line after dnssec-validation yes; option in named.conf.

key-directory "/var/named/keys";

On the slave server don’t do anything.
Add these lines in the same file (named.conf) on the master server only. They should be below file directive where your zone is defined (at the bottom of the file). These lines are part of your zone config, not separate directives.

auto-dnssec maintain;
inline-signing yes;

Let’s create the KSK and ZSK keys. Master server only.

cd /var/named/keys
dnssec-keygen -r /dev/urandom -a RSASHA256 -b 2048 -f KSK -n ZONE cloudwerk.us
ls -l
echo "Take a note of the key file that ends with .key"
dnssec-keygen -r /dev/urandom -a RSASHA256 -b 2048 -n ZONE cloudwerk.us

You’ll see 4 files created. Scribble down the first key file that was created (.key), see the output from above. Reload the config on the master but make sure BIND can access the keys.

chown -R named:named /var/named/keys
rndc reload

You’ll see two more files (.jbk and .signed) under /var/named. These are the signed files. If you check the slaves directory on the slave node, you’ll see a signed file there as well.
Let’s do a test. Go to this link, type your domain name and hit enter.
You should see something like this.

Now, stop the master server with rndc stop and refresh the web page. The test will take a little bit longer, because NS1 is down, but eventually you’ll get something like this.

That’s normal. It tells us that ns1 is down. What’s not normal is the red X mark that says “No DS records found for coudwerk.us in the us zone”. This translates as, yes you signed your zone, but no one can validate it. It’s like a fake certificate. So, we need to tell our registrant that we really own the domain and that we signed it.
This portion is also registrant’s dependent. I’ll describe how to overcome this error with Go Daddy.
So, go to the keys directory and execute this command against the first file, that one that you took note about.

cd /var/named/keys
dnssec-dsfromkey Kcloudwerk.us.+008+23956.key
cloudwerk.us. IN DS 23956 8 1 2DA94FECA549E0A2B137FA6C64D09D2CBADF8864
cloudwerk.us. IN DS 23956 8 2 8CC2CB79E1B8A6767F11230485C7499D52162135E1F6DAA503ADEC4470E417A9

Now go to the DNS management panel for your domain at Go Daddy and you’ll see something like this. Click on DNSSEC and then click ADD.

Looks confusing but it’s very simple. Just look at the output from the dnssec-dsfromkey and add accordingly. You’ll have to add both hash values.

For Key Tag enter 23956, Algorithm is 8, Digest Type is 1, Digest is the hash value and Key Data Algorithm is also 8, the value of Algorithm. Click Update , do the 2nd line of the output and wait 30-45 mins. Sometimes, even more. It depends on your registrant.

At the end it should look like this.

And if you analyze your domain again, the result should look like this.

Or if you use dig, use the Google’s DNS server to check.

dig +trace +noadditional DS cloudwerk.us. @8.8.8.8 | grep DS

CentOS – Blog of Kliment Andreev – A place so I won't forget things

Ansible: Quick Start Guide for FreeBSD, CentOS and Ubuntu

Control and managed nodes

CentOS 8

Ubuntu

FreeBSD

SSH keys

Inventory and the config file

Playbooks

Service handlers

Variables

Roles

Error handling

Tags

Ansible Vault

Prompts

Useful options

CentOS, FreeBSD: Secure SSH with fail2ban and blacklistd with MFA (Google Authenticator)

CentOS

Log with SSH keys (optional)

Google Authenticator

fail2ban

FreeBSD

Log with SSH keys (optional)

Google Authenticator

blacklistd

CentOS: Create disposable e-mail addresses using postfix, dovecot and squirrelmail

Valid domain and MX records

Firewall and SELinux

postfix Part I

MySQL

Scripts

postfix Part II

dovecot

Test e-mail delivery

squirrelmail (optional)

Note about POP3/IMAP clients

AWS: squid transparent proxy for DNS filtering/whitelisting

VPC, subnets, public route and Internet Gateway

squid instance

Private route

Source/Destination Check

Install squid from source

Test

AWS, CentOS: Create your own radio station and deploy it on Alexa (optional)

CentOS 7

icecast

ices

MP3 to OGG conversion (sox)

nginx as reverse proxy

SSL certificate

Alexa

Cleanup

FreeBSD, CentOS: Install Jenkins behind a nginx reverse proxy using SSL certificates

CentOS

nginx as reverse proxy

FreeBSD

nginx as reverse proxy

Alibaba Cloud, CentOS: Build WordPress site using NFS, RDS and Load Balancers

VPC

Security Groups

SSH Keys

NFS instance

RDS

WordPress instances

Load Balancer

CentOS: Upgrade PostgreSQL from 9.2 to 9.6

CentOS: Apache Guacamole – Remote Desktop Gateway

Install Apache Guacamole

MySQL server

Start the application

nginx as reverse proxy

CentOS: Install an authoritative DNS server (BIND)

Installation, startup and control

Zone file

DNSSEC (optional)