freebsd – Blog of Kliment Andreev – A place so I won't forget things

FreeBSD: OpenVPN server in a LAN (local network) environment

Kliment Andreev — Fri, 22 Nov 2024 16:36:12 +0000

In this post I’ll describe a use case that I am currently using. I have a standard Internet connection that goes to my Eero mesh router and behind that I have a homelab on 192.168.1.0/24 network as shown in the diagram. I have a port forwarding configured on the router so I can access one of my Linux servers, but if I want to access other servers, I have to either port forward SSH on some other-than-22/tcp port or use the Linux server as a jump-box. None of this is good, so I decided to create an OpenVPN server. Anytime I VPN to this server I can ssh/rdp to any of the internal servers. The OpenVPN server requires one nic.
NOTE: This post is very FreeBSD specific so if you want to use some Linux server as OpenVPN server, this guide won’t help you at all.

IMPORTANT: On your home router port-forward port 1194/udp to your OpenVPN server.

OpenVPN server

Let’s install OpenVPN server first.

pkg install openvpn

Make sure it starts on boot and enable the NAT.

sysrc openvpn_enable=YES
sysrc gateway_enable=YES

Next, we need a directory where we’ll store our config files and the PKI.

mkdir /usr/local/etc/openvpn
cd /usr/local/etc/openvpn

Create the PKI structure.

easyrsa init-pki

Go to the newly created pki directory and make a copy of the vars.example file.

cd pki
cp vars.example vars

Now edit this file and uncomment the following lines and change to match your needs. It’s optional and you can keep everything-as-is.

set_var EASYRSA_REQ_COUNTRY     "US"
set_var EASYRSA_REQ_PROVINCE    "NJ"
set_var EASYRSA_REQ_CITY        "Allentown"
set_var EASYRSA_REQ_ORG         "iAndreev"
set_var EASYRSA_REQ_EMAIL       "mail@somemail.com"
set_var EASYRSA_REQ_OU          "iAndreev"
set_var EASYRSA_CA_EXPIRE       3650
set_var EASYRSA_CERT_EXPIRE     825

Now create the Certificate Authority based on these values.

cd /usr/local/etc/openvpn
easyrsa build-ca

This command will prompt you to create a password that you will you to sign the certificates for the server and the clients.
Then create the DH parameters. It might take 30+ seconds.

easyrsa gen-dh

Create the server request and sign it.

easyrsa build-server-full server nopass

You have to type yes to confirm and then use the password that you created above to sign the cert.
The command will create some files under the pki directory and we’ll need those later.
Next, we’ll create the same for the client. In my case it’s a Dell laptop, so I use the word dell, but you can use client, client1, something-whatever…

easyrsa build-client-full dell nopass

Same thing, answer yes and then use the password to sign the certs.
Copy the OpenVPN config example file that comes with the package.

cp /usr/local/share/examples/openvpn/sample-config-files/server.conf /usr/local/etc/openvpn/openvpn.conf

Edit openvpn.conf and make sure these values are changed and match the certs you created earlier.

ca /usr/local/etc/openvpn/pki/ca.crt
cert /usr/local/etc/openvpn/pki/issued/server.crt
key /usr/local/etc/openvpn/pki/private/server.key

dh /usr/local/etc/openvpn/pki/dh.pem

In addition to that, find the line that says push route and create a new one that matches your internal subnet. In my case it looks like this.

push "route 192.168.1.0 255.255.255.0"

If you have internal DNS and you want to send that as well, uncomment the following line and change to match your DNS IP.

push "dhcp-option DNS 208.67.222.222"

Now, you can start the OpenVPN server. Make sure you check the syslog for any errors.

service openvpn start
tail /var/log/messages

OpenVPN client

On your laptop that you want to use as a VPN client, download the OpenVPN client from here. When you install the client, you’ll have a generic config file under C:\Program Files\OpenVPN\sample-config folder. Copy that file under c:\users\[your_username]\OpenVPN\config.
Then, edit the file and change the following lines.

remote public-ip-or-hostname-of-your-router 1194
ca ca.crt
cert dell.crt
key dell.key

NOTE: Make sure you copy ca.crt from /usr/local/etc/openvpn/pki, dell.crt or client.crt or whatever.crt from /usr/local/etc/openvpn/pki/issued and dell.key or client.key or whatever.key from /usr/local/etc/openvpn/pki/private directories to the same c:\users\[your-user-name]\OpenVPN\config folder. There should be 4 files there. The config file, the CA certificate, the client certificate and the private key for the client.
Start the OpenVPN client and connect. You shouldn’t have any problems and you’ll get 10.8.0.2 IP assigned to your laptop. If you try to ping 10.8.0.1 you’ll get a response. Do route print and you’ll see that the routes for 192.168.1.0 are going over 10.8.0.1. But, if you try to ping some server internally, e.g. 192.168.1.21, you won’t get any response. That’s because that server doesn’t know the route back. You can easily add the route to each server but it’s not practical. Let’s make one more change to the server.

OpenVPN server

Add these two lines to /etc/rc.conf. That’s the FreeBSD pf firewall.

pf_enable="YES"
pf_rules="/etc/pf.conf"

Now, create this config file in /etc/pf.conf. Many thanks to this guy.

vpnclients = "10.8.0.0/24"
# The name of your NIC. I am using a VM on ESXi and vmx0 is my NIC.
wanint = "vmx0"
# put your tunnel interface here, it is usually tun0
vpnint = "tun0"
# OpenVPN by default runs on udp port 1194
udpopen = "{1194}"
# Open the SSH port.
tcpopen = "{22}"
icmptypes = "{echoreq, unreach}"

set skip on lo
# the essential line
nat on $wanint inet from $vpnclients to any -> $wanint

block in
pass in on $wanint proto udp from any to $wanint port $udpopen
pass in on $wanint proto tcp from any to $wanint port $tcpopen
# the following two lines could be made stricter if you don't trust the clients
pass out quick
pass in on $vpnint from any to any
pass in inet proto icmp all icmp-type $icmptypes

Reboot the OpenVPN server and if you reconnect from the client, you can use the internal IPs directly to connect.

Ansible: Quick Start Guide for FreeBSD, CentOS and Ubuntu

Kliment Andreev — Mon, 08 Feb 2021 17:04:37 +0000

In this post/howto, I’ll explain how to install Ansible as control and managed node on FreeBSD 12, CentOS 8 and Ubuntu 18. Then, I’ll explain how to create SSH keys so the nodes can communicate and some basic tasks. Then, I’ll show you how to create a playbook to install the latest updates and also install an Apache server with the default settings. Finally, I’ll show an example of how to use variables.

Control and managed nodes

A control node is where you execute all of your Ansible commands and eventually keep your playbooks, configs, inventory etc. It’s pretty much your workstation. A managed node is where the actual playbooks are executed. These are the main concepts and the terminology.
In my case, I have 4 VMs/instances. The main one which is the control node and 3 managed nodes.

CentOS 8

If you want to install Ansible on the control node, you have to install Python 3.x first.

sudo dnf install python3

This will also install pip. Type python3 to test, CTRL-D to exit and then install ansible.

sudo pip3 install ansible

Type ansible to test.

Ubuntu

Ubuntu comes with python installed, but not with pip. Install pip with:

sudo apt install python3-pip

Then install ansible with:

sudo pip3 install ansible

Type ansible to test.

FreeBSD

FreeBSD comes with python installed but pip is not. Check the version and then install the same pip version.

ls -l /usr/local/bin/python*

If your output is for example python37, install the same pip version.

pkg install py37-pip

Then install ansible.

pip install ansible

Type ansible to test.

SSH keys

While Ansible can use standard *nix username/password authentication, it’s recommended that you use SSH keys to communicate from control node to the managed nodes. For that, you’ll have to create your SSH keys. Let’s say you have an account on your control node and the username is admin. You also want to use the user ansible on the managed nodes. It really doesn’t matter what usernames you are going to choose. You can always override the keys to use, but in this case, I’ll create a key on the control node and send it to all managed nodes.
On the control node, regardless of your OS, do:

ssh-keygen -b 4096

This will create a subfolder .ssh with two files: id_rsa and id_rsa.pub. The former is your private key and the later is your public key.
Copy the key to your managed nodes.

ssh-copy-id ansible@nodeX.andreev.local

This will copy my key for the user admin to the node X under the ansible user. Then test the passwordless connection.

ssh ansible@nodeX.andreev.local

Mind that the use of FQDN (nodex.andreev.local) vs. hostname (nodex) is important. For SSH these two are different. Once you log to the managed node, the node will be added to the list of known hosts in the file .ssh/known_hosts.

Inventory and the config file

Ansible uses the inventory files to execute an action against using the options and parameters specified in the config file. The config file is /etc/ansible/ansible.cfg for CentOS and Ubuntu and /usr/local/etc/ansible/ansible.cfg for FreeBSD. You can also put the inventory file in the same directory and name it as you wish, but you have to specify the inventory as a parameter on the command line or an entry in the config file. In addition, you can have your config file in your current directory or under the .ansible directory in your home folder. ansible.cfg in the current directory has a precedence over .ansible.cfg in the home directory which has a precedence over /etc/ansible/ansible.cfg. Here is how that looks.
Let’s list all the inventory.

ansible --list-hosts all

You’ll get a message that there is no inventory file. Let’s create one, we’ll name it inventory.txt.

[freebsd]
node1.andreev.local
[centos]
node2.andreev.local
[ubuntu]
node3.andreev.local
[bsd]
node1.andreev.local
[linux]
node2.andreev.local
node3.andreev.local

If we specify the inventory file, we’ll get this.

ansible --list-hosts all -i inventory.txt
  hosts (3):
    node1.andreev.local
    node2.andreev.local
    node3.andreev.local

If we create a config file, we can tell ansible where to look for the inventory. Create a file ansible.cfg in the same directory.

[defaults]
inventory=/home//inventory.txt

If you do ansible –list-hosts all now, you’ll get the same result as before, but without specifying the inventory file.
Or something like this.

ansible all -m shell -a "uname -a"
node1.andreev.local | CHANGED | rc=0 >>
FreeBSD node1.andreev.local 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64
node2.andreev.local | CHANGED | rc=0 >>
Linux node2.andreev.local 4.18.0-193.14.2.el8_2.x86_64 #1 SMP Sun Jul 26 03:54:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
node3.andreev.local | CHANGED | rc=0 >>
Linux node3.andreev.local 4.15.0-129-generic #132-Ubuntu SMP Thu Dec 10 14:02:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

By default, ansible runs on the managed node with the currently logged user that executes the playbook on the control node. If you get an error saying that the previous command cannot connect to the host, you have to specify the same user that you used when you test the connection with ssh -user-@managednode. So, edit ansible.cfg and add this line.

remote_user=

We can target only the group linux which consists of Linux hosts only in the inventory file.

ansible linux -m shell -a "date"
node3.andreev.local | CHANGED | rc=0 >>
Fri Jan  8 15:23:40 UTC 2021
node2.andreev.local | CHANGED | rc=0 >>
Fri Jan  8 10:23:40 EST 2021

Playbooks

The playbooks are the blueprints of the automation tasks. Instead of running the ansible command to execute each task separately, we combine these tasks in a YAML file and execute them sequentially. Here are 3 playbooks that update each of our managed nodes. There is no update module for FreeBSD, so we use the shell command.

# freebsd-update.yml
---
  - hosts: freebsd
    become: yes
    tasks:
      - name: Fetch all packages
        shell: freebsd-update fetch
      - name: Install FreeBSD updates
        shell: freebsd-update install
      - name: Reboot
        reboot:

For CentOS we’ll use yum.

# centos-update.yml
---
  - hosts: centos
    become: yes
    tasks:
      - name: Update all packages
        yum: name=* state=latest
      - name: Reboot
        reboot:

…and for Ubuntu we’ll use apt.

# ubuntu-update.yml
---
  - hosts: ubuntu
    become: yes
    tasks:
      - name: Update all packages
        apt: name=* state=latest
      - name: Reboot
        reboot:

Save these files with a YAML extension and you can execute them with the following command.

ansible-playbook

All of them will probably fail. That’s because your ansible user on the managed nodes is required a password when executing a sudo command. In order to fix that, you’ll have to add a line in the sudoers file. Edit this file using the visudo command.

visudo

…and then add this line right before the @includedir <...> which is the last line in the file.

ansible ALL=(ALL) NOPASSWD:ALL

Where ansible is the user that runs the playbooks on the managed nodes. FreeBSD doesn’t come with sudo preinstalled, so you’ll have to install it first on the managed node.

pkg install sudo

These playbooks will update the OS and the packages for the Linux. For FreeBSD, it will update only the OS. Here is another example of playbooks that will install Apache server in a default configuration and change the ServerName and ServerAdmin lines. We’ll also install PHP and test our server.
If you have a firewall enabled, make sure you open it up first on CentOS. Ubuntu and FreeBSD do not come with the firewall enabled.

firewall-cmd --zone=public --permanent --add-service=http
firewall-cmd --reload

For FreeBSD, the playbook looks like this.

# freebsd-apache.yml
---
  - hosts: freebsd
    become: yes
    tasks:
      - name: Install apache and php
        pkgng:
          name:
            - apache24
            - php74
            - mod_php74
          state:  present
      - name: Start on reboot
        service: name=apache24 enabled=yes
      - name: Copy index.php
        copy:
          src: ../files/index.php
          dest: /usr/local/www/apache24/data
          mode: 0755
      - name: Copy mod_php.conf
        copy:
          src: ../files/mod_php.conf
          dest: /usr/local/etc/apache24/modules.d
          mode: 0755
      - name: Start apache now
        service: name=apache24 state=started

For CentOS, it looks like this.

# centos-apache.yml
---
  - hosts: centos
    become: yes
    tasks:
      - name: Install apache and php
        yum:
          name:
            - httpd
            - php
          state:  present
      - name: Start apache now and on reboot
        service: name=httpd state=started enabled=yes
      - name: Copy index.php
        copy:
          src: ../files/index.php
          dest: /var/www/html
          mode: 0755

…and for Ubuntu it looks like this.

# ubuntu-apache.yml
---
  - hosts: ubuntu
    become: yes
    tasks:
      - name: Install apache and php
        apt:
          name:
            - apache2
            - php
          state:  present
      - name: Start apache now and on reboot
        service: name=apache2 state=started enabled=yes
      - name: Copy index.php
        copy:
          src: ../files/index.php
          dest: /var/www/html
          mode: 0755

You will also need these two files in a directory called files. In my case it’s one level above the directory where I keep my playbooks.
index.php

001_mod-php.conf


    DirectoryIndex index.php index.html
    
        SetHandler application/x-httpd-php
    
    
        SetHandler application/x-httpd-php-source

The index.php file is the standard test file to test the PHP distributions and the 001_mod-php.conf is needed for FreeBSD only. As you can see the playbooks differ quite a bit for these three OSes. Once you deploy the playbooks, you can test the result by going to http://[nodeX]/index.php.
Looks like this.

Service handlers

Service handlers are used only when a change is made on the managed node. For example, we can restart a service only if the config file was changed. If the file is not changed, then there is no need to restart. Here is an example of a service handler. We’ll restart postfix service only if the main.cf file was changed.

# service-handler.yml
---
- hosts: centos
  become: yes
  tasks:
  - name: Configure main.cf
    lineinfile:
      path: /etc/postfix/main.cf
      regexp: ^#mydomain
      line: 'mydomain = example.com'
    notify: restart postfix

  handlers:
  - name: restart postfix
    service: name=postfix state=restarted

Make sure that the name for the handler is the same (lines 11 and 14), so Ansible knows what service handlers is referred.

Variables

When ansible runs a playbook on a managed node, the first task is to gather info about the managed node. The info is a bunch of settings that we can use in our playbooks. For example, if you execute the following command, you can see the IP address, the CPU model, python version etc.

ansible -m setup

We can use these settings and use them as variables if we need them. For example, this playbook displays the hostname and the IP.

# showip.yml
---
  - hosts: freebsd
    become: yes
    tasks:
      - name: Show the IP address
        debug:
          msg: "The hostname is {{ inventory_hostname}}  and the IP is {{ansible_default_ipv4.address }}"

And if you run the playbook, you’ll see something like this.

TASK [Show the IP address] *********************************************************************************************
ok: [node1.andreev.local] => {
    "msg": "The hostname is node1.andreev.local  and the IP is 192.168.1.211"
}

Here is another example of using variables. In this case, we’ll specify a file and change the ownership and the mode.

# owner.yml
---
  - hosts: centos
    become: yes
    vars:
      filename: "/var/www/html/index.php"
    tasks:
      - name: Change the owner of the file
        file:
          path: "{{ filename }}"
          owner: apache
          group: apache
          mode: '0755'

In case we want to assign an output to a variable, we’ll have to use the keyword register. Here is an example of how to get the output from a command and print it on the screen with the keyword debug.

# variables.yml
---
  - hosts: freebsd

    tasks:
    - name: Get the uptime manually
      command: uptime
      register: var_uptime

    - name: Print the uptime
      debug:
        msg: The uptime is "{{ var_uptime }}"

Roles

Roles let you automatically load related vars_files, tasks, handlers, and other Ansible artifacts based on a known file structure. Once you group your content in roles, you can easily reuse them and share them with other users. The idea is to separate the tasks, handlers and vars in different files. Let’s see this playbook for example. It changes a line in main.cf file, restarts postfix and copies a file under the postfix main directory.

# roles.yml
---
  - hosts: centos
    become: yes

    vars:
      filevd: "/etc/postfix/virtual_domains"
      cfgpostfix: "/etc/postfix/main.cf"


    tasks:
    - name: Configure main.cf
      lineinfile:
        path: "{{ cfgpostfix }}"
        regexp: ^#mydomain
        line: 'mydomain = example.com'
      notify: restart postfix
    - name: Copy virtual_domains
      copy:
        src: ../files/virtual_domains
        dest: "{{ filevd }}"
        mode: 0755

    handlers:
    - name: restart postfix
      service: name=postfix state=restarted

We can rewrite this file by separating the tasks, variables, files and handlers. Run this command.

ansible-galaxy role init postfix

If you look at the file/directory structure of the newly created directory postfix, it looks like this.

tree postfix
postfix
├── defaults
│   └── main.yml
├── files
├── handlers
│   └── main.yml
├── meta
│   └── main.yml
├── README.md
├── tasks
│   └── main.yml
├── templates
├── tests
│   ├── inventory
│   └── test.yml
└── vars
    └── main.yml

Create a file virtual_domains under the files directory.

echo "mydomain.com" > postfix/files/virtual_domains

Move the config for vars, handlers and tasks in the separate main.yml files. For example, this is how my files look like.
postfix/vars/main.yml

---
# vars file for postfix

  filevd: "/etc/postfix/virtual_domains"
  cfgpostfix: "/etc/postfix/main.cf"

As you can see the keyword vars: does not exists. Ansible knows that this file is for vars so there is no need to enter the vars keyboard.
postfix/tasks/main.yml

---
# tasks file for postfix

  - name: Configure main.cf
    lineinfile:
      path: "{{ cfgpostfix }}"
      regexp: ^#mydomain
      line: 'mydomain = example.com'
    notify: restart postfix
  - name: Copy virtual_domains
    copy:
      src: ../files/virtual_domains
      dest: "{{ filevd }}"
      mode: 0755

postfix/handlers/main.yml

---
# handlers file for postfix

  - name: restart postfix
    service: name=postfix state=restarted

Finally, create a file called something.yml that will be your main file. This file has to be outside the postfix directory structure.
In my case it looks like this.
something.yml

# something.yml
---
  - hosts: centos
    become: yes
    roles:
      - postfix

Now, if you execute this playbook, ansible will automatically execute the rest of the dependant playbooks as well.

ansible-playbook something.yml

Error handling

Sometimes we want certain changes to be ignored. Sometimes, we know the behavior of certain commands and we know that they might return non-zero code and we want that ignored. For example, consider this part of a playbook.

- hosts: centos
  tasks: 
  - name: Type something that will fail
    command: thiscommanddoesntexist
    ignore_errors: yes

  - name: Run command remotely
    command: /usr/local/bin/somecommand
    register: cmd_result
    changed_when: cmd_result == 2

Ansible would report a task as changed as long as the command (or) script gives zero return code.
In the first part, we know that the task will fail, but we decide to ignore it using the keyword ignore_errors. No matter what the command returns, ignore_errors: yes will never report to ansible that the command failed.
In the second command we can ignore the error based on the output of the command. For example, if the output is 2, the the error will be ignored. If cmd_result is not equal to 2, the task will be marked as changed.
So whenever this condition is true, the task will be marked as changed.

Ansible Vault

Ansible Vault encrypts variables and files so you can protect sensitive content such as passwords or keys rather than leaving it visible as plaintext in playbooks or roles.
First, you have to create a vaulted file where we’ll store the passwords. When you run this command it will ask you to create a password and then an empty file will show up.

ansible-vault create secrets.yml

Add some passwords there and save the file.

mysql_pwd: "DifficultPassword"
ht_pwd: "PasswordXYZ"

If you look at the file now, you’ll see that it’s encrypted and you can’t see the passwords anymore.
If you want to edit the file do ansible-vault edit secrets.yaml and enter the vault password.
Create a small playbook that displays the password.

# centos-vault.yml
---
- hosts: centos
  vars_files:
    - secrets.yml

  tasks:
  - name: Show mysql pwd
    debug:
      msg: "{{ mysql_pwd }}"

If you run the playbook now, ansible will throw an error saying ERROR! Attempting to decrypt but no vault secrets found. You have to specify the parameter –ask-vault-pass and enter the vault password when prompted.

Prompts

In case you need to pause the playbook execution and ask the user for some input such as confirmation or password, use prompts.
If you want the output to echo, use private: no, otherwise what you type won’t show up on the screen. Here is an example of a playbook that asks you to confirm if a file needs to be copied to the node. If you type yes and hit enter, the file will be copied, otherwise it won’t.

# centos-prompt.yml
---
- hosts: centos
  become: yes

  vars_prompt:
    name: upload
    private: no
    prompt: "Do you want to upload xyz.txt?"

  tasks:
  - name: Upload xyz.txt
    copy:
      src: xyz.txt
      dest: /var/log
    when: upload == "yes"

Useful options

ansible-playbook  --syntax-check # checks the syntax of the playbook
ansible-playbook  --check # does a dry run, reports the will-be changes, but the playbook is not executed
ansible-playbook  --step # ask to confirm each-step

FreeBSD, pfSense: Site-to-site VPN IPsec tunnel between FreeBSD and pfSense using strongswan

Kliment Andreev — Fri, 28 Aug 2020 20:49:59 +0000

I have written a lot about pfSense and different types of VPN scenarios (AWS, Azure), but never created a post about a site-to-site VPN tunnel with FreeBSD running strongswan and pfSense. The scenario described here works with FreeBSD, but it will work with any other Linux distribution with some small OS dependand changes. The syntax for strongswan is the same, it’s just the configuration file names and locations that are different across distributions. For CentOS, see my other post.

Prerequisites

Both the pfSense box and FreeBSD need to have public IPs. While it’s possible to have them behind NAT, this scenario only covers configurations with public IPs. You should disable the firewall on FreeBSD. Both of pfSense and FreeBSD box need two network interfaces. In my case this is how it looks like.

It’s much easier if you enter these values in a small spreadsheet so you don’t get confused later.

FreeBSD configuration

First, we have to install strongswan, configure the 2nd internal NIC if it’s not configured and allow FreeBSD to act as a gateway for other servers behind it (e.g. servers srv1 and srv2).
Install strongswan from packages.

pkg install strongswan

Edit /etc/rc.conf and add this line, so strongswan starts on boot.

strongswan_enable="YES"

While editing that file, add this line too.

gateway_enable="YES"

…and if your internal NIC is not configured, you can assign the IP there too.

ifconfig_vtnet1="inet 10.39.96.3 netmask 255.255.240.0"

where vtnet1 is the name of the NIC. You can see the name of your NIC by doing ifconfig -a.
Next, we’ll configure strongswan. The config files are under /usr/local/etc. Edit ipsec.conf and delete everything then paste these lines.

config setup
 
conn %default
        ikelifetime=60m
        keylife=20m
        rekeymargin=3m
        keyingtries=1
        authby=secret
        keyexchange=ikev2
        mobike=no
 
conn net-net
        left=140.82.31.124
        leftsubnet=10.39.96.0/20
        leftid=140.82.31.124
        leftfirewall=no
        right=67.81.194.50
        rightsubnet=192.168.1.0/24
        rightid=67.81.194.50
        ike=aes256-sha256-modp2048!
        esp=aes256-sha256!
        auto=start
        lifetime=1h

If you look at lines 13, 14 and 15 you’ll see that line 13 is the external IP of our FreeBSD box, line 14 is the internal subnet for FreeBSD and line 15 is just a description for this connection. In my case I used the IP. The same applies for lines 17, 18 and 19. Line 17 is the external IP of the pfSense, line 18 is the internal subnet for the resources behind pfSense and line 19 is just an ID for this connection.
NOTE: if you change the IDs, e.g. lines 15 and 19, then you have to change the same in the secrets file.
The secrets file is a password that will be used to communicate between FreeBSD and pfSense. Another option is to use certificates and it’s not that difficult to reconfigure that, but that’s not in this scope. So, edit /usr/local/etc/ipsec.secrets and add this line.

140.82.31.124 67.81.194.50 : PSK "supersecretpassword"

Don’t use tabs. Spaces only or it won’t work. As you can see you are specifying the leftid, then the rightid and some password. PSK means pre-shared key or password. The parameter leftid and rightid in ipsec.conf must be the same with the parameters here. So, if I change the line 14 to be leftid=@freebsd, I have to do the same in ipsec.secrets. I have to specify @freebsd instead of 140.82.31.124.

pfSense

Now that the FreeBSD strongswan box is configured, we can configure pfSense. In pfSense, go to VPN | IPSec from the menu and click on Add P1 button. Fill out the General Information section, so it looks like this.

As you can see, we use IKEv2 and our remote gateway is 140.82.31.124 which is the FreeBSD strongswan box.
For Phase 1 Proposal (Authentication and Algorithms), fill out the section so it looks like this. The Pre-Shared Key is what you put in /usr/local/etc/ipsec.secrets file, remember?

Then for Phase 1 Proposal (Algorithms), use these values. If you look at your ipsec.conf, you’ll see how they match.

…and use the defaults for the Advanced Options.

Once you click Save at the bottom, pfSense will bring you back. Just click Apply Changes.
Do the phase 2 now, click on Show Phase 2 Entries and then click Add P2.

For the phase 2, enter the following information. Choose Network instead of LAN and add the subnets for both pfSense and FreeBSD. In my case it looks like this. Phase 2 is very important because you have to make sure you enter the proper values. One extra checkmark and you’ll be endlessly troubleshooting why it’s not working. So, pay attention please!

For the Phase 2 Proposal (SA/Key Exchange) i have AES 256 bit instead of Auto and SHA256 instead of SHA1. Lifetime is also 3600.

And you can leave the Advanced Configuration as is. If you notice that your tunnel drops because there is 0 activity, you can enter the IP address of the FreeBSD box or an IP for a server that’s behind FreeBSD. This will send ping ICMP packets to the destination so it keeps the tunnel alive. It’s not necessary, but you never know. Click Save then Apply Changes again.
By default, pfSense will block the IPsec traffic, so you have to enable it. From the menu go to Firewall | Rules and click on IPSec submenu. Click Add and fill out the form so it looks like this.

…and this.

Click Save and go back to the FreeBSD box. Reboot the FreeBSD.

Site-to-site VPN tunnel

Once FreeBSD comes back up, do:

ipsec status
Security Associations (1 up, 0 connecting):
     net-net[1]: ESTABLISHED 8 seconds ago, 140.82.31.124[140.82.31.124]...67.81.194.50[67.81.194.50]
     net-net{1}:  INSTALLED, TUNNEL, reqid 1, ESP SPIs: ca009d9e_i cc8a8b76_o
     net-net{1}:   10.39.96.0/20 === 192.168.1.0/24

…and if you go to pfSense, under Status | IPsec you can see the tunnel established.

If you try to ping the FreeBSD from a server behind your pfSense, you’ll get a response. But if you try to ping something behind pfSense from the FreeBSD box, it won’t work.
Check the routes in FreeBSD.

netstat -nr | grep 192
192.168.1.0/24     140.82.30.1        US       vtnet0

This is a wrong route that strongswan injected. Delete the route and add the proper one.

route delete 192.168.1.0
delete net 192.168.1.0
route add -net 192.168.1.0/24 10.39.96.3
add net 192.168.1.0: gateway 10.39.96.3

Now if you ping something, it will work. But wait! Once we reboot, everything will be messed up again? Correct!
First, add this line in /etc/rc.conf. It creates a permanent static route.

route_vpn="-net 192.168.1.0/24 10.39.96.3"

Second, prevent strongswan to mess up with the routes. Edit /usr/local/etc/strongswan.conf and above line load_modular = yes, add install_routes = no. It looks like this.

......
charon {
        install_routes = no
        load_modular = yes
        plugins {
                include strongswan.d/charon/*.conf
        }
}
......

After this change, restart strongswan.

service strongswan restart

NOTE: For any resource behind FreeBSD, you have to add a static route for the 192.168.1.0/24 network. For example, for Windows type route add 192.168.1.0/24 10.39.96.3 -p otherwise that resource will use the default gateway.

CentOS, FreeBSD: Secure SSH with fail2ban and blacklistd with MFA (Google Authenticator)

Kliment Andreev — Fri, 27 Mar 2020 18:57:52 +0000

I have some servers that are exposed to the Internet on port 22 and I see a lot of brute force attacks coming. The easiest and best way to protect is to block port 22 for everyone except a handful of IPs or safe subnets. But sometimes, these servers need to be available for some other people who work from who knows where, so you can’t just restrict IPs anymore. In this case, the best practice is to enforce MFA (multi factor authentication). But, this won’t work if you have some service accounts that need to log to the server. You can’t MFA a service account.To balance between security and availability, you can also install some tools that will block the repeated offenders, IPs that crawl the network and try to login to your network over SSH. In this post I’ll describe how to use fail2ban for CentOS and blacklistd for FreeBSD. fail2ban for FreeBSD is too complex to setup while blacklistd is a native tool and does the same job.

CentOS

I have a CentOS box with both SELinux and firewalld enabled. I am not using keys to log to the server, I have a generic user that uses password and the root is NOT allowed to log to the server using ssh (PermitRootLogin no in /etc/ssh/sshd_config).

Log with SSH keys (optional)

If you want to have some users that need to log with SSH keys, you have to do the following. Log as that user and run ssh-keygen and hit enter for the defaults.

ssh-keygen -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/home/test/.ssh/id_rsa):
Created directory '/home/test/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/test/.ssh/id_rsa.
Your public key has been saved in /home/test/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:CmYzEZq7u4IilphSEC/cdTFKQY2+fjQgkw8acCXZjIo test@localhost.localdomain
The key's randomart image is:
+---[RSA 4096]----+
|  .*+++o.        |
|o ++++.o.        |
|+=+ =..          |
|Eo.B +           |
| o+ @ o S        |
| ..+ * +         |
|ooo . o .        |
|Oo . . .         |
|=.o.  .          |
+----[SHA256]-----+

You have two files created for the test user, a private (id_rsa) and a public (id_rsa.pub) file.
You need the private file in order to use it with putty. But putty uses it’s own PPK file, so you have to convert that with puttygen. You can use the Windows version or you can install putty and convert it. NOTE: putty is a GUI program so don’t install it on a server because it will install a bunch of GUI libraries.
Install epel-release repo.

yum install -y epel-release

…and then install putty.

yum install -y putty

Log as the user and convert the file from PEM to PPK.

puttygen .ssh/id_rsa -o private.ppk -O private

Your key will be in the home directory as private.ppk. Use this key to log to your server. But, you have to authorize your public key first. Logged as test user do:

cat .ssh/id_rsa.pub >> .ssh/authorized_keys
chmod 0600 .ssh/authorized_keys

Now, you can log as test user with a key using putty. This will affect only the test user to log with a key. The other users will continue logging as they used to do before. Mind that at this point you can log as a test user using a key and the password. If you want to completely disable password logins make sure that these settings are configured under /etc/ssh/sshd_config.

PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes

Restart sshd (systemctl restart sshd) anytime you make a change in that file.

Google Authenticator

In order to enable MFA, we’ll have to install Google Authenticator on the server and a client on a phone or a desktop. I use authy for that. You can use any MFA client you want on your phone. I will also use key only logins and no passwords so go ahead and change those 3 lines above in sshd_config. Also, make sure that you can login with at least one user that can sudo or you’ll be locked. The only way to get in will be thru the console which does not use ssh so it’s not affected.
To install Google Authenticator, you need the epel repo. Skip this step if you already completed any of the previous steps.

yum install -y epel-release

Then install Google Authenticator.

yum -y install google-authenticator

Now, you can go two ways. Mandate that all accounts use MFA or only certain ones. It depends on your server usage case. If you have some automated scripts that use SSH to do some work, the first option is not viable. For each account that you want to use MFA, you have to configure the authenticator. I’ll use my test account. Logged as the test user, type google-authenticator. I answered y on all of the questions.

google-authenticator
Do you want authentication tokens to be time-based (y/n) y
Warning: pasting the following URL into your browser exposes the OTP secret to Google:
https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/test@localhost.l....

Your new secret key is: HF52123FWAGRKLGX5EJ4567CLE
Your verification code is 880933
Your emergency scratch codes are:
94310385
95472197
98736692
31351434
83561212

Do you want me to update your "/home/test/.google_authenticator" file? (y/n) y

Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n) y

By default, a new token is generated every 30 seconds by the mobile app.
In order to compensate for possible time-skew between the client and the server,
we allow an extra token before and after the current time. This allows for a
time skew of up to 30 seconds between authentication server and client. If you
experience problems with poor time synchronization, you can increase the window
from its default size of 3 permitted codes (one previous code, the current
code, the next code) to 17 permitted codes (the 8 previous codes, the current
code, and the 8 next codes). This will permit for a time skew of up to 4 minutes
between client and server.
Do you want to do so? (y/n) y

If the computer that you are logging into isn't hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting? (y/n) y

In a console session you’ll see a barcode but it’s garbled. Instead scroll up and you’ll see a URL (line 4). Go to that URL and scan the barcode with your MFA app.
Open /etc/ssh/sshd_config, scroll all the way down and add this line as the last line.

AuthenticationMethods publickey,password publickey,keyboard-interactive

Then change this line to yes.

ChallengeResponseAuthentication yes

Edit /etc/pam.d/sshd and after these two lines, add the auth line.

# Used with polkit to reauthorize users in remote sessions
-session   optional     pam_reauthorize.so prepare
auth required pam_google_authenticator.so nullok

nullok means that you are allowing some accounts to log as usual, without MFA. If you remove nullok, all accounts will be required to use MFA.
Also, comment this line like this.

#auth       substack     password-auth

Restart the sshd daemon now.

systemctl restart sshd

fail2ban

In order to install fail2ban you have to install the epel-release first. Skip this step if you followed the previous step because you already installed epel repo.

yum -y install epel-release

Then install fail2ban.

yum -y install fail2ban

Make sure it starts on boot.

systemctl enable fail2ban

The config files for fail2ban are under /etc/fail2ban, but you shouldn’t change any of the existing files. Instead create a new file called sshd.conf under /etc/fail2ban/jail.d directory. Here is mine.

[sshd]
enabled = true

# "ignoreip" can be a list of IP addresses, CIDR masks or DNS hosts. Fail2ban
# will not ban a host which matches an address in this list. Several addresses
# can be defined using space (and/or comma) separator.
ignoreip = 127.0.0.1/8

# "bantime" is the number of seconds that a host is banned.
# add m for minutes or hr for hours
bantime  = 10m

# A host is banned if it has generated "maxretry" during the last "findtime"
# seconds.
findtime = 300

# "maxretry" is the number of failures before a host get banned.
maxretry = 3

It is very self explanatory with the included comments. In my example, I’ll ban an IP for 10 minutes if I see that IP try and fail 3 times to login in 300 seconds (5 mins). You can start/restart the service now.

systemctl start fail2ban

The fail2ban log file is under /var/log/fail2ban.log and the SSH logs are under /var/log/secure.
Most of the hits that you’ll see in the /var/log/secure are IPs that will try to log as admin or root, but only once in a hour or two. So, they won’t be caught with these settings. You should add all of your subnets under ignoreip setting and change the maxretry to 1 and bantime to 8760h (1 year). It will ultimately block the IP for a year for anyone that makes any unauthorized attempt. It’s up to you what settings you use.
Here is an example in the /var/log/secure of an IP (3.92.82.165) trying to log as root.

Mar 21 10:26:13 localhost sshd[13207]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Mar 21 10:26:15 localhost sshd[13207]: Failed password for root from 3.92.82.165 port 60362 ssh2
Mar 21 10:26:16 localhost unix_chkpwd[13211]: password check failed for user (root)
Mar 21 10:26:16 localhost sshd[13207]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Mar 21 10:26:18 localhost sshd[13207]: Failed password for root from 3.92.82.165 port 60362 ssh2
Mar 21 10:26:18 localhost sshd[13207]: Connection closed by 3.92.82.165 port 60362 [preauth]
Mar 21 10:26:18 localhost sshd[13207]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=ec2-3-92-82-165.compute-1.amazonaws.com  user=root

…and the IP got banned. You can see that from /var/log/fail2ban.

2020-03-21 10:26:12,259 fail2ban.filter         [13131]: INFO    [sshd] Found 3.92.82.165 - 2020-03-21 10:26:12
2020-03-21 10:26:15,382 fail2ban.filter         [13131]: INFO    [sshd] Found 3.92.82.165 - 2020-03-21 10:26:15
2020-03-21 10:26:18,887 fail2ban.filter         [13131]: INFO    [sshd] Found 3.92.82.165 - 2020-03-21 10:26:18
2020-03-21 10:26:19,286 fail2ban.actions        [13131]: NOTICE  [sshd] Ban 3.92.82.165

You can also see the banned IPs using this command. sshd is the name of the jail. If you scroll up, you’ll see that I named my jail like that [sshd] in the conf file above.

fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 1
|  |- Total failed:     8
|  `- Journal matches:  _SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
   |- Currently banned: 2
   |- Total banned:     2
   `- Banned IP list:   3.81.85.240 3.92.82.165

But mistakes happen. If someone gets banned, you can unban the IP with this command, where sshd is the jail name and the IP you want unbanned.

fail2ban-client set sshd unbanip 3.92.82.165
1

You should get 1 as an output. Means OK, True. If you get 0, it means the IP wasn’t there. If you can’t see all banned IPs, do:

zgrep 'Ban' /var/log/fail2ban.log*

FreeBSD

I have a FreeBSD box with no firewall enabled for now. For the blacklistd part, I’ll use pf. If you use ipfirewall or ipfilter, then you have to find another guide. I am not using keys to log to the server, I have a generic user that uses password and the root is NOT allowed to log to the server using ssh (PermitRootLogin no in /etc/ssh/sshd_config).

Log with SSH keys (optional)

If you want to have some users that need to log with SSH keys, you have to do the following. Log as that user and run ssh-keygen and hit enter for the defaults.

ssh-keygen -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/home/test/.ssh/id_rsa):
Created directory '/home/test/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/test/.ssh/id_rsa.
Your public key has been saved in /home/test/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:rT2m3a/KrC/jk1lxHUPMwuXM5g/5LpjkurrC29EO1z8 test@bsd.andreev.local
The key's randomart image is:
+---[RSA 4096]----+
|                 |
|                 |
|           .     |
|        ..o o    |
|     . =S..+     |
|.   . + B+.      |
|o. o + @+.+o     |
|..o * o.OE+o     |
| ..o.+o.+@@.oo.  |
+----[SHA256]-----+

You have two files created for the test user, a private (id_rsa) and a public (id_rsa.pub) file.
You need the private file in order to use it with putty. But putty uses it’s own PPK file, so you have to convert that with puttygen. You can use the Windows version or you can install putty and convert it.

pkg install putty-nogtk

Log as the user and convert the file from PEM to PPK.

puttygen .ssh/id_rsa -o private.ppk -O private

Your key will be in the home directory as private.ppk. Use this key to log to your server. But, you have to authorize your public key first. Logged as test user do:

cat .ssh/id_rsa.pub >> .ssh/authorized_keys
chmod 0600 .ssh/authorized_keys

PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes

Restart sshd (service sshd restart) anytime you make a change in that file.

Google Authenticator

pkg install pam_google_authenticator

$ google-authenticator
Do you want authentication tokens to be time-based (y/n) y
Warning: pasting the following URL into your browser exposes the OTP secret to Google:
https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/test@bsd.andreev.local%3Fsecret%3D24345SOFKCTVX44YXYZVYOQKCA%26issuer%3Dbsd.andreev.local
Your new secret key is: 24HJ3SOFKCTVZ33YVBKVYOKEDB
Enter code from app (-1 to skip): -1
Code confirmation skipped
Your emergency scratch codes are:
34344229
32681543
80840666
32593123
29994070

Do you want me to update your "/home/test/.google_authenticator" file? (y/n) y

In a console session you’ll see a barcode but it’s garbled. Instead scroll up and you’ll see a URL (line 4). Go to that URL and scan the barcode with your MFA app.
Now, log as root or sudo to open /etc/ssh/sshd_config file. Change this line to yes.

ChallengeResponseAuthentication yes

Scroll all the way down and add this line as the last line.

AuthenticationMethods publickey,password publickey,keyboard-interactive

Edit /etc/pam.d/sshd and after these two lines, add the last auth line but commend the line above so it looks like this.

#auth           sufficient      pam_ssh.so              no_warn try_first_pass
#auth            required        pam_unix.so             no_warn try_first_pass
auth            required        pam_google_authenticator.so nullok

nullok means that you are allowing some accounts to log as usual, without MFA. If you remove nullok, all accounts will be required to use MFA. Restart the sshd daemon now.

service sshd restart

At this point, you’ll be able to login only with a public key and an MFA for the users that have google-authenticator and a public key configured. The password users won’t be able to log in. The console logins are not affected. So, if you have some users without a public key and google-authneticator, they’ll still be able to login with a password from a console.
If you don’t want to use public key and prefer passwords and MFA, then remove the last line from sshd_config AuthenticationMethods and uncomment pam_unix.so in /etc/pam.d/sshd above google_authenticator line. Restart sshd.

blacklistd

fail2ban is available for FreeBSD, but it takes some time to configure it right with the proper firewalls. Instead, I use blacklistd which comes preinstalled. The only prerequisite is a firewall and I’ll be using pf for that. So, first let’s enable pf and blacklistd by adding these lines in /etc/rc.conf.

pf_enable="YES"
pflog_enable="YES"
blacklistd_enable="YES"
blacklistd_flags="-r"

Let’s create a very simple pf config file /etc/pf.conf.

ext_if = "vmx0"
set skip on lo0
anchor "blacklistd/*" in on $ext_if
table  persist file "/etc/blocked_subnets"
block all
block in log quick on $ext_if from  to any
block out log quick on $ext_if from any to 
pass in on $ext_if proto tcp from any to any port {ssh} keep state
pass out quick on $ext_if keep state

Make sure you replace the name of your network card under line #1. You can get the name from ifconfig output. Also, if you want to block certain subnets by default, you can list them under /etc/blocked_subnets file. The subnet(s) should be in CIDR format. For example, if you want to block certain countries, you can look at this website. BTW, the subnets blocking is part of pf not blacklistd. You can reboot now to make sure everything comes back OK and you are able to login. But before doing that, edit the sshd daemon and let it know that we’ll use blacklistd. Find UseBlacklist in /etc/ssh/sshd_conf and change it to:

UseBlacklist yes

The reboot will start pf and blacklistd. The blacklistd conf file is /etc/blacklistd.conf. You can read more about blacklistd here. For example, if I want to whitelist certain subnets from being blacklisted, I’ll add them under the [remote] section. In my case, I am whitelisting my home subnet and an external subnet.

[remote]
172.16.1.0/24:ssh      *       *       *               =       *       *
154.21.3.0/24:ssh      *       *       *               =       *       *

You can see your SSH activity in /var/log/auth.log. To see who is blocked, run:

blacklistctl dump -br
        address/ma:port id      nfail   remaining time
  213.74.176.36/32:22   OK      8/3     23h58m57s

This IP will be blocked for 24hrs by default. You can change that under the disable column in /etc/blacklistd.conf.
You can also see the blocked IPs by looking at the pf blacklistd table.

pfctl -a blacklistd/22 -t port22 -T show
   213.74.176.36

And if you want to remove an IP or a subnet do:

pfctl -a blacklistd/22 -t port22 -T delete 213.74.176.36/32
1/1 addresses deleted.

FreeBSD, CentOS: Install Jenkins behind a nginx reverse proxy using SSL certificates

Kliment Andreev — Sat, 20 Jul 2019 13:06:24 +0000

In this post, I’ll explain how to install Jenkins on FreeBSD and CentOS. Jenkins runs on port 8080 (8180 in FreeBSD), so sometimes it’s not possible to access these ports because of corporate firewalls. We’ll put Jenkins behind nginx that will act as a reverse proxy. We’ll use port 80 (HTTP) and 443 (HTTPS) if you want to have SSL certificates. The installation is very simple.

CentOS

The Jenkins package is not part of the default CentOS repository, so we have to download that one first, install Jenkins, enable it to start on boot and then start it. You can skip the first line if you have wget installed. By default, it is not installed.

yum -y install wget
wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key
yum -y install jenkins java
systemctl enable jenkins
systemctl start jenkins

If you have a firewall enabled, you have to open the port for 8080.

firewall-cmd --permanent --zone=public --add-port=8080/tcp
firewall-cmd --reload

Now, go to http://[ip]:8080 and replace the with the IP of your server and you should see this greeting page.

Cat out that file, copy & paste the value in the browser and click Continue.

cat /var/lib/jenkins/secrets/initialAdminPassword
7f09cb85a4b843bdab66c768020a7c6e

And that’s it. Follow the prompts to install plugins, provide admin password etc…

nginx as reverse proxy

In order to access the server with out the need to specify the port 8080 and use a DNS name, we’ll have to use nginx that will listen on port 80 or 443 and redirect the traffic to 8080. Let’s install and configure it. In my case, I’ll access the server as jenkins.domain.com.

yum -y install epel-release
yum -y install nginx
systemctl enable nginx

Edit /etc/nginx/nginx.conf and delete everything after this line around line 36. The include line should stay and the last line “}” should stay as well. So, the last lines should look like this.

       include /etc/nginx/conf.d/*.conf;
}

Go to /etc/nginx/conf.d folder and create a new file called jenkins.conf.
This is how my jenkins.conf file looks like. Change the domain in the highlighted line.

server {
    listen 80;
    server_name jenkins.domain.com;

    location / {
        proxy_set_header   X-Forwarded-For $remote_addr;
        proxy_set_header   Host $http_host;
        proxy_pass         "http://127.0.0.1:8080";
    }
}

Don’t forget to open port 80 in your firewall.

firewall-cmd --permanent --zone=public --add-port=80/tcp
firewall-cmd --reload

If you have SELinux, you’ll have to allow HTTP traffic.

setsebool -P httpd_can_network_connect 1

And restart nginx for changes to take effect.

systemctl restart nginx

Now, you can access the server as jenkins.domain.com.
For HTTPS traffic, the configuration is different. You’ll need certificates specified in lines 15 and 16 (under /etc/nginx) and the domain changed in 7,13 and 31. Here is the config (/etc/nginx/conf.d/jenkins.conf) in order to access jenkins over SSL.

upstream jenkins {
  server 127.0.0.1:8080 fail_timeout=0;
}

server {
  listen 80;
  server_name jenkins.domain.com;
  return 301 https://$host$request_uri;
}

server {
  listen 443 ssl;
  server_name jenkins.domain.com;

  ssl_certificate public_cert.pem;
  ssl_certificate_key private_cert.key;
  client_max_body_size 4M;

  location / {
    proxy_set_header        Host $host:$server_port;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto $scheme;
    proxy_redirect http:// https://;
    proxy_pass              http://jenkins;
    # Required for new HTTP-based CLI
    proxy_http_version 1.1;
    proxy_request_buffering off;
    proxy_buffering off; # Required for HTTP-based CLI to work over SSL
    # workaround for https://issues.jenkins-ci.org/browse/JENKINS-45651
    add_header 'X-SSH-Endpoint' 'jenkins.domain.com:50022' always;
  }
}

Make sure you open port 443 on the firewall as we did for port 80 and change SELinux with setsebool.

firewall-cmd --permanent --zone=public --add-port=443/tcp
firewall-cmd --reload

SELinux change.

setsebool -P httpd_can_network_connect 1

You should also restart nginx for the changes to take effect.

systemctl restart nginx

That’s it. You should be able to access Jenkins as jenkins.domain.com over https.

FreeBSD

The installation on FreeBSD is a little bit different. You can install Jenkins from the packages. It will install JDK as well.

pkg install jenkins

The OpenJDK installation depends on these “memory filesystems”. Mount them now.

mount -t fdescfs fdesc /dev/fd
mount -t procfs proc /proc

…and make sure they are mounted on boot.

cat <> /etc/fstab
fdesc   /dev/fd         fdescfs         rw      0       0
proc    /proc           procfs          rw      0       0
EOF

We need to make sure jenkins starts on boot.

sysrc jenkins_enable=YES

Now, we can start jenkins.

service jenkins start

Go to http://[IP]:8180/jenkins and you should see the welcome screen. Replace the [IP] with the IP address of your server. You should see this screen.

Cat out that file, copy & paste the value in the browser and click Continue.

cat /usr/local/jenkins/secrets/initialAdminPassword
c33a584df234433d9a88e19d8e14c289

And that’s it. Follow the prompts to install plugins, provide admin password etc…The jenkins installation on FreeBSD is accessed with a suffix /jenkins. If you try to access it without the suffix, you’ll get this error.

HTTP ERROR 404
Problem accessing /. Reason:
    Not Found

It’s much easier to do the redirects with nginx without the suffix, so we’ll change it.
Edit this file /usr/local/etc/rc.d/jenkins and around line 54 find this.

: ${jenkins_args="--webroot=${jenkins_home}/war --httpPort=8180 --prefix=/jenkins"}

Change it so it looks like this.

: ${jenkins_args="--webroot=${jenkins_home}/war --httpPort=8180 --prefix=/"}

Restart Jenkins and you’ll be able to access it as http://[IP]:8180.

service jenkins restart

nginx as reverse proxy

In order to access the server with out the need to specify the port 8180 and use a DNS name, we’ll have to use nginx that will listen on port 80 or 443 and redirect the traffic to 8180. Let’s install and configure nginx. In my case, I’ll access the server as jenkins.domain.com.

pkg install nginx

Run on boot.

sysrc nginx_enable=YES

Edit /usr/local/etc/nginx/nginx.conf and delete everything after this line around line 39 after #gzip on directive. Add the include line, so the last three lines look like this.

    #gzip  on;
    include /usr/local/etc/nginx/conf.d/*.conf;
}

Create a new conf.d folder mkdir /usr/local/etc/nginx/conf.d and create a new file called jenkins.conf.
This is how my jenkins.conf file looks like. Change the domain in the highlighted line.

server {
    listen 80;
    server_name jenkins.domain.com;

    location / {
        proxy_set_header   X-Forwarded-For $remote_addr;
        proxy_set_header   Host $http_host;
        proxy_pass         "http://127.0.0.1:8180";
    }
}

Restart Jenkins and you should be able to access the site now with http://jenkins.domain.com.
For HTTPS traffic, the configuration is different. You’ll need certificates specified in lines 15 and 16 (under /usr/local/etc/nginx) and the domain changed in 7,13 and 31. Here is the config (/usr/local/etc/nginx/conf.d/jenkins.conf) in order to access jenkins over SSL.

upstream jenkins {
  server 127.0.0.1:8180 fail_timeout=0;
}

server {
  listen 80;
  server_name jenkins.domain.com;
  return 301 https://$host$request_uri;
}

server {
  listen 443 ssl;
  server_name jenkins.domain.com;

  ssl_certificate public_cert.pem;
  ssl_certificate_key private_cert.key;
  client_max_body_size 4M;

  location / {
    proxy_set_header        Host $host:$server_port;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto $scheme;
    proxy_redirect http:// https://;
    proxy_pass              http://jenkins;
    # Required for new HTTP-based CLI
    proxy_http_version 1.1;
    proxy_request_buffering off;
    proxy_buffering off; # Required for HTTP-based CLI to work over SSL
    # workaround for https://issues.jenkins-ci.org/browse/JENKINS-45651
    add_header 'X-SSH-Endpoint' 'jenkins.domain.com:50022' always;
  }
}

You should also restart nginx for the changes to take effect.

service nginx restart

That’s it. You should be able to access Jenkins as jenkins.domain.com over https.

FreeBSD: Install Prometheus, Node Exporter and Grafana

Kliment Andreev — Sat, 20 Apr 2019 22:49:05 +0000

FreeBSD comes out of the box with three great tools for monitoring. If you need more info about how these tools work, please read the official documentation. I’ll explain the installation only and creating a simple dashboard.
The installation is very simple. These amazing tools don’t need any external libraries. We’ll install the packages and enable them to start on boot.

pkg install prometheus node_exporter grafana5
sysrc prometheus_enable=YES
sysrc node_exporter_enable=YES
sysrc grafana_enable=YES

Add these lines at the very end in /usr/local/etc/prometheus.yml. Make sure you have the proper spacing. Check the job_name above. They have to be in line.

- job_name: 'node_exporter'
  scrape_interval: 5s
  static_configs:
    - targets: ['localhost:9100']

Looks like this.

Now start the services.

service prometheus start
service node_exporter start
service grafana start

Go to http://your_ip:9090. This is how it should look like.

Type prometheus_http_requests_total or node_memory_active_bytes in the Expression field and click on Execute. You should see some output below.

Now go to Grafana at http://your_ip:3000. Log as admin/admin and you’ll be prompted to change your password. From the Home Dashboard click on Add Source, right in the middle.

Click on the Prometheus icons from the choice of the others. You’ll be prompted to fill out this form. Just make sure the URL is http://localhost:9090. Click Save & Test and make sure it’s OK.

Now, in the upper left corner where the Grafana logo is, click on the Plus (+) sign and choose Import. In another browser tab go to this link and on the right side click on Download JSON. Save it somewhere. Go back to your Grafana dashboard and click on Upload .json file.

Find the JSON file that you saved in the previous step and upload it. You’ll see this screen. Just make sure to choose Prometheus under localhost parameter and click Import.

That’s it. You’ll see the node_exporter’s dashboard.

Now, you can add dashboards. Click on the (+) on the left, click on Dashboard, click on the arrow next to Panel Title and select Edit. Add some metrics such as this example below and click on Add Query.

Finally click on the floppy icon on top to save the dashboard. It looks like this after.

You can also go to this site and use some community dashboards for various metrics and applications.

FreeBSD: Install an authoritative DNS server (BIND)

Kliment Andreev — Tue, 23 Oct 2018 14:13:54 +0000

In this post I’ll explain how to install and configure BIND DNS server to act as an authoritative server for a public domain in a master/slave configuration. This is not a setup for a server that will act as a DNS server in your local environment and does DNS resolution for your local network. In this post, the DNS server won’t resolve any DNS queries except for the domain that it is authoritative for. The domain that I’ll use is cloudwerk.us registered with Go Daddy, so some of the configurations will apply to Go Daddy only. The IPs and keys that I’ll use will be visible. The setup will be gone by the time you read this.
So, I have my master server, ns1.cloudwerk.us (104.248.47.54) and the slave server ns2.cloudwerk.us (142.93.201.231).
I will be using FreeBSD 11.2 and BIND 9.12.2-P2. Before you start, make sure you have your domain purchased and you have 2 servers with public IPs up and running. You can leave the default registrant’s DNS servers for now.

Installation, startup and control

First, I’ll install BIND on both servers. All commands below are executed as root.

pkg install bind912

When the installation is done after 10-15 seconds, you’ll see some text that requires attention. I am not using a jailed install, so I can skip that part. The only part that’s important is to create the key for the rndc (Remote Name Daemon Control) utility. This utility allows management of the named (DNS) daemon. Moreover, you can manage remote DNS servers and I’ll explain how. So, let’s create the keys (highlighted line is what you type, the rest is output). Do this on the master server only.

rndc-confgen -a
wrote key file "/usr/local/etc/namedb/rndc.key"

As you can see, there is a file called rndc.key created under /usr/local/etc/namedb directory. This is the default directory where BIND expects to find its config files and zones.
Now, copy this file to the second server in the same location. You can use scp, copy & paste in a terminal session, it doesn’t matter. What matters is that you have to change the permissions and the owner of the file. BIND runs under the bind user, not root. Do this on both servers.

cd /usr/local/etc/namedb
chown root:bind rndc.key
chmod 640 rndc.key

Let’s make sure that BIND starts on boot. On both servers, execute:

sysrc named_enable="YES"

Now, we can start the BIND daemon on both.

service named start

Check the logs.

tail /var/log/messages
Oct 22 15:11:11 ns1 named[1161]: ----------------------------------------------------
Oct 22 15:11:11 ns1 named[1161]: BIND 9 is maintained by Internet Systems Consortium,
Oct 22 15:11:11 ns1 named[1161]: Inc. (ISC), a non-profit 501(c)(3) public-benefit
Oct 22 15:11:11 ns1 named[1161]: corporation.  Support and training for BIND 9 are
Oct 22 15:11:11 ns1 named[1161]: available at https://www.isc.org/support
Oct 22 15:11:11 ns1 named[1161]: ----------------------------------------------------
Oct 22 15:11:11 ns1 named[1161]: command channel listening on 127.0.0.1#953
Oct 22 15:11:11 ns1 named[1161]: command channel listening on ::1#953
Oct 22 15:11:11 ns1 named[1161]: all zones loaded
Oct 22 15:11:11 ns1 named[1161]: running

We can’t start the server with the rndc utility, but we can stop it if needed with rndc stop or simply service named stop.
To control the slave server from the master, we have to modify the main BIND configuration file, named.conf on the slave server. Also, the keys have to be the same, but we already took care of that part above.
So, in the same /usr/local/etc/namedb directory, edit named.conf and add these lines, right before the options directive. Do this on the slave server, not the master.

include "/usr/local/etc/namedb/rndc.key";

controls {
        inet 127.0.0.1 allow { localhost; } keys { "rndc-key"; };
        inet 142.93.201.231 allow { 104.248.47.54; } keys { "rndc-key"; };
};

If you look at my IPs below, this means that the rndc utility will be accepted from the localhost and from the the IP 104.248.47.54 which is the master server. But, we also have to tell BIND to listen on the public IP. By default, it listens on the localhost only.
Do this on BOTH servers. In named.conf find the options directive and modify the listen-on option so it looks like this. You should put the public IP of each server there, so the line will differ on both servers.

//Master server
listen-on       { 127.0.0.1; 104.248.47.54; };

//Slave server
listen-on       { 127.0.0.1; 142.93.201.231; };

Restart BIND on both.

service named restart

Now, you can manage them with rndc.

rndc reload
rndc status

From the master server, try to reload the slave server.

rndc -s 142.93.201.231 reload
server reload successful

If you have a firewall, make sure port 953 TCP is open for communication.
What we also want to do is tell the OS that from now on, the default DNS server will be our local DNS.
So, edit /etc/resolv.conf and add this line as the first line on both.

nameserver 127.0.0.1

If we try to ping google.com we should get a response or at least an IP resolved. That’s good. From another computer, try to use any of our servers as a DNS. This is what you should get: Query refused. That’s good too, because we don’t want our server to be used as a DNS server for other domains. But, also, we want to access the Internet from the server, for example to update the server. Name resolution for other domains from localhost is OK, but not OK from outside.

C:\Users\Elite>nslookup
Default Server:  ns-cache.above.net
Address:  207.126.96.248

> server 142.93.201.231
Default Server:  ns2.cloudwerk.us
Address:  142.93.201.231

> www.google.com
Server:  ns2.cloudwerk.us
Address:  142.93.201.231

*** ns2.cloudwerk.us can't find www.google.com: Query refused

You can allow this server to act as a forwarder by including recursion yes; option under options and allow-access to all IPs, but that’s not the intended use of an authoritative DNS server.

Zone file

Now that we have the servers up and running, let’s create the zone file. The zone file tells where the records can be found. On the master server only, edit named.conf and add these two lines right after the listen-on option. We are allowing the transfer of the zones to the slave server only

allow-transfer { localhost; 142.93.201.231; };
notify yes;

Go all the way at the bottom and add these line to define the zone file.
NOTE: If you don’t want to use the dynamic directory, make sure that your zone file directory is owned by the bind user and group (e.g. mkdir somedir, chown -R bind:bind somedir).

zone "cloudwerk.us" {
        type master;
        file "/usr/local/etc/namedb/dynamic/db.cloudwerk.us";
};

Now, go to the /usr/local/etc/namedb/dynamic directory and create the zone file to look like this. The file should be called db.cloudwerk.us. Do this on the master server only. You might want to read the following link if you are not familiar with the syntax of a zone file.

$TTL    604800
@       IN      SOA     ns1.cloudwerk.us.       admin.cloudwerk.us. (
                        2018101901;     Serial
                        3H;             Refresh
                        15M;            Retry
                        2W;             Expiry
                        1D );           Minimum

; name servers - NS records
        IN      NS      ns1.cloudwerk.us.
        IN      NS      ns2.cloudwerk.us.

; name servers - A records
ns1.cloudwerk.us.       IN      A       104.248.47.54
ns2.cloudwerk.us.       IN      A       142.93.201.231

; other servers - A records
www.cloudwerk.us.       IN      A       52.71.63.84

All you have to do now is to reload the new config with rndc reload. At this point we have the master server up and running. Now, we have to configure the slave server. Edit the named.conf file and add these lines at the bottom. Very self-explanatory.

zone "cloudwerk.us" {
        type slave;
        masters { 104.248.47.54; };
        file "/usr/local/etc/namedb/dynamic/db.cloudwerk.us";
};

Restart the named daemon and if you go to /usr/local/etc/namedb/dynamic directory, you’ll see that the zone file replicated from the master.
At this point, we have both the master and the slave servers configured, up and running. But no one on the Internet knows that our servers are authoritative servers. You have to make this change at your domain registrant. In my case, I’ve purchased the domain from Go Daddy, so I went to the DNS management panel and add two A records for ns1.cloudwerk.us and ns2.cloudwerk.us. Then, at the bottom of the web contol panel, I’ve changed the DNS from xy.domaincontrol.com to custom and added my servers. I’ve encountered some issues, but the on-line chat with the support resolved this in 5 minutes. In essence, if you open up the DNS console, scroll all the way down and under Advanced Features, you’ll see Host names link. Click on this one and create two records, ns1 and ns2 with your IPs. Now, you can switch to Custom Nameserves on the same page.
Leave some time for replication. It might take up to 24 hrs, but in my case it took less than 20 mins.
You should know that you are in charge when these commands return your servers. In Windows:

C:\Users\Elite>nslookup
Default Server:  ns-cache.above.net
Address:  207.126.96.248

> set query=soa
> cloudwerk.us
Server:  ns-cache.above.net
Address:  207.126.96.248

Non-authoritative answer:
cloudwerk.us
        primary name server = ns1.cloudwerk.us
        responsible mail addr = admin.cloudwerk.us
        serial  = 2018101901
        refresh = 10800 (3 hours)
        retry   = 900 (15 mins)
        expire  = 1209600 (14 days)
        default TTL = 86400 (1 day)

cloudwerk.us    nameserver = ns2.cloudwerk.us
cloudwerk.us    nameserver = ns1.cloudwerk.us
ns1.cloudwerk.us        internet address = 104.248.47.54
ns2.cloudwerk.us        internet address = 149.93.201.231

In Linux/BSD if you use dig:

dig +short NS cloudwerk.us
ns2.cloudwerk.us.
ns1.cloudwerk.us.

Or, if you want to do it on-line.
This is what I got.

It says that the slave server is leaking the zone which means anyone can list all the records. In order to prevent that, we have to edit named.conf on the slave server and add this line after listen-on option.

allow-transfer { localhost; 104.248.47.54; };

Do rndc reload after and if you re-execute the test, you’ll see that everything looks OK.

DNSSEC (optional)

I won’t go into details of what is DNSSEC, I’ll just explain how to configure and secure your DNS server. It’s pretty much a PKI infrastructure for securing your records by signing them with a private key. If you need more info, you can find tons of info by googling DNSSEC.
On the master server, add these lines after listen-on option in named.conf.

dnssec-enable yes;
key-directory "/usr/local/etc/namedb/keys";

On the slave server, add this line after listen-on option in named.conf.

dnssec-enable yes;

Add these lines in the same file on the master server only. They should be below file directive where your zone is defined (at the bottom of the file).

auto-dnssec maintain;
inline-signing yes;

On the master server only, create they keys directory.

mkdir /usr/local/etc/namedb/keys
cd /usr/local/etc/namedb/keys

Let’s create the KSK and ZSK keys. Master server only.

dnssec-keygen -r /dev/urandom -a RSASHA256 -b 2048 -f KSK -n ZONE cloudwerk.us
ls -l
echo "Take a note of the key file"
dnssec-keygen -r /dev/urandom -a RSASHA256 -b 2048 -n ZONE cloudwerk.us

You’ll see 4 files created. Scribble down the first key file that was created, see the output from above. Step one directory up and change the ownership so BIND can read these files.

cd /usr/local/etc/namedb
chown -R bind:bind keys
rndc reload

If you go to the dynamic directory where your zone file is, you’ll see three more files. These are the signed files.

ls -l /usr/local/etc/namedb/dynamic

If you go on the slave server in the same dynamic directory, you’ll see that signed .jnl file also transferred.
Let’s do a test. Go to this link, type your domain name and hit enter.
You should see something like this.

Now, stop the master server with rndc stop and refresh the web page. The test will take a little bit longer, because NS1 is down, but eventually you’ll get something like this.

That’s normal. It tells us that ns1 is down. What’s not normal is the red X mark that says “No DS records found for coudwerk.us in the us zone”. This translates as, yes you signed your zone, but no one can validate it. It’s like a fake certificate. So, we need to tell our registrant that we really own the domain and that we signed it.
This portion is also registrant’s dependent. I’ll describe how to overcome this error with Go Daddy.
So, go to the keys directory and execute this command against the first file.

cd /usr/local/etc/namedb/keys
dnssec-dsfromkey Kcloudwerk.us.+008+23956.key
cloudwerk.us. IN DS 23956 8 1 2DA94FECA549E0A2B137FA6C64D09D2CBADF8864
cloudwerk.us. IN DS 23956 8 2 8CC2CB79E1B8A6767F11230485C7499D52162135E1F6DAA503ADEC4470E417A9

Now go to the DNS management panel for your domain at Go Daddy and you’ll see something like this. Click on DNSSEC and then click ADD.

Looks confusing but it’s very simple. Just look at the output from the dnssec-dsfromkey and add accordingly. You’ll have to add both hash values.

For Key Tag enter 23956, Algorithm is 8, Digest Type is 1, Digest is the hash value and Key Data Algorithm is also 8, the value of Algorithm. Click Update , do the 2nd line of the output and wait 30-45 mins. Sometimes, even more. It depends on your registrant.

At the end it should look like this.

And if you analyze your domain again, the result should look like this.

Or if you use dig, use the Google’s DNS server to check.

dig +trace +noadditional DS cloudwerk.us. @8.8.8.8 | grep DS

Windows, CentOS, FreeBSD: ssh, RDP – Bypass your corporate firewall

Kliment Andreev — Fri, 23 Mar 2018 17:04:18 +0000

If you are behind a firewall and want to access some resources over SSH or RDP, most likely you won’t be able to do that. It’s because most of the corporate firewalls allow only ports 80 and 443 outbound. But you can still bypass that by tunneling everything over port 443. You will need a software called shellinabox to tunnel SSH over 443 and RDP gateway to tunnel the RDP traffic over 443. I’ll present the shellinabox solution for both CentOS 7 and FreeBSD 11. Both of these servers will be sitting somewhere in the cloud or behind your home firewall. The only requirement is to have port 443 opened and accessible on Internet. You might have a public IP or your home firewall will forward the traffic to 443, it doesn’t matter. Another option for connecting to any port is to use putty and a Linux/FreeBSD instance. I’ll describe that option last.

CentOS 7

shellinabox doesn’t come up with the default packages, so you have to install the EPEL release first.

yum install epel-release
yum install shellinabox
systemctl enable shellinaboxd

Edit the configuration file for shellinabox which is /etc/sysconfig/shellinaboxd. Make sure it looks like this.

# Shell in a box daemon configuration
# For details see shellinaboxd man page
# Basic options
USER=shellinabox
GROUP=shellinabox
CERTDIR=/var/lib/shellinabox
PORT=4200
OPTS="--css white-on-black.css -t -s /:SSH:localhost"
#OPTS="--css color.css -t -s /:SSH:localhost"

shellinabox runs on port 4200 by default. You can change it to run on 443, but you have to run as root. A better solution is to install Apache and use the mod_proxy so the Apache will listen on 443 and forward the traffic to 4200.

yum install httpd mod_ssl
systemctl enable httpd

Create the configuration file for the Apache server. In my case, I’ll access the SSH over HTTPS as sshtest.iandreev.com. Change the config below to suit your needs.

cd /etc/httpd/conf.d
touch httpd-vhosts.conf

Edit httpd-vhosts.conf and paste the following.


    SSLEngine On
    SSLCertificateFile /etc/pki/tls/certs/sshtest.iandreev.com.crt
    ServerAdmin klimenta@iandreev.com
    ServerName sshtest.iandreev.com
    ErrorLog "/var/log/httpd/sshtest.iandreev.com-error_log"
    CustomLog "/var/log/httpd/sshtest.iandreev.com-access_log" combined
    ProxyRequests On
    ProxyPreserveHost On
    
        AuthUserFile /var/www/sshtest.iandreev.com/.htpasswd
        AuthName EnterPassword
        AuthType Basic
        require user ssh.admin
        Order deny,allow
        Allow from all
    
    ProxyPass / http://localhost:4200/
    ProxyPassReverse / http://localhost:4200/

HTTPS requires a certificate, we’ll create a fake one. If you have a valid certificate, just put it under /etc/pki/tls/certs as sshtest.iandreev.com.crt.

cd /etc/pki/tls/certs
./make-dummy-cert sshtest.iandreev.com.crt

shellinabox when started will give you a SSH prompt so you can login to your server. A more secure solution is to protect the access even more with a username and password. Anytime you access your server, you’ll get prompted with a username and password and then you’ll get prompted with your SSH credentials.

cd /var/www
mkdir sshtest.iandreev.com
cd sshtest.iandreev.com
htpasswd -c .htpasswd ssh.admin
cd ..
chown -R apache:apache sshtest.iandreev.com

I’ve created a user called ssh.admin and the commands above will ask you for a password.
Now, it’s time to start shellinabox.

systemctl start httpd
systemctl start shellinaboxd

On a laptop behind your comporate firewall, go to https://sshtest.yourdomain.com and you should get prompted for ssh.admin’s password. Once you pass that you’ll see the login prompt in your browser. From here you can SSH to any server that has port 22 opened.
shellinabox comes with two styles. If you see above in it’s config, we provided these two lines.

OPTS="--css white-on-black.css -t -s /:SSH:localhost"
#OPTS="--css color.css -t -s /:SSH:localhost"

If you prefer black on white background, uncomment the last line, save the config file and restart shellinabox.
Depending on your CentOS install, you might have firewall and SElinux enabled. If these are not configured, shellinabox won’t work. For the firewall, you’ll have to allow port 443 inbound.

firewall-cmd --add-service=https --permanent
firewall-cmd --reload

For SElinux, you’ll have to allow Apache to make outbound connections.

/usr/sbin/setsebool -P httpd_can_network_connect 1

At the end, it will look like this.

…and this.

FreeBSD 11

We have to install Apache and shellinabox first.

pkg install shellinabox apache24

Make sure they start on boot. Add these two lines in /etc/rc.d

apache24_enable="YES"
shellinaboxd_enable="YES"
shellinaboxd_flags="--disable-ssl --css=/usr/local/www/sshtest.iandreev.com/style.css"

Go to the Apache config directory and edit the config file /usr/local/etc/apache24/httpd.conf. Make sure these lines are uncommented.

Include etc/apache24/extra/httpd-vhosts.conf
LoadModule authn_socache_module libexec/apache24/mod_authn_socache.so
LoadModule socache_shmcb_module libexec/apache24/mod_socache_shmcb.so
LoadModule ssl_module libexec/apache24/mod_ssl.so
Include etc/apache24/extra/httpd-ssl.conf
LoadModule proxy_module libexec/apache24/mod_proxy.so
LoadModule proxy_http_module libexec/apache24/mod_proxy_http.so
ServerName www.example.com:80

Go to /usr/local/etc/apache24/extra folder and make sure you have the definitiopn for the virtual host there. Change it to suit your needs.


    SSLEngine On
    SSLCertificateFile /usr/local/share/certs/sshtest.iandreev.com.crt
    SSLCertificateKeyFile /usr/local/share/certs/sshtest.iandreev.com.key
    ServerAdmin klimenta@iandreev.com
    ServerName sshtest.iandreev.com
    ErrorLog "/var/log/sshtest.iandreev.com-error_log"
    CustomLog "/var/log/sshtest.iandreev.com-access_log" combined
    ProxyRequests On
    ProxyPreserveHost On
    
        AuthUserFile /usr/local/www/sshtest.iandreev.com/.htpasswd
        AuthName EnterPassword
        AuthType Basic
        require user ssh.admin
        Order deny,allow
        Allow from all
    
    ProxyPass / http://localhost:4200/
    ProxyPassReverse / http://localhost:4200/

In the same directory, edit httpd-ssl.conf file and make sure it looks like this.

SSLRandomSeed startup file:/dev/urandom 512
Listen 443
SSLCipherSuite HIGH:MEDIUM:!MD5:!RC4
SSLProxyCipherSuite HIGH:MEDIUM:!MD5:!RC4
SSLHonorCipherOrder on
SSLProtocol all -SSLv3
SSLProxyProtocol all -SSLv3
SSLPassPhraseDialog  builtin
SSLSessionCache        "shmcb:/var/run/ssl_scache(512000)"
SSLSessionCacheTimeout  300

We’ll protect shellinabox with extra username (ssh.admin) and password.

cd /usr/local/www
mkdir sshtest.iandreev.com
cd sshtest.iandreev.com
htpasswd -c .htpasswd ssh.admin

We’ll need a certificate for the HTTPS site. Use your own or create a fake one. Hit ENTER for everything prompted. It’s a fake certificate.

cd /usr/local/share/certs
openssl genrsa -out sshtest.iandreev.com.key 2048
openssl req -new -key sshtest.iandreev.com.key -out sshtest.iandreev.com.csr
openssl x509 -req -days 3650 -in sshtest.iandreev.com.csr -signkey sshtest.iandreev.com.key -out sshtest.iandreev.com.crt

Unlike CentOS, FreeBSD shellinabox doesn’t come up with CSS files for the color, so we can use these two. Copy these files under /usr/local/www/sshtest.iandreev.com as blackonwhite.css and whiteonblack.css.
This is blackonwhite.css. Click to expand.

#vt100 .ansiDefR {
  color:            #ffffff;
}

#vt100 .bgAnsiDefR {
  background-color: #123450;
}

#vt100 #scrollable.inverted .ansiDefR {
  color:            #000000;
}

#vt100 #scrollable.inverted .bgAnsiDefR {
  background-color: #ffffff;
}

#vt100 .ansiDefR {
  color:            #ffdfd0;
}

#vt100 .bgAnsiDefR {
  background-color: #010203;
}

#vt100 #scrollable.inverted .ansiDefR {
  color:            #002030;
}

#vt100 #scrollable.inverted .bgAnsiDefR {
  background-color: #1f1fff;
}

This is whiteonblack.css. Click to expand.

#vt100 #cursor.bright {
  background-color: white;
  color:            black;
}

#vt100 #cursor.dim {
  background-color: black;
  opacity:          0.2;
  -moz-opacity:     0.2;
  filter:           alpha(opacity=20);
}

#vt100 #scrollable {
  color:            #ffffff;
  background-color: #000000;
}

#vt100 #scrollable.inverted {
  color:            #000000;
  background-color: #ffffff;
}

#vt100 .ansiDef {
  color:            #ffffff;
}

#vt100 .ansiDefR {
  color:            #000000;
}

#vt100 .bgAnsiDef {
  background-color: #000000;
}

#vt100 .bgAnsiDefR {
  background-color: #ffffff;
}

#vt100 #scrollable.inverted .ansiDef {
  color:            #000000;
}

#vt100 #scrollable.inverted .ansiDefR {
  color:            #ffffff;
}

#vt100 #scrollable.inverted .bgAnsiDef {
  background-color: #ffffff;
}

#vt100 #scrollable.inverted .bgAnsiDefR {
  background-color: #000000;
}

Copy one of them to be your style. Anytime you change the style, restart shellinabox.

cd /usr/local/www/sshtest.iandreev.com
cp blackonwhite.css style.css
cd ..
chown -R www:www sshtest.iandreev.com

Finally, start Apache and shellinabox.

service shellinaboxd start
service apache24 start

Access your server from a laptop behind your corporate firewall as https://sshtest.domain.com.

Windows 2016

You will need a Windows 2016 server with a public IP and port 443 allowed or you can use a Windows server behind your home network as long as port 443 is allowed. In order to bypass the RDP restriction, we’ll tunnel the RDP traffic over HTTPS using Remote Desktop Gateway.
From the Server Manager, go to Add Roles and Features. Select Remote Desktop Services.

Click Next 2-3 times and then select Remote Desktop Gateway. Click Next again and accept all the defaults. Windows will install some other components for you.
Once everything is installed, from the Server Manager’s menu click on Tools, Remote Desktop Services and then Remote Desktop Gateway Manager.
Click on the server name and in the middle pane you’ll see what you have to do.

Click on the first link, View or modify certificate properties. Choose to create a fake certificate or you can import your own. It has to be in p12 format, not PEM.

If you decide to go with a fake certificate, enter the FQDN of the server, e.g sshtest.iandreev.com. You will have to make sure that sshtest.iandreev.com resolves to the public IP of the Windows box or if you have an internal server in your home lab, then the external IP of your cable/DSL modem. Then just click on the button Create and Import Certificate, enter the FQDN sshtest.iandreev.com and then click OK when prompted. Click Apply and OK to go back.
At this point, you might want to create a user or a group that you can allow access to the Gateway.
I created a user called RDP. Back in the RD Gateway Manager, select Policies under the server name, right-click on it and choose Create New Authorization Policies. Choose the option to create both RD CAP and RD RAP policies. Here is what I did in the wizard config.

I choose BUILTIN\Users to be able to use the Gateway. The generic user RDP that I created is by default a member of the users group.

Select the default Enable device redirection for all client devices.
Check both checkmarks for Idle Timeout and Session Timeout. This is optional, but it’s good to have.
Click Next and then create the Resource Authorization Policy.

Accept the same group (BUILTIN\Users).
Choose the option at the bottom, Allow users to connect to any network resource (computer).
Choose the first option, Allow connections only to port 3389.
Click Next and Finish.
So, how do you use this solution now? Easy…
All you have to do is go to your corporate laptop and create a new RDP connection. Under the General tab enter the IP address of the Windows box that you want to reach. This is the box that listens on 3389 and that you are not able to reach directly. Mind that the Windows server that we just built is a gateway, so the Windows RD Gateway server should be able to talk to the destination server over 3389 and your corporate laptop will talk to Windows RD Gateway server over 443.

Click on the Advanced tab and then the Settings button. Select to Use these RD Gateway server settings and enter the FQDN (sshtest.iandreev.com) of the RD Gateway server that we just built.

Once you are done, click Connect and you should get prompted to enter the credentials for the RD Gateway (in my case the username and password for the RDP user) and then you’ll have to enter the username and password for the destination server.
NOTE: Make sure you use .\rdp for the username, not rdp. You can also get an error saying that the identity of the RD Gateway can’t be verified. This is most likely if you messed up the certificate and it doesn’t match the hostname. In that case, the RDP client will allow you to view the certificate. Then copy it to a file and import it on the local machine under the Trusted Root Cert Authorities.

Tunnel over putty

Let’s say that your corporate firewall allows only port 443 outbound. In this case, we’ll build a Linux/BSD server with a public IP and change the SSH listener port from 22 to 443. For this, edit sshd_config under /etc/ssh/sshd_config or /usr/local/etc/ssh/sshd_config. Look for the directive Port 22 and change it to Port 443. Restart the ssh service.
On your client machine behind the corporate firewall, open putty and under Session enter the public IP of your Linux/BSD server and port 443 so it looks like this.

Now, expand the Connection on the left side, then SSH and select Tunnels. For the source port enter 3390 and for the destination enter the IP of your Windows box where you want to connect.

Finally, save this session, open it and login to the Linux server to establish the tunnel. You should be able to run the RDP client (mstsc) and RDP to 127.0.0.1:3390.

CentOS, FreeBSD, Windows: rsyslog server and client

Kliment Andreev — Sun, 03 Dec 2017 23:56:22 +0000

In this post, I’ll explain how to configure a rsyslog server and client on various operating systems. For the servers, I’ll chose 2 Red Hat 7 servers and the clients will be AWS Linux, FreeBSD and Windows 2016.
rsyslog is an open-source utility for logging, a derivate of the original syslog. As defined by RFC 3164, each message includes the facility code and severity level.

The severity levels are these. (Both screenshots are from Wikipedia).

RHEL 7 rsyslog servers

RHEL 7 comes with rsyslog installed by default. Make sure it’s running.

systemctl status rsyslog

The configuration file is /etc/rsyslog.conf. Make sure that these lines are uncommented. They tell the rsyslog daemon to accept TCP and UDP connections.

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

If you make changes in /etc/rsyslog.conf, make sure you verify the config.

rsyslogd -N 1

If you make a change, you’ll have to restart the rsyslog daemon.

systemctl restart rsyslog

To test if rsyslog works fine, try a test.

logger -p local0.notice -t from_cmd_line "Test Message"

If you check the /var/log/messages, a file where by default rsyslog writes the output, you’ll see the message.

tail /var/log/messages

OK, so we didn’t have to make any changes on both RHEL servers. Let’s move to the clients.
NOTE: If you have firewalld enabled, you’ll have to open the ports for the clients.

firewall-cmd --permanent --zone=public --add-port=514/tcp
firewall-cmd --permanent --zone=public --add-port=514/udp
firewall-cmd --reload

AWS Linux, RHEL 6, CentOS 6

AWS Linux is based on RHEL 6 so the instructions will apply for all of them. Check if rsyslog is running.

service rsyslog status

Edit /etc/rsyslog.conf and make sure these lines are commented. We don’t want a server, we want a client.

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514

Add these lines (11 to 14) somewhere at the end.

#$WorkDirectory /var/lib/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

*.*     @@10.0.0.170
$ActionExecOnlyWhenPreviousIsSuspended on
& @@10.0.0.225
$ActionExecOnlyWhenPreviousIsSuspended off

# Finally include all config files in /etc/rsyslog.d. This allows overrides
# of the default configuration above.
$IncludeConfig /etc/rsyslog.d/*.conf

These four lines means that my primary rsyslog server has an IP of 10.0.0.170 and in case it’s not reachable, use the server 10.0.0.225. Line 11 tells the rsyslog client to log everything to the server. If you want only certain messages logged, than you can change it, e.g. instead of *.* @@10.0.0.170 you can use something like *.info @@10.0.0.170 or *.info;auth.err @@10.0.0.170. Look at those two tables above. If you run this command on the client.

logger -p local0.notice -t from_aws_linux "Test Message"

and do

tail -f /var/log/messages

on the server, you’ll see the message on the server, not on the client.

FreeBSD

FreeBSD uses the old syslog, not rsyslog, but we can still redirect the logs to our RHEL servers. Edit /etc/syslog.conf and all the way at the end add this line.

*.* @10.0.0.170

As you can see, syslog uses one @, not two @@s. Restart the service and send a test message. You’ll see that the message goes to the RHEL server now.

service syslogd restart
logger -p local0.notice -t from_FreeBSD "Test Message"

Windows

Go to this link and download the rsyslog client. You can see the link for the download and also for the manual.

The rsyslog agent for Windows is very detailed and requires some studying. Use the manual, there are some examples there. I’ll show you how to forward the event ID 7036. This event gets triggered when a service changes its state. So, download the client and open up the GUI. Expand the following and click on Rsyslog. Enter the primary and secondary rsyslog server’s IPs. Click Save in the upper left corner and then click Restart.

If you check the logs on your rsyslog server, you’ll see a lot of logging going on. Pretty much everything that Windows does. We just want to filter all these logs to a single event ID.
So, in the Default Rule Set, click Filters, then right-click AND, choose Add Filter, Event Log Monitor and then Event ID.

Enter the Event ID 7036, click Save and Restart.

Now, go to Services (services.msc) and restart Windows Update service. You should see this on the rsyslog server.

FreeBSD: Install nginx – virtual hosts, PHP, MySQL and WordPress

Kliment Andreev — Sun, 29 Jan 2017 15:40:02 +0000

I’ve decided to play with nginx to see how different is from Apache. The best way to experiment is to have an end goal, e.g. a working WordPress site. I will be using FreeBSD 11.0 and for that I’ve created a public server in AWS cloud. The hostname for my server is test.iandreev.com and I also have 4 different hostnames that point to the same IP (test1, test2, test3 and test4).

Installation

To install nginx, type:

pkg install nginx

Make sure it starts at boot by adding nginx_enable=”YES” in /etc/rc.conf.
To see the version, type:

nginx -v
nginx version: nginx/1.10.2

To see the modules that were installed with the package, type:

nginx -V

You can start and stop the service with the standard service command. When making changes in the config file, use reload, e.g. service nginx reload.
By default, nginx creates the log files under /var/log/nginx. There are two files, access.log and error.log. The config files are under /usr/local/etc/nginx. The main config file is nginx.conf.
Let’s start the nginx.

service nginx start
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Starting nginx.

If everything is OK, you can telnet localhost 80. Type something and hit ENTER and you’ll see a response.

telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
asdf
HTTP/1.1 400 Bad Request
Server: nginx/1.10.2
Date: Sun, 29 Jan 2017 13:56:04 GMT
Content-Type: text/html
Content-Length: 173
Connection: close


400 Bad Request

400 Bad Request
nginx/1.10.2


Connection closed by foreign host.

Check the nginx process.

ps -waux | grep nginx
root     778   0.0  0.7 29640 6984  -  Is   13:55    0:00.00 nginx: master process /usr/local/sbin/nginx
www      779   0.0  0.7 29640 7452  -  I    13:55    0:00.00 nginx: worker process (nginx)

As you can see the master process runs as root, but then it forks the connections to the www user.

Single host

This is the easiest setup. You don’t have to do anything except the steps above. nginx works right out of the box with the default configuration. Go to http://server_IP_or_hostname and you’ll see the welcome page.

But, this scenario has no real use, so we’ll create two virtual hosts, test1 and test2.

Virtual hosts

First, we’ll create the two directories for the virtual hosts and put some index files there. I use the same root directory as Apache. Make sure that the locations of the virtual hosts are owned by the user www and group www.

cd /usr/local/www
mkdir test1.iandreev.com
mkdir test2.iandreev.com
echo "test1" > test1.iandreev.com/index.htm
echo "test2" > test2.iandreev.com/index.htm
chown -R www:www test*

Now, we have to edit the main config file and tell nginx where to look for these sites.
Go to /usr/local/etc/nginx/ and edit nginx.conf. Change the values for worker_processes from 1 to auto and uncomment gzip on directive. See what worker_processes mean here. By uncommenting gzip directive, you are enabling compression on the pages. To test if your web server serves compressed pages, check this website.
Next, we have to add our two virtual hosts. Scroll all the way down, just before the last accolade “}” and add these lines.

    server {
        access_log /var/log/nginx/test1.iandreev.access.log;
        error_log  /var/log/nginx/test1.iandreev.error.log;
        listen 80;
        server_name     test1.iandreev.com;
        location / {
            root /usr/local/www/test1.iandreev.com;
            index index.htm index.html;
        }
    }
    server {
        access_log /var/log/nginx/test2.iandreev.access.log;
        error_log  /var/log/nginx/test2.iandreev.error.log;
        listen 80;
        server_name test2.iandreev.com;
        location / {
            root /usr/local/www/test2.iandreev.com;
            index index.htm index.html;
        }
    }

Reload the config with:

service nginx reload
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful

and make sure everything is OK.
If everything is OK, you should be able to browse to your virtual sites.

If you check one of the error logs, you’ll see an error that favicon.ico is missing for the site. You can ignore this as it’s not an error, it’s just a warning that you don’t have a favicon for your site.

tail test2.iandreev.error.log
2017/01/29 14:18:59 [error] 838#100084: *4 open() "/usr/local/www/test2.iandreev.com/favicon.ico" failed (2: No such file or directory), client: 240.14.17.24, server: test2.iandreev.com, request: "GET /favicon.ico HTTP/1.1", host: "test2.iandreev.com"

You can get a favicon icon from this website or you can generate one from this one.

nginx and PHP

Unlike Apache and the php module, nginx is using PHP-FPM as a proxy to requests PHP dynamic code over FCGI protocol. PHP-FPM comes bundled with PHP, so let’s install php 5.6.

pkg install php56

Add this line in /etc/rc.conf so PHP-FPM starts on boot.

php_fpm_enable="YES"

Create the php.ini file from php.ini-production.

cp /usr/local/etc/php.ini-production /usr/local/etc/php.ini

Now, edit /usr/local/etc/php-fpm.conf and make these changes. Replace listen = 127.0.0.1:9000 with listen = /var/run/php-fpm.sock. In addition, uncomment these three lines.

listen.owner = www
listen.group = www
listen.mode = 0660

Finally, start the php-fpm service.

service php-fpm start
Performing sanity check on php-fpm configuration:
[28-Jan-2017 23:41:31] NOTICE: configuration file /usr/local/etc/php-fpm.conf test is successful

Let’s create a test website that will serve PHP content.

cd /usr/local/www
mkdir test3.iandreev.com
echo "" > test3.iandreev.com/index.php
chown -R www:www test3.iandreev.com

Add a new virtual host in nginx.

server {
        access_log /var/log/nginx/test3.iandreev.access.log;
        error_log  /var/log/nginx/test3.iandreev.error.log;
        listen       80;
        server_name  test3.iandreev.com;
        root /usr/local/www/test3.iandreev.com;
        index index.php index.html index.htm;
        location / {
            try_files $uri $uri/ =404;
        }
        error_page      500 502 503 504  /50x.html;
        location = /50x.html {
            root /usr/local/www/nginx-dist;
        }
        location ~ \.php$ {
                try_files $uri =404;
                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                fastcgi_pass unix:/var/run/php-fpm.sock;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $request_filename;
                include fastcgi_params;
        }
    }

and reload the config changes.

service nginx reload

Go to your new virtual host, in my case http://test3.iandreev.com and you should see this.

nginx and WordPress

Now that we have PHP support we can create a WordPress site. We’ll use MySQL as a DB backend. Please see my other post on how to install MySQL first.
Once you have the MySQL installed, create the database. Replace the values of wordpress, wpuser and secret to suit your needs.

CREATE DATABASE wordpress;
GRANT ALL PRIVILEGES ON wordpress.* TO "wpuser"@"localhost" IDENTIFIED BY "secret";
FLUSH PRIVILEGES;
QUIT;

Install the following php modules, otherwise when you start WordPress you’ll be presented with a blank page.

pkg install php56-extensions php56-mysql php56-mysqli php56-xml php56-hash php56-gd php56-curl php56-tokenizer php56-zlib php56-zip

Download the latest WordPress install. If you don’t have wget, install it first with pkg install wget.

cd /usr/local/www
wget www.wordpress.org/latest.zip --no-check-certificate
unzip latest.zip
mv wordpress test4.iandreev.com
chown -R www:www test4.iandreev.com
rm latest.zip

Add the following in nginx.conf to create the virtual host for the WordPress site.

server {
        access_log /var/log/nginx/test4.iandreev.access.log;
        error_log  /var/log/nginx/test4.iandreev.error.log;
        listen       80;
        server_name  test4.iandreev.com;
        root /usr/local/www/test4.iandreev.com;
        index index.php index.html index.htm;
        location / {
            try_files $uri $uri/ =404;
        }
        error_page      500 502 503 504  /50x.html;
        location = /50x.html {
            root /usr/local/www/nginx-dist;
        }
        location ~ \.php$ {
                try_files $uri =404;
                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                fastcgi_pass unix:/var/run/php-fpm.sock;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $request_filename;
                include fastcgi_params;
        }
    }

Reload the config and restart php-fpm service.

service nginx reload
service php-fpm restart

Go to your website and you’ll be presented with the WordPress install page.

freebsd – Blog of Kliment Andreev – A place so I won't forget things

FreeBSD: OpenVPN server in a LAN (local network) environment

OpenVPN server

OpenVPN client

OpenVPN server

Ansible: Quick Start Guide for FreeBSD, CentOS and Ubuntu

Control and managed nodes

CentOS 8

Ubuntu

FreeBSD

SSH keys

Inventory and the config file

Playbooks

Service handlers

Variables

Roles

Error handling

Tags

Ansible Vault

Prompts

Useful options

FreeBSD, pfSense: Site-to-site VPN IPsec tunnel between FreeBSD and pfSense using strongswan

Prerequisites

FreeBSD configuration

pfSense

Site-to-site VPN tunnel

CentOS, FreeBSD: Secure SSH with fail2ban and blacklistd with MFA (Google Authenticator)

CentOS

Log with SSH keys (optional)

Google Authenticator

fail2ban

FreeBSD

Log with SSH keys (optional)

Google Authenticator

blacklistd

FreeBSD, CentOS: Install Jenkins behind a nginx reverse proxy using SSL certificates

CentOS

nginx as reverse proxy

FreeBSD

nginx as reverse proxy

FreeBSD: Install Prometheus, Node Exporter and Grafana

FreeBSD: Install an authoritative DNS server (BIND)

Installation, startup and control

Zone file

DNSSEC (optional)

Windows, CentOS, FreeBSD: ssh, RDP – Bypass your corporate firewall

CentOS 7

FreeBSD 11

Windows 2016

Tunnel over putty

CentOS, FreeBSD, Windows: rsyslog server and client

RHEL 7 rsyslog servers

AWS Linux, RHEL 6, CentOS 6

FreeBSD

Windows

FreeBSD: Install nginx – virtual hosts, PHP, MySQL and WordPress

Installation

400 Bad Request

Single host

Virtual hosts

nginx and PHP

nginx and WordPress