As the title says, I’ll create two web servers in a private subnet, put an Application Load Balancer in front and protect the content with Web Application Firewall (WAF). In addition, I’ll create a bastion host so I can access the web servers using SSH. Web Servers are not reachable from the Internet.

Elastic IP

First thing to do is to create an Elastic IP for the NAT gateway. NAT gateway is needed if you want to give the Web Servers Internet access so they can be patched. Go to Elastic IPs under Network & Security in EC2 menu and click on Allocate Elastic IP address in the top right corner. Accept the defaults and click Allocate.

VPC

We’ll create a VPC with two public and two private subnets. Minimum two public subnets are needed for the Application Load Balancer. You can’t create an ALB with one public subnet.
Click on Launch VPC Wizard in the VPC menu and select VPC with Public and Private Subnets. I’ll use this CIDR (192.168.100.0/22) which gives me 1024 hosts enough for 4 Class C /24 subnets, you can use whatever CIDR you want. As you can see I’ll use 192.168.100.0/24 for the first public subnet and 192.168.102.0/24 for the first private subnet. I’ll create the other public and private subnet later. You’ll have to associate the Elastic IP that you created in the previous step to the NAT gateway. Click Create VPC in the bottom right corner.

Go to Subnets menu click on the public subnet and look at the Availability Zone.

When you create the next subnet, you have to put it on another zone.
Click on the Route table tab. Remember/write-down the route table id (rtb-0e3cbsomething). It will be different for you.

Then click on Create subnet in the upper right corner. Choose your VPC, name the subnet (in my case subPublic2), assign an Availability Zone (e.g. us-east-1a) and add the CIDR 192.168.101.0/24. Make sure that your 2nd public Availability Zone is different than the 1st Availability Zone. Click Create subnet.
If you click on the new subnet now and then click on the Route table tab, you’ll see that AWS created a route table for you. But as you can notice below, the destination 0.0.0.0/0 goes through the NAT gateway.

We don’t want that. We want the public subnet to use the Internet gateway, not the NAT gateway. So click on the Edit route table association on the right. From the drop down choose the other route table, that’s the Internet gateway, that’s the one that you have wrote down earlier. When you change to the other route table, you’ll see that the destination for 0.0.0.0/0 changes to igw which is the Internet gateway. Click Save.

Now, do the same for the other private subnet. Click on the private subnet and look at the Availability Zone. In my case it’s us-east-1b. This means, I have to create my 2nd private subnet in another zone.

So, I am creating the 2nd private subnet within my VPC and I named my subnet subPvt2 in us-east-1a and 192.168.103.0/24 CIDR.

Check the route table for this new private subnet. The route table for 0.0.0.0/0 should be the NAT gateway which is correct, but just in case…
So these are my subnets now.

Bastion host

You use a bastion host as a jump server to get to the web servers. If you have any other means to get to them such as VPN or Systems Manager, then go for it. This is just an example of how to get to the web servers.
I’ll use the AWS Linux for this. It really doesn’t matter as long you have a SSH client installed on the bastion host. Launch a simple t2.micro instance in any public subnet. Make sure the bastion host has a public IP. Also make sure that this bastion host is accessible from a single IP or some dedicated subnet. You don’t want this bastion host to be exposed to the world. In my case, the security group for the bastion host is restricted to my home IP only on port 22 for SSH.

Web Servers

I’ll use two AWS Linux instances deployed in the private subnets. Do not assign a public IP for these. That’s what the bastion host is for. Create a security group that allows traffic for SSH from the security group that you created for the bastion host. In my case it looks like this. Start typing sg in the circled field and choose the security group that you create earlier. If you named it properly, you’ll see it. If not, go get the security group ID for the bastion host.

Launch two instances in two different private subnets.

Test connection

Log to the bastion host using the key you created. Now, we’ll connect to the web servers from the bastion host. If you used the same key for bastion host and web servers, copy and paste the PEM certificate in a file called key.pem or whatever. If you used different keys, it doesn’t matter, just make sure you have the keys as a file on the bastion host. I used the same key, which is not the best practice, but for the sake of simplicity, I have a file called key.pem on the bastion host. Log to each web server using ssh ec2-user@192.168.102.176 -i key.pem where ec2-user is the user for the AWS Linux (it differs for different Linux distros), 192.168.102.176 is the IP of the first web server and the parameter -i tells which key file certificate to use. Make sure you do chmod 600 key.pem so you can log in. You should be able to log in to both web servers.

Install Apache web server

Log to each web server and execute these commands to install Apache web server, start it and enable to start on boot.

sudo yum -y install httpd
sudo systemctl start httpd
sudo systemctl enable httpd

Switch to root.

sudo su

Then on each server do this. Do Server 2 on the second server. This is just an example. You need the same code on both web servers. The difference is because we’ll test the ALB later and see how the page changes.

echo "Server 1" > /var/www/html/index.html

For the 2nd server.

echo "Server 2" > /var/www/html/index.html

Application Load Balancer

Now, we are moving to the ALB. Before we create the ALB, let’s create a security group that will allow port 80 from everywhere. Looks like this. Create the security group from the EC2 menu. This is a security group that we’ll assign to the load balancer.

From the EC2 menu on the left side, go to Load Balancing and click on Load Balancers. Click on Create Load Balancer. Choose Application Load Balancer. Make sure you choose the public subnets and for the security group use the one that we created earlier for the load balancer.

If you scroll down, you’ll see that you need to create a target group for the ALB. The target is our web servers. Click on Create target group. It will open up a new browser window. Choose Instances, name the target group, leave HTTP1 for protocol and for the Health checks type /index.html. Remember, we created these files earlier.


Click Next, select the web servers and click on Include as pending below.

Finally, click on Create target group.
Go back to the original tab when you created the load balancer, click on the refresh button and choose the target group you just created.

Scroll down and click on Create load balancer.
Wait for a couple minutes until you see that the state changes to Active. Click Refresh on the right side.

While you are waiting, if you click on the Description tab for the load balancer and scroll a little bit down, you’ll see the security group that we create for the load balancer.

We need to add this security group to the web servers. Right now, the web servers are configured to accept traffic from the bastion host only, but we need for the web servers to accept HTTP traffic as well. Now, the security group for the load balancer allows HTTP from everywhere, but that doesn’t matter. When we add this security group to the web servers they’ll also allow HTTP request from everywhere. But the web servers have no public access so it really doesn’t matter. Does this mean that any other server in these public/private subnets will have access to HTTP to the web servers? Yes. If I create another server in the private/public subnet, it will have access to HTTP to the web servers.
Let’s add the ALB security group to the instances. In my case it looks like this where the bottom security group is the security group for allowing HTTP from everywhere, the one that we assigned to the ALB.

Before you do anything, go to the Target Groups menu under Load Balancing. You should see both instances as healthy.

If this looks good, go back to the load balancer and get the URL.

Open up a new browser window and go to that URL. Click refresh in your browser several times. You’ll see how it changes from Server 1 to Server 2.


If everything is OK, proceed further.

Web Application Firewall

The good news is that you can directly create your rules from the ALB. Click on the load balancer you created and click on the Integrated services tab. Click on Create Web ACL.

Name the ACL and click on Add AWS resources where we gonna add the ALB.

Here we are. Click Next after.

And now you are on your own. You have to add rules. There are some that are templated and there are some that you can buy from the AWS Marketplace.

Once you add these rules, you can see what’s blocked and what’s not under Cloud Watch | All metrics | WAFV2.

In this post I’ll explain how to run a WordPress site using Azure services for managed MySQL and App Services for a PHP application.
The basic offerings for MySQL and App Services create these resources in a public subnet, so in order to integrate them with your VNETs, you’ll have to use the more expensive options. In this post, we’ll use just that. We’ll create the MySQL database without any public access and integrate the App Services to connect to the SQL database internally.

VNET/subnets

First, we’ll create a VNET with three subnets. When you integrate an App Services in a VNET, the subnet where the App Services will reside must be empty. The same applies for Flex MySQL servers. You can have multiple Apps in the same dedicated subnet. So, you need a dedicated subnet for both App Service and MySQL Flex, but not for MySQL single server. Single MySQL server can share the subnet with other resources.
For MySQL, there are two options. We can use a private endpoint or a VNET integration. We’ll go with a private endpoint first. We don’t need a dedicated subnet for this.
In my case, I’ll create a VNET with 192.168.100.0/22 CIDR and three subnets, 192.168.100.0/24 for some Infra/MySQL stuff, 192.168.101.0/24 for the App Services and 192.168.102.0/24 for Flex MySQL. I’ll use a new resource group called rgWordPress. The VNET will be called vnetWordPress and the subnets, subInfra, subApp and subMySQL respectively.

VM

I will also create a VM in the subInfra subnet. This VM is not needed at all, it will be used to test connectivity. In my case, I’ll spin a new Debian 1CPU, 1GB VM and I’ll install the MySQL client.

sudo apt install default-mysql-client

Managed Single server MySQL

When it comes to MySQL, there are two options: Flexible server and Single server. Look at this link to determine what you want to use. In most of the cases, you’ll use a Flexible server. But there are some limitations in both use cases. For example, a Flexible server can’t use private endpoints. You are limited to VNET integration and Single server can’t be VNET integrated. On top of that, only General Purpose and Memory Optimized servers can have private endpoints. The basic single server can be public only. Flex VNET integrated servers require dedicated and empty subnets. There are other limitations as well, so take a look at the link above.
In my case, I am using a General Purpose server.

Once you create the server, it will be public. As you can see from the Overview for the server, there is a public DNS record.

On the left side/blade, under Security, click on Private endpoint connections and create one.
Name the private link and put it in the same resource group as the other resources.

For the resource type Microsoft.DBforMySQL/servers and you’ll be able to select the newly created MySQL server from the dropdown under Resource.

Finally, choose the virtual network and subnet where you want to put the private link. Do not use the subnet for App Services. In addition, make sure that the integration with the private DNS zone is set to Yes.

If you log to the server that we just built and try to resolve the public DNS for the MySQL server, you’ll see that it resolves as a private IP. That’s because you have a private DNZ zone that takes precedence over the public DNS.

In case, you are still getting the public IP, go to the Private DNS zone and make sure that it’s linked to the actual VNET.

Now you can try to connect to the MySQL database.

mysql -u klimenta@mysqlwordpress-single -h mysqlwordpress-single.mysql.database.azure.com -p

Where -u specifies the username and the -h is the host. These parameters are under Overview on the right side. See the screenshot above.
You might get the following error. ERROR 9002 (28000): SSL connection is required. Please specify SSL options and retry.
In that case, download the SSL certificate and try again.

wget https://www.digicert.com/CACerts/BaltimoreCyberTrustRoot.crt.pem
mysql -u klimenta@mysqlwordpress-single -h mysqlwordpress-single.mysql.database.azure.com -p --ssl-ca=BaltimoreCyberTrustRoot.crt.pem

In case you are wondering, wait, why this cert? – The answer is here.
As you can see, I am able to connect now.

It is also recommended to disable the public access. Go to Connection security under Settings and select Yes for Deny public network access.

App Services

Go to App Services and create one. Make sure you use Premium offering so you can do a VNET integration. The initial parameters are self explaining. Use the defaults for the rest of the screens. You won’t be able to do much with PHP anyway.

Once the app is provisioned, on the left side/blade, go to the Networking menu under Settings. You’ll see that the VNET integration is not configured. Click on the link.

Pretty much, you have to select the VNET and an empty subnet. As you can see, I can only select subApp from the dropdown.

Close the blade and from the App menu, click on SSH under Development Tools on the left. This will open a new tab in your browser with a SSH session.

We have to create the WordPress database now. You can do that from the test server that we used initially or from the actual SSH session. Let’s use the SSH session. It already comes with mysql client preinstalled.

wget https://www.digicert.com/CACerts/BaltimoreCyberTrustRoot.crt.pem
mysql -u klimenta@mysqlwordpress-single -h mysqlwordpress-single.mysql.database.azure.com -p --ssl-ca=BaltimoreCyberTrustRoot.crt.pem

Once in MySQL client, execute these SQL statements.

CREATE DATABASE dbwordpress;
GRANT ALL PRIVILEGES ON dbwordpress.* TO "wordpressuser"@"%" IDENTIFIED BY "SuperSecret";
FLUSH PRIVILEGES;
EXIT

Replace the values with yours for the database name, MySQL host name, username and password.
Now, you can download WordPress and extract it.

cd /home/site/wwwroot
wget -c http://WordPress.org/latest.tar.gz
tar -xzvf latest.tar.gz
mv WordPress/* /home/site/wwwroot/
rm -rf WordPress
rm -rf latest.tar.gz

Once completed, go to the URL for the app. It should be under the Overview. You have to provide the database name (dbwordpress), username (wordpressuser@mysqlwordpress-single), password (SuperSecret) and the database host (mysqlwordpress-single.mysql.database.azure.com).
Most likely you’ll see an error. The easiest way is to disable the SSL requirement for the MySQL database from Connection security under Settings for the database.

If this doesn’t work for you, you’ll have to manually specify the parameters in wp-config.php. You can search this on your own.
Once you are done, specify the values in WordPress install screen and you are all set.

Flex MySQL server

Let’s see how Flex MySQL server works. Go to Azure MySQL but this time provision a flex server.
This is how my setup looks like.

…and this.

A flexible MySQL server can’t have a private endpoint, but it can be VNET integrated, same as the Azure App Services. And if you choose for the flex server to be VNET integrated, you’ll need a dedicated and empty subnet, same as Azure App Services.

Once you provision the flex MySQL server, you’ll be able to manage it the same as a single server with a private endpoint. You can also see the entry in the private DNS zone for a flex server, same as with the single server. A flex server also allows you to have a hot standby HA pair but you can’t use this hot standby as a replica. A HA will double the price and you won’t be able to create replicas. The HA is in a different zone but in a same region and it allows forced and automatic failovers. And if you want to use a read only replica (up to 10), you have to take care of the SQL proxying using ProxySQL, HAProxy or Nginx.
So, if you want some sort of HA for the MySQL, go with the flexible MySQL. You’ll have automatic failover within the region. But if you want to recover from a regional failure, use a single MySQL server with a read replica in another region (flexible MySQL can’t have replica in another region) and when you need to failover, just promote the replica in a standalone server (you’ll also need an Azure App Service in the DR region). The bad thing is that once you promote a replica in a standalone server and the original regions comes back up, you can’t revert the DR MySQL server in a replica again.

In this post/howto, I’ll explain how to install Ansible as control and managed node on FreeBSD 12, CentOS 8 and Ubuntu 18. Then, I’ll explain how to create SSH keys so the nodes can communicate and some basic tasks. Then, I’ll show you how to create a playbook to install the latest updates and also install an Apache server with the default settings. Finally, I’ll show an example of how to use variables.

Control and managed nodes

A control node is where you execute all of your Ansible commands and eventually keep your playbooks, configs, inventory etc. It’s pretty much your workstation. A managed node is where the actual playbooks are executed. These are the main concepts and the terminology.
In my case, I have 4 VMs/instances. The main one which is the control node and 3 managed nodes.

CentOS 8

If you want to install Ansible on the control node, you have to install Python 3.x first.

sudo dnf install python3

This will also install pip. Type python3 to test, CTRL-D to exit and then install ansible.

sudo pip3 install ansible

Type ansible to test.

Ubuntu

Ubuntu comes with python installed, but not with pip. Install pip with:

sudo apt install python3-pip

Then install ansible with:

sudo pip3 install ansible

Type ansible to test.

FreeBSD

FreeBSD comes with python installed but pip is not. Check the version and then install the same pip version.

ls -l /usr/local/bin/python*

If your output is for example python37, install the same pip version.

pkg install py37-pip

Then install ansible.

pip install ansible

Type ansible to test.

SSH keys

While Ansible can use standard *nix username/password authentication, it’s recommended that you use SSH keys to communicate from control node to the managed nodes. For that, you’ll have to create your SSH keys. Let’s say you have an account on your control node and the username is admin. You also want to use the user ansible on the managed nodes. It really doesn’t matter what usernames you are going to choose. You can always override the keys to use, but in this case, I’ll create a key on the control node and send it to all managed nodes.
On the control node, regardless of your OS, do:

ssh-keygen -b 4096

This will create a subfolder .ssh with two files: id_rsa and id_rsa.pub. The former is your private key and the later is your public key.
Copy the key to your managed nodes.

ssh-copy-id ansible@nodeX.andreev.local

This will copy my key for the user admin to the node X under the ansible user. Then test the passwordless connection.

ssh ansible@nodeX.andreev.local

Mind that the use of FQDN (nodex.andreev.local) vs. hostname (nodex) is important. For SSH these two are different. Once you log to the managed node, the node will be added to the list of known hosts in the file .ssh/known_hosts.

Inventory and the config file

Ansible uses the inventory files to execute an action against using the options and parameters specified in the config file. The config file is /etc/ansible/ansible.cfg for CentOS and Ubuntu and /usr/local/etc/ansible/ansible.cfg for FreeBSD. You can also put the inventory file in the same directory and name it as you wish, but you have to specify the inventory as a parameter on the command line or an entry in the config file. In addition, you can have your config file in your current directory or under the .ansible directory in your home folder. ansible.cfg in the current directory has a precedence over .ansible.cfg in the home directory which has a precedence over /etc/ansible/ansible.cfg. Here is how that looks.
Let’s list all the inventory.

ansible --list-hosts all

You’ll get a message that there is no inventory file. Let’s create one, we’ll name it inventory.txt.

[freebsd]
node1.andreev.local
[centos]
node2.andreev.local
[ubuntu]
node3.andreev.local
[bsd]
node1.andreev.local
[linux]
node2.andreev.local
node3.andreev.local

If we specify the inventory file, we’ll get this.

ansible --list-hosts all -i inventory.txt
  hosts (3):
    node1.andreev.local
    node2.andreev.local
    node3.andreev.local

If we create a config file, we can tell ansible where to look for the inventory. Create a file ansible.cfg in the same directory.

[defaults]
inventory=/home/<somewhere>/inventory.txt

If you do ansible –list-hosts all now, you’ll get the same result as before, but without specifying the inventory file.
Or something like this.

ansible all -m shell -a "uname -a"
node1.andreev.local | CHANGED | rc=0 >>
FreeBSD node1.andreev.local 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64
node2.andreev.local | CHANGED | rc=0 >>
Linux node2.andreev.local 4.18.0-193.14.2.el8_2.x86_64 #1 SMP Sun Jul 26 03:54:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
node3.andreev.local | CHANGED | rc=0 >>
Linux node3.andreev.local 4.15.0-129-generic #132-Ubuntu SMP Thu Dec 10 14:02:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

By default, ansible runs on the managed node with the currently logged user that executes the playbook on the control node. If you get an error saying that the previous command cannot connect to the host, you have to specify the same user that you used when you test the connection with ssh -user-@managednode. So, edit ansible.cfg and add this line.

remote_user=<user>

We can target only the group linux which consists of Linux hosts only in the inventory file.

ansible linux -m shell -a "date"
node3.andreev.local | CHANGED | rc=0 >>
Fri Jan  8 15:23:40 UTC 2021
node2.andreev.local | CHANGED | rc=0 >>
Fri Jan  8 10:23:40 EST 2021

Playbooks

The playbooks are the blueprints of the automation tasks. Instead of running the ansible command to execute each task separately, we combine these tasks in a YAML file and execute them sequentially. Here are 3 playbooks that update each of our managed nodes. There is no update module for FreeBSD, so we use the shell command.

# freebsd-update.yml
---
  - hosts: freebsd
    become: yes
    tasks:
      - name: Fetch all packages
        shell: freebsd-update fetch
      - name: Install FreeBSD updates
        shell: freebsd-update install
      - name: Reboot
        reboot:

For CentOS we’ll use yum.

# centos-update.yml
---
  - hosts: centos
    become: yes
    tasks:
      - name: Update all packages
        yum: name=* state=latest
      - name: Reboot
        reboot:

…and for Ubuntu we’ll use apt.

# ubuntu-update.yml
---
  - hosts: ubuntu
    become: yes
    tasks:
      - name: Update all packages
        apt: name=* state=latest
      - name: Reboot
        reboot:

Save these files with a YAML extension and you can execute them with the following command.

ansible-playbook <filename>

All of them will probably fail. That’s because your ansible user on the managed nodes is required a password when executing a sudo command. In order to fix that, you’ll have to add a line in the sudoers file. Edit this file using the visudo command.

visudo

…and then add this line right before the @includedir <...> which is the last line in the file.

ansible ALL=(ALL) NOPASSWD:ALL

Where ansible is the user that runs the playbooks on the managed nodes. FreeBSD doesn’t come with sudo preinstalled, so you’ll have to install it first on the managed node.

pkg install sudo

These playbooks will update the OS and the packages for the Linux. For FreeBSD, it will update only the OS. Here is another example of playbooks that will install Apache server in a default configuration and change the ServerName and ServerAdmin lines. We’ll also install PHP and test our server.
If you have a firewall enabled, make sure you open it up first on CentOS. Ubuntu and FreeBSD do not come with the firewall enabled.

firewall-cmd --zone=public --permanent --add-service=http
firewall-cmd --reload

For FreeBSD, the playbook looks like this.

# freebsd-apache.yml
---
  - hosts: freebsd
    become: yes
    tasks:
      - name: Install apache and php
        pkgng:
          name:
            - apache24
            - php74
            - mod_php74
          state:  present
      - name: Start on reboot
        service: name=apache24 enabled=yes
      - name: Copy index.php
        copy:
          src: ../files/index.php
          dest: /usr/local/www/apache24/data
          mode: 0755
      - name: Copy mod_php.conf
        copy:
          src: ../files/mod_php.conf
          dest: /usr/local/etc/apache24/modules.d
          mode: 0755
      - name: Start apache now
        service: name=apache24 state=started

For CentOS, it looks like this.

# centos-apache.yml
---
  - hosts: centos
    become: yes
    tasks:
      - name: Install apache and php
        yum:
          name:
            - httpd
            - php
          state:  present
      - name: Start apache now and on reboot
        service: name=httpd state=started enabled=yes
      - name: Copy index.php
        copy:
          src: ../files/index.php
          dest: /var/www/html
          mode: 0755

…and for Ubuntu it looks like this.

# ubuntu-apache.yml
---
  - hosts: ubuntu
    become: yes
    tasks:
      - name: Install apache and php
        apt:
          name:
            - apache2
            - php
          state:  present
      - name: Start apache now and on reboot
        service: name=apache2 state=started enabled=yes
      - name: Copy index.php
        copy:
          src: ../files/index.php
          dest: /var/www/html
          mode: 0755

You will also need these two files in a directory called files. In my case it’s one level above the directory where I keep my playbooks.
index.php

<?php
  phpinfo();
?>

001_mod-php.conf

<IfModule dir_module>
    DirectoryIndex index.php index.html
    <FilesMatch "\.php$">
        SetHandler application/x-httpd-php
    </FilesMatch>
    <FilesMatch "\.phps$">
        SetHandler application/x-httpd-php-source
    </FilesMatch>
</IfModule>

The index.php file is the standard test file to test the PHP distributions and the 001_mod-php.conf is needed for FreeBSD only. As you can see the playbooks differ quite a bit for these three OSes. Once you deploy the playbooks, you can test the result by going to http://[nodeX]/index.php.
Looks like this.

Service handlers

Service handlers are used only when a change is made on the managed node. For example, we can restart a service only if the config file was changed. If the file is not changed, then there is no need to restart. Here is an example of a service handler. We’ll restart postfix service only if the main.cf file was changed.

# service-handler.yml
---
- hosts: centos
  become: yes
  tasks:
  - name: Configure main.cf
    lineinfile:
      path: /etc/postfix/main.cf
      regexp: ^#mydomain
      line: 'mydomain = example.com'
    notify: restart postfix

  handlers:
  - name: restart postfix
    service: name=postfix state=restarted

Make sure that the name for the handler is the same (lines 11 and 14), so Ansible knows what service handlers is referred.

Variables

When ansible runs a playbook on a managed node, the first task is to gather info about the managed node. The info is a bunch of settings that we can use in our playbooks. For example, if you execute the following command, you can see the IP address, the CPU model, python version etc.

ansible -m setup <node>

We can use these settings and use them as variables if we need them. For example, this playbook displays the hostname and the IP.

# showip.yml
---
  - hosts: freebsd
    become: yes
    tasks:
      - name: Show the IP address
        debug:
          msg: "The hostname is {{ inventory_hostname}}  and the IP is {{ansible_default_ipv4.address }}"

And if you run the playbook, you’ll see something like this.

TASK [Show the IP address] *********************************************************************************************
ok: [node1.andreev.local] => {
    "msg": "The hostname is node1.andreev.local  and the IP is 192.168.1.211"
}

Here is another example of using variables. In this case, we’ll specify a file and change the ownership and the mode.

# owner.yml
---
  - hosts: centos
    become: yes
    vars:
      filename: "/var/www/html/index.php"
    tasks:
      - name: Change the owner of the file
        file:
          path: "{{ filename }}"
          owner: apache
          group: apache
          mode: '0755'

In case we want to assign an output to a variable, we’ll have to use the keyword register. Here is an example of how to get the output from a command and print it on the screen with the keyword debug.

# variables.yml
---
  - hosts: freebsd

    tasks:
    - name: Get the uptime manually
      command: uptime
      register: var_uptime

    - name: Print the uptime
      debug:
        msg: The uptime is "{{ var_uptime }}"

Roles

Roles let you automatically load related vars_files, tasks, handlers, and other Ansible artifacts based on a known file structure. Once you group your content in roles, you can easily reuse them and share them with other users. The idea is to separate the tasks, handlers and vars in different files. Let’s see this playbook for example. It changes a line in main.cf file, restarts postfix and copies a file under the postfix main directory.

# roles.yml
---
  - hosts: centos
    become: yes

    vars:
      filevd: "/etc/postfix/virtual_domains"
      cfgpostfix: "/etc/postfix/main.cf"


    tasks:
    - name: Configure main.cf
      lineinfile:
        path: "{{ cfgpostfix }}"
        regexp: ^#mydomain
        line: 'mydomain = example.com'
      notify: restart postfix
    - name: Copy virtual_domains
      copy:
        src: ../files/virtual_domains
        dest: "{{ filevd }}"
        mode: 0755

    handlers:
    - name: restart postfix
      service: name=postfix state=restarted

We can rewrite this file by separating the tasks, variables, files and handlers. Run this command.

ansible-galaxy role init postfix

If you look at the file/directory structure of the newly created directory postfix, it looks like this.

tree postfix
postfix
├── defaults
│   └── main.yml
├── files
├── handlers
│   └── main.yml
├── meta
│   └── main.yml
├── README.md
├── tasks
│   └── main.yml
├── templates
├── tests
│   ├── inventory
│   └── test.yml
└── vars
    └── main.yml

Create a file virtual_domains under the files directory.

echo "mydomain.com" > postfix/files/virtual_domains

Move the config for vars, handlers and tasks in the separate main.yml files. For example, this is how my files look like.
postfix/vars/main.yml

---
# vars file for postfix

  filevd: "/etc/postfix/virtual_domains"
  cfgpostfix: "/etc/postfix/main.cf"

As you can see the keyword vars: does not exists. Ansible knows that this file is for vars so there is no need to enter the vars keyboard.
postfix/tasks/main.yml

---
# tasks file for postfix

  - name: Configure main.cf
    lineinfile:
      path: "{{ cfgpostfix }}"
      regexp: ^#mydomain
      line: 'mydomain = example.com'
    notify: restart postfix
  - name: Copy virtual_domains
    copy:
      src: ../files/virtual_domains
      dest: "{{ filevd }}"
      mode: 0755

postfix/handlers/main.yml

---
# handlers file for postfix

  - name: restart postfix
    service: name=postfix state=restarted

Finally, create a file called something.yml that will be your main file. This file has to be outside the postfix directory structure.
In my case it looks like this.
something.yml

# something.yml
---
  - hosts: centos
    become: yes
    roles:
      - postfix

Now, if you execute this playbook, ansible will automatically execute the rest of the dependant playbooks as well.

ansible-playbook something.yml

Error handling

Sometimes we want certain changes to be ignored. Sometimes, we know the behavior of certain commands and we know that they might return non-zero code and we want that ignored. For example, consider this part of a playbook.

- hosts: centos
  tasks: 
  - name: Type something that will fail
    command: thiscommanddoesntexist
    ignore_errors: yes

  - name: Run command remotely
    command: /usr/local/bin/somecommand
    register: cmd_result
    changed_when: cmd_result == 2

Ansible would report a task as changed as long as the command (or) script gives zero return code.
In the first part, we know that the task will fail, but we decide to ignore it using the keyword ignore_errors. No matter what the command returns, ignore_errors: yes will never report to ansible that the command failed.
In the second command we can ignore the error based on the output of the command. For example, if the output is 2, the the error will be ignored. If cmd_result is not equal to 2, the task will be marked as changed.
So whenever this condition is true, the task will be marked as changed.

Tags

Tags are used when you have a playbook with several tasks and you need to run only specific parts of it instead of running the entire playbook. You use tags to execute or skip selected tasks. Let’s say we have this playbook that installs Docker on Centos and has multiple tasks. As you can notice in lines 12, 24, 37 and 45 we have a new line with a keyword tags that we use to tag certain tasks. The purpose of this is to include or exclude these tasks from the playbook.

# centos-docker.yml
---
- name: Install docker
  hosts: centos
  become: true

  tasks:
    - name: Install yum utils
      yum:
        name: yum-utils
        state: latest
      tags: install

    - name: Install device-mapper-persistent-data
      yum:
        name: device-mapper-persistent-data
        state: latest
      tags: install

    - name: Install lvm2
      yum:
        name: lvm2
        state: latest
      tags: install

    - name: Add Docker repo
      get_url:
        url: https://download.docker.com/linux/centos/docker-ce.repo
        dest: /etc/yum.repos.d/docer-ce.repo
      become: yes

    - name: Install Docker
      package:
        name: docker-ce
        state: latest
      become: yes
      tags: install

    - name: Start Docker service
      service:
        name: docker
        state: started
        enabled: yes
      become: yes
      tags: start

Now, with the command below, we can execute the playbook and only the tasks tagged with install will be executed.

ansible-playbook centos-docker.yaml --tags install

We can also tell ansible to NOT run those tasks tagged with install.

ansible-playbook centos-docker.yaml --skip-tags install

You can add multiple tags per task, e.g.

tags:
  - cleanup_app
  - cleanup_web

Ansible reserves two tag names for special behavior: always and never. If you assign the always tag to a task or play, Ansible will always run that task or play, unless you specifically skip it (–skip-tags always). If you assign the never tag to a task or play, Ansible will skip that task or play unless you specifically request it (–tags never).

Ansible Vault

Ansible Vault encrypts variables and files so you can protect sensitive content such as passwords or keys rather than leaving it visible as plaintext in playbooks or roles.
First, you have to create a vaulted file where we’ll store the passwords. When you run this command it will ask you to create a password and then an empty file will show up.

ansible-vault create secrets.yml

Add some passwords there and save the file.

mysql_pwd: "DifficultPassword"
ht_pwd: "PasswordXYZ"

If you look at the file now, you’ll see that it’s encrypted and you can’t see the passwords anymore.
If you want to edit the file do ansible-vault edit secrets.yaml and enter the vault password.
Create a small playbook that displays the password.

# centos-vault.yml
---
- hosts: centos
  vars_files:
    - secrets.yml

  tasks:
  - name: Show mysql pwd
    debug:
      msg: "{{ mysql_pwd }}"

If you run the playbook now, ansible will throw an error saying ERROR! Attempting to decrypt but no vault secrets found. You have to specify the parameter –ask-vault-pass and enter the vault password when prompted.

Prompts

In case you need to pause the playbook execution and ask the user for some input such as confirmation or password, use prompts.
If you want the output to echo, use private: no, otherwise what you type won’t show up on the screen. Here is an example of a playbook that asks you to confirm if a file needs to be copied to the node. If you type yes and hit enter, the file will be copied, otherwise it won’t.

# centos-prompt.yml
---
- hosts: centos
  become: yes

  vars_prompt:
    name: upload
    private: no
    prompt: "Do you want to upload xyz.txt?"

  tasks:
  - name: Upload xyz.txt
    copy:
      src: xyz.txt
      dest: /var/log
    when: upload == "yes"

Useful options

ansible-playbook <name> --syntax-check # checks the syntax of the playbook
ansible-playbook <name> --check # does a dry run, reports the will-be changes, but the playbook is not executed
ansible-playbook <name> --step # ask to confirm each-step

In this short article I’ll describe briefly how to stream/broadcast from your phone’s camera to the world. In order to do that, you’ll need the Larix Broadcaster app installed on your phone, you will need a server with a public IP and a Haivision Media Gateway installed and for the end users, they’ll need standard VLC Player install.
The Larix Broadcaster app and the VLC player are free, but Haivision Media Gateway is not. In addition, if you have your own server and want to install it as a VM, you’ll have to contact the vendor directly. I’ll use a much easier option and host it in AWS where it comes as a template from the AWS Marketplace.

Haivision Media Gateway in AWS

In AWS, launch a new instance from the AWS Marketplace and search for SRT. You’ll see several options(SRT Gateway vs. Media Gateway and BYOL as PAYG license). I’ll choose Pay-as-You-Go Media Gateway.

As you can see, the software is not that cheap and you pay as much as you used. So if you are not streaming, just shut down the instance. Another important thing is the size of the instance, how much horse power you need. Is it 2 CPUs, 4 CPUs, 8? Well, nobody knows your needs, but it also depends on what type of stream are you sending, is it going to be a low-res or 1080p, how many users will consume that stream, etc. I strongly suggest that you go over this document. It describes how to size your environment and it gives you practical guidance on how to configure the Media Gateway. It’ not a necessary read to continue with this post, but it’s strongly recommended once you have everything up and running.
OK, in my case I’ll go with a t3.medium instance which is 2 CPUs and 4GB RAM. Create the instance as you normally do, make sure you have a public IP and the ports that are recommended in the security group given are open (22, 80 and 443). Once the instance boots up log with the default username haiadmin and the password is your instance id which is i-0123something.

I won’t go into details what each option does and what each keyword means, I’ll just guide you through creating a route for you to be able to broadcast. So, from the screen that you see, click on the button to add a route.

These are my parameters. For the Route Name, enter a description and make sure the checkmark is checked for Start Route. Further down, enter a description for the Source Name, in my case it’s Larix Cam or whatever you want to put here. The Protocol is TS Over SRT and the port is 50000. The Mode is Listener and the rest are the defaults. As I said earlier, if you want a better throughput, read the SRT deployment guide doc that I’ve sent earlier. It will tell you how to tweak these parameters. So, this tells the Media Gateway to listen on port UDP 50000 for any incoming broadcast.
Now, click on the Destination button.

The Destination is where the broadcast will be distributed. So, you got the point now. A camera on your phone sends the video to our instance in AWS and this instance (Media Gateway) distributes the feed to numerous viewers. You can’t stream from your phone directly, because your phone’s bandwidth and horse power are minuscule, so you need a gateway.
Here is how the Destination looks like.

Name the destination however you like, make sure the Protocol is TS over SRT and the Port is 50001. The Media Gateway passes the signal from UDP 50000 (source) to UDP 50001 (destination). Can you use some other ports? Yes, you can as long as they are not the same and greater than 1024 and less than 65535. Change the Type to Listener and leave the rest of the parameters as-is. Again, read the SRT Deployment Guide. Click Add and once you go back, click Create to create the route.
SUPER IMPORTANT: Make sure you edit the security group for this instance and open the ports for the source and destination. In my case, you have to add UDP ports 50000 and 50001 to the rules of the security group.

Larix Broadcaster app

On your phone download this app from Google Play or App Store. Go to Settings (the gear icon), Connections and click on the (+) sign to add a connection.
Enter a name for your connection, and for the URL enter the public IP of your instance and the port of the listener, so it looks like srt://public_IP:port. Make sure you specify srt:// as prefix. The defaults are 640×480 with audio, but you can change that in the settings for the app. Now, that you have a connection, make sure it’s active by selecting it and after that just hit the Record button. If something is not right, you’ll see it right away on your camera. It will tell you that the app can’t connect to the server. And if everything is OK after about 8-10 seconds you’ll see a green dot next to your source in the Media Gateway.

VLC PLayer

There is really nothing much to say here. Go to this website, download and install VLC and from the menu click on Media | Open Network Stream. Enter the IP address of the AWS instance and specify the destination port like srt://public_IP:port. Make sure you specify the destination port, not the source port and the protocol should be srt.

Stream a video file (optional)

In case you want your source to be a video file instead of a camera stream, reconfigure the source or add another route with new source and destination of which the source is RTMP. Configure the source so it looks like this.

From another server send the video stream to the Media Gateway using this command (you need ffmpeg installed).

ffmpeg -re -i video-file.mp4 -c copy -f flv rtmp://media_gateway_ip/live/stream

In VLC, use the srt://media_gateway_ip:port to watch the stream.

I have written a lot about pfSense and different types of VPN scenarios (AWS, Azure), but never created a post about a site-to-site VPN tunnel with CentOS running strongswan and pfSense. The scenario described here works with CentOS, but it will work with any other Linux of BSD distribution. The syntax for strongswan is the same, it’s just the configuration file names and locations that are different across distributions.

Prerequisites

Both the pfSense box and CentOS need to have public IPs. While it’s possible to have them behind NAT, this scenario only covers configurations with public IPs. You should disable the firewalld on CentOS (initially). Both of them need two network interfaces. In my case this is how it looks like.

It’s also easier if you create a small spreadsheet so you don’t loose track of the IPs and subnets.

CentOS configuration

I am using CentOS 7. First, we have to install strongswan and disable the firewall temporarily.

yum -y install epel-release
yum -y install strongswan
systemctl enable strongswan
systemctl stop firewalld
systemctl disable firewalld

Once installed, change these system variables so you allow IP forwarding and prevent ICMP redirects.

echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
echo "net.ipv4.conf.all.accept_redirects=0" >> /etc/sysctl.conf
echo "net.ipv4.conf.all.send_redirects=0" >> /etc/sysctl.conf
sysctl -p /etc/sysctl.conf

strongswan on CentOS places its config under /etc/strongswan directory. We’ll use a config with pre-shared key because it’s easier to implement. So, edit the /etc/strongswan/ipsec.conf file, remove everything there and paste this config. Of course, this assumes that you don’t have any working configs there. Now, look at the highlighted lines. That’s where you have to replace the IPs and the subnets with your values. E.g. line 13 is the private IP of the CentOS server, 14 is the subnet, 15 is the ID of the server (IP or hostname), 17 is the external IP of the pfSense server, 18 is its subnet and line 19 is the ID of the pfSense server (IP or hostname). This is where the small spreadsheet above comes handy.

config setup

conn %default
        ikelifetime=60m
        keylife=20m
        rekeymargin=3m
        keyingtries=1
        authby=secret
        keyexchange=ikev2
        mobike=no

conn net-net
        left=148.27.56.195
        leftsubnet=10.1.96.0/20
        leftid=148.27.56.195
        leftfirewall=no
        right=23.183.125.249
        rightsubnet=192.168.1.1/24
        rightid=23.183.125.249
        ike=aes256-sha256-modp2048!
        esp=aes256-sha256!
        pfs=no
        auto=start
        lifetime=1h

Then, edit the /etc/strongswan/ipsec.secret file and specify the pre-shared key. This is a password that the servers will use to authenticate. Make sure it’s something more complex in production. In my case, the password is “secret“.

	
# ipsec.secrets - strongSwan IPsec secrets file
148.27.56.195 23.183.125.249 : PSK "secret"

pfSense

Now that the CentOS strongswan box is configured, we can configure pfSense. In pfSense, go to VPN | IPSec from the menu and click on Add P1 button. Fill out the General Information section, so it looks like this.

As you can see, we use IKEv2 and our remote gateway is 148.27.56.195 which is the CentOS strongswan box.
For Phase 1 Proposal (Authentication and Algorithms), fill out the section so it looks like this.

From the screenshot above, you can see that the values correspond to what we have in ipsec.conf file.
For the Advanced Options section, use the defaults.

Click Save and then from the main IPSec | Tunnels menu, click on Add P2 button.
For Phase 2, fill out these values for the General Information section.

Change the values for Local Network and Remote Network to suit your needs. These are the local subnets behind pfSense and strongswan.
For Phase 2 Proposal (SA/Key Exchange) section, choose these values.

For the Advanced Configuration section, you can leave it as is, or put the private IP of the CentOS box so the IPSec protocol sends keep-alive pings. It’s not mandatory, but if your tunnel fails frequently, you can configure this field.

Click Save and the VPN config is done. But, we have to tell pfSense to allow IPSec traffic. So, from the menu go to Firewall | Rules and click on IPSec submenu. Click Add and fill out the form so it looks like this.

…and this.

Click Save and go back to the CentOS box.

Site-to-site VPN tunnel

At this point you can start the VPN.

systemctl start strongswan

If everything is OK, the log will look like this.

tail -f /var/log/messages
Mar  9 16:15:00 titan charon: 09[NET] sending packet: from 148.27.56.195[500] to 23.183.125.249[500] (256 bytes)
Mar  9 16:15:00 titan charon: 10[NET] received packet: from 23.183.125.249[500] to 148.27.56.195[500] (224 bytes)
Mar  9 16:15:00 titan charon: 10[ENC] parsed IKE_AUTH response 1 [ IDr AUTH N(ESP_TFC_PAD_N) SA TSi TSr ]
Mar  9 16:15:00 titan charon: 10[IKE] authentication of '23.183.125.249' with pre-shared key successful
Mar  9 16:15:00 titan charon: 10[IKE] IKE_SA net-net[1] established between 148.27.56.195[148.27.56.195]...23.183.125.249[23.183.125.249]
Mar  9 16:15:00 titan charon: 10[IKE] scheduling reauthentication in 3350s
Mar  9 16:15:00 titan charon: 10[IKE] maximum IKE_SA lifetime 3530s
Mar  9 16:15:00 titan charon: 10[IKE] received ESP_TFC_PADDING_NOT_SUPPORTED, not using ESPv3 TFC padding
Mar  9 16:15:00 titan charon: 10[CFG] selected proposal: ESP:AES_CBC_256/HMAC_SHA2_256_128/NO_EXT_SEQ
Mar  9 16:15:00 titan charon: 10[IKE] CHILD_SA net-net{1} established with SPIs c82fdbae_i ce57b856_o and TS 10.1.96.0/20 === 192.168.1.0/24

Check the status.

strongswan status
Security Associations (1 up, 0 connecting):
     net-net[3]: ESTABLISHED 3 minutes ago, 148.27.56.195[148.27.56.195]...23.183.125.249[23.183.125.249]
     net-net{3}:  INSTALLED, TUNNEL, reqid 2, ESP SPIs: c818e2d7_i c34c4c3d_o
     net-net{3}:   10.1.96.0/20 === 192.168.1.0/24

If the current CentOS box is not your Internet gateway for the servers behind, you have to add a static route to the pfSense subnet. So, on each server behind CentOS, do something like this.

route add 192.168.1.0 mask 255.255.255.0 10.1.96.3 -p

Check the documentation for other OSes. You have to tell the OS to use 10.1.96.3 (internal strongswan IP) as a gateway to 192.168.1.0 subnet. Because my pfSense box is also my gateway, I don’t have to make these changes.

Firewall

Now, everything is up and running, but we have the firewall disabled and the CentOS box is exposed to Internet. Not good. In order to fix that, we have to make some changes and allow the IPSec protocol to flow through the firewall.

systemctl enable firewalld
systemctl start firewalld
firewall-cmd --set-default-zone=dmz
firewall-cmd --zone=dmz --permanent --add-rich-rule='rule protocol value="esp" accept' 
firewall-cmd --zone=dmz --permanent --add-rich-rule='rule protocol value="ah" accept' 
firewall-cmd --zone=dmz --permanent --add-port=500/udp 
firewall-cmd --zone=dmz --permanent --add-port=4500/udp 
firewall-cmd --permanent --add-service="ipsec"
firewall-cmd --zone=dmz --permanent --add-masquerade
firewall-cmd --permanent --direct --add-rule ipv4 nat POSTROUTING 0 -m policy --pol ipsec --dir out -j ACCEPT
firewall-cmd --reload

…and you are all set.

NFS (Network File System) is a distributed file system protocol originally developed by Sun Microsystems in 1984, allowing a user on a client computer to access files over a computer network much like local storage is accessed. NFS, like many other protocols, builds on the Open Network Computing Remote Procedure Call (ONC RPC) system. The NFS is an open standard defined in Request for Comments (RFC), allowing anyone to implement the protocol. That’s what Wikipedia says.
NFS is used as a file system when two or more instances need access to the same file system. E.g. SQL clusters, VM clusters, Docker Swarm, etc…
In this post, I’ll describe a very quick install on a server and show you how to mount that on another CentOS box.

Server Install

On the server (in my case 192.168.1.91) I’ll install the NFS server, start it, make sure it starts at boot and then I’ll allow the firewall to accept the NFS clients. If you have SELinux enabled, it’s automatically configured.

yum -y install nfs-utils
systemctl start nfs-server
systemctl enable nfs-server
firewall-cmd --permanent --zone=public --add-service nfs
firewall-cmd --reload

Let’s create an empty directory, configure the permissions and create the NFS share.

mkdir /nfs
chown nfsnobody:nfsnobody /nfs
chmod 755 /nfs

We have to tell the NFS server about the share. Execute this line that will add the correct line in /etc/exports. That’s where the NFS server expects its shares defined.

echo "/nfs    *(rw,sync,no_root_squash,no_subtree_check)" >> /etc/exports

The * means that all IPs can have access to this share. Do man 5 exports to see all available options. We have to tell the NFS server to export the share now.

exportfs -a

Check the exported share.

exportfs -s
/nfs  *(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)

Client install

On the client, we’ll have to install the NFS client. It’s the same package.

yum -y install nfs-utils

There is no need to start any service or change any firewall settings.
Let’s mount the share. I’ll mount it under /mnt/nfs which is a new directory on the client.

mkdir -p /mnt/nfs
mount 192.168.1.91:/nfs /mnt/nfs

If you create a file on the share now and go back to the server you’ll see that the file is there.

echo "NFS test" > /mnt/nfs/nfs.txt

In order to mount this share anytime you boot the client, you’ll have to add it to /etc/fstab so it mounts on boot.

echo "192.168.1.91:/nfs       /mnt/nfs    nfs     defaults 0 0" >> /etc/fstab

This post is a slight modification of the official wiki for setting up Samba as an AD Domain Member.

In my lab, I built a Windows 2012R2 domain controller/DNS and a CentOS 7.1 VM running Samba 4.2.10.
My Windows domain is kdomain.local and the NETBIOS name is MYDOMAIN. They are on the same subnet and the IP of the CentOS VM is in the DNS. Let’s verify the prerequisites before we install Samba. Make sure that everything checks out.

Check the hostname first.

hostname
centos7dd.kdomain.local

If you need to change the hostname, use hostnamectl set-hostname your_hostname
Make sure your resolver points to a valid domain controller.

cat /etc/resolv.conf
search kdomain.local
nameserver 192.168.1.19

Check your IP.

ip addr show | grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
    inet 192.168.1.6/24 brd 192.168.1.255 scope global eno16777736

Check the resolving. dc03 is my domain controller, centos7dd is my CentOS member server, the one that I am configuring.

host -t A dc03.kdomain.local
dc03.kdomain.local has address 192.168.1.19
host -t A centos7dd.kdomain.local
centos7dd.kdomain.local has address 192.168.1.6

If you get an error that host is an unknown command, install bind-utils with yum install bind-utils. You don’t have to, but make sure that all IPs and resolving works fine.
Some more checks, to see if you don’t have something messed up in /etc/hosts file.

getent hosts centos7dd
192.168.1.6     centos7dd.kdomain.local

Now it’s time to install samba.

yum install samba samba-devel samba-winbind samba-winbind-clients

Before you go further, you need to know what’s realm, NETBIOS and workgroup in Samba terms. They are kind of misleading as you can see below from the samba config file. On the domain controller, open up a command prompt and type:

set | find  "DOMAIN"
USERDNSDOMAIN=KDOMAIN.LOCAL
USERDOMAIN=MYDOMAIN
USERDOMAIN_ROAMINGPROFILE=MYDOMAIN

My domain is KDOMAIN.LOCAL and the NETBIOS domain name is MYDOMAIN. The hostname of my domain controller is DC03, but that’s irrelevant.
Find where the config file is stored.

smbd -b | grep CONFIGFILE
   CONFIGFILE: /etc/samba/smb.conf

Rename the original file as something else and edit smb.conf so it’s an empty file. Add the following lines there.

[global]
        netbios name = centos7dd
        realm = KDOMAIN.LOCAL
        workgroup = MYDOMAIN
        security = ADS
        winbind enum groups = Yes
        winbind enum users = Yes
        winbind nss info = rfc2307
        idmap config *:range = 2000-9999
        idmap config * : backend = tdb

Now, you can join the VM to the domain.

net ads join -U administrator
Enter administrator's password:
Using short domain name -- MYDOMAIN
Joined 'CENTOS7DD' to dns domain 'kdomain.local'

Let’s make sure that Samba starts now and on each boot.

systemctl start smb.service
systemctl start nmb.service
systemctl start winbind.service
systemctl enable smb.service
systemctl enable nmb.service
systemctl enable winbind.service

Make sure you get the domain users and groups back.

wbinfo -u
MYDOMAIN\administrator
MYDOMAIN\guest
MYDOMAIN\klimenta
MYDOMAIN\krbtgt

wbinfo -g
MYDOMAIN\winrmremotewmiusers__
MYDOMAIN\domain computers
MYDOMAIN\domain controllers
MYDOMAIN\schema admins
MYDOMAIN\enterprise admins
....

Now, edit /etc/nsswitch.conf and modify these three lines. Just add winbind after sss for passwd , shadow and group entries, so it looks like this. Don’t change the lines where it says #passwd: db files nisplus nis.

passwd:     files sss winbind
shadow:     files sss winbind
group:      files sss winbind

Now, do getent passwd and getent group. You should recieve both local and domain users and groups as output.

Finally, if you have a firewall, add an exception for Samba services.

firewall-cmd --permanent --zone=public --add-service=samba
firewall-cmd --reload

At this point, you’ll have the OS fully configured as a domain member, but you still can’t log in to CentOS with a domain account. In order to do that, we’ll have to run authconfig-tui. NOTE: For CentOS 8 use authselect select winbind --force.

authconfig-tui

Make sure these options are set.

…and these. You don’t have to join to the domain, just select OK.

This utility will change a lot of files, especially those in /etc/pam.d directory and it will also change smb.conf. Restart the winbind service in order for these changes to take effect.

systemctl restart winbind.service

Make sure you have these lines in smb.conf.

template shell = /bin/sh
template homedir = /home/%D/%U

Reload Samba config.

systemctl reload smb.service

Because of the template homedir directive above, all home directories for the domain users will be under /home/MYDOMAIN (or whatever domain you have).
Finally, create the home dir and log as the user.

mkdir -p /home/MYDOMAIN/klimenta
cd /home/MYDOMAIN
chown -R klimenta@kdomain.local klimenta

Now, you can log as the domain user.

login as: klimenta@kdomain.local
klimenta@kdomain.local@192.168.1.6's password:
Last login: Wed Sep  7 11:08:10 2016 from 192.168.1.101
-bash-4.2$ pwd
/home/MYDOMAIN/klimenta

What if we want to share our home or some other folder on the CentOS server? We have to add a config change in smb.conf file. I’ll explain how to share the home folder. If you want to share some other folder, make sure that the domain user has rights to that folder. In my case, I’ll have the domain user klimenta access the shared home folder for the domain user klimenta on the CentOS server. The permissions are already in place. Edit /etc/samba/smb.conf file and add these lines at the end.

[share]
        comment = klimenta domain user share
        path =  /home/MYDOMAIN/klimenta
        valid users = klimenta@kdomain.local
        guest ok = no
        writable = yes
        browsable = yes

Restart both services

systemctl restart smb.service
systemctl restart nmb.service

Now, from a Windows workstation go to \\ and you’ll see the folder opened with all the files in there.

Recently I was playing with the new Elastic File System (EFS) that AWS provides. It’s an NFS v4 implementation and my goal was to use it as a shared repository between two different systems. The access is based on security groups and by default when you create a new file system, the mount targets are created in all availability zones for that region.

Create NFS security group

NFS access is based on security groups. So, if you have an instance that needs access to an NFS target, you have to make sure that the instance belongs to the same security group that NFS target belongs. We’ll create a security group that grants access to TCP port 2049. This is the port that NFS uses.
Go to the main AWS menu, click on VPC, then Security Groups and click on Create Security Group.

Once created, click on the Inbound Rules tab and then click Edit.
From the first drop down box on the left choose NFS (2049), for the source choose the same Group ID as the group itself and click Save.

Create EFS

From the main AWS menu, click on Elastic File System.

Then, click on Create file system. This is how my default looks like.

At this point, you can change the availability zone for the mount target. If you plan to launch instances in different zones, then go with the defaults. Remove the default security group on the right and add the security group that we just created for all of the zones, so the right part looks like this.

Click on Next Step and fill out the optional settings for the tags and the performance.

Click on Next Step again, review and click Create File System.
This is what you will have once you create your file system.

If you click on DNS names, you’ll see your server names (for the mount target).

And that’s all what we need. If you click on EC2 mount instruction, you’ll get a quick guide on how to mount your new file system.

FreeBSD

For FreeBSD, we’ll have to put this one line in /etc/rc.conf.

nfs_client_enable="YES"

and then start the nfs client with

service nfsclient start

In order to mount the NFS target, we’ll need to know what zone our instance belongs to. As you can remember from the above, each zone has a different DNS server name for the target. So, if you need your zone you can see that from the EC2 menu or if you have curl installed, you can browse your instance metadata.

curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone/

Install curl with pkg install curl if you don’t have it.
We also need a mount point. In my case I’ll mount my new file system under /root/efs. I don’t have the directory created, so I’ll do.

mkdir /root/efs

The easiest way to mount the target without looking at the zone where the instance belongs is to run this one liner.

mount -t nfs -o nfsv4 `curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone`.fs-0b864a42.efs.us-east-1.amazonaws.com:/ /root/efs

This means mount a new file system of type NFS, make sure it’s NFSv4 (-o), get the zone with curl and append it to the DNS server name of the NFS server and mount it under /root/efs.
If you want to unmount the filesystem, do

umount /root/efs

For some reason, under FreeBSD I was getting RPC time out errors, so if you want to remove the mount instantly, force it.

umount -f /root/efs

Now, if you reboot your server the mount points are gone, so you’ll have to make sure that on each reboot your NFS mounts on. Edit /etc/fstab and add this line.

us-east-1d.fs-0b864a42.efs.us-east-1.amazonaws.com:/ /root/efs nfs rw,nfsv4 0 0

What if you have an image in a load balancing scenario and you have it configured to boot up in a random zone. The above /etc/fstab entry won’t work and you can’t query that with curl in fstab. It’s a text file, not an executable script. The solution is to put this little script in your image AMI under User Data. Any time an image launches, this will be executed once on the first boot.

#!/bin/sh
echo "`curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone`.fs-0b864a42.efs.us-east-1.amazonaws.com:/ /root/efs nfs rw,nfsv4 0 0" >> /etc/fstab
mount -a

RHEL/CentOS

Red Hat and CentOS don’t come up with NFS client installed, so you’ll have to install it first.

yum install nfs-utils

In order to mount the NFS target for Red Hat and CentOS servers, we’ll need to know what zone our instance belongs to. As you can remember from the above, each zone has a different DNS server name for the target. So, if you need your zone you can see that from the EC2 menu or with curl you can browse your instance metadata.

curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone/

We also need a mount point. In my case I’ll mount my new file system under /root/efs. I don’t have the directory created, so I’ll do.

mkdir /root/efs

The easiest way to mount the target without looking at the zone where the instance belongs is to run this one liner.

mount -t nfs4 -o nfsvers=4.1 $(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone).fs-0b864a42.efs.us-east-1.amazonaws.com:/ /root/efs

This means mount a new file system of type NFS, make sure it’s NFSv4 (-o), get the zone with curl and append it to the DNS server name of the NFS server and mount it under /root/efs.
If you want to unmount the filesystem, do

umount /root/efs

Now, if you reboot your server, the mount points are gone, so you’ll have to make sure that on each reboot, your NFS mounts on. Edit /etc/fstab and add this line.

us-east-1d.fs-0b864a42.efs.us-east-1.amazonaws.com:/ /root/efs nfs defaults,vers=4.1 0 0

What if you have an image in a load balancing scenario and you have it configured to boot up in a random zone. The above /etc/fstab entry won’t work and you can’t query that with curl in fstab. It’s a text file, not an executable script. The solution is to put this little script in your image AMI under User Data. Anytime an image launches, this will be executed once on the first boot.

#!/bin/bash
echo "`$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone).fs-0b864a42.efs.us-east-1.amazonaws.com:/ /root/efs nfs defaults,vers=4.1 0 0" >> /etc/fstab
mount -a

AWS Linux

See above for Red Hat and CentOS and just skip the part to install the nfs client. AWS Linux comes up with these utils preinstalled.

Recently, when I tried to ssh to my AWS Linux AMI, I got an error that the key is not valid. First, I thought that somehow my private key was stolen, but upon further investigation in the logs, I found these errors.

/ containts a file system with errors, check forced

The AMI will try to fsck automatically. Most of the time, this will fix your problems. In my case it failed and I got a useless AMI that won’t boot. If fsck doesn’t work, check the console. I saw that one of the checks failed.

According to AWS and this troubleshooting AWS Troubleshooting, first stop the instance.

Go to Volumes, Detach volume and wait until it says available.

Create a new instance, same AMI type, same resources. Right click the newly created AMI, choose to Attach volume and select the instance where this volume will be attached.

Once attached, in the new AMI do fdisk -l to see the device name for your attached disk. In my case I see /dev/xvdf.

Try to repair it from here. In my case I got the following errors.

fsck.ext4 -v /dev/xvdf
e2fsck 1.42.12 (29-Aug-2014)
ext2fs_open2: Bad magic number in super-block
fsck.ext4: Superblock invalid, trying backup blocks...
fsck.ext4: Bad magic number in super-block while trying to open /dev/xvdf

The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

We have to find the superblocks that are corrupt. Now, let’s see where my superblocks are kept.

mke2fs -n /dev/xvdf
mke2fs 1.42.12 (29-Aug-2014)
Found a gpt partition table in /dev/xvdf
Proceed anyway? (y,n) y
Creating filesystem with 2097152 4k blocks and 524288 inodes
Filesystem UUID: 350b4d59-28c8-422f-a3af-ede9daae43e3
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

NOTE: You’ll get different numbers. Use yours not mine.

Next step is to try to restore the superblocks.

e2fsck -b 1605632 /dev/xvdf
e2fsck 1.42.12 (29-Aug-2014)
e2fsck: Invalid argument while trying to open /dev/xvdf

The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

I tried all superblock numbers and all failed. Finally, I tried with the last resort.

mke2fs -S /dev/xvdf
mke2fs 1.42.12 (29-Aug-2014)
Found a gpt partition table in /dev/xvdf
Proceed anyway? (y,n) y
Creating filesystem with 2097152 4k blocks and 524288 inodes
Filesystem UUID: 97fc1ad1-b8dc-44ec-ba46-33061541cd60
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done
Writing superblocks and filesystem accounting information: done

Force the checks on the filesystem.

e2fsck -f /dev/xvdf -y
/dev/xvdf: ***** FILE SYSTEM WAS MODIFIED *****
/dev/xvdf: 9001/524288 files (5.0% non-contiguous), 114693/2097152 blocks

At the end, mount the volume and see if you see your data. In my case, I got nothing.

ls -l
total 148
drwx------ 516 root root 147456 May 13 23:23 lost+found

VMware Tools is a suite of utilities that enhances the performance of the virtual machines guest operating system. Usually, it is a very simple install, especially on Windows OS. The port for FreeBSD is also easy, but there are some quirks that needs to be resolved. First, the instructions for the install assume that you have the cdrom already installed. Second, the tools depend on Perl and since 10.1, Perl was removed from the default install and /usr/bin. So, in order to install VMware Tools on FreeBSD 10.1 or 10.2, we have to do some work. For FreeBSD 11.x and higher, scroll all the way down.

FreeBSD 10.x

First, install the compatibility libraries.

pkg install compat6x-amd64

If you have a 32-bit FreeBSD, use compat6x-i386 instead.
Then, install perl from the packages.

pkg install perl5-5.20.3_8

If there is a problem with this version, check what’s available and install the latest.

pkg search ^perl

Check the name of the cdrom drive.

dmesg | grep cd

In my case it was cd0. Most likely it’s cd0, unless you specified some SCSI cd when creating a custom VM.
Right click the VM and choose Install VMware Tools.

Create a mount point, mount the tools, copy the tools on the disk and extract them.

mkdir /tmp/vt
mount -t cd9660 /dev/cd0 /tmp/vt
cd /tmp/vt
cp vmware-freebsd-tools.tar.gz ../
cd ..
umount /tmp/vt
rm -Rf /tmp/vt
tar xzvf vmware-freebsd-tools.tar.gz
rm vmware-freebsd-tools.tar.gz
cd vmware-tools-distrib

If you try to run the tools with ./vmware-install.pl, you’ll get an error “command not found”. That’s because the provided script is looking for perl under /usr/bin. As of FreeBSD 10.1, perl is no longer a system binary and after the install, the binary is under /usr/local/bin.
So, edit the following files and replace /usr/bin/perl with /usr/local/bin/perl in the following three files. It’s the first line in all three files.

vmware-install.pl
vmware-install.real.pl (if you have VMware Workstation 12)
bin/vmware-config-tools.pl
bin/vmware-uninstall-tools.pl

Once replaced, you can start the installation with:

./vmware-install.pl

Look at the choices, but most likely you’ll accept the defaults and then reboot.

FreeBSD 11.x

If you are using FreeBSD 11.x and higher you can install it with or without X11 support (nox11).

pkg install open-vm-tools
pkg install open-vm-tools-nox11

Once installed, add these lines in /etc/rc.conf.

vmware_guest_vmblock_enable="YES"
vmware_guest_vmhgfs_enable="NO"
vmware_guest_vmmemctl_enable="YES"
vmware_guest_vmxnet_enable="YES"
vmware_guestd_enable="YES"

Reboot after making changes in /etc/rc.conf.

CentOS

The installation for CentOS is much simpler.

yum install open-vm-tools