Docker – Blog of Kliment Andreev – A place so I won't forget things

Install Kubernetes with CRI-O and Cilium on Rocky Linux 9 / VMware / Bare Metal

Kliment Andreev — Mon, 09 Oct 2023 16:18:16 +0000

In this post, I’ll build a Kubernetes cluster with one master and two nodes using CRI-O and Cilium. I have DNS in my environment but you can use the hosts files if needed.
The OS will be Rocky Linux 9 running on ESXi 8.x but this should work on any other bare metal server too. The config for the nodes is:

– Master: 2 CPUs, 4GB RAM
– Workers: 2 CPUs, 8GB RAM.

First, we’ll have to disable swap and put SELinux in permissive mode.
Do this on all nodes (master + workers).

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

Now, we have to open some firewall ports for Kubernetes..
Do this on the master node only.

sudo firewall-cmd --permanent --add-port=6443/tcp
sudo firewall-cmd --permanent --add-port=2379-2380/tcp
sudo firewall-cmd --permanent --add-port=10250/tcp
sudo firewall-cmd --permanent --add-port=10259/tcp
sudo firewall-cmd --permanent --add-port=10257/tcp
sudo firewall-cmd --reload

Do this on the workers only.

sudo firewall-cmd --permanent --add-port=10250/tcp
sudo firewall-cmd --permanent --add-port=30000-32767/tcp   
sudo firewall-cmd --reload

In addition, we’ll have to open the firewall ports for Cilium.
Do this on all nodes.

sudo firewall-cmd --permanent --add-port=4240/tcp 
sudo firewall-cmd --permanent --add-port=8472/udp 
sudo firewall-cmd --reload

Now, we have to make some changes that are also required. Some necessary modules and IPtables change.
Do this on all nodes.

cat <
We have to install CRI-O container runtime.

Do this on all nodes.
VERSION=1.22
sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/CentOS_8/devel:kubic:libcontainers:stable.repo
sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:${VERSION}.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:${VERSION}/CentOS_8/devel:kubic:libcontainers:stable:cri-o:${VERSION}.repo
sudo dnf -y install cri-o cri-tools
sudo systemctl enable --now crio
sudo systemctl status crio

Then, we have to install Kubernetes.

Do this on all nodes.
cat <
On the master node only, initialize the cluster. Replace master.homelab.com with your master IP if you don’t use DNS.
sudo kubeadm init --control-plane-endpoint master.homelab.local:6443

You’ll get an output that says how to join the nodes, something like kubeadm join master… + a token

On the master node, add these lines.
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Join the nodes to the cluster.

Do this on workers only.
sudo kubeadm join master.homelab.local:6443 --token t2yir0.7n49rn6r57msy4j4 \
        --discovery-token-ca-cert-hash sha256:563360fcf60c49be91cdcf6486a4954c579a80c54503127ee0682ab8f86ec840

Check the nodes.

Do this on the master node.
kubectl get nodes
NAME                   STATUS   ROLES           AGE    VERSION
master.homelab.local   Ready    control-plane   116s   v1.28.2
node1.homelab.local    Ready              89s    v1.28.2
node2.homelab.local    Ready              85s    v1.28.2

…then check all the pods.

Do this on the master node.
kubectl get pods --all-namespaces
NAMESPACE     NAME                                           READY   STATUS    RESTARTS   AGE
kube-system   coredns-5dd5756b68-5ljgg                       1/1     Running   0          2m
kube-system   coredns-5dd5756b68-jp8tq                       1/1     Running   0          2m
kube-system   etcd-master.homelab.local                      1/1     Running   0          2m6s
kube-system   kube-apiserver-master.homelab.local            1/1     Running   0          2m6s
kube-system   kube-controller-manager-master.homelab.local   1/1     Running   0          2m6s
kube-system   kube-proxy-2z64r                               1/1     Running   0          2m
kube-system   kube-proxy-mrwjv                               1/1     Running   0          96s
kube-system   kube-proxy-nj5q4                               1/1     Running   0          100s
kube-system   kube-scheduler-master.homelab.local            1/1     Running   0          2m6s

Install Cilium on the master node only.
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
cilium install --version 1.14.2

Check the status.
cilium status --wait

You should see something like this. 
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium             Desired: 3, Ready: 3/3, Available: 3/3
Deployment             cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
Containers:            cilium             Running: 3
                       cilium-operator    Running: 1
Cluster Pods:          2/2 managed by Cilium
Helm chart version:    1.14.2
Image versions         cilium             quay.io/cilium/cilium:v1.14.2@sha256:6263f3a3d5d63b267b538298dbeb5ae87da3efacf09a2c620446c873ba807d35: 3
                       cilium-operator    quay.io/cilium/operator-generic:v1.14.2@sha256:52f70250dea22e506959439a7c4ea31b10fe8375db62f5c27ab746e3a2af866d: 1

Then you can run the connectivity test. Some of the egress test will fail because we don’t have a public IP on the cluster.
cilium connectivity test

You should be able to run your deployments now. If you need to access the pods using URL use something like NodePort or port-forwarding. It’s very cumbersome and inefficient, but you can use a load balancer with MetalLB. Read this post to see how to do that. It’s separated from this post, because it’s a separate topic than just k8s, container runtime and CNI.

CentOS: Install Docker and Kubernetes

Kliment Andreev — Sat, 26 Sep 2020 18:58:12 +0000

I have another post regarding Docker and Kubernetes, but it’s related to CentOS 7. In this post, I’ll explain how to install Docker and Kubernetes on CentOS 8. There is not that much difference in the installation process, but some things changed so it’s easier to write a new post than modify the original one.
I’ll explain how to install Docker and Kubernetes on one master and one node. You can have as many nodes you want. All commands will be executed as root but we’ll create a user called devops that will ultimately manage both Docker and Kubernetes.

Docker

Log as root and execute this command on both master and the node server. The highlighted line is what you type. The rest is the output.

dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
Adding repo from: https://download.docker.com/linux/centos/docker-ce.repo

Then install docker. This will also install Docker CLI and containerd.io. Do this on both servers too.

dnf -y install docker-ce

Let’s make sure that it starts now and on boot.

systemctl enable --now docker

If you check the status, you’ll see that Docker is running.

systemctl status docker | grep active
   Active: active (running) since Sat 2020-09-26 09:28:50 EDT; 50s ago

Let’s make sure it’s running and it’s functioning. Do this on one server.

docker run hello-world

You’ll see a bunch of output, some greetings and Hello from Docker! message. You don’t need this container and image so you can delete it.

docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS                          PORTS               NAMES
bb436fac0476        hello-world         "/hello"            About a minute ago   Exited (0) About a minute ago                       quirky_chatelet

Look at the end. In my case, the container name is quirky_chatelet. In your case it’s something different. Remove the container.

docker rm quirky_chatelet
quirky_chatelet

Remove the image.

docker rmi hello-world
Untagged: hello-world:latest
Untagged: hello-world@sha256:4cf9c47f86df71d48364001ede3a4fcd85ae80ce02ebad74156906caff5378bc
Deleted: sha256:bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b
Deleted: sha256:9c27e219663c25e0f28493790cc0b88bc973ba3b1686355f221c38a36978ac63

We also have to make some changes how Docker uses the storage and cgroups. By default, it’s not using systemd.

docker info | grep Storage
 Storage Driver: overlay2
docker info | grep Cgroup
 Cgroup Driver: cgroupfs

Overlay2 is fine but cgroupfs is not. Make these changes on both servers. Just copy and paste the whole snippet and it will create a file for you.

cat < /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF

Restart Docker daemon on both servers and check the changes.

systemctl restart docker
docker info | grep -i cgroup
 Cgroup Driver: systemd

Finally, we need a user that will manage Docker. We don’t want to use root for that. If you don’t have a user created, do it with adduser command. In addition, we need to add this user to be a member of the group docker that was created when we installed docker. In my case the user is devops. Do this on both servers.

usermod -aG docker devops

If you log as the devops user and execute docker ps -a, you’ll see that Docker accepted your command. Otherwise, you’ll see some access denied error.

Kubernetes

In order to run Kubernetes (k8s) we need to make some changes to the OS. There are several prerequisites.
First, we’ll have to enable bridge networking. Do this on both servers and make sure you are logged as root.

modprobe br_netfilter
echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables
echo '1' > /proc/sys/net/bridge/bridge-nf-call-ip6tables

Then we have to disable SELinux security.

setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

We also need to disable swap disk.

swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

Now it’s time to install Kubernetes. Copy & paste this snippet that will add the Kubernetes repo to CentOS. Do this on both servers logged as root.

cat < /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

Install Kubernetes on both.

dnf install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

(Optional) Install this handy tool that can help with Kubernetes commands auto-complete. For example, you type kubectl then g hit tab and it will give you a choice of commands that start with g.

dnf install -y bash-completion

Before we finish the installation, we’ll have to make some changes to the firewall. If you don’t have the firewall enabled, you can skip this section.

Firewall

On the master node only, open these ports.

firewall-cmd --add-port=6443/tcp --permanent
firewall-cmd --add-port=2379-2380/tcp --permanent
firewall-cmd --add-port=10250-10252/tcp --permanent
firewall-cmd --reload

On the worker nodes only, open these ports.

firewall-cmd --add-port=10250/tcp --permanent
firewall-cmd --add-port=30000-32767/tcp --permanent
firewall-cmd --reload

Network

Now, it’s time to install the networking component. Kubernetes has several of them, such as flannel, callico, WeaveNet etc… We’ll go with flannel.
Do this on the master node only. There is no need to install the network component on the worker nodes. Kubernetes will take care of that.
Initialize flannel with this subnet on the master node only. This subnet will be where pods will be running. It takes a couple of minutes.

kubeadm init --pod-network-cidr=10.244.0.0/16

At the end you’ll see an output that tells how to join the worker nodes to the master node. In my case it was something like this. Copy this code and the token somewhere safe. You’ll need it anytime you need to join a worker node. Don’t execute this yet.

kubeadm join 192.168.1.115:6443 --token dgwe20.8pw4vd8ks2savzbf \
    --discovery-token-ca-cert-hash sha256:6948547d7ff058a56aba374f44a583459f87e63aacaa53179e7adfeb5cf935a6

Let’s start Kubernetes and make sure it starts on boot. Do this on the master and the worker node logged as root.

systemctl enable --now kubelet

On the master node, log as your standard user that you made to run Docker, in my case it was devops.
Run these commands, one by one.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
echo 'source <(kubectl completion bash)' >>~/.bashrc

Now, we have to install flannel. Do this on the master node only logged as the regular user.

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Check the nodes.

kubectl get nodes
NAME                    STATUS   ROLES    AGE   VERSION
jupiter.andreev.local   Ready    master   12m   v1.19.2

As you can see, we have only the master node. Go to the worker node and as root run this command. The command that you saved somewhere.

kubeadm join 192.168.1.115:6443 --token dgwe20.8pw4vd8ks2savzbf \
    --discovery-token-ca-cert-hash sha256:6948547d7ff058a56aba374f44a583459f87e63aacaa53179e7adfeb5cf935a6

After a minute or two, if you run the same command (get nodes) again, you’ll see the worker node too.

kubectl get nodes
NAME                     STATUS   ROLES    AGE     VERSION
ganymede.andreev.local   Ready       47s     v1.19.2
jupiter.andreev.local    Ready    master   3m47s   v1.19.2

DevOps: CI/CD using git, Jenkins, Docker, Kubernetes

Kliment Andreev — Sun, 03 Nov 2019 21:34:00 +0000

In this post I’ll explain how to create a Node.js chat program, put the source on Github and then using Jenkins we’ll deploy the code to Docker Hub and from there to Docker containers that are managed by Kubernetes. There are many prerequisites for this.

GitHub

DockerHub

See my other posts on how to install Jenkins and Kubernetes cluster.

The chat program

For this purpose I’ve found a simple Node.js chat program that uses Express framework. The program can be found here, but we’ll modify it a bit. On a Linux workstation (I use CentOS, so some commands might differ), log as your regular user and create some necessary directories, a directory called chitchat and two subdirectories, public and test and two subdirectories for public, called js and css.

mkdir -p chitchat/test
mkdir -p chitchat/public/{js,css} 
cd chitchat

Create the source files for the chat program. First, let’s create the HTML. Click the (+) sign to expand. Just copy and paste everything. The first line (cat) will take care of the file creation.

cat << EOF > index.html


    
        Chit Chat App
        
        
    
    
        
        
            
            
        
        
        
    

EOF

Then, the Node.js code.

cat << EOF > index.js
var express = require('express');
var app = express();
var http = require('http').Server(app);
var io = require('socket.io')(http);
var port = process.env.PORT || 4141;
var path = require('path');

app.get('/', function(req, res){
    res.sendFile(__dirname + '/index.html');
});

app.use(express.static(path.join(__dirname + '/public')));


io.on('connection', function(socket) {

    socket.on('user_join', function(data) {
        this.username = data;
        socket.broadcast.emit('user_join', data);
    });

    socket.on('chat_message', function(data) {
        data.username = this.username;
        socket.broadcast.emit('chat_message', data);
    });

    socket.on('disconnect', function(data) {
        socket.broadcast.emit('user_leave', this.username);
    });
});

http.listen(port, function() {
    console.log('Listening on *:' + port);
});
EOF

Then, the actual code for the app.

cat << EOF > public/js/app.js
const form = document.querySelector("form");
const input = document.querySelector(".input");
const messages = document.querySelector(".messages");
const username = prompt("Please enter a nickname: ", "");
const socket = io();

form.addEventListener("submit", function(event) {
        event.preventDefault();

        addMessage(username + ": " + input.value);

        socket.emit("chat_message", {
                        message: input.value
        });

        input.value = "";
        return false;
}, false);

socket.on("chat_message", function(data) {
        addMessage(data.username + ": " + data.message);
});

socket.on("user_join", function(data) {
        addMessage(data + " just joined the chat!");
});

socket.on("user_leave", function(data) {
        addMessage(data + " has left the chat.");
});

addMessage("You have joined the chat as '" + username  + "'.");
socket.emit("user_join", username);

function addMessage(message) {
        const li = document.createElement("li");
        li.innerHTML = message;
        messages.appendChild(li);
        window.scrollTo(0, document.body.scrollHeight);
}
EOF

And finally the CSS stylesheet.

cat << EOF > public/css/styles.css
body {
    margin: 0;
    font-family: sans-serif;
}

form {
    position: fixed;
    bottom: 0;
    left: 0;
    width: 100%;
    display: flex;
    box-sizing: border-box;
    padding: 0.25rem;
}

form input {
    border: 0;
    padding: 0.5rem;
    width: 100%;
    outline: 0;
    margin-right: 0.5rem;
    border-radius: 0.25rem;
    background: #ccc;
}

form button {
    width: 6rem;
    background-color: #1b8c00;
    color: white;
    border: none;
    padding: 0.5rem;
    cursor: pointer;
    border-radius: 0.25rem;
    text-transform: uppercase;
}

form button:hover {
    background-color: #166d01;
}

.messages {
    margin: 0;
    padding: 0;
    margin-bottom: 3rem;
}

.messages li {
    padding: 0.5rem;
}

.messages li:nth-child(odd) {
    background: #eee;
}
EOF

If you don’t have Node.js installed, then install it. If you already have Node.js, skip this step. While still logged as the regular user, sudo to install Node.js and npm and check the versions. Make sure you are in chitchat directory.

curl -sL https://rpm.nodesource.com/setup_10.x | sudo bash -
sudo yum -y install nodejs
node -v && npm -v

Initiate a new project. Make sure you are in chitchat directory.

npm init

Answer the questions for the initialization. Just hit Enter for every question, we’ll replace this file later. Then install Express and socket.io.

npm install -save express
npm install -save socket.io

Finally, start the program. If you have a firewall enabled, you have to poke a hole.

sudo firewall-cmd --add-port=4141/tcp --zone=public --permanent
sudo firewall-cmd --reload
node index.js

The program should listen on port 4141. Open up a browser and access the Node.js app. (http://workstation_ip:4141).
I’ve opened two windows and logged as UserA and UserB to test the functionality. It looks like this.

The GitHub repository

Now that we have the program ready, let’s upload it on GitHub. Log with your account and create a new repository. In my case, I’ll name it chitchat.
On your workstation, initiate the git repo, add the files, do the first commit and push the files to GitHub. It should look like this. Replace your username and the repo name in line 5.

cd chitchat
git init
git add .
git commit -m "First commit."
git remote add origin https://github.com//
git push -u origin master

Jenkins

Install Node.js plugin, Git and Docker

If you installed Jenkins with the default settings, the Git and Docker plugins are probably installed. Node.js plugin is not installed by default so we’ll have to take care of that. Go to Manage Jenkins from the menu and then Manage Plugins. From the Available tab, find the NodeJS plugin and install it.

Again, go to Manage Jenkins, Global Tool Configuration and click on Add NodeJS.

Enter a name and choose the the latest 10.x version. Make sure that the Install automatically is checked. Also, I named my Node.js installation “node” and I’ll reference it as “node” further in the pipeline script. If you name your installation differently, you’ll have to change that in line 4 in the script below.

Again, go to Manage Jenkins and then Manage Plugins in the center. Click on the Installed tab and look for Docker Pipeline plugin and Git plugin. They should be there and already installed (see the grayed checkmark on the left).

If not, click on the Available tab, search for these plugins and install them similarly to what you did with Node.js plugin. Once installed, SSH to the Jenkins server and install Git first. We’ll install it from the source. This is for CentOS 7 only.

yum -y groupinstall "Development Tools"
yum -y install gettext-devel openssl-devel perl-CPAN perl-devel zlib-devel wget
wget https://github.com/git/git/archive/v2.23.0.tar.gz -O /tmp/git.tgz
cd /tmp
tar xzvf git.tgz 
cd git*
make configure
./configure
make install
cd /tmp
rm -Rf git*

Run git ‐‐version to check if it works properly.
Then, install Docker.

yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum -y install docker-ce
systemctl enable docker
systemctl start docker
usermod jenkins -g docker
systemctl restart jenkins

Now, Jenkins can execute Docker commands.

Jenkins pipeline

Once installed from the main menu, in the upper-left corner, click on New Item. Enter the project name and choose Pipeline.

Scroll down until you find the Pipeline script and paste this code. We just want to test pulling the repo from Github. Make sure you replace your username and repo name in line 10.

pipeline {
    agent any

    tools {nodejs "node"}

    stages {

        stage('Cloning Git') {
            steps {
                git 'https://github.com//'
            }
        }
    }
}

It looks like this.

Click on Save and once back, click on Build Now from the menu.

Once completed, you’ll see that everything is running OK.

Click on Console Output to see the actual logs from git.
Now, that we have everything verified, we want to add another step of testing the application. We’ll have to install the pre-requisites for the Node.js and create a test scenario. On the workstation where your project is, create a test scenario. We’ll use mocha and chai for this.

cat << EOF > test/test.js
var chai = require('chai');
var io = require('socket.io-client')
var app = require('../index');
var client = io('http://localhost:4141');

console.log('Server started...Waiting for client connection...');
it('Client connected...', function(done) {
  client.on('connect', function (data) {
    done();
  });
});
EOF

We also need to define the pre-requisites. Change some of the lines here to suit your needs. This is the file that we are replacing, the one that npm init creates.

cat << EOF > package.json
{
  "name": "chitchat",
  "version": "1.0.0",
  "description": "A simple chat program",
  "main": "index.js",
  "scripts": {
    "start": "node index.js",
    "test": "mocha -R spec test/test.js --exit"
  },
  "author": "Kliment Andreev",
  "license": "ISC",
  "dependencies": {
    "express": "^4.17.1",
    "socket.io": "^2.3.0"
  },
  "devDependencies": {
    "chai": "^4.2.0",
    "mocha": "^5.2.0"
  }
}
EOF

As you can see the first file goes under chitchat/test directory and the package.json file is in the root of chitchat directory. Don’t forget to push these files to GitHub.

git add .
git commit -m "Added package.json and test unit"
git push -u origin master

Now, go back to your project in Jenkins (click Configure on the left) and replace the pipeline script with this one. Make sure you change your name and repo in line 10.

pipeline {
    agent any

    tools {nodejs "node"}

    stages {

        stage('Cloning Git') {
            steps {
                git 'https://github.com//'
            }
        }

        stage('Install dependencies') {
            steps {
                sh 'npm install'
            }
        }

        stage('Test') {
            steps {
                sh 'npm test'
            }
        }
    }
}

Click on Build Now and you’ll see that all stages completed fine. But, we want to go a step further and automate the build process anytime we make a change in the source code. In order to do that we need a webhook between GitHub and our Jenkins server. Make sure your Jenkins server is publicly available, otherwise this integration won’t work.
But before we do that, let’s add the script above as a file under our source code for the chat program. Make sure you change your name and repo in line 11. This file goes in the root of the chitchat directory.

cat << EOF > Jenkinsfile
pipeline {
    agent any

    tools {nodejs "node"}

    stages {

        stage('Cloning Git') {
            steps {
                git 'https://github.com//'
            }
        }

        stage('Install dependencies') {
            steps {
                sh 'npm install'
            }
        }

        stage('Test') {
            steps {
                sh 'npm test'
            }
        }
    }
}
EOF

Add this file to Github.

git add .
git commit -m "Added Jenkinsfile"
git push -u origin master

GitHub integration with Jenkins

Log in to Github and from the upper right corner, click on your avatar and click on Settings. Then on the menu on the left side, all the way to the bottom click on Developer settings. Click on Personal access tokens and click on Generate new token.

Give a name to your token (I’ll name mine Jenkins Token) and click on admin:repo_hook.

Click on Generate token after. You’ll get a hex token, click on the clipboard icon next to it. You’ll need this token for the Jenkins configuration.
Go to Jenkins and click on Manage Jenkins on the left followed by Configure System in the middle. Scroll down to the Git section, click on Add GitHub server, type GitHub for Name and next to Credentials click on Add, then Jenkins.

When this window shows up, change the Kind to Secret text. Paste the token under Secret and enter something for ID and Description. Click Add.

Once back, select the credential that we just created and click on Test connection on the right. You should get a message that the connection is fine. Make sure that Manage hooks is checked. Click on Save at the bottom once everything is OK.

Go back to your project in Jenkins and click Configure on the left. Scroll down where the Pipeline is and change the Definition from Pipeline script to Pipeline script from SCM. SCM means Source Code Management. Change SCM to Git, enter the repo URL under Repositories . If your repo is not public, you’ll have to specify some credentials under Credentials. If your repo is public, you can leave them as none.

Then scroll up a bit and check GitHub hook trigger for GitScm polling and then click Save.

If you go back to GitHub, click on your project and all the way on the right click on Settings.

Then click on Webhooks and you’ll see a green checkmark next to your Jenkins URL.

Go to your workstation and edit the index.html file. Change the line 4 for example so instead of Chit Chat App, replace the blank with a hyphen. e.g. Chit Chat to Chit-Chat. Save it and push to GitHib.

git add .
git commit -m "Minor change in index.html"
git push -u origin master

If you go back to Jenkins, you’ll see that there is a build there triggered automatically. This part is CI (Continous Integration). But, let’s go further and expand our CI with creating a Docker image.

Docker image and Docker Hub

In order to create a Docker image, we need a Dockerfile. I won’t explain the meaning of each line in the Dockerfile, so just copy & paste the following snippet in the chitchat directory.

cat << EOF > Dockerfile
FROM node:10
WORKDIR /usr/src/app
COPY package*.json ./

RUN npm install
COPY . .

EXPOSE 4141
CMD [ "npm", "start" ]
EOF

This file will tell Docker exactly what to do when creating the image. But, we also have to tell Jenkins that we need to build the image, test the image and deploy it to Docker Hub. So, your new Jenkinsfile will look like this. Change the URLs in lines 6 and 7 to suit your needs. Before you deploy the script, go to Jenkins, click on Credentials on the left, click on System below, click on Global credentials (unrestricted) in the middle and then Add Credentials on the left. Fill out the form with your Docker Hub credentials. Whatever you put as ID for these username/password pair is what you have to put in line 8 in the script.

And, now copy & paste the script in the chitchat directory. If everything goes well, you’ll see your image deployed in Docker Hub and tagged as whatever your latest build was and the tag latest too.

cat << 'EOF' > Jenkinsfile
pipeline {

    environment {
        dockerregistry = 'https://registry.hub.docker.com'
        dockerhuburl = "klimenta/chitchat"
        githuburl = "klimenta/chitchat"
        dockerhubcrd = 'dockerhub'
    }

    agent any

    tools {nodejs "node"}

    stages {

        stage('Clone git repo') {
            steps {
                git 'https://github.com/' + githuburl
            }
        }

        stage('Install Node.js dependencies') {
            steps {
                sh 'npm install'
            }
        }

        stage('Test App') {
            steps {
                sh 'npm test'
            }
        }

        stage('Build image') {
          steps{
            script {
              dockerImage = docker.build(dockerhuburl + ":$BUILD_NUMBER")
            }
          }
        }

        stage('Test image') {
            steps {
                sh 'docker run -i ' + dockerhuburl + ':$BUILD_NUMBER npm test'
            }
        }

        stage('Deploy image') {
          steps{
            script {
              docker.withRegistry(dockerregistry, dockerhubcrd ) {
                dockerImage.push("${env.BUILD_NUMBER}")
                dockerImage.push("latest")
              }
            }
          }
        }

        stage('Remove image') {
          steps{
            sh "docker rmi $dockerhuburl:$BUILD_NUMBER"
          }
        }
    }
}
EOF

Of course, you’ll have to tell GitHub that there is a change is a file.

git add .
git commit -m "Added Jenkinsfile for Docker"
git push -u origin master

Deployment to Kubernetes using Jenkins

Now that we are finished with the CI (Continuous Integration), it’s time for the CD (Continuous Deployment). We’ll use the existing Docker image that we put on Docker Hub. First thing first, we’ll have to install a plugin for a Kubernetes integration. Go to Manage Jenkins on the left, then click on Manage plugins in the center and click on Available tab. Install the Kubernetes Continuous Deploy plugin.

Once installed and Jenkins is restarted, click on Credentials on the left, then System just below and then click on Global credentials (unrestricted) in the middle. Click on Add Credentials on the left. When this window pops up, change the Kind to Kubernetes configuration (kubeconfig), enter an ID, in my case it’s k8s. Remember the ID, you’ll reference it in the Jenkinsfile later, type a description and select Enter directly. Before you click OK, we’ll have to go to our Kubernetes cluster and log to it.

Once you are logged to your master node, log as the user that you use to manage the k8s cluster and run this command.

cat .kube/config

You’ll get a long list of text. Paste this back in the Jenkins window under Content and click OK. This is how Jenkins will communicate with your Kubernetes cluster. We also need to tell Kubernetes how we want our app deployed. As you know this is done with a YAML file. So, we have to add that YAML file under the chitchat directory.Change the IP in line 25 to match your master k8s node IP. I am deploying a load balancer that will listen on port 8080 and 2 pods in a deployment.

cat << 'EOF' > k8s.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: chitchat
spec:
  replicas: 2
  template:
    metadata:
      name: chitchat
      labels:
        app: chitchat
    spec:
      containers:
      - image: $dockerhuburl:$BUILD_NUMBER
        name: nodejs
---
apiVersion: v1
kind: Service
metadata:
  name: loadbalancer
spec:
  type: LoadBalancer
  externalIPs:
  - 192.168.1.7
  selector:
    app: chitchat
  ports:
  - port: 8080
    targetPort: 4141
EOF

And we also want to tell Jenkins that we are adding an extra step to deploy it to the cluster, so the final Jenkins file looks like this.

cat << 'EOF' > Jenkinsfile
pipeline {

  environment {
    dockerregistry = 'https://registry.hub.docker.com'
    dockerhuburl = 'klimenta/chitchat'
    githuburl = 'klimenta/chitchat'
    dockerhubcrd = 'dockerhub'
    dockerImage = ''
  }

  agent any

  tools {nodejs "node"}

  stages {

    stage('Clone git repo') {
      steps {
         git 'https://github.com/' + githuburl
      }
    }

    stage('Install Node.js dependencies') {
      steps {
        sh 'npm install'
      }
    }

    stage('Test App') {
        steps {
            sh 'npm test'
        }
    }

    stage('Build image') {
      steps{
        script {
          dockerImage = docker.build(dockerhuburl + ":$BUILD_NUMBER")
        }
      }
    }

    stage('Test image') {
      steps {
        sh 'docker run -i ' + dockerhuburl + ':$BUILD_NUMBER npm test'
      }
    }

    stage('Deploy image') {
      steps{
        script {
          docker.withRegistry(dockerregistry, dockerhubcrd ) {
            dockerImage.push("${env.BUILD_NUMBER}")
            dockerImage.push("latest")
          }
        }
      }
    }

    stage('Remove image') {
      steps{
        sh "docker rmi $dockerhuburl:$BUILD_NUMBER"
      }
    }

    stage('Deploy k8s') {
      steps {
        kubernetesDeploy(
          kubeconfigId: 'k8s',
          configs: 'k8s.yaml',
          enableConfigSubstitution: true
        )
      }
    }
  }
}
EOF

Trigger the change.

git add .
git commit -m "Added Jenkinsfile for kubernetes and yaml"
git push -u origin master

After a minute or so, you’ll see your pods running and if you go to your master node on port 8080 you’ll see the app happily running.

CentOS: Kubernetes, Flannel and Calico in a single master configuration

Kliment Andreev — Tue, 09 Apr 2019 15:46:01 +0000

Kubernetes (k8s) is getting a lot of attention and it’s becoming more and more popular even in enterprises that are quite IT conservative. In this post I’ll explain how to install Kubernetes on a single master and one node on CentOS. Then, we’ll install the networking plugins (CNI), Flannel or Calico. At the end I’ll show an example of how to deploy a simple Node.js app and do rollout update and undoing the rollout.

NOTE ABOUT VERSIONS

Kubernetes and the surrounding components are changed on a daily basis. What works today, might not work tomorrow.

This tutorial assumes that you use CentOS 7, Docker 18.x and Kubernetes 1.14.

For this post, there are some pre-requisites. You will need 2 servers with 2 CPUs and at least 2GB RAM. It is also recommended to have a working DNS. If you don’t have DNS in your lab, make sure you use /etc/hosts for hostname resolution, but you can get away if you use IPs only (not recommended).

Pre-requisites

On a fresh installed CentOS 7, do these pre-req commands on both the master and the node at the same time. You need to be logged as root.
Make sure SELinux is disabled.

setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

Kubernetes doesn’t like swap, so if you have it in /etc/fstab, disable the swap.

swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

Enable the bridge network module.

modprobe br_netfilter
echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables
echo '1' > /proc/sys/net/bridge/bridge-nf-call-ip6tables

Install Docker and change the cgroup from cfsgroup to systemd.

yum -y install yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum -y install docker-ce
mkdir /etc/docker
cat < /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
systemctl daemon-reload
systemctl enable docker && systemctl start docker

Try this line and make sure the output says systemd.

docker info | grep -i cgroup
Cgroup Driver: systemd

Add the Kubernetes repo.

cat < /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

Master Node

Open the firewall ports.

firewall-cmd --add-port=6443/tcp --permanent
firewall-cmd --add-port=2379-2380/tcp --permanent
firewall-cmd --add-port=10250-10252/tcp --permanent
firewall-cmd --reload

From the repo install kubelet, kubectl and kubeadm and make sure Kubernetes starts on boot.

yum -y install kubelet kubectl kubeadm
systemctl enable kubelet

Don’t start Kubernetes yet. It will fail with a message that it can’t find a config yaml file. Just initialize the cluster. This will also start the kubelet service. Pick one choice (Flannel or Calico).
NOTE: This line initializes the cluster to be used for Flannel.

kubeadm init --pod-network-cidr=10.244.0.0/16

NOTE: This line initializes the cluster to be used for Calico.

kubeadm init --pod-network-cidr=192.168.0.0/16

Look at the bottom of the output. You should see something like this. Lines 5,6,7 and 15 and 16 are important.

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.1.175:6443 --token 0z3iov.28rz29hxw9ft4jmg \
    --discovery-token-ca-cert-hash sha256:782d37b5c870ebebd2f17cc3ba1424f305e6a3e293afc04fc2030edfab6bf4b0

This means that the cluster initialized OK.
Check the status of both Docker and Kubernetes.

systemctl status docker | grep Active
systemctl status kubelet | grep Active

Make sure they are both running. Check /var/log/messages if you have any issues.

Nodes (workers)

On the worker nodes, make sure you do the same as you did on the master (swap, SELinux, Docker) except that you don’t have to install kubectl.

yum -y install kubelet kubeadm
systemctl enable kubelet

Open the firewall ports.

firewall-cmd --add-port=10250/tcp --permanent
firewall-cmd --add-port=30000-32767/tcp --permanent
firewall-cmd --reload

Now, you can join the cluster. Use the command that was the output from the kubeadm init on the master (see above – lines 15 and 16).

kubeadm join 192.168.1.175:6443 --token 0z3iov.28rz29hxw9ft4jmg \
    --discovery-token-ca-cert-hash sha256:782d37b5c870ebebd2f17cc3ba1424f305e6a3e293afc04fc2030edfab6bf4b0

That’s how you join nodes to the master. Replace 192.168.1.175 with the IP or hostname of your master node. If everything is OK, you’ll see something like this.

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

Kubernetes user

While still logged as root on the master, create the Kubernetes user that you will use for managing the k8s cluster. In my case, I’ll create a user called k8s with secret as password.

useradd k8s -g docker
usermod -aG wheel k8s
echo -e "secret\nsecret" | passwd k8s

Log as this user (k8s) and execute these commands. These lines were also an output of the kubeadm init command above (5,6 and 7).

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

If you want to add another user to manage the Kubernetes cluster, make sure you execute these 3 lines above for that user. Check if everything looks good.

docker ps

You should see a bunch of Kubernetes system containers running (etcd, scheduler, API server).
Then check the nodes.

kubectl get nodes
NAME                      STATUS     ROLES    AGE   VERSION
k8smaster.andreev.local   NotReady   master   12m   v1.14.0
k8snode1.andreev.local    NotReady      10m   v1.14.0

The reason the master and the node are not ready is because we don’t have a network for the cluster.

Network CNI

Depending on how you’ve initialized the cluster, pick one of the network plugins (Flannel or Calico).

Flannel

For the network to work, we’ll have to use one of the CNI plugins. There are many, Flannel, Weave Net, Calico etc.
Let’s install Flannel. Do this on the master only logged as k8s user. The master will take care of the nodes.

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Start this little infinite loop and you’ll see that after 20-30 seconds, both the master and the node will change their status to Ready. Hit Ctrl-C to end.

while true
do
kubectl get nodes
sleep 3
done

Now, you have a fully working cluster ready.

Calico

kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

Start this little infinite loop and you’ll see that after 20-30 seconds, both the master and the node will change their status to Ready. Hit Ctrl-C to end.

while true
do
kubectl get nodes
sleep 3
done

Now, you have a fully working cluster ready.

Deployment

In this example, I’ll create a small container that runs a Node.js app that when run it will display “Hello from ” the hostname of the container. On top of that we’ll create a load balancer, so we can see how that works.
First, let’s create the container based on Node.js image. Create a file named Dockerfile with this content.

FROM node:latest
LABEL maintainer "kliment@andreev.it"
ADD appv1.js /app.js
ENTRYPOINT ["node", "app.js"]

This is our application. Save it as appv1.js.

const http = require('http');
const os = require('os');
const port = 3000;

const server = http.createServer((req, res) => {
  res.statusCode = 200;
  res.end('Hello from ' + os.hostname() + '\n');
});

server.listen(port);

Create the container. You’ll need a valid Docker Hub login. In my case, my username is klimenta. Replace it with yours.

docker build -t klimenta/appv1:latest .

It’s time to login to Docker Hub and upload the image there. You’ll be prompted for a username and password.

docker login

Upload the image.

docker push klimenta/appv1:latest

Create a Kubernetes deployment file named deployment.yaml.

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: appv1
spec:
  replicas: 3
  template:
    metadata:
      name: appv1
      labels:
        app: appv1
    spec:
      containers:
      - image: klimenta/appv1:latest
        name: nodejs
---
apiVersion: v1
kind: Service
metadata:
  name: loadbalancer
spec:
  type: LoadBalancer
  selector:
    app: appv1
  ports:
  - port: 80
    targetPort: 3000

We are creating a deployment with 3 replicas and a load balancer that listens on port 80 and sends the traffic to port 3000 on the pods with our application.
Create the deployment and the load balanced service.

kubectl create -f deployment.yaml

After about 30 seconds, you’ll see that your pods are ready.

kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
appv1-596dd64666-4k7qn   1/1     Running   0          93m
appv1-596dd64666-gn5gr   1/1     Running   0          93m
appv1-596dd64666-vv9h5   1/1     Running   0          93m

The load balancer is also ready.

kubectl get svc
NAME           TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes     ClusterIP      10.96.0.1               443/TCP        102m
loadbalancer   LoadBalancer   10.111.212.41        80:30310/TCP   94m

If you hit the load balancer, you’ll see a response. Replace the IP with yours.

curl http://10.111.212.41
Hello from appv1-596dd64666-gn5gr

Now, let’s say that we created a new version of our application. Copy appv1.js as appv2.js and change appv2.js a little bit.

cp appv1.js appv2.js

The appv2.js should look like this.

const http = require('http');
const os = require('os');
const port = 3000;

const server = http.createServer((req, res) => {
  res.statusCode = 200;
  res.end('Greetings from ' + os.hostname() + '\n');
});

server.listen(port);

Change the Dockerfile to look like this.

FROM node:latest
LABEL maintainer "kliment@andreev.it"
ADD appv2.js /app.js
ENTRYPOINT ["node", "app.js"]

Build the new image and upload it to Docker Hub.

docker build -t klimenta/appv2:latest .
docker push klimenta/appv2:latest

Deploy the new application.

kubectl set image deployment appv1 nodejs=klimenta/appv2:latest

If you check the app now, you’ll see that it reflects the new version.

curl http://10.111.212.41
Greetings from appv1-5d949774f5-b794k

But what if there is a bug in our application and we want to revert it back to the initial one? Easy.

kubectl rollout undo deployment appv1
deployment.extensions/appv1 rolled back
curl http://10.111.212.41
Hello from appv1-596dd64666-94gqh