Kubernetes (k8s) is getting a lot of attention and it’s becoming more and more popular even in enterprises that are quite IT conservative. In this post I’ll explain how to install Kubernetes on a single master and one node on CentOS. Then, we’ll install the networking plugins (CNI), Flannel or Calico. At the end I’ll show an example of how to deploy a simple Node.js app and do rollout update and undoing the rollout.

For this post, there are some pre-requisites. You will need 2 servers with 2 CPUs and at least 2GB RAM. It is also recommended to have a working DNS. If you don’t have DNS in your lab, make sure you use /etc/hosts for hostname resolution, but you can get away if you use IPs only (not recommended).

Pre-requisites

On a fresh installed CentOS 7, do these pre-req commands on both the master and the node at the same time. You need to be logged as root.
Make sure SELinux is disabled.

setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

Kubernetes doesn’t like swap, so if you have it in /etc/fstab, disable the swap.

swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

Enable the bridge network module.

modprobe br_netfilter
echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables
echo '1' > /proc/sys/net/bridge/bridge-nf-call-ip6tables

Install Docker and change the cgroup from cfsgroup to systemd.

yum -y install yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum -y install docker-ce
mkdir /etc/docker
cat <<EOF > /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
systemctl daemon-reload
systemctl enable docker && systemctl start docker

Try this line and make sure the output says systemd.

docker info | grep -i cgroup
Cgroup Driver: systemd

Add the Kubernetes repo.

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

Master Node

Open the firewall ports.

firewall-cmd --add-port=6443/tcp --permanent
firewall-cmd --add-port=2379-2380/tcp --permanent
firewall-cmd --add-port=10250-10252/tcp --permanent
firewall-cmd --reload

From the repo install kubelet, kubectl and kubeadm and make sure Kubernetes starts on boot.

yum -y install kubelet kubectl kubeadm
systemctl enable kubelet

Don’t start Kubernetes yet. It will fail with a message that it can’t find a config yaml file. Just initialize the cluster. This will also start the kubelet service. Pick one choice (Flannel or Calico).
NOTE: This line initializes the cluster to be used for Flannel.

kubeadm init --pod-network-cidr=10.244.0.0/16

NOTE: This line initializes the cluster to be used for Calico.

kubeadm init --pod-network-cidr=192.168.0.0/16

Look at the bottom of the output. You should see something like this. Lines 5,6,7 and 15 and 16 are important.

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.1.175:6443 --token 0z3iov.28rz29hxw9ft4jmg \
    --discovery-token-ca-cert-hash sha256:782d37b5c870ebebd2f17cc3ba1424f305e6a3e293afc04fc2030edfab6bf4b0

This means that the cluster initialized OK.
Check the status of both Docker and Kubernetes.

systemctl status docker | grep Active
systemctl status kubelet | grep Active

Make sure they are both running. Check /var/log/messages if you have any issues.

Nodes (workers)

On the worker nodes, make sure you do the same as you did on the master (swap, SELinux, Docker) except that you don’t have to install kubectl.

yum -y install kubelet kubeadm
systemctl enable kubelet

Open the firewall ports.

firewall-cmd --add-port=10250/tcp --permanent
firewall-cmd --add-port=30000-32767/tcp --permanent
firewall-cmd --reload

Now, you can join the cluster. Use the command that was the output from the kubeadm init on the master (see above – lines 15 and 16).

kubeadm join 192.168.1.175:6443 --token 0z3iov.28rz29hxw9ft4jmg \
    --discovery-token-ca-cert-hash sha256:782d37b5c870ebebd2f17cc3ba1424f305e6a3e293afc04fc2030edfab6bf4b0

That’s how you join nodes to the master. Replace 192.168.1.175 with the IP or hostname of your master node. If everything is OK, you’ll see something like this.

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

Kubernetes user

While still logged as root on the master, create the Kubernetes user that you will use for managing the k8s cluster. In my case, I’ll create a user called k8s with secret as password.

useradd k8s -g docker
usermod -aG wheel k8s
echo -e "secret\nsecret" | passwd k8s

Log as this user (k8s) and execute these commands. These lines were also an output of the kubeadm init command above (5,6 and 7).

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

If you want to add another user to manage the Kubernetes cluster, make sure you execute these 3 lines above for that user. Check if everything looks good.

docker ps

You should see a bunch of Kubernetes system containers running (etcd, scheduler, API server).
Then check the nodes.

kubectl get nodes
NAME                      STATUS     ROLES    AGE   VERSION
k8smaster.andreev.local   NotReady   master   12m   v1.14.0
k8snode1.andreev.local    NotReady   <none>   10m   v1.14.0

The reason the master and the node are not ready is because we don’t have a network for the cluster.

Network CNI

Depending on how you’ve initialized the cluster, pick one of the network plugins (Flannel or Calico).

Flannel

For the network to work, we’ll have to use one of the CNI plugins. There are many, Flannel, Weave Net, Calico etc.
Let’s install Flannel. Do this on the master only logged as k8s user. The master will take care of the nodes.

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Start this little infinite loop and you’ll see that after 20-30 seconds, both the master and the node will change their status to Ready. Hit Ctrl-C to end.

while true
do
kubectl get nodes
sleep 3
done

Now, you have a fully working cluster ready.

Calico

For the network to work, we’ll have to use one of the CNI plugins. There are many, Flannel, Weave Net, Calico etc.
Let’s install Flannel. Do this on the master only logged as k8s user. The master will take care of the nodes.

kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

Start this little infinite loop and you’ll see that after 20-30 seconds, both the master and the node will change their status to Ready. Hit Ctrl-C to end.

while true
do
kubectl get nodes
sleep 3
done

Now, you have a fully working cluster ready.

Deployment

In this example, I’ll create a small container that runs a Node.js app that when run it will display “Hello from ” the hostname of the container. On top of that we’ll create a load balancer, so we can see how that works.
First, let’s create the container based on Node.js image. Create a file named Dockerfile with this content.

FROM node:latest
LABEL maintainer "kliment@andreev.it"
ADD appv1.js /app.js
ENTRYPOINT ["node", "app.js"]

This is our application. Save it as appv1.js.

const http = require('http');
const os = require('os');
const port = 3000;

const server = http.createServer((req, res) => {
  res.statusCode = 200;
  res.end('Hello from ' + os.hostname() + '\n');
});

server.listen(port);

Create the container. You’ll need a valid Docker Hub login. In my case, my username is klimenta. Replace it with yours.

docker build -t klimenta/appv1:latest .

It’s time to login to Docker Hub and upload the image there. You’ll be prompted for a username and password.

docker login

Upload the image.

docker push klimenta/appv1:latest

Create a Kubernetes deployment file named deployment.yaml.

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: appv1
spec:
  replicas: 3
  template:
    metadata:
      name: appv1
      labels:
        app: appv1
    spec:
      containers:
      - image: klimenta/appv1:latest
        name: nodejs
---
apiVersion: v1
kind: Service
metadata:
  name: loadbalancer
spec:
  type: LoadBalancer
  selector:
    app: appv1
  ports:
  - port: 80
    targetPort: 3000

We are creating a deployment with 3 replicas and a load balancer that listens on port 80 and sends the traffic to port 3000 on the pods with our application.
Create the deployment and the load balanced service.

kubectl create -f deployment.yaml

After about 30 seconds, you’ll see that your pods are ready.

kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
appv1-596dd64666-4k7qn   1/1     Running   0          93m
appv1-596dd64666-gn5gr   1/1     Running   0          93m
appv1-596dd64666-vv9h5   1/1     Running   0          93m

The load balancer is also ready.

kubectl get svc
NAME           TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes     ClusterIP      10.96.0.1       <none>        443/TCP        102m
loadbalancer   LoadBalancer   10.111.212.41   <pending>     80:30310/TCP   94m

If you hit the load balancer, you’ll see a response. Replace the IP with yours.

curl http://10.111.212.41
Hello from appv1-596dd64666-gn5gr

Now, let’s say that we created a new version of our application. Copy appv1.js as appv2.js and change appv2.js a little bit.

cp appv1.js appv2.js

The appv2.js should look like this.

const http = require('http');
const os = require('os');
const port = 3000;

const server = http.createServer((req, res) => {
  res.statusCode = 200;
  res.end('Greetings from ' + os.hostname() + '\n');
});

server.listen(port);

Change the Dockerfile to look like this.

FROM node:latest
LABEL maintainer "kliment@andreev.it"
ADD appv2.js /app.js
ENTRYPOINT ["node", "app.js"]

Build the new image and upload it to Docker Hub.

docker build -t klimenta/appv2:latest .
docker push klimenta/appv2:latest

Deploy the new application.

kubectl set image deployment appv1 nodejs=klimenta/appv2:latest

If you check the app now, you’ll see that it reflects the new version.

curl http://10.111.212.41
Greetings from appv1-5d949774f5-b794k

But what if there is a bug in our application and we want to revert it back to the initial one? Easy.

kubectl rollout undo deployment appv1
deployment.extensions/appv1 rolled back
curl http://10.111.212.41
Hello from appv1-596dd64666-94gqh