In this post, I’ll build a Kubernetes cluster with one master and two nodes using CRI-O and Cilium. I have DNS in my environment but you can use the hosts files if needed.
The OS will be Rocky Linux 9 running on ESXi 8.x but this should work on any other bare metal server too. The config for the nodes is:

– Master: 2 CPUs, 4GB RAM
– Workers: 2 CPUs, 8GB RAM.

First, we’ll have to disable swap and put SELinux in permissive mode.
Do this on all nodes (master + workers).

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

Now, we have to open some firewall ports for Kubernetes..
Do this on the master node only.

sudo firewall-cmd --permanent --add-port=6443/tcp
sudo firewall-cmd --permanent --add-port=2379-2380/tcp
sudo firewall-cmd --permanent --add-port=10250/tcp
sudo firewall-cmd --permanent --add-port=10259/tcp
sudo firewall-cmd --permanent --add-port=10257/tcp
sudo firewall-cmd --reload

Do this on the workers only.

sudo firewall-cmd --permanent --add-port=10250/tcp
sudo firewall-cmd --permanent --add-port=30000-32767/tcp   
sudo firewall-cmd --reload

In addition, we’ll have to open the firewall ports for Cilium.
Do this on all nodes.

sudo firewall-cmd --permanent --add-port=4240/tcp 
sudo firewall-cmd --permanent --add-port=8472/udp 
sudo firewall-cmd --reload

Now, we have to make some changes that are also required. Some necessary modules and IPtables change.
Do this on all nodes.

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# Apply sysctl params without reboot
sudo sysctl --system

We have to install CRI-O container runtime.
Do this on all nodes.

VERSION=1.22
sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/CentOS_8/devel:kubic:libcontainers:stable.repo
sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:${VERSION}.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:${VERSION}/CentOS_8/devel:kubic:libcontainers:stable:cri-o:${VERSION}.repo
sudo dnf -y install cri-o cri-tools
sudo systemctl enable --now crio
sudo systemctl status crio

Then, we have to install Kubernetes.
Do this on all nodes.

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF
sudo dnf install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
sudo systemctl enable --now kubelet

On the master node only, initialize the cluster. Replace master.homelab.com with your master IP if you don’t use DNS.

sudo kubeadm init --control-plane-endpoint master.homelab.local:6443

You’ll get an output that says how to join the nodes, something like kubeadm join master… + a token
On the master node, add these lines.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Join the nodes to the cluster.
Do this on workers only.

sudo kubeadm join master.homelab.local:6443 --token t2yir0.7n49rn6r57msy4j4 \
        --discovery-token-ca-cert-hash sha256:563360fcf60c49be91cdcf6486a4954c579a80c54503127ee0682ab8f86ec840

Check the nodes.
Do this on the master node.

kubectl get nodes
NAME                   STATUS   ROLES           AGE    VERSION
master.homelab.local   Ready    control-plane   116s   v1.28.2
node1.homelab.local    Ready    <none>          89s    v1.28.2
node2.homelab.local    Ready    <none>          85s    v1.28.2

…then check all the pods.
Do this on the master node.

kubectl get pods --all-namespaces
NAMESPACE     NAME                                           READY   STATUS    RESTARTS   AGE
kube-system   coredns-5dd5756b68-5ljgg                       1/1     Running   0          2m
kube-system   coredns-5dd5756b68-jp8tq                       1/1     Running   0          2m
kube-system   etcd-master.homelab.local                      1/1     Running   0          2m6s
kube-system   kube-apiserver-master.homelab.local            1/1     Running   0          2m6s
kube-system   kube-controller-manager-master.homelab.local   1/1     Running   0          2m6s
kube-system   kube-proxy-2z64r                               1/1     Running   0          2m
kube-system   kube-proxy-mrwjv                               1/1     Running   0          96s
kube-system   kube-proxy-nj5q4                               1/1     Running   0          100s
kube-system   kube-scheduler-master.homelab.local            1/1     Running   0          2m6s

Install Cilium on the master node only.

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
cilium install --version 1.14.2

Check the status.

cilium status --wait

You should see something like this.

    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium             Desired: 3, Ready: 3/3, Available: 3/3
Deployment             cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
Containers:            cilium             Running: 3
                       cilium-operator    Running: 1
Cluster Pods:          2/2 managed by Cilium
Helm chart version:    1.14.2
Image versions         cilium             quay.io/cilium/cilium:v1.14.2@sha256:6263f3a3d5d63b267b538298dbeb5ae87da3efacf09a2c620446c873ba807d35: 3
                       cilium-operator    quay.io/cilium/operator-generic:v1.14.2@sha256:52f70250dea22e506959439a7c4ea31b10fe8375db62f5c27ab746e3a2af866d: 1

Then you can run the connectivity test. Some of the egress test will fail because we don’t have a public IP on the cluster.

cilium connectivity test

You should be able to run your deployments now. If you need to access the pods using URL use something like NodePort or port-forwarding. It’s very cumbersome and inefficient, but you can use a load balancer with MetalLB. Read this post to see how to do that. It’s separated from this post, because it’s a separate topic than just k8s, container runtime and CNI.