Calico – Blog of Kliment Andreev – A place so I won't forget things

AWS: EKS running out of IPs, increase pod density

Kliment Andreev — Tue, 06 Jun 2023 21:04:16 +0000

When I was doing some proof of concept using EKS, I’ve noticed that there is a limitation on how many pods can a node run and how EKS CNI assigns the subnet IPs. Apparently, a node can run a maximum of XX pods (based on type) which is very low, e.g. a t3.large instance with 2 CPUs and 8GB RAM can run only 35 pods. On top of that, the IPs that are assigned for the pods are used from the existing pool of IPs and if you have a /24 subnet for the EKS cluster, you’ll run out of IPs in no time. Here is an example of what I did and options how to solve this issue.

Option 1 – Calico 3rd party CNI

I’ll provision an EKS cluster in a /24 subnet with 3 public and 3 private subnets.

eksctl create cluster --name eksECIC --region us-east-2 --instance-types t3.large \
    --managed --vpc-cidr 192.168.100.0/24 --node-private-networking --version 1.24 --without-nodegroup

I am using the Ohio region which has 3 availability zones. The public and private subnets (total of 6) have /27 subnet which is 32 IPs. Not all of them are usable of course. Let’s create a node group with 3 nodes, and because t3.large is a Nitro instance, we can set the max number of pods to be 110.

eksctl create nodegroup --cluster eksECIC --name old-nodegroup --nodes 3 \
    --node-type t3.large --node-private-networking --managed --max-pods-per-node 200

These are the available 31 IPs that I have in the private subnets.

Let’s provision a small deployment with a single pod and 10 replicas and a network load balancer.

kubectl apply -f demo.yaml

…and this is the deployment file demo.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
spec:
  replicas: 10
  selector:
    matchLabels:
      run: demo
  template:
    metadata:
      labels:
        run: demo
    spec:
      containers:
      - name: demo
        image: klimenta/serverip
        ports:
        - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: loadbalancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
spec:
  ports:
    - port: 80
      targetPort: 3000
      protocol: TCP
  type: LoadBalancer
  selector:
    run: demo

If you look at the available address now, there are 7. So from 31 to 7 with just 10 pods. Check this guy’s blog to see what’s going on.

Let’s scale up the cluster to 100 pods.

kubectl scale --replicas=100 deployment/demo

After a minute or two, check the running pods.

kubectl get pods --field-selector=status.phase=Running | wc -l
69

The console will show 0 available subnets.

If you check the pods with kubectl get pods, you’ll see the remaining containers have a ContainerCreating status.
And if you check one of them, you’ll see this message.

kubectl describe pod demo-698f6fc958-z8smb
....
d9b65ccfeccb7e9e7433ec61fef78faf83d1bae": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  2m29s (x17 over 5m59s)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "edfa987c3eddf415068691f914d740fa9ca9ca6ffdfb8db44d1340d913ece0b0": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container

Get the load balancer service.

kubectl get svc
NAME           TYPE           CLUSTER-IP      EXTERNAL-IP                                                                     PORT(S)        AGE
kubernetes     ClusterIP      10.100.0.1                                                                                443/TCP        37m
loadbalancer   LoadBalancer   10.100.25.241   a105789cd3a264ac596c367fce463640-80d2b88acdc17771.elb.us-east-2.amazonaws.com   80:32396/TCP   17m

If you go to a105789cd3a264…amazonaws.com URL, you’ll see the load balancer hitting different pods. Wait for 2-3 minutes if you see that the page can’t be opened. It takes time for DNS to propagate.

Let’s delete the deployment and the old node group.

kubectl delete -f demo.yaml
eksctl delete nodegroup --name old-nodegroup --cluster eksECIC

At this point, we still have the EKS cluster, the etcd database, networking etc, we just don’t have the nodes. Now it’s time to remove the EKS CNI, install Calico and add nodes that will use Calico.
Installation is very simple. Remove the AWS CNI.

kubectl delete daemonset -n kube-system aws-node

Check the pods in all namespaces.

kubectl get pods --all-namespaces
NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
kube-system   coredns-5c5677bc78-2k88b   0/1     Pending   0          4m
kube-system   coredns-5c5677bc78-2zz2n   0/1     Pending   0          4m

You will see coredns in pending state, it can’t be deployed anywhere.
Deploy Calico.

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/tigera-operator.yaml

…keep going…

kubectl create -f - <
Deploy the new node group.
eksctl create nodegroup --cluster eksECIC --name new-nodegroup --nodes 3 \
--node-type t3.large --node-private-networking --managed --max-pods-per-node 200

Let’s deploy the same demo deployment now.
kubectl apply -f demo.yaml

…scale it up to 601 pods.
kubectl scale --replicas=601 deployment/demo

Wait 2-3 mins and check the running ones.
kubectl get pods --field-selector=status.phase=Running | wc -l

There will be 585 pods running, 195 per node. Much, much better. This time the remaining containers will be in Pending state.
kubectl describe pod demo-698f6fc958-smlbq
....
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 3m27s (x3 over 13m) default-scheduler 0/3 nodes are available: 3 Too many pods. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.

Option 2 – CNI Custom networking and prefix assignment mode
In this case, we’ll keep the AWS CNI network plugin, but we’ll use what is called CNI Custom networking, meaning we’ll use another CIDR attached to the VPC where pods will run. NOTE: The nodes will still run in the 192.168/24 subnets, it’s just the pods that will run in a much bigger IP space. In addition, this CNI Custom networking won’t solve the density of the pods that we can run on the nodes. We’ll need something called prefix assignment mode that will allow us to run much more pods on a node than usual. See this and this links for more information. For more info regarding prefix assignment mode check out this link. The solution described below is much more complicated than if you just use Calico.

Let’s provision a cluster with the same VPC CIDR and no nodegroups.
eksctl create cluster --name eksECIC --region us-east-2 --instance-types t3.large \
--managed --vpc-cidr 192.168.100.0/24 --node-private-networking --version 1.24 --without-nodegroup

Add a new CIDR and associate it with the VPC. I am using the console but you can use aws ec2 associate-vpc-cidr-block command.Go to the VPC where the EKS cluster resides. From the Actions button, select Edit CIDRs and then click the Add new IPv4 CIDR button.

I’ll add 100.64.0.0/16 CIDR and then I’ll create 3 /19 subnets.

These 3 subnets will be where the pods will run. I also want them to initiate connections to Internet, so I’ll modify the route table and add the Internet Gateway. The three new subnets 100.64/19 that we created have the same route table.

Just add the route to 0.0.0.0/0 to go over Internet Gateway.

Let’s configure the custom networking on the cluster.
kubectl set env daemonset aws-node -n kube-system AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true

Get the EKS cluster security group.
sec_grp=$(aws eks describe-cluster --name eksECIC --query cluster.resourcesVpcConfig.clusterSecurityGroupId --output text)
echo $sec_grp

For each of the subnets create a custom resource. Define the three 100.64 subnet-ids as variables.
export sub1=subnet-0f3ec5d75caf1b981
export sub2=subnet-0fc39edc5a1ce91c0
export sub3=subnet-0bd6bdfd5543aa086

cat >us-east-2a.yaml <
2nd subnet.
cat >us-east-2b.yaml <
3rd subnet.
cat >us-east-2c.yaml <
NOTE:Make sure you name the metadata in line 5 as I did based on your region and AZ.

Deploy the custom resources.
kubectl apply -f us-east-2a.yaml
kubectl apply -f us-east-2b.yaml
kubectl apply -f us-east-2c.yaml

Confirm it looks good.
kubectl get ENIConfigs
NAME AGE
us-east-2a 16s
us-east-2b 12s
us-east-2c 8s

Update your aws-node DaemonSet to automatically apply the ENIConfig for an Availability Zone to any new Amazon EC2 nodes created in your cluster.
kubectl set env daemonset aws-node -n kube-system ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone

Enable prefix assignment mode. This will allow much more pods per node.
kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

Another change.
kubectl set env ds aws-node -n kube-system WARM_PREFIX_TARGET=1

With the default setting, WARM_PREFIX_TARGET will allocate one additional complete (/28) prefix even if the existing prefix is used by only one pod. If the ENI does not have enough space to assign a prefix, a new ENI is generated.

Provision the node group.
eksctl create nodegroup --cluster eksECIC --name my-nodegroup --nodes 3 \
--node-type t3.large --node-private-networking --managed --max-pods-per-node 200

Check the nodes, they are running in the 192.168/24 original VPC CIDR. See the IP in the names of the nodes.
kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-100-122.us-east-2.compute.internal Ready 7m39s v1.24.13-eks-0a21954
ip-192-168-100-155.us-east-2.compute.internal Ready 7m39s v1.24.13-eks-0a21954
ip-192-168-100-167.us-east-2.compute.internal Ready 7m37s v1.24.13-eks-0a21954

Check the pods.
kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system aws-node-txwrd 1/1 Running 0 9m4s 192.168.100.167 ip-192-168-100-167.us-east-2.compute.internal
kube-system aws-node-vdmv7 1/1 Running 0 9m6s 192.168.100.155 ip-192-168-100-155.us-east-2.compute.internal
kube-system aws-node-xss7c 1/1 Running 0 9m6s 192.168.100.122 ip-192-168-100-122.us-east-2.compute.internal
kube-system coredns-5c5677bc78-2lkcx 1/1 Running 0 80m 100.64.76.225 ip-192-168-100-122.us-east-2.compute.internal
kube-system coredns-5c5677bc78-mj6wr 1/1 Running 0 80m 100.64.76.224 ip-192-168-100-122.us-east-2.compute.internal
kube-system kube-proxy-lt7v4 1/1 Running 0 9m6s 192.168.100.122 ip-192-168-100-122.us-east-2.compute.internal
kube-system kube-proxy-m8pf6 1/1 Running 0 9m4s 192.168.100.167 ip-192-168-100-167.us-east-2.compute.internal
kube-system kube-proxy-nlvw7 1/1 Running 0 9m6s 192.168.100.155 ip-192-168-100-155.us-east-2.compute.internal

Look at coredns pods. They are running in the new CIDR.

Deploy the demo.
kubectl apply -f demo.yaml

Get the load balancer service.
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.100.0.1 443/TCP 83m
loadbalancer LoadBalancer 10.100.206.187 a7ad3ecf1d52e4fc58dbbdc550237009-ac4778e11af2c0ad.elb.us-east-2.amazonaws.com 80:30230/TCP 45s

If you go to a7ad3ecf1d…amazonaws.com URL, you’ll see the load balancer hitting different pods. Wait for 2-3 minutes if you see that the page can’t be opened. It takes time for DNS to propagate.

The pods are running in the 100.64 subnet.

Scale up to 601 pods.
kubectl scale --replicas=601 deployment/demo

Wait 2-3 mins and check the running ones.
kubectl get pods --field-selector=status.phase=Running | wc -l

I got 593 total pods. Slightly better than 585 with Calico. The ones that are in pending state show up an error that there are no more resources.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 3m49s default-scheduler 0/3 nodes are available: 3 Too many pods. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.

CentOS: Kubernetes, Flannel and Calico in a single master configuration

Kliment Andreev — Tue, 09 Apr 2019 15:46:01 +0000

Kubernetes (k8s) is getting a lot of attention and it’s becoming more and more popular even in enterprises that are quite IT conservative. In this post I’ll explain how to install Kubernetes on a single master and one node on CentOS. Then, we’ll install the networking plugins (CNI), Flannel or Calico. At the end I’ll show an example of how to deploy a simple Node.js app and do rollout update and undoing the rollout.

NOTE ABOUT VERSIONS

Kubernetes and the surrounding components are changed on a daily basis. What works today, might not work tomorrow.

This tutorial assumes that you use CentOS 7, Docker 18.x and Kubernetes 1.14.

For this post, there are some pre-requisites. You will need 2 servers with 2 CPUs and at least 2GB RAM. It is also recommended to have a working DNS. If you don’t have DNS in your lab, make sure you use /etc/hosts for hostname resolution, but you can get away if you use IPs only (not recommended).

Pre-requisites

On a fresh installed CentOS 7, do these pre-req commands on both the master and the node at the same time. You need to be logged as root.
Make sure SELinux is disabled.

setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

Kubernetes doesn’t like swap, so if you have it in /etc/fstab, disable the swap.

swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

Enable the bridge network module.

modprobe br_netfilter
echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables
echo '1' > /proc/sys/net/bridge/bridge-nf-call-ip6tables

Install Docker and change the cgroup from cfsgroup to systemd.

yum -y install yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum -y install docker-ce
mkdir /etc/docker
cat < /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
systemctl daemon-reload
systemctl enable docker && systemctl start docker

Try this line and make sure the output says systemd.

docker info | grep -i cgroup
Cgroup Driver: systemd

Add the Kubernetes repo.

cat < /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

Master Node

Open the firewall ports.

firewall-cmd --add-port=6443/tcp --permanent
firewall-cmd --add-port=2379-2380/tcp --permanent
firewall-cmd --add-port=10250-10252/tcp --permanent
firewall-cmd --reload

From the repo install kubelet, kubectl and kubeadm and make sure Kubernetes starts on boot.

yum -y install kubelet kubectl kubeadm
systemctl enable kubelet

Don’t start Kubernetes yet. It will fail with a message that it can’t find a config yaml file. Just initialize the cluster. This will also start the kubelet service. Pick one choice (Flannel or Calico).
NOTE: This line initializes the cluster to be used for Flannel.

kubeadm init --pod-network-cidr=10.244.0.0/16

NOTE: This line initializes the cluster to be used for Calico.

kubeadm init --pod-network-cidr=192.168.0.0/16

Look at the bottom of the output. You should see something like this. Lines 5,6,7 and 15 and 16 are important.

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.1.175:6443 --token 0z3iov.28rz29hxw9ft4jmg \
    --discovery-token-ca-cert-hash sha256:782d37b5c870ebebd2f17cc3ba1424f305e6a3e293afc04fc2030edfab6bf4b0

This means that the cluster initialized OK.
Check the status of both Docker and Kubernetes.

systemctl status docker | grep Active
systemctl status kubelet | grep Active

Make sure they are both running. Check /var/log/messages if you have any issues.

Nodes (workers)

On the worker nodes, make sure you do the same as you did on the master (swap, SELinux, Docker) except that you don’t have to install kubectl.

yum -y install kubelet kubeadm
systemctl enable kubelet

Open the firewall ports.

firewall-cmd --add-port=10250/tcp --permanent
firewall-cmd --add-port=30000-32767/tcp --permanent
firewall-cmd --reload

Now, you can join the cluster. Use the command that was the output from the kubeadm init on the master (see above – lines 15 and 16).

kubeadm join 192.168.1.175:6443 --token 0z3iov.28rz29hxw9ft4jmg \
    --discovery-token-ca-cert-hash sha256:782d37b5c870ebebd2f17cc3ba1424f305e6a3e293afc04fc2030edfab6bf4b0

That’s how you join nodes to the master. Replace 192.168.1.175 with the IP or hostname of your master node. If everything is OK, you’ll see something like this.

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

Kubernetes user

While still logged as root on the master, create the Kubernetes user that you will use for managing the k8s cluster. In my case, I’ll create a user called k8s with secret as password.

useradd k8s -g docker
usermod -aG wheel k8s
echo -e "secret\nsecret" | passwd k8s

Log as this user (k8s) and execute these commands. These lines were also an output of the kubeadm init command above (5,6 and 7).

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

If you want to add another user to manage the Kubernetes cluster, make sure you execute these 3 lines above for that user. Check if everything looks good.

docker ps

You should see a bunch of Kubernetes system containers running (etcd, scheduler, API server).
Then check the nodes.

kubectl get nodes
NAME                      STATUS     ROLES    AGE   VERSION
k8smaster.andreev.local   NotReady   master   12m   v1.14.0
k8snode1.andreev.local    NotReady      10m   v1.14.0

The reason the master and the node are not ready is because we don’t have a network for the cluster.

Network CNI

Depending on how you’ve initialized the cluster, pick one of the network plugins (Flannel or Calico).

Flannel

For the network to work, we’ll have to use one of the CNI plugins. There are many, Flannel, Weave Net, Calico etc.
Let’s install Flannel. Do this on the master only logged as k8s user. The master will take care of the nodes.

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Start this little infinite loop and you’ll see that after 20-30 seconds, both the master and the node will change their status to Ready. Hit Ctrl-C to end.

while true
do
kubectl get nodes
sleep 3
done

Now, you have a fully working cluster ready.

Calico

kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

Start this little infinite loop and you’ll see that after 20-30 seconds, both the master and the node will change their status to Ready. Hit Ctrl-C to end.

while true
do
kubectl get nodes
sleep 3
done

Now, you have a fully working cluster ready.

Deployment

In this example, I’ll create a small container that runs a Node.js app that when run it will display “Hello from ” the hostname of the container. On top of that we’ll create a load balancer, so we can see how that works.
First, let’s create the container based on Node.js image. Create a file named Dockerfile with this content.

FROM node:latest
LABEL maintainer "kliment@andreev.it"
ADD appv1.js /app.js
ENTRYPOINT ["node", "app.js"]

This is our application. Save it as appv1.js.

const http = require('http');
const os = require('os');
const port = 3000;

const server = http.createServer((req, res) => {
  res.statusCode = 200;
  res.end('Hello from ' + os.hostname() + '\n');
});

server.listen(port);

Create the container. You’ll need a valid Docker Hub login. In my case, my username is klimenta. Replace it with yours.

docker build -t klimenta/appv1:latest .

It’s time to login to Docker Hub and upload the image there. You’ll be prompted for a username and password.

docker login

Upload the image.

docker push klimenta/appv1:latest

Create a Kubernetes deployment file named deployment.yaml.

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: appv1
spec:
  replicas: 3
  template:
    metadata:
      name: appv1
      labels:
        app: appv1
    spec:
      containers:
      - image: klimenta/appv1:latest
        name: nodejs
---
apiVersion: v1
kind: Service
metadata:
  name: loadbalancer
spec:
  type: LoadBalancer
  selector:
    app: appv1
  ports:
  - port: 80
    targetPort: 3000

We are creating a deployment with 3 replicas and a load balancer that listens on port 80 and sends the traffic to port 3000 on the pods with our application.
Create the deployment and the load balanced service.

kubectl create -f deployment.yaml

After about 30 seconds, you’ll see that your pods are ready.

kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
appv1-596dd64666-4k7qn   1/1     Running   0          93m
appv1-596dd64666-gn5gr   1/1     Running   0          93m
appv1-596dd64666-vv9h5   1/1     Running   0          93m

The load balancer is also ready.

kubectl get svc
NAME           TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes     ClusterIP      10.96.0.1               443/TCP        102m
loadbalancer   LoadBalancer   10.111.212.41        80:30310/TCP   94m

If you hit the load balancer, you’ll see a response. Replace the IP with yours.

curl http://10.111.212.41
Hello from appv1-596dd64666-gn5gr

Now, let’s say that we created a new version of our application. Copy appv1.js as appv2.js and change appv2.js a little bit.

cp appv1.js appv2.js

The appv2.js should look like this.

const http = require('http');
const os = require('os');
const port = 3000;

const server = http.createServer((req, res) => {
  res.statusCode = 200;
  res.end('Greetings from ' + os.hostname() + '\n');
});

server.listen(port);

Change the Dockerfile to look like this.

FROM node:latest
LABEL maintainer "kliment@andreev.it"
ADD appv2.js /app.js
ENTRYPOINT ["node", "app.js"]

Build the new image and upload it to Docker Hub.

docker build -t klimenta/appv2:latest .
docker push klimenta/appv2:latest

Deploy the new application.

kubectl set image deployment appv1 nodejs=klimenta/appv2:latest

If you check the app now, you’ll see that it reflects the new version.

curl http://10.111.212.41
Greetings from appv1-5d949774f5-b794k

But what if there is a bug in our application and we want to revert it back to the initial one? Easy.

kubectl rollout undo deployment appv1
deployment.extensions/appv1 rolled back
curl http://10.111.212.41
Hello from appv1-596dd64666-94gqh