EKS – Blog of Kliment Andreev – A place so I won't forget things

AWS: Install EKS cluster and Ingress Controller using Terraform and GitLab using OIDC and web identity

Kliment Andreev — Tue, 21 Nov 2023 18:39:16 +0000

In this post, I’ll explain how to create an EKS cluster in AWS and add an Ingress Controller. We’ll use GitLab as a repo and execute the CI/CD pipeline from there.
For this you’ll need:

GitLab and AWS account
Terraform, AWS CLI, helm installed

AWS

From the IAM console, create a new Identity Provider in AWS and use https://gitlab.com for both Provider URL and Audience. For self hosted GitLab use gitlab.domain.com or whatever is your GitLab URL.
Click on Get thumbprint before clicking on Add provider.

In the IAM console, create a role. I’ll name the role rolGitLab, name it whatever you like. When you create the policy, choose Custom trust policy.

Now, here is what you have to decide. If you want to manage the cluster from GitLab only, you can ommit lines 16-23 where I allow the IAM user called gitlab to assume this role. But if you want some IAM user to manage the cluster using aws cli or kubectl, add the ARN of that IAM user in line 20. Add AdministratorAccess to that gitlab user but you can fine-tune those policies later. If you want to manage the cluster from GitLab, you have to register an agent and provide a config file. You can read how to do that here.
Replace the default custom trust policy with this JSON.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Principal": {
				"Federated": "arn:aws:iam::123456789012:oidc-provider/gitlab.com"
			},
			"Action": "sts:AssumeRoleWithWebIdentity",
			"Condition": {
				"StringLike": {
					"gitlab.com:sub": "project_path:*/eks-gitlab:ref_type:branch:ref:main"
				}
			}
		},
		{
			"Sid": "",
			"Effect": "Allow",
			"Principal": {
				"AWS": "arn:aws:iam::123456789012:user/gitlab"
			},
			"Action": "sts:AssumeRole"
		}
	]
}

In line 7 and 20, change the 123456789012 with your account number and in line 7 change gitlab.com with however you named the Identity Provider (IdP). In line 12 replace eks-gitlab with how you plan to name your GitLab project. In line 20 change the gitlab user to however you named your IAM user to access the EKS cluster. Click Next and when asked for a permission policy, choose AdministratorAccess. Best practice is to limit access to only what’s needed, but EKS has so many policies, it will take a lot to list them all.
In case you created the gitlab IAM user, attach the AdministratorPolicy (fine tune later) and this inline policy. Change the ARN in line 7. Name this inline policy something like polAssumeGitLabRole.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::123456789012:role/rolGitLab"
        }
    ]
}

GitLab

Create a new blank project in Gitlab with the default settings. In my case the project name is called eks-gitlab.

Go to Settings on the left, then CI/CD and then Variables. Add these two variables.

AWS_PROFILE and a value of oidc. Yes, the value is oidc. Just type these 4 characters for the value.
ROLE_ARN and a value of the role ARN from the role you just created for rolGitLab

Create a .gitlab-ci.yml file with the following content. Change the domain for gitlab.com at line 7 if you use self-hosted GitLab.

assume role:
  image: 
    name: amazon/aws-cli:latest
    entrypoint: [""]
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com
  before_script:
    - mkdir -p ~/.aws
    - echo "${GITLAB_OIDC_TOKEN}" > /tmp/web_identity_token
    - echo -e "[profile oidc]\nrole_arn=${ROLE_ARN}\nweb_identity_token_file=/tmp/web_identity_token" > ~/.aws/config
  script:
      - aws sts get-caller-identity
      - aws s3 ls

By the time you commit changes, the pipeline will start and if everything is OK, you’ll see your S3 buckets in the logs of the job if you go to Build | Jobs.

One more thing related to Terraform state as we’ll keep the state in GitLab is that we have to create an Access Token. If you have a self-hosted or paid GitLab account, create an Access Token from the Project | Settings | Access Tokens.
If you have a free GitLab account, you have to create a Personal Access Token and you do that if you click on your avatar in the upper left corner, then select Edit Profile and then click Access Tokens on the left. Click Add new token, name it GITLAB_PAT and select API only. Click Create personal access token. Once the token has been created click on the eye icon where it says Your new personal access token and copy & paste somewhere safe.

Go to the project settings and add these 2 variables.

TF_USERNAME which is your username, click on the avatar, look at the @ under your name, your username is without the @ sign
TF_PASSWORD which is the value of your personal access token you just created

Local machine

Now that we know the pipeline is working properly, let’s clone the project on our local machine.
If you plan to manage the cluster from your local machine, now it’s time to configure the gitlab IAM user on your local machine using access keys.
You need to have a valid key in order to clone your private repo.
Upload your .pub key to GitLab by clicking your avatar, then Edit Profile, click SSH Keys and then Add new key.
Finally, check the connection with the following command. It doesn’t matter what your local Linux user is, just use git@gitlab.com as the username@hostname. Type as is.

ssh -T git@gitlab.com
Welcome to GitLab, @[your_gitlab_username_here]

Once you clone the repo add the following files locally.
FILE: .gitignore

# Local .terraform directories
**/.terraform/*

# .tfstate files
*.tfstate
*.tfstate.*

# Crash log files
crash.log
crash.*.log

# Exclude all .tfvars files, which are likely to contain sensitive data, such as
# password, private keys, and other secrets. These should not be part of version 
# control as they are data points which are potentially sensitive and subject 
# to change depending on the environment.
*.tfvars
*.tfvars.json

# Ignore override files as they are usually used to override resources locally and so
# are not checked in
override.tf
override.tf.json
*_override.tf
*_override.tf.json

# Include override files you do wish to add to version control using negated pattern
# !example_override.tf

# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
# example: *tfplan*

# Ignore CLI configuration files
.terraformrc
terraform.rc

FILE: LICENSE

This is free and unencumbered software released into the public domain.

Anyone is free to copy, modify, publish, use, compile, sell, or
distribute this software, either in source code form or as a compiled
binary, for any purpose, commercial or non-commercial, and by any
means.

In jurisdictions that recognize copyright laws, the author or authors
of this software dedicate any and all copyright interest in the
software to the public domain. We make this dedication for the benefit
of the public at large and to the detriment of our heirs and
successors. We intend this dedication to be an overt act of
relinquishment in perpetuity of all present and future rights to this
software under copyright law.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.

For more information, please refer to

FILE: alb_ingress_iam_policy.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:CreateServiceLinkedRole"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeAccountAttributes",
                "ec2:DescribeAddresses",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeInternetGateways",
                "ec2:DescribeVpcs",
                "ec2:DescribeVpcPeeringConnections",
                "ec2:DescribeSubnets",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeInstances",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeTags",
                "ec2:GetCoipPoolUsage",
                "ec2:DescribeCoipPools",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeLoadBalancerAttributes",
                "elasticloadbalancing:DescribeListeners",
                "elasticloadbalancing:DescribeListenerCertificates",
                "elasticloadbalancing:DescribeSSLPolicies",
                "elasticloadbalancing:DescribeRules",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeTargetGroupAttributes",
                "elasticloadbalancing:DescribeTargetHealth",
                "elasticloadbalancing:DescribeTags"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "cognito-idp:DescribeUserPoolClient",
                "acm:ListCertificates",
                "acm:DescribeCertificate",
                "iam:ListServerCertificates",
                "iam:GetServerCertificate",
                "waf-regional:GetWebACL",
                "waf-regional:GetWebACLForResource",
                "waf-regional:AssociateWebACL",
                "waf-regional:DisassociateWebACL",
                "wafv2:GetWebACL",
                "wafv2:GetWebACLForResource",
                "wafv2:AssociateWebACL",
                "wafv2:DisassociateWebACL",
                "shield:GetSubscriptionState",
                "shield:DescribeProtection",
                "shield:CreateProtection",
                "shield:DeleteProtection"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:RevokeSecurityGroupIngress"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateSecurityGroup"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags"
            ],
            "Resource": "arn:aws:ec2:*:*:security-group/*",
            "Condition": {
                "StringEquals": {
                    "ec2:CreateAction": "CreateSecurityGroup"
                },
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags",
                "ec2:DeleteTags"
            ],
            "Resource": "arn:aws:ec2:*:*:security-group/*",
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:DeleteSecurityGroup"
            ],
            "Resource": "*",
            "Condition": {
                "Null": {
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:CreateLoadBalancer",
                "elasticloadbalancing:CreateTargetGroup"
            ],
            "Resource": "*",
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:CreateListener",
                "elasticloadbalancing:DeleteListener",
                "elasticloadbalancing:CreateRule",
                "elasticloadbalancing:DeleteRule"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:AddTags",
                "elasticloadbalancing:RemoveTags"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ],
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:AddTags",
                "elasticloadbalancing:RemoveTags"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:ModifyLoadBalancerAttributes",
                "elasticloadbalancing:SetIpAddressType",
                "elasticloadbalancing:SetSecurityGroups",
                "elasticloadbalancing:SetSubnets",
                "elasticloadbalancing:DeleteLoadBalancer",
                "elasticloadbalancing:ModifyTargetGroup",
                "elasticloadbalancing:ModifyTargetGroupAttributes",
                "elasticloadbalancing:DeleteTargetGroup"
            ],
            "Resource": "*",
            "Condition": {
                "Null": {
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:AddTags"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ],
            "Condition": {
                "StringEquals": {
                    "elasticloadbalancing:CreateAction": [
                        "CreateTargetGroup",
                        "CreateLoadBalancer"
                    ]
                },
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:RegisterTargets",
                "elasticloadbalancing:DeregisterTargets"
            ],
            "Resource": "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:SetWebAcl",
                "elasticloadbalancing:ModifyListener",
                "elasticloadbalancing:AddListenerCertificates",
                "elasticloadbalancing:RemoveListenerCertificates",
                "elasticloadbalancing:ModifyRule"
            ],
            "Resource": "*"
        }
    ]
}

FILE: backend.tf

terraform {
  backend "http" {
  }
}

FILE: data.tf

data "aws_availability_zones" "available" {
}

data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_name
  depends_on = [module.eks.cluster_name]
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_name
  depends_on = [module.eks.cluster_name]
}

FILE: main.tf NOTE: Make sure you add lines 40-41 and specify the rolGitlab role created earlier. If you don’t do this when you destroy the cluster, the KMS key for EKS will stay there and there is no way you can delete it unless you call AWS support.

# Create a VPC
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.0.0"

  name = var.vpcName

  cidr = var.vpcCIDR
  azs  = slice(data.aws_availability_zones.available.names, 0, 3)

  private_subnets = var.privateSubnets
  public_subnets  = var.publicSubnets

  enable_nat_gateway   = var.enable_nat_gateway
  single_nat_gateway   = var.single_nat_gateway
  enable_dns_hostnames = var.enable_dns_hostnames

  public_subnet_tags = {
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
    "kubernetes.io/role/elb"                    = 1
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
    "kubernetes.io/role/internal-elb"           = 1
  }
}

# Create an EKS cluster
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "19.15.3"

  cluster_name    = var.cluster_name
  cluster_version = var.cluster_version

  vpc_id                         = module.vpc.vpc_id
  subnet_ids                     = module.vpc.private_subnets
  cluster_endpoint_public_access = var.cluster_endpoint_public_access
  kms_key_administrators         = ["arn:aws:iam::123456789012:role/rolGitLab"]
  kms_key_enable_default_policy  = true

  eks_managed_node_group_defaults = {
    ami_type = var.ami_type
  }

  eks_managed_node_groups = {
    one = {
      name = var.nodeGroup

      instance_types = var.instanceTypes

      min_size     = var.minSize
      max_size     = var.maxSize
      desired_size = var.desiredSize
    }
  }
}

# Create an IAM policy for the ALB ingress
resource "aws_iam_policy" "worker_policy" {
  name        = "worker-policy"
  description = "Worker policy for the ALB Ingress"

  policy = file(var.iamALBIngressPolicyFileName)
}

resource "aws_iam_role_policy_attachment" "additional" {
  for_each = module.eks.eks_managed_node_groups

  policy_arn = aws_iam_policy.worker_policy.arn
  role       = each.value.iam_role_name
}

# Crete the Ingress using Helm chart from AWS
resource "helm_release" "ingress" {
  name       = "ingress"
  chart      = "aws-load-balancer-controller"
  repository = "https://aws.github.io/eks-charts"
  version    = "1.6.1"

  set {
    name  = "autoDiscoverAwsRegion"
    value = "true"
  }

  set {
    name  = "autoDiscoverAwsVpcID"
    value = "true"
  }

  set {
    name  = "clusterName"
    value = module.eks.cluster_name
  }
}

FILE: outputs.tf

output "cluster_endpoint" {
  description = "Endpoint for EKS control plane"
  value       = module.eks.cluster_endpoint
}

output "cluster_security_group_id" {
  description = "Security group ids attached to the cluster control plane"
  value       = module.eks.cluster_security_group_id
}

output "region" {
  description = "AWS region"
  value       = var.region
}

output "cluster_name" {
  description = "Kubernetes Cluster Name"
  value       = module.eks.cluster_name
}

FILE: providers.tf

provider "aws" {
  region = var.region

  shared_config_files      = ["~/.aws/config"]  
  profile                  = "oidc"
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    token                  = data.aws_eks_cluster_auth.cluster.token
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  }
}

FILE: variables.tf

variable "region" {
  type    = string
  default = "us-east-2"
}

variable "vpcName" {
  type    = string
  default = "vpcFromGitLab"
}

variable "vpcCIDR" {
  type    = string
  default = "10.0.0.0/16"
}

variable "privateSubnets" {
  type    = list(string)
  default = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
}

variable "publicSubnets" {
  type    = list(string)
  default = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]
}

variable "enable_nat_gateway" {
  type    = bool
  default = true
}

variable "single_nat_gateway" {
  type    = bool
  default = false
}

variable "enable_dns_hostnames" {
  type    = bool
  default = true
}

variable "cluster_name" {
  type    = string
  default = "eksFromGitLab"
}

variable "cluster_version" {
  type    = string
  default = "1.27"
}

variable "cluster_endpoint_public_access" {
  type    = bool
  default = true
}

variable "ami_type" {
  type    = string
  default = "AL2_x86_64"
}

variable "nodeGroup" {
  type    = string
  default = "ngFromGitLab"
}

variable "instanceTypes" {
  type    = list(string)
  default = ["t3.small"]
}

variable "minSize" {
  type    = number
  default = 1
}

variable "maxSize" {
  type    = number
  default = 3
}

variable "desiredSize" {
  type    = number
  default = 3
}


variable "iamALBIngressPolicyFileName" {
  type    = string
  default = "alb_ingress_iam_policy.json"
}

…and finally your .gitlab-ci.yml file.
FILE: .gitlab-ci.yml. NOTE: Remove line 38 if you don’t use it, don’t keep it with a comment or you’ll get an error in the pipeline.

stages:          
  - validate
  - plan
  - apply
  - destroy

image:
  name: hashicorp/terraform:1.6
  entrypoint:
    - "/usr/bin/env"
    - "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

variables:
  TF_DIR: ${CI_PROJECT_DIR}
  TF_STATE_NAME: "eks-tf-state"
  TF_ADDRESS: "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/terraform/state/${TF_STATE_NAME}"

default:
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com

  before_script:
    - terraform --version
    - cd ${TF_DIR}
    - terraform init
      -reconfigure 
      -backend-config="address=${TF_ADDRESS}" 
      -backend-config="lock_address=${TF_ADDRESS}/lock" 
      -backend-config="unlock_address=${TF_ADDRESS}/lock" 
      -backend-config="username=${TF_USERNAME}" 
      -backend-config="password=${TF_PASSWORD}" 
      -backend-config="lock_method=POST" 
      -backend-config="unlock_method=DELETE" 
      -backend-config="retry_wait_min=5"
    - mkdir -p ~/.aws
    - echo "${GITLAB_OIDC_TOKEN}" > /tmp/web_identity_token
    - # echo "${GITLAB_OIDC_TOKEN}" | base64 - If you want to see the token, reveal it this way and then use base64 --decode to get the real one
    - echo -e "[profile oidc]\nrole_arn=${ROLE_ARN}\nweb_identity_token_file=/tmp/web_identity_token" > ~/.aws/config

job-validate:
  stage: validate
  script:
    - terraform validate

job-plan:
  stage: plan
  script:
    - terraform plan -out "tf_plan"
  dependencies:
    - job-validate
  artifacts:
    paths:
      - tf_plan

job-apply:
  stage: apply
  script:
    - terraform apply "tf_plan"
  dependencies:
    - job-plan
  allow_failure: true

job-destroy:
  stage: destroy
  script:
    - terraform destroy  -auto-approve
  dependencies:
    #- job-plan
    #- job-apply
  when: manual

Push the code back to Gitlab (git add, git commit, git push) and once everything completes successfully, you’ll have a working EKS cluster after 10-15 mins.
If you choose to manage the EKS from GitLab, you can skip to the next section. If you decide to manage the cluster outside GitLab and you already have another IAM user added to the role rolGitLab as described above, then do the following. Replace account # in line 1.

aws eks update-kubeconfig --region us-east-2 --name eksFromGitLab --role-arn arn:aws:iam::123456789012:role/rolGitLab
aws eks list-clusters
eksctl get cluster --name=eksFromGitLab --region us-east-2
kubectl get svc

Manage EKS from GitLab

Generally, you would want this to be in another project. Our first project was to build the cluster, but our second project builds on top of that and manages the cluster. In addition, the second project also uses the CI/CD pipeline file .gitlab-ci.yml so in order to avoid a single complex project – just create a new project. I’ll name my new project – project1. Clone the project on your local machine using git clone.
Create an empty config file for the agent on your local machine. It has to be under this folder structure.

mkdir -p .gitlab/agents/eks-agent && touch .gitlab/agents/eks-agent/config.yaml

In my case the name of the agent is eks-agent. Use whatever name you want but follow DNS naming convention.
Push the local changes to GitLab.
On the left side in GitLab, go to Operate | Kubernetes clusters and then click Connect a cluster from the upper right corner.
Select the agent from the drop down and click Register.

You’ll get an access token and a bunch of helm commands to install the agent on the cluster. Copy and paste and run those commands.

helm repo add gitlab https://charts.gitlab.io
helm repo update
helm upgrade --install eks-agent gitlab/gitlab-agent \
    --namespace gitlab-agent-eks-agent \
    --create-namespace \
    --set image.tag=v16.9.0-rc2 \
    --set config.token=glagent-2r_uW63e1w_mkNzR_keU2hmaNhhcKauSpujg8OdLQG74wrF73g \
    --set config.kasAddress=wss://kas.gitlab.com

Refresh the GitLab screen after the helm chart has been installed and you’ll see your cluster connected.

Now you need to authorize the agent to access our project.
In the blank config.yaml file on the local machine under .gitlab/agents/eks-agent/ directory, add these lines. Change the path the project in line 3 (look at the URL, that’s your project name after gitlab.com) and change the manifest file if you want in line 6. You can also change the namespace in line 5 as well.

gitops:
  manifest_projects:
  - id: andreevkliment/project1
    default_namespace: default
    paths:
      - glob: 'manifest.yaml'

Then create the manifest file under the root of the project. Save it as manifest.yaml or however you referenced it in line 6 in the previous file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

If you push this code back to GitLab, the agent will execute it and you’ll (with kubect get pods) see the nginx pods running on your cluster.
Another way of managing the EKS cluster is using .gitlab-ci.yml file.
Create .gitlab-ci.yml with these lines. Replace your username, project and the agent name in line 7.

deploy:
  image:
    name: bitnami/kubectl:latest
    entrypoint: ['']
  script:
    - kubectl config get-contexts
    - kubectl config use-context andreevkliment/project1:eks-agent
    - kubectl get pods

If you push this back to GitLab, the pipeline will start and you’ll see the pods from the cluster in the logs. This example doesn’t execute manifest.yaml. You can have both ways of interacting with the cluster or you can use them separately.

AWS: EKS running out of IPs, increase pod density

Kliment Andreev — Tue, 06 Jun 2023 21:04:16 +0000

When I was doing some proof of concept using EKS, I’ve noticed that there is a limitation on how many pods can a node run and how EKS CNI assigns the subnet IPs. Apparently, a node can run a maximum of XX pods (based on type) which is very low, e.g. a t3.large instance with 2 CPUs and 8GB RAM can run only 35 pods. On top of that, the IPs that are assigned for the pods are used from the existing pool of IPs and if you have a /24 subnet for the EKS cluster, you’ll run out of IPs in no time. Here is an example of what I did and options how to solve this issue.

Option 1 – Calico 3rd party CNI

I’ll provision an EKS cluster in a /24 subnet with 3 public and 3 private subnets.

eksctl create cluster --name eksECIC --region us-east-2 --instance-types t3.large \
    --managed --vpc-cidr 192.168.100.0/24 --node-private-networking --version 1.24 --without-nodegroup

I am using the Ohio region which has 3 availability zones. The public and private subnets (total of 6) have /27 subnet which is 32 IPs. Not all of them are usable of course. Let’s create a node group with 3 nodes, and because t3.large is a Nitro instance, we can set the max number of pods to be 110.

eksctl create nodegroup --cluster eksECIC --name old-nodegroup --nodes 3 \
    --node-type t3.large --node-private-networking --managed --max-pods-per-node 200

These are the available 31 IPs that I have in the private subnets.

Let’s provision a small deployment with a single pod and 10 replicas and a network load balancer.

kubectl apply -f demo.yaml

…and this is the deployment file demo.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
spec:
  replicas: 10
  selector:
    matchLabels:
      run: demo
  template:
    metadata:
      labels:
        run: demo
    spec:
      containers:
      - name: demo
        image: klimenta/serverip
        ports:
        - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: loadbalancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
spec:
  ports:
    - port: 80
      targetPort: 3000
      protocol: TCP
  type: LoadBalancer
  selector:
    run: demo

If you look at the available address now, there are 7. So from 31 to 7 with just 10 pods. Check this guy’s blog to see what’s going on.

Let’s scale up the cluster to 100 pods.

kubectl scale --replicas=100 deployment/demo

After a minute or two, check the running pods.

kubectl get pods --field-selector=status.phase=Running | wc -l
69

The console will show 0 available subnets.

If you check the pods with kubectl get pods, you’ll see the remaining containers have a ContainerCreating status.
And if you check one of them, you’ll see this message.

kubectl describe pod demo-698f6fc958-z8smb
....
d9b65ccfeccb7e9e7433ec61fef78faf83d1bae": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  2m29s (x17 over 5m59s)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "edfa987c3eddf415068691f914d740fa9ca9ca6ffdfb8db44d1340d913ece0b0": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container

Get the load balancer service.

kubectl get svc
NAME           TYPE           CLUSTER-IP      EXTERNAL-IP                                                                     PORT(S)        AGE
kubernetes     ClusterIP      10.100.0.1                                                                                443/TCP        37m
loadbalancer   LoadBalancer   10.100.25.241   a105789cd3a264ac596c367fce463640-80d2b88acdc17771.elb.us-east-2.amazonaws.com   80:32396/TCP   17m

If you go to a105789cd3a264…amazonaws.com URL, you’ll see the load balancer hitting different pods. Wait for 2-3 minutes if you see that the page can’t be opened. It takes time for DNS to propagate.

Let’s delete the deployment and the old node group.

kubectl delete -f demo.yaml
eksctl delete nodegroup --name old-nodegroup --cluster eksECIC

At this point, we still have the EKS cluster, the etcd database, networking etc, we just don’t have the nodes. Now it’s time to remove the EKS CNI, install Calico and add nodes that will use Calico.
Installation is very simple. Remove the AWS CNI.

kubectl delete daemonset -n kube-system aws-node

Check the pods in all namespaces.

kubectl get pods --all-namespaces
NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
kube-system   coredns-5c5677bc78-2k88b   0/1     Pending   0          4m
kube-system   coredns-5c5677bc78-2zz2n   0/1     Pending   0          4m

You will see coredns in pending state, it can’t be deployed anywhere.
Deploy Calico.

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/tigera-operator.yaml

…keep going…

kubectl create -f - <
Deploy the new node group.
eksctl create nodegroup --cluster eksECIC --name new-nodegroup --nodes 3 \
--node-type t3.large --node-private-networking --managed --max-pods-per-node 200

Let’s deploy the same demo deployment now.
kubectl apply -f demo.yaml

…scale it up to 601 pods.
kubectl scale --replicas=601 deployment/demo

Wait 2-3 mins and check the running ones.
kubectl get pods --field-selector=status.phase=Running | wc -l

There will be 585 pods running, 195 per node. Much, much better. This time the remaining containers will be in Pending state.
kubectl describe pod demo-698f6fc958-smlbq
....
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 3m27s (x3 over 13m) default-scheduler 0/3 nodes are available: 3 Too many pods. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.

Option 2 – CNI Custom networking and prefix assignment mode
In this case, we’ll keep the AWS CNI network plugin, but we’ll use what is called CNI Custom networking, meaning we’ll use another CIDR attached to the VPC where pods will run. NOTE: The nodes will still run in the 192.168/24 subnets, it’s just the pods that will run in a much bigger IP space. In addition, this CNI Custom networking won’t solve the density of the pods that we can run on the nodes. We’ll need something called prefix assignment mode that will allow us to run much more pods on a node than usual. See this and this links for more information. For more info regarding prefix assignment mode check out this link. The solution described below is much more complicated than if you just use Calico.

Let’s provision a cluster with the same VPC CIDR and no nodegroups.
eksctl create cluster --name eksECIC --region us-east-2 --instance-types t3.large \
--managed --vpc-cidr 192.168.100.0/24 --node-private-networking --version 1.24 --without-nodegroup

Add a new CIDR and associate it with the VPC. I am using the console but you can use aws ec2 associate-vpc-cidr-block command.Go to the VPC where the EKS cluster resides. From the Actions button, select Edit CIDRs and then click the Add new IPv4 CIDR button.

I’ll add 100.64.0.0/16 CIDR and then I’ll create 3 /19 subnets.

These 3 subnets will be where the pods will run. I also want them to initiate connections to Internet, so I’ll modify the route table and add the Internet Gateway. The three new subnets 100.64/19 that we created have the same route table.

Just add the route to 0.0.0.0/0 to go over Internet Gateway.

Let’s configure the custom networking on the cluster.
kubectl set env daemonset aws-node -n kube-system AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true

Get the EKS cluster security group.
sec_grp=$(aws eks describe-cluster --name eksECIC --query cluster.resourcesVpcConfig.clusterSecurityGroupId --output text)
echo $sec_grp

For each of the subnets create a custom resource. Define the three 100.64 subnet-ids as variables.
export sub1=subnet-0f3ec5d75caf1b981
export sub2=subnet-0fc39edc5a1ce91c0
export sub3=subnet-0bd6bdfd5543aa086

cat >us-east-2a.yaml <
2nd subnet.
cat >us-east-2b.yaml <
3rd subnet.
cat >us-east-2c.yaml <
NOTE:Make sure you name the metadata in line 5 as I did based on your region and AZ.

Deploy the custom resources.
kubectl apply -f us-east-2a.yaml
kubectl apply -f us-east-2b.yaml
kubectl apply -f us-east-2c.yaml

Confirm it looks good.
kubectl get ENIConfigs
NAME AGE
us-east-2a 16s
us-east-2b 12s
us-east-2c 8s

Update your aws-node DaemonSet to automatically apply the ENIConfig for an Availability Zone to any new Amazon EC2 nodes created in your cluster.
kubectl set env daemonset aws-node -n kube-system ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone

Enable prefix assignment mode. This will allow much more pods per node.
kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

Another change.
kubectl set env ds aws-node -n kube-system WARM_PREFIX_TARGET=1

With the default setting, WARM_PREFIX_TARGET will allocate one additional complete (/28) prefix even if the existing prefix is used by only one pod. If the ENI does not have enough space to assign a prefix, a new ENI is generated.

Provision the node group.
eksctl create nodegroup --cluster eksECIC --name my-nodegroup --nodes 3 \
--node-type t3.large --node-private-networking --managed --max-pods-per-node 200

Check the nodes, they are running in the 192.168/24 original VPC CIDR. See the IP in the names of the nodes.
kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-100-122.us-east-2.compute.internal Ready 7m39s v1.24.13-eks-0a21954
ip-192-168-100-155.us-east-2.compute.internal Ready 7m39s v1.24.13-eks-0a21954
ip-192-168-100-167.us-east-2.compute.internal Ready 7m37s v1.24.13-eks-0a21954

Check the pods.
kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system aws-node-txwrd 1/1 Running 0 9m4s 192.168.100.167 ip-192-168-100-167.us-east-2.compute.internal
kube-system aws-node-vdmv7 1/1 Running 0 9m6s 192.168.100.155 ip-192-168-100-155.us-east-2.compute.internal
kube-system aws-node-xss7c 1/1 Running 0 9m6s 192.168.100.122 ip-192-168-100-122.us-east-2.compute.internal
kube-system coredns-5c5677bc78-2lkcx 1/1 Running 0 80m 100.64.76.225 ip-192-168-100-122.us-east-2.compute.internal
kube-system coredns-5c5677bc78-mj6wr 1/1 Running 0 80m 100.64.76.224 ip-192-168-100-122.us-east-2.compute.internal
kube-system kube-proxy-lt7v4 1/1 Running 0 9m6s 192.168.100.122 ip-192-168-100-122.us-east-2.compute.internal
kube-system kube-proxy-m8pf6 1/1 Running 0 9m4s 192.168.100.167 ip-192-168-100-167.us-east-2.compute.internal
kube-system kube-proxy-nlvw7 1/1 Running 0 9m6s 192.168.100.155 ip-192-168-100-155.us-east-2.compute.internal

Look at coredns pods. They are running in the new CIDR.

Deploy the demo.
kubectl apply -f demo.yaml

Get the load balancer service.
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.100.0.1 443/TCP 83m
loadbalancer LoadBalancer 10.100.206.187 a7ad3ecf1d52e4fc58dbbdc550237009-ac4778e11af2c0ad.elb.us-east-2.amazonaws.com 80:30230/TCP 45s

If you go to a7ad3ecf1d…amazonaws.com URL, you’ll see the load balancer hitting different pods. Wait for 2-3 minutes if you see that the page can’t be opened. It takes time for DNS to propagate.

The pods are running in the 100.64 subnet.

Scale up to 601 pods.
kubectl scale --replicas=601 deployment/demo

Wait 2-3 mins and check the running ones.
kubectl get pods --field-selector=status.phase=Running | wc -l

I got 593 total pods. Slightly better than 585 with Calico. The ones that are in pending state show up an error that there are no more resources.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 3m49s default-scheduler 0/3 nodes are available: 3 Too many pods. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.

Kubernetes: Monitoring Node.js application with Prometheus and Grafana, Helm charts in AWS EKS

Kliment Andreev — Tue, 24 Jan 2023 16:01:05 +0000

Recently, we were migrating some Node.js applications to a different AWS account and we had a need to monitor the application with Prometheus and Grafana. In this post, I’ll explain how to install both Prometheus and Grafana using Helm charts, then we’ll create a simple Express.js web app, create a Helm chart for it and deploy it on an EKS cluster in AWS. I am using an EKS cluster, but you can use any Kubernetes cluster that you can manage it with kubectl. You might have to make some changes though in how you access the Load Balancer for Grafana.
Requirements:

Create EKS cluster

This is optional, if you already have a running Kubernetes cluster, you can skip this step.

eksctl create cluster --name eksTest \
--region us-east-2 \
--instance-types t3.medium \
--nodes 2 \
--managed \
--version 1.22

This command will create a 2 node cluster, with t3.medium instances in Ohio region using Amazon Linux 2 image. It will take about 15 minutes.
Verify that everything looks OK.

kubectl get nodes
NAME                                           STATUS   ROLES    AGE     VERSION
ip-192-168-10-4.us-east-2.compute.internal     Ready       8m54s   v1.22.15-eks-fb459a0
ip-192-168-74-160.us-east-2.compute.internal   Ready       8m51s   v1.22.15-eks-fb459a0

Install Prometheus and Grafana

Add the Helm repos for Prometheus, Grafana and update any changes.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

We’ll create a separate namespace for Prometheus and install the chart there.

kubectl create namespace prometheus
helm install prometheus prometheus-community/prometheus \
    --namespace prometheus \
    --set alertmanager.persistentVolume.storageClass="gp2" \
    --set server.persistentVolume.storageClass="gp2"

Create this file. It tells Grafana what to use as a source (url). It’s the Prometheus endpoint service that we just installed.

cat << EOF > grafana.yaml
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-server.prometheus.svc.cluster.local
      access: proxy
      isDefault: true
EOF

Now, let’s create a namespace for Grafana and then install Grafana in that namespace. Change the admin password. In my case it’s Password1$.

kubectl create namespace grafana
helm install grafana grafana/grafana \
    --namespace grafana \
    --set persistence.storageClassName="gp2" \
    --set persistence.enabled=true \
    --set adminPassword='Password1$' \
    --values grafana.yaml \
    --set service.type=LoadBalancer

You’ll get something like this on the screen when the install ends.

export SERVICE_IP=$(kubectl get svc --namespace grafana grafana -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

That’s your load balancer IP so you can access Grafana. If you use EKS, your IP is different.

kubectl get services -n grafana
NAME      TYPE           CLUSTER-IP     EXTERNAL-IP                                                               PORT(S)        AGE
grafana   LoadBalancer   10.100.20.53   a074996f790b74793a74bec584c04460-2034633908.us-east-2.elb.amazonaws.com   80:31916/TCP   3m21s

The IP is the gibberish URL that ends with amazonaws.com. If you go to that URL, you can log as admin and your password.

Go to Dashboards, click on the New button on the right and click Import.
Add the following dashboard IDs (one by one then click Load) and choose Prometheus as a source. The IDs are 3119 (Kubernetes cluster), 6417 (pods) and 11159 (Node.js). For the first dashboard, you’ll have some metrics. The Node.js dashboard is empty and our goal is to get the metrics from the application.

Node.js application

Create an empty folder and add this file.

mkdir mynodeapp
cd mynodeapp
cat << EOF > package.json
{
  "name": "mynodeapp",
  "version": "1.0.0",
  "description": "Node.js on Docker",
  "author": "Kliment Andreev ",
  "main": "server.js",
  "scripts": {
    "start": "node server.js"
  },
  "dependencies": {
    "express": "^4.16.1",
    "prom-client": "^14.1.1"
  }
}
EOF

I use Node v18.12. Install the dependencies.

npm install

This is the simple web application. It listens on port 3000.

cat << "EOF" > server.js
'use strict'
const express = require('express')
const prom = require('prom-client')

const collectDefaultMetrics = prom.collectDefaultMetrics();

const app = express()
const port = process.env.PORT || 3000

app.get('/', (req, res, next) => {
  res.send('Hello World!');
});

app.get('/metrics', async (req, res) => {
  try {
	  res.set('Content-Type', prom.register.contentType);
	  res.end(await prom.register.metrics());
  } 
  catch (ex) {
	  res.status(500).end(ex);
  }
});

const server = app.listen(port, () => {
  console.log(`mynodeapp listening on port ${port}!`)
})
EOF

The app is a super simple web app that prints Hello World! The important part is the /metrics router. That’s how you get the metrics from the app. When we created the package.json file, we specify the prom-client as dependency. That’s the part that collects the metrics. Go to this link for more info.
Start the app with node server.js.

node server.js
mynodeapp listening on port 3000!

Go to the localhost or the server IP where this app is running and you’ll see Hello World! printed out.

And if you go to the /metrics URL, you’ll see this text. That’s what Prometheus expects.

Now that we know the app is working, let’s create a Docker image.

Docker image and container

Create this Dockerfile.

cat << EOF > Dockerfile
FROM node:18-alpine

# Create app directory
WORKDIR /app

# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm@5+)
COPY package*.json ./

RUN npm install
# If you are building your code for production
# RUN npm install --only=production

# Bundle app source
COPY server.js /app

EXPOSE 3000
CMD [ "npm", "start", "server.js" ]
EOF

Create a Docker image based on the Dockerfile above. Replace the username with your Docker Hub username.

docker build -t /mynodeapp:latest .

Run the container from the image.

docker run -p 3000:3000 /mynodeapp:latest

> mynodeapp@1.0.0 start
> node server.js server.js

mynodeapp listening on port 3000!

Use the same test as before by going to the localhost or IP URL on port 3000.
If everything is OK, publish the image on Dockerhub.

docker login
docker push /mynodeapp

Helm chart

Now that we have the Docker image, we’ll create a Helm chart so we can deploy the web app to the Kubernetes cluster.
Create the necessary structure.

helm create mynodeapp
Creating mynodeapp

Go to the sub-directory mynodeapp and edit the values.yaml file first. Look for the image key first and replace it so it looks like this. Make sure you replace the with your Docker Hub username.

image:
  repository: /mynodeapp
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: "latest"

Then look for podAnnotations: {} line and replace it so it looks like this.

podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/path: "/metrics"
  prometheus.io/port: "3000"

Look for the service parameter and change it so it looks like this.

service:
  type: LoadBalancer
  port: 80
  targetPort: 3000
  name: mynodeapp-service

Look for the resources key and uncomment the defaults.

resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
limits:
  cpu: 100m
  memory: 128Mi
requests:
  cpu: 100m
  memory: 128Mi

Edit the templates/deployment.yaml file and find these lines.

ports:
  - name: http
    containerPort: {{ .Values.service.port }}

Replace the containerPort to look like this.

containerPort: 3000

Go back to the root of the mynodeapp folder and check the chart.

helm install mynodeapp --generate-name
NAME: mynodeapp-1674575241
LAST DEPLOYED: Tue Jan 24 10:47:22 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
     NOTE: It may take a few minutes for the LoadBalancer IP to be available.
           You can watch the status of by running 'kubectl get --namespace default svc -w mynodeapp-1674575241'
  export SERVICE_IP=$(kubectl get svc --namespace default mynodeapp-1674575241 --template "{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}")
  echo http://$SERVICE_IP:80

Execute the export SERVICE_IP command above and echo the IP. That’s your URL. If you go to that URL you’ll see the Hello World! greetings.
What we care is the pod. Get the pod running. It’s in the default namespace.

kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
mynodeapp-1674575241-5ff9d7469d-pddcz   1/1     Running   0          3m

Get the annotations. Replace with your pod name.

kubectl get pod mynodeapp-1674575241-5ff9d7469d-pddcz -o jsonpath='{.metadata.annotations}'
{"kubernetes.io/psp":"eks.privileged","prometheus.io/path":"/metrics","prometheus.io/port":"3000","prometheus.io/scrape":"true"}

You can see the prometheus annotations.
And if you go to Grafana and open up the NodeJS dash you’ll see the metrics.

AWS: EKS monitoring and logging

Kliment Andreev — Mon, 03 Oct 2022 23:09:44 +0000

In this post I’ll explain several things related to EKS monitoring and logging.

In order to do that, we’ll need the following CLI tools.

You can easily find how to install and configure these tools for various OSes.

Create the EKS cluster

I’ll create a managed cluster called eksWordPress in us-east-2 region with two t3.medium nodes.

eksctl create cluster --name eksWordPress --region us-east-2 --instance-types t3.medium --nodes 2 --managed --version 1.22

If you get an error that the last supported version is 1.21, update eksctl tool. The cluster creation took about 20 mins for me.

Send EKS logs to CloudWatch

Once the cluster was created and up and running, enabling the EKS control plane logs is easy. There are 5 log types (API server, Audit, Authenticator, Controller Manager and Scheduler logs). The logged info is a lot, so it’s up to you what you want to log. There is no way to choose what to log, e.g. info, warning, errors. Everything is logged. To enable logging from the console, go to the EKS cluster, select it, click the Logging tab and then click on the Manage logging button.

If you want to do it from a CLI, type this command and specify what types of logs you want sent to CloudWatch. Specify the region and the cluster name.

aws eks update-cluster-config \
    --region  \
    --name  \
    --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'

In addition, because we provisioned the cluster with the eksctl tool, we can also enable the logs using eksctl.

eksctl utils update-cluster-logging --enable-types all --cluster  --approve

Or for certain types, use…

eksctl utils update-cluster-logging --enable-types  --cluster  --approve

You can use the same types as in the AWS CLI command above (api, audit, scheduler…).
If you go to CloudWatch and then the Log groups, you’ll see the log group with the name of the EKS cluster (/aws/eks/eksWordPress/cluster).
Make sure you change the retention from Never to some value. You probably don’t want to keep these logs indefinitely. If you look at the logs streams, you’ll see that there is a lot of info there. Most of the stuff is useless.

Send the logs to OpenSearch cluster

Let’s create a public OpenSearch cluster with anonymous access that only you can access. Replace the IP at the end of the statement with your IP address. It takes less than 10 mins for the OpenSearch cluster to be provisioned.

aws opensearch create-domain --domain-name oswordpress --engine-version OpenSearch_1.3 \
    --auto-tune-options DesiredState="ENABLED" --cluster-config InstanceType=t3.small.search,InstanceCount=2 \
    --ebs-options EBSEnabled=true,VolumeType=gp3,VolumeSize=10,Iops=3000 \
    --access-policies '{"Version": "2012-10-17", "Statement": [{"Action": "es:*", "Principal":"*","Effect": "Allow", "Condition": {"IpAddress":{"aws:SourceIp":["2.18.2.19/32"]}}}]}'

Once the cluster is provisioned you can get the public URL with:

echo `aws es describe-elasticsearch-domain --domain-name oswordpress --output text --query "DomainStatus.Endpoint"`\\_dashboards

You can also get the dashboard URL from the cluster settings as well.

Create the following IAM policy. Save the policy below as policy-file.json but change the account_no in line 9 to match your AWS account.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "es:*"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:es:us-east-2::domain/oswordpress/*"
        }
    ]
}

And then create the IAM policy.

aws iam create-policy --policy-name polOpenSearch --policy-document file://policy-file.json

Create the role. Save the policy below as policy-trust.json.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

And then create the role.

aws iam create-role --role-name rolOpenSearch --assume-role-policy file://policy-trust.json

Finally, attach the policy to the role.

aws iam put-role-policy --role-name rolOpenSearch --policy-name polOpenSearch --policy-document file://policy-file.json

Go to CloudWatch, select the Log group, from the Actions button select Subscription filters and then Create Amazon OpenSearch service subscription filter.

Choose the OpenSearch cluster that we created, the log format is JSON, subscription filter pattern is ” “ for all events and type all subscription filter name or whatever you want to name this pattern. Click Start streaming button.
Now, go to OpenSearch dashboard and from the hamburger menu in the upper left corner, click on Visualize. In the middle of the screen, you’ll be prompted to create an index pattern. Type cwl* and click Next step.

Select @timestamp from the drop down menu and click on Create index pattern. You’ll see the fields and the index.

Click on the Discover from the hamburger menu and you’ll see your data there. You can use queries to search your data, but that’s out of the scope of this post.

Container Insights and FluentBit

Container Insights is a CloudWatch agent that we’ll install on the EKS cluster. It will collect all kinds of metrics and then FluentBit as a log forwarder will ship those logs to CloudWatch. See this link for more info.
Let’s do the prerequisites work first.
Get the nodes. Type the first line only, the rest is my output.

kubectl get nodes                                                                                                                                                               
NAME                                           STATUS   ROLES    AGE     VERSION
ip-192-168-56-250.us-east-2.compute.internal   Ready       3h39m   v1.22.12-eks-ba74326
ip-192-168-66-251.us-east-2.compute.internal   Ready       3h39m   v1.22.12-eks-ba74326

Get the instance ID from any of the instances, it doesn’t matter. Replace the name accordingly after Values=.

aws ec2 describe-instances --filters 'Name=private-dns-name,Values=ip-192-168-56-250.us-east-2.compute.internal' \
    --output text --query 'Reservations[*].Instances[*].InstanceId'
i-05adaa0a823a8549a

Once you have the instance ID of any node, get the Arn of the IAM role that’s the attached to that node.

aws ec2 describe-instances --region us-east-2 --instance-ids i-05adaa0a823a8549a --query 'Reservations[*].Instances[*].IamInstanceProfile.Id'
[
    [
        "AIPAXFRN6SYD75I5N4BUN"
    ]
]

Get the role name. Replace the ID (AIPAX…) with your value above.

aws iam list-instance-profiles --query 'InstanceProfiles[?InstanceProfileId==`AIPAXFRN6SYD75I5N4BUN`].Roles[*].RoleName' 
[
    [
        "eksctl-eksWordPress-nodegroup-ng-NodeInstanceRole-LWJZESQ468Y2"
    ]
]

We want to add a policy that allows nodes to write to CloudWatch logs group. Replace the –role-name with the value that you got above.

aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy --role-name eksctl-eksWordPress-nodegroup-ng-NodeInstanceRole-LWJZESQ468Y2

oidc_id=$(aws eks describe-cluster --name eksWordPress --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)

Type this line and see if there is any output. If no output, execute the command after. If yes-output, then do nothing.

aws iam list-open-id-connect-providers | grep $oidc_id

If no output, type this.

eksctl utils associate-iam-oidc-provider --cluster eksWordPress --approve

Then install CloudWatch Container Insights and FluentBit. Change the CLUSTER_NAME and REGION in lines 1 and 2.

ClusterName=
RegionName=
FluentBitHttpPort='2020'
FluentBitReadFromHead='Off'
[[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
[[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluent-bit-quickstart.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/;s/{{region_name}}/'${RegionName}'/;s/{{http_server_toggle}}/"'${FluentBitHttpServer}'"/;s/{{http_server_port}}/"'${FluentBitHttpPort}'"/;s/{{read_from_head}}/"'${FluentBitReadFromHead}'"/;s/{{read_from_tail}}/"'${FluentBitReadFromTail}'"/' | kubectl apply -f -

Download a config map and edit it the file called cwagent-configmap.yaml.

curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-configmap.yaml

In line 11, change the variable so it points to your cluster. In my case it looks like this.

"cluster_name": "{{eksWordPress}}",

Save the changes and apply the config.

kubectl apply -f cwagent-configmap.yaml

Then deploy it as a DaemonSet.

kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-daemonset.yaml

Verify that it’s running. Type the first line only. The rest is my output.

kubectl get pods -n amazon-cloudwatch
NAME                     READY   STATUS    RESTARTS   AGE
cloudwatch-agent-dfkzv   1/1     Running   0          18m
cloudwatch-agent-nfnf7   1/1     Running   0          18m
fluent-bit-8vd2n         1/1     Running   0          18m
fluent-bit-tvtpv         1/1     Running   0          18m

Check the logs.

kubectl logs   -n amazon-cloudwatch

Or in my case…

kubectl logs cloudwatch-agent-dfkzv -n amazon-cloudwatch
[2022/10/03 13:28:56] [ info] [output:cloudwatch_logs:cloudwatch_logs.2] Created log stream ip-192-168-73-121.us-east-2.compute.internal.host.messages
[2022/10/03 13:29:06] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-73-121.us-east-2.compute.internal-application.var.log.containers.cloudwatch-agent-7fvxd_amazon-cloudwatch_cloudwatch-agent-5409bed9d4733a51602e4bc0cccde5e5580eb3e9282cd5cc4c1a4f2d2e28e8ea.log in log group /aws/containerinsights/eksWordPress/application
[2022/10/03 13:29:06] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Created log stream ip-192-168-73-121.us-east-2.compute.internal-application.var.log.containers.cloudwatch-agent-7fvxd_amazon-cloudwatch_cloudwatch-agent-5409bed9d4733a51602e4bc0cccde5e5580eb3e9282cd5cc4c1a4f2d2e28e8ea.log

If you go to CloudWatch now you’ll see 4 new log groups.

Prometheus and Grafana

The Kubernetes API can also be monitored using Prometheus. We’ll install it using helm.
Create a namespace first.

kubectl create namespace prometheus

Add the Prometheus repo.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Deploy Prometheus.

helm upgrade -i prometheus prometheus-community/prometheus --namespace prometheus --set alertmanager.persistentVolume.storageClass="gp2",server.persistentVolume.storageClass="gp2"

Check if everything is OK.

kubectl get pods -n prometheus
NAME                                            READY   STATUS    RESTARTS   AGE
prometheus-alertmanager-5c57cc6945-v9lcb        2/2     Running   0          4m55s
prometheus-kube-state-metrics-77ddf69b4-jsgrp   1/1     Running   0          4m55s
prometheus-node-exporter-68dk9                  1/1     Running   0          4m55s
prometheus-node-exporter-m98xk                  1/1     Running   0          4m55s
prometheus-pushgateway-ff89cc976-4sfhl          1/1     Running   0          4m55s
prometheus-server-6c99667b9b-mpw97              2/2     Running   0          4m55s

Type this command and open up a browser and go to localhost:9090.

kubectl --namespace=prometheus port-forward deploy/prometheus-server 9090

You can CTRL-C out of the command prompt once you verify it’s OK.
Let’s add the Grafana repo first.

helm repo add grafana https://grafana.github.io/helm-charts

Copy, paste and save this as grafana.yaml.

datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-server.prometheus.svc.cluster.local
      access: proxy
      isDefault: true

Create a namespace for grafana.

kubectl create namespace grafana

Deploy using helm. Look at the adminPassword parameter. Change it to something else. In my case it’s admin123!.

helm install grafana grafana/grafana --namespace grafana --set persistence.storageClassName="gp2" --set persistence.enabled=true --set adminPassword='admin123!' --values grafana.yaml --set service.type=LoadBalancer

Check if everything is OK.

kubectl get all -n grafana

Get the URL of the classic LB that was just created.

export ELB=$(kubectl get svc -n grafana grafana -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
echo "http://$ELB"

Go to that URL and you’ll see the Grafana URL.
You should see the Grafana landing page. Log as admin and the password you specified when pods were deployed.
In Grafana create a new dashboard and Import the dashboard with ID 3119. Choose Prometheus as source.

For pods monitoring use the same method but this time specify 6417 as a dashboard ID. This is how it looks like under my account.

WordPress

Let’s deploy WordPress.

kubectl create namespace wordpress

Add the Bitnami helm chart.

helm repo add bitnami https://charts.bitnami.com/bitnami

Deploy WordPress in its own namespace

helm -n wordpress install understood-zebu bitnami/wordpress

Wait for 3-4 mins and do this command. This is your ELB, get the URL, something like *zdasdfa*.elb.amazonaws.com

kubectl get svc --namespace wordpress -w understood-zebu-wordpress

The username is user and get the password with:

echo Password: $(kubectl get secret --namespace wordpress understood-zebu-wordpress -o jsonpath="{.data.wordpress-password}" | base64 -d)

If you go to the ELB URL, you’ll hit WordPress main page, if you want to login, add /wp-login.php as suffix to the above URL.
Go to CloudWatch and check the /aws/containerinsights/eksWordPress/application log group. You’ll see a bunch of references for WordPress. You can ship those to OpenSearch if you want and alert on errors or whatever you want to do.

Delete EKS and OpenSearch cluster

Detach the policy, delete the daemonset and delete the EKS cluster.

aws iam detach-role-policy --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy --role-name eksctl-eksWordPress-nodegroup-ng-NodeInstanceRole-LWJZESQ468Y2
ClusterName=
RegionName=
FluentBitHttpPort='2020'
FluentBitReadFromHead='Off'
[[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
[[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluent-bit-quickstart.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/;s/{{region_name}}/'${LogRegion}'/;s/{{http_server_toggle}}/"'${FluentBitHttpServer}'"/;s/{{http_server_port}}/"'${FluentBitHttpPort}'"/;s/{{read_from_head}}/"'${FluentBitReadFromHead}'"/;s/{{read_from_tail}}/"'${FluentBitReadFromTail}'"/' | kubectl delete -f -
eksctl delete cluster --name eksWordPress --region=us-east-2

Delete policy.

aws iam delete-policy --policy-arn arn:aws:iam::492943873543:policy/polOpenSearch

Delete CloudWatch log groups.

EKS_CLUSTER=eksWordPress
aws logs delete-log-group --log-group-name "/aws/containerinsights/$EKS_CLUSTER/application"
aws logs delete-log-group --log-group-name "/aws/containerinsights/$EKS_CLUSTER/dataplane"
aws logs delete-log-group --log-group-name "/aws/containerinsights/$EKS_CLUSTER/host"
aws logs delete-log-group --log-group-name "/aws/containerinsights/$EKS_CLUSTER/performance"
aws logs delete-log-group --log-group-name "/aws/eks/$EKS_CLUSTER/cluster"

Delete the OpenSearch cluster.

aws opensearch delete-domain --domain-name oswordpress