DevOps – Blog of Kliment Andreev – A place so I won't forget things

AWS: Install EKS cluster and Ingress Controller using Terraform and GitLab using OIDC and web identity

Kliment Andreev — Tue, 21 Nov 2023 18:39:16 +0000

In this post, I’ll explain how to create an EKS cluster in AWS and add an Ingress Controller. We’ll use GitLab as a repo and execute the CI/CD pipeline from there.
For this you’ll need:

GitLab and AWS account
Terraform, AWS CLI, helm installed

AWS

From the IAM console, create a new Identity Provider in AWS and use https://gitlab.com for both Provider URL and Audience. For self hosted GitLab use gitlab.domain.com or whatever is your GitLab URL.
Click on Get thumbprint before clicking on Add provider.

In the IAM console, create a role. I’ll name the role rolGitLab, name it whatever you like. When you create the policy, choose Custom trust policy.

Now, here is what you have to decide. If you want to manage the cluster from GitLab only, you can ommit lines 16-23 where I allow the IAM user called gitlab to assume this role. But if you want some IAM user to manage the cluster using aws cli or kubectl, add the ARN of that IAM user in line 20. Add AdministratorAccess to that gitlab user but you can fine-tune those policies later. If you want to manage the cluster from GitLab, you have to register an agent and provide a config file. You can read how to do that here.
Replace the default custom trust policy with this JSON.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Principal": {
				"Federated": "arn:aws:iam::123456789012:oidc-provider/gitlab.com"
			},
			"Action": "sts:AssumeRoleWithWebIdentity",
			"Condition": {
				"StringLike": {
					"gitlab.com:sub": "project_path:*/eks-gitlab:ref_type:branch:ref:main"
				}
			}
		},
		{
			"Sid": "",
			"Effect": "Allow",
			"Principal": {
				"AWS": "arn:aws:iam::123456789012:user/gitlab"
			},
			"Action": "sts:AssumeRole"
		}
	]
}

In line 7 and 20, change the 123456789012 with your account number and in line 7 change gitlab.com with however you named the Identity Provider (IdP). In line 12 replace eks-gitlab with how you plan to name your GitLab project. In line 20 change the gitlab user to however you named your IAM user to access the EKS cluster. Click Next and when asked for a permission policy, choose AdministratorAccess. Best practice is to limit access to only what’s needed, but EKS has so many policies, it will take a lot to list them all.
In case you created the gitlab IAM user, attach the AdministratorPolicy (fine tune later) and this inline policy. Change the ARN in line 7. Name this inline policy something like polAssumeGitLabRole.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::123456789012:role/rolGitLab"
        }
    ]
}

GitLab

Create a new blank project in Gitlab with the default settings. In my case the project name is called eks-gitlab.

Go to Settings on the left, then CI/CD and then Variables. Add these two variables.

AWS_PROFILE and a value of oidc. Yes, the value is oidc. Just type these 4 characters for the value.
ROLE_ARN and a value of the role ARN from the role you just created for rolGitLab

Create a .gitlab-ci.yml file with the following content. Change the domain for gitlab.com at line 7 if you use self-hosted GitLab.

assume role:
  image: 
    name: amazon/aws-cli:latest
    entrypoint: [""]
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com
  before_script:
    - mkdir -p ~/.aws
    - echo "${GITLAB_OIDC_TOKEN}" > /tmp/web_identity_token
    - echo -e "[profile oidc]\nrole_arn=${ROLE_ARN}\nweb_identity_token_file=/tmp/web_identity_token" > ~/.aws/config
  script:
      - aws sts get-caller-identity
      - aws s3 ls

By the time you commit changes, the pipeline will start and if everything is OK, you’ll see your S3 buckets in the logs of the job if you go to Build | Jobs.

One more thing related to Terraform state as we’ll keep the state in GitLab is that we have to create an Access Token. If you have a self-hosted or paid GitLab account, create an Access Token from the Project | Settings | Access Tokens.
If you have a free GitLab account, you have to create a Personal Access Token and you do that if you click on your avatar in the upper left corner, then select Edit Profile and then click Access Tokens on the left. Click Add new token, name it GITLAB_PAT and select API only. Click Create personal access token. Once the token has been created click on the eye icon where it says Your new personal access token and copy & paste somewhere safe.

Go to the project settings and add these 2 variables.

TF_USERNAME which is your username, click on the avatar, look at the @ under your name, your username is without the @ sign
TF_PASSWORD which is the value of your personal access token you just created

Local machine

Now that we know the pipeline is working properly, let’s clone the project on our local machine.
If you plan to manage the cluster from your local machine, now it’s time to configure the gitlab IAM user on your local machine using access keys.
You need to have a valid key in order to clone your private repo.
Upload your .pub key to GitLab by clicking your avatar, then Edit Profile, click SSH Keys and then Add new key.
Finally, check the connection with the following command. It doesn’t matter what your local Linux user is, just use git@gitlab.com as the username@hostname. Type as is.

ssh -T git@gitlab.com
Welcome to GitLab, @[your_gitlab_username_here]

Once you clone the repo add the following files locally.
FILE: .gitignore

# Local .terraform directories
**/.terraform/*

# .tfstate files
*.tfstate
*.tfstate.*

# Crash log files
crash.log
crash.*.log

# Exclude all .tfvars files, which are likely to contain sensitive data, such as
# password, private keys, and other secrets. These should not be part of version 
# control as they are data points which are potentially sensitive and subject 
# to change depending on the environment.
*.tfvars
*.tfvars.json

# Ignore override files as they are usually used to override resources locally and so
# are not checked in
override.tf
override.tf.json
*_override.tf
*_override.tf.json

# Include override files you do wish to add to version control using negated pattern
# !example_override.tf

# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
# example: *tfplan*

# Ignore CLI configuration files
.terraformrc
terraform.rc

FILE: LICENSE

This is free and unencumbered software released into the public domain.

Anyone is free to copy, modify, publish, use, compile, sell, or
distribute this software, either in source code form or as a compiled
binary, for any purpose, commercial or non-commercial, and by any
means.

In jurisdictions that recognize copyright laws, the author or authors
of this software dedicate any and all copyright interest in the
software to the public domain. We make this dedication for the benefit
of the public at large and to the detriment of our heirs and
successors. We intend this dedication to be an overt act of
relinquishment in perpetuity of all present and future rights to this
software under copyright law.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.

For more information, please refer to

FILE: alb_ingress_iam_policy.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:CreateServiceLinkedRole"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeAccountAttributes",
                "ec2:DescribeAddresses",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeInternetGateways",
                "ec2:DescribeVpcs",
                "ec2:DescribeVpcPeeringConnections",
                "ec2:DescribeSubnets",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeInstances",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeTags",
                "ec2:GetCoipPoolUsage",
                "ec2:DescribeCoipPools",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeLoadBalancerAttributes",
                "elasticloadbalancing:DescribeListeners",
                "elasticloadbalancing:DescribeListenerCertificates",
                "elasticloadbalancing:DescribeSSLPolicies",
                "elasticloadbalancing:DescribeRules",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeTargetGroupAttributes",
                "elasticloadbalancing:DescribeTargetHealth",
                "elasticloadbalancing:DescribeTags"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "cognito-idp:DescribeUserPoolClient",
                "acm:ListCertificates",
                "acm:DescribeCertificate",
                "iam:ListServerCertificates",
                "iam:GetServerCertificate",
                "waf-regional:GetWebACL",
                "waf-regional:GetWebACLForResource",
                "waf-regional:AssociateWebACL",
                "waf-regional:DisassociateWebACL",
                "wafv2:GetWebACL",
                "wafv2:GetWebACLForResource",
                "wafv2:AssociateWebACL",
                "wafv2:DisassociateWebACL",
                "shield:GetSubscriptionState",
                "shield:DescribeProtection",
                "shield:CreateProtection",
                "shield:DeleteProtection"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:RevokeSecurityGroupIngress"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateSecurityGroup"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags"
            ],
            "Resource": "arn:aws:ec2:*:*:security-group/*",
            "Condition": {
                "StringEquals": {
                    "ec2:CreateAction": "CreateSecurityGroup"
                },
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags",
                "ec2:DeleteTags"
            ],
            "Resource": "arn:aws:ec2:*:*:security-group/*",
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:DeleteSecurityGroup"
            ],
            "Resource": "*",
            "Condition": {
                "Null": {
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:CreateLoadBalancer",
                "elasticloadbalancing:CreateTargetGroup"
            ],
            "Resource": "*",
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:CreateListener",
                "elasticloadbalancing:DeleteListener",
                "elasticloadbalancing:CreateRule",
                "elasticloadbalancing:DeleteRule"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:AddTags",
                "elasticloadbalancing:RemoveTags"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ],
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:AddTags",
                "elasticloadbalancing:RemoveTags"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:ModifyLoadBalancerAttributes",
                "elasticloadbalancing:SetIpAddressType",
                "elasticloadbalancing:SetSecurityGroups",
                "elasticloadbalancing:SetSubnets",
                "elasticloadbalancing:DeleteLoadBalancer",
                "elasticloadbalancing:ModifyTargetGroup",
                "elasticloadbalancing:ModifyTargetGroupAttributes",
                "elasticloadbalancing:DeleteTargetGroup"
            ],
            "Resource": "*",
            "Condition": {
                "Null": {
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:AddTags"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ],
            "Condition": {
                "StringEquals": {
                    "elasticloadbalancing:CreateAction": [
                        "CreateTargetGroup",
                        "CreateLoadBalancer"
                    ]
                },
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:RegisterTargets",
                "elasticloadbalancing:DeregisterTargets"
            ],
            "Resource": "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:SetWebAcl",
                "elasticloadbalancing:ModifyListener",
                "elasticloadbalancing:AddListenerCertificates",
                "elasticloadbalancing:RemoveListenerCertificates",
                "elasticloadbalancing:ModifyRule"
            ],
            "Resource": "*"
        }
    ]
}

FILE: backend.tf

terraform {
  backend "http" {
  }
}

FILE: data.tf

data "aws_availability_zones" "available" {
}

data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_name
  depends_on = [module.eks.cluster_name]
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_name
  depends_on = [module.eks.cluster_name]
}

FILE: main.tf NOTE: Make sure you add lines 40-41 and specify the rolGitlab role created earlier. If you don’t do this when you destroy the cluster, the KMS key for EKS will stay there and there is no way you can delete it unless you call AWS support.

# Create a VPC
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.0.0"

  name = var.vpcName

  cidr = var.vpcCIDR
  azs  = slice(data.aws_availability_zones.available.names, 0, 3)

  private_subnets = var.privateSubnets
  public_subnets  = var.publicSubnets

  enable_nat_gateway   = var.enable_nat_gateway
  single_nat_gateway   = var.single_nat_gateway
  enable_dns_hostnames = var.enable_dns_hostnames

  public_subnet_tags = {
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
    "kubernetes.io/role/elb"                    = 1
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
    "kubernetes.io/role/internal-elb"           = 1
  }
}

# Create an EKS cluster
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "19.15.3"

  cluster_name    = var.cluster_name
  cluster_version = var.cluster_version

  vpc_id                         = module.vpc.vpc_id
  subnet_ids                     = module.vpc.private_subnets
  cluster_endpoint_public_access = var.cluster_endpoint_public_access
  kms_key_administrators         = ["arn:aws:iam::123456789012:role/rolGitLab"]
  kms_key_enable_default_policy  = true

  eks_managed_node_group_defaults = {
    ami_type = var.ami_type
  }

  eks_managed_node_groups = {
    one = {
      name = var.nodeGroup

      instance_types = var.instanceTypes

      min_size     = var.minSize
      max_size     = var.maxSize
      desired_size = var.desiredSize
    }
  }
}

# Create an IAM policy for the ALB ingress
resource "aws_iam_policy" "worker_policy" {
  name        = "worker-policy"
  description = "Worker policy for the ALB Ingress"

  policy = file(var.iamALBIngressPolicyFileName)
}

resource "aws_iam_role_policy_attachment" "additional" {
  for_each = module.eks.eks_managed_node_groups

  policy_arn = aws_iam_policy.worker_policy.arn
  role       = each.value.iam_role_name
}

# Crete the Ingress using Helm chart from AWS
resource "helm_release" "ingress" {
  name       = "ingress"
  chart      = "aws-load-balancer-controller"
  repository = "https://aws.github.io/eks-charts"
  version    = "1.6.1"

  set {
    name  = "autoDiscoverAwsRegion"
    value = "true"
  }

  set {
    name  = "autoDiscoverAwsVpcID"
    value = "true"
  }

  set {
    name  = "clusterName"
    value = module.eks.cluster_name
  }
}

FILE: outputs.tf

output "cluster_endpoint" {
  description = "Endpoint for EKS control plane"
  value       = module.eks.cluster_endpoint
}

output "cluster_security_group_id" {
  description = "Security group ids attached to the cluster control plane"
  value       = module.eks.cluster_security_group_id
}

output "region" {
  description = "AWS region"
  value       = var.region
}

output "cluster_name" {
  description = "Kubernetes Cluster Name"
  value       = module.eks.cluster_name
}

FILE: providers.tf

provider "aws" {
  region = var.region

  shared_config_files      = ["~/.aws/config"]  
  profile                  = "oidc"
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    token                  = data.aws_eks_cluster_auth.cluster.token
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  }
}

FILE: variables.tf

variable "region" {
  type    = string
  default = "us-east-2"
}

variable "vpcName" {
  type    = string
  default = "vpcFromGitLab"
}

variable "vpcCIDR" {
  type    = string
  default = "10.0.0.0/16"
}

variable "privateSubnets" {
  type    = list(string)
  default = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
}

variable "publicSubnets" {
  type    = list(string)
  default = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]
}

variable "enable_nat_gateway" {
  type    = bool
  default = true
}

variable "single_nat_gateway" {
  type    = bool
  default = false
}

variable "enable_dns_hostnames" {
  type    = bool
  default = true
}

variable "cluster_name" {
  type    = string
  default = "eksFromGitLab"
}

variable "cluster_version" {
  type    = string
  default = "1.27"
}

variable "cluster_endpoint_public_access" {
  type    = bool
  default = true
}

variable "ami_type" {
  type    = string
  default = "AL2_x86_64"
}

variable "nodeGroup" {
  type    = string
  default = "ngFromGitLab"
}

variable "instanceTypes" {
  type    = list(string)
  default = ["t3.small"]
}

variable "minSize" {
  type    = number
  default = 1
}

variable "maxSize" {
  type    = number
  default = 3
}

variable "desiredSize" {
  type    = number
  default = 3
}


variable "iamALBIngressPolicyFileName" {
  type    = string
  default = "alb_ingress_iam_policy.json"
}

…and finally your .gitlab-ci.yml file.
FILE: .gitlab-ci.yml. NOTE: Remove line 38 if you don’t use it, don’t keep it with a comment or you’ll get an error in the pipeline.

stages:          
  - validate
  - plan
  - apply
  - destroy

image:
  name: hashicorp/terraform:1.6
  entrypoint:
    - "/usr/bin/env"
    - "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

variables:
  TF_DIR: ${CI_PROJECT_DIR}
  TF_STATE_NAME: "eks-tf-state"
  TF_ADDRESS: "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/terraform/state/${TF_STATE_NAME}"

default:
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com

  before_script:
    - terraform --version
    - cd ${TF_DIR}
    - terraform init
      -reconfigure 
      -backend-config="address=${TF_ADDRESS}" 
      -backend-config="lock_address=${TF_ADDRESS}/lock" 
      -backend-config="unlock_address=${TF_ADDRESS}/lock" 
      -backend-config="username=${TF_USERNAME}" 
      -backend-config="password=${TF_PASSWORD}" 
      -backend-config="lock_method=POST" 
      -backend-config="unlock_method=DELETE" 
      -backend-config="retry_wait_min=5"
    - mkdir -p ~/.aws
    - echo "${GITLAB_OIDC_TOKEN}" > /tmp/web_identity_token
    - # echo "${GITLAB_OIDC_TOKEN}" | base64 - If you want to see the token, reveal it this way and then use base64 --decode to get the real one
    - echo -e "[profile oidc]\nrole_arn=${ROLE_ARN}\nweb_identity_token_file=/tmp/web_identity_token" > ~/.aws/config

job-validate:
  stage: validate
  script:
    - terraform validate

job-plan:
  stage: plan
  script:
    - terraform plan -out "tf_plan"
  dependencies:
    - job-validate
  artifacts:
    paths:
      - tf_plan

job-apply:
  stage: apply
  script:
    - terraform apply "tf_plan"
  dependencies:
    - job-plan
  allow_failure: true

job-destroy:
  stage: destroy
  script:
    - terraform destroy  -auto-approve
  dependencies:
    #- job-plan
    #- job-apply
  when: manual

Push the code back to Gitlab (git add, git commit, git push) and once everything completes successfully, you’ll have a working EKS cluster after 10-15 mins.
If you choose to manage the EKS from GitLab, you can skip to the next section. If you decide to manage the cluster outside GitLab and you already have another IAM user added to the role rolGitLab as described above, then do the following. Replace account # in line 1.

aws eks update-kubeconfig --region us-east-2 --name eksFromGitLab --role-arn arn:aws:iam::123456789012:role/rolGitLab
aws eks list-clusters
eksctl get cluster --name=eksFromGitLab --region us-east-2
kubectl get svc

Manage EKS from GitLab

Generally, you would want this to be in another project. Our first project was to build the cluster, but our second project builds on top of that and manages the cluster. In addition, the second project also uses the CI/CD pipeline file .gitlab-ci.yml so in order to avoid a single complex project – just create a new project. I’ll name my new project – project1. Clone the project on your local machine using git clone.
Create an empty config file for the agent on your local machine. It has to be under this folder structure.

mkdir -p .gitlab/agents/eks-agent && touch .gitlab/agents/eks-agent/config.yaml

In my case the name of the agent is eks-agent. Use whatever name you want but follow DNS naming convention.
Push the local changes to GitLab.
On the left side in GitLab, go to Operate | Kubernetes clusters and then click Connect a cluster from the upper right corner.
Select the agent from the drop down and click Register.

You’ll get an access token and a bunch of helm commands to install the agent on the cluster. Copy and paste and run those commands.

helm repo add gitlab https://charts.gitlab.io
helm repo update
helm upgrade --install eks-agent gitlab/gitlab-agent \
    --namespace gitlab-agent-eks-agent \
    --create-namespace \
    --set image.tag=v16.9.0-rc2 \
    --set config.token=glagent-2r_uW63e1w_mkNzR_keU2hmaNhhcKauSpujg8OdLQG74wrF73g \
    --set config.kasAddress=wss://kas.gitlab.com

Refresh the GitLab screen after the helm chart has been installed and you’ll see your cluster connected.

Now you need to authorize the agent to access our project.
In the blank config.yaml file on the local machine under .gitlab/agents/eks-agent/ directory, add these lines. Change the path the project in line 3 (look at the URL, that’s your project name after gitlab.com) and change the manifest file if you want in line 6. You can also change the namespace in line 5 as well.

gitops:
  manifest_projects:
  - id: andreevkliment/project1
    default_namespace: default
    paths:
      - glob: 'manifest.yaml'

Then create the manifest file under the root of the project. Save it as manifest.yaml or however you referenced it in line 6 in the previous file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

If you push this code back to GitLab, the agent will execute it and you’ll (with kubect get pods) see the nginx pods running on your cluster.
Another way of managing the EKS cluster is using .gitlab-ci.yml file.
Create .gitlab-ci.yml with these lines. Replace your username, project and the agent name in line 7.

deploy:
  image:
    name: bitnami/kubectl:latest
    entrypoint: ['']
  script:
    - kubectl config get-contexts
    - kubectl config use-context andreevkliment/project1:eks-agent
    - kubectl get pods

If you push this back to GitLab, the pipeline will start and you’ll see the pods from the cluster in the logs. This example doesn’t execute manifest.yaml. You can have both ways of interacting with the cluster or you can use them separately.

Install MetalLB on Kubernetes cluster running on VMWare VMs or bare metal server and nginx ingress

Kliment Andreev — Sat, 21 Oct 2023 23:05:18 +0000

In this post I’ll explain how to install MetalLB which is a load balancer for bare metal/VM Kubernetes servers. If you have your own home lab to play with Kubernetes and you are not using AKS, EKS or GKE, then this post is for you.
The prerequisite is to have a running Kubernetes cluster. In my earlier post, I’ve described how to install Kubernetes with CRI-O and Cilium as a CNI, so you can follow that post or feel free to have your own install. Mind that if you use other CNIs such as Calico or Weave, there are some things you have to check first and see if it applies to you. Look at this link.
We’ll install MetalLB now. You probably have your k8s master and the nodes ready.

Install MetalLB

Do this first on the master.

kubectl edit configmap -n kube-system kube-proxy

…and change strictARP to true.

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"
ipvs:
  strictARP: true

Deploy MetalLB.

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.11/config/manifests/metallb-native.yaml

You’ll see the pods running in metallb-system namespace.

kubectl get pods -n metallb-system
NAME                          READY   STATUS    RESTARTS   AGE
controller-7d56b4f464-gkfrq   1/1     Running   0          55s
speaker-l5954                 1/1     Running   0          55s
speaker-xrswp                 1/1     Running   0          55s
speaker-z8vdt                 1/1     Running   0          55s

Let’s configure our load balancer to give IPs in this 192.168.1.50-192.168.1.60 range. Change to your needs accordingly in line 8.
Save it as a lb-config.yaml file.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: first-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.1.50-192.168.1.60
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: example
  namespace: metallb-system

Deploy the config.

kubectl apply -f lb-config.yaml

Now, we can deploy a simple app and a load balancer. Save the file as demo.yaml.
This is an app that I made in Node.js that listens on port 3000 to display the IP of the pod that the load balancer hits.
You can use it for any type of a load balancer, not just MetalLB. It’s a super simple app that prints the IP of the pod where it’s running.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
spec:
  replicas: 6
  selector:
    matchLabels:
      run: demo
  template:
    metadata:
      labels:
        run: demo
    spec:
      containers:
      - name: demo
        image: klimenta/serverip
        ports:
        - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: loadbalancer
spec:
  ports:
    - port: 80
      targetPort: 3000
      protocol: TCP
  type: LoadBalancer
  selector:
    run: demo

Deploy the app.

kubectl apply -f demo.yaml

Check the load balancer service.

kubectl get svc
NAME           TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)        AGE
kubernetes     ClusterIP      10.96.0.1               443/TCP        33m
loadbalancer   LoadBalancer   10.106.36.58   192.168.1.50   80:30770/TCP   4m33s

…and if you go to that IP (192.168.1.50) in your browser, you’ll see the IP of the pods that the load balancer hits.

Install nginx ingress reverse proxy

The problem with the scenario above is that for every service you need to expose, you need a different IP from the IP pool that we’ve assigned to MetalLB (.50-.60).
A better solution is to install an nginx ingress that will act as a reverse proxy, but for this you’ll need a working DNS server as the services will share the same IP, but different FQDN.
So, in practice it looks like this.
Install nginx ingress first.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.2/deploy/static/provider/cloud/deploy.yaml
kubectl get pods -n ingress-nginx
kubectl get service ingress-nginx-controller -n=ingress-nginx

Don’t worry if the two nginx ports are not running and it says – “completed”.
You’ll see the external IP that’s assigned to nginx from MetalLB.
For some reason, nginx ingress won’t work without this line. Many people have this problem in a bare-metal environment and this is the workaround.

kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission

Now, let’s create an nginx web server. This has nothing to do with nginx ingress, it’s completely separate.
I am exposing the nginx deploymenty and creating a rule that will hit the deployment on port 80 with nginx.homelab.local as FQDN.
NOTE: If your browser can’t resolve this URL, it won’t work. So, make sure you have a DNS or /etc/hosts file or some type of name resolution.

kubectl create deployment nginx --image=nginx --port=80
kubectl expose deployment nginx
kubectl create ingress nginx --class=nginx --rule nginx.homelab.local/=nginx:80

Then, let’s install Apache server that runs on the same IP and same port 80.

kubectl create deployment httpd --image=httpd --port=80
kubectl expose deployment httpd
kubectl create ingress httpd --class=nginx --rule httpd.homelab.local/=httpd:80

If everything is OK, you’ll be able to access both web servers using their URLs.
If you want to put Grafana and/or KubeCost behind nginx ingress, read these short posts here and here.
If you want to use yaml for deployment, here is an example of that. It creates 6 replicas of a pod that prints the IP where the pod is running. The service runs on port 3000. Then it creates a ClusterIP on the same port and puts the ClusterIP behind the ingress. Change the hostname at line 39.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
spec:
  replicas: 6
  selector:
    matchLabels:
      run: demo
  template:
    metadata:
      labels:
        run: demo
    spec:
      containers:
      - name: demo
        image: klimenta/getmyip
        ports:
        - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: demo
spec:
  type: ClusterIP
  ports:
  - port: 3000
    targetPort: 3000
  selector:
    run: demo
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-demo
spec:
  rules:
  - host: demo.homelab.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: demo
            port:
              number: 3000
  ingressClassName: nginx

SSL certificates

The examples above allow you to run multiple websites behind a reverse proxy on port 80, but in reality you want everything on port 443 and with SSL certificates. I’ll describe two scenarios: with a valid cert and self-signed certificate.Fortunately, nginx ingress listens on port 443 and does automatic redirect from http->https so you don’t have to worry about any changes.

Valid SSL certificate

You will need a valid SSL certificate for this. And you will need the private key, the certificate, the CA root certificate and the intermediate certificate. You can get these from the vendor where you purchased the certificate. In my case, I’ll have the private key saved as domain.key, the certificate as domain.pem, the root CA certificate as domain.ca and the intermediate certificate as domain.inter.
IMPORTANT: nginx expects the certificate chain in exact order: PEM -> INTERMEDIATE -> ROOT CA. If you have just the private key and the certificate (without inter and root CA), it might work, but you might get some errors when using curl, such as that the certificate authority cannot be recognized.
So, create a file called domain.full.pem or whatever that has all 3 certificates in that order, e.g. cat domain.pem domain.inter domain.ca > domain.full.pem.
If you received a bundle file from the certificate vendor, create a chain from two files: domain.pem plus the bundle file. The bundle file contains the intermediate and the root certificate.
Now that you have these files, you need to create a TLS secret in Kubernetes where you’ll store your certificate. You can do that with kubectl or with a YAML file.

kubectl create secret tls demo-tls --key domain.key --cert domain.full.pem

Keep the secret, ingress and the deployment in the same namespace.
If you use YAML, which most likely you’ll do if you use automation, you’ll have to encode the certificates.
This is the template YAML file. Name it as tls.yaml.

apiVersion: v1
kind: Secret
metadata:
  name: demo-tls
type: kubernetes.io/tls
data:
  tls.crt: |
    
  tls.key: |

Don’t change data keys tls.crt and tls.key. You have to use them, not something like key.crt or something.crt. Use exactly tls.crt and tls.key.
Encode the certificate with base64.

base64 -w0 domain.full.pem

Encode the key with base64.

base64 -w0 domain.key

You’ll get two long strings that start with LS0…Copy & paste these strings into YAML and make sure they are properly indented.

Now, you can create the secret.

kubectl apply -f tls.yaml

All yopu have to do now is to reference the TLS secret that we created. Compare the highlighted lines in this YAML with the previous one.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-demo
spec:
  tls:
  - hosts:
    - demo.iandreev.com
    secretName: demo-tls
  rules:
  - host: demo.iandreev.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: demo
            port:
              number: 3000
  ingressClassName: nginx

Change the hostname in line 11 and 14 to match your cert and you can deploy the app.

Self-signed certificates

First, we have to create a Certificate Authority (CA). I’ll use homelab.local as my domain.

openssl req -x509 -nodes -sha512 -days 365 -newkey rsa:4096 \
 -keyout rootCA.key -out rootCA.crt -subj "/C=US/CN=HomeLab Certificate Authority"

You’ll get two files, rootCA.key (the private key) and rootCA.crt.
Create a private key for the wildcard homelab.local domain and a certificate signing request file.
You’ll get two files, star.homelab.local.key (the private key for the domain) and star.homelab.local.csr (the certificate signing request).

openssl req -new -nodes -sha512 -days 365 -newkey rsa:4096 \
 -keyout star.homelab.local.key -out star.homelab.local.csr -subj "/C=US/ST=NJ/L=Trenton/O=HomeLab Org/CN=*.homelab.local"

Create this file where you’ll specify your domain name and eventually your IP, but the IP is not necessary.

cat << EOF > v3.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = *.homelab.local
IP.1 = 192.168.1.50
EOF

Sign the certificate signing request and create the certificate file. You’ll get one certificate file star.homelab.local.crt.

openssl x509 -req -sha512 -days 365 -in star.homelab.local.csr -CA rootCA.crt -CAkey rootCA.key \
 -CAcreateserial -out star.homelab.local.crt  -extfile v3.ext

Combine the certificate and the CA certificate in a single file.

cat star.homelab.local.crt rootCA.crt > star.homelab.local.chain.crt

Create this tls.yaml template file.

apiVersion: v1
kind: Secret
metadata:
  name: demo-tls
type: kubernetes.io/tls
data:
  tls.crt: |
    
  tls.key: |

Encode both the domain certificate chain file and the domain private key file.

base64 -w0 star.homelab.local.chain.crt
base64 -w0 star.homelab.local.key

You’ll get two very long single lines. Add these two lines to the tls.yaml template file but make sure they are properly indented (see the screenshot in the previous section).
Now, you can create the secret.

kubectl apply -f tls.yaml

You can also use kubectl to create the secret, see the previous section.
Save this file as demo-local.yaml and deploy it using kubectl apply -f demo-local.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
spec:
  replicas: 6
  selector:
    matchLabels:
      run: demo
  template:
    metadata:
      labels:
        run: demo
    spec:
      containers:
      - name: demo
        image: klimenta/getmyip
        ports:
        - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: demo
spec:
  type: ClusterIP
  ports:
  - port: 3000
    targetPort: 3000
  selector:
    run: demo
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-demo
spec:
  tls:
  - hosts:
    - demo.homelab.local
    secretName: demo-tls
  rules:
  - host: demo.homelab.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: demo
            port:
              number: 3000
  ingressClassName: nginx

You’ll be able to go to your browser and check demo.homelab.local and see the page after the SSL warning.
But if you double-click the rootCA.crt file on your local machine, you can import the certificate in the Trusted Root Certificates store and you won’t get that warning anymore.

Install Kubernetes with CRI-O and Cilium on Rocky Linux 9 / VMware / Bare Metal

Kliment Andreev — Mon, 09 Oct 2023 16:18:16 +0000

In this post, I’ll build a Kubernetes cluster with one master and two nodes using CRI-O and Cilium. I have DNS in my environment but you can use the hosts files if needed.
The OS will be Rocky Linux 9 running on ESXi 8.x but this should work on any other bare metal server too. The config for the nodes is:

– Master: 2 CPUs, 4GB RAM
– Workers: 2 CPUs, 8GB RAM.

First, we’ll have to disable swap and put SELinux in permissive mode.
Do this on all nodes (master + workers).

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

Now, we have to open some firewall ports for Kubernetes..
Do this on the master node only.

sudo firewall-cmd --permanent --add-port=6443/tcp
sudo firewall-cmd --permanent --add-port=2379-2380/tcp
sudo firewall-cmd --permanent --add-port=10250/tcp
sudo firewall-cmd --permanent --add-port=10259/tcp
sudo firewall-cmd --permanent --add-port=10257/tcp
sudo firewall-cmd --reload

Do this on the workers only.

sudo firewall-cmd --permanent --add-port=10250/tcp
sudo firewall-cmd --permanent --add-port=30000-32767/tcp   
sudo firewall-cmd --reload

In addition, we’ll have to open the firewall ports for Cilium.
Do this on all nodes.

sudo firewall-cmd --permanent --add-port=4240/tcp 
sudo firewall-cmd --permanent --add-port=8472/udp 
sudo firewall-cmd --reload

Now, we have to make some changes that are also required. Some necessary modules and IPtables change.
Do this on all nodes.

cat <
We have to install CRI-O container runtime.

Do this on all nodes.
VERSION=1.22
sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/CentOS_8/devel:kubic:libcontainers:stable.repo
sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:${VERSION}.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:${VERSION}/CentOS_8/devel:kubic:libcontainers:stable:cri-o:${VERSION}.repo
sudo dnf -y install cri-o cri-tools
sudo systemctl enable --now crio
sudo systemctl status crio

Then, we have to install Kubernetes.

Do this on all nodes.
cat <
On the master node only, initialize the cluster. Replace master.homelab.com with your master IP if you don’t use DNS.
sudo kubeadm init --control-plane-endpoint master.homelab.local:6443

You’ll get an output that says how to join the nodes, something like kubeadm join master… + a token

On the master node, add these lines.
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Join the nodes to the cluster.

Do this on workers only.
sudo kubeadm join master.homelab.local:6443 --token t2yir0.7n49rn6r57msy4j4 \
        --discovery-token-ca-cert-hash sha256:563360fcf60c49be91cdcf6486a4954c579a80c54503127ee0682ab8f86ec840

Check the nodes.

Do this on the master node.
kubectl get nodes
NAME                   STATUS   ROLES           AGE    VERSION
master.homelab.local   Ready    control-plane   116s   v1.28.2
node1.homelab.local    Ready              89s    v1.28.2
node2.homelab.local    Ready              85s    v1.28.2

…then check all the pods.

Do this on the master node.
kubectl get pods --all-namespaces
NAMESPACE     NAME                                           READY   STATUS    RESTARTS   AGE
kube-system   coredns-5dd5756b68-5ljgg                       1/1     Running   0          2m
kube-system   coredns-5dd5756b68-jp8tq                       1/1     Running   0          2m
kube-system   etcd-master.homelab.local                      1/1     Running   0          2m6s
kube-system   kube-apiserver-master.homelab.local            1/1     Running   0          2m6s
kube-system   kube-controller-manager-master.homelab.local   1/1     Running   0          2m6s
kube-system   kube-proxy-2z64r                               1/1     Running   0          2m
kube-system   kube-proxy-mrwjv                               1/1     Running   0          96s
kube-system   kube-proxy-nj5q4                               1/1     Running   0          100s
kube-system   kube-scheduler-master.homelab.local            1/1     Running   0          2m6s

Install Cilium on the master node only.
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
cilium install --version 1.14.2

Check the status.
cilium status --wait

You should see something like this. 
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium             Desired: 3, Ready: 3/3, Available: 3/3
Deployment             cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
Containers:            cilium             Running: 3
                       cilium-operator    Running: 1
Cluster Pods:          2/2 managed by Cilium
Helm chart version:    1.14.2
Image versions         cilium             quay.io/cilium/cilium:v1.14.2@sha256:6263f3a3d5d63b267b538298dbeb5ae87da3efacf09a2c620446c873ba807d35: 3
                       cilium-operator    quay.io/cilium/operator-generic:v1.14.2@sha256:52f70250dea22e506959439a7c4ea31b10fe8375db62f5c27ab746e3a2af866d: 1

Then you can run the connectivity test. Some of the egress test will fail because we don’t have a public IP on the cluster.
cilium connectivity test

You should be able to run your deployments now. If you need to access the pods using URL use something like NodePort or port-forwarding. It’s very cumbersome and inefficient, but you can use a load balancer with MetalLB. Read this post to see how to do that. It’s separated from this post, because it’s a separate topic than just k8s, container runtime and CNI.

Kubernetes: Monitoring Node.js application with Prometheus and Grafana, Helm charts in AWS EKS

Kliment Andreev — Tue, 24 Jan 2023 16:01:05 +0000

Recently, we were migrating some Node.js applications to a different AWS account and we had a need to monitor the application with Prometheus and Grafana. In this post, I’ll explain how to install both Prometheus and Grafana using Helm charts, then we’ll create a simple Express.js web app, create a Helm chart for it and deploy it on an EKS cluster in AWS. I am using an EKS cluster, but you can use any Kubernetes cluster that you can manage it with kubectl. You might have to make some changes though in how you access the Load Balancer for Grafana.
Requirements:

Create EKS cluster

This is optional, if you already have a running Kubernetes cluster, you can skip this step.

eksctl create cluster --name eksTest \
--region us-east-2 \
--instance-types t3.medium \
--nodes 2 \
--managed \
--version 1.22

This command will create a 2 node cluster, with t3.medium instances in Ohio region using Amazon Linux 2 image. It will take about 15 minutes.
Verify that everything looks OK.

kubectl get nodes
NAME                                           STATUS   ROLES    AGE     VERSION
ip-192-168-10-4.us-east-2.compute.internal     Ready       8m54s   v1.22.15-eks-fb459a0
ip-192-168-74-160.us-east-2.compute.internal   Ready       8m51s   v1.22.15-eks-fb459a0

Install Prometheus and Grafana

Add the Helm repos for Prometheus, Grafana and update any changes.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

We’ll create a separate namespace for Prometheus and install the chart there.

kubectl create namespace prometheus
helm install prometheus prometheus-community/prometheus \
    --namespace prometheus \
    --set alertmanager.persistentVolume.storageClass="gp2" \
    --set server.persistentVolume.storageClass="gp2"

Create this file. It tells Grafana what to use as a source (url). It’s the Prometheus endpoint service that we just installed.

cat << EOF > grafana.yaml
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-server.prometheus.svc.cluster.local
      access: proxy
      isDefault: true
EOF

Now, let’s create a namespace for Grafana and then install Grafana in that namespace. Change the admin password. In my case it’s Password1$.

kubectl create namespace grafana
helm install grafana grafana/grafana \
    --namespace grafana \
    --set persistence.storageClassName="gp2" \
    --set persistence.enabled=true \
    --set adminPassword='Password1$' \
    --values grafana.yaml \
    --set service.type=LoadBalancer

You’ll get something like this on the screen when the install ends.

export SERVICE_IP=$(kubectl get svc --namespace grafana grafana -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

That’s your load balancer IP so you can access Grafana. If you use EKS, your IP is different.

kubectl get services -n grafana
NAME      TYPE           CLUSTER-IP     EXTERNAL-IP                                                               PORT(S)        AGE
grafana   LoadBalancer   10.100.20.53   a074996f790b74793a74bec584c04460-2034633908.us-east-2.elb.amazonaws.com   80:31916/TCP   3m21s

The IP is the gibberish URL that ends with amazonaws.com. If you go to that URL, you can log as admin and your password.

Go to Dashboards, click on the New button on the right and click Import.
Add the following dashboard IDs (one by one then click Load) and choose Prometheus as a source. The IDs are 3119 (Kubernetes cluster), 6417 (pods) and 11159 (Node.js). For the first dashboard, you’ll have some metrics. The Node.js dashboard is empty and our goal is to get the metrics from the application.

Node.js application

Create an empty folder and add this file.

mkdir mynodeapp
cd mynodeapp
cat << EOF > package.json
{
  "name": "mynodeapp",
  "version": "1.0.0",
  "description": "Node.js on Docker",
  "author": "Kliment Andreev ",
  "main": "server.js",
  "scripts": {
    "start": "node server.js"
  },
  "dependencies": {
    "express": "^4.16.1",
    "prom-client": "^14.1.1"
  }
}
EOF

I use Node v18.12. Install the dependencies.

npm install

This is the simple web application. It listens on port 3000.

cat << "EOF" > server.js
'use strict'
const express = require('express')
const prom = require('prom-client')

const collectDefaultMetrics = prom.collectDefaultMetrics();

const app = express()
const port = process.env.PORT || 3000

app.get('/', (req, res, next) => {
  res.send('Hello World!');
});

app.get('/metrics', async (req, res) => {
  try {
	  res.set('Content-Type', prom.register.contentType);
	  res.end(await prom.register.metrics());
  } 
  catch (ex) {
	  res.status(500).end(ex);
  }
});

const server = app.listen(port, () => {
  console.log(`mynodeapp listening on port ${port}!`)
})
EOF

The app is a super simple web app that prints Hello World! The important part is the /metrics router. That’s how you get the metrics from the app. When we created the package.json file, we specify the prom-client as dependency. That’s the part that collects the metrics. Go to this link for more info.
Start the app with node server.js.

node server.js
mynodeapp listening on port 3000!

Go to the localhost or the server IP where this app is running and you’ll see Hello World! printed out.

And if you go to the /metrics URL, you’ll see this text. That’s what Prometheus expects.

Now that we know the app is working, let’s create a Docker image.

Docker image and container

Create this Dockerfile.

cat << EOF > Dockerfile
FROM node:18-alpine

# Create app directory
WORKDIR /app

# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm@5+)
COPY package*.json ./

RUN npm install
# If you are building your code for production
# RUN npm install --only=production

# Bundle app source
COPY server.js /app

EXPOSE 3000
CMD [ "npm", "start", "server.js" ]
EOF

Create a Docker image based on the Dockerfile above. Replace the username with your Docker Hub username.

docker build -t /mynodeapp:latest .

Run the container from the image.

docker run -p 3000:3000 /mynodeapp:latest

> mynodeapp@1.0.0 start
> node server.js server.js

mynodeapp listening on port 3000!

Use the same test as before by going to the localhost or IP URL on port 3000.
If everything is OK, publish the image on Dockerhub.

docker login
docker push /mynodeapp

Helm chart

Now that we have the Docker image, we’ll create a Helm chart so we can deploy the web app to the Kubernetes cluster.
Create the necessary structure.

helm create mynodeapp
Creating mynodeapp

Go to the sub-directory mynodeapp and edit the values.yaml file first. Look for the image key first and replace it so it looks like this. Make sure you replace the with your Docker Hub username.

image:
  repository: /mynodeapp
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: "latest"

Then look for podAnnotations: {} line and replace it so it looks like this.

podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/path: "/metrics"
  prometheus.io/port: "3000"

Look for the service parameter and change it so it looks like this.

service:
  type: LoadBalancer
  port: 80
  targetPort: 3000
  name: mynodeapp-service

Look for the resources key and uncomment the defaults.

resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
limits:
  cpu: 100m
  memory: 128Mi
requests:
  cpu: 100m
  memory: 128Mi

Edit the templates/deployment.yaml file and find these lines.

ports:
  - name: http
    containerPort: {{ .Values.service.port }}

Replace the containerPort to look like this.

containerPort: 3000

Go back to the root of the mynodeapp folder and check the chart.

helm install mynodeapp --generate-name
NAME: mynodeapp-1674575241
LAST DEPLOYED: Tue Jan 24 10:47:22 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
     NOTE: It may take a few minutes for the LoadBalancer IP to be available.
           You can watch the status of by running 'kubectl get --namespace default svc -w mynodeapp-1674575241'
  export SERVICE_IP=$(kubectl get svc --namespace default mynodeapp-1674575241 --template "{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}")
  echo http://$SERVICE_IP:80

Execute the export SERVICE_IP command above and echo the IP. That’s your URL. If you go to that URL you’ll see the Hello World! greetings.
What we care is the pod. Get the pod running. It’s in the default namespace.

kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
mynodeapp-1674575241-5ff9d7469d-pddcz   1/1     Running   0          3m

Get the annotations. Replace with your pod name.

kubectl get pod mynodeapp-1674575241-5ff9d7469d-pddcz -o jsonpath='{.metadata.annotations}'
{"kubernetes.io/psp":"eks.privileged","prometheus.io/path":"/metrics","prometheus.io/port":"3000","prometheus.io/scrape":"true"}

You can see the prometheus annotations.
And if you go to Grafana and open up the NodeJS dash you’ll see the metrics.

AWS: EKS monitoring and logging

Kliment Andreev — Mon, 03 Oct 2022 23:09:44 +0000

In this post I’ll explain several things related to EKS monitoring and logging.

In order to do that, we’ll need the following CLI tools.

You can easily find how to install and configure these tools for various OSes.

Create the EKS cluster

I’ll create a managed cluster called eksWordPress in us-east-2 region with two t3.medium nodes.

eksctl create cluster --name eksWordPress --region us-east-2 --instance-types t3.medium --nodes 2 --managed --version 1.22

If you get an error that the last supported version is 1.21, update eksctl tool. The cluster creation took about 20 mins for me.

Send EKS logs to CloudWatch

Once the cluster was created and up and running, enabling the EKS control plane logs is easy. There are 5 log types (API server, Audit, Authenticator, Controller Manager and Scheduler logs). The logged info is a lot, so it’s up to you what you want to log. There is no way to choose what to log, e.g. info, warning, errors. Everything is logged. To enable logging from the console, go to the EKS cluster, select it, click the Logging tab and then click on the Manage logging button.

If you want to do it from a CLI, type this command and specify what types of logs you want sent to CloudWatch. Specify the region and the cluster name.

aws eks update-cluster-config \
    --region  \
    --name  \
    --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'

In addition, because we provisioned the cluster with the eksctl tool, we can also enable the logs using eksctl.

eksctl utils update-cluster-logging --enable-types all --cluster  --approve

Or for certain types, use…

eksctl utils update-cluster-logging --enable-types  --cluster  --approve

You can use the same types as in the AWS CLI command above (api, audit, scheduler…).
If you go to CloudWatch and then the Log groups, you’ll see the log group with the name of the EKS cluster (/aws/eks/eksWordPress/cluster).
Make sure you change the retention from Never to some value. You probably don’t want to keep these logs indefinitely. If you look at the logs streams, you’ll see that there is a lot of info there. Most of the stuff is useless.

Send the logs to OpenSearch cluster

Let’s create a public OpenSearch cluster with anonymous access that only you can access. Replace the IP at the end of the statement with your IP address. It takes less than 10 mins for the OpenSearch cluster to be provisioned.

aws opensearch create-domain --domain-name oswordpress --engine-version OpenSearch_1.3 \
    --auto-tune-options DesiredState="ENABLED" --cluster-config InstanceType=t3.small.search,InstanceCount=2 \
    --ebs-options EBSEnabled=true,VolumeType=gp3,VolumeSize=10,Iops=3000 \
    --access-policies '{"Version": "2012-10-17", "Statement": [{"Action": "es:*", "Principal":"*","Effect": "Allow", "Condition": {"IpAddress":{"aws:SourceIp":["2.18.2.19/32"]}}}]}'

Once the cluster is provisioned you can get the public URL with:

echo `aws es describe-elasticsearch-domain --domain-name oswordpress --output text --query "DomainStatus.Endpoint"`\\_dashboards

You can also get the dashboard URL from the cluster settings as well.

Create the following IAM policy. Save the policy below as policy-file.json but change the account_no in line 9 to match your AWS account.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "es:*"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:es:us-east-2::domain/oswordpress/*"
        }
    ]
}

And then create the IAM policy.

aws iam create-policy --policy-name polOpenSearch --policy-document file://policy-file.json

Create the role. Save the policy below as policy-trust.json.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

And then create the role.

aws iam create-role --role-name rolOpenSearch --assume-role-policy file://policy-trust.json

Finally, attach the policy to the role.

aws iam put-role-policy --role-name rolOpenSearch --policy-name polOpenSearch --policy-document file://policy-file.json

Go to CloudWatch, select the Log group, from the Actions button select Subscription filters and then Create Amazon OpenSearch service subscription filter.

Choose the OpenSearch cluster that we created, the log format is JSON, subscription filter pattern is ” “ for all events and type all subscription filter name or whatever you want to name this pattern. Click Start streaming button.
Now, go to OpenSearch dashboard and from the hamburger menu in the upper left corner, click on Visualize. In the middle of the screen, you’ll be prompted to create an index pattern. Type cwl* and click Next step.

Select @timestamp from the drop down menu and click on Create index pattern. You’ll see the fields and the index.

Click on the Discover from the hamburger menu and you’ll see your data there. You can use queries to search your data, but that’s out of the scope of this post.

Container Insights and FluentBit

Container Insights is a CloudWatch agent that we’ll install on the EKS cluster. It will collect all kinds of metrics and then FluentBit as a log forwarder will ship those logs to CloudWatch. See this link for more info.
Let’s do the prerequisites work first.
Get the nodes. Type the first line only, the rest is my output.

kubectl get nodes                                                                                                                                                               
NAME                                           STATUS   ROLES    AGE     VERSION
ip-192-168-56-250.us-east-2.compute.internal   Ready       3h39m   v1.22.12-eks-ba74326
ip-192-168-66-251.us-east-2.compute.internal   Ready       3h39m   v1.22.12-eks-ba74326

Get the instance ID from any of the instances, it doesn’t matter. Replace the name accordingly after Values=.

aws ec2 describe-instances --filters 'Name=private-dns-name,Values=ip-192-168-56-250.us-east-2.compute.internal' \
    --output text --query 'Reservations[*].Instances[*].InstanceId'
i-05adaa0a823a8549a

Once you have the instance ID of any node, get the Arn of the IAM role that’s the attached to that node.

aws ec2 describe-instances --region us-east-2 --instance-ids i-05adaa0a823a8549a --query 'Reservations[*].Instances[*].IamInstanceProfile.Id'
[
    [
        "AIPAXFRN6SYD75I5N4BUN"
    ]
]

Get the role name. Replace the ID (AIPAX…) with your value above.

aws iam list-instance-profiles --query 'InstanceProfiles[?InstanceProfileId==`AIPAXFRN6SYD75I5N4BUN`].Roles[*].RoleName' 
[
    [
        "eksctl-eksWordPress-nodegroup-ng-NodeInstanceRole-LWJZESQ468Y2"
    ]
]

We want to add a policy that allows nodes to write to CloudWatch logs group. Replace the –role-name with the value that you got above.

aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy --role-name eksctl-eksWordPress-nodegroup-ng-NodeInstanceRole-LWJZESQ468Y2

oidc_id=$(aws eks describe-cluster --name eksWordPress --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)

Type this line and see if there is any output. If no output, execute the command after. If yes-output, then do nothing.

aws iam list-open-id-connect-providers | grep $oidc_id

If no output, type this.

eksctl utils associate-iam-oidc-provider --cluster eksWordPress --approve

Then install CloudWatch Container Insights and FluentBit. Change the CLUSTER_NAME and REGION in lines 1 and 2.

ClusterName=
RegionName=
FluentBitHttpPort='2020'
FluentBitReadFromHead='Off'
[[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
[[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluent-bit-quickstart.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/;s/{{region_name}}/'${RegionName}'/;s/{{http_server_toggle}}/"'${FluentBitHttpServer}'"/;s/{{http_server_port}}/"'${FluentBitHttpPort}'"/;s/{{read_from_head}}/"'${FluentBitReadFromHead}'"/;s/{{read_from_tail}}/"'${FluentBitReadFromTail}'"/' | kubectl apply -f -

Download a config map and edit it the file called cwagent-configmap.yaml.

curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-configmap.yaml

In line 11, change the variable so it points to your cluster. In my case it looks like this.

"cluster_name": "{{eksWordPress}}",

Save the changes and apply the config.

kubectl apply -f cwagent-configmap.yaml

Then deploy it as a DaemonSet.

kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-daemonset.yaml

Verify that it’s running. Type the first line only. The rest is my output.

kubectl get pods -n amazon-cloudwatch
NAME                     READY   STATUS    RESTARTS   AGE
cloudwatch-agent-dfkzv   1/1     Running   0          18m
cloudwatch-agent-nfnf7   1/1     Running   0          18m
fluent-bit-8vd2n         1/1     Running   0          18m
fluent-bit-tvtpv         1/1     Running   0          18m

Check the logs.

kubectl logs   -n amazon-cloudwatch

Or in my case…

kubectl logs cloudwatch-agent-dfkzv -n amazon-cloudwatch
[2022/10/03 13:28:56] [ info] [output:cloudwatch_logs:cloudwatch_logs.2] Created log stream ip-192-168-73-121.us-east-2.compute.internal.host.messages
[2022/10/03 13:29:06] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-73-121.us-east-2.compute.internal-application.var.log.containers.cloudwatch-agent-7fvxd_amazon-cloudwatch_cloudwatch-agent-5409bed9d4733a51602e4bc0cccde5e5580eb3e9282cd5cc4c1a4f2d2e28e8ea.log in log group /aws/containerinsights/eksWordPress/application
[2022/10/03 13:29:06] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Created log stream ip-192-168-73-121.us-east-2.compute.internal-application.var.log.containers.cloudwatch-agent-7fvxd_amazon-cloudwatch_cloudwatch-agent-5409bed9d4733a51602e4bc0cccde5e5580eb3e9282cd5cc4c1a4f2d2e28e8ea.log

If you go to CloudWatch now you’ll see 4 new log groups.

Prometheus and Grafana

The Kubernetes API can also be monitored using Prometheus. We’ll install it using helm.
Create a namespace first.

kubectl create namespace prometheus

Add the Prometheus repo.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Deploy Prometheus.

helm upgrade -i prometheus prometheus-community/prometheus --namespace prometheus --set alertmanager.persistentVolume.storageClass="gp2",server.persistentVolume.storageClass="gp2"

Check if everything is OK.

kubectl get pods -n prometheus
NAME                                            READY   STATUS    RESTARTS   AGE
prometheus-alertmanager-5c57cc6945-v9lcb        2/2     Running   0          4m55s
prometheus-kube-state-metrics-77ddf69b4-jsgrp   1/1     Running   0          4m55s
prometheus-node-exporter-68dk9                  1/1     Running   0          4m55s
prometheus-node-exporter-m98xk                  1/1     Running   0          4m55s
prometheus-pushgateway-ff89cc976-4sfhl          1/1     Running   0          4m55s
prometheus-server-6c99667b9b-mpw97              2/2     Running   0          4m55s

Type this command and open up a browser and go to localhost:9090.

kubectl --namespace=prometheus port-forward deploy/prometheus-server 9090

You can CTRL-C out of the command prompt once you verify it’s OK.
Let’s add the Grafana repo first.

helm repo add grafana https://grafana.github.io/helm-charts

Copy, paste and save this as grafana.yaml.

datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-server.prometheus.svc.cluster.local
      access: proxy
      isDefault: true

Create a namespace for grafana.

kubectl create namespace grafana

Deploy using helm. Look at the adminPassword parameter. Change it to something else. In my case it’s admin123!.

helm install grafana grafana/grafana --namespace grafana --set persistence.storageClassName="gp2" --set persistence.enabled=true --set adminPassword='admin123!' --values grafana.yaml --set service.type=LoadBalancer

Check if everything is OK.

kubectl get all -n grafana

Get the URL of the classic LB that was just created.

export ELB=$(kubectl get svc -n grafana grafana -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
echo "http://$ELB"

Go to that URL and you’ll see the Grafana URL.
You should see the Grafana landing page. Log as admin and the password you specified when pods were deployed.
In Grafana create a new dashboard and Import the dashboard with ID 3119. Choose Prometheus as source.

For pods monitoring use the same method but this time specify 6417 as a dashboard ID. This is how it looks like under my account.

WordPress

Let’s deploy WordPress.

kubectl create namespace wordpress

Add the Bitnami helm chart.

helm repo add bitnami https://charts.bitnami.com/bitnami

Deploy WordPress in its own namespace

helm -n wordpress install understood-zebu bitnami/wordpress

Wait for 3-4 mins and do this command. This is your ELB, get the URL, something like *zdasdfa*.elb.amazonaws.com

kubectl get svc --namespace wordpress -w understood-zebu-wordpress

The username is user and get the password with:

echo Password: $(kubectl get secret --namespace wordpress understood-zebu-wordpress -o jsonpath="{.data.wordpress-password}" | base64 -d)

If you go to the ELB URL, you’ll hit WordPress main page, if you want to login, add /wp-login.php as suffix to the above URL.
Go to CloudWatch and check the /aws/containerinsights/eksWordPress/application log group. You’ll see a bunch of references for WordPress. You can ship those to OpenSearch if you want and alert on errors or whatever you want to do.

Delete EKS and OpenSearch cluster

Detach the policy, delete the daemonset and delete the EKS cluster.

aws iam detach-role-policy --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy --role-name eksctl-eksWordPress-nodegroup-ng-NodeInstanceRole-LWJZESQ468Y2
ClusterName=
RegionName=
FluentBitHttpPort='2020'
FluentBitReadFromHead='Off'
[[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
[[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluent-bit-quickstart.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/;s/{{region_name}}/'${LogRegion}'/;s/{{http_server_toggle}}/"'${FluentBitHttpServer}'"/;s/{{http_server_port}}/"'${FluentBitHttpPort}'"/;s/{{read_from_head}}/"'${FluentBitReadFromHead}'"/;s/{{read_from_tail}}/"'${FluentBitReadFromTail}'"/' | kubectl delete -f -
eksctl delete cluster --name eksWordPress --region=us-east-2

Delete policy.

aws iam delete-policy --policy-arn arn:aws:iam::492943873543:policy/polOpenSearch

Delete CloudWatch log groups.

EKS_CLUSTER=eksWordPress
aws logs delete-log-group --log-group-name "/aws/containerinsights/$EKS_CLUSTER/application"
aws logs delete-log-group --log-group-name "/aws/containerinsights/$EKS_CLUSTER/dataplane"
aws logs delete-log-group --log-group-name "/aws/containerinsights/$EKS_CLUSTER/host"
aws logs delete-log-group --log-group-name "/aws/containerinsights/$EKS_CLUSTER/performance"
aws logs delete-log-group --log-group-name "/aws/eks/$EKS_CLUSTER/cluster"

Delete the OpenSearch cluster.

aws opensearch delete-domain --domain-name oswordpress

Ansible: Quick Start Guide for FreeBSD, CentOS and Ubuntu

Kliment Andreev — Mon, 08 Feb 2021 17:04:37 +0000

In this post/howto, I’ll explain how to install Ansible as control and managed node on FreeBSD 12, CentOS 8 and Ubuntu 18. Then, I’ll explain how to create SSH keys so the nodes can communicate and some basic tasks. Then, I’ll show you how to create a playbook to install the latest updates and also install an Apache server with the default settings. Finally, I’ll show an example of how to use variables.

Control and managed nodes

A control node is where you execute all of your Ansible commands and eventually keep your playbooks, configs, inventory etc. It’s pretty much your workstation. A managed node is where the actual playbooks are executed. These are the main concepts and the terminology.
In my case, I have 4 VMs/instances. The main one which is the control node and 3 managed nodes.

CentOS 8

If you want to install Ansible on the control node, you have to install Python 3.x first.

sudo dnf install python3

This will also install pip. Type python3 to test, CTRL-D to exit and then install ansible.

sudo pip3 install ansible

Type ansible to test.

Ubuntu

Ubuntu comes with python installed, but not with pip. Install pip with:

sudo apt install python3-pip

Then install ansible with:

sudo pip3 install ansible

Type ansible to test.

FreeBSD

FreeBSD comes with python installed but pip is not. Check the version and then install the same pip version.

ls -l /usr/local/bin/python*

If your output is for example python37, install the same pip version.

pkg install py37-pip

Then install ansible.

pip install ansible

Type ansible to test.

SSH keys

While Ansible can use standard *nix username/password authentication, it’s recommended that you use SSH keys to communicate from control node to the managed nodes. For that, you’ll have to create your SSH keys. Let’s say you have an account on your control node and the username is admin. You also want to use the user ansible on the managed nodes. It really doesn’t matter what usernames you are going to choose. You can always override the keys to use, but in this case, I’ll create a key on the control node and send it to all managed nodes.
On the control node, regardless of your OS, do:

ssh-keygen -b 4096

This will create a subfolder .ssh with two files: id_rsa and id_rsa.pub. The former is your private key and the later is your public key.
Copy the key to your managed nodes.

ssh-copy-id ansible@nodeX.andreev.local

This will copy my key for the user admin to the node X under the ansible user. Then test the passwordless connection.

ssh ansible@nodeX.andreev.local

Mind that the use of FQDN (nodex.andreev.local) vs. hostname (nodex) is important. For SSH these two are different. Once you log to the managed node, the node will be added to the list of known hosts in the file .ssh/known_hosts.

Inventory and the config file

Ansible uses the inventory files to execute an action against using the options and parameters specified in the config file. The config file is /etc/ansible/ansible.cfg for CentOS and Ubuntu and /usr/local/etc/ansible/ansible.cfg for FreeBSD. You can also put the inventory file in the same directory and name it as you wish, but you have to specify the inventory as a parameter on the command line or an entry in the config file. In addition, you can have your config file in your current directory or under the .ansible directory in your home folder. ansible.cfg in the current directory has a precedence over .ansible.cfg in the home directory which has a precedence over /etc/ansible/ansible.cfg. Here is how that looks.
Let’s list all the inventory.

ansible --list-hosts all

You’ll get a message that there is no inventory file. Let’s create one, we’ll name it inventory.txt.

[freebsd]
node1.andreev.local
[centos]
node2.andreev.local
[ubuntu]
node3.andreev.local
[bsd]
node1.andreev.local
[linux]
node2.andreev.local
node3.andreev.local

If we specify the inventory file, we’ll get this.

ansible --list-hosts all -i inventory.txt
  hosts (3):
    node1.andreev.local
    node2.andreev.local
    node3.andreev.local

If we create a config file, we can tell ansible where to look for the inventory. Create a file ansible.cfg in the same directory.

[defaults]
inventory=/home//inventory.txt

If you do ansible –list-hosts all now, you’ll get the same result as before, but without specifying the inventory file.
Or something like this.

ansible all -m shell -a "uname -a"
node1.andreev.local | CHANGED | rc=0 >>
FreeBSD node1.andreev.local 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64
node2.andreev.local | CHANGED | rc=0 >>
Linux node2.andreev.local 4.18.0-193.14.2.el8_2.x86_64 #1 SMP Sun Jul 26 03:54:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
node3.andreev.local | CHANGED | rc=0 >>
Linux node3.andreev.local 4.15.0-129-generic #132-Ubuntu SMP Thu Dec 10 14:02:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

By default, ansible runs on the managed node with the currently logged user that executes the playbook on the control node. If you get an error saying that the previous command cannot connect to the host, you have to specify the same user that you used when you test the connection with ssh -user-@managednode. So, edit ansible.cfg and add this line.

remote_user=

We can target only the group linux which consists of Linux hosts only in the inventory file.

ansible linux -m shell -a "date"
node3.andreev.local | CHANGED | rc=0 >>
Fri Jan  8 15:23:40 UTC 2021
node2.andreev.local | CHANGED | rc=0 >>
Fri Jan  8 10:23:40 EST 2021

Playbooks

The playbooks are the blueprints of the automation tasks. Instead of running the ansible command to execute each task separately, we combine these tasks in a YAML file and execute them sequentially. Here are 3 playbooks that update each of our managed nodes. There is no update module for FreeBSD, so we use the shell command.

# freebsd-update.yml
---
  - hosts: freebsd
    become: yes
    tasks:
      - name: Fetch all packages
        shell: freebsd-update fetch
      - name: Install FreeBSD updates
        shell: freebsd-update install
      - name: Reboot
        reboot:

For CentOS we’ll use yum.

# centos-update.yml
---
  - hosts: centos
    become: yes
    tasks:
      - name: Update all packages
        yum: name=* state=latest
      - name: Reboot
        reboot:

…and for Ubuntu we’ll use apt.

# ubuntu-update.yml
---
  - hosts: ubuntu
    become: yes
    tasks:
      - name: Update all packages
        apt: name=* state=latest
      - name: Reboot
        reboot:

Save these files with a YAML extension and you can execute them with the following command.

ansible-playbook

All of them will probably fail. That’s because your ansible user on the managed nodes is required a password when executing a sudo command. In order to fix that, you’ll have to add a line in the sudoers file. Edit this file using the visudo command.

visudo

…and then add this line right before the @includedir <...> which is the last line in the file.

ansible ALL=(ALL) NOPASSWD:ALL

Where ansible is the user that runs the playbooks on the managed nodes. FreeBSD doesn’t come with sudo preinstalled, so you’ll have to install it first on the managed node.

pkg install sudo

These playbooks will update the OS and the packages for the Linux. For FreeBSD, it will update only the OS. Here is another example of playbooks that will install Apache server in a default configuration and change the ServerName and ServerAdmin lines. We’ll also install PHP and test our server.
If you have a firewall enabled, make sure you open it up first on CentOS. Ubuntu and FreeBSD do not come with the firewall enabled.

firewall-cmd --zone=public --permanent --add-service=http
firewall-cmd --reload

For FreeBSD, the playbook looks like this.

# freebsd-apache.yml
---
  - hosts: freebsd
    become: yes
    tasks:
      - name: Install apache and php
        pkgng:
          name:
            - apache24
            - php74
            - mod_php74
          state:  present
      - name: Start on reboot
        service: name=apache24 enabled=yes
      - name: Copy index.php
        copy:
          src: ../files/index.php
          dest: /usr/local/www/apache24/data
          mode: 0755
      - name: Copy mod_php.conf
        copy:
          src: ../files/mod_php.conf
          dest: /usr/local/etc/apache24/modules.d
          mode: 0755
      - name: Start apache now
        service: name=apache24 state=started

For CentOS, it looks like this.

# centos-apache.yml
---
  - hosts: centos
    become: yes
    tasks:
      - name: Install apache and php
        yum:
          name:
            - httpd
            - php
          state:  present
      - name: Start apache now and on reboot
        service: name=httpd state=started enabled=yes
      - name: Copy index.php
        copy:
          src: ../files/index.php
          dest: /var/www/html
          mode: 0755

…and for Ubuntu it looks like this.

# ubuntu-apache.yml
---
  - hosts: ubuntu
    become: yes
    tasks:
      - name: Install apache and php
        apt:
          name:
            - apache2
            - php
          state:  present
      - name: Start apache now and on reboot
        service: name=apache2 state=started enabled=yes
      - name: Copy index.php
        copy:
          src: ../files/index.php
          dest: /var/www/html
          mode: 0755

You will also need these two files in a directory called files. In my case it’s one level above the directory where I keep my playbooks.
index.php

001_mod-php.conf


    DirectoryIndex index.php index.html
    
        SetHandler application/x-httpd-php
    
    
        SetHandler application/x-httpd-php-source

The index.php file is the standard test file to test the PHP distributions and the 001_mod-php.conf is needed for FreeBSD only. As you can see the playbooks differ quite a bit for these three OSes. Once you deploy the playbooks, you can test the result by going to http://[nodeX]/index.php.
Looks like this.

Service handlers

Service handlers are used only when a change is made on the managed node. For example, we can restart a service only if the config file was changed. If the file is not changed, then there is no need to restart. Here is an example of a service handler. We’ll restart postfix service only if the main.cf file was changed.

# service-handler.yml
---
- hosts: centos
  become: yes
  tasks:
  - name: Configure main.cf
    lineinfile:
      path: /etc/postfix/main.cf
      regexp: ^#mydomain
      line: 'mydomain = example.com'
    notify: restart postfix

  handlers:
  - name: restart postfix
    service: name=postfix state=restarted

Make sure that the name for the handler is the same (lines 11 and 14), so Ansible knows what service handlers is referred.

Variables

When ansible runs a playbook on a managed node, the first task is to gather info about the managed node. The info is a bunch of settings that we can use in our playbooks. For example, if you execute the following command, you can see the IP address, the CPU model, python version etc.

ansible -m setup

We can use these settings and use them as variables if we need them. For example, this playbook displays the hostname and the IP.

# showip.yml
---
  - hosts: freebsd
    become: yes
    tasks:
      - name: Show the IP address
        debug:
          msg: "The hostname is {{ inventory_hostname}}  and the IP is {{ansible_default_ipv4.address }}"

And if you run the playbook, you’ll see something like this.

TASK [Show the IP address] *********************************************************************************************
ok: [node1.andreev.local] => {
    "msg": "The hostname is node1.andreev.local  and the IP is 192.168.1.211"
}

Here is another example of using variables. In this case, we’ll specify a file and change the ownership and the mode.

# owner.yml
---
  - hosts: centos
    become: yes
    vars:
      filename: "/var/www/html/index.php"
    tasks:
      - name: Change the owner of the file
        file:
          path: "{{ filename }}"
          owner: apache
          group: apache
          mode: '0755'

In case we want to assign an output to a variable, we’ll have to use the keyword register. Here is an example of how to get the output from a command and print it on the screen with the keyword debug.

# variables.yml
---
  - hosts: freebsd

    tasks:
    - name: Get the uptime manually
      command: uptime
      register: var_uptime

    - name: Print the uptime
      debug:
        msg: The uptime is "{{ var_uptime }}"

Roles

Roles let you automatically load related vars_files, tasks, handlers, and other Ansible artifacts based on a known file structure. Once you group your content in roles, you can easily reuse them and share them with other users. The idea is to separate the tasks, handlers and vars in different files. Let’s see this playbook for example. It changes a line in main.cf file, restarts postfix and copies a file under the postfix main directory.

# roles.yml
---
  - hosts: centos
    become: yes

    vars:
      filevd: "/etc/postfix/virtual_domains"
      cfgpostfix: "/etc/postfix/main.cf"


    tasks:
    - name: Configure main.cf
      lineinfile:
        path: "{{ cfgpostfix }}"
        regexp: ^#mydomain
        line: 'mydomain = example.com'
      notify: restart postfix
    - name: Copy virtual_domains
      copy:
        src: ../files/virtual_domains
        dest: "{{ filevd }}"
        mode: 0755

    handlers:
    - name: restart postfix
      service: name=postfix state=restarted

We can rewrite this file by separating the tasks, variables, files and handlers. Run this command.

ansible-galaxy role init postfix

If you look at the file/directory structure of the newly created directory postfix, it looks like this.

tree postfix
postfix
├── defaults
│   └── main.yml
├── files
├── handlers
│   └── main.yml
├── meta
│   └── main.yml
├── README.md
├── tasks
│   └── main.yml
├── templates
├── tests
│   ├── inventory
│   └── test.yml
└── vars
    └── main.yml

Create a file virtual_domains under the files directory.

echo "mydomain.com" > postfix/files/virtual_domains

Move the config for vars, handlers and tasks in the separate main.yml files. For example, this is how my files look like.
postfix/vars/main.yml

---
# vars file for postfix

  filevd: "/etc/postfix/virtual_domains"
  cfgpostfix: "/etc/postfix/main.cf"

As you can see the keyword vars: does not exists. Ansible knows that this file is for vars so there is no need to enter the vars keyboard.
postfix/tasks/main.yml

---
# tasks file for postfix

  - name: Configure main.cf
    lineinfile:
      path: "{{ cfgpostfix }}"
      regexp: ^#mydomain
      line: 'mydomain = example.com'
    notify: restart postfix
  - name: Copy virtual_domains
    copy:
      src: ../files/virtual_domains
      dest: "{{ filevd }}"
      mode: 0755

postfix/handlers/main.yml

---
# handlers file for postfix

  - name: restart postfix
    service: name=postfix state=restarted

Finally, create a file called something.yml that will be your main file. This file has to be outside the postfix directory structure.
In my case it looks like this.
something.yml

# something.yml
---
  - hosts: centos
    become: yes
    roles:
      - postfix

Now, if you execute this playbook, ansible will automatically execute the rest of the dependant playbooks as well.

ansible-playbook something.yml

Error handling

Sometimes we want certain changes to be ignored. Sometimes, we know the behavior of certain commands and we know that they might return non-zero code and we want that ignored. For example, consider this part of a playbook.

- hosts: centos
  tasks: 
  - name: Type something that will fail
    command: thiscommanddoesntexist
    ignore_errors: yes

  - name: Run command remotely
    command: /usr/local/bin/somecommand
    register: cmd_result
    changed_when: cmd_result == 2

Ansible would report a task as changed as long as the command (or) script gives zero return code.
In the first part, we know that the task will fail, but we decide to ignore it using the keyword ignore_errors. No matter what the command returns, ignore_errors: yes will never report to ansible that the command failed.
In the second command we can ignore the error based on the output of the command. For example, if the output is 2, the the error will be ignored. If cmd_result is not equal to 2, the task will be marked as changed.
So whenever this condition is true, the task will be marked as changed.

Ansible Vault

Ansible Vault encrypts variables and files so you can protect sensitive content such as passwords or keys rather than leaving it visible as plaintext in playbooks or roles.
First, you have to create a vaulted file where we’ll store the passwords. When you run this command it will ask you to create a password and then an empty file will show up.

ansible-vault create secrets.yml

Add some passwords there and save the file.

mysql_pwd: "DifficultPassword"
ht_pwd: "PasswordXYZ"

If you look at the file now, you’ll see that it’s encrypted and you can’t see the passwords anymore.
If you want to edit the file do ansible-vault edit secrets.yaml and enter the vault password.
Create a small playbook that displays the password.

# centos-vault.yml
---
- hosts: centos
  vars_files:
    - secrets.yml

  tasks:
  - name: Show mysql pwd
    debug:
      msg: "{{ mysql_pwd }}"

If you run the playbook now, ansible will throw an error saying ERROR! Attempting to decrypt but no vault secrets found. You have to specify the parameter –ask-vault-pass and enter the vault password when prompted.

Prompts

In case you need to pause the playbook execution and ask the user for some input such as confirmation or password, use prompts.
If you want the output to echo, use private: no, otherwise what you type won’t show up on the screen. Here is an example of a playbook that asks you to confirm if a file needs to be copied to the node. If you type yes and hit enter, the file will be copied, otherwise it won’t.

# centos-prompt.yml
---
- hosts: centos
  become: yes

  vars_prompt:
    name: upload
    private: no
    prompt: "Do you want to upload xyz.txt?"

  tasks:
  - name: Upload xyz.txt
    copy:
      src: xyz.txt
      dest: /var/log
    when: upload == "yes"

Useful options

ansible-playbook  --syntax-check # checks the syntax of the playbook
ansible-playbook  --check # does a dry run, reports the will-be changes, but the playbook is not executed
ansible-playbook  --step # ask to confirm each-step

Visual Studio and BitBucket

Kliment Andreev — Sat, 11 Jan 2020 12:43:11 +0000

In this post I’ll explain how to use Visual Studio and BitBucket. I host my own BitBucket server, but it should be the same.
First, create a BitBucket repository and name it. The repo will have to be the same name as the project in Visual Studio.
So, in my case, I’ll create a repo called test123.

Then, open Visual Studio 2019 and choose Continue without code from the bottom right.

If you don’t have the BitBucket plugin, go to Extensions | Manage Extensions from the menu and install Visual Studio Bitbucket Extension. Once installed, you’ll have to close and re-open Visual Studio.

Go to View | Team Explorer from the menu and click Login for the BitBucket exptension.

Depending on your BitBucket server (cloud or on-prem), enter the necessary parameters and click to Connect.

Once logged in, click on Clone.

Select the repo and click Clone.

Create a new project (File | New | Project) and name it the same as the repo.

Click Next and name the project the same again, then click Create.

You can push the initial code now. First click File | Save All. Then in Team Explorer, click where it says Connect and switch it to Change.

Type something for the first commit and click on Commit All and Push.

If you go back to BitBucket, you’ll see your code there.

AWS: Deploy nginx HA cluster with Packer, Terraform and Ansible in a new VPC environment

Kliment Andreev — Sun, 29 Dec 2019 23:36:48 +0000

In this post, I’ll explain how to create a whole environment consisting of a new VPC, two public subnets, two private subnets, a bastion host and two application hosts that run nginx web server. These application hosts will be behind an application load balancer. The purpose of the bastion host in the public subnet is to be able to access the application hosts which are in the private subnet and have no public access directly. We’ll use the official CentOS 7 image from the marketplace, make some modifications with packer and bake our own image that has a local ansible. Once these servers are deployed, they’ll already have nginx installed and started and a fully functional template website from GitHub (courtesy of free-css.com). This is how it looks like in a diagram.

Before you start, I suggest you create a directory where you are going to place all these files. Make sure to execute each command from that directory unless told otherwise.

Packer and ansible

Packer is super easy to install. Just get the executable and place it somewhere. Once ready to create an image, specify your AWS access key and secret key. The IAM associated with these keys should be able to create an AMI. This is my packer file.

{
  "variables": {
    "aws_access_key": "",
    "aws_secret_key": "",
    "ami_name": "centos7-USWTA",
    "region": "us-east-1",
    "source_ami": "ami-02eac2c0129f6376b",
    "instance_type": "t2.micro"
  },
  "builders": [{
    "access_key": "{{user `aws_access_key`}}",
    "secret_key": "{{user `aws_secret_key`}}",
    "ami_name": "{{user `ami_name`}}.{{timestamp}}",
    "region" : "{{user `region`}}",
    "source_ami": "{{user `source_ami`}}",
    "instance_type": "{{user `instance_type`}}",
    "ssh_username": "centos",
    "type": "amazon-ebs"
  }],
  "provisioners": [
    {
     "type": "shell",
      "inline": "sudo yum -y install epel-release"
    },
    {
      "type": "shell",
      "inline": "sudo yum -y install ansible"
    },
    {
      "type": "file",
      "source": "ansible.yml",
      "destination": "/home/centos/ansible.yml"
    },
    {
      "type": "ansible-local",
      "playbook_file": "ansible.yml"
    }
  ]
}

You can specify your access keys in the script, but you don’t want to. You can specify the name of the AMI image, the region and the instance type. Do not change the source AMI variable, that’s the id of the CentOS official image. In the provisioners section, we’ll install the epel repo so we can install ansible. In addition, packer will get a local ansible playbook and place it under /home/centos in the new image. So, you need the ansible.yml playbook in the same directory as packer.json file. This is the ansible playbook. You definitely won’t do it this way in production. In my case I rename the original HTML root and clone a github repo. You’ll probably want to clone the site in a separate directory and use a modified nginx.conf file that suits your needs.

---
- hosts: localhost
  become: true

  tasks:
    - name: install nginx
      yum:
        name: nginx
        state: present

    - name: start nginx
      service:
        name: nginx
        state: started
        enabled: yes

    - name: install git
      yum:
        name: git
        state: present

    - name: remove the original html
      shell: |
        mv /usr/share/nginx/html /usr/share/nginx/html.old

    - name: clone web template
      git:
        repo: https://github.com/klimenta/webtemplate
        dest: /usr/share/nginx/html
        version: master

This playbook installs nginx, git and clones a website template from my github account. So, with both files ready (ansible.yml and packer.json), you can create your own image. Replace your own access key and secret and run this command.

packer build -var 'aws_access_key=AKIA.....' -var 'aws_secret_key=Dp123Sm4.....' packer.json

It takes about 5-6 minutes to bake the AMI. If you go to your AWS account, you’ll see it there.

With our image ready, we are ready to deploy the infrastructure environment using Terraform. But before we do that, we want to create the keys that we will use to log in to our servers. In a production environment you’ll choose to have separate keys for the bastion host and the app hosts, but for the sake of clarity, we’ll use one key to rule them all.
Type this command and hit Enter twice to skip the passphrase.

ssh-keygen -t rsa -f $PWD/keyUSWTA  -b 2048

You’ll have two files, keyUSWTA and keyUSWTA.pub. keyUSWTA is your private key, don’t share it with anyone. The public key will be used to spin-up the instances from our image. If you use putty instead of ssh, you have to convert your private key in a ppk format.

Terraform

Terraform is also super easy to install. Just copy the file somewhere in your path and you are ready to go. For Terraform we’ll have 4 separate files to build the infra. The first file is variables.tf. This is where we define all the variables that are in use. The second file is terraform.tfvars. This file is where we assign values to our variables. The first file can go to GitHub or any other version control system, the second one can not, especially if you have passwords and keys there. But it’s up to you. The third file is outputs.tf. Terraform dumps the values of this file when it’s done provisioning. For example, you want to see what’s your external IP from the instance you just created instead of going to AWS console to find out. And the fourth file is main.tf where we define our infrastructure. So, here they are.
variables.tf

variable "aws_region" {}
variable "aws_az1" {}
variable "aws_az2" {}
variable "aws_profile" {}
variable "vpc_cidr" {}
variable "sub_private1_cidr" {}
variable "sub_private2_cidr" {}
variable "sub_public1_cidr" {}
variable "sub_public2_cidr" {}
variable "bastion_ami_id" {}
variable "instance_type" {}
variable "ip_address" {}

terraform.tfvars

aws_profile = "default"
aws_region  = "us-east-1"
aws_az1 = "us-east-1a"
aws_az2 = "us-east-1b"
vpc_cidr = "192.168.200.0/23"
sub_public1_cidr = "192.168.200.0/25"
sub_public2_cidr = "192.168.200.128/25"
sub_private1_cidr = "192.168.201.0/25"
sub_private2_cidr = "192.168.201.128/25"
bastion_ami_id = "ami-02eac2c0129f6376b"
instance_type = "t2.micro"
ip_address = "1.2.3.4/32"

This means I’ll use my default profile for AWS CLI, the region for deployment is us-east-1, then the CIDRs for the VPC and the public and private subnets. The bastion host is based on the AMI there, which is CentOS 7 and t2.micro. And that host will have access from 1.2.3.4 IP only. Change to suit your needs.
outputs.tf

output "aws_lb" {
  value = aws_lb.albUSWTA.dns_name
}
output "aws_ip" {
  value = aws_instance.ec2USWTABastion.public_ip
}
output "aws_app1_ip" {
  value = aws_instance.ec2USWTAApplication1.private_ip
}
output "aws_app2_ip" {
  value = aws_instance.ec2USWTAApplication2.private_ip
}

These are the values that will be dumped on your screen, once Terraform completes. You’ll see the URL of your load balancer which is how you’ll access your website. Then, the public IP of the bastion host and the private IPs of your app instances in the private subnet.
And finally, this is the main.tf file that provisions the infrastructure. Click the (+) sign to expand.

provider "aws" {
  region  = var.aws_region
  profile = var.aws_profile
}

# VPC
resource "aws_vpc" "vpcUSWTA" {
  cidr_block = var.vpc_cidr

  tags = {
    Name = "vpcUSWTA"
  }
}

# Public subnet 1
resource "aws_subnet" "subPublic1" {
  vpc_id = aws_vpc.vpcUSWTA.id
  cidr_block = var.sub_public1_cidr
  availability_zone = var.aws_az1

  tags = {
    Name = "subPublic - USWTA - 1"
  }
}

# Public subnet 2
resource "aws_subnet" "subPublic2" {
  vpc_id = aws_vpc.vpcUSWTA.id
  cidr_block = var.sub_public2_cidr
  availability_zone = var.aws_az2

  tags = {
    Name = "subPublic - USWTA - 2"
  }
}

# Private subnet 1
resource "aws_subnet" "subPrivate1" {
  vpc_id = aws_vpc.vpcUSWTA.id
  cidr_block = var.sub_private1_cidr
  availability_zone = var.aws_az1

  tags = {
    Name = "subPrivate - USWTA - 1"
  }
}

# Private subnet 2
resource "aws_subnet" "subPrivate2" {
  vpc_id = aws_vpc.vpcUSWTA.id
  cidr_block = var.sub_private2_cidr
  availability_zone = var.aws_az2

  tags = {
    Name = "subPrivate - USWTA - 2"
  }
}

# Internet gateway
resource "aws_internet_gateway" "igwUSWTA" {
  vpc_id = aws_vpc.vpcUSWTA.id

  tags = {
    Name = "igwInternetGateway - USWTA"
  }
}

# Key pair
resource "aws_key_pair" "keyUSWTA" {
  key_name   = "Key for USWTA"
  public_key = file("${path.module}/keyUSWTA.pub")
}

# Security group for bastion host
resource "aws_security_group" "sgUSWTABastion" {
  name = "sgUSWTA - Bastion"
  description = "Allow access on port 22 from restricted IP"
  vpc_id = aws_vpc.vpcUSWTA.id

  ingress {
    from_port = 22
    to_port = 22
    protocol = "tcp"
    cidr_blocks = [var.ip_address]
  }

  egress {
    from_port       = 0
    to_port         = 0
    protocol        = "-1"
    cidr_blocks     = ["0.0.0.0/0"]
  }

  tags = {
    Name = "Allow access on port 22 from my IP"
  }
}

# Security group for application load balancer
resource "aws_security_group" "sgUSWTAALB" {
  name = "sgUSWTA - ALB"
  description = "Allow access on port 80 from everywhere"
  vpc_id = aws_vpc.vpcUSWTA.id

  ingress {
    from_port = 80
    to_port = 80
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port       = 0
    to_port         = 0
    protocol        = "-1"
    cidr_blocks     = ["0.0.0.0/0"]
  }

  tags = {
    Name = "Allow HTTP access from everywhere"
  }
}

# Security group for application hosts
resource "aws_security_group" "sgUSWTAApplication" {
  name = "sgUSWTA - Application"
  description = "Allow access on ports 22 and 80"
  vpc_id = aws_vpc.vpcUSWTA.id

  ingress {
    from_port = 22
    to_port = 22
    protocol = "tcp"
    cidr_blocks = [var.sub_public1_cidr]
  }

  ingress {
    from_port = 80
    to_port = 80
    protocol = "tcp"
    security_groups = [aws_security_group.sgUSWTAALB.id]
  }

  egress {
    from_port       = 0
    to_port         = 0
    protocol        = "-1"
    cidr_blocks     = ["0.0.0.0/0"]
  }

  tags = {
    Name = "Allow access on ports 22 and 80"
  }
}

# Get the latest image
data "aws_ami" "image" {
  most_recent = true
  owners = ["self"]
  filter {
    name = "name"
    values = ["centos7-USWTA*"]
  }
}

# ec2 instance - bastion host
resource "aws_instance" "ec2USWTABastion" {
  ami = var.bastion_ami_id
  instance_type = var.instance_type
  key_name = aws_key_pair.keyUSWTA.key_name
  vpc_security_group_ids = [aws_security_group.sgUSWTABastion.id]
  subnet_id = aws_subnet.subPublic1.id
  associate_public_ip_address = true

  root_block_device {
    delete_on_termination = true
  }

  tags = {
    Name = "ec2USWTA - Bastion"
  }
}

# ec2 instance - app host 1
resource "aws_instance" "ec2USWTAApplication1" {
  ami = data.aws_ami.image.id
  instance_type = var.instance_type
  key_name = aws_key_pair.keyUSWTA.key_name
  vpc_security_group_ids = [aws_security_group.sgUSWTAApplication.id]
  subnet_id = aws_subnet.subPrivate1.id
  associate_public_ip_address = false

  root_block_device {
    delete_on_termination = true
  }

  tags = {
    Name = "ec2USWTA - Application - 1"
  }
}

# ec2 instance - app host 2
resource "aws_instance" "ec2USWTAApplication2" {
  ami = data.aws_ami.image.id
  instance_type = var.instance_type
  key_name = aws_key_pair.keyUSWTA.key_name
  vpc_security_group_ids = [aws_security_group.sgUSWTAApplication.id]
  subnet_id = aws_subnet.subPrivate2.id
  associate_public_ip_address = false

  root_block_device {
    delete_on_termination = true
  }

  tags = {
    Name = "ec2USWTA - Application - 2"
  }
}

# Elastic IP for the NAT gateway
resource "aws_eip" "eipUSWTA" {
  vpc = true

  tags = {
    Name = "eipUSWTA"
  }
}

# NAT gateway
resource "aws_nat_gateway" "ngwUSWTA" {
  allocation_id = aws_eip.eipUSWTA.id
  subnet_id     = aws_subnet.subPublic1.id

  tags = {
    Name = "ngwUSWTA"
  }
}

# Add route to Internet to main route table
resource "aws_route" "rtMainRoute" {
  route_table_id = aws_vpc.vpcUSWTA.main_route_table_id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id = aws_nat_gateway.ngwUSWTA.id
}

# Create public route table
resource "aws_route_table" "rtPublic" {
  vpc_id = aws_vpc.vpcUSWTA.id

  tags = {
    Name = "rtPublic - USWTA"
  }
}

# Add route to Internet to public route table
resource "aws_route" "rtPublicRoute" {
  route_table_id = aws_route_table.rtPublic.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id = aws_internet_gateway.igwUSWTA.id
}

# Associate public route table with public subnet 1
resource "aws_route_table_association" "rtPubAssoc1" {
  subnet_id   = aws_subnet.subPublic1.id
  route_table_id = aws_route_table.rtPublic.id
}

# Associate public route table with public subnet 2
resource "aws_route_table_association" "rtPubAssoc2" {
  subnet_id   = aws_subnet.subPublic2.id
  route_table_id = aws_route_table.rtPublic.id
}

# Application Load Balancer
resource "aws_lb" "albUSWTA" {
  name               = "albUSWTA"
  internal           = false
  load_balancer_type = "application"
  subnets            = [aws_subnet.subPublic1.id, aws_subnet.subPublic2.id]
  security_groups = [aws_security_group.sgUSWTAALB.id]

  tags = {
    Name = "Application Load Balancer for USWTA"
  }
}

# Target group
resource "aws_lb_target_group" "tgUSWTA" {
  name = "tgUSWTA"
  port = "80"
  protocol = "HTTP"
  vpc_id = aws_vpc.vpcUSWTA.id

  health_check {
    healthy_threshold   = 5
    unhealthy_threshold = 2
    timeout             = 5
    path                = "/index.html"
    port = 80
    matcher = "200"
    interval            = 30
  }
}

# Listener
resource "aws_lb_listener" "lisUSWTA" {
  load_balancer_arn = aws_lb.albUSWTA.arn
  port = "80"
  protocol = "HTTP"

  default_action {
    type = "forward"
    target_group_arn = aws_lb_target_group.tgUSWTA.arn
  }
}

# Add instance 1 to target group
resource "aws_lb_target_group_attachment" "tgaUSWTA1" {
  target_group_arn = aws_lb_target_group.tgUSWTA.arn
  target_id        = aws_instance.ec2USWTAApplication1.id
  port             = "80"
}

# Add instance 2 to target group
resource "aws_lb_target_group_attachment" "tgaUSWTA2" {
  target_group_arn = aws_lb_target_group.tgUSWTA.arn
  target_id        = aws_instance.ec2USWTAApplication2.id
  port             = "80"
}

The comments tell you what is being done. Terraform has a great documentation so if you google any of the resources created you’ll see ample documentation. So, make changes to suit your needs and once you have everything ready initialize the terraform so it downloads the necessary plugins.

terraform init

Pay special attention to lines 68-72, it expects the keys to be the same exact filename and 156-164 it looks for an image that matches that name. That’s the AMI that we create with packer. Then create the infra. You have to type yes to proceed.

terraform apply

Once completed (in my case it took 4 minutes and 30 seconds), you’ll see the variables from the outputs.tf.

Apply complete! Resources: 25 added, 0 changed, 0 destroyed.

Outputs:

aws_app1_ip = 192.168.201.91
aws_app2_ip = 192.168.201.187
aws_ip = 54.145.191.102
aws_lb = albUSWTA-1695850209.us-east-1.elb.amazonaws.com

Go to http://albUSWTA-1695850209.us-east-1.elb.amazonaws.com which is the output from the LB from above and voila, your website is up and running on two HA servers.

Go to your AWS console and you’ll see all of your resources there. e.g. the instances and the targets behind the load balancers.

Once you are done playing, destroy the resources.

terraform destroy

DevOps: CI/CD using git, Jenkins, Docker, Kubernetes

Kliment Andreev — Sun, 03 Nov 2019 21:34:00 +0000

In this post I’ll explain how to create a Node.js chat program, put the source on Github and then using Jenkins we’ll deploy the code to Docker Hub and from there to Docker containers that are managed by Kubernetes. There are many prerequisites for this.

GitHub

DockerHub

See my other posts on how to install Jenkins and Kubernetes cluster.

The chat program

For this purpose I’ve found a simple Node.js chat program that uses Express framework. The program can be found here, but we’ll modify it a bit. On a Linux workstation (I use CentOS, so some commands might differ), log as your regular user and create some necessary directories, a directory called chitchat and two subdirectories, public and test and two subdirectories for public, called js and css.

mkdir -p chitchat/test
mkdir -p chitchat/public/{js,css} 
cd chitchat

Create the source files for the chat program. First, let’s create the HTML. Click the (+) sign to expand. Just copy and paste everything. The first line (cat) will take care of the file creation.

cat << EOF > index.html


    
        Chit Chat App
        
        
    
    
        
        
            
            
        
        
        
    

EOF

Then, the Node.js code.

cat << EOF > index.js
var express = require('express');
var app = express();
var http = require('http').Server(app);
var io = require('socket.io')(http);
var port = process.env.PORT || 4141;
var path = require('path');

app.get('/', function(req, res){
    res.sendFile(__dirname + '/index.html');
});

app.use(express.static(path.join(__dirname + '/public')));


io.on('connection', function(socket) {

    socket.on('user_join', function(data) {
        this.username = data;
        socket.broadcast.emit('user_join', data);
    });

    socket.on('chat_message', function(data) {
        data.username = this.username;
        socket.broadcast.emit('chat_message', data);
    });

    socket.on('disconnect', function(data) {
        socket.broadcast.emit('user_leave', this.username);
    });
});

http.listen(port, function() {
    console.log('Listening on *:' + port);
});
EOF

Then, the actual code for the app.

cat << EOF > public/js/app.js
const form = document.querySelector("form");
const input = document.querySelector(".input");
const messages = document.querySelector(".messages");
const username = prompt("Please enter a nickname: ", "");
const socket = io();

form.addEventListener("submit", function(event) {
        event.preventDefault();

        addMessage(username + ": " + input.value);

        socket.emit("chat_message", {
                        message: input.value
        });

        input.value = "";
        return false;
}, false);

socket.on("chat_message", function(data) {
        addMessage(data.username + ": " + data.message);
});

socket.on("user_join", function(data) {
        addMessage(data + " just joined the chat!");
});

socket.on("user_leave", function(data) {
        addMessage(data + " has left the chat.");
});

addMessage("You have joined the chat as '" + username  + "'.");
socket.emit("user_join", username);

function addMessage(message) {
        const li = document.createElement("li");
        li.innerHTML = message;
        messages.appendChild(li);
        window.scrollTo(0, document.body.scrollHeight);
}
EOF

And finally the CSS stylesheet.

cat << EOF > public/css/styles.css
body {
    margin: 0;
    font-family: sans-serif;
}

form {
    position: fixed;
    bottom: 0;
    left: 0;
    width: 100%;
    display: flex;
    box-sizing: border-box;
    padding: 0.25rem;
}

form input {
    border: 0;
    padding: 0.5rem;
    width: 100%;
    outline: 0;
    margin-right: 0.5rem;
    border-radius: 0.25rem;
    background: #ccc;
}

form button {
    width: 6rem;
    background-color: #1b8c00;
    color: white;
    border: none;
    padding: 0.5rem;
    cursor: pointer;
    border-radius: 0.25rem;
    text-transform: uppercase;
}

form button:hover {
    background-color: #166d01;
}

.messages {
    margin: 0;
    padding: 0;
    margin-bottom: 3rem;
}

.messages li {
    padding: 0.5rem;
}

.messages li:nth-child(odd) {
    background: #eee;
}
EOF

If you don’t have Node.js installed, then install it. If you already have Node.js, skip this step. While still logged as the regular user, sudo to install Node.js and npm and check the versions. Make sure you are in chitchat directory.

curl -sL https://rpm.nodesource.com/setup_10.x | sudo bash -
sudo yum -y install nodejs
node -v && npm -v

Initiate a new project. Make sure you are in chitchat directory.

npm init

Answer the questions for the initialization. Just hit Enter for every question, we’ll replace this file later. Then install Express and socket.io.

npm install -save express
npm install -save socket.io

Finally, start the program. If you have a firewall enabled, you have to poke a hole.

sudo firewall-cmd --add-port=4141/tcp --zone=public --permanent
sudo firewall-cmd --reload
node index.js

The program should listen on port 4141. Open up a browser and access the Node.js app. (http://workstation_ip:4141).
I’ve opened two windows and logged as UserA and UserB to test the functionality. It looks like this.

The GitHub repository

Now that we have the program ready, let’s upload it on GitHub. Log with your account and create a new repository. In my case, I’ll name it chitchat.
On your workstation, initiate the git repo, add the files, do the first commit and push the files to GitHub. It should look like this. Replace your username and the repo name in line 5.

cd chitchat
git init
git add .
git commit -m "First commit."
git remote add origin https://github.com//
git push -u origin master

Jenkins

Install Node.js plugin, Git and Docker

If you installed Jenkins with the default settings, the Git and Docker plugins are probably installed. Node.js plugin is not installed by default so we’ll have to take care of that. Go to Manage Jenkins from the menu and then Manage Plugins. From the Available tab, find the NodeJS plugin and install it.

Again, go to Manage Jenkins, Global Tool Configuration and click on Add NodeJS.

Enter a name and choose the the latest 10.x version. Make sure that the Install automatically is checked. Also, I named my Node.js installation “node” and I’ll reference it as “node” further in the pipeline script. If you name your installation differently, you’ll have to change that in line 4 in the script below.

Again, go to Manage Jenkins and then Manage Plugins in the center. Click on the Installed tab and look for Docker Pipeline plugin and Git plugin. They should be there and already installed (see the grayed checkmark on the left).

If not, click on the Available tab, search for these plugins and install them similarly to what you did with Node.js plugin. Once installed, SSH to the Jenkins server and install Git first. We’ll install it from the source. This is for CentOS 7 only.

yum -y groupinstall "Development Tools"
yum -y install gettext-devel openssl-devel perl-CPAN perl-devel zlib-devel wget
wget https://github.com/git/git/archive/v2.23.0.tar.gz -O /tmp/git.tgz
cd /tmp
tar xzvf git.tgz 
cd git*
make configure
./configure
make install
cd /tmp
rm -Rf git*

Run git ‐‐version to check if it works properly.
Then, install Docker.

yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum -y install docker-ce
systemctl enable docker
systemctl start docker
usermod jenkins -g docker
systemctl restart jenkins

Now, Jenkins can execute Docker commands.

Jenkins pipeline

Once installed from the main menu, in the upper-left corner, click on New Item. Enter the project name and choose Pipeline.

Scroll down until you find the Pipeline script and paste this code. We just want to test pulling the repo from Github. Make sure you replace your username and repo name in line 10.

pipeline {
    agent any

    tools {nodejs "node"}

    stages {

        stage('Cloning Git') {
            steps {
                git 'https://github.com//'
            }
        }
    }
}

It looks like this.

Click on Save and once back, click on Build Now from the menu.

Once completed, you’ll see that everything is running OK.

Click on Console Output to see the actual logs from git.
Now, that we have everything verified, we want to add another step of testing the application. We’ll have to install the pre-requisites for the Node.js and create a test scenario. On the workstation where your project is, create a test scenario. We’ll use mocha and chai for this.

cat << EOF > test/test.js
var chai = require('chai');
var io = require('socket.io-client')
var app = require('../index');
var client = io('http://localhost:4141');

console.log('Server started...Waiting for client connection...');
it('Client connected...', function(done) {
  client.on('connect', function (data) {
    done();
  });
});
EOF

We also need to define the pre-requisites. Change some of the lines here to suit your needs. This is the file that we are replacing, the one that npm init creates.

cat << EOF > package.json
{
  "name": "chitchat",
  "version": "1.0.0",
  "description": "A simple chat program",
  "main": "index.js",
  "scripts": {
    "start": "node index.js",
    "test": "mocha -R spec test/test.js --exit"
  },
  "author": "Kliment Andreev",
  "license": "ISC",
  "dependencies": {
    "express": "^4.17.1",
    "socket.io": "^2.3.0"
  },
  "devDependencies": {
    "chai": "^4.2.0",
    "mocha": "^5.2.0"
  }
}
EOF

As you can see the first file goes under chitchat/test directory and the package.json file is in the root of chitchat directory. Don’t forget to push these files to GitHub.

git add .
git commit -m "Added package.json and test unit"
git push -u origin master

Now, go back to your project in Jenkins (click Configure on the left) and replace the pipeline script with this one. Make sure you change your name and repo in line 10.

pipeline {
    agent any

    tools {nodejs "node"}

    stages {

        stage('Cloning Git') {
            steps {
                git 'https://github.com//'
            }
        }

        stage('Install dependencies') {
            steps {
                sh 'npm install'
            }
        }

        stage('Test') {
            steps {
                sh 'npm test'
            }
        }
    }
}

Click on Build Now and you’ll see that all stages completed fine. But, we want to go a step further and automate the build process anytime we make a change in the source code. In order to do that we need a webhook between GitHub and our Jenkins server. Make sure your Jenkins server is publicly available, otherwise this integration won’t work.
But before we do that, let’s add the script above as a file under our source code for the chat program. Make sure you change your name and repo in line 11. This file goes in the root of the chitchat directory.

cat << EOF > Jenkinsfile
pipeline {
    agent any

    tools {nodejs "node"}

    stages {

        stage('Cloning Git') {
            steps {
                git 'https://github.com//'
            }
        }

        stage('Install dependencies') {
            steps {
                sh 'npm install'
            }
        }

        stage('Test') {
            steps {
                sh 'npm test'
            }
        }
    }
}
EOF

Add this file to Github.

git add .
git commit -m "Added Jenkinsfile"
git push -u origin master

GitHub integration with Jenkins

Log in to Github and from the upper right corner, click on your avatar and click on Settings. Then on the menu on the left side, all the way to the bottom click on Developer settings. Click on Personal access tokens and click on Generate new token.

Give a name to your token (I’ll name mine Jenkins Token) and click on admin:repo_hook.

Click on Generate token after. You’ll get a hex token, click on the clipboard icon next to it. You’ll need this token for the Jenkins configuration.
Go to Jenkins and click on Manage Jenkins on the left followed by Configure System in the middle. Scroll down to the Git section, click on Add GitHub server, type GitHub for Name and next to Credentials click on Add, then Jenkins.

When this window shows up, change the Kind to Secret text. Paste the token under Secret and enter something for ID and Description. Click Add.

Once back, select the credential that we just created and click on Test connection on the right. You should get a message that the connection is fine. Make sure that Manage hooks is checked. Click on Save at the bottom once everything is OK.

Go back to your project in Jenkins and click Configure on the left. Scroll down where the Pipeline is and change the Definition from Pipeline script to Pipeline script from SCM. SCM means Source Code Management. Change SCM to Git, enter the repo URL under Repositories . If your repo is not public, you’ll have to specify some credentials under Credentials. If your repo is public, you can leave them as none.

Then scroll up a bit and check GitHub hook trigger for GitScm polling and then click Save.

If you go back to GitHub, click on your project and all the way on the right click on Settings.

Then click on Webhooks and you’ll see a green checkmark next to your Jenkins URL.

Go to your workstation and edit the index.html file. Change the line 4 for example so instead of Chit Chat App, replace the blank with a hyphen. e.g. Chit Chat to Chit-Chat. Save it and push to GitHib.

git add .
git commit -m "Minor change in index.html"
git push -u origin master

If you go back to Jenkins, you’ll see that there is a build there triggered automatically. This part is CI (Continous Integration). But, let’s go further and expand our CI with creating a Docker image.

Docker image and Docker Hub

In order to create a Docker image, we need a Dockerfile. I won’t explain the meaning of each line in the Dockerfile, so just copy & paste the following snippet in the chitchat directory.

cat << EOF > Dockerfile
FROM node:10
WORKDIR /usr/src/app
COPY package*.json ./

RUN npm install
COPY . .

EXPOSE 4141
CMD [ "npm", "start" ]
EOF

This file will tell Docker exactly what to do when creating the image. But, we also have to tell Jenkins that we need to build the image, test the image and deploy it to Docker Hub. So, your new Jenkinsfile will look like this. Change the URLs in lines 6 and 7 to suit your needs. Before you deploy the script, go to Jenkins, click on Credentials on the left, click on System below, click on Global credentials (unrestricted) in the middle and then Add Credentials on the left. Fill out the form with your Docker Hub credentials. Whatever you put as ID for these username/password pair is what you have to put in line 8 in the script.

And, now copy & paste the script in the chitchat directory. If everything goes well, you’ll see your image deployed in Docker Hub and tagged as whatever your latest build was and the tag latest too.

cat << 'EOF' > Jenkinsfile
pipeline {

    environment {
        dockerregistry = 'https://registry.hub.docker.com'
        dockerhuburl = "klimenta/chitchat"
        githuburl = "klimenta/chitchat"
        dockerhubcrd = 'dockerhub'
    }

    agent any

    tools {nodejs "node"}

    stages {

        stage('Clone git repo') {
            steps {
                git 'https://github.com/' + githuburl
            }
        }

        stage('Install Node.js dependencies') {
            steps {
                sh 'npm install'
            }
        }

        stage('Test App') {
            steps {
                sh 'npm test'
            }
        }

        stage('Build image') {
          steps{
            script {
              dockerImage = docker.build(dockerhuburl + ":$BUILD_NUMBER")
            }
          }
        }

        stage('Test image') {
            steps {
                sh 'docker run -i ' + dockerhuburl + ':$BUILD_NUMBER npm test'
            }
        }

        stage('Deploy image') {
          steps{
            script {
              docker.withRegistry(dockerregistry, dockerhubcrd ) {
                dockerImage.push("${env.BUILD_NUMBER}")
                dockerImage.push("latest")
              }
            }
          }
        }

        stage('Remove image') {
          steps{
            sh "docker rmi $dockerhuburl:$BUILD_NUMBER"
          }
        }
    }
}
EOF

Of course, you’ll have to tell GitHub that there is a change is a file.

git add .
git commit -m "Added Jenkinsfile for Docker"
git push -u origin master

Deployment to Kubernetes using Jenkins

Now that we are finished with the CI (Continuous Integration), it’s time for the CD (Continuous Deployment). We’ll use the existing Docker image that we put on Docker Hub. First thing first, we’ll have to install a plugin for a Kubernetes integration. Go to Manage Jenkins on the left, then click on Manage plugins in the center and click on Available tab. Install the Kubernetes Continuous Deploy plugin.

Once installed and Jenkins is restarted, click on Credentials on the left, then System just below and then click on Global credentials (unrestricted) in the middle. Click on Add Credentials on the left. When this window pops up, change the Kind to Kubernetes configuration (kubeconfig), enter an ID, in my case it’s k8s. Remember the ID, you’ll reference it in the Jenkinsfile later, type a description and select Enter directly. Before you click OK, we’ll have to go to our Kubernetes cluster and log to it.

Once you are logged to your master node, log as the user that you use to manage the k8s cluster and run this command.

cat .kube/config

You’ll get a long list of text. Paste this back in the Jenkins window under Content and click OK. This is how Jenkins will communicate with your Kubernetes cluster. We also need to tell Kubernetes how we want our app deployed. As you know this is done with a YAML file. So, we have to add that YAML file under the chitchat directory.Change the IP in line 25 to match your master k8s node IP. I am deploying a load balancer that will listen on port 8080 and 2 pods in a deployment.

cat << 'EOF' > k8s.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: chitchat
spec:
  replicas: 2
  template:
    metadata:
      name: chitchat
      labels:
        app: chitchat
    spec:
      containers:
      - image: $dockerhuburl:$BUILD_NUMBER
        name: nodejs
---
apiVersion: v1
kind: Service
metadata:
  name: loadbalancer
spec:
  type: LoadBalancer
  externalIPs:
  - 192.168.1.7
  selector:
    app: chitchat
  ports:
  - port: 8080
    targetPort: 4141
EOF

And we also want to tell Jenkins that we are adding an extra step to deploy it to the cluster, so the final Jenkins file looks like this.

cat << 'EOF' > Jenkinsfile
pipeline {

  environment {
    dockerregistry = 'https://registry.hub.docker.com'
    dockerhuburl = 'klimenta/chitchat'
    githuburl = 'klimenta/chitchat'
    dockerhubcrd = 'dockerhub'
    dockerImage = ''
  }

  agent any

  tools {nodejs "node"}

  stages {

    stage('Clone git repo') {
      steps {
         git 'https://github.com/' + githuburl
      }
    }

    stage('Install Node.js dependencies') {
      steps {
        sh 'npm install'
      }
    }

    stage('Test App') {
        steps {
            sh 'npm test'
        }
    }

    stage('Build image') {
      steps{
        script {
          dockerImage = docker.build(dockerhuburl + ":$BUILD_NUMBER")
        }
      }
    }

    stage('Test image') {
      steps {
        sh 'docker run -i ' + dockerhuburl + ':$BUILD_NUMBER npm test'
      }
    }

    stage('Deploy image') {
      steps{
        script {
          docker.withRegistry(dockerregistry, dockerhubcrd ) {
            dockerImage.push("${env.BUILD_NUMBER}")
            dockerImage.push("latest")
          }
        }
      }
    }

    stage('Remove image') {
      steps{
        sh "docker rmi $dockerhuburl:$BUILD_NUMBER"
      }
    }

    stage('Deploy k8s') {
      steps {
        kubernetesDeploy(
          kubeconfigId: 'k8s',
          configs: 'k8s.yaml',
          enableConfigSubstitution: true
        )
      }
    }
  }
}
EOF

Trigger the change.

git add .
git commit -m "Added Jenkinsfile for kubernetes and yaml"
git push -u origin master

After a minute or so, you’ll see your pods running and if you go to your master node on port 8080 you’ll see the app happily running.

FreeBSD, CentOS: Install Jenkins behind a nginx reverse proxy using SSL certificates

Kliment Andreev — Sat, 20 Jul 2019 13:06:24 +0000

In this post, I’ll explain how to install Jenkins on FreeBSD and CentOS. Jenkins runs on port 8080 (8180 in FreeBSD), so sometimes it’s not possible to access these ports because of corporate firewalls. We’ll put Jenkins behind nginx that will act as a reverse proxy. We’ll use port 80 (HTTP) and 443 (HTTPS) if you want to have SSL certificates. The installation is very simple.

CentOS

The Jenkins package is not part of the default CentOS repository, so we have to download that one first, install Jenkins, enable it to start on boot and then start it. You can skip the first line if you have wget installed. By default, it is not installed.

yum -y install wget
wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key
yum -y install jenkins java
systemctl enable jenkins
systemctl start jenkins

If you have a firewall enabled, you have to open the port for 8080.

firewall-cmd --permanent --zone=public --add-port=8080/tcp
firewall-cmd --reload

Now, go to http://[ip]:8080 and replace the with the IP of your server and you should see this greeting page.

Cat out that file, copy & paste the value in the browser and click Continue.

cat /var/lib/jenkins/secrets/initialAdminPassword
7f09cb85a4b843bdab66c768020a7c6e

And that’s it. Follow the prompts to install plugins, provide admin password etc…

nginx as reverse proxy

In order to access the server with out the need to specify the port 8080 and use a DNS name, we’ll have to use nginx that will listen on port 80 or 443 and redirect the traffic to 8080. Let’s install and configure it. In my case, I’ll access the server as jenkins.domain.com.

yum -y install epel-release
yum -y install nginx
systemctl enable nginx

Edit /etc/nginx/nginx.conf and delete everything after this line around line 36. The include line should stay and the last line “}” should stay as well. So, the last lines should look like this.

       include /etc/nginx/conf.d/*.conf;
}

Go to /etc/nginx/conf.d folder and create a new file called jenkins.conf.
This is how my jenkins.conf file looks like. Change the domain in the highlighted line.

server {
    listen 80;
    server_name jenkins.domain.com;

    location / {
        proxy_set_header   X-Forwarded-For $remote_addr;
        proxy_set_header   Host $http_host;
        proxy_pass         "http://127.0.0.1:8080";
    }
}

Don’t forget to open port 80 in your firewall.

firewall-cmd --permanent --zone=public --add-port=80/tcp
firewall-cmd --reload

If you have SELinux, you’ll have to allow HTTP traffic.

setsebool -P httpd_can_network_connect 1

And restart nginx for changes to take effect.

systemctl restart nginx

Now, you can access the server as jenkins.domain.com.
For HTTPS traffic, the configuration is different. You’ll need certificates specified in lines 15 and 16 (under /etc/nginx) and the domain changed in 7,13 and 31. Here is the config (/etc/nginx/conf.d/jenkins.conf) in order to access jenkins over SSL.

upstream jenkins {
  server 127.0.0.1:8080 fail_timeout=0;
}

server {
  listen 80;
  server_name jenkins.domain.com;
  return 301 https://$host$request_uri;
}

server {
  listen 443 ssl;
  server_name jenkins.domain.com;

  ssl_certificate public_cert.pem;
  ssl_certificate_key private_cert.key;
  client_max_body_size 4M;

  location / {
    proxy_set_header        Host $host:$server_port;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto $scheme;
    proxy_redirect http:// https://;
    proxy_pass              http://jenkins;
    # Required for new HTTP-based CLI
    proxy_http_version 1.1;
    proxy_request_buffering off;
    proxy_buffering off; # Required for HTTP-based CLI to work over SSL
    # workaround for https://issues.jenkins-ci.org/browse/JENKINS-45651
    add_header 'X-SSH-Endpoint' 'jenkins.domain.com:50022' always;
  }
}

Make sure you open port 443 on the firewall as we did for port 80 and change SELinux with setsebool.

firewall-cmd --permanent --zone=public --add-port=443/tcp
firewall-cmd --reload

SELinux change.

setsebool -P httpd_can_network_connect 1

You should also restart nginx for the changes to take effect.

systemctl restart nginx

That’s it. You should be able to access Jenkins as jenkins.domain.com over https.

FreeBSD

The installation on FreeBSD is a little bit different. You can install Jenkins from the packages. It will install JDK as well.

pkg install jenkins

The OpenJDK installation depends on these “memory filesystems”. Mount them now.

mount -t fdescfs fdesc /dev/fd
mount -t procfs proc /proc

…and make sure they are mounted on boot.

cat <> /etc/fstab
fdesc   /dev/fd         fdescfs         rw      0       0
proc    /proc           procfs          rw      0       0
EOF

We need to make sure jenkins starts on boot.

sysrc jenkins_enable=YES

Now, we can start jenkins.

service jenkins start

Go to http://[IP]:8180/jenkins and you should see the welcome screen. Replace the [IP] with the IP address of your server. You should see this screen.

Cat out that file, copy & paste the value in the browser and click Continue.

cat /usr/local/jenkins/secrets/initialAdminPassword
c33a584df234433d9a88e19d8e14c289

And that’s it. Follow the prompts to install plugins, provide admin password etc…The jenkins installation on FreeBSD is accessed with a suffix /jenkins. If you try to access it without the suffix, you’ll get this error.

HTTP ERROR 404
Problem accessing /. Reason:
    Not Found

It’s much easier to do the redirects with nginx without the suffix, so we’ll change it.
Edit this file /usr/local/etc/rc.d/jenkins and around line 54 find this.

: ${jenkins_args="--webroot=${jenkins_home}/war --httpPort=8180 --prefix=/jenkins"}

Change it so it looks like this.

: ${jenkins_args="--webroot=${jenkins_home}/war --httpPort=8180 --prefix=/"}

Restart Jenkins and you’ll be able to access it as http://[IP]:8180.

service jenkins restart

nginx as reverse proxy

In order to access the server with out the need to specify the port 8180 and use a DNS name, we’ll have to use nginx that will listen on port 80 or 443 and redirect the traffic to 8180. Let’s install and configure nginx. In my case, I’ll access the server as jenkins.domain.com.

pkg install nginx

Run on boot.

sysrc nginx_enable=YES

Edit /usr/local/etc/nginx/nginx.conf and delete everything after this line around line 39 after #gzip on directive. Add the include line, so the last three lines look like this.

    #gzip  on;
    include /usr/local/etc/nginx/conf.d/*.conf;
}

Create a new conf.d folder mkdir /usr/local/etc/nginx/conf.d and create a new file called jenkins.conf.
This is how my jenkins.conf file looks like. Change the domain in the highlighted line.

server {
    listen 80;
    server_name jenkins.domain.com;

    location / {
        proxy_set_header   X-Forwarded-For $remote_addr;
        proxy_set_header   Host $http_host;
        proxy_pass         "http://127.0.0.1:8180";
    }
}

Restart Jenkins and you should be able to access the site now with http://jenkins.domain.com.
For HTTPS traffic, the configuration is different. You’ll need certificates specified in lines 15 and 16 (under /usr/local/etc/nginx) and the domain changed in 7,13 and 31. Here is the config (/usr/local/etc/nginx/conf.d/jenkins.conf) in order to access jenkins over SSL.

upstream jenkins {
  server 127.0.0.1:8180 fail_timeout=0;
}

server {
  listen 80;
  server_name jenkins.domain.com;
  return 301 https://$host$request_uri;
}

server {
  listen 443 ssl;
  server_name jenkins.domain.com;

  ssl_certificate public_cert.pem;
  ssl_certificate_key private_cert.key;
  client_max_body_size 4M;

  location / {
    proxy_set_header        Host $host:$server_port;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto $scheme;
    proxy_redirect http:// https://;
    proxy_pass              http://jenkins;
    # Required for new HTTP-based CLI
    proxy_http_version 1.1;
    proxy_request_buffering off;
    proxy_buffering off; # Required for HTTP-based CLI to work over SSL
    # workaround for https://issues.jenkins-ci.org/browse/JENKINS-45651
    add_header 'X-SSH-Endpoint' 'jenkins.domain.com:50022' always;
  }
}

You should also restart nginx for the changes to take effect.

service nginx restart

That’s it. You should be able to access Jenkins as jenkins.domain.com over https.