In this post I’ll describe a use case that I am currently using. I have a standard Internet connection that goes to my Eero mesh router and behind that I have a homelab on 192.168.1.0/24 network as shown in the diagram. I have a port forwarding configured on the router so I can access one of my Linux servers, but if I want to access other servers, I have to either port forward SSH on some other-than-22/tcp port or use the Linux server as a jump-box. None of this is good, so I decided to create an OpenVPN server. Anytime I VPN to this server I can ssh/rdp to any of the internal servers. The OpenVPN server requires one nic.
NOTE: This post is very FreeBSD specific so if you want to use some Linux server as OpenVPN server, this guide won’t help you at all.

IMPORTANT: On your home router port-forward port 1194/udp to your OpenVPN server.

OpenVPN server

Let’s install OpenVPN server first.

pkg install openvpn

Make sure it starts on boot and enable the NAT.

sysrc openvpn_enable=YES
sysrc gateway_enable=YES

Next, we need a directory where we’ll store our config files and the PKI.

mkdir /usr/local/etc/openvpn
cd /usr/local/etc/openvpn

Create the PKI structure.

easyrsa init-pki

Go to the newly created pki directory and make a copy of the vars.example file.

cd pki
cp vars.example vars

Now edit this file and uncomment the following lines and change to match your needs. It’s optional and you can keep everything-as-is.

set_var EASYRSA_REQ_COUNTRY     "US"
set_var EASYRSA_REQ_PROVINCE    "NJ"
set_var EASYRSA_REQ_CITY        "Allentown"
set_var EASYRSA_REQ_ORG         "iAndreev"
set_var EASYRSA_REQ_EMAIL       "mail@somemail.com"
set_var EASYRSA_REQ_OU          "iAndreev"
set_var EASYRSA_CA_EXPIRE       3650
set_var EASYRSA_CERT_EXPIRE     825

Now create the Certificate Authority based on these values.

cd /usr/local/etc/openvpn
easyrsa build-ca

This command will prompt you to create a password that you will you to sign the certificates for the server and the clients.
Then create the DH parameters. It might take 30+ seconds.

easyrsa gen-dh

Create the server request and sign it.

easyrsa build-server-full server nopass

You have to type yes to confirm and then use the password that you created above to sign the cert.
The command will create some files under the pki directory and we’ll need those later.
Next, we’ll create the same for the client. In my case it’s a Dell laptop, so I use the word dell, but you can use client, client1, something-whatever…

easyrsa build-client-full dell nopass

Same thing, answer yes and then use the password to sign the certs.
Copy the OpenVPN config example file that comes with the package.

cp /usr/local/share/examples/openvpn/sample-config-files/server.conf /usr/local/etc/openvpn/openvpn.conf

Edit openvpn.conf and make sure these values are changed and match the certs you created earlier.

ca /usr/local/etc/openvpn/pki/ca.crt
cert /usr/local/etc/openvpn/pki/issued/server.crt
key /usr/local/etc/openvpn/pki/private/server.key

dh /usr/local/etc/openvpn/pki/dh.pem

In addition to that, find the line that says push route and create a new one that matches your internal subnet. In my case it looks like this.

push "route 192.168.1.0 255.255.255.0"

If you have internal DNS and you want to send that as well, uncomment the following line and change to match your DNS IP.

push "dhcp-option DNS 208.67.222.222"

Now, you can start the OpenVPN server. Make sure you check the syslog for any errors.

service openvpn start
tail /var/log/messages

OpenVPN client

On your laptop that you want to use as a VPN client, download the OpenVPN client from here. When you install the client, you’ll have a generic config file under C:\Program Files\OpenVPN\sample-config folder. Copy that file under c:\users\[your_username]\OpenVPN\config.
Then, edit the file and change the following lines.

remote public-ip-or-hostname-of-your-router 1194
ca ca.crt
cert dell.crt
key dell.key

NOTE: Make sure you copy ca.crt from /usr/local/etc/openvpn/pki, dell.crt or client.crt or whatever.crt from /usr/local/etc/openvpn/pki/issued and dell.key or client.key or whatever.key from /usr/local/etc/openvpn/pki/private directories to the same c:\users\[your-user-name]\OpenVPN\config folder. There should be 4 files there. The config file, the CA certificate, the client certificate and the private key for the client.
Start the OpenVPN client and connect. You shouldn’t have any problems and you’ll get 10.8.0.2 IP assigned to your laptop. If you try to ping 10.8.0.1 you’ll get a response. Do route print and you’ll see that the routes for 192.168.1.0 are going over 10.8.0.1. But, if you try to ping some server internally, e.g. 192.168.1.21, you won’t get any response. That’s because that server doesn’t know the route back. You can easily add the route to each server but it’s not practical. Let’s make one more change to the server.

OpenVPN server

Add these two lines to /etc/rc.conf. That’s the FreeBSD pf firewall.

pf_enable="YES"
pf_rules="/etc/pf.conf"

Now, create this config file in /etc/pf.conf. Many thanks to this guy.

vpnclients = "10.8.0.0/24"
# The name of your NIC. I am using a VM on ESXi and vmx0 is my NIC.
wanint = "vmx0"
# put your tunnel interface here, it is usually tun0
vpnint = "tun0"
# OpenVPN by default runs on udp port 1194
udpopen = "{1194}"
# Open the SSH port.
tcpopen = "{22}"
icmptypes = "{echoreq, unreach}"

set skip on lo
# the essential line
nat on $wanint inet from $vpnclients to any -> $wanint

block in
pass in on $wanint proto udp from any to $wanint port $udpopen
pass in on $wanint proto tcp from any to $wanint port $tcpopen
# the following two lines could be made stricter if you don't trust the clients
pass out quick
pass in on $vpnint from any to any
pass in inet proto icmp all icmp-type $icmptypes

Reboot the OpenVPN server and if you reconnect from the client, you can use the internal IPs directly to connect.

In this post I’ll create a site-to-site (S2S) tunnel between AWS and Azure. It’s an HA tunnel so it’s regionally resilient. I’ll use Terraform for the deployment using a set of modules (download here). The Terraform files will be separated in two different folders. One for Azure and one for AWS.

The architectural diagram looks like this.

Azure

For Azure, we will provision a public subnet (10.1.0.0/24) and a Gateway subnet (10.1.1.0/24). Then, we’ll provision one VNET Gateway with active-active BGP tunnel. The Azure setup requires that we provision 4 local network gateways and S2S connections, but the output will return only 2 public IPs. These are the files.
provider.tf

terraform {
  required_version = "~>1.9.5"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~>3.110.0"
    }
  }
}

provider "azurerm" {
  features {
    resource_group {
      prevent_deletion_if_contains_resources = false
    }
  }
}

azure.tf

# Create a resource group
module "rg" {
  source = "../../modules/azure/rg-0.0.4"

  az_rg_name     = var.rg_ResourceGroupName
  az_rg_location = var.rgn_RegionLocation
}

# Create a VNET
module "vnet" {
  source = "../../modules/azure/vnet-4.0.0"

  vnet_name           = var.vnet_Name
  resource_group_name = var.rg_ResourceGroupName
  vnet_location       = var.rgn_RegionLocation
  use_for_each        = false
  address_space       = [var.vnet_CIDR]
  subnet_prefixes     = [var.sub_PublicCIDR, var.sub_GatewayCIDR]
  subnet_names        = [var.sub_PublicName, var.sub_GatewayName]

  depends_on = [module.rg]
}

# Create a virtual network gateway
module "vnetgw" {
  source = "../../modules/azure/vnetgw-0.5.1"

  location                  = var.rgn_RegionLocation
  name                      = var.vgw_Name
  subnet_address_prefix     = var.sub_GatewayCIDR
  sku                       = var.vgw_SKU
  type                      = var.vgw_Type
  virtual_network_id        = module.vnet.vnet_id
  vpn_generation            = var.vgw_Generation
  vpn_type                  = var.vpn_Type
  subnet_creation_enabled   = false
  vpn_active_active_enabled = true
  vpn_bgp_enabled           = true
  vpn_bgp_settings = {
    asn = var.asn_Azure
  }
  # Create virtual network gateway configurations
  ip_configurations = {
    "ip_config_01" = {
      name            = var.gwc_BGPName1
      apipa_addresses = var.ip_APIPAAddresses1
      public_ip = {
        allocation_method = var.ip_allocation_method
      }
    },
    "ip_config_02" = {
      name            = var.gwc_BGPName2
      apipa_addresses = var.ip_APIPAAddresses2
      public_ip = {
        allocation_method = var.ip_allocation_method
      }
    }
  }
  # Create local network gateways and connections
  local_network_gateways = {
    # AWS Tunnel 1 to Azure Instance 0
    "lngw_01" = {
      name            = var.lng_AWST1A0
      gateway_address = var.ip_AWST1A0
      bgp_settings = {
        asn                 = var.asn_AWS
        bgp_peering_address = var.ip_peering_AWST1A0
      }
      connection = {
        name       = var.conn_AWST1A0
        type       = var.conn_Type
        enable_bgp = true
        shared_key = var.key_SharedKey
        custom_bgp_addresses = {
          primary   = var.ip_primary_AWST1A0
          secondary = var.ip_secondary_AWST1A0
        }
      }
    },
    # AWS Tunnel 2 to Azure Instance 0
    "lngw_02" = {
      name            = var.lng_AWST2A0
      gateway_address = var.ip_AWST2A0
      bgp_settings = {
        asn                 = var.asn_AWS
        bgp_peering_address = var.ip_peering_AWST2A0
      }
      connection = {
        name       = var.conn_AWST2A0
        type       = var.conn_Type
        enable_bgp = true
        shared_key = var.key_SharedKey
        custom_bgp_addresses = {
          primary = var.ip_primary_AWST2A0
        secondary = var.ip_secondary_AWST2A0 }
      }
    },
    # AWS Tunnel 1 to Azure Instance 1
    "lngw_03" = {
      name            = var.lng_AWST1A1
      gateway_address = var.ip_AWST1A1
      bgp_settings = {
        asn                 = var.asn_AWS
        bgp_peering_address = var.ip_peering_AWST1A1
      }
      connection = {
        name       = var.conn_AWST1A1
        type       = var.conn_Type
        enable_bgp = true
        shared_key = var.key_SharedKey
        custom_bgp_addresses = {
          primary = var.ip_primary_AWST1A1
        secondary = var.ip_secondary_AWST1A1 }
      }
    },
    # AWS Tunnel 2 to Azure Instance 1
    "lngw_04" = {
      name            = var.lng_AWST2A1
      gateway_address = var.ip_AWST2A1
      bgp_settings = {
        asn                 = var.asn_AWS
        bgp_peering_address = var.ip_peering_AWST2A1
      }
      connection = {
        name       = var.conn_AWST2A1
        type       = var.conn_Type
        enable_bgp = true
        shared_key = var.key_SharedKey
        custom_bgp_addresses = {
          primary = var.ip_primary_AWST2A1
        secondary = var.ip_secondary_AWST2A1 }
      }
    }
  }

  depends_on = [module.vnet]
}

variables.tf

# Resource group
variable "rg_ResourceGroupName" {
  type        = string
  default     = "rgVPN"
  description = "Resource group name"
}

variable "rgn_RegionLocation" {
  type        = string
  default     = "eastus2"
  description = "Azure region"
}

# VNET
variable "vnet_Name" {
  type        = string
  default     = "vnetVPN"
  description = "VNET name"
}

variable "vnet_CIDR" {
  type        = string
  default     = "10.1.0.0/16"
  description = "VNET CIDR"
}

variable "sub_PublicCIDR" {
  type        = string
  default     = "10.1.0.0/24"
  description = "CIDR of the public subnet"
}

variable "sub_GatewayCIDR" {
  type        = string
  default     = "10.1.1.0/24"
  description = "CIDR of the Gateway subnet"
}

variable "sub_PublicName" {
  type        = string
  default     = "subPublic"
  description = "Name of the public subnet"
}

variable "sub_GatewayName" {
  type        = string
  default     = "GatewaySubnet"
  description = "Name of the Gateway subnet - DO NOT CHANGE. It has to stay as GatewaySubnet"
}

# Virtual network gateway
variable "vgw_Name" {
  type        = string
  default     = "vgwVPN"
  description = "Name of the virtual gateway"
}

variable "vgw_SKU" {
  type        = string
  default     = "VpnGw2AZ"
  description = "Virtual Gateway SKU"
}

variable "vgw_Type" {
  type        = string
  default     = "Vpn"
  description = "Virtual Gateway Type (VPN or ExpressRoute)"
}

variable "vgw_Generation" {
  type        = string
  default     = "Generation2"
  description = "Virtual Gateway Generation"
}

variable "vpn_Type" {
  type        = string
  default     = "RouteBased"
  description = "VPN type - RouteBased or PolicyBased"
}

variable "asn_Azure" {
  type        = string
  default     = "65000"
  description = "Azure ASN"
}

variable "asn_AWS" {
  type        = string
  default     = "64512"
  description = "AWS ASN"
}

# Virtual network gateway config 1
variable "gwc_BGPName1" {
  type        = string
  default     = "vnetGatewayConfig01"
  description = "Name of the virtual network gateway configuration for the first tunnel"
}

variable "ip_APIPAAddresses1" {
  type        = list(string)
  default     = ["169.254.21.2", "169.254.22.2"]
  description = "APIPA address for the first tunnel"
}

# Virtual network gateway config 2
variable "gwc_BGPName2" {
  type        = string
  default     = "vnetGatewayConfig02"
  description = "Name of the virtual network gateway configuration for the second tunnel"
}

variable "ip_APIPAAddresses2" {
  type        = list(string)
  default     = ["169.254.21.6", "169.254.22.6"]
  description = "APIPA address for the second tunnel"
}

variable "ip_allocation_method" {
  type        = string
  default     = "Static"
  description = "Static or Dynamic public IP"
}

# Local network gateway common settings
variable "conn_Type" {
  type        = string
  default     = "IPsec"
  description = "Connection type: IPsec or Vnet2Vnet"
}

variable "key_SharedKey" {
  type        = string
  default     = "KlimentAndreev1970_"
  description = "Make sure it's the same as 'key_SharedKey' variable in variables.tf under the AWS folder"
  # Don't go crazy with complexity. AWS side can support only 
  # between 8 and 64 characters in length and cannot start with zero (0). 
  # Allowed characters are alphanumeric characters, periods (.), and underscores (_).
}

# Local network gateway AWS Tunnel 1 to Azure Instance 0
# lngw_01
variable "lng_AWST1A0" {
  type        = string
  default     = "lng_AWSTunnel1ToAzureInstance0"
  description = "From AWS Tunnel 1 to Azure Instance 0"
}

###
### REPLACE FROM THE OUTPUT OF AWS IN STEP 2
###
variable "ip_AWST1A0" {
  type        = string
  default     = "13.58.80.164"
  description = "Enter the output of ip_AWST1IP1 from the AWS output.tf script"
}

variable "ip_peering_AWST1A0" {
  type        = string
  default     = "169.254.21.1"
  description = "BGP Peering IP for AWS Tunnel1 to Azure Instance 0"
}

variable "ip_primary_AWST1A0" {
  type        = string
  default     = "169.254.21.2"
  description = "Primary Custom BGP Address for AWS Tunnel1 to Azure Instance 0"
}

variable "ip_secondary_AWST1A0" {
  type        = string
  default     = "169.254.21.6"
  description = "Secondary Custom BGP Address for AWS Tunnel1 to Azure Instance 0"
}

variable "conn_AWST1A0" {
  type        = string
  default     = "connAWSTunnel1toAzureInstance0"
  description = "Connection name for AWS Tunnel1 to Azure Instance 0"
}

# Local network gateway AWS Tunnel 2 to Azure Instance 0
# lngw_02
variable "lng_AWST2A0" {
  type        = string
  default     = "lng_AWSTunnel2ToAzureInstance0"
  description = "From AWS Tunnel 2 to Azure Instance 0"
}

###
### REPLACE FROM THE OUTPUT OF AWS IN STEP 2
###
variable "ip_AWST2A0" {
  type        = string
  default     = "18.119.84.239"
  description = "Enter the output of ip_AWST2A0 from the AWS output.tf script"
}

variable "ip_peering_AWST2A0" {
  type        = string
  default     = "169.254.22.1"
  description = "BGP Peering IP for AWS Tunnel2 to Azure Instance 0"
}

variable "ip_primary_AWST2A0" {
  type        = string
  default     = "169.254.22.2"
  description = "Primary Custom BGP Address for AWS Tunnel2 to Azure Instance 0"
}

variable "ip_secondary_AWST2A0" {
  type        = string
  default     = "169.254.21.6"
  description = "Secondary Custom BGP Address for AWS Tunnel1 to Azure Instance 1"
}

variable "conn_AWST2A0" {
  type        = string
  default     = "connAWSTunnel2toAzureInstance0"
  description = "Connection name for AWS Tunnel2 to Azure Instance 0"
}

# Local network gateway AWS Tunnel 1 to Azure Instance 1
# lngw_03
variable "lng_AWST1A1" {
  type        = string
  default     = "lng_AWSTunnel1ToAzureInstance1"
  description = "From AWS Tunnel 1 to Azure Instance 1"
}

###
### REPLACE FROM THE OUTPUT OF AWS IN STEP 2
###
variable "ip_AWST1A1" {
  type        = string
  default     = "18.119.2.80"
  description = "Enter the output of ip_AWST1A1 from the AWS output.tf script"
}

variable "ip_peering_AWST1A1" {
  type        = string
  default     = "169.254.21.5"
  description = "BGP Peering IP for AWS Tunnel1 to Azure Instance 1"
}

variable "ip_primary_AWST1A1" {
  type        = string
  default     = "169.254.21.2"
  description = "Primary Custom BGP Address for AWS Tunnel1 to Azure Instance 1"
}

variable "ip_secondary_AWST1A1" {
  type        = string
  default     = "169.254.21.6"
  description = "Secondary Custom BGP Address for AWS Tunnel1 to Azure Instance 1"
}

variable "conn_AWST1A1" {
  type        = string
  default     = "connAWSTunnel1toAzureInstance1"
  description = "Connection name for AWS Tunnel1 to Azure Instance 1"
}

# Local network gateway AWS Tunnel 2 to Azure Instance 1
# lngw_04
variable "lng_AWST2A1" {
  type        = string
  default     = "lng_AWSTunnel2ToAzureInstance1"
  description = "From AWS Tunnel 2 to Azure Instance 1"
}

###
### REPLACE FROM THE OUTPUT OF AWS IN STEP 2
###
variable "ip_AWST2A1" {
  type        = string
  default     = "18.220.73.188"
  description = "Enter the output of ip_AWST2A1 from the AWS ouput.tf script"
}

variable "ip_peering_AWST2A1" {
  type        = string
  default     = "169.254.22.5"
  description = "BGP Peering IP for AWS Tunnel2 to Azure Instance 1"
}

variable "ip_primary_AWST2A1" {
  type        = string
  default     = "169.254.21.2"
  description = "Primary Custom BGP Address for AWS Tunnel2 to Azure Instance 1"
}

variable "ip_secondary_AWST2A1" {
  type        = string
  default     = "169.254.22.6"
  description = "Secondary Custom BGP Address for AWS Tunnel2 to Azure Instance 1"
}

variable "conn_AWST2A1" {
  type        = string
  default     = "connAWSTunnel2toAzureInstance1"
  description = "Connection name for AWS Tunnel2 to Azure Instance 1"
}

outputs.tf

output "vgw_publicip_1" {
  value       = module.vnetgw.public_ip_addresses.ip_config_01.ip_address
  description = "Use this output to populate the same variable in AWS variables.tf file."
}

output "vgw_publicip_2" {
  value       = module.vnetgw.public_ip_addresses.ip_config_02.ip_address
  description = "Use this output to populate the same variable in AWS variables.tf file."
}

Once you run the initial terraform apply it will take about 35-40 mins and you’ll get two IPs as output. You’ll need these momentarily in the file variables.tf under 02.AWS folder.

vgw_publicip_1 = "48.211.240.129"
vgw_publicip_2 = "48.211.240.186"

AWS

In AWS we’ll provision a private and a public subnet as well as two VPN tunnels. This is the Terraform code for that.
provider.tf

terraform {
  required_version = "~>1.9.5"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~>5.57.0"
    }
  }
}

provider "aws" {
  region = var.rgn_RegionLocation
}

aws.tf

# Create VPC
module "vpc" {

  source = "../../modules/aws/vpc-5.5.1"

  name = var.vpc_Name
  cidr = var.vpc_CIDR

  azs             = var.vpc_AZs
  private_subnets = var.sub_pvtCIDR
  public_subnets  = var.sub_pubCIDR

  enable_nat_gateway = true
  enable_vpn_gateway = true
  amazon_side_asn    = var.asn_AWS
}

# Change the route to enable propagation to public route table
# resource "aws_vpn_gateway_route_propagation" "rtPropg" {
#   vpn_gateway_id = module.vpc.vgw_id
#   route_table_id = module.vpc.public_route_table_ids[0]
# }

# Change the route to enable propagation to private route table
resource "aws_vpn_gateway_route_propagation" "rtPvtPropg" {
  vpn_gateway_id = module.vpc.vgw_id
  route_table_id = module.vpc.private_route_table_ids[0]
}

# Add the NAT gateway as a route to Internet to the private route table
# resource "aws_route" "r" {
#   route_table_id         = module.vpc.private_route_table_ids[0]
#   destination_cidr_block = "0.0.0.0/0"
#   nat_gateway_id         = module.vpc.natgw_ids[0]
# }

# Create 2 customer gateways
module "cgw" {
  source = "../../modules/aws/cgw-2.0.1/"

  name = var.cgw_Name

  customer_gateways = {
    IP1 = {
      bgp_asn    = var.asn_Azure
      ip_address = var.vgw_publicip_1
    },
    IP2 = {
      bgp_asn    = var.asn_Azure
      ip_address = var.vgw_publicip_2
    }
  }
}

module "vpngw1" {
  source = "../../modules/aws/vpngw-3.7.2/"

  customer_gateway_id               = module.cgw.ids[0]
  vpc_id                            = module.vpc.vpc_id
  vpn_gateway_id                    = module.vpc.vgw_id
  vpn_connection_static_routes_only = false
  local_ipv4_network_cidr           = "0.0.0.0/0"

  tunnel1_inside_cidr   = var.ip_AWST1A0
  tunnel2_inside_cidr   = var.ip_AWST2A0
  tunnel1_preshared_key = var.key_SharedKey
  tunnel2_preshared_key = var.key_SharedKey

  tags = {
    Name = var.vpn_AWSA0Name
  }

  depends_on = [module.cgw]

}

module "vpngw2" {
  source = "../../modules/aws/vpngw-3.7.2/"

  customer_gateway_id               = module.cgw.ids[1]
  vpc_id                            = module.vpc.vpc_id
  vpn_gateway_id                    = module.vpc.vgw_id
  vpn_connection_static_routes_only = false
  local_ipv4_network_cidr           = "0.0.0.0/0"

  tunnel1_inside_cidr   = var.ip_AWST1A1
  tunnel2_inside_cidr   = var.ip_AWST2A1
  tunnel1_preshared_key = var.key_SharedKey
  tunnel2_preshared_key = var.key_SharedKey

  tags = {
    Name = var.vpn_AWSA1Name
  }

  depends_on = [module.cgw]

}

variables.tf – replace the IPs in line 50 and 59 from the previous Azure output

# VPC
variable "rgn_RegionLocation" {
  type        = string
  default     = "us-east-2"
  description = "AWS region"
}

variable "vpc_Name" {
  type        = string
  default     = "vpcVPN"
  description = "Name of the VPC in AWS"
}

variable "vpc_CIDR" {
  type        = string
  default     = "10.2.0.0/16"
  description = "CIDR of the VPC in AWS"
}

variable "vpc_AZs" {
  type        = list(string)
  default     = ["us-east-2a"]
  description = "Location of available zones"
}

variable "sub_pubCIDR" {
  type        = list(string)
  default     = ["10.2.0.0/24"]
  description = "Public subnet CIDR"
}

variable "sub_pvtCIDR" {
  type        = list(string)
  default     = ["10.2.1.0/24"]
  description = "Private subnet CIDR"
}

# Customer gateways
variable "cgw_Name" {
  type        = string
  default     = "cgwVPN"
  description = "Name of the Customer Gateway"
}

####
#### REPLACE WITH OUTPUT FROM AZURE
####
variable "vgw_publicip_1" {
  type        = string
  default     = "20.10.220.96"
  description = "Use the output from previous Azure script. First public IP of the Virtual Network Gateway in Azure"
}

####
#### REPLACE WITH OUTPUT FROM AZURE
####
variable "vgw_publicip_2" {
  type        = string
  default     = "4.152.124.252"
  description = "Use the output from previous Azure script. Second public IP of the Virtual Network Gateway in Azure"
}

variable "asn_AWS" {
  type        = string
  default     = "64512"
  description = "AWS ASN"
}

variable "asn_Azure" {
  type        = string
  default     = "65000"
  description = "Azure ASN"
}

# VPN site to site 1
variable "key_SharedKey" {
  type        = string
  default     = "KlimentAndreev1970_"
  description = "Make sure it's the same as vartiables.tf in  Azure folder"
  # Don't go crazy with complexity. AWS side can support only 
  # between 8 and 64 characters in length and cannot start with zero (0). 
  # Allowed characters are alphanumeric characters, periods (.), and underscores (_).

}

variable "vpn_AWSA0Name" {
  type        = string
  default     = "vpnToAzureInstance0"
  description = "AWS Tunnel to Azure Instance 0 name"
}

variable "ip_AWST1A0" {
  type    = string
  default = "169.254.21.0/30"
}

variable "ip_AWST2A0" {
  type    = string
  default = "169.254.22.0/30"
}

# VPN site to site 2
variable "vpn_AWSA1Name" {
  type        = string
  default     = "vpnToAzureInstance1"
  description = "AWS Tunnel to Azure Instance 1 name"
}

variable "ip_AWST1A1" {
  type    = string
  default = "169.254.21.4/30"
}

variable "ip_AWST2A1" {
  type    = string
  default = "169.254.22.4/30"
}

outputs.tf

output "ip_AWST1A0" {
  value = module.vpngw1.vpn_connection_tunnel1_address
}

output "ip_AWST2A0" {
  value = module.vpngw1.vpn_connection_tunnel2_address
}

output "ip_AWST1A1" {
  value = module.vpngw2.vpn_connection_tunnel1_address
}

output "ip_AWST2A1" {
  value = module.vpngw2.vpn_connection_tunnel2_address
}

Once you run terraform apply you’ll get 4 IPs as an output.

ip_AWST1A0 = "3.137.10.4"
ip_AWST1A1 = "3.20.47.74"
ip_AWST2A0 = "18.220.233.198"
ip_AWST2A1 = "18.190.37.23"

You have to go back and replace these 4 IPs in variables.tf in the previous Azure folder. So, you run terraform apply in 01.Azure folder first, then 02.AWS second and then again terraform apply in 01.Azure.

Azure

Edit variables.tf file and add the 4 IPs. Make sure that the variables match, do not just assign IPs randomly. Replace the IPs in the lines 155, 196, 237 and 278. The second time you run terraform apply it will take about a minute or two.

Verification

Now, go to AWS and go to VPC | VPN | Site-to-Site VPN connections. You’ll see two tunnels there. For each one select the tunnel and then click on the Tunnel details tab. Both should be up for each tunnel.

In Azure, go to Virtual Network Gateways, click on the vgw, then Monitoring and BGP peers.
You should see something like this.

If you have a use case where you need to transfer a lot of data back-and-forth between various resources and an S3 bucket, you will definitely benefit if you use gateway or interface endpoints for S3. The monthly bill for regional and zonal transfer will be much less and on top of that it’s much more secure.
From the AWS documentation, “Amazon S3 supports both gateway endpoints and interface endpoints. With a gateway endpoint, you can access Amazon S3 from your VPC, without requiring an internet gateway or NAT device for your VPC, and with no additional cost. However, gateway endpoints do not allow access from on-premises networks, from peered VPCs in other AWS Regions, or through a transit gateway. For those scenarios, you must use an interface endpoint, which is available for an additional cost.”
Keep this table handy as it explains the differences between gateway and interface S3 endpoint.

Now that we know the use case and the differences, we’ll see three different scenarios. For this, you’ll need access to two different accounts and two different regions.

• An S3 bucket in account A and an ec2 instance in the same account A and the same region A
• An S3 bucket in account A and an ec2 instance in a different account B, but the same region A
• An S3 bucket in account A and an ec2 instance in a different account B and a different region B

The diagram looks like this.

An S3 bucket in account A, ec2 in account A, both in the same region

Let’s create an S3 bucket first. We’ll use AWS CLI with different profiles.

BUCKET_NAME="myuniquenameforthebucket"
REGION_A="us-east-2"
aws s3api create-bucket \
    --bucket $BUCKET_NAME \
    --region $REGION_A \
    --create-bucket-configuration LocationConstraint=$REGION_A

Let’s copy a file there.

dd if=/dev/zero of=somefile bs=1024 count=1
aws s3 cp somefile "s3://$BUCKET_NAME/somefile"

Using the same profile, create a VPC with one public subnet with the same CIDR as the VPC.

CIDR_A="192.168.10.0/24"
VPCA_ID=$(aws ec2 create-vpc \
    --cidr-block $CIDR_A \
    --region $REGION_A \
    --tag-specification 'ResourceType=vpc,Tags=[{Key=Name,Value=VPC-A}]' \
    --output text --query 'Vpc.VpcId')
echo $VPCA_ID

Enable DNS and hostnames resolution, needed for SSM. Two separate lines needed.

aws ec2 modify-vpc-attribute --enable-dns-hostnames "{\"Value\":true}"  --vpc-id $VPCA_ID
aws ec2 modify-vpc-attribute --enable-dns-support "{\"Value\":true}"  --vpc-id $VPCA_ID

Create a default network ACL.

aws ec2 create-network-acl --vpc-id $VPCA_ID

Create an Internet Gateway and attach it to the VPC.

IGWA_ID=$(aws ec2 create-internet-gateway \
    --region $REGION_A \
    --tag-specifications 'ResourceType=internet-gateway,Tags=[{Key=Name,Value=igw-A}]' \
    --output text --query 'InternetGateway.InternetGatewayId')
echo $IGWA_ID
aws ec2 attach-internet-gateway \
    --internet-gateway-id $IGWA_ID \
    --vpc-id $VPCA_ID

Let’s create a subnet.

SUBA_ID=$(aws ec2 create-subnet \
    --vpc-id $VPCA_ID \
    --cidr-block 192.168.10.0/24 \
    --region $REGION_A \
    --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=subPubA}]' \
    --output text --query 'Subnet.SubnetId')
echo $SUBA_ID

Create a route table.

RTA_ID=$(aws ec2 create-route-table \
     --vpc-id $VPCA_ID \
     --output text --query 'RouteTable.RouteTableId')
echo $RTA_ID
aws ec2 create-route --route-table-id $RTA_ID --destination-cidr-block 0.0.0.0/0 --gateway-id $IGWA_ID

Associate this route table to the public subnet.

aws ec2 associate-route-table --route-table-id $RTA_ID --subnet-id $SUBA_ID

Now, we have to create an EC2 instance where we can test the S3 access. We won’t be using keys and SSH to access, we’ll use SSM. For that we need the instance to have access to the Internet which means we need a security group that allows access to Internet.

SGA_ID=$(aws ec2 create-security-group \
    --group-name sgAllowICMPandOutboundAccess --description "Allows ICMP and outbound access" \
    --vpc-id $VPCA_ID \
    --output text --query 'GroupId')
echo $SGA_ID

Allow unrestricted egress. By default this rule is there, so don’t execute the code below.

#aws ec2 authorize-security-group-egress \
    --group-id $SGA_ID \
    --protocol all \
    --cidr "0.0.0.0/0" 

Do this one, so we can ping instances from each other when we do VPC peering.

aws ec2 authorize-security-group-ingress \
  --group-id $SGA_ID \
  --ip-permissions IpProtocol=icmp,FromPort=-1,ToPort=-1,IpRanges="[{CidrIp=192.168.10.0/24},{CidrIp=192.168.20.0/24},{CidrIp=192.168.30.0/24}]"

Let’s create a role so we can access the EC2 over SSM. First we need a policy that the role can assume.

cat << 'EOF' >> trust-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

Then create the role and attach the AWS Managed policy for SSM (AmazonSSMManagerInstanceCore).

aws iam create-role --role-name rolSSM --assume-role-policy-document file://trust-policy.json
aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore --role-name rolSSM

Create instance profile.

aws iam create-instance-profile --instance-profile-name ipEC2

Add the role.

aws iam add-role-to-instance-profile --role-name rolSSM --instance-profile-name ipEC2

Create an instance.

AMI_ID="ami-033fabdd332044f06"
INS_TYPE="t3.micro"
aws ec2 run-instances \
    --image-id $AMI_ID \
    --instance-type $INS_TYPE \
    --security-group-ids $SGA_ID \
    --subnet-id $SUBA_ID \
    --iam-instance-profile Name="ipEC2" \
    --associate-public-ip-address  \
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=ec2-A}]'

We’ll need an IAM user that we’ll use to access the S3 bucket.

IAM_USER="usrMyBucket"
aws iam create-user --user-name $IAM_USER

We’ll also need a policy that gives access to that specific bucket only.

cat >> bucket-policy.json << EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::$BUCKET_NAME",
                "arn:aws:s3:::$BUCKET_NAME/*"
            ]
        },
        {
            "Effect": "Deny",
            "NotAction": "s3:*",
            "NotResource": [
                "arn:aws:s3:::$BUCKET_NAME",
                "arn:aws:s3:::$BUCKET_NAME/*"
            ]
        }
    ]
}
EOF

Create the policy.

POL_ARN=$(aws iam create-policy \
    --policy-name polMyBucketAccess \
    --policy-document file://bucket-policy.json \
    --output text --query 'Policy.Arn')
echo $POL_ARN

Attach the policy to the user.

aws iam attach-user-policy --policy-arn $POL_ARN --user-name $IAM_USER

Create access keys.

aws iam create-access-key --user-name $IAM_USER --output table --query ['AccessKey.AccessKeyId','AccessKey.SecretAccessKey']

Copy these two values.
Now, go to AWS console, select the EC2 instance and click Connect. Use the 2nd tab Session Manager and click Connect.
Type aws configure and paste the copied values for the IAM user you just created.

If you try to do aws s3 ls, you’ll get access denied, but if you try aws s3 ls s3://myuniquenameforthebucket, you’ll see that it works.

Now, this type of access goes over the Internet and you’ll be charged for the data transfers. In order to avoid that, we’ll create an S3 gateway endpoint.
Go to VPC menu and from the left side choose Endpoints.
Create the endpoint, name the endpoint, choose AWS services, filter by Type = Gateway and select the S3 service name. Then select the VPC where you created your EC2 and the route table that’s associated with the subnet where EC2 resides. For policy choose Full Access. This means, the S3 gateway endpoint will have access to all S3 buckets. Finally, click to create the endpoint.

If you check your route table, you’ll see that there is another entry there. Wait 10 seconds or less if you don’t see it.

If you go back to the EC2 instance and do the aws s3 ls s3://myuniquenameforthebucket command again, you’ll see that nothing changed. So, how do you know that we did the right thing? It’s simple. Click on the endpoint that you’ve created, click on the Policy tab and then click the Edit Policy button.

Change line 5 from Allow to Deny and click Save.
Go back to the EC2 instance and do the aws s3 ls s3://myuniquenameforthebucket again. You’ll get access denied. There you go…Traffic goes over the endpoint. Revert the changes back from Deny to Allow. Let’s move to the 2nd use case.

An S3 bucket in account A, an ec2 instance in a different account B, but both in the same region A

NOTE:Make sure you switch your AWS CLI to a different profile, because we’ll be building the EC2 instance in another account. The S3 bucket stays the same.
Let’s build the 2nd VPC and EC2.

REGION_A="us-east-2"
CIDR_B="192.168.20.0/24"
VPCB_ID=$(aws ec2 create-vpc \
    --cidr-block $CIDR_B \
    --region $REGION_A \
    --tag-specification 'ResourceType=vpc,Tags=[{Key=Name,Value=VPC-B}]' \
    --output text --query 'Vpc.VpcId')
echo $VPCB_ID

Enable DNS and hostnames resolution, needed for SSM. Two separate lines needed.

aws ec2 modify-vpc-attribute --enable-dns-hostnames "{\"Value\":true}"  --vpc-id $VPCB_ID
aws ec2 modify-vpc-attribute --enable-dns-support "{\"Value\":true}"  --vpc-id $VPCB_ID

Create a default network ACL.

aws ec2 create-network-acl --vpc-id $VPCB_ID

Create an Internet Gateway and attach it to the VPC.

IGWB_ID=$(aws ec2 create-internet-gateway \
    --region $REGION_A \
    --tag-specifications 'ResourceType=internet-gateway,Tags=[{Key=Name,Value=igw-B}]' \
    --output text --query 'InternetGateway.InternetGatewayId')
echo $IGWB_ID
aws ec2 attach-internet-gateway \
    --internet-gateway-id $IGWB_ID \
    --vpc-id $VPCB_ID

Let’s create a subnet.

SUBB_ID=$(aws ec2 create-subnet \
    --vpc-id $VPCB_ID \
    --cidr-block 192.168.20.0/24 \
    --region $REGION_A \
    --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=subPubB}]' \
    --output text --query 'Subnet.SubnetId')
echo $SUBB_ID

Create a route table.

RTB_ID=$(aws ec2 create-route-table \
     --vpc-id $VPCB_ID \
     --output text --query 'RouteTable.RouteTableId')
echo $RTB_ID
aws ec2 create-route --route-table-id $RTB_ID --destination-cidr-block 0.0.0.0/0 --gateway-id $IGWB_ID

Associate this route table to the public subnet.

aws ec2 associate-route-table --route-table-id $RTB_ID --subnet-id $SUBB_ID

Now, we have to create an EC2 instance where we can test the S3 access. We won’t be using keys and SSH to access, we’ll use SSM. For that we need the instance to have access to the Internet which means we need a security group that allows access to Internet.

SGB_ID=$(aws ec2 create-security-group \
    --group-name sgAllowICMPandOutboundAccess --description "Allows ICMP and outbound access" \
    --vpc-id $VPCB_ID \
    --output text --query 'GroupId')
echo $SGB_ID

Allow unrestricted egress. By default this rule is there, so don’t execute the code below.

#aws ec2 authorize-security-group-egress \
    --group-id $SGB_ID \
    --protocol all \
    --cidr "0.0.0.0/0" 

Do this one, so we can ping instances from each other when we do VPC peering.

aws ec2 authorize-security-group-ingress \
  --group-id $SGB_ID \
  --ip-permissions IpProtocol=icmp,FromPort=-1,ToPort=-1,IpRanges="[{CidrIp=192.168.10.0/24},{CidrIp=192.168.20.0/24},{CidrIp=192.168.30.0/24}]"

We can reuse the same policy to create a role that will allow us to use SSM.

cat trust-policy.json

Then create the role and attach the AWS Managed policy for SSM (AmazonSSMManagerInstanceCore).

aws iam create-role --role-name rolSSM --assume-role-policy-document file://trust-policy.json
aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore --role-name rolSSM

Create instance profile. We’ll use the same names, because they are different accounts.

aws iam create-instance-profile --instance-profile-name ipEC2

Add the role.

aws iam add-role-to-instance-profile --role-name rolSSM --instance-profile-name ipEC2

Create an instance. The AMI_ID is the same because it’s the same region.

AMI_ID="ami-033fabdd332044f06"
INS_TYPE="t3.micro"
aws ec2 run-instances \
    --image-id $AMI_ID \
    --instance-type $INS_TYPE \
    --security-group-ids $SGB_ID \
    --subnet-id $SUBB_ID \
    --iam-instance-profile Name="ipEC2" \
    --associate-public-ip-address  \
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=ec2-B}]'

Wait for a couple of minutes and then connect from SSM, same as you did with the first instance. Configure AWS again using the same set of credentials that you did before and then try to list the content of the bucket. You should be able to see the file.

BUCKET_NAME="myuniquenameforthebucket"
aws s3 ls s3://$BUCKET_NAME

Again, the traffic goes over the Internet. So we need another gateway S3 endpoint. You create the gateway S3 endpoint where the client is located, not the destination.
Let’s create another S3 gateway endpoint as described above.

Check the route table and you’ll see the route created. At this point, you are accessing the bucket over the endpoint. Do the same test. Change the policy to Deny for the endpoint and you’ll get access denied.

An S3 bucket in account A, an ec2 instance in a different account B and a different region B

For the 3rd use case, we’ll use an interface endpoint (PrivateLink), not a gateway endpoint. This is what AWS documentation has to say. “With AWS PrivateLink for Amazon S3, you can provision interface VPC endpoints (interface endpoints) in your virtual private cloud (VPC). These endpoints are directly accessible from applications that are on premises over VPN and AWS Direct Connect, or in a different AWS Region over VPC peering. Interface endpoints are represented by one or more elastic network interfaces (ENIs) that are assigned private IP addresses from subnets in your VPC. Requests to Amazon S3 over interface endpoints stay on the Amazon network. You can also access interface endpoints in your VPC from on-premises applications through AWS Direct Connect or AWS Virtual Private Network (AWS VPN).”
Let’s create the resources. VPC and EC2. Change the AWS CLI profile so it’s using the same account B and a new region B.

REGION_B="us-east-1"
CIDR_C="192.168.30.0/24"
VPCC_ID=$(aws ec2 create-vpc \
    --cidr-block $CIDR_C \
    --region $REGION_B \
    --tag-specification 'ResourceType=vpc,Tags=[{Key=Name,Value=VPC-C}]' \
    --output text --query 'Vpc.VpcId')
echo $VPCC_ID

Enable DNS and hostnames resolution, needed for SSM. Two separate lines needed.

aws ec2 modify-vpc-attribute --enable-dns-hostnames "{\"Value\":true}"  --vpc-id $VPCC_ID
aws ec2 modify-vpc-attribute --enable-dns-support "{\"Value\":true}"  --vpc-id $VPCC_ID

Create a default network ACL.

aws ec2 create-network-acl --vpc-id $VPCC_ID

Create an Internet Gateway and attach it to the VPC.

IGWC_ID=$(aws ec2 create-internet-gateway \
    --region $REGION_B \
    --tag-specifications 'ResourceType=internet-gateway,Tags=[{Key=Name,Value=igw-C}]' \
    --output text --query 'InternetGateway.InternetGatewayId')
echo $IGWC_ID
aws ec2 attach-internet-gateway \
    --internet-gateway-id $IGWC_ID \
    --vpc-id $VPCC_ID

Let’s create a subnet.

SUBC_ID=$(aws ec2 create-subnet \
    --vpc-id $VPCC_ID \
    --cidr-block 192.168.30.0/24 \
    --region $REGION_B \
    --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=subPubC}]' \
    --output text --query 'Subnet.SubnetId')
echo $SUBC_ID

Create a route table.

RTC_ID=$(aws ec2 create-route-table \
     --vpc-id $VPCC_ID \
     --output text --query 'RouteTable.RouteTableId')
echo $RTC_ID
aws ec2 create-route --route-table-id $RTC_ID --destination-cidr-block 0.0.0.0/0 --gateway-id $IGWC_ID

Associate this route table to the public subnet.

aws ec2 associate-route-table --route-table-id $RTC_ID --subnet-id $SUBC_ID

Now, we have to create an EC2 instance where we can test the S3 access. We won’t be using keys and SSH to access, we’ll use SSM. For that we need the instance to have access to the Internet which means we need a security group that allows access to Internet.

SGC_ID=$(aws ec2 create-security-group \
    --group-name sgAllowICMPandOutboundAccess --description "Allows ICMP and outbound access" \
    --vpc-id $VPCC_ID \
    --output text --query 'GroupId')
echo $SGC_ID

Allow unrestricted egress. By default this rule is there, so don’t execute the code below.

#aws ec2 authorize-security-group-egress \
    --group-id $SGB_ID \
    --protocol all \
    --cidr "0.0.0.0/0" 

Do this one, so we can ping instances from each other when we do VPC peering.

aws ec2 authorize-security-group-ingress \
  --group-id $SGC_ID \
  --ip-permissions IpProtocol=icmp,FromPort=-1,ToPort=-1,IpRanges="[{CidrIp=192.168.10.0/24},{CidrIp=192.168.20.0/24},{CidrIp=192.168.30.0/24}]"

We don’t have to create any roles. We’ll use the existing ones that we built for the 2nd use case.
Create the EC2. The AMI ID will change this time.

AMI_ID="ami-08a0d1e16fc3f61ea"
INS_TYPE="t3.micro"
aws ec2 run-instances \
    --image-id $AMI_ID \
    --instance-type $INS_TYPE \
    --security-group-ids $SGC_ID \
    --subnet-id $SUBC_ID \
    --iam-instance-profile Name="ipEC2" \
    --associate-public-ip-address  \
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=ec2-C}]'

Same thing as we did before, connect from SSM, configure AWS CLI using the previous credentials and try to list the bucket.

BUCKET_NAME="myuniquenameforthebucket"
aws s3 ls s3://$BUCKET_NAME

You should see the file there. But, this access goes over Internet. Now, we have to create an interface endpoint in the destination VPC which is VPC-A. When we create an interface endpoint a private IP will be generated for this interface endpoint. In order this private IP to access the bucket, we need some sort of connection between regions, e.g. VPC peering or Transit Gateway. Let’s do the peering between account A (VPC-A) and account B (VPC-C). Enter the 12 digit account number for account A where the S3 bucket resides.

echo $VPCA_ID
echo $VPCC_ID
echo $REGION_A
ACCOUNT_A="123456789012"
echo $ACCOUNT_A

Make sure all of the above returns some value. Do the peering.

aws ec2 create-vpc-peering-connection \
  --vpc-id $VPCC_ID --peer-vpc-id $VPCA_ID \
  --peer-owner-id $ACCOUNT_A --peer-region $REGION_A

Log to the account A and under VPC | Peering menu on the left, select the peering and accept it. But this is not enough. You need to tell the VPC to use the routes over the peering connection. Go to account A, VPC A, find the route for the subnet where the EC2 instance is and add a route for destination 192.168.30.0/24 to go over peering. As you can see this is the same route that we modified initially in our first use case.

Now, log to account B and VPC C and modify the route but this time, the destination is VPC A (192.168.10.0/24). At this point you should be able to ping EC2-A from EC2-C. Don’t worry if you can’t ping EC2-C from EC2-A.
At this point we can create the interface endpoint where the bucket is. This time choose interface as type, VPC-A, our security group, full access policy and the subnet where EC2 is. This subnet has nothing to do with the bucket access, we are just creating a network interface there. We can choose any subnet in that VPC, we just need the IP assigned to the endpoint interface.

There will be 2 DNS names created for you. One is referring the zone and the other the region. Use the regional DNS name, the first one and add the word bucket instead of the * character.

From instance EC2-C, this is how you will access the bucket.

aws s3 ls s3://myuniquenameforthebucket --endpoint-url https://bucket.vpce-038d5f94d78d8b216-vyvk5uon.s3.us-east-2.vpce.amazonaws.com

If you try from the instance EC2-C, this won’t work. Why? Because of the security group that we used. We have to allow HTTPS traffic from 192.168.30.0/24. Go find the first security group, that’s the one we used for the interface endpoint and allow HTTPS from 192.168.30.0/24. After the change, you’ll be able to access the bucket.

In this post I’ll explain how to configure a static web app in Azure and put the app behind Entra ID (Azure AD) SSO authentication. You’ll have the same protection as logging with your Entra ID credentials to Azure and Office 365. There is no need for any code change in the app. As long as you are an user or a member of an Entra ID group assigned to the Enterprise Application, then you can access the Static Web App URL. I’ll use Azure DevOps to host the files for the web site, but you can use any other repo.

Azure DevOps

Log to your ADO organization and create a new project. I called mine Static Web App. Click on Initialize and then copy the path so you can clone it locally.

Once cloned, go to the repo locally and you’ll see a README.md file. Clone this web site template or just create an index.html file.

git clone https://github.com/designmodo/html-website-templates.git

Then move the index file back to the root of our repo.

mv html-website-templates/Animated\ Landing\ Page\ Website\ Template/* .
rm -rf html-website-templates/

You should have the index.html file in the root of our repo. Push to ADO.

git add .
git commit -m "Updated website"
git push

Azure Static Web App

Log to your Azure subscription and create or use an existing resource group. I’ll call mine rgStaticWebApp. Go to the Static Web Apps menu and create a new Static Web App. Specify a subscription, a resource group and name it. In my case it’s called mystaticwebapp.

Make sure you choose the Standard hosting plan. SSO won’t work with the free plan. Choose the repo location, the branch and HTML for Build Preset.
Use the defaults for the rest and click on Create. The static web site will be deployed in 5-10 seconds and if you go to the resource, click Overview, you’ll see the URL of your website. Go to that URL and you’ll see our website is public.

Custom domain (optional)

In case you want a custom domain, then these changes are required. This is not a requirement for the SSO, but it’s better if you have your own domain URL.
Click on Custom Domains and then choose where is your domain hosted. In my case I have it with CloudFlare.

Enter how you want to access your website, in my case it is mysite.andreev.it.
You’ll have to create a CNAME in CloudFlare.

Do not proxy it thru CloudFlare for now.

Click on Add at the bottom and it might take some time for Azure to verify the new CNAME record. In my case it took less than a minute.

…and if you go to whateveryou.namedyour.site, you should see the same static web page.

Entra ID (Azure AD) SSO

In order to add SSO, we have to add a file in the root of our static web site and then make some changes in Azure. Refer to this link for more information.
Create the following file in the root of the static web site and name it staticwebapp.config.json. It has to be this name.
This is what the file looks like.

{
  "routes": [
    {
      "route": "/*",
      "allowedRoles": ["authenticated"]
    }
  ],
  "responseOverrides": {
    "401": {
      "statusCode": 302,
      "redirect": "/.auth/login/aad"
    }
  },
  "auth": {
    "identityProviders": {
      "azureActiveDirectory": {
        "registration": {
          "openIdIssuer": "https://login.microsoftonline.com/TENANT_ID/v2.0",
          "clientIdSettingName": "AZURE_CLIENT_ID",
          "clientSecretSettingName": "AZURE_CLIENT_SECRET"
        }
      }
    }
  }
}

We need 3 values. The first value, TENANT_ID needs to go to this file hardcoded and the other two values for the CLIENT_ID and CLIENT_SECRET will be added in Azure. Here is how to fine these values. Go to Entra ID and click Overview. This is your TENANT_ID value. Copy and paste that value in line 18 and save the file. After, git add/commit/push the file back to Azure DevOps repo.

Go to App Registrations and click on New registration.
Enter these values and click Register. Replace the URL with yours even if it’s not a custom domain. Add .auth/login/aad/callback as a suffix.

The app registers right away and if you click the Overview you’ll see the app client ID. Copy this value in notepad or somewhere.

From the menu on the left, click Certificates & secrets and then click on New client secret.

Name the secret and choose for how long it’s valid.

You’ll see that a secret has been created and you’ll have a value and a secret ID. Copy the value not the ID in notepad or somewhere.
Now, go to the Static Web App and click on Environment variables. Add both AZURE_CLIENT_ID and AZURE_CLIENT_SECRET and click Apply at the bottom.

Finally, go to App Registration, find your app there, click Authentication on the left and make sure that ID tokens option is checked.

Now, if you go to the URL of the app, you’ll be prompted to log in and after successfully logging in, you’ll see our static web page. But every user from your tenant can log in, the only restriction is that you are member of the Entra ID tenant. If you want to restrict the website only to specific people, do the following. Go to Enterprise Applications, find your application and click on Properties. Make sure Assignment required is set to Yes. Click Save on top.

Then, click on Users and Groups and assign a group or users to the app. Mind that if you use the free version of Entra ID, you won’t be able to assign groups, only users.

In this post I’ll describe how to integrate Okta and your AWS account. In order everything to work, you’ll have to enable AWS Organization on the master account and then you can add multiple sub accounts in AWS.
Best practice says that the AWS master account should be used for SSO only and you shouldn’t run any workloads. First thing first, let’s start with Okta.

Okta

Log to your Okta account or sign up for a 30 day trial. Once logged in, if you haven’t already done so, change the sign-in domain from trail*-okta.com to your domain. Or you can skip this step.
Click on Admin from the upper right corner and then under Customizations, click on Domain. I’ve used the Okta-managed customization because it’s easier to manage.

After you click Next, you’ll be prompted to add two DNS entries for your domain.

Change the DNS. This will allow us to access Okta dashboard using our domain in my case it’s okta.andreev.it. From the same menu on the left, click on Applications, then Applications again and click on Browse App Catalog.

Search for AWS IAM and choose AWS IAM Identity Center.

Click on Add Integration and then Done.
Click on the Sign On tab and scroll all the way down to SAML Signing Certificates. Click on Actions drop-down and then View IdP metadata.

This will open a new tab with an XML file. Save the file as metadata.xml.

AWS

Go to console.aws.amazon.com and log with your root account. Go to IAM Identity Center and click on Enable and then select Enable with AWS Organizations.


Click on Settings on the left, then Actions drop-down and click on Customize AWS access portal URL. Change the URL so it’s something that you can easily remember, e.g. yourorg or aws-master.

Click on Settings on the left, then Actions drop-down and click on Change identity source.

Choose External identity provider and click Next.

You’ll land on a page with two parts, Service provider metadata (that’s AWS) and Identity provider metadata (that’s Okta).
Click on Download metadata file from upper right. You’ll need this in Okta. The file will be named with some date prefix and some characters plus metadata.xml
Copy the values for the 2nd and 3rd entry. You’ll also need these for Okta.

Click on Choose file under Identity provider metadata and upload the metadata.xml file that you’ve downloaded earlier from Okta.

Click Next, type ACCEPT and finally click Change identity source.

Okta

Go back to Okta, Sign On tab and right bellow click Edit. Enter the two URL entries that you copied from AWS. Click Save after.

AWS

Go back to AWS, then IAM Identity Center, click on Settings and click to Enable the Automatic Provisioning.

Click on Show Token and copy both values for the SCIM endpoint and the Access Token.

Okta

Go back to Okta and for the AWS app, click on Provisioning and then click on Configure API Integration.

Click Enable API Integration and enter the SCIM endpoint and the token value in the corresponding fields. Click Save. Remove the trailing slash from the end of the Base URL (SCIM endpoint) if you get an error (Base URL: Does not match required pattern)
Under Provisioning tab, click on To App, click Edit and select all three checkmarks. Click Save.

Then, go to Assignments tab and assign a group or a user to this AWS app.

AWS

Go back to AWS and under IAM Identity Center and then Users, you can see the user that we just assigned.

But this user has no rights in AWS, so we have to create a Permission set and then assign this user to that permission set and the account that this user has access to. Click on Permission sets on the left and then Create permission set.

I chose AdministratorAccess from the Predefined permission sets.

Go back to AWS accounts, click on the account and choose Assign users or groups.

Select the User tab, click on the user, click Next.

Select the permission set that we just created and click Next then Submit.

We just told AWS that the user xyz will have access to the account abc with AdministratorAccess permissions.

Test SSO

Open a new tab and go to the URL that you configured earlier (Customize AWS access portal). It should be https://whatever.awsapps.com/start.
AWS will redirect you to Okta where you’ll enter your Okta username.

Once authenticated, you’ll see the AWS accounts in your org and the permissions that you have.

I wrote about this topic a couple of years ago, but some things changed so I am writing this post again. In this post I’ll explain how to log to AWS Console and AWS cli with Entra ID (former Azure AD) credentials. For this you’ll need admin access to both the AWS account and Entra ID.

IAM Identity Center

The former AWS Single Sign-on is now IAM Identity Center. You need to enable this in order to configure SSO. Go to IAM Identity Center and click on the Enable button.

Enable it with AWS Organizations.

What you want to do first is to change the AWS access portal URL. Click on the Edit button and change the URL. It has to be unique, so don’t expect you can type aws and be done with it.

Another setting that you have to change is the Identity source. On the left side of the IAM Identity Center, you are on the Dashboard screen. Click on Settings and then click on Change identity source.

Choose External identity provider and click Next.

You will see this screen which consists of two parts, service provider metatada (that’s AWS) and Identity provider metadata (that’s Entra ID). Click on Download metadata file. You’ll get an XML file.

Keep the page open and in a new browser window go to portal.azure.com.

Entra ID

Once logged to Azure, go to Enterprise Applications and search for AWS IAM Identity Center (successor to AWS Single Sign-On). Click to install. When you are presented with the Overview of the application, click on Assign users and groups. Depending on your Entra ID level (P1 or P2) you might not be able to assign groups, so you have to assign users.

The next step is to setup the SSO. Click on 2. Set up single sign on and then SAML. You’ll be presented with this page and eventually you’ll see a prompt if you want to save some SSO settings. If you don’t see the prompt, no worries. If you see the prompt, click Yes.

Click on Upload metadata file and point to the XML file that you saved from AWS. Click Save and don’t worry about any of the fields there. If you see a popup to test the config, just click No, I’ll test later. Under step 3, SAML Certificates, click to Download Federation Metadata XML.

You’ll get another XML file called AWS IAM Identity Center (successor to AWS Single Sign-On).xml.

Back to AWS

Go back to the same page where you’ve downloaded the XML metadata file, but this time look at the bottom part where it says Identity provider metadata and IdP SAML metadata. Click on Choose file and upload the file from Azure. Click Next, type ACCEPT and click Change identity source.
Once completed click to Enable Automatic provisioning. Copy the SCIM endpoint and the access token.

Back to Azure

Under the AWS SSO application. on the left side click on Provisioning. Change the Provisioning Mode to Automatic. Expand Admin credentials and for Tenant ID enter the SCIM endpoint and for Secret Token, enter the token. Click Test Connection and upon successful test, click Save. If you see clientsecret instead of Tenant ID, it means you are not using the right AWS application and you use the AWS Single Account application instead. Start from scratch.
Anytime you assign a new user or a group to AWS application, Azure will sync the user to AWS. But, instead of waiting (40 mins by default) for the initial sync, click on the Overview and click Start provisioning.

Back to AWS

If your provisioning works fine, you’ll see your assigned Azure user under Users in IAM Identity Center. On the left side of the IAM Identity Center, click on Permission sets. Click Create Permission set and choose Predefined permission set, then AdministratorAccess. Change the name, description, session duration if needed and then click Next and then Create.
Finally, click on AWS accounts, select an account from the organization, click Assign users or groups, click Users or Groups, select a user and click Next. Assign a permission set, click Next and Submit.

Now, test the SSO by going to whatever.awsapps.com/start# and you’ll be redirected to the Microsoft sign in prompt. Follow the authentication process for Entra ID (username, password, MFA etc) and once you pass the authentication process you’ll see the AWS account that you have access to.

AWS CLI and SSO

Follow the instructions to install AWS CLI for your OS, then configure SSO. Name your session, enter the URL you configured earlier, enter the region where you configured SSO and hit enter for the registration scopes.

aws configure sso
SSO session name (Recommended): sso
SSO start URL [None]: https://your_url.awsapps.com/start
SSO region [None]: us-east-1
SSO registration scopes [sso:account:access]:
Attempting to automatically open the SSO authorization page in your default browser.
If the browser does not open or you wish to use a different device to authorize this request, open the following URL:

https://device.sso.us-east-1.amazonaws.com/

Then enter the code:

PWKG-CAFZ

After successful authentication, you’ll be presented with the accounts available to you. Pick one.

You’ll be prompted to choose the roles available to you and then specify your default region, default output and the profile. \
NOTE: If you’ll be working with multiple accounts, maybe you should specify a different profile than the default. It’s up to you. E.g. specify prod and dev profiles.

There are 2 AWS accounts available to you.
Using the account ID 123456789012
The only role available to you is: AdministratorAccess
Using the role name "AdministratorAccess"
CLI default client Region [None]: us-east-2
CLI default output format [None]: json
CLI profile name [AdministratorAccess-123456789012]: default

To use this profile, specify the profile name using --profile, as shown:

aws s3 ls --profile default

After your SSO session expires, you have to log back in.

aws sso login

If you are like me and always uses the default profile so I don’t have to type –profile parameter, then anytime you want to use a different account, just aws configure sso again, but specify the other account now.

In one of my previous posts, I wrote about accessing DynamoDB and AWS Gateway/Lambda using IAM roles and secrets from EC2/EKS Node.js application. The idea is to reference passwords in your code and never store and secrets/passwords in the code. By using roles, we can access the secrets from the AWS Secrets Manager and use them dynamically in the code. However, that solution has its own pros and cons. The pros are that you don’t have to do anything extra except creating an IAM role and secrets. The cons are that you need AWS SDK as part of your application. You will use AWS SDK for Node.js to get the credentials from the Secrets Manager. This means you have to maintain up-to-date AWS SDK and add extra code. Another way is to use the k8s CSI driver and ASCP (provider) that pretty much gets the secret and mounts it as a file. You still need IAM roles to access the Secrets Manager but you don’t need AWS SDK. You can read the secrets as you read a file.

Create an EKS cluster

We’ll create an EKS cluster using the eksctl tool.

CLUSTER_NAME="eksDemoCluster"
REGION="us-east-2"
NAMESPACE="default"

eksctl create cluster --name $CLUSTER_NAME \
  --region $REGION --version 1.28 \
  --node-type t3.small --nodes 2

We’ll use helm to install the CSI driver and the specific ASCP provider for AWS.

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm repo add aws-secrets-manager https://aws.github.io/secrets-store-csi-driver-provider-aws
helm repo update
helm install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver
helm install -n kube-system secrets-provider-aws aws-secrets-manager/secrets-store-csi-driver-provider-aws

Check if everything is OK.

kubectl --namespace=kube-system get pods -l "app=secrets-store-csi-driver"
NAME                                               READY   STATUS    RESTARTS   AGE
csi-secrets-store-secrets-store-csi-driver-nhq7b   3/3     Running   0          19s
csi-secrets-store-secrets-store-csi-driver-tft6q   3/3     Running   0          19s

Create a secret

Let’s create a secret that we’ll be using in a mock-up app.

SECRET_NAME="demoapp/mysecret"
SECRET_ARN=$(aws secretsmanager create-secret  --name $SECRET_NAME \
  --secret-string '{"username":"Administrator", "password":"SuperSecretPassword$"}' \
  --output text --query 'ARN' --region $REGION)
echo $SECRET_ARN
arn:aws:secretsmanager:us-east-2:123456789012:secret:demoapp/mysecret-lboyfd

Create an IAM policy

We also need to define an IAM policy that will allow our EKS pods to access only this particular secret and nothing else.

cat << EOF > polDemoSecret.json
{
    "Version": "2012-10-17",
    "Statement": [ {
        "Effect": "Allow",
        "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
        "Resource": ["$SECRET_ARN"]
    } ]
}
EOF

Create the policy.

POLICY_ARN=$(aws iam create-policy --policy-name polDemoSecret \
  --policy-document file://polDemoSecret.json \
  --output text --query 'Policy.Arn')
echo $POLICY_ARN
arn:aws:iam::123456789012:policy/polDemoSecret

Create an IAM OIDC provider

Let’s determine whether we have an existing IAM OIDC provider for our cluster.

aws eks describe-cluster --name $CLUSTER_NAME --region $REGION \
  --query "cluster.identity.oidc.issuer" --output text
https://oidc.eks.us-east-2.amazonaws.com/id/D7FAD6F2BEF430FC1CB673777A9E4FED

When we provisioned the cluster with eksctl, the OIDC was created automatically. We need the ID of the OIDC which is the hex value at the end.

OIDCID=$(aws eks describe-cluster --name $CLUSTER_NAME --region $REGION \
  --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)
echo $OIDCID
E9B5E80C37A71FA3932B94EFF5BA3437

Check if the IAM OIDC provider is already configured. It shouldn’t if you provisioned a brand new cluster.

aws iam list-open-id-connect-providers | grep $OIDCID 

If for whatever reason, you had an output from the command above, skip this associate-iam-oidc-provider step below.
If you don’t have any output, associate the OIDC provider with the cluster.

eksctl utils associate-iam-oidc-provider --cluster $CLUSTER_NAME --region $REGION --approve
2023-12-22 12:27:47 [ℹ]  will create IAM Open ID Connect provider for cluster "eksDemoCluster" in "us-east-2"
2023-12-22 12:27:47 [✔]  created IAM Open ID Connect provider for cluster "eksDemoCluster" in "us-east-2"

Cluster service account

We also need a service account for the cluster and we need to attache the policy for the secrets that we created before.

SA_NAME="sademosecret"
eksctl create iamserviceaccount --name $SA_NAME --namespace $NAMESPACE \
--cluster $CLUSTER_NAME  --region $REGION \
--attach-policy-arn $POLICY_ARN --approve --override-existing-serviceaccounts
kubectl get sa

Service Provider Class

…and finally we need a service provider class that we’ll deploy on the EKS cluster.

cat << EOF > spc.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: spc-demosecret
  namespace: $NAMESPACE
spec:
  provider: aws
  parameters:
    objects: |
        - objectName: "$SECRET_NAME"
          objectType: "secretsmanager"
EOF
kubectl apply -f spc.yaml

Test with busybox pod

Create this YAML file that will deploy a pod. Pay attention to highlighted lines. We have to provide the namespace, the service account and the service provider class that specifies the secret.Check this link on how to mount and specify different secrets.

cat << EOF > app.yaml
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: $NAMESPACE
spec:
  serviceAccountName: $SA_NAME
  volumes:
  - name: secretsvolume
    csi:
       driver: secrets-store.csi.k8s.io
       readOnly: true
       volumeAttributes:
         secretProviderClass: "spc-demosecret"
  containers:
  - image: public.ecr.aws/docker/library/busybox:1.36
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox
    volumeMounts:
    - name: secretsvolume
      mountPath: "/mnt/secrets-store"
      readOnly: true
  restartPolicy: Always
EOF
kubectl apply -f app.yaml

Test the secrets retrieval. As you can see, the secret that we created was demoapp/mysecret, but in this case the forward slash would mean a directory, so the actual secret is demoapp_mysecret… and it’s simple as that. The secret is just a file under the /mnt/secrets-store directory.

kubectl exec -it $(kubectl get pods | awk '/busybox/{print $1}' | head -1) -- cat /mnt/secrets-store/demoapp_mysecret; echo
{"username":"Administrator", "password":"SuperSecretPassword$"}

Node.js app

Create a folder and add this file. Save it as server.js.

'use strict';
const express = require('express')
const app = express()
const port = 3000
 
var fs = require('fs');
try {  
    var data = fs.readFileSync('/mnt/secrets-store/demoapp_mysecret', 'utf8');
    console.log(data.toString());    
} catch(e) {
    console.log('Error:', e.stack);
}

app.get('/', (req, res) => {
    res.send(JSON.stringify(data));
})
 
app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

Let’s create the Dockerfile so we can build the image.

FROM node:20
  
# Create app directory
WORKDIR /usr/src/app
  
# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm@5+)
COPY package*.json ./
  
RUN npm install
# If you are building your code for production
# RUN npm ci --only=production
  
# Bundle app source
COPY . .
  
EXPOSE 3000
CMD [ "node", "server.js" ]

Create a .dockerignore file.

node_modules
npm-debug.log

Now, we can create the image and push to Dockerhub or some other image repo. Use your own repo here at line 1.

sudo docker build . -t klimenta/secretsdemo
sudo docker images
sudo docker push klimenta/secretsdemo

Create a file to deploy 2 replicas and a load balancer. Save it as something.yaml and deploy it. Analyze the file and see how we use the service account, secret provider class and the image.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demosecret
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      run: demosecret
  template:
    metadata:
      labels:
        run: demosecret
    spec:
      serviceAccountName: sademosecret
      volumes:
      - name: secretsvolume
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "spc-demosecret"
      containers:
      - name: demosecret
        image: klimenta/secretsdemo
        volumeMounts:
        - name: secretsvolume
          mountPath: "/mnt/secrets-store"
          readOnly: true
        ports:
        - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: loadbalancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
spec:
  ports:
    - port: 80
      targetPort: 3000
      protocol: TCP
  type: LoadBalancer
  selector:
    run: demosecret

Once you deploy with kubectl apply -f something.yaml, type kubectl get svc and you’ll see a public URL.
If you go to that URL, you’ll see our secret.

Recently I was playing with Kubernetes and racked a substantial bill in AWS and Azure so I decided to bring the lab home. I needed a small but powerful PC that can run multiple test VMs and Kubernetes clusters. Kubernetes on a bare metal is different than EKS or AKS, especially when it comes to handling load balancers, ingress, IAM roles etc. Some of these things don’t exist on a bare metal machine.
I bought a NUC 13 small factor PC from Amazon and added Samsung NVMe 2TB SSD and Crucial 64GB RAM. After 2 months (these things are expensive!!!), I bought a Synology NAS with 2 x 8GB SATA disks. Next, is to upgrade the NAS with SSD that will act as cache, but I am not in a rush. Nevertheless…

Initial SSD, RAM setup

First thing you need to do is to CAREFULLY open the PC and make sure you don’t pull too hard and break the ribbon. Good thing is that the screws are not coming off and there is a spring mechanism so it’s super easy to open and close the bottom.

Add the NVMe SSD on the left side, but remove the screw first, bottom left. You’ll use it to tighten the SSD after.

Add the memory on the right side, you can add up to 64GB RAM hence 2 x 32GB modules. If you boot the PC even with a bootable USB stick, you’ll get an error saying “A bootable device has not been detected”.

Hit CTRL-ALT-DEL and then F2 to enter BIOS. From there, go to Boot menu, then Boot Priority.

Check Boot USB Devices First.

If you boot from ESXi 8 ISO now, most likely you’ll get an Fatal CPU mismatch on feature PSOD error. So, while still in the BIOS, go to Power, Performance and Cooling tab. Expand the first item External Ambient Temperature Tolerance and change the temperature setting to User Defined.

Now, you can enter the Performance menu which was grayed if the setting is not User Defined for the External Ambient Temperature Tolerance.

Once there, go to Processor.

Once there, change the number of Active Efficient Cores to 0.
In addition, when you install ESXi 8 and when you access the GUI, you’ll see a nagging warning that TPM cannot be contacted or something similar. No matter how many times you acknowledge this warning it keeps coming back. So, while you in the BIOS, go to Security tab, then Security Features and uncheck Intel Platform Trust Technology.
Save the settings with F10.

Install ESXi 8

At this point, reboot and boot from ESXi ISO USB. Check this guy for more info (link and link).
I have a VMUG Advantage subscription so I can freely download all VMWare software from here. As long as you pay the yearly fee, you can have the licenses and software indefinitely.
Choose where to install the ESXi (Samsung SSD).

Choose a root password.

Remove the USB stick and reboot when the installation is over and you’ll see this screen after.

This is my DHCP assigned address, so just change it to a static IP providing DNS and the default gateway. Then, change your FQDN as well. In my case I named my box as esxi1.homelab.local.

At this point you have a functioning ESXi server. Open up a browser, go to the ESXi IP and log with root and the password you specified.
If you have a license, add it now by going to Manage (upper left corner), then Licensing tab and click Assign license or skip this part and use the evaluation mode.

Optional: Click on Storage then datastore1, Actions from the menu and rename the datastore. In my case I’ll use ESX1-LOCAL.

Install vCenter 8.0 appliance

Now, we can install vCenter. Mount the vCenter ISO image, go to vcsa-ui-installer folder and choose the OS (win32, mac, lin64). Run the installer and click Install icon.

Click Next and Accept the legal terms. Then enter the IP address or FQDN if you have your DNS configured for the ESXi server you just installed. Specify the root password too.

Name the vCenter and specify the vCenter root password. This is not the password that you will use to log to vCenter. This is a password for the appliance itself. You’ll need this only if you have to troubleshoot the appliance or enable/disable SSH on the appliance.

Choose deployment size, for me the smallest size is acceptable. After all, I plan to use this primarily for Kubernetes.

Choose the datastore.

Configure network settings, IP, gateway, DNS, FQDN…

Confirm and click Finish. After some time, you’ll get a prompt to configure vCenter. I chose to use NTP and sync with some public NTP servers. Use whatever is closest to you.

I’ll create a new SSO domain and I’ll use the defaults. I just have to provide a password.

Once vCenter is deployed go to the FQDN or the IP from a browser. Your username is no longer root, it’s administrator@vspehere.local and your password is the one you specified when you installed vCenter SSO in the previous step. Enter the license by going to Administration | Licenses from the left.

Once you enter the license from the Licenses tab, go to Assets tab and click Assign License.

Go to Inventory from the hamburger menu on the left, right-click the vCenter IP or FQDN and choose New Datacenter.

Then right-click on the newly created datacenter and choose New Cluster.
Same thing again, right-click the cluster and choose Add Hosts.

Specify the IP or FQDN and the root password when you installed ESXi initially.

Optional: iSCSI

Click on the ESXi hosts, then Configure tab, click Add Software Adapter and then Add iSCSI adapter.

In my case, I have to use static discovery so click on that tab and then Add.

Enter the iSCSI IP and the target name.

Finally, click on Rescan storagein the menu above. If you need to provide your initiator, this is where you can find it under iSCSI Name (iqn.1998—_.

To learn how you can install Kubernetes on your NUC and how to use k8s Load Balancing see the links from my blog here and here.

In this post, I’ll explain how to create an EKS cluster in AWS and add an Ingress Controller. We’ll use GitLab as a repo and execute the CI/CD pipeline from there.
For this you’ll need:

• GitLab and AWS account
• Terraform, AWS CLI, helm installed

AWS

From the IAM console, create a new Identity Provider in AWS and use https://gitlab.com for both Provider URL and Audience. For self hosted GitLab use gitlab.domain.com or whatever is your GitLab URL.
Click on Get thumbprint before clicking on Add provider.

In the IAM console, create a role. I’ll name the role rolGitLab, name it whatever you like. When you create the policy, choose Custom trust policy.

Now, here is what you have to decide. If you want to manage the cluster from GitLab only, you can ommit lines 16-23 where I allow the IAM user called gitlab to assume this role. But if you want some IAM user to manage the cluster using aws cli or kubectl, add the ARN of that IAM user in line 20. Add AdministratorAccess to that gitlab user but you can fine-tune those policies later. If you want to manage the cluster from GitLab, you have to register an agent and provide a config file. You can read how to do that here.
Replace the default custom trust policy with this JSON.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Principal": {
				"Federated": "arn:aws:iam::123456789012:oidc-provider/gitlab.com"
			},
			"Action": "sts:AssumeRoleWithWebIdentity",
			"Condition": {
				"StringLike": {
					"gitlab.com:sub": "project_path:*/eks-gitlab:ref_type:branch:ref:main"
				}
			}
		},
		{
			"Sid": "",
			"Effect": "Allow",
			"Principal": {
				"AWS": "arn:aws:iam::123456789012:user/gitlab"
			},
			"Action": "sts:AssumeRole"
		}
	]
}

In line 7 and 20, change the 123456789012 with your account number and in line 7 change gitlab.com with however you named the Identity Provider (IdP). In line 12 replace eks-gitlab with how you plan to name your GitLab project. In line 20 change the gitlab user to however you named your IAM user to access the EKS cluster. Click Next and when asked for a permission policy, choose AdministratorAccess. Best practice is to limit access to only what’s needed, but EKS has so many policies, it will take a lot to list them all.
In case you created the gitlab IAM user, attach the AdministratorPolicy (fine tune later) and this inline policy. Change the ARN in line 7. Name this inline policy something like polAssumeGitLabRole.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::123456789012:role/rolGitLab"
        }
    ]
}

GitLab

Create a new blank project in Gitlab with the default settings. In my case the project name is called eks-gitlab.

Go to Settings on the left, then CI/CD and then Variables. Add these two variables.

• AWS_PROFILE and a value of oidc. Yes, the value is oidc. Just type these 4 characters for the value.
• ROLE_ARN and a value of the role ARN from the role you just created for rolGitLab

Create a .gitlab-ci.yml file with the following content. Change the domain for gitlab.com at line 7 if you use self-hosted GitLab.

assume role:
  image: 
    name: amazon/aws-cli:latest
    entrypoint: [""]
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com
  before_script:
    - mkdir -p ~/.aws
    - echo "${GITLAB_OIDC_TOKEN}" > /tmp/web_identity_token
    - echo -e "[profile oidc]\nrole_arn=${ROLE_ARN}\nweb_identity_token_file=/tmp/web_identity_token" > ~/.aws/config
  script:
      - aws sts get-caller-identity
      - aws s3 ls

By the time you commit changes, the pipeline will start and if everything is OK, you’ll see your S3 buckets in the logs of the job if you go to Build | Jobs.

One more thing related to Terraform state as we’ll keep the state in GitLab is that we have to create an Access Token. If you have a self-hosted or paid GitLab account, create an Access Token from the Project | Settings | Access Tokens.
If you have a free GitLab account, you have to create a Personal Access Token and you do that if you click on your avatar in the upper left corner, then select Edit Profile and then click Access Tokens on the left. Click Add new token, name it GITLAB_PAT and select API only. Click Create personal access token. Once the token has been created click on the eye icon where it says Your new personal access token and copy & paste somewhere safe.

Go to the project settings and add these 2 variables.

• TF_USERNAME which is your username, click on the avatar, look at the @ under your name, your username is without the @ sign
• TF_PASSWORD which is the value of your personal access token you just created

Local machine

Now that we know the pipeline is working properly, let’s clone the project on our local machine.
If you plan to manage the cluster from your local machine, now it’s time to configure the gitlab IAM user on your local machine using access keys.
You need to have a valid key in order to clone your private repo.
Upload your .pub key to GitLab by clicking your avatar, then Edit Profile, click SSH Keys and then Add new key.
Finally, check the connection with the following command. It doesn’t matter what your local Linux user is, just use git@gitlab.com as the username@hostname. Type as is.

ssh -T git@gitlab.com
Welcome to GitLab, @[your_gitlab_username_here]

Once you clone the repo add the following files locally.
FILE: .gitignore

# Local .terraform directories
**/.terraform/*

# .tfstate files
*.tfstate
*.tfstate.*

# Crash log files
crash.log
crash.*.log

# Exclude all .tfvars files, which are likely to contain sensitive data, such as
# password, private keys, and other secrets. These should not be part of version 
# control as they are data points which are potentially sensitive and subject 
# to change depending on the environment.
*.tfvars
*.tfvars.json

# Ignore override files as they are usually used to override resources locally and so
# are not checked in
override.tf
override.tf.json
*_override.tf
*_override.tf.json

# Include override files you do wish to add to version control using negated pattern
# !example_override.tf

# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
# example: *tfplan*

# Ignore CLI configuration files
.terraformrc
terraform.rc

FILE: LICENSE

This is free and unencumbered software released into the public domain.

Anyone is free to copy, modify, publish, use, compile, sell, or
distribute this software, either in source code form or as a compiled
binary, for any purpose, commercial or non-commercial, and by any
means.

In jurisdictions that recognize copyright laws, the author or authors
of this software dedicate any and all copyright interest in the
software to the public domain. We make this dedication for the benefit
of the public at large and to the detriment of our heirs and
successors. We intend this dedication to be an overt act of
relinquishment in perpetuity of all present and future rights to this
software under copyright law.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.

For more information, please refer to <https://unlicense.org>

FILE: alb_ingress_iam_policy.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:CreateServiceLinkedRole"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeAccountAttributes",
                "ec2:DescribeAddresses",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeInternetGateways",
                "ec2:DescribeVpcs",
                "ec2:DescribeVpcPeeringConnections",
                "ec2:DescribeSubnets",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeInstances",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeTags",
                "ec2:GetCoipPoolUsage",
                "ec2:DescribeCoipPools",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeLoadBalancerAttributes",
                "elasticloadbalancing:DescribeListeners",
                "elasticloadbalancing:DescribeListenerCertificates",
                "elasticloadbalancing:DescribeSSLPolicies",
                "elasticloadbalancing:DescribeRules",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeTargetGroupAttributes",
                "elasticloadbalancing:DescribeTargetHealth",
                "elasticloadbalancing:DescribeTags"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "cognito-idp:DescribeUserPoolClient",
                "acm:ListCertificates",
                "acm:DescribeCertificate",
                "iam:ListServerCertificates",
                "iam:GetServerCertificate",
                "waf-regional:GetWebACL",
                "waf-regional:GetWebACLForResource",
                "waf-regional:AssociateWebACL",
                "waf-regional:DisassociateWebACL",
                "wafv2:GetWebACL",
                "wafv2:GetWebACLForResource",
                "wafv2:AssociateWebACL",
                "wafv2:DisassociateWebACL",
                "shield:GetSubscriptionState",
                "shield:DescribeProtection",
                "shield:CreateProtection",
                "shield:DeleteProtection"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:RevokeSecurityGroupIngress"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateSecurityGroup"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags"
            ],
            "Resource": "arn:aws:ec2:*:*:security-group/*",
            "Condition": {
                "StringEquals": {
                    "ec2:CreateAction": "CreateSecurityGroup"
                },
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags",
                "ec2:DeleteTags"
            ],
            "Resource": "arn:aws:ec2:*:*:security-group/*",
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:DeleteSecurityGroup"
            ],
            "Resource": "*",
            "Condition": {
                "Null": {
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:CreateLoadBalancer",
                "elasticloadbalancing:CreateTargetGroup"
            ],
            "Resource": "*",
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:CreateListener",
                "elasticloadbalancing:DeleteListener",
                "elasticloadbalancing:CreateRule",
                "elasticloadbalancing:DeleteRule"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:AddTags",
                "elasticloadbalancing:RemoveTags"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ],
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:AddTags",
                "elasticloadbalancing:RemoveTags"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:ModifyLoadBalancerAttributes",
                "elasticloadbalancing:SetIpAddressType",
                "elasticloadbalancing:SetSecurityGroups",
                "elasticloadbalancing:SetSubnets",
                "elasticloadbalancing:DeleteLoadBalancer",
                "elasticloadbalancing:ModifyTargetGroup",
                "elasticloadbalancing:ModifyTargetGroupAttributes",
                "elasticloadbalancing:DeleteTargetGroup"
            ],
            "Resource": "*",
            "Condition": {
                "Null": {
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:AddTags"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ],
            "Condition": {
                "StringEquals": {
                    "elasticloadbalancing:CreateAction": [
                        "CreateTargetGroup",
                        "CreateLoadBalancer"
                    ]
                },
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:RegisterTargets",
                "elasticloadbalancing:DeregisterTargets"
            ],
            "Resource": "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:SetWebAcl",
                "elasticloadbalancing:ModifyListener",
                "elasticloadbalancing:AddListenerCertificates",
                "elasticloadbalancing:RemoveListenerCertificates",
                "elasticloadbalancing:ModifyRule"
            ],
            "Resource": "*"
        }
    ]
}

FILE: backend.tf

terraform {
  backend "http" {
  }
}

FILE: data.tf

data "aws_availability_zones" "available" {
}

data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_name
  depends_on = [module.eks.cluster_name]
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_name
  depends_on = [module.eks.cluster_name]
}

FILE: main.tf NOTE: Make sure you add lines 40-41 and specify the rolGitlab role created earlier. If you don’t do this when you destroy the cluster, the KMS key for EKS will stay there and there is no way you can delete it unless you call AWS support.

# Create a VPC
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.0.0"

  name = var.vpcName

  cidr = var.vpcCIDR
  azs  = slice(data.aws_availability_zones.available.names, 0, 3)

  private_subnets = var.privateSubnets
  public_subnets  = var.publicSubnets

  enable_nat_gateway   = var.enable_nat_gateway
  single_nat_gateway   = var.single_nat_gateway
  enable_dns_hostnames = var.enable_dns_hostnames

  public_subnet_tags = {
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
    "kubernetes.io/role/elb"                    = 1
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
    "kubernetes.io/role/internal-elb"           = 1
  }
}

# Create an EKS cluster
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "19.15.3"

  cluster_name    = var.cluster_name
  cluster_version = var.cluster_version

  vpc_id                         = module.vpc.vpc_id
  subnet_ids                     = module.vpc.private_subnets
  cluster_endpoint_public_access = var.cluster_endpoint_public_access
  kms_key_administrators         = ["arn:aws:iam::123456789012:role/rolGitLab"]
  kms_key_enable_default_policy  = true

  eks_managed_node_group_defaults = {
    ami_type = var.ami_type
  }

  eks_managed_node_groups = {
    one = {
      name = var.nodeGroup

      instance_types = var.instanceTypes

      min_size     = var.minSize
      max_size     = var.maxSize
      desired_size = var.desiredSize
    }
  }
}

# Create an IAM policy for the ALB ingress
resource "aws_iam_policy" "worker_policy" {
  name        = "worker-policy"
  description = "Worker policy for the ALB Ingress"

  policy = file(var.iamALBIngressPolicyFileName)
}

resource "aws_iam_role_policy_attachment" "additional" {
  for_each = module.eks.eks_managed_node_groups

  policy_arn = aws_iam_policy.worker_policy.arn
  role       = each.value.iam_role_name
}

# Crete the Ingress using Helm chart from AWS
resource "helm_release" "ingress" {
  name       = "ingress"
  chart      = "aws-load-balancer-controller"
  repository = "https://aws.github.io/eks-charts"
  version    = "1.6.1"

  set {
    name  = "autoDiscoverAwsRegion"
    value = "true"
  }

  set {
    name  = "autoDiscoverAwsVpcID"
    value = "true"
  }

  set {
    name  = "clusterName"
    value = module.eks.cluster_name
  }
}

FILE: outputs.tf

output "cluster_endpoint" {
  description = "Endpoint for EKS control plane"
  value       = module.eks.cluster_endpoint
}

output "cluster_security_group_id" {
  description = "Security group ids attached to the cluster control plane"
  value       = module.eks.cluster_security_group_id
}

output "region" {
  description = "AWS region"
  value       = var.region
}

output "cluster_name" {
  description = "Kubernetes Cluster Name"
  value       = module.eks.cluster_name
}

FILE: providers.tf

provider "aws" {
  region = var.region

  shared_config_files      = ["~/.aws/config"]  
  profile                  = "oidc"
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    token                  = data.aws_eks_cluster_auth.cluster.token
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  }
}

FILE: variables.tf

variable "region" {
  type    = string
  default = "us-east-2"
}

variable "vpcName" {
  type    = string
  default = "vpcFromGitLab"
}

variable "vpcCIDR" {
  type    = string
  default = "10.0.0.0/16"
}

variable "privateSubnets" {
  type    = list(string)
  default = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
}

variable "publicSubnets" {
  type    = list(string)
  default = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]
}

variable "enable_nat_gateway" {
  type    = bool
  default = true
}

variable "single_nat_gateway" {
  type    = bool
  default = false
}

variable "enable_dns_hostnames" {
  type    = bool
  default = true
}

variable "cluster_name" {
  type    = string
  default = "eksFromGitLab"
}

variable "cluster_version" {
  type    = string
  default = "1.27"
}

variable "cluster_endpoint_public_access" {
  type    = bool
  default = true
}

variable "ami_type" {
  type    = string
  default = "AL2_x86_64"
}

variable "nodeGroup" {
  type    = string
  default = "ngFromGitLab"
}

variable "instanceTypes" {
  type    = list(string)
  default = ["t3.small"]
}

variable "minSize" {
  type    = number
  default = 1
}

variable "maxSize" {
  type    = number
  default = 3
}

variable "desiredSize" {
  type    = number
  default = 3
}


variable "iamALBIngressPolicyFileName" {
  type    = string
  default = "alb_ingress_iam_policy.json"
}

…and finally your .gitlab-ci.yml file.
FILE: .gitlab-ci.yml. NOTE: Remove line 38 if you don’t use it, don’t keep it with a comment or you’ll get an error in the pipeline.

stages:          
  - validate
  - plan
  - apply
  - destroy

image:
  name: hashicorp/terraform:1.6
  entrypoint:
    - "/usr/bin/env"
    - "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

variables:
  TF_DIR: ${CI_PROJECT_DIR}
  TF_STATE_NAME: "eks-tf-state"
  TF_ADDRESS: "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/terraform/state/${TF_STATE_NAME}"

default:
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com

  before_script:
    - terraform --version
    - cd ${TF_DIR}
    - terraform init
      -reconfigure 
      -backend-config="address=${TF_ADDRESS}" 
      -backend-config="lock_address=${TF_ADDRESS}/lock" 
      -backend-config="unlock_address=${TF_ADDRESS}/lock" 
      -backend-config="username=${TF_USERNAME}" 
      -backend-config="password=${TF_PASSWORD}" 
      -backend-config="lock_method=POST" 
      -backend-config="unlock_method=DELETE" 
      -backend-config="retry_wait_min=5"
    - mkdir -p ~/.aws
    - echo "${GITLAB_OIDC_TOKEN}" > /tmp/web_identity_token
    - # echo "${GITLAB_OIDC_TOKEN}" | base64 - If you want to see the token, reveal it this way and then use base64 --decode to get the real one
    - echo -e "[profile oidc]\nrole_arn=${ROLE_ARN}\nweb_identity_token_file=/tmp/web_identity_token" > ~/.aws/config

job-validate:
  stage: validate
  script:
    - terraform validate

job-plan:
  stage: plan
  script:
    - terraform plan -out "tf_plan"
  dependencies:
    - job-validate
  artifacts:
    paths:
      - tf_plan

job-apply:
  stage: apply
  script:
    - terraform apply "tf_plan"
  dependencies:
    - job-plan
  allow_failure: true

job-destroy:
  stage: destroy
  script:
    - terraform destroy  -auto-approve
  dependencies:
    #- job-plan
    #- job-apply
  when: manual

Push the code back to Gitlab (git add, git commit, git push) and once everything completes successfully, you’ll have a working EKS cluster after 10-15 mins.
If you choose to manage the EKS from GitLab, you can skip to the next section. If you decide to manage the cluster outside GitLab and you already have another IAM user added to the role rolGitLab as described above, then do the following. Replace account # in line 1.

aws eks update-kubeconfig --region us-east-2 --name eksFromGitLab --role-arn arn:aws:iam::123456789012:role/rolGitLab
aws eks list-clusters
eksctl get cluster --name=eksFromGitLab --region us-east-2
kubectl get svc

Manage EKS from GitLab

Generally, you would want this to be in another project. Our first project was to build the cluster, but our second project builds on top of that and manages the cluster. In addition, the second project also uses the CI/CD pipeline file .gitlab-ci.yml so in order to avoid a single complex project – just create a new project. I’ll name my new project – project1. Clone the project on your local machine using git clone.
Create an empty config file for the agent on your local machine. It has to be under this folder structure.

mkdir -p .gitlab/agents/eks-agent && touch .gitlab/agents/eks-agent/config.yaml

In my case the name of the agent is eks-agent. Use whatever name you want but follow DNS naming convention.
Push the local changes to GitLab.
On the left side in GitLab, go to Operate | Kubernetes clusters and then click Connect a cluster from the upper right corner.
Select the agent from the drop down and click Register.

You’ll get an access token and a bunch of helm commands to install the agent on the cluster. Copy and paste and run those commands.

helm repo add gitlab https://charts.gitlab.io
helm repo update
helm upgrade --install eks-agent gitlab/gitlab-agent \
    --namespace gitlab-agent-eks-agent \
    --create-namespace \
    --set image.tag=v16.9.0-rc2 \
    --set config.token=glagent-2r_uW63e1w_mkNzR_keU2hmaNhhcKauSpujg8OdLQG74wrF73g \
    --set config.kasAddress=wss://kas.gitlab.com

Refresh the GitLab screen after the helm chart has been installed and you’ll see your cluster connected.

Now you need to authorize the agent to access our project.
In the blank config.yaml file on the local machine under .gitlab/agents/eks-agent/ directory, add these lines. Change the path the project in line 3 (look at the URL, that’s your project name after gitlab.com) and change the manifest file if you want in line 6. You can also change the namespace in line 5 as well.

gitops:
  manifest_projects:
  - id: andreevkliment/project1
    default_namespace: default
    paths:
      - glob: 'manifest.yaml'

Then create the manifest file under the root of the project. Save it as manifest.yaml or however you referenced it in line 6 in the previous file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

If you push this code back to GitLab, the agent will execute it and you’ll (with kubect get pods) see the nginx pods running on your cluster.
Another way of managing the EKS cluster is using .gitlab-ci.yml file.
Create .gitlab-ci.yml with these lines. Replace your username, project and the agent name in line 7.

deploy:
  image:
    name: bitnami/kubectl:latest
    entrypoint: ['']
  script:
    - kubectl config get-contexts
    - kubectl config use-context andreevkliment/project1:eks-agent
    - kubectl get pods

If you push this back to GitLab, the pipeline will start and you’ll see the pods from the cluster in the logs. This example doesn’t execute manifest.yaml. You can have both ways of interacting with the cluster or you can use them separately.

In this post I’ll explain how to install MetalLB which is a load balancer for bare metal/VM Kubernetes servers. If you have your own home lab to play with Kubernetes and you are not using AKS, EKS or GKE, then this post is for you.
The prerequisite is to have a running Kubernetes cluster. In my earlier post, I’ve described how to install Kubernetes with CRI-O and Cilium as a CNI, so you can follow that post or feel free to have your own install. Mind that if you use other CNIs such as Calico or Weave, there are some things you have to check first and see if it applies to you. Look at this link.
We’ll install MetalLB now. You probably have your k8s master and the nodes ready.

Install MetalLB

Do this first on the master.

kubectl edit configmap -n kube-system kube-proxy

…and change strictARP to true.

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"
ipvs:
  strictARP: true

Deploy MetalLB.

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.11/config/manifests/metallb-native.yaml

You’ll see the pods running in metallb-system namespace.

kubectl get pods -n metallb-system
NAME                          READY   STATUS    RESTARTS   AGE
controller-7d56b4f464-gkfrq   1/1     Running   0          55s
speaker-l5954                 1/1     Running   0          55s
speaker-xrswp                 1/1     Running   0          55s
speaker-z8vdt                 1/1     Running   0          55s

Let’s configure our load balancer to give IPs in this 192.168.1.50-192.168.1.60 range. Change to your needs accordingly in line 8.
Save it as a lb-config.yaml file.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: first-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.1.50-192.168.1.60
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: example
  namespace: metallb-system

Deploy the config.

kubectl apply -f lb-config.yaml

Now, we can deploy a simple app and a load balancer. Save the file as demo.yaml.
This is an app that I made in Node.js that listens on port 3000 to display the IP of the pod that the load balancer hits.
You can use it for any type of a load balancer, not just MetalLB. It’s a super simple app that prints the IP of the pod where it’s running.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
spec:
  replicas: 6
  selector:
    matchLabels:
      run: demo
  template:
    metadata:
      labels:
        run: demo
    spec:
      containers:
      - name: demo
        image: klimenta/serverip
        ports:
        - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: loadbalancer
spec:
  ports:
    - port: 80
      targetPort: 3000
      protocol: TCP
  type: LoadBalancer
  selector:
    run: demo

Deploy the app.

kubectl apply -f demo.yaml

Check the load balancer service.

kubectl get svc
NAME           TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)        AGE
kubernetes     ClusterIP      10.96.0.1      <none>         443/TCP        33m
loadbalancer   LoadBalancer   10.106.36.58   192.168.1.50   80:30770/TCP   4m33s

…and if you go to that IP (192.168.1.50) in your browser, you’ll see the IP of the pods that the load balancer hits.

Install nginx ingress reverse proxy

The problem with the scenario above is that for every service you need to expose, you need a different IP from the IP pool that we’ve assigned to MetalLB (.50-.60).
A better solution is to install an nginx ingress that will act as a reverse proxy, but for this you’ll need a working DNS server as the services will share the same IP, but different FQDN.
So, in practice it looks like this.
Install nginx ingress first.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.2/deploy/static/provider/cloud/deploy.yaml
kubectl get pods -n ingress-nginx
kubectl get service ingress-nginx-controller -n=ingress-nginx

Don’t worry if the two nginx ports are not running and it says – “completed”.
You’ll see the external IP that’s assigned to nginx from MetalLB.
For some reason, nginx ingress won’t work without this line. Many people have this problem in a bare-metal environment and this is the workaround.

kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission

Now, let’s create an nginx web server. This has nothing to do with nginx ingress, it’s completely separate.
I am exposing the nginx deploymenty and creating a rule that will hit the deployment on port 80 with nginx.homelab.local as FQDN.
NOTE: If your browser can’t resolve this URL, it won’t work. So, make sure you have a DNS or /etc/hosts file or some type of name resolution.

kubectl create deployment nginx --image=nginx --port=80
kubectl expose deployment nginx
kubectl create ingress nginx --class=nginx --rule nginx.homelab.local/=nginx:80

Then, let’s install Apache server that runs on the same IP and same port 80.

kubectl create deployment httpd --image=httpd --port=80
kubectl expose deployment httpd
kubectl create ingress httpd --class=nginx --rule httpd.homelab.local/=httpd:80

If everything is OK, you’ll be able to access both web servers using their URLs.
If you want to put Grafana and/or KubeCost behind nginx ingress, read these short posts here and here.
If you want to use yaml for deployment, here is an example of that. It creates 6 replicas of a pod that prints the IP where the pod is running. The service runs on port 3000. Then it creates a ClusterIP on the same port and puts the ClusterIP behind the ingress. Change the hostname at line 39.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
spec:
  replicas: 6
  selector:
    matchLabels:
      run: demo
  template:
    metadata:
      labels:
        run: demo
    spec:
      containers:
      - name: demo
        image: klimenta/getmyip
        ports:
        - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: demo
spec:
  type: ClusterIP
  ports:
  - port: 3000
    targetPort: 3000
  selector:
    run: demo
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-demo
spec:
  rules:
  - host: demo.homelab.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: demo
            port:
              number: 3000
  ingressClassName: nginx

SSL certificates

The examples above allow you to run multiple websites behind a reverse proxy on port 80, but in reality you want everything on port 443 and with SSL certificates. I’ll describe two scenarios: with a valid cert and self-signed certificate.Fortunately, nginx ingress listens on port 443 and does automatic redirect from http->https so you don’t have to worry about any changes.

Valid SSL certificate

You will need a valid SSL certificate for this. And you will need the private key, the certificate, the CA root certificate and the intermediate certificate. You can get these from the vendor where you purchased the certificate. In my case, I’ll have the private key saved as domain.key, the certificate as domain.pem, the root CA certificate as domain.ca and the intermediate certificate as domain.inter.
IMPORTANT: nginx expects the certificate chain in exact order: PEM -> INTERMEDIATE -> ROOT CA. If you have just the private key and the certificate (without inter and root CA), it might work, but you might get some errors when using curl, such as that the certificate authority cannot be recognized.
So, create a file called domain.full.pem or whatever that has all 3 certificates in that order, e.g. cat domain.pem domain.inter domain.ca > domain.full.pem.
If you received a bundle file from the certificate vendor, create a chain from two files: domain.pem plus the bundle file. The bundle file contains the intermediate and the root certificate.
Now that you have these files, you need to create a TLS secret in Kubernetes where you’ll store your certificate. You can do that with kubectl or with a YAML file.

kubectl create secret tls demo-tls --key domain.key --cert domain.full.pem

Keep the secret, ingress and the deployment in the same namespace.
If you use YAML, which most likely you’ll do if you use automation, you’ll have to encode the certificates.
This is the template YAML file. Name it as tls.yaml.

apiVersion: v1
kind: Secret
metadata:
  name: demo-tls
type: kubernetes.io/tls
data:
  tls.crt: |
    <insert_base64_string>
  tls.key: |
    <insert_base64_string>

Don’t change data keys tls.crt and tls.key. You have to use them, not something like key.crt or something.crt. Use exactly tls.crt and tls.key.
Encode the certificate with base64.

base64 -w0 domain.full.pem

Encode the key with base64.

base64 -w0 domain.key

You’ll get two long strings that start with LS0…Copy & paste these strings into YAML and make sure they are properly indented.

Now, you can create the secret.

kubectl apply -f tls.yaml

All yopu have to do now is to reference the TLS secret that we created. Compare the highlighted lines in this YAML with the previous one.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-demo
spec:
  tls:
  - hosts:
    - demo.iandreev.com
    secretName: demo-tls
  rules:
  - host: demo.iandreev.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: demo
            port:
              number: 3000
  ingressClassName: nginx

Change the hostname in line 11 and 14 to match your cert and you can deploy the app.

Self-signed certificates

First, we have to create a Certificate Authority (CA). I’ll use homelab.local as my domain.

openssl req -x509 -nodes -sha512 -days 365 -newkey rsa:4096 \
 -keyout rootCA.key -out rootCA.crt -subj "/C=US/CN=HomeLab Certificate Authority"

You’ll get two files, rootCA.key (the private key) and rootCA.crt.
Create a private key for the wildcard homelab.local domain and a certificate signing request file.
You’ll get two files, star.homelab.local.key (the private key for the domain) and star.homelab.local.csr (the certificate signing request).

openssl req -new -nodes -sha512 -days 365 -newkey rsa:4096 \
 -keyout star.homelab.local.key -out star.homelab.local.csr -subj "/C=US/ST=NJ/L=Trenton/O=HomeLab Org/CN=*.homelab.local"

Create this file where you’ll specify your domain name and eventually your IP, but the IP is not necessary.

cat << EOF > v3.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = *.homelab.local
IP.1 = 192.168.1.50
EOF

Sign the certificate signing request and create the certificate file. You’ll get one certificate file star.homelab.local.crt.

openssl x509 -req -sha512 -days 365 -in star.homelab.local.csr -CA rootCA.crt -CAkey rootCA.key \
 -CAcreateserial -out star.homelab.local.crt  -extfile v3.ext

Combine the certificate and the CA certificate in a single file.

cat star.homelab.local.crt rootCA.crt > star.homelab.local.chain.crt

Create this tls.yaml template file.

apiVersion: v1
kind: Secret
metadata:
  name: demo-tls
type: kubernetes.io/tls
data:
  tls.crt: |
    <insert_base64_string>
  tls.key: |
    <insert_base64_string>

Encode both the domain certificate chain file and the domain private key file.

base64 -w0 star.homelab.local.chain.crt
base64 -w0 star.homelab.local.key

You’ll get two very long single lines. Add these two lines to the tls.yaml template file but make sure they are properly indented (see the screenshot in the previous section).
Now, you can create the secret.

kubectl apply -f tls.yaml

You can also use kubectl to create the secret, see the previous section.
Save this file as demo-local.yaml and deploy it using kubectl apply -f demo-local.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
spec:
  replicas: 6
  selector:
    matchLabels:
      run: demo
  template:
    metadata:
      labels:
        run: demo
    spec:
      containers:
      - name: demo
        image: klimenta/getmyip
        ports:
        - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: demo
spec:
  type: ClusterIP
  ports:
  - port: 3000
    targetPort: 3000
  selector:
    run: demo
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-demo
spec:
  tls:
  - hosts:
    - demo.homelab.local
    secretName: demo-tls
  rules:
  - host: demo.homelab.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: demo
            port:
              number: 3000
  ingressClassName: nginx

You’ll be able to go to your browser and check demo.homelab.local and see the page after the SSL warning.
But if you double-click the rootCA.crt file on your local machine, you can import the certificate in the Trusted Root Certificates store and you won’t get that warning anymore.