Recently I was playing with Kubernetes and racked a substantial bill in AWS and Azure so I decided to bring the lab home. I needed a small but powerful PC that can run multiple test VMs and Kubernetes clusters. Kubernetes on a bare metal is different than EKS or AKS, especially when it comes to handling load balancers, ingress, IAM roles etc. Some of these things don’t exist on a bare metal machine.
I bought a NUC 13 small factor PC from Amazon and added Samsung NVMe 2TB SSD and Crucial 64GB RAM. After 2 months (these things are expensive!!!), I bought a Synology NAS with 2 x 8GB SATA disks. Next, is to upgrade the NAS with SSD that will act as cache, but I am not in a rush. Nevertheless…

Initial SSD, RAM setup

First thing you need to do is to CAREFULLY open the PC and make sure you don’t pull too hard and break the ribbon. Good thing is that the screws are not coming off and there is a spring mechanism so it’s super easy to open and close the bottom.

Add the NVMe SSD on the left side, but remove the screw first, bottom left. You’ll use it to tighten the SSD after.

Add the memory on the right side, you can add up to 64GB RAM hence 2 x 32GB modules. If you boot the PC even with a bootable USB stick, you’ll get an error saying “A bootable device has not been detected”.

Hit CTRL-ALT-DEL and then F2 to enter BIOS. From there, go to Boot menu, then Boot Priority.

Check Boot USB Devices First.

If you boot from ESXi 8 ISO now, most likely you’ll get an Fatal CPU mismatch on feature PSOD error. So, while still in the BIOS, go to Power, Performance and Cooling tab. Expand the first item External Ambient Temperature Tolerance and change the temperature setting to User Defined.

Now, you can enter the Performance menu which was grayed if the setting is not User Defined for the External Ambient Temperature Tolerance.

Once there, go to Processor.

Once there, change the number of Active Efficient Cores to 0.
In addition, when you install ESXi 8 and when you access the GUI, you’ll see a nagging warning that TPM cannot be contacted or something similar. No matter how many times you acknowledge this warning it keeps coming back. So, while you in the BIOS, go to Security tab, then Security Features and uncheck Intel Platform Trust Technology.
Save the settings with F10.

Install ESXi 8

At this point, reboot and boot from ESXi ISO USB. Check this guy for more info (link and link).
I have a VMUG Advantage subscription so I can freely download all VMWare software from here. As long as you pay the yearly fee, you can have the licenses and software indefinitely.
Choose where to install the ESXi (Samsung SSD).

Choose a root password.

Remove the USB stick and reboot when the installation is over and you’ll see this screen after.

This is my DHCP assigned address, so just change it to a static IP providing DNS and the default gateway. Then, change your FQDN as well. In my case I named my box as esxi1.homelab.local.

At this point you have a functioning ESXi server. Open up a browser, go to the ESXi IP and log with root and the password you specified.
If you have a license, add it now by going to Manage (upper left corner), then Licensing tab and click Assign license or skip this part and use the evaluation mode.

Optional: Click on Storage then datastore1, Actions from the menu and rename the datastore. In my case I’ll use ESX1-LOCAL.

Install vCenter 8.0 appliance

Now, we can install vCenter. Mount the vCenter ISO image, go to vcsa-ui-installer folder and choose the OS (win32, mac, lin64). Run the installer and click Install icon.

Click Next and Accept the legal terms. Then enter the IP address or FQDN if you have your DNS configured for the ESXi server you just installed. Specify the root password too.

Name the vCenter and specify the vCenter root password. This is not the password that you will use to log to vCenter. This is a password for the appliance itself. You’ll need this only if you have to troubleshoot the appliance or enable/disable SSH on the appliance.

Choose deployment size, for me the smallest size is acceptable. After all, I plan to use this primarily for Kubernetes.

Choose the datastore.

Configure network settings, IP, gateway, DNS, FQDN…

Confirm and click Finish. After some time, you’ll get a prompt to configure vCenter. I chose to use NTP and sync with some public NTP servers. Use whatever is closest to you.

I’ll create a new SSO domain and I’ll use the defaults. I just have to provide a password.

Once vCenter is deployed go to the FQDN or the IP from a browser. Your username is no longer root, it’s administrator@vspehere.local and your password is the one you specified when you installed vCenter SSO in the previous step. Enter the license by going to Administration | Licenses from the left.

Once you enter the license from the Licenses tab, go to Assets tab and click Assign License.

Go to Inventory from the hamburger menu on the left, right-click the vCenter IP or FQDN and choose New Datacenter.

Then right-click on the newly created datacenter and choose New Cluster.
Same thing again, right-click the cluster and choose Add Hosts.

Specify the IP or FQDN and the root password when you installed ESXi initially.

Optional: iSCSI

Click on the ESXi hosts, then Configure tab, click Add Software Adapter and then Add iSCSI adapter.

In my case, I have to use static discovery so click on that tab and then Add.

Enter the iSCSI IP and the target name.

Finally, click on Rescan storagein the menu above. If you need to provide your initiator, this is where you can find it under iSCSI Name (iqn.1998—_.

To learn how you can install Kubernetes on your NUC and how to use k8s Load Balancing see the links from my blog here and here.

In this post I’ll explain how to install MetalLB which is a load balancer for bare metal/VM Kubernetes servers. If you have your own home lab to play with Kubernetes and you are not using AKS, EKS or GKE, then this post is for you.
The prerequisite is to have a running Kubernetes cluster. In my earlier post, I’ve described how to install Kubernetes with CRI-O and Cilium as a CNI, so you can follow that post or feel free to have your own install. Mind that if you use other CNIs such as Calico or Weave, there are some things you have to check first and see if it applies to you. Look at this link.
We’ll install MetalLB now. You probably have your k8s master and the nodes ready.

Install MetalLB

Do this first on the master.

kubectl edit configmap -n kube-system kube-proxy

…and change strictARP to true.

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"
ipvs:
  strictARP: true

Deploy MetalLB.

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.11/config/manifests/metallb-native.yaml

You’ll see the pods running in metallb-system namespace.

kubectl get pods -n metallb-system
NAME                          READY   STATUS    RESTARTS   AGE
controller-7d56b4f464-gkfrq   1/1     Running   0          55s
speaker-l5954                 1/1     Running   0          55s
speaker-xrswp                 1/1     Running   0          55s
speaker-z8vdt                 1/1     Running   0          55s

Let’s configure our load balancer to give IPs in this 192.168.1.50-192.168.1.60 range. Change to your needs accordingly in line 8.
Save it as a lb-config.yaml file.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: first-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.1.50-192.168.1.60
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: example
  namespace: metallb-system

Deploy the config.

kubectl apply -f lb-config.yaml

Now, we can deploy a simple app and a load balancer. Save the file as demo.yaml.
This is an app that I made in Node.js that listens on port 3000 to display the IP of the pod that the load balancer hits.
You can use it for any type of a load balancer, not just MetalLB. It’s a super simple app that prints the IP of the pod where it’s running.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
spec:
  replicas: 6
  selector:
    matchLabels:
      run: demo
  template:
    metadata:
      labels:
        run: demo
    spec:
      containers:
      - name: demo
        image: klimenta/serverip
        ports:
        - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: loadbalancer
spec:
  ports:
    - port: 80
      targetPort: 3000
      protocol: TCP
  type: LoadBalancer
  selector:
    run: demo

Deploy the app.

kubectl apply -f demo.yaml

Check the load balancer service.

kubectl get svc
NAME           TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)        AGE
kubernetes     ClusterIP      10.96.0.1      <none>         443/TCP        33m
loadbalancer   LoadBalancer   10.106.36.58   192.168.1.50   80:30770/TCP   4m33s

…and if you go to that IP (192.168.1.50) in your browser, you’ll see the IP of the pods that the load balancer hits.

Install nginx ingress reverse proxy

The problem with the scenario above is that for every service you need to expose, you need a different IP from the IP pool that we’ve assigned to MetalLB (.50-.60).
A better solution is to install an nginx ingress that will act as a reverse proxy, but for this you’ll need a working DNS server as the services will share the same IP, but different FQDN.
So, in practice it looks like this.
Install nginx ingress first.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.2/deploy/static/provider/cloud/deploy.yaml
kubectl get pods -n ingress-nginx
kubectl get service ingress-nginx-controller -n=ingress-nginx

Don’t worry if the two nginx ports are not running and it says – “completed”.
You’ll see the external IP that’s assigned to nginx from MetalLB.
For some reason, nginx ingress won’t work without this line. Many people have this problem in a bare-metal environment and this is the workaround.

kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission

Now, let’s create an nginx web server. This has nothing to do with nginx ingress, it’s completely separate.
I am exposing the nginx deploymenty and creating a rule that will hit the deployment on port 80 with nginx.homelab.local as FQDN.
NOTE: If your browser can’t resolve this URL, it won’t work. So, make sure you have a DNS or /etc/hosts file or some type of name resolution.

kubectl create deployment nginx --image=nginx --port=80
kubectl expose deployment nginx
kubectl create ingress nginx --class=nginx --rule nginx.homelab.local/=nginx:80

Then, let’s install Apache server that runs on the same IP and same port 80.

kubectl create deployment httpd --image=httpd --port=80
kubectl expose deployment httpd
kubectl create ingress httpd --class=nginx --rule httpd.homelab.local/=httpd:80

If everything is OK, you’ll be able to access both web servers using their URLs.
If you want to put Grafana and/or KubeCost behind nginx ingress, read these short posts here and here.
If you want to use yaml for deployment, here is an example of that. It creates 6 replicas of a pod that prints the IP where the pod is running. The service runs on port 3000. Then it creates a ClusterIP on the same port and puts the ClusterIP behind the ingress. Change the hostname at line 39.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
spec:
  replicas: 6
  selector:
    matchLabels:
      run: demo
  template:
    metadata:
      labels:
        run: demo
    spec:
      containers:
      - name: demo
        image: klimenta/getmyip
        ports:
        - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: demo
spec:
  type: ClusterIP
  ports:
  - port: 3000
    targetPort: 3000
  selector:
    run: demo
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-demo
spec:
  rules:
  - host: demo.homelab.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: demo
            port:
              number: 3000
  ingressClassName: nginx

SSL certificates

The examples above allow you to run multiple websites behind a reverse proxy on port 80, but in reality you want everything on port 443 and with SSL certificates. I’ll describe two scenarios: with a valid cert and self-signed certificate.Fortunately, nginx ingress listens on port 443 and does automatic redirect from http->https so you don’t have to worry about any changes.

Valid SSL certificate

You will need a valid SSL certificate for this. And you will need the private key, the certificate, the CA root certificate and the intermediate certificate. You can get these from the vendor where you purchased the certificate. In my case, I’ll have the private key saved as domain.key, the certificate as domain.pem, the root CA certificate as domain.ca and the intermediate certificate as domain.inter.
IMPORTANT: nginx expects the certificate chain in exact order: PEM -> INTERMEDIATE -> ROOT CA. If you have just the private key and the certificate (without inter and root CA), it might work, but you might get some errors when using curl, such as that the certificate authority cannot be recognized.
So, create a file called domain.full.pem or whatever that has all 3 certificates in that order, e.g. cat domain.pem domain.inter domain.ca > domain.full.pem.
If you received a bundle file from the certificate vendor, create a chain from two files: domain.pem plus the bundle file. The bundle file contains the intermediate and the root certificate.
Now that you have these files, you need to create a TLS secret in Kubernetes where you’ll store your certificate. You can do that with kubectl or with a YAML file.

kubectl create secret tls demo-tls --key domain.key --cert domain.full.pem

Keep the secret, ingress and the deployment in the same namespace.
If you use YAML, which most likely you’ll do if you use automation, you’ll have to encode the certificates.
This is the template YAML file. Name it as tls.yaml.

apiVersion: v1
kind: Secret
metadata:
  name: demo-tls
type: kubernetes.io/tls
data:
  tls.crt: |
    <insert_base64_string>
  tls.key: |
    <insert_base64_string>

Don’t change data keys tls.crt and tls.key. You have to use them, not something like key.crt or something.crt. Use exactly tls.crt and tls.key.
Encode the certificate with base64.

base64 -w0 domain.full.pem

Encode the key with base64.

base64 -w0 domain.key

You’ll get two long strings that start with LS0…Copy & paste these strings into YAML and make sure they are properly indented.

Now, you can create the secret.

kubectl apply -f tls.yaml

All yopu have to do now is to reference the TLS secret that we created. Compare the highlighted lines in this YAML with the previous one.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-demo
spec:
  tls:
  - hosts:
    - demo.iandreev.com
    secretName: demo-tls
  rules:
  - host: demo.iandreev.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: demo
            port:
              number: 3000
  ingressClassName: nginx

Change the hostname in line 11 and 14 to match your cert and you can deploy the app.

Self-signed certificates

First, we have to create a Certificate Authority (CA). I’ll use homelab.local as my domain.

openssl req -x509 -nodes -sha512 -days 365 -newkey rsa:4096 \
 -keyout rootCA.key -out rootCA.crt -subj "/C=US/CN=HomeLab Certificate Authority"

You’ll get two files, rootCA.key (the private key) and rootCA.crt.
Create a private key for the wildcard homelab.local domain and a certificate signing request file.
You’ll get two files, star.homelab.local.key (the private key for the domain) and star.homelab.local.csr (the certificate signing request).

openssl req -new -nodes -sha512 -days 365 -newkey rsa:4096 \
 -keyout star.homelab.local.key -out star.homelab.local.csr -subj "/C=US/ST=NJ/L=Trenton/O=HomeLab Org/CN=*.homelab.local"

Create this file where you’ll specify your domain name and eventually your IP, but the IP is not necessary.

cat << EOF > v3.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = *.homelab.local
IP.1 = 192.168.1.50
EOF

Sign the certificate signing request and create the certificate file. You’ll get one certificate file star.homelab.local.crt.

openssl x509 -req -sha512 -days 365 -in star.homelab.local.csr -CA rootCA.crt -CAkey rootCA.key \
 -CAcreateserial -out star.homelab.local.crt  -extfile v3.ext

Combine the certificate and the CA certificate in a single file.

cat star.homelab.local.crt rootCA.crt > star.homelab.local.chain.crt

Create this tls.yaml template file.

apiVersion: v1
kind: Secret
metadata:
  name: demo-tls
type: kubernetes.io/tls
data:
  tls.crt: |
    <insert_base64_string>
  tls.key: |
    <insert_base64_string>

Encode both the domain certificate chain file and the domain private key file.

base64 -w0 star.homelab.local.chain.crt
base64 -w0 star.homelab.local.key

You’ll get two very long single lines. Add these two lines to the tls.yaml template file but make sure they are properly indented (see the screenshot in the previous section).
Now, you can create the secret.

kubectl apply -f tls.yaml

You can also use kubectl to create the secret, see the previous section.
Save this file as demo-local.yaml and deploy it using kubectl apply -f demo-local.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
spec:
  replicas: 6
  selector:
    matchLabels:
      run: demo
  template:
    metadata:
      labels:
        run: demo
    spec:
      containers:
      - name: demo
        image: klimenta/getmyip
        ports:
        - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: demo
spec:
  type: ClusterIP
  ports:
  - port: 3000
    targetPort: 3000
  selector:
    run: demo
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-demo
spec:
  tls:
  - hosts:
    - demo.homelab.local
    secretName: demo-tls
  rules:
  - host: demo.homelab.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: demo
            port:
              number: 3000
  ingressClassName: nginx

You’ll be able to go to your browser and check demo.homelab.local and see the page after the SSL warning.
But if you double-click the rootCA.crt file on your local machine, you can import the certificate in the Trusted Root Certificates store and you won’t get that warning anymore.

In this post, I’ll build a Kubernetes cluster with one master and two nodes using CRI-O and Cilium. I have DNS in my environment but you can use the hosts files if needed.
The OS will be Rocky Linux 9 running on ESXi 8.x but this should work on any other bare metal server too. The config for the nodes is:

– Master: 2 CPUs, 4GB RAM
– Workers: 2 CPUs, 8GB RAM.

First, we’ll have to disable swap and put SELinux in permissive mode.
Do this on all nodes (master + workers).

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

Now, we have to open some firewall ports for Kubernetes..
Do this on the master node only.

sudo firewall-cmd --permanent --add-port=6443/tcp
sudo firewall-cmd --permanent --add-port=2379-2380/tcp
sudo firewall-cmd --permanent --add-port=10250/tcp
sudo firewall-cmd --permanent --add-port=10259/tcp
sudo firewall-cmd --permanent --add-port=10257/tcp
sudo firewall-cmd --reload

Do this on the workers only.

sudo firewall-cmd --permanent --add-port=10250/tcp
sudo firewall-cmd --permanent --add-port=30000-32767/tcp   
sudo firewall-cmd --reload

In addition, we’ll have to open the firewall ports for Cilium.
Do this on all nodes.

sudo firewall-cmd --permanent --add-port=4240/tcp 
sudo firewall-cmd --permanent --add-port=8472/udp 
sudo firewall-cmd --reload

Now, we have to make some changes that are also required. Some necessary modules and IPtables change.
Do this on all nodes.

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# Apply sysctl params without reboot
sudo sysctl --system

We have to install CRI-O container runtime.
Do this on all nodes.

VERSION=1.22
sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/CentOS_8/devel:kubic:libcontainers:stable.repo
sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:${VERSION}.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:${VERSION}/CentOS_8/devel:kubic:libcontainers:stable:cri-o:${VERSION}.repo
sudo dnf -y install cri-o cri-tools
sudo systemctl enable --now crio
sudo systemctl status crio

Then, we have to install Kubernetes.
Do this on all nodes.

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF
sudo dnf install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
sudo systemctl enable --now kubelet

On the master node only, initialize the cluster. Replace master.homelab.com with your master IP if you don’t use DNS.

sudo kubeadm init --control-plane-endpoint master.homelab.local:6443

You’ll get an output that says how to join the nodes, something like kubeadm join master… + a token
On the master node, add these lines.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Join the nodes to the cluster.
Do this on workers only.

sudo kubeadm join master.homelab.local:6443 --token t2yir0.7n49rn6r57msy4j4 \
        --discovery-token-ca-cert-hash sha256:563360fcf60c49be91cdcf6486a4954c579a80c54503127ee0682ab8f86ec840

Check the nodes.
Do this on the master node.

kubectl get nodes
NAME                   STATUS   ROLES           AGE    VERSION
master.homelab.local   Ready    control-plane   116s   v1.28.2
node1.homelab.local    Ready    <none>          89s    v1.28.2
node2.homelab.local    Ready    <none>          85s    v1.28.2

…then check all the pods.
Do this on the master node.

kubectl get pods --all-namespaces
NAMESPACE     NAME                                           READY   STATUS    RESTARTS   AGE
kube-system   coredns-5dd5756b68-5ljgg                       1/1     Running   0          2m
kube-system   coredns-5dd5756b68-jp8tq                       1/1     Running   0          2m
kube-system   etcd-master.homelab.local                      1/1     Running   0          2m6s
kube-system   kube-apiserver-master.homelab.local            1/1     Running   0          2m6s
kube-system   kube-controller-manager-master.homelab.local   1/1     Running   0          2m6s
kube-system   kube-proxy-2z64r                               1/1     Running   0          2m
kube-system   kube-proxy-mrwjv                               1/1     Running   0          96s
kube-system   kube-proxy-nj5q4                               1/1     Running   0          100s
kube-system   kube-scheduler-master.homelab.local            1/1     Running   0          2m6s

Install Cilium on the master node only.

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
cilium install --version 1.14.2

Check the status.

cilium status --wait

You should see something like this.

    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium             Desired: 3, Ready: 3/3, Available: 3/3
Deployment             cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
Containers:            cilium             Running: 3
                       cilium-operator    Running: 1
Cluster Pods:          2/2 managed by Cilium
Helm chart version:    1.14.2
Image versions         cilium             quay.io/cilium/cilium:v1.14.2@sha256:6263f3a3d5d63b267b538298dbeb5ae87da3efacf09a2c620446c873ba807d35: 3
                       cilium-operator    quay.io/cilium/operator-generic:v1.14.2@sha256:52f70250dea22e506959439a7c4ea31b10fe8375db62f5c27ab746e3a2af866d: 1

Then you can run the connectivity test. Some of the egress test will fail because we don’t have a public IP on the cluster.

cilium connectivity test

You should be able to run your deployments now. If you need to access the pods using URL use something like NodePort or port-forwarding. It’s very cumbersome and inefficient, but you can use a load balancer with MetalLB. Read this post to see how to do that. It’s separated from this post, because it’s a separate topic than just k8s, container runtime and CNI.

Rocky Linux was finally release today (5/1/2021) and I’ve decided to try it in my lab. I’ve downloaded the minimal ISO (Rocky-8.3-x86_64-minimal) and installed it under my vCenter version 6.x.
I chose CentOS 8 as a base OS and after I mounted the ISO, I got an error saying ERROR Verification failed: (0x1A) Security Violation. If this is the case, shut down the guest, go to Settings | VM Options and if you expand Boot Options, change the Firmware from EFI to BIOS. This should do the trick.

What you’ll see next is pretty much the same as any other CentOS 7/8 installation.

NOTE: The current version is not considered for production.
Do the same as you do with any other CentOS install. Partition the hard drive, change the root password, configure the network and add a user.

Once the server is up, you’ll see the login prompt.

Check the kernel version and the release.

uname -a
Linux rocky.andreev.local 4.18.0-240.22.1.el8.x86_64 #1 SMP Mon Apr 12 04:29:16 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
cat /etc/redhat-release
Rocky Linux release 8.3

By default, both the firewall and SELinux are enabled.

systemctl status firewalld
sestatus

You can use dnf, yum or rpm to manage the packages. If you need to change the network settings, use nmtui.
Let’s install WordPress now. As root or using sudo, install Apache, PHP and MariaDB.

dnf install httpd php php-mysqlnd mariadb-server wget php-json

Start the Apache and MariaDB and enable them on both to start on reboot.

systemctl enable httpd --now
systemctl enable mariadb --now

Open port 80 on the firewall for the Apache.

firewall-cmd --permanent --zone=public --add-service=http --permanent
firewall-cmd --reload

Do the initian MariaDB configuration and specify the new root password for the SQL server.

mysql_secure_installation

Log as the SQL root user to MariaDB.

mysql -u root -p

Create the database and the WordPress user.

CREATE DATABASE wordpress;
GRANT ALL ON wordpress.* TO 'wpadmin'@'localhost' IDENTIFIED BY 'supersecret';
FLUSH PRIVILEGES;

Get the latest WordPress release under /tmp, unzip it and copy it where it belongs (root Apache data directory).

cd /tmp
wget https://www.wordpress.org/latest.zip
unzip latest.zip
mv wordpress/* /var/www/html/
chown -R apache:apache /var/www/html

Change the SELinux context for the root Apache data directory.

chcon -t httpd_sys_rw_content_t /var/www/html -R

At this point, you can access your WordPress install at http://your_IP_or_hostname.

If you have some on-prem workloads (either virtual or physical servers) and you want to migrate them to Azure, this post might be for you. I’ll describe how to migrate these servers in Azure using Azure Migrate and Azure Site Recovery. These tools (actual VMs) will have to be installed in your on-prem environment and they’ll do the heavy lifting for you. Before I begin, there are some pre-requisites. You’ll need a valid Azure subscription, supported VMware vCenter (v5.5 and above) and resources to install two VMs. On my ESXi host, using thin provisioned storage, I needed 8 CPUs, 32GB RAM and 20GB-80GB thin provisioned storage for Azure Migrate: Server Assessment and 8CPUs, 32GB RAM and 40GB disk for Azure Migrate: Server Migration tool. It will take much more space if you have a lots of VMs. I have about 15. You’ll also need a VNET(s) and subnet(s) where you’ll migrate the VMs.
I’ll configure that first. I’ll give you an example of how to migrate a two-tier Windows application (web server + MS SQL server) and a WordPress server running on two CentOS 7 servers.

Azure VNET/subnet

Log to your Azure subscription and create a resource group for the migration. I’ll name mine rgMigration.

Then create a new VNET and a subnet where you’ll migrate your VMs. You can use the same subnets as on-prem but only if you don’t have a ExpressRoute or VPN between on-prem and Azure. In my case, I have a VPN between my home-lab and Azure, so I have to choose a different subnet. I named my VNET vnetMigration and the subnet subServers.

Azure Migrate

Once completed, in the Azure portal in the middle upper search field, start typing migrate and select Azure Migrate.

You’ll be presented with different options to migrate data, SQL, servers etc. Choose this one that says migrate servers.

Click on Add tool(s).

You’ll be prompted to choose the subscription, the resource group, name the project and choose the region. Use whatever suits your needs and click Next.

There are several tools there, we’ll use the Microsoft Migrate: Server Assessment. Click on it to highlight and click Next.

For the migration tool, we’ll choose Azure Migrate: Server Migration. Select, click Next and finally click Add tool(s).

It takes about a minute to provision the tools. You’ll see two tiles on your screen, one for the Assessment tool and the other for the migration.

We’ll use the first tool to assess the on-prem servers. This tool will tell us if we have servers that can be actually moved to Azure cloud and it will recommend the size of those servers.

Server Assessment

Click on Discover first. You’ll see another screen where you can download an OVA file for your vCenter. Download the file and deploy the OVA file (v2.19.07.30). I won’t go into details how to do this, it’s super easy. Once you have your VM from the OVA file up and running, open the console for that VM, accept the EULA, change the admin password and log in as Administrator with the password that you just specified. You might also want to change the IP from DHCP to static before proceeding. In addition, disable the IE Enhanced Security Configuration from the Server Manager or you’ll get bombarded with annoying popups from IE.
On the desktop, there should be two files, one text file with the appliance version and a shortcut to Microsoft Azure Appliance Configuration Manager. Accept the license terms for the appliance.

Wait for the appliance to do the pre-flight checks and then install VMware vSphere Virtual Disk Development Kit.

Pretty much, you’ll have to download a ZIP file from VMware site, extract the ZIP and put it under C:\Program Files\VMWare\VMWare Virtual Disk Development Kit.

Click Verify and Install and Continue. Then, login to Azure portal from the appliance to register it.

Fill out the subscription, specify the migration project and enter a name for this appliance. Click Register and once successfully registered click Continue.

Specify the vCenter where your VMs reside, enter the username and password and click to Validate Connection.

Finally if everything is OK, click on Save and start discovery.
NOTE from the appliance: You will be able to review the discovered machines on the Azure portal and create as-on-premises assessments within 15 minutes. Collection of performance data will be initiated and it is recommended that you create performance-based assessments after 24-hours.
It’s important to leave the appliance running for some time so it can properly determine the usage and size the destination VM for you. If you already know what size your VM will be in Azure, you don’t have to worry about this.
Now, go back to Azure portal and find the Azure Migrate service. Click on Servers and then on Discovered servers under Assessment tools.

You’ll see a list of all servers in your vCenter and their specs. We’ll create a new assessment group for two Linux servers that I have (apache and mysql) and that are part of my local WordPress installation.

Then, create an assessment for that same group from the top menu.

Make sure you change the assessment properties. That’s where you specify the region. Just click on View all and change the settings there.

Now if you go back to the Azure Migrate, you’ll see that you have one group and one assessment.

Click on the Groups and then on the actual group. You’ll see something like this.

If you want to see what your server is doing in terms of connections to others servers over TCP/UDP, you have to install the agent. It’s not necessary, but you’ll see the dependencies. It’s useful when you migrate some servers as a group and then you realize that you need to migrate another server because there is a dependency.

Dependencies agent

Click on the link Requires agent installation and you’ll see this page. Pretty much you have to install two agents on each server and configure them. But, first you have to configure the OMS Workspace that gathers metadata from the VMs.

Click on Configure OMS workspace first. Type a name a region and click Configure at the bottom.

Linux

Download both files. You’ll get redirected to github for the first agent. Download omsagent-1.11.0-9.universal.x64.sh for a 64-bit Linux OS or whatever is the latest version. This is a bash script file with an embedded tar binary file. Transfer it to the Linux box, make sure it’s executable and run the script. Check this link for your version of Linux/Unix. Make sure you specify the Workspace ID and the Workspace Key, which you can get at the bottom of the page from the Dependencies screenshot (see the previous screenshot above).

wget https://github.com/microsoft/OMS-Agent-for-Linux/releases/download/OMSAgent_v1.11.0-9/omsagent-1.11.0-9.universal.x64.sh
chmod +x omsagent-1.11.0-9.universal.x64.sh
./omsagent-1.11.0-9.universal.x64.sh --install --enable-opsmgr -w <workspaceID> -s <workspaceKey>

You should see a message saying Shell bundle exiting with code 0.
For the second agent, download it from the Microsoft site, change the permission to execute and run.

wget http://aka.ms/dependencyagentlinux -O InstallDependencyAgent-Linux64.bin
chmod +x InstallDependencyAgent-Linux64.bin
./InstallDependencyAgent-Linux64.bin

Type YES when prompted and you should be good.

Windows

Download both files (MMASetup-AMD64.exe and InstallDependencyAgent-Windows.exe). Install on each server. For the MMA agent, check to connect the agent to Azure Log Analytics (OMS).

Enter the Workspace ID and the key and choose Azure Commericial.

Installing the dependency agent is even easier, just double-click the EXE and accept the agreement. From the Discovered servers list, click on the server where it says View dependencies and you’ll something like this.

Click on each block to see more details.

Server Migration

Once we have all the pre-requisites in place and we are all confident in the assessment based on the confidence rating, we can proceed with the migration.

Now, you can do the migration within Azure Migrate: Server Assessment tool. Just click on Discovered Servers under the tool and click Replicate from the menu.
Specify the type of the hypervisor and the on-prem appliance.

As you can see from the appliance suffix, the migration option of the assessment tool is in preview. You can proceed further. In the screenshot below, I’ll be migrating my Linux servers.

You just have to specify where you want your servers migrated.

Choose the size of the servers, the disk size and you can proceed.

This is probably the way to go, but because it’s in preview, I’ll explain the old way too.
Under the migration tool, click Discover.

Specify where do you want your VMs migrated. This is an important step. Click on Create resources.

Finish step 1. Download the OVA and install it in your vCenter. Same as with the previous OVA, log to the console, and change the IP to static. First, you’ll be prompted to name the appliance.

Then log in to Azure portal.

Configure the appliance using the Site Recovery icon on the desktop. It’s gonna ask you to log (again) to Azure and provide the subscription, resource group and the vault. Make sure you choose the vault that’s provided in the same dialog box where you downloaded the OVA. If you fail to provide the proper vault, you’ll have to deal with deleting hidden resources etc…

Next step is to agree to install MySQL server on the appliance and specify the vCenter.

You should also add the credentials for Windows and Linux servers. For example, my username has full access to all Windows VMs and root has access to all Linux servers.

Once completed, you can go to the Azure Portal and if you click on Replicate, you’ll see that our new appliance showed up.

If you are having problems finding out where to register the new appliance, go to the Replicate menu. Fill out the pre-requisites and you’ll get a warning that your appliance is not registered. Click Finalize registration from here.

As you can see, with this tool, migrate is grayed. First, we’ll have to replicate the servers.

Fill out the settings and proceed. Make sure you choose the right credentials. I’ll be replicating Linux instances, so I chose Linux-CREDS. These were defined when we configured the appliance.

Choose the VMs.

Choose the VNET, subnet etc…

Choose the rest of the requirements (disk etc…) and click on Replicate.
You can check the status from the tool. Click on Overview, then Replicating machines, then click the Other tab.

Once the replication is completed, you can do a Test migration or full migration. Click on the menu (three dots) on the right of each VM and do a Test migration first. If you a confident, you can do full migration as well.

For a test migration, you have to specify a VNET. When you do a test migration, the original VM stays up and the new VM in Azure has a -test suffix. Here I have my apache server test-migrated in Azure.

And I can access it over my VPN.

When you a done, you can clean up the test migration from the same menu above and you’ll get an option to delete the test VM. Finally, once everything is OK, you can do a full migration. Click Migrate and you’ll be all set.
NOTE: If you can’t RDP to your VM in Azure, it’s because RDP port was disabled. You can re-enable it from the console. Go to the VM menu and all the way down, click on Reset password. Do Reset configuration only and after a minute or so, you’ll be able to RDP.

And if you get this error, The remote computer requires Network Level Authentication…blah, blah, domain controller.

…then log as local administrator and check the DNS settings. Most likely the DNS settings were messed up when migrating. Here is an example of what I got from my VM in Azure.

  Default Gateway . . . . . . . . . : 192.168.10.1
  DHCP Server . . . . . . . . . . . : 168.63.129.16
  DHCPv6 IAID . . . . . . . . . . . : 352324922
  DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-25-0F-72-AF-00-50-56-B9-F7-0F
  DNS Servers . . . . . . . . . . . : 168.63.129.16

Azure assigned a DNS in Azure. Change it to your DNS and you should be OK.

Migrating a physical server

In order to migrate a physical server, you’ll have to follow these steps. In essence, you’ll have to log to the Azure Site recovery VM and go to this folder (C:\ProgramData\ASR\home\svsystems\pushinstallsvc\repository). Transfer the ASR file to each physical instance. While logged in the appliance, get your key.

C:\ProgramData\ASR\home\svsystems\bin\genpassphrase.exe -v

Don’t run the EXE on the VM yet. You’ll have to do it from a command prompt. Open up a command prompt (with admin rights) and execute these lines.

mkdir c:\temp
cd c:\<path_to_EXE>
ren Mi<hit_TAB_for_autocompletion> MobilityServiceInstaller.exe 
move MobilityServiceInstaller.exe c:\temp
cd c:\temp
MobilityServiceInstaller.exe /q /x:C:\Temp\
UnifiedAgent.exe /Role "MS" /Silent

The last command will open a new terminal window and install the agent. Wait to finish and you can exit. Then in the GUI go to c:\program files (x86)\Microsoft Azure Site Recovery\agent folder. Double-click the UnifiedAgentConfigurator.exe file and enter the IP of your Site Recovery appliance and the passphrase that you got from the command above.

After 10 seconds you should have everything ready.

Go back to the portal and in the Azure Migrate: Server Migration tool, click on Replicate. Under the Source settings tab, select that your machines are NOT virtualized. Choose the Site Recovery appliance and specify the credentials that will be used to access the physical server.

You don’t have an assessment for the physical server, so choose to specify the migrations settings manually. My physical server is called W2012R2.

Choose the subscription, VNET, subnet under Target settings. Choose automatic or specify the compute type of your instance under Compute tab and do the same for the disks under the next tab.

Click Replicate when ready. After a minute or so, you can check the status under Replicating servers.

…and same with the other VMs that were replicated, you can do a test migration and full migration. Once you power up the replicated server, make sure you uninstall any HW related software for the arrays, WMI management etc… And pretty much that’s it!

In my home lab, I run vCenter 6.7 with several ESXi hosts and since I built the lab I never did any backups.
Now that my VMs are not test VMs and they became fairly important, I’ve decided to take care of the backups.
Initially, I was thinking of using the Veeam Free Backup, but this program can’t do any schedules in the free version. Then I decided to play with OVF Tool, but I was having multiple issues with this tool and logging with a domain account. So, finally, I used PowerCLI and AWS CLI.
The idea is to create a OVA export of each VM and then upload it to an S3 bucket. The source is well commented, so you shouldn’t have any issues in the script’s logic.

######################################################################################################
# vCenter2AWS v0.1
# K.Andreev - 06/30/2018
# Description:  Creates an OVA backup of all or selected vCenter VMs 
#				and uploads them to a bucket in AWS (optional)
# Prerequisites: PowerCLI, AWS CLI (configured)
# NOTE: The VM being backed up will be shut down and powered up when the OVA file is complete
# 		If using PowerCLI 10.x execute this line once
# 			Set-PowerCLIConfiguration -InvalidCertificateAction Ignore
######################################################################################################
# Function for logging
Function LogWrite {
   Param ([string]$logstring)
   Add-content $LogFile -value ((Get-Date -format "[dd/MMM/yyyy:HH:mm:ss] ").ToString() + $logstring)
}
######################################################################################################
########## IMPORTANT PARAMETERS. DO NOT IGNORE. MODIFY TO SUIT YOUR NEEDS. ###########################
######################################################################################################
# vCenter hostname
$VCenter = "vcenter.domain.local"
# vCenter username
$Username = "myusername@domain.local"
# vCenter password
$Password = "SecretPassword"
# Destination for the OVA backup files, make sure you have trailing \ at the end of this string
$Destination = "D:\VMs\"
# Exclude the following VMs from backups
$Exclude = @("VMware vCenter Server Appliance")
# Name of the S3 bucket (make sure it exists, lower case only, starts with s3://)
$S3Bucket = "s3://vspherebackups"
# Do you want to upload to S3
$Upload = $true
# Specify if the OVA file contains a time stamp 
# e.g. vmname.YYYYMMDDHHmm.ova ($true) or vmname.ova ($fasle)
$TimeStamp = $true
# Do you want the OVA files to be deleted after upload to S3
$DeleteOVA = $true
# Log file path and name
$LogFile = "D:\VMs\vcenter2aws.log"
######################################################################################################
################### EXPORT VMs AS OVA FILES ##########################################################
######################################################################################################
# Connect to vCenter
$Server = Connect-VIServer $VCenter -User $Username -Password $Password
LogWrite ("Connected to " + $Server)
# Get all VMs
$VMs = Get-VM
# Loop through each of the VMs
Foreach ($VM in $VMs) {
# Get the name of each VMs	
$Name = $VM.Name
    # Skip the backup for the VMs that are in the array variable defined above
    if ($Exclude.Contains($Name)) {
        LogWrite ("Backup skipped for "+ $Name)
        Continue
    }
    # Stop the VM in order to take a backup
    Stop-VM -VM $Name -Confirm:$false
    LogWrite ("Stopping VM " + $Name)
    # Calculates the time stamp that will become part of the name
    if ($TimeStamp) {
        $TimeStampName = Get-Date -UFormat "%Y%m%d%H%M"
        $TimeStampName = "." + $TimeStampName
    } else
    # otherwise the time stamp will be blank
    {
        $TimeStampName = ""
    }
    LogWrite ("Backup starts for " + $Name)
    # Exports the VM as OVA file, overwrites if there is a same file at the destination
	Get-VM -Name $Name | Export-VApp -Destination ($Destination + $Name + $TimeStampName + ".ova") -Format Ova -Force:$true
    LogWrite ("Backup ends for " + $Name)
    # Once the export completes, power up the VM
	Start-VM -VM $Name -RunAsync
    LogWrite ("Starting VM " + $Name)
}
# Disconnect from vCenter
Disconnect-VIServer -Server $Server -Confirm:$false
LogWrite ("Disconnected from " + $Server)
######################################################################################################
################################ UPLOAD TO AWS #######################################################
######################################################################################################
# Exit if upload is not needed
if (!$Upload) {
    LogWrite "Upload to S3 skipped, exiting..."
    exit
}
LogWrite ("Upload to S3 bucket " + $S3Bucket + " starts")
# Use AWS CLI command to sync the destination folder and S3. 
aws s3 sync $Destination $S3Bucket
LogWrite ("Upload to S3 bucket " + $S3Bucket + " ends")
######################################################################################################
################################ DELETE THE OVA FILES ################################################
######################################################################################################
# Exit if delete is not needed
if (!$DeleteOVA) {
    LogWrite "Deleting OVA files skipped, exiting..."
    exit
}
Get-ChildItem $Destination -Include *.ova -Recurse | Foreach ($_) {Remove-Item $_.Fullname}
LogWrite ("Deleting OVA files from " + $Destination)
LogWrite "Script completed...Have a nice day!"

Recently, I wrote a blog post about migrating VMs from VMware vCenter to AWS using basic export OVA/OVF files. Amazon made a connector that ties your vCenter to AWS so the export is kind of simplified, but the configuration is not. While using the export as OVA/OVF requires you to change the NIC from vmxnet* to e1000 and changing the static IP of your server to DHCP, with AWS Connector you can leave everything as is. Also, the export as OVA/OVF requires your VM to be powered off, while with AWS Connector you can migrate your VMs while they are powered on (using snaps). I also found that AWS Connector is slower than plain export as OVA/OVF. In this post I’ll show you how to configure the AWS Connector.

AWS Connector is not just about VM migrations. It has some other useful functions. Check this web page for more information and this PDF for the install/user guide. So, head to this page first and scroll all the way down to the bottom.

Step 1, Download the AWS Connector. Step 2, click on Setup AWS Management Portal for vCenter. Log in with your AWS account. Click on Get Started Now.

I will use the first option and most likely you’ll do. Option 2 is if you have a SSO (Single Sign-On) to AWS.

Make sure you have some IAM users defined, otherwise you’ll receive this error. As a matter of fact, you’ll want a separate IAM user for this. This user acts as a proxy between AWS and your vCenter.

In my case, I’ve created a new IAM user called vCenter. Check Programmatic Access.

On the next screen, click on the Attach existing policies directly and in the Filter: Policy type, start typing AWSConnector. This is an AWS managed policy. Click on the check mark and proceed.

Make sure you download your credentials for this IAM user. They’ll be saved as credentials.csv file on your computer.

Go back to your first tab (or the initial URL) and hit refresh. Type the username vCenter and click on the check mark that you agree.

The next step tells you to add the admins for AWS Management Portal. Make sure that these accounts are admins in vCenter. In my case, my vCenter is joined to the domain and I am using my own domain account to manage vCenter. You can create a separate user for this, but make sure it’s a vCenter admin.

Now, this part is very important. I’ve spent hours troubleshooting this. It was right in front of my eyes, but I didn’t pay attention.

The easiest way is to open a command prompt from your desktop and type:

set | find "USERDOMAIN"

My domain is not andreev. It is actually ANDREEV, so I had to correct the entry.

The last entry will ask you to create an AMP-Connector Key. Just type some name.

At the end, you’ll have to download the configuration file. It’s a text file with some key text in it.
So, you have two files now, one with the credentials for the proxy user and one with the configuration key. After you are done with this, you’ll land up on this page. You can expand the settings and make sure everything looks OK. You can ignore the Configure Instance Export menu. That’s for S3 exports.

Now, you will need another proxy user, but this time on the vCenter side. It can be a local vCenter user or in my case, I prefer to have an AD account. So, I’ve created an AD account called awsconnector. No special rights needed and don’t add this account to vCenter. Just create a plain AD account as a Domain User.

At this point, you should be OK on the AWS side. Head back to your vCenter and deploy the OVA file that you’ve downloaded.

Once the appliance is provisioned, power it up. If you have a DHCP on your network and you are fine with the DHCP assigned IP on this VM, you don’t have to do anything. But, I prefer to have a static IP so open the console of this VM and log as ec2-user with ec2pass as a password. Type sudo setup.rb. Type ec2pass for password. You should see this menu. Choose option 2 to assign a static IP if you need. If you are OK with a DHCP, it’s fine. It’s up to you.

Once you have the AWS Connector on the network go to https://aws_connector_IP address. Make sure to use https. Use your vCenter admin to log in.

Click Trust when you get a certificate warning.

You’ll be prompted to change the web front-end password. There is no user name for this next time you log to your AWS Connector.

Once you log in, this will pop up.

Click Upload the configuration file, select the file that you’ve downloaded from AWS. It’s called AMP_Setup_Configuration.txt and click Next. Again, log with your vCenter admin account.

Click Trust.

Now, enter that AD account that you’ve just created.

Use the credentials from credentials.csv file to enter the username and password. The Trust Role ARN field should be automatically populated.

Click on I agree to install the vCenter SSL certificates… otherwise you won’t be able to proceed and click Register.

You should get this pop-up which means you are good.

Open the vCenter client and from the Plug-ins menu click Manage Plug-ins…The AWS Connector should be enabled.

In the web client it should look like this.

Now, it’s time to get some action. Pick a VM that you want to migrate to AWS. Did I say that you need RDP/SSH enabled?
Right-click on this VM and choose Migrate to EC2.

You’ll get this error. AWS Connector is not yet configured.

All right, so go to Home -> AWS Management Portal in vCenter and add the preferred region.

Click on Create an environment.

Select your VPC and choose all subnets.

Go back to your VM and try to migrate it to EC2.

Oh, snap!

Go to vCenter roles, and follow up the instructions from above.

Then again, go back to your VM, right-click and choose Migrate to EC2. You’ll get this warning. It pretty much tells you that while migrating the VM will be slow.

Once the migration starts you’ll see this.

Do that. Go to Home -> AWS Management Portal -> Dashboard in vCenter. You’ll see the progress. If you scroll to the right you can actually see how much is uploaded. AWS is uploading the snap, so my VM has a 30GB disk (thick provisioned), but it was a plain vanilla OS with nothing installed except a text file on the desktop. So the actual size for upload is 4GB.


Once your migration completes, you can find your instance/VM under IMAGES | AMIs in AWS console.


Right-click on the AMI, choose Launch, follow the prompts and you should be all set. You’ll have to specify a key, but you can use your local admin credentials to log in.

I needed to move some VMs from my home lab (vCenter 5.5) to my Amazon AWS cloud account. This is what I did.
First, you have to prepare the VM. If you have any vmxnet* network card, you have to delete it from the properties of the VM in vCenter. Second, make sure that you have RDP or SSH enabled on the VM and third, uninstall the VMware tools. Shut down the VM, add an E1000 NIC and power it up again. Make sure that the TCP/IP and DNS settings for the new NIC are set for DHCP. Verify that all these requirements apply and then shut down the VM again.
In vCenter, select the VM for export, click on File | Export and then Export OVF template. Choose Single file (OVA).

Once the export is completed, you have to upload the OVA file to a S3 bucket. I created a new bucket for this, called uploadvmstuff with the default settings. You’ll have to pick a unique name for the bucket, but you probably know that. Upload your OVA file to S3. It might take a while depending how big the file is.

I am using my admin account in AWS to import the VM, but I still have to assign some policies. So, when the upload is complete, go to the command prompt or PowerShell on your desktop. Make sure you have AWS CLI installed. You can check my other blog post on how to do this. Very simple.

Create these three files in Notepad and save them on the desktop. The first file is called trust-policy.json. See this link for more info.

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": { "Service": "vmie.amazonaws.com" },
         "Action": "sts:AssumeRole",
         "Condition": {
            "StringEquals":{
               "sts:Externalid": "vmimport"
            }
         }
      }
   ]
}

Execute this command from the command prompt. Make sure that you are in the same folder in the command prompt where you saved your files, e.g. cd c:\users\whatever\desktop.

aws iam create-role --role-name vmimport --assume-role-policy-document file://trust-policy.json

Create the 2nd file called role-policy.json so it looks like this. Replace the two occurrences of disk-image-file-bucket with your bucket name, in my case it was uploadvmstuff.

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Action": [
            "s3:ListBucket",
            "s3:GetBucketLocation"
         ],
         "Resource": [
            "arn:aws:s3:::disk-image-file-bucket"
         ]
      },
      {
         "Effect": "Allow",
         "Action": [
            "s3:GetObject"
         ],
         "Resource": [
            "arn:aws:s3:::disk-image-file-bucket/*"
         ]
      },
      {
         "Effect": "Allow",
         "Action":[
            "ec2:ModifySnapshotAttribute",
            "ec2:CopySnapshot",
            "ec2:RegisterImage",
            "ec2:Describe*"
         ],
         "Resource": "*"
      }
   ]
}

Execute this command from the command prompt.

aws iam put-role-policy --role-name vmimport --policy-name vmimport --policy-document file://role-policy.json

The third file is called containers.json and it looks like this. Change the lines so it matches your settings. For Description use whatever you want, change the S3Bucket line and the S3Key. The S3Key is the name of the ova file that you’ve exported from vCenter.

[
  {
    "Description": "Windows 2008R2 Import",
    "Format": "ova",
    "UserBucket": {
        "S3Bucket": "uploadvmstuff",
        "S3Key": "nvm.ova"
    }
}]

Now, you are ready to import the file from S3 to EC2. Execute this line from the command prompt.

aws ec2 import-image --description "Windows 2008R2 VMDKs" --license-type BYOL --disk-containers file://containers.json

You should get your prompt back after 2-3 seconds. But, this means that the import started, not that it finished. To check the status, execute this line.

aws ec2 describe-import-image-tasks

It takes some time, for my OVA file that was created from a 40GB disk and 3.7GB OVA file it took about 20-25 mins. You will see the Status change to completed once everything is OK. Make sure that your AWS CLI is configured to do the output in JSON so it’s easy to see.

Once completed, you will see your import in the AMI section of EC2. Go to EC2 section, click Launch New Instance and choose the AMI that you just created. Follow the standard prompts for the type of the instance, security groups, volumes, etc… and once your instance is launched you can RDP to it. You will use the same admin password as before to RDP.

Recently, we moved to CommVault for backing up VMs. We use a CommVault solution that interacts with NetApp for taking snaps. Once the backup runs, it time-stamps the VM with the last backup taken. If you don’t need an e-mail in HTML format with the report, remove the lines from 103 to 113. Finally, remove the comment line # from line 57 if you want the report on the screen.

Based on this info, I made a script that loops through all clusters and VMs and dumps the report in a mail. You can remove lines 5 to 14 if you don’t want the script scheduled.

# PowerCLI script that lists all VMs under clusters that start with USCORPROD*
# and prints the Last Backup attribute from CommVault
# If the difference between the run time of the script and the last backup is
# more than $HoursDifference, write the output with red color
Add-PSSnapin VMware.VimAutomation.Core
Add-PSSnapin VMware.VimAutomation.Vds
if(get-item HKLM:\SOFTWARE\Microsoft\PowerShell\1\PowerShellSnapIns\VMware.VimAutomation.Core){
	. ((get-item HKLM:\SOFTWARE\Microsoft\PowerShell\1\PowerShellSnapIns\VMware.VimAutomation.Core).GetValue("ApplicationBase")+"\Scripts\Initialize-PowerCLIEnvironment.ps1")
}
else
{
	write-warning "PowerCLI Path not found in registry, please set path to Initialize-PowerCLIEnvironment.ps1 manually. Is PowerCli aleady installed?"
	. "D:\Programs (x86)\VMware\Infrastructure\vSphere PowerCLI\Scripts\Initialize-PowerCLIEnvironment.ps1"
}

# Specify the vCenter server
$server = "vcenter"
# Specify the difference in hours to compare
$HoursDifferenceCompare = 24
# Cluster filter
$ClusterFilter = "*"
# Connect to vCenter
Connect-VIServer -Server $server -user domain\username -password vcpassword
# Initalize the main HTML content
$midHTML=""
# Get each cluster that starts with $ClusterFilter and sort them. 
# Make sure only unique values are returned
ForEach ($Cluster in Get-Cluster $ClusterFilter | Sort -unique)
{
	# Get all the VMs per cluster
	$VMs = Get-Cluster $Cluster | Get-VM 
	# Loop each VM
	ForEach ($VM in $VMs)
	{
		# Get the last backup attribute
		$LastBackup = Get-VM $VM | Get-Annotation -CustomAttribute "Last Backup"
		# Parse the output, trinm the string
		$LastBackupTimeStamp = $LastBackup -replace "Last Backup:"
		# Get the date/time when the script started
		$StartDate=(Get-Date)
		# The end date is the Last Backup time for the VM
		# If the VM was never backed up, assume 01/01/70
		if ($LastBackupTimeStamp -eq '') {
			$EndDate = [datetime]'January 1, 1970'
			$LastBackupTimeStamp = "NEVER"}
		else {
			$EndDate = [datetime]$LastBackupTimeStamp 
			}
		# Calculate the difference
		$TimeSpan = New-TimeSpan -Start $StartDate -End $EndDate
		# It's a negative value, so get the absolute
		$DaysDifference = [math]::abs($TimeSpan.Days)
		$HoursDifference = [math]::abs($TimeSpan.Hours) + ($DaysDifference * 24)
		# Print the output
		if ($HoursDifference -gt $HoursDifferenceCompare) {
			# Remove for debugging if needed
			#Write-Host $Cluster " " $VM " " $LastBackupTimeStamp
			# Build the HTML
			$midHTML = $midHTML + '<font color="red">'
			$midHTML = $midHTML + "<p>" + $Cluster + " " + $VM.Name.PadRight(30,"-") + " " + $LastBackupTimeStamp + "</p>"
			$midHTML = $midHTML + "</font>"
		} else {
			$midHTML = $midHTML + "<p>" + $Cluster + " " + $VM.Name.PadRight(30,"-") + " " + $LastBackupTimeStamp + "</p>"
		}
	}	
}

# Top HTML header
$topHTML = @"
<!DOCTYPE html>
<html>
<body>
<font face="Courier New, Courier, monospace">
<style>
p
{
  margin:0;
  padding:0;
  font-size:12px;
    line-height:12px;
}
div
{
  margin:0;
  padding:0;
}
</style>
<div>
<p>&#160;&#160;&#160;CLUSTER NAME&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;SERVER NAME&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;DATE OF LAST BACKUP</p>
<p>-----------------------------------------------------------------------</p>
"@

# Bottom HTML footer
$bottomHTML = @"
<p>-----------------------------------------------------------------------</p>
<p>Script by K.Andreev - 20160630</p>
</div>
</font>
</body>
</html>
"@

# e-mail setup
$EmailFrom ="sender@domain.com"
$EmailTo ="receiver@domain.com"
$msg = New-Object System.Net.Mail.MailMessage ($EmailFrom,$EmailTo)
$msg.subject = "CommVault Report " + $StartDate
$msg.IsBodyHtml = $true;
$msg.Body =$topHTML + $midHTML + $bottomHTML
$smtpClient = New-Object System.Net.Mail.SmtpClient
$smtpClient.Host = "smtp.server.com"
$smtpClient.EnableSsl = $false
$smtpClient.Send($msg)

# Disconnect from vCenter
Disconnect-VIServer -Server $Server -Confirm:$false

I manage a terminal server that Service Desk is using to access some applications using RDP client. Recently, they needed audio enabled. These steps will help accomplish that goal .

First, log to the server and make sure that Windows Audio service is set to start as Automatic instead of Manual. Then, start the service.

Go to Administrative Tools, Remote Desktop and start the Remote Desktop Session Host Configuration. Select the RDP connection, right-click and choose Properties.

In the dialog box, select Client Settings and uncheck Audio and Video Playback.

You’ll get a message that you have to log off and log back on for these changes to take effect.

Also, make sure that you don’t inherit any domain policies that prevent the RDP audio settings. Click Start, Run and type:

gpedit.msc

Navigate to Computer Configuration | Administrative Templates | Windows Components | Remote Desktop Services | Remote Desktop Session Host | Device and Resource Configuration.

If it says Not Configured or Enabled, it’s fine. If it says Disabled, you’ll have to modify the domain GPO.

Finally, on the client computers, make sure that the RDP client is set to play the audio on the remote machine and not on the server.

Sources:

http://technet.microsoft.com/en-us/library/cc770631.aspx
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004839