AWS – Blog of Kliment Andreev – A place so I won't forget things

AWS: Access an S3 bucket using gateway and interface endpoints (PrivateLink)

Kliment Andreev — Tue, 18 Jun 2024 21:19:14 +0000

If you have a use case where you need to transfer a lot of data back-and-forth between various resources and an S3 bucket, you will definitely benefit if you use gateway or interface endpoints for S3. The monthly bill for regional and zonal transfer will be much less and on top of that it’s much more secure.
From the AWS documentation, “Amazon S3 supports both gateway endpoints and interface endpoints. With a gateway endpoint, you can access Amazon S3 from your VPC, without requiring an internet gateway or NAT device for your VPC, and with no additional cost. However, gateway endpoints do not allow access from on-premises networks, from peered VPCs in other AWS Regions, or through a transit gateway. For those scenarios, you must use an interface endpoint, which is available for an additional cost.”
Keep this table handy as it explains the differences between gateway and interface S3 endpoint.

Now that we know the use case and the differences, we’ll see three different scenarios. For this, you’ll need access to two different accounts and two different regions.

An S3 bucket in account A and an ec2 instance in the same account A and the same region A
An S3 bucket in account A and an ec2 instance in a different account B, but the same region A
An S3 bucket in account A and an ec2 instance in a different account B and a different region B

The diagram looks like this.

An S3 bucket in account A, ec2 in account A, both in the same region

Let’s create an S3 bucket first. We’ll use AWS CLI with different profiles.

BUCKET_NAME="myuniquenameforthebucket"
REGION_A="us-east-2"
aws s3api create-bucket \
    --bucket $BUCKET_NAME \
    --region $REGION_A \
    --create-bucket-configuration LocationConstraint=$REGION_A

Let’s copy a file there.

dd if=/dev/zero of=somefile bs=1024 count=1
aws s3 cp somefile "s3://$BUCKET_NAME/somefile"

Using the same profile, create a VPC with one public subnet with the same CIDR as the VPC.

CIDR_A="192.168.10.0/24"
VPCA_ID=$(aws ec2 create-vpc \
    --cidr-block $CIDR_A \
    --region $REGION_A \
    --tag-specification 'ResourceType=vpc,Tags=[{Key=Name,Value=VPC-A}]' \
    --output text --query 'Vpc.VpcId')
echo $VPCA_ID

Enable DNS and hostnames resolution, needed for SSM. Two separate lines needed.

aws ec2 modify-vpc-attribute --enable-dns-hostnames "{\"Value\":true}"  --vpc-id $VPCA_ID
aws ec2 modify-vpc-attribute --enable-dns-support "{\"Value\":true}"  --vpc-id $VPCA_ID

Create a default network ACL.

aws ec2 create-network-acl --vpc-id $VPCA_ID

Create an Internet Gateway and attach it to the VPC.

IGWA_ID=$(aws ec2 create-internet-gateway \
    --region $REGION_A \
    --tag-specifications 'ResourceType=internet-gateway,Tags=[{Key=Name,Value=igw-A}]' \
    --output text --query 'InternetGateway.InternetGatewayId')
echo $IGWA_ID
aws ec2 attach-internet-gateway \
    --internet-gateway-id $IGWA_ID \
    --vpc-id $VPCA_ID

Let’s create a subnet.

SUBA_ID=$(aws ec2 create-subnet \
    --vpc-id $VPCA_ID \
    --cidr-block 192.168.10.0/24 \
    --region $REGION_A \
    --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=subPubA}]' \
    --output text --query 'Subnet.SubnetId')
echo $SUBA_ID

Create a route table.

RTA_ID=$(aws ec2 create-route-table \
     --vpc-id $VPCA_ID \
     --output text --query 'RouteTable.RouteTableId')
echo $RTA_ID
aws ec2 create-route --route-table-id $RTA_ID --destination-cidr-block 0.0.0.0/0 --gateway-id $IGWA_ID

Associate this route table to the public subnet.

aws ec2 associate-route-table --route-table-id $RTA_ID --subnet-id $SUBA_ID

Now, we have to create an EC2 instance where we can test the S3 access. We won’t be using keys and SSH to access, we’ll use SSM. For that we need the instance to have access to the Internet which means we need a security group that allows access to Internet.

SGA_ID=$(aws ec2 create-security-group \
    --group-name sgAllowICMPandOutboundAccess --description "Allows ICMP and outbound access" \
    --vpc-id $VPCA_ID \
    --output text --query 'GroupId')
echo $SGA_ID

Allow unrestricted egress. By default this rule is there, so don’t execute the code below.

#aws ec2 authorize-security-group-egress \
    --group-id $SGA_ID \
    --protocol all \
    --cidr "0.0.0.0/0"

Do this one, so we can ping instances from each other when we do VPC peering.

aws ec2 authorize-security-group-ingress \
  --group-id $SGA_ID \
  --ip-permissions IpProtocol=icmp,FromPort=-1,ToPort=-1,IpRanges="[{CidrIp=192.168.10.0/24},{CidrIp=192.168.20.0/24},{CidrIp=192.168.30.0/24}]"

Let’s create a role so we can access the EC2 over SSM. First we need a policy that the role can assume.

cat << 'EOF' >> trust-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

Then create the role and attach the AWS Managed policy for SSM (AmazonSSMManagerInstanceCore).

aws iam create-role --role-name rolSSM --assume-role-policy-document file://trust-policy.json
aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore --role-name rolSSM

Create instance profile.

aws iam create-instance-profile --instance-profile-name ipEC2

Add the role.

aws iam add-role-to-instance-profile --role-name rolSSM --instance-profile-name ipEC2

Create an instance.

AMI_ID="ami-033fabdd332044f06"
INS_TYPE="t3.micro"
aws ec2 run-instances \
    --image-id $AMI_ID \
    --instance-type $INS_TYPE \
    --security-group-ids $SGA_ID \
    --subnet-id $SUBA_ID \
    --iam-instance-profile Name="ipEC2" \
    --associate-public-ip-address  \
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=ec2-A}]'

We’ll need an IAM user that we’ll use to access the S3 bucket.

IAM_USER="usrMyBucket"
aws iam create-user --user-name $IAM_USER

We’ll also need a policy that gives access to that specific bucket only.

cat >> bucket-policy.json << EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::$BUCKET_NAME",
                "arn:aws:s3:::$BUCKET_NAME/*"
            ]
        },
        {
            "Effect": "Deny",
            "NotAction": "s3:*",
            "NotResource": [
                "arn:aws:s3:::$BUCKET_NAME",
                "arn:aws:s3:::$BUCKET_NAME/*"
            ]
        }
    ]
}
EOF

Create the policy.

POL_ARN=$(aws iam create-policy \
    --policy-name polMyBucketAccess \
    --policy-document file://bucket-policy.json \
    --output text --query 'Policy.Arn')
echo $POL_ARN

Attach the policy to the user.

aws iam attach-user-policy --policy-arn $POL_ARN --user-name $IAM_USER

Create access keys.

aws iam create-access-key --user-name $IAM_USER --output table --query ['AccessKey.AccessKeyId','AccessKey.SecretAccessKey']

Copy these two values.
Now, go to AWS console, select the EC2 instance and click Connect. Use the 2nd tab Session Manager and click Connect.
Type aws configure and paste the copied values for the IAM user you just created.

If you try to do aws s3 ls, you’ll get access denied, but if you try aws s3 ls s3://myuniquenameforthebucket, you’ll see that it works.

Now, this type of access goes over the Internet and you’ll be charged for the data transfers. In order to avoid that, we’ll create an S3 gateway endpoint.
Go to VPC menu and from the left side choose Endpoints.
Create the endpoint, name the endpoint, choose AWS services, filter by Type = Gateway and select the S3 service name. Then select the VPC where you created your EC2 and the route table that’s associated with the subnet where EC2 resides. For policy choose Full Access. This means, the S3 gateway endpoint will have access to all S3 buckets. Finally, click to create the endpoint.

If you check your route table, you’ll see that there is another entry there. Wait 10 seconds or less if you don’t see it.

If you go back to the EC2 instance and do the aws s3 ls s3://myuniquenameforthebucket command again, you’ll see that nothing changed. So, how do you know that we did the right thing? It’s simple. Click on the endpoint that you’ve created, click on the Policy tab and then click the Edit Policy button.

Change line 5 from Allow to Deny and click Save.
Go back to the EC2 instance and do the aws s3 ls s3://myuniquenameforthebucket again. You’ll get access denied. There you go…Traffic goes over the endpoint. Revert the changes back from Deny to Allow. Let’s move to the 2nd use case.

An S3 bucket in account A, an ec2 instance in a different account B, but both in the same region A

NOTE:Make sure you switch your AWS CLI to a different profile, because we’ll be building the EC2 instance in another account. The S3 bucket stays the same.
Let’s build the 2nd VPC and EC2.

REGION_A="us-east-2"
CIDR_B="192.168.20.0/24"
VPCB_ID=$(aws ec2 create-vpc \
    --cidr-block $CIDR_B \
    --region $REGION_A \
    --tag-specification 'ResourceType=vpc,Tags=[{Key=Name,Value=VPC-B}]' \
    --output text --query 'Vpc.VpcId')
echo $VPCB_ID

Enable DNS and hostnames resolution, needed for SSM. Two separate lines needed.

aws ec2 modify-vpc-attribute --enable-dns-hostnames "{\"Value\":true}"  --vpc-id $VPCB_ID
aws ec2 modify-vpc-attribute --enable-dns-support "{\"Value\":true}"  --vpc-id $VPCB_ID

Create a default network ACL.

aws ec2 create-network-acl --vpc-id $VPCB_ID

Create an Internet Gateway and attach it to the VPC.

IGWB_ID=$(aws ec2 create-internet-gateway \
    --region $REGION_A \
    --tag-specifications 'ResourceType=internet-gateway,Tags=[{Key=Name,Value=igw-B}]' \
    --output text --query 'InternetGateway.InternetGatewayId')
echo $IGWB_ID
aws ec2 attach-internet-gateway \
    --internet-gateway-id $IGWB_ID \
    --vpc-id $VPCB_ID

Let’s create a subnet.

SUBB_ID=$(aws ec2 create-subnet \
    --vpc-id $VPCB_ID \
    --cidr-block 192.168.20.0/24 \
    --region $REGION_A \
    --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=subPubB}]' \
    --output text --query 'Subnet.SubnetId')
echo $SUBB_ID

Create a route table.

RTB_ID=$(aws ec2 create-route-table \
     --vpc-id $VPCB_ID \
     --output text --query 'RouteTable.RouteTableId')
echo $RTB_ID
aws ec2 create-route --route-table-id $RTB_ID --destination-cidr-block 0.0.0.0/0 --gateway-id $IGWB_ID

Associate this route table to the public subnet.

aws ec2 associate-route-table --route-table-id $RTB_ID --subnet-id $SUBB_ID

SGB_ID=$(aws ec2 create-security-group \
    --group-name sgAllowICMPandOutboundAccess --description "Allows ICMP and outbound access" \
    --vpc-id $VPCB_ID \
    --output text --query 'GroupId')
echo $SGB_ID

Allow unrestricted egress. By default this rule is there, so don’t execute the code below.

#aws ec2 authorize-security-group-egress \
    --group-id $SGB_ID \
    --protocol all \
    --cidr "0.0.0.0/0"

Do this one, so we can ping instances from each other when we do VPC peering.

aws ec2 authorize-security-group-ingress \
  --group-id $SGB_ID \
  --ip-permissions IpProtocol=icmp,FromPort=-1,ToPort=-1,IpRanges="[{CidrIp=192.168.10.0/24},{CidrIp=192.168.20.0/24},{CidrIp=192.168.30.0/24}]"

We can reuse the same policy to create a role that will allow us to use SSM.

cat trust-policy.json

Then create the role and attach the AWS Managed policy for SSM (AmazonSSMManagerInstanceCore).

aws iam create-role --role-name rolSSM --assume-role-policy-document file://trust-policy.json
aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore --role-name rolSSM

Create instance profile. We’ll use the same names, because they are different accounts.

aws iam create-instance-profile --instance-profile-name ipEC2

Add the role.

aws iam add-role-to-instance-profile --role-name rolSSM --instance-profile-name ipEC2

Create an instance. The AMI_ID is the same because it’s the same region.

AMI_ID="ami-033fabdd332044f06"
INS_TYPE="t3.micro"
aws ec2 run-instances \
    --image-id $AMI_ID \
    --instance-type $INS_TYPE \
    --security-group-ids $SGB_ID \
    --subnet-id $SUBB_ID \
    --iam-instance-profile Name="ipEC2" \
    --associate-public-ip-address  \
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=ec2-B}]'

Wait for a couple of minutes and then connect from SSM, same as you did with the first instance. Configure AWS again using the same set of credentials that you did before and then try to list the content of the bucket. You should be able to see the file.

BUCKET_NAME="myuniquenameforthebucket"
aws s3 ls s3://$BUCKET_NAME

Again, the traffic goes over the Internet. So we need another gateway S3 endpoint. You create the gateway S3 endpoint where the client is located, not the destination.
Let’s create another S3 gateway endpoint as described above.

Check the route table and you’ll see the route created. At this point, you are accessing the bucket over the endpoint. Do the same test. Change the policy to Deny for the endpoint and you’ll get access denied.

An S3 bucket in account A, an ec2 instance in a different account B and a different region B

For the 3rd use case, we’ll use an interface endpoint (PrivateLink), not a gateway endpoint. This is what AWS documentation has to say. “With AWS PrivateLink for Amazon S3, you can provision interface VPC endpoints (interface endpoints) in your virtual private cloud (VPC). These endpoints are directly accessible from applications that are on premises over VPN and AWS Direct Connect, or in a different AWS Region over VPC peering. Interface endpoints are represented by one or more elastic network interfaces (ENIs) that are assigned private IP addresses from subnets in your VPC. Requests to Amazon S3 over interface endpoints stay on the Amazon network. You can also access interface endpoints in your VPC from on-premises applications through AWS Direct Connect or AWS Virtual Private Network (AWS VPN).”
Let’s create the resources. VPC and EC2. Change the AWS CLI profile so it’s using the same account B and a new region B.

REGION_B="us-east-1"
CIDR_C="192.168.30.0/24"
VPCC_ID=$(aws ec2 create-vpc \
    --cidr-block $CIDR_C \
    --region $REGION_B \
    --tag-specification 'ResourceType=vpc,Tags=[{Key=Name,Value=VPC-C}]' \
    --output text --query 'Vpc.VpcId')
echo $VPCC_ID

Enable DNS and hostnames resolution, needed for SSM. Two separate lines needed.

aws ec2 modify-vpc-attribute --enable-dns-hostnames "{\"Value\":true}"  --vpc-id $VPCC_ID
aws ec2 modify-vpc-attribute --enable-dns-support "{\"Value\":true}"  --vpc-id $VPCC_ID

Create a default network ACL.

aws ec2 create-network-acl --vpc-id $VPCC_ID

Create an Internet Gateway and attach it to the VPC.

IGWC_ID=$(aws ec2 create-internet-gateway \
    --region $REGION_B \
    --tag-specifications 'ResourceType=internet-gateway,Tags=[{Key=Name,Value=igw-C}]' \
    --output text --query 'InternetGateway.InternetGatewayId')
echo $IGWC_ID
aws ec2 attach-internet-gateway \
    --internet-gateway-id $IGWC_ID \
    --vpc-id $VPCC_ID

Let’s create a subnet.

SUBC_ID=$(aws ec2 create-subnet \
    --vpc-id $VPCC_ID \
    --cidr-block 192.168.30.0/24 \
    --region $REGION_B \
    --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=subPubC}]' \
    --output text --query 'Subnet.SubnetId')
echo $SUBC_ID

Create a route table.

RTC_ID=$(aws ec2 create-route-table \
     --vpc-id $VPCC_ID \
     --output text --query 'RouteTable.RouteTableId')
echo $RTC_ID
aws ec2 create-route --route-table-id $RTC_ID --destination-cidr-block 0.0.0.0/0 --gateway-id $IGWC_ID

Associate this route table to the public subnet.

aws ec2 associate-route-table --route-table-id $RTC_ID --subnet-id $SUBC_ID

SGC_ID=$(aws ec2 create-security-group \
    --group-name sgAllowICMPandOutboundAccess --description "Allows ICMP and outbound access" \
    --vpc-id $VPCC_ID \
    --output text --query 'GroupId')
echo $SGC_ID

Allow unrestricted egress. By default this rule is there, so don’t execute the code below.

#aws ec2 authorize-security-group-egress \
    --group-id $SGB_ID \
    --protocol all \
    --cidr "0.0.0.0/0"

Do this one, so we can ping instances from each other when we do VPC peering.

aws ec2 authorize-security-group-ingress \
  --group-id $SGC_ID \
  --ip-permissions IpProtocol=icmp,FromPort=-1,ToPort=-1,IpRanges="[{CidrIp=192.168.10.0/24},{CidrIp=192.168.20.0/24},{CidrIp=192.168.30.0/24}]"

We don’t have to create any roles. We’ll use the existing ones that we built for the 2nd use case.
Create the EC2. The AMI ID will change this time.

AMI_ID="ami-08a0d1e16fc3f61ea"
INS_TYPE="t3.micro"
aws ec2 run-instances \
    --image-id $AMI_ID \
    --instance-type $INS_TYPE \
    --security-group-ids $SGC_ID \
    --subnet-id $SUBC_ID \
    --iam-instance-profile Name="ipEC2" \
    --associate-public-ip-address  \
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=ec2-C}]'

Same thing as we did before, connect from SSM, configure AWS CLI using the previous credentials and try to list the bucket.

BUCKET_NAME="myuniquenameforthebucket"
aws s3 ls s3://$BUCKET_NAME

You should see the file there. But, this access goes over Internet. Now, we have to create an interface endpoint in the destination VPC which is VPC-A. When we create an interface endpoint a private IP will be generated for this interface endpoint. In order this private IP to access the bucket, we need some sort of connection between regions, e.g. VPC peering or Transit Gateway. Let’s do the peering between account A (VPC-A) and account B (VPC-C). Enter the 12 digit account number for account A where the S3 bucket resides.

echo $VPCA_ID
echo $VPCC_ID
echo $REGION_A
ACCOUNT_A="123456789012"
echo $ACCOUNT_A

Make sure all of the above returns some value. Do the peering.

aws ec2 create-vpc-peering-connection \
  --vpc-id $VPCC_ID --peer-vpc-id $VPCA_ID \
  --peer-owner-id $ACCOUNT_A --peer-region $REGION_A

Log to the account A and under VPC | Peering menu on the left, select the peering and accept it. But this is not enough. You need to tell the VPC to use the routes over the peering connection. Go to account A, VPC A, find the route for the subnet where the EC2 instance is and add a route for destination 192.168.30.0/24 to go over peering. As you can see this is the same route that we modified initially in our first use case.

Now, log to account B and VPC C and modify the route but this time, the destination is VPC A (192.168.10.0/24). At this point you should be able to ping EC2-A from EC2-C. Don’t worry if you can’t ping EC2-C from EC2-A.
At this point we can create the interface endpoint where the bucket is. This time choose interface as type, VPC-A, our security group, full access policy and the subnet where EC2 is. This subnet has nothing to do with the bucket access, we are just creating a network interface there. We can choose any subnet in that VPC, we just need the IP assigned to the endpoint interface.

There will be 2 DNS names created for you. One is referring the zone and the other the region. Use the regional DNS name, the first one and add the word bucket instead of the * character.

From instance EC2-C, this is how you will access the bucket.

aws s3 ls s3://myuniquenameforthebucket --endpoint-url https://bucket.vpce-038d5f94d78d8b216-vyvk5uon.s3.us-east-2.vpce.amazonaws.com

If you try from the instance EC2-C, this won’t work. Why? Because of the security group that we used. We have to allow HTTPS traffic from 192.168.30.0/24. Go find the first security group, that’s the one we used for the interface endpoint and allow HTTPS from 192.168.30.0/24. After the change, you’ll be able to access the bucket.

Okta and AWS SSO (AWS IAM Identity Center)

Kliment Andreev — Wed, 20 Mar 2024 17:25:23 +0000

In this post I’ll describe how to integrate Okta and your AWS account. In order everything to work, you’ll have to enable AWS Organization on the master account and then you can add multiple sub accounts in AWS.
Best practice says that the AWS master account should be used for SSO only and you shouldn’t run any workloads. First thing first, let’s start with Okta.

Okta

Log to your Okta account or sign up for a 30 day trial. Once logged in, if you haven’t already done so, change the sign-in domain from trail*-okta.com to your domain. Or you can skip this step.
Click on Admin from the upper right corner and then under Customizations, click on Domain. I’ve used the Okta-managed customization because it’s easier to manage.

After you click Next, you’ll be prompted to add two DNS entries for your domain.

Change the DNS. This will allow us to access Okta dashboard using our domain in my case it’s okta.andreev.it. From the same menu on the left, click on Applications, then Applications again and click on Browse App Catalog.

Search for AWS IAM and choose AWS IAM Identity Center.

Click on Add Integration and then Done.
Click on the Sign On tab and scroll all the way down to SAML Signing Certificates. Click on Actions drop-down and then View IdP metadata.

This will open a new tab with an XML file. Save the file as metadata.xml.

AWS

Go to console.aws.amazon.com and log with your root account. Go to IAM Identity Center and click on Enable and then select Enable with AWS Organizations.

Click on Settings on the left, then Actions drop-down and click on Customize AWS access portal URL. Change the URL so it’s something that you can easily remember, e.g. yourorg or aws-master.

Click on Settings on the left, then Actions drop-down and click on Change identity source.

Choose External identity provider and click Next.

You’ll land on a page with two parts, Service provider metadata (that’s AWS) and Identity provider metadata (that’s Okta).
Click on Download metadata file from upper right. You’ll need this in Okta. The file will be named with some date prefix and some characters plus metadata.xml
Copy the values for the 2nd and 3rd entry. You’ll also need these for Okta.

Click on Choose file under Identity provider metadata and upload the metadata.xml file that you’ve downloaded earlier from Okta.

Click Next, type ACCEPT and finally click Change identity source.

Okta

Go back to Okta, Sign On tab and right bellow click Edit. Enter the two URL entries that you copied from AWS. Click Save after.

AWS

Go back to AWS, then IAM Identity Center, click on Settings and click to Enable the Automatic Provisioning.

Click on Show Token and copy both values for the SCIM endpoint and the Access Token.

Okta

Go back to Okta and for the AWS app, click on Provisioning and then click on Configure API Integration.

Click Enable API Integration and enter the SCIM endpoint and the token value in the corresponding fields. Click Save. Remove the trailing slash from the end of the Base URL (SCIM endpoint) if you get an error (Base URL: Does not match required pattern)
Under Provisioning tab, click on To App, click Edit and select all three checkmarks. Click Save.

Then, go to Assignments tab and assign a group or a user to this AWS app.

AWS

Go back to AWS and under IAM Identity Center and then Users, you can see the user that we just assigned.

But this user has no rights in AWS, so we have to create a Permission set and then assign this user to that permission set and the account that this user has access to. Click on Permission sets on the left and then Create permission set.

I chose AdministratorAccess from the Predefined permission sets.

Go back to AWS accounts, click on the account and choose Assign users or groups.

Select the User tab, click on the user, click Next.

Select the permission set that we just created and click Next then Submit.

We just told AWS that the user xyz will have access to the account abc with AdministratorAccess permissions.

Test SSO

Open a new tab and go to the URL that you configured earlier (Customize AWS access portal). It should be https://whatever.awsapps.com/start.
AWS will redirect you to Okta where you’ll enter your Okta username.

Once authenticated, you’ll see the AWS accounts in your org and the permissions that you have.

Entra ID (Azure AD) and AWS SSO (AWS IAM Identity Center)

Kliment Andreev — Mon, 11 Mar 2024 18:58:39 +0000

I wrote about this topic a couple of years ago, but some things changed so I am writing this post again. In this post I’ll explain how to log to AWS Console and AWS cli with Entra ID (former Azure AD) credentials. For this you’ll need admin access to both the AWS account and Entra ID.

IAM Identity Center

The former AWS Single Sign-on is now IAM Identity Center. You need to enable this in order to configure SSO. Go to IAM Identity Center and click on the Enable button.

Enable it with AWS Organizations.

What you want to do first is to change the AWS access portal URL. Click on the Edit button and change the URL. It has to be unique, so don’t expect you can type aws and be done with it.

Another setting that you have to change is the Identity source. On the left side of the IAM Identity Center, you are on the Dashboard screen. Click on Settings and then click on Change identity source.

Choose External identity provider and click Next.

You will see this screen which consists of two parts, service provider metatada (that’s AWS) and Identity provider metadata (that’s Entra ID). Click on Download metadata file. You’ll get an XML file.

Keep the page open and in a new browser window go to portal.azure.com.

Entra ID

Once logged to Azure, go to Enterprise Applications and search for AWS IAM Identity Center (successor to AWS Single Sign-On). Click to install. When you are presented with the Overview of the application, click on Assign users and groups. Depending on your Entra ID level (P1 or P2) you might not be able to assign groups, so you have to assign users.

The next step is to setup the SSO. Click on 2. Set up single sign on and then SAML. You’ll be presented with this page and eventually you’ll see a prompt if you want to save some SSO settings. If you don’t see the prompt, no worries. If you see the prompt, click Yes.

Click on Upload metadata file and point to the XML file that you saved from AWS. Click Save and don’t worry about any of the fields there. If you see a popup to test the config, just click No, I’ll test later. Under step 3, SAML Certificates, click to Download Federation Metadata XML.

You’ll get another XML file called AWS IAM Identity Center (successor to AWS Single Sign-On).xml.

Back to AWS

Go back to the same page where you’ve downloaded the XML metadata file, but this time look at the bottom part where it says Identity provider metadata and IdP SAML metadata. Click on Choose file and upload the file from Azure. Click Next, type ACCEPT and click Change identity source.
Once completed click to Enable Automatic provisioning. Copy the SCIM endpoint and the access token.

Back to Azure

Under the AWS SSO application. on the left side click on Provisioning. Change the Provisioning Mode to Automatic. Expand Admin credentials and for Tenant ID enter the SCIM endpoint and for Secret Token, enter the token. Click Test Connection and upon successful test, click Save. If you see clientsecret instead of Tenant ID, it means you are not using the right AWS application and you use the AWS Single Account application instead. Start from scratch.
Anytime you assign a new user or a group to AWS application, Azure will sync the user to AWS. But, instead of waiting (40 mins by default) for the initial sync, click on the Overview and click Start provisioning.

Back to AWS

If your provisioning works fine, you’ll see your assigned Azure user under Users in IAM Identity Center. On the left side of the IAM Identity Center, click on Permission sets. Click Create Permission set and choose Predefined permission set, then AdministratorAccess. Change the name, description, session duration if needed and then click Next and then Create.
Finally, click on AWS accounts, select an account from the organization, click Assign users or groups, click Users or Groups, select a user and click Next. Assign a permission set, click Next and Submit.

Now, test the SSO by going to whatever.awsapps.com/start# and you’ll be redirected to the Microsoft sign in prompt. Follow the authentication process for Entra ID (username, password, MFA etc) and once you pass the authentication process you’ll see the AWS account that you have access to.

AWS CLI and SSO

Follow the instructions to install AWS CLI for your OS, then configure SSO. Name your session, enter the URL you configured earlier, enter the region where you configured SSO and hit enter for the registration scopes.

aws configure sso
SSO session name (Recommended): sso
SSO start URL [None]: https://your_url.awsapps.com/start
SSO region [None]: us-east-1
SSO registration scopes [sso:account:access]:
Attempting to automatically open the SSO authorization page in your default browser.
If the browser does not open or you wish to use a different device to authorize this request, open the following URL:

https://device.sso.us-east-1.amazonaws.com/

Then enter the code:

PWKG-CAFZ

After successful authentication, you’ll be presented with the accounts available to you. Pick one.

You’ll be prompted to choose the roles available to you and then specify your default region, default output and the profile. \
NOTE: If you’ll be working with multiple accounts, maybe you should specify a different profile than the default. It’s up to you. E.g. specify prod and dev profiles.

There are 2 AWS accounts available to you.
Using the account ID 123456789012
The only role available to you is: AdministratorAccess
Using the role name "AdministratorAccess"
CLI default client Region [None]: us-east-2
CLI default output format [None]: json
CLI profile name [AdministratorAccess-123456789012]: default

To use this profile, specify the profile name using --profile, as shown:

aws s3 ls --profile default

After your SSO session expires, you have to log back in.

aws sso login

If you are like me and always uses the default profile so I don’t have to type –profile parameter, then anytime you want to use a different account, just aws configure sso again, but specify the other account now.

AWS: Get secrets from Secrets Manager using CSI Driver and ASCP

Kliment Andreev — Wed, 03 Jan 2024 21:02:53 +0000

In one of my previous posts, I wrote about accessing DynamoDB and AWS Gateway/Lambda using IAM roles and secrets from EC2/EKS Node.js application. The idea is to reference passwords in your code and never store and secrets/passwords in the code. By using roles, we can access the secrets from the AWS Secrets Manager and use them dynamically in the code. However, that solution has its own pros and cons. The pros are that you don’t have to do anything extra except creating an IAM role and secrets. The cons are that you need AWS SDK as part of your application. You will use AWS SDK for Node.js to get the credentials from the Secrets Manager. This means you have to maintain up-to-date AWS SDK and add extra code. Another way is to use the k8s CSI driver and ASCP (provider) that pretty much gets the secret and mounts it as a file. You still need IAM roles to access the Secrets Manager but you don’t need AWS SDK. You can read the secrets as you read a file.

Create an EKS cluster

We’ll create an EKS cluster using the eksctl tool.

CLUSTER_NAME="eksDemoCluster"
REGION="us-east-2"
NAMESPACE="default"

eksctl create cluster --name $CLUSTER_NAME \
  --region $REGION --version 1.28 \
  --node-type t3.small --nodes 2

We’ll use helm to install the CSI driver and the specific ASCP provider for AWS.

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm repo add aws-secrets-manager https://aws.github.io/secrets-store-csi-driver-provider-aws
helm repo update
helm install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver
helm install -n kube-system secrets-provider-aws aws-secrets-manager/secrets-store-csi-driver-provider-aws

Check if everything is OK.

kubectl --namespace=kube-system get pods -l "app=secrets-store-csi-driver"
NAME                                               READY   STATUS    RESTARTS   AGE
csi-secrets-store-secrets-store-csi-driver-nhq7b   3/3     Running   0          19s
csi-secrets-store-secrets-store-csi-driver-tft6q   3/3     Running   0          19s

Create a secret

Let’s create a secret that we’ll be using in a mock-up app.

SECRET_NAME="demoapp/mysecret"
SECRET_ARN=$(aws secretsmanager create-secret  --name $SECRET_NAME \
  --secret-string '{"username":"Administrator", "password":"SuperSecretPassword$"}' \
  --output text --query 'ARN' --region $REGION)
echo $SECRET_ARN
arn:aws:secretsmanager:us-east-2:123456789012:secret:demoapp/mysecret-lboyfd

Create an IAM policy

We also need to define an IAM policy that will allow our EKS pods to access only this particular secret and nothing else.

cat << EOF > polDemoSecret.json
{
    "Version": "2012-10-17",
    "Statement": [ {
        "Effect": "Allow",
        "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
        "Resource": ["$SECRET_ARN"]
    } ]
}
EOF

Create the policy.

POLICY_ARN=$(aws iam create-policy --policy-name polDemoSecret \
  --policy-document file://polDemoSecret.json \
  --output text --query 'Policy.Arn')
echo $POLICY_ARN
arn:aws:iam::123456789012:policy/polDemoSecret

Create an IAM OIDC provider

Let’s determine whether we have an existing IAM OIDC provider for our cluster.

aws eks describe-cluster --name $CLUSTER_NAME --region $REGION \
  --query "cluster.identity.oidc.issuer" --output text
https://oidc.eks.us-east-2.amazonaws.com/id/D7FAD6F2BEF430FC1CB673777A9E4FED

When we provisioned the cluster with eksctl, the OIDC was created automatically. We need the ID of the OIDC which is the hex value at the end.

OIDCID=$(aws eks describe-cluster --name $CLUSTER_NAME --region $REGION \
  --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)
echo $OIDCID
E9B5E80C37A71FA3932B94EFF5BA3437

Check if the IAM OIDC provider is already configured. It shouldn’t if you provisioned a brand new cluster.

aws iam list-open-id-connect-providers | grep $OIDCID

If for whatever reason, you had an output from the command above, skip this associate-iam-oidc-provider step below.
If you don’t have any output, associate the OIDC provider with the cluster.

eksctl utils associate-iam-oidc-provider --cluster $CLUSTER_NAME --region $REGION --approve
2023-12-22 12:27:47 [ℹ]  will create IAM Open ID Connect provider for cluster "eksDemoCluster" in "us-east-2"
2023-12-22 12:27:47 [✔]  created IAM Open ID Connect provider for cluster "eksDemoCluster" in "us-east-2"

Cluster service account

We also need a service account for the cluster and we need to attache the policy for the secrets that we created before.

SA_NAME="sademosecret"
eksctl create iamserviceaccount --name $SA_NAME --namespace $NAMESPACE \
--cluster $CLUSTER_NAME  --region $REGION \
--attach-policy-arn $POLICY_ARN --approve --override-existing-serviceaccounts
kubectl get sa

Service Provider Class

…and finally we need a service provider class that we’ll deploy on the EKS cluster.

cat << EOF > spc.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: spc-demosecret
  namespace: $NAMESPACE
spec:
  provider: aws
  parameters:
    objects: |
        - objectName: "$SECRET_NAME"
          objectType: "secretsmanager"
EOF
kubectl apply -f spc.yaml

Test with busybox pod

Create this YAML file that will deploy a pod. Pay attention to highlighted lines. We have to provide the namespace, the service account and the service provider class that specifies the secret.Check this link on how to mount and specify different secrets.

cat << EOF > app.yaml
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: $NAMESPACE
spec:
  serviceAccountName: $SA_NAME
  volumes:
  - name: secretsvolume
    csi:
       driver: secrets-store.csi.k8s.io
       readOnly: true
       volumeAttributes:
         secretProviderClass: "spc-demosecret"
  containers:
  - image: public.ecr.aws/docker/library/busybox:1.36
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox
    volumeMounts:
    - name: secretsvolume
      mountPath: "/mnt/secrets-store"
      readOnly: true
  restartPolicy: Always
EOF
kubectl apply -f app.yaml

Test the secrets retrieval. As you can see, the secret that we created was demoapp/mysecret, but in this case the forward slash would mean a directory, so the actual secret is demoapp_mysecret… and it’s simple as that. The secret is just a file under the /mnt/secrets-store directory.

kubectl exec -it $(kubectl get pods | awk '/busybox/{print $1}' | head -1) -- cat /mnt/secrets-store/demoapp_mysecret; echo
{"username":"Administrator", "password":"SuperSecretPassword$"}

Node.js app

Create a folder and add this file. Save it as server.js.

'use strict';
const express = require('express')
const app = express()
const port = 3000
 
var fs = require('fs');
try {  
    var data = fs.readFileSync('/mnt/secrets-store/demoapp_mysecret', 'utf8');
    console.log(data.toString());    
} catch(e) {
    console.log('Error:', e.stack);
}

app.get('/', (req, res) => {
    res.send(JSON.stringify(data));
})
 
app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

Let’s create the Dockerfile so we can build the image.

FROM node:20
  
# Create app directory
WORKDIR /usr/src/app
  
# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm@5+)
COPY package*.json ./
  
RUN npm install
# If you are building your code for production
# RUN npm ci --only=production
  
# Bundle app source
COPY . .
  
EXPOSE 3000
CMD [ "node", "server.js" ]

Create a .dockerignore file.

node_modules
npm-debug.log

Now, we can create the image and push to Dockerhub or some other image repo. Use your own repo here at line 1.

sudo docker build . -t klimenta/secretsdemo
sudo docker images
sudo docker push klimenta/secretsdemo

Create a file to deploy 2 replicas and a load balancer. Save it as something.yaml and deploy it. Analyze the file and see how we use the service account, secret provider class and the image.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demosecret
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      run: demosecret
  template:
    metadata:
      labels:
        run: demosecret
    spec:
      serviceAccountName: sademosecret
      volumes:
      - name: secretsvolume
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "spc-demosecret"
      containers:
      - name: demosecret
        image: klimenta/secretsdemo
        volumeMounts:
        - name: secretsvolume
          mountPath: "/mnt/secrets-store"
          readOnly: true
        ports:
        - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: loadbalancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
spec:
  ports:
    - port: 80
      targetPort: 3000
      protocol: TCP
  type: LoadBalancer
  selector:
    run: demosecret

Once you deploy with kubectl apply -f something.yaml, type kubectl get svc and you’ll see a public URL.
If you go to that URL, you’ll see our secret.

AWS: Install EKS cluster and Ingress Controller using Terraform and GitLab using OIDC and web identity

Kliment Andreev — Tue, 21 Nov 2023 18:39:16 +0000

In this post, I’ll explain how to create an EKS cluster in AWS and add an Ingress Controller. We’ll use GitLab as a repo and execute the CI/CD pipeline from there.
For this you’ll need:

GitLab and AWS account
Terraform, AWS CLI, helm installed

AWS

From the IAM console, create a new Identity Provider in AWS and use https://gitlab.com for both Provider URL and Audience. For self hosted GitLab use gitlab.domain.com or whatever is your GitLab URL.
Click on Get thumbprint before clicking on Add provider.

In the IAM console, create a role. I’ll name the role rolGitLab, name it whatever you like. When you create the policy, choose Custom trust policy.

Now, here is what you have to decide. If you want to manage the cluster from GitLab only, you can ommit lines 16-23 where I allow the IAM user called gitlab to assume this role. But if you want some IAM user to manage the cluster using aws cli or kubectl, add the ARN of that IAM user in line 20. Add AdministratorAccess to that gitlab user but you can fine-tune those policies later. If you want to manage the cluster from GitLab, you have to register an agent and provide a config file. You can read how to do that here.
Replace the default custom trust policy with this JSON.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Principal": {
				"Federated": "arn:aws:iam::123456789012:oidc-provider/gitlab.com"
			},
			"Action": "sts:AssumeRoleWithWebIdentity",
			"Condition": {
				"StringLike": {
					"gitlab.com:sub": "project_path:*/eks-gitlab:ref_type:branch:ref:main"
				}
			}
		},
		{
			"Sid": "",
			"Effect": "Allow",
			"Principal": {
				"AWS": "arn:aws:iam::123456789012:user/gitlab"
			},
			"Action": "sts:AssumeRole"
		}
	]
}

In line 7 and 20, change the 123456789012 with your account number and in line 7 change gitlab.com with however you named the Identity Provider (IdP). In line 12 replace eks-gitlab with how you plan to name your GitLab project. In line 20 change the gitlab user to however you named your IAM user to access the EKS cluster. Click Next and when asked for a permission policy, choose AdministratorAccess. Best practice is to limit access to only what’s needed, but EKS has so many policies, it will take a lot to list them all.
In case you created the gitlab IAM user, attach the AdministratorPolicy (fine tune later) and this inline policy. Change the ARN in line 7. Name this inline policy something like polAssumeGitLabRole.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::123456789012:role/rolGitLab"
        }
    ]
}

GitLab

Create a new blank project in Gitlab with the default settings. In my case the project name is called eks-gitlab.

Go to Settings on the left, then CI/CD and then Variables. Add these two variables.

AWS_PROFILE and a value of oidc. Yes, the value is oidc. Just type these 4 characters for the value.
ROLE_ARN and a value of the role ARN from the role you just created for rolGitLab

Create a .gitlab-ci.yml file with the following content. Change the domain for gitlab.com at line 7 if you use self-hosted GitLab.

assume role:
  image: 
    name: amazon/aws-cli:latest
    entrypoint: [""]
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com
  before_script:
    - mkdir -p ~/.aws
    - echo "${GITLAB_OIDC_TOKEN}" > /tmp/web_identity_token
    - echo -e "[profile oidc]\nrole_arn=${ROLE_ARN}\nweb_identity_token_file=/tmp/web_identity_token" > ~/.aws/config
  script:
      - aws sts get-caller-identity
      - aws s3 ls

By the time you commit changes, the pipeline will start and if everything is OK, you’ll see your S3 buckets in the logs of the job if you go to Build | Jobs.

One more thing related to Terraform state as we’ll keep the state in GitLab is that we have to create an Access Token. If you have a self-hosted or paid GitLab account, create an Access Token from the Project | Settings | Access Tokens.
If you have a free GitLab account, you have to create a Personal Access Token and you do that if you click on your avatar in the upper left corner, then select Edit Profile and then click Access Tokens on the left. Click Add new token, name it GITLAB_PAT and select API only. Click Create personal access token. Once the token has been created click on the eye icon where it says Your new personal access token and copy & paste somewhere safe.

Go to the project settings and add these 2 variables.

TF_USERNAME which is your username, click on the avatar, look at the @ under your name, your username is without the @ sign
TF_PASSWORD which is the value of your personal access token you just created

Local machine

Now that we know the pipeline is working properly, let’s clone the project on our local machine.
If you plan to manage the cluster from your local machine, now it’s time to configure the gitlab IAM user on your local machine using access keys.
You need to have a valid key in order to clone your private repo.
Upload your .pub key to GitLab by clicking your avatar, then Edit Profile, click SSH Keys and then Add new key.
Finally, check the connection with the following command. It doesn’t matter what your local Linux user is, just use git@gitlab.com as the username@hostname. Type as is.

ssh -T git@gitlab.com
Welcome to GitLab, @[your_gitlab_username_here]

Once you clone the repo add the following files locally.
FILE: .gitignore

# Local .terraform directories
**/.terraform/*

# .tfstate files
*.tfstate
*.tfstate.*

# Crash log files
crash.log
crash.*.log

# Exclude all .tfvars files, which are likely to contain sensitive data, such as
# password, private keys, and other secrets. These should not be part of version 
# control as they are data points which are potentially sensitive and subject 
# to change depending on the environment.
*.tfvars
*.tfvars.json

# Ignore override files as they are usually used to override resources locally and so
# are not checked in
override.tf
override.tf.json
*_override.tf
*_override.tf.json

# Include override files you do wish to add to version control using negated pattern
# !example_override.tf

# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
# example: *tfplan*

# Ignore CLI configuration files
.terraformrc
terraform.rc

FILE: LICENSE

This is free and unencumbered software released into the public domain.

Anyone is free to copy, modify, publish, use, compile, sell, or
distribute this software, either in source code form or as a compiled
binary, for any purpose, commercial or non-commercial, and by any
means.

In jurisdictions that recognize copyright laws, the author or authors
of this software dedicate any and all copyright interest in the
software to the public domain. We make this dedication for the benefit
of the public at large and to the detriment of our heirs and
successors. We intend this dedication to be an overt act of
relinquishment in perpetuity of all present and future rights to this
software under copyright law.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.

For more information, please refer to

FILE: alb_ingress_iam_policy.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:CreateServiceLinkedRole"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeAccountAttributes",
                "ec2:DescribeAddresses",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeInternetGateways",
                "ec2:DescribeVpcs",
                "ec2:DescribeVpcPeeringConnections",
                "ec2:DescribeSubnets",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeInstances",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeTags",
                "ec2:GetCoipPoolUsage",
                "ec2:DescribeCoipPools",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeLoadBalancerAttributes",
                "elasticloadbalancing:DescribeListeners",
                "elasticloadbalancing:DescribeListenerCertificates",
                "elasticloadbalancing:DescribeSSLPolicies",
                "elasticloadbalancing:DescribeRules",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeTargetGroupAttributes",
                "elasticloadbalancing:DescribeTargetHealth",
                "elasticloadbalancing:DescribeTags"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "cognito-idp:DescribeUserPoolClient",
                "acm:ListCertificates",
                "acm:DescribeCertificate",
                "iam:ListServerCertificates",
                "iam:GetServerCertificate",
                "waf-regional:GetWebACL",
                "waf-regional:GetWebACLForResource",
                "waf-regional:AssociateWebACL",
                "waf-regional:DisassociateWebACL",
                "wafv2:GetWebACL",
                "wafv2:GetWebACLForResource",
                "wafv2:AssociateWebACL",
                "wafv2:DisassociateWebACL",
                "shield:GetSubscriptionState",
                "shield:DescribeProtection",
                "shield:CreateProtection",
                "shield:DeleteProtection"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:RevokeSecurityGroupIngress"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateSecurityGroup"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags"
            ],
            "Resource": "arn:aws:ec2:*:*:security-group/*",
            "Condition": {
                "StringEquals": {
                    "ec2:CreateAction": "CreateSecurityGroup"
                },
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags",
                "ec2:DeleteTags"
            ],
            "Resource": "arn:aws:ec2:*:*:security-group/*",
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:DeleteSecurityGroup"
            ],
            "Resource": "*",
            "Condition": {
                "Null": {
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:CreateLoadBalancer",
                "elasticloadbalancing:CreateTargetGroup"
            ],
            "Resource": "*",
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:CreateListener",
                "elasticloadbalancing:DeleteListener",
                "elasticloadbalancing:CreateRule",
                "elasticloadbalancing:DeleteRule"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:AddTags",
                "elasticloadbalancing:RemoveTags"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ],
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:AddTags",
                "elasticloadbalancing:RemoveTags"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:ModifyLoadBalancerAttributes",
                "elasticloadbalancing:SetIpAddressType",
                "elasticloadbalancing:SetSecurityGroups",
                "elasticloadbalancing:SetSubnets",
                "elasticloadbalancing:DeleteLoadBalancer",
                "elasticloadbalancing:ModifyTargetGroup",
                "elasticloadbalancing:ModifyTargetGroupAttributes",
                "elasticloadbalancing:DeleteTargetGroup"
            ],
            "Resource": "*",
            "Condition": {
                "Null": {
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:AddTags"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ],
            "Condition": {
                "StringEquals": {
                    "elasticloadbalancing:CreateAction": [
                        "CreateTargetGroup",
                        "CreateLoadBalancer"
                    ]
                },
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:RegisterTargets",
                "elasticloadbalancing:DeregisterTargets"
            ],
            "Resource": "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:SetWebAcl",
                "elasticloadbalancing:ModifyListener",
                "elasticloadbalancing:AddListenerCertificates",
                "elasticloadbalancing:RemoveListenerCertificates",
                "elasticloadbalancing:ModifyRule"
            ],
            "Resource": "*"
        }
    ]
}

FILE: backend.tf

terraform {
  backend "http" {
  }
}

FILE: data.tf

data "aws_availability_zones" "available" {
}

data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_name
  depends_on = [module.eks.cluster_name]
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_name
  depends_on = [module.eks.cluster_name]
}

FILE: main.tf NOTE: Make sure you add lines 40-41 and specify the rolGitlab role created earlier. If you don’t do this when you destroy the cluster, the KMS key for EKS will stay there and there is no way you can delete it unless you call AWS support.

# Create a VPC
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.0.0"

  name = var.vpcName

  cidr = var.vpcCIDR
  azs  = slice(data.aws_availability_zones.available.names, 0, 3)

  private_subnets = var.privateSubnets
  public_subnets  = var.publicSubnets

  enable_nat_gateway   = var.enable_nat_gateway
  single_nat_gateway   = var.single_nat_gateway
  enable_dns_hostnames = var.enable_dns_hostnames

  public_subnet_tags = {
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
    "kubernetes.io/role/elb"                    = 1
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
    "kubernetes.io/role/internal-elb"           = 1
  }
}

# Create an EKS cluster
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "19.15.3"

  cluster_name    = var.cluster_name
  cluster_version = var.cluster_version

  vpc_id                         = module.vpc.vpc_id
  subnet_ids                     = module.vpc.private_subnets
  cluster_endpoint_public_access = var.cluster_endpoint_public_access
  kms_key_administrators         = ["arn:aws:iam::123456789012:role/rolGitLab"]
  kms_key_enable_default_policy  = true

  eks_managed_node_group_defaults = {
    ami_type = var.ami_type
  }

  eks_managed_node_groups = {
    one = {
      name = var.nodeGroup

      instance_types = var.instanceTypes

      min_size     = var.minSize
      max_size     = var.maxSize
      desired_size = var.desiredSize
    }
  }
}

# Create an IAM policy for the ALB ingress
resource "aws_iam_policy" "worker_policy" {
  name        = "worker-policy"
  description = "Worker policy for the ALB Ingress"

  policy = file(var.iamALBIngressPolicyFileName)
}

resource "aws_iam_role_policy_attachment" "additional" {
  for_each = module.eks.eks_managed_node_groups

  policy_arn = aws_iam_policy.worker_policy.arn
  role       = each.value.iam_role_name
}

# Crete the Ingress using Helm chart from AWS
resource "helm_release" "ingress" {
  name       = "ingress"
  chart      = "aws-load-balancer-controller"
  repository = "https://aws.github.io/eks-charts"
  version    = "1.6.1"

  set {
    name  = "autoDiscoverAwsRegion"
    value = "true"
  }

  set {
    name  = "autoDiscoverAwsVpcID"
    value = "true"
  }

  set {
    name  = "clusterName"
    value = module.eks.cluster_name
  }
}

FILE: outputs.tf

output "cluster_endpoint" {
  description = "Endpoint for EKS control plane"
  value       = module.eks.cluster_endpoint
}

output "cluster_security_group_id" {
  description = "Security group ids attached to the cluster control plane"
  value       = module.eks.cluster_security_group_id
}

output "region" {
  description = "AWS region"
  value       = var.region
}

output "cluster_name" {
  description = "Kubernetes Cluster Name"
  value       = module.eks.cluster_name
}

FILE: providers.tf

provider "aws" {
  region = var.region

  shared_config_files      = ["~/.aws/config"]  
  profile                  = "oidc"
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    token                  = data.aws_eks_cluster_auth.cluster.token
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  }
}

FILE: variables.tf

variable "region" {
  type    = string
  default = "us-east-2"
}

variable "vpcName" {
  type    = string
  default = "vpcFromGitLab"
}

variable "vpcCIDR" {
  type    = string
  default = "10.0.0.0/16"
}

variable "privateSubnets" {
  type    = list(string)
  default = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
}

variable "publicSubnets" {
  type    = list(string)
  default = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]
}

variable "enable_nat_gateway" {
  type    = bool
  default = true
}

variable "single_nat_gateway" {
  type    = bool
  default = false
}

variable "enable_dns_hostnames" {
  type    = bool
  default = true
}

variable "cluster_name" {
  type    = string
  default = "eksFromGitLab"
}

variable "cluster_version" {
  type    = string
  default = "1.27"
}

variable "cluster_endpoint_public_access" {
  type    = bool
  default = true
}

variable "ami_type" {
  type    = string
  default = "AL2_x86_64"
}

variable "nodeGroup" {
  type    = string
  default = "ngFromGitLab"
}

variable "instanceTypes" {
  type    = list(string)
  default = ["t3.small"]
}

variable "minSize" {
  type    = number
  default = 1
}

variable "maxSize" {
  type    = number
  default = 3
}

variable "desiredSize" {
  type    = number
  default = 3
}


variable "iamALBIngressPolicyFileName" {
  type    = string
  default = "alb_ingress_iam_policy.json"
}

…and finally your .gitlab-ci.yml file.
FILE: .gitlab-ci.yml. NOTE: Remove line 38 if you don’t use it, don’t keep it with a comment or you’ll get an error in the pipeline.

stages:          
  - validate
  - plan
  - apply
  - destroy

image:
  name: hashicorp/terraform:1.6
  entrypoint:
    - "/usr/bin/env"
    - "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

variables:
  TF_DIR: ${CI_PROJECT_DIR}
  TF_STATE_NAME: "eks-tf-state"
  TF_ADDRESS: "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/terraform/state/${TF_STATE_NAME}"

default:
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com

  before_script:
    - terraform --version
    - cd ${TF_DIR}
    - terraform init
      -reconfigure 
      -backend-config="address=${TF_ADDRESS}" 
      -backend-config="lock_address=${TF_ADDRESS}/lock" 
      -backend-config="unlock_address=${TF_ADDRESS}/lock" 
      -backend-config="username=${TF_USERNAME}" 
      -backend-config="password=${TF_PASSWORD}" 
      -backend-config="lock_method=POST" 
      -backend-config="unlock_method=DELETE" 
      -backend-config="retry_wait_min=5"
    - mkdir -p ~/.aws
    - echo "${GITLAB_OIDC_TOKEN}" > /tmp/web_identity_token
    - # echo "${GITLAB_OIDC_TOKEN}" | base64 - If you want to see the token, reveal it this way and then use base64 --decode to get the real one
    - echo -e "[profile oidc]\nrole_arn=${ROLE_ARN}\nweb_identity_token_file=/tmp/web_identity_token" > ~/.aws/config

job-validate:
  stage: validate
  script:
    - terraform validate

job-plan:
  stage: plan
  script:
    - terraform plan -out "tf_plan"
  dependencies:
    - job-validate
  artifacts:
    paths:
      - tf_plan

job-apply:
  stage: apply
  script:
    - terraform apply "tf_plan"
  dependencies:
    - job-plan
  allow_failure: true

job-destroy:
  stage: destroy
  script:
    - terraform destroy  -auto-approve
  dependencies:
    #- job-plan
    #- job-apply
  when: manual

Push the code back to Gitlab (git add, git commit, git push) and once everything completes successfully, you’ll have a working EKS cluster after 10-15 mins.
If you choose to manage the EKS from GitLab, you can skip to the next section. If you decide to manage the cluster outside GitLab and you already have another IAM user added to the role rolGitLab as described above, then do the following. Replace account # in line 1.

aws eks update-kubeconfig --region us-east-2 --name eksFromGitLab --role-arn arn:aws:iam::123456789012:role/rolGitLab
aws eks list-clusters
eksctl get cluster --name=eksFromGitLab --region us-east-2
kubectl get svc

Manage EKS from GitLab

Generally, you would want this to be in another project. Our first project was to build the cluster, but our second project builds on top of that and manages the cluster. In addition, the second project also uses the CI/CD pipeline file .gitlab-ci.yml so in order to avoid a single complex project – just create a new project. I’ll name my new project – project1. Clone the project on your local machine using git clone.
Create an empty config file for the agent on your local machine. It has to be under this folder structure.

mkdir -p .gitlab/agents/eks-agent && touch .gitlab/agents/eks-agent/config.yaml

In my case the name of the agent is eks-agent. Use whatever name you want but follow DNS naming convention.
Push the local changes to GitLab.
On the left side in GitLab, go to Operate | Kubernetes clusters and then click Connect a cluster from the upper right corner.
Select the agent from the drop down and click Register.

You’ll get an access token and a bunch of helm commands to install the agent on the cluster. Copy and paste and run those commands.

helm repo add gitlab https://charts.gitlab.io
helm repo update
helm upgrade --install eks-agent gitlab/gitlab-agent \
    --namespace gitlab-agent-eks-agent \
    --create-namespace \
    --set image.tag=v16.9.0-rc2 \
    --set config.token=glagent-2r_uW63e1w_mkNzR_keU2hmaNhhcKauSpujg8OdLQG74wrF73g \
    --set config.kasAddress=wss://kas.gitlab.com

Refresh the GitLab screen after the helm chart has been installed and you’ll see your cluster connected.

Now you need to authorize the agent to access our project.
In the blank config.yaml file on the local machine under .gitlab/agents/eks-agent/ directory, add these lines. Change the path the project in line 3 (look at the URL, that’s your project name after gitlab.com) and change the manifest file if you want in line 6. You can also change the namespace in line 5 as well.

gitops:
  manifest_projects:
  - id: andreevkliment/project1
    default_namespace: default
    paths:
      - glob: 'manifest.yaml'

Then create the manifest file under the root of the project. Save it as manifest.yaml or however you referenced it in line 6 in the previous file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

If you push this code back to GitLab, the agent will execute it and you’ll (with kubect get pods) see the nginx pods running on your cluster.
Another way of managing the EKS cluster is using .gitlab-ci.yml file.
Create .gitlab-ci.yml with these lines. Replace your username, project and the agent name in line 7.

deploy:
  image:
    name: bitnami/kubectl:latest
    entrypoint: ['']
  script:
    - kubectl config get-contexts
    - kubectl config use-context andreevkliment/project1:eks-agent
    - kubectl get pods

If you push this back to GitLab, the pipeline will start and you’ll see the pods from the cluster in the logs. This example doesn’t execute manifest.yaml. You can have both ways of interacting with the cluster or you can use them separately.

AWS: EKS running out of IPs, increase pod density

Kliment Andreev — Tue, 06 Jun 2023 21:04:16 +0000

When I was doing some proof of concept using EKS, I’ve noticed that there is a limitation on how many pods can a node run and how EKS CNI assigns the subnet IPs. Apparently, a node can run a maximum of XX pods (based on type) which is very low, e.g. a t3.large instance with 2 CPUs and 8GB RAM can run only 35 pods. On top of that, the IPs that are assigned for the pods are used from the existing pool of IPs and if you have a /24 subnet for the EKS cluster, you’ll run out of IPs in no time. Here is an example of what I did and options how to solve this issue.

Option 1 – Calico 3rd party CNI

I’ll provision an EKS cluster in a /24 subnet with 3 public and 3 private subnets.

eksctl create cluster --name eksECIC --region us-east-2 --instance-types t3.large \
    --managed --vpc-cidr 192.168.100.0/24 --node-private-networking --version 1.24 --without-nodegroup

I am using the Ohio region which has 3 availability zones. The public and private subnets (total of 6) have /27 subnet which is 32 IPs. Not all of them are usable of course. Let’s create a node group with 3 nodes, and because t3.large is a Nitro instance, we can set the max number of pods to be 110.

eksctl create nodegroup --cluster eksECIC --name old-nodegroup --nodes 3 \
    --node-type t3.large --node-private-networking --managed --max-pods-per-node 200

These are the available 31 IPs that I have in the private subnets.

Let’s provision a small deployment with a single pod and 10 replicas and a network load balancer.

kubectl apply -f demo.yaml

…and this is the deployment file demo.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
spec:
  replicas: 10
  selector:
    matchLabels:
      run: demo
  template:
    metadata:
      labels:
        run: demo
    spec:
      containers:
      - name: demo
        image: klimenta/serverip
        ports:
        - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: loadbalancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
spec:
  ports:
    - port: 80
      targetPort: 3000
      protocol: TCP
  type: LoadBalancer
  selector:
    run: demo

If you look at the available address now, there are 7. So from 31 to 7 with just 10 pods. Check this guy’s blog to see what’s going on.

Let’s scale up the cluster to 100 pods.

kubectl scale --replicas=100 deployment/demo

After a minute or two, check the running pods.

kubectl get pods --field-selector=status.phase=Running | wc -l
69

The console will show 0 available subnets.

If you check the pods with kubectl get pods, you’ll see the remaining containers have a ContainerCreating status.
And if you check one of them, you’ll see this message.

kubectl describe pod demo-698f6fc958-z8smb
....
d9b65ccfeccb7e9e7433ec61fef78faf83d1bae": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  2m29s (x17 over 5m59s)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "edfa987c3eddf415068691f914d740fa9ca9ca6ffdfb8db44d1340d913ece0b0": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container

Get the load balancer service.

kubectl get svc
NAME           TYPE           CLUSTER-IP      EXTERNAL-IP                                                                     PORT(S)        AGE
kubernetes     ClusterIP      10.100.0.1                                                                                443/TCP        37m
loadbalancer   LoadBalancer   10.100.25.241   a105789cd3a264ac596c367fce463640-80d2b88acdc17771.elb.us-east-2.amazonaws.com   80:32396/TCP   17m

If you go to a105789cd3a264…amazonaws.com URL, you’ll see the load balancer hitting different pods. Wait for 2-3 minutes if you see that the page can’t be opened. It takes time for DNS to propagate.

Let’s delete the deployment and the old node group.

kubectl delete -f demo.yaml
eksctl delete nodegroup --name old-nodegroup --cluster eksECIC

At this point, we still have the EKS cluster, the etcd database, networking etc, we just don’t have the nodes. Now it’s time to remove the EKS CNI, install Calico and add nodes that will use Calico.
Installation is very simple. Remove the AWS CNI.

kubectl delete daemonset -n kube-system aws-node

Check the pods in all namespaces.

kubectl get pods --all-namespaces
NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
kube-system   coredns-5c5677bc78-2k88b   0/1     Pending   0          4m
kube-system   coredns-5c5677bc78-2zz2n   0/1     Pending   0          4m

You will see coredns in pending state, it can’t be deployed anywhere.
Deploy Calico.

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.25.1/manifests/tigera-operator.yaml

…keep going…

kubectl create -f - <
Deploy the new node group.
eksctl create nodegroup --cluster eksECIC --name new-nodegroup --nodes 3 \
--node-type t3.large --node-private-networking --managed --max-pods-per-node 200

Let’s deploy the same demo deployment now.
kubectl apply -f demo.yaml

…scale it up to 601 pods.
kubectl scale --replicas=601 deployment/demo

Wait 2-3 mins and check the running ones.
kubectl get pods --field-selector=status.phase=Running | wc -l

There will be 585 pods running, 195 per node. Much, much better. This time the remaining containers will be in Pending state.
kubectl describe pod demo-698f6fc958-smlbq
....
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 3m27s (x3 over 13m) default-scheduler 0/3 nodes are available: 3 Too many pods. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.

Option 2 – CNI Custom networking and prefix assignment mode
In this case, we’ll keep the AWS CNI network plugin, but we’ll use what is called CNI Custom networking, meaning we’ll use another CIDR attached to the VPC where pods will run. NOTE: The nodes will still run in the 192.168/24 subnets, it’s just the pods that will run in a much bigger IP space. In addition, this CNI Custom networking won’t solve the density of the pods that we can run on the nodes. We’ll need something called prefix assignment mode that will allow us to run much more pods on a node than usual. See this and this links for more information. For more info regarding prefix assignment mode check out this link. The solution described below is much more complicated than if you just use Calico.

Let’s provision a cluster with the same VPC CIDR and no nodegroups.
eksctl create cluster --name eksECIC --region us-east-2 --instance-types t3.large \
--managed --vpc-cidr 192.168.100.0/24 --node-private-networking --version 1.24 --without-nodegroup

Add a new CIDR and associate it with the VPC. I am using the console but you can use aws ec2 associate-vpc-cidr-block command.Go to the VPC where the EKS cluster resides. From the Actions button, select Edit CIDRs and then click the Add new IPv4 CIDR button.

I’ll add 100.64.0.0/16 CIDR and then I’ll create 3 /19 subnets.

These 3 subnets will be where the pods will run. I also want them to initiate connections to Internet, so I’ll modify the route table and add the Internet Gateway. The three new subnets 100.64/19 that we created have the same route table.

Just add the route to 0.0.0.0/0 to go over Internet Gateway.

Let’s configure the custom networking on the cluster.
kubectl set env daemonset aws-node -n kube-system AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true

Get the EKS cluster security group.
sec_grp=$(aws eks describe-cluster --name eksECIC --query cluster.resourcesVpcConfig.clusterSecurityGroupId --output text)
echo $sec_grp

For each of the subnets create a custom resource. Define the three 100.64 subnet-ids as variables.
export sub1=subnet-0f3ec5d75caf1b981
export sub2=subnet-0fc39edc5a1ce91c0
export sub3=subnet-0bd6bdfd5543aa086

cat >us-east-2a.yaml <
2nd subnet.
cat >us-east-2b.yaml <
3rd subnet.
cat >us-east-2c.yaml <
NOTE:Make sure you name the metadata in line 5 as I did based on your region and AZ.

Deploy the custom resources.
kubectl apply -f us-east-2a.yaml
kubectl apply -f us-east-2b.yaml
kubectl apply -f us-east-2c.yaml

Confirm it looks good.
kubectl get ENIConfigs
NAME AGE
us-east-2a 16s
us-east-2b 12s
us-east-2c 8s

Update your aws-node DaemonSet to automatically apply the ENIConfig for an Availability Zone to any new Amazon EC2 nodes created in your cluster.
kubectl set env daemonset aws-node -n kube-system ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone

Enable prefix assignment mode. This will allow much more pods per node.
kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true

Another change.
kubectl set env ds aws-node -n kube-system WARM_PREFIX_TARGET=1

With the default setting, WARM_PREFIX_TARGET will allocate one additional complete (/28) prefix even if the existing prefix is used by only one pod. If the ENI does not have enough space to assign a prefix, a new ENI is generated.

Provision the node group.
eksctl create nodegroup --cluster eksECIC --name my-nodegroup --nodes 3 \
--node-type t3.large --node-private-networking --managed --max-pods-per-node 200

Check the nodes, they are running in the 192.168/24 original VPC CIDR. See the IP in the names of the nodes.
kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-100-122.us-east-2.compute.internal Ready 7m39s v1.24.13-eks-0a21954
ip-192-168-100-155.us-east-2.compute.internal Ready 7m39s v1.24.13-eks-0a21954
ip-192-168-100-167.us-east-2.compute.internal Ready 7m37s v1.24.13-eks-0a21954

Check the pods.
kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system aws-node-txwrd 1/1 Running 0 9m4s 192.168.100.167 ip-192-168-100-167.us-east-2.compute.internal
kube-system aws-node-vdmv7 1/1 Running 0 9m6s 192.168.100.155 ip-192-168-100-155.us-east-2.compute.internal
kube-system aws-node-xss7c 1/1 Running 0 9m6s 192.168.100.122 ip-192-168-100-122.us-east-2.compute.internal
kube-system coredns-5c5677bc78-2lkcx 1/1 Running 0 80m 100.64.76.225 ip-192-168-100-122.us-east-2.compute.internal
kube-system coredns-5c5677bc78-mj6wr 1/1 Running 0 80m 100.64.76.224 ip-192-168-100-122.us-east-2.compute.internal
kube-system kube-proxy-lt7v4 1/1 Running 0 9m6s 192.168.100.122 ip-192-168-100-122.us-east-2.compute.internal
kube-system kube-proxy-m8pf6 1/1 Running 0 9m4s 192.168.100.167 ip-192-168-100-167.us-east-2.compute.internal
kube-system kube-proxy-nlvw7 1/1 Running 0 9m6s 192.168.100.155 ip-192-168-100-155.us-east-2.compute.internal

Look at coredns pods. They are running in the new CIDR.

Deploy the demo.
kubectl apply -f demo.yaml

Get the load balancer service.
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.100.0.1 443/TCP 83m
loadbalancer LoadBalancer 10.100.206.187 a7ad3ecf1d52e4fc58dbbdc550237009-ac4778e11af2c0ad.elb.us-east-2.amazonaws.com 80:30230/TCP 45s

If you go to a7ad3ecf1d…amazonaws.com URL, you’ll see the load balancer hitting different pods. Wait for 2-3 minutes if you see that the page can’t be opened. It takes time for DNS to propagate.

The pods are running in the 100.64 subnet.

Scale up to 601 pods.
kubectl scale --replicas=601 deployment/demo

Wait 2-3 mins and check the running ones.
kubectl get pods --field-selector=status.phase=Running | wc -l

I got 593 total pods. Slightly better than 585 with Calico. The ones that are in pending state show up an error that there are no more resources.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 3m49s default-scheduler 0/3 nodes are available: 3 Too many pods. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.

Kubernetes: Monitoring Node.js application with Prometheus and Grafana, Helm charts in AWS EKS

Kliment Andreev — Tue, 24 Jan 2023 16:01:05 +0000

Recently, we were migrating some Node.js applications to a different AWS account and we had a need to monitor the application with Prometheus and Grafana. In this post, I’ll explain how to install both Prometheus and Grafana using Helm charts, then we’ll create a simple Express.js web app, create a Helm chart for it and deploy it on an EKS cluster in AWS. I am using an EKS cluster, but you can use any Kubernetes cluster that you can manage it with kubectl. You might have to make some changes though in how you access the Load Balancer for Grafana.
Requirements:

Create EKS cluster

This is optional, if you already have a running Kubernetes cluster, you can skip this step.

eksctl create cluster --name eksTest \
--region us-east-2 \
--instance-types t3.medium \
--nodes 2 \
--managed \
--version 1.22

This command will create a 2 node cluster, with t3.medium instances in Ohio region using Amazon Linux 2 image. It will take about 15 minutes.
Verify that everything looks OK.

kubectl get nodes
NAME                                           STATUS   ROLES    AGE     VERSION
ip-192-168-10-4.us-east-2.compute.internal     Ready       8m54s   v1.22.15-eks-fb459a0
ip-192-168-74-160.us-east-2.compute.internal   Ready       8m51s   v1.22.15-eks-fb459a0

Install Prometheus and Grafana

Add the Helm repos for Prometheus, Grafana and update any changes.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

We’ll create a separate namespace for Prometheus and install the chart there.

kubectl create namespace prometheus
helm install prometheus prometheus-community/prometheus \
    --namespace prometheus \
    --set alertmanager.persistentVolume.storageClass="gp2" \
    --set server.persistentVolume.storageClass="gp2"

Create this file. It tells Grafana what to use as a source (url). It’s the Prometheus endpoint service that we just installed.

cat << EOF > grafana.yaml
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-server.prometheus.svc.cluster.local
      access: proxy
      isDefault: true
EOF

Now, let’s create a namespace for Grafana and then install Grafana in that namespace. Change the admin password. In my case it’s Password1$.

kubectl create namespace grafana
helm install grafana grafana/grafana \
    --namespace grafana \
    --set persistence.storageClassName="gp2" \
    --set persistence.enabled=true \
    --set adminPassword='Password1$' \
    --values grafana.yaml \
    --set service.type=LoadBalancer

You’ll get something like this on the screen when the install ends.

export SERVICE_IP=$(kubectl get svc --namespace grafana grafana -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

That’s your load balancer IP so you can access Grafana. If you use EKS, your IP is different.

kubectl get services -n grafana
NAME      TYPE           CLUSTER-IP     EXTERNAL-IP                                                               PORT(S)        AGE
grafana   LoadBalancer   10.100.20.53   a074996f790b74793a74bec584c04460-2034633908.us-east-2.elb.amazonaws.com   80:31916/TCP   3m21s

The IP is the gibberish URL that ends with amazonaws.com. If you go to that URL, you can log as admin and your password.

Go to Dashboards, click on the New button on the right and click Import.
Add the following dashboard IDs (one by one then click Load) and choose Prometheus as a source. The IDs are 3119 (Kubernetes cluster), 6417 (pods) and 11159 (Node.js). For the first dashboard, you’ll have some metrics. The Node.js dashboard is empty and our goal is to get the metrics from the application.

Node.js application

Create an empty folder and add this file.

mkdir mynodeapp
cd mynodeapp
cat << EOF > package.json
{
  "name": "mynodeapp",
  "version": "1.0.0",
  "description": "Node.js on Docker",
  "author": "Kliment Andreev ",
  "main": "server.js",
  "scripts": {
    "start": "node server.js"
  },
  "dependencies": {
    "express": "^4.16.1",
    "prom-client": "^14.1.1"
  }
}
EOF

I use Node v18.12. Install the dependencies.

npm install

This is the simple web application. It listens on port 3000.

cat << "EOF" > server.js
'use strict'
const express = require('express')
const prom = require('prom-client')

const collectDefaultMetrics = prom.collectDefaultMetrics();

const app = express()
const port = process.env.PORT || 3000

app.get('/', (req, res, next) => {
  res.send('Hello World!');
});

app.get('/metrics', async (req, res) => {
  try {
	  res.set('Content-Type', prom.register.contentType);
	  res.end(await prom.register.metrics());
  } 
  catch (ex) {
	  res.status(500).end(ex);
  }
});

const server = app.listen(port, () => {
  console.log(`mynodeapp listening on port ${port}!`)
})
EOF

The app is a super simple web app that prints Hello World! The important part is the /metrics router. That’s how you get the metrics from the app. When we created the package.json file, we specify the prom-client as dependency. That’s the part that collects the metrics. Go to this link for more info.
Start the app with node server.js.

node server.js
mynodeapp listening on port 3000!

Go to the localhost or the server IP where this app is running and you’ll see Hello World! printed out.

And if you go to the /metrics URL, you’ll see this text. That’s what Prometheus expects.

Now that we know the app is working, let’s create a Docker image.

Docker image and container

Create this Dockerfile.

cat << EOF > Dockerfile
FROM node:18-alpine

# Create app directory
WORKDIR /app

# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm@5+)
COPY package*.json ./

RUN npm install
# If you are building your code for production
# RUN npm install --only=production

# Bundle app source
COPY server.js /app

EXPOSE 3000
CMD [ "npm", "start", "server.js" ]
EOF

Create a Docker image based on the Dockerfile above. Replace the username with your Docker Hub username.

docker build -t /mynodeapp:latest .

Run the container from the image.

docker run -p 3000:3000 /mynodeapp:latest

> mynodeapp@1.0.0 start
> node server.js server.js

mynodeapp listening on port 3000!

Use the same test as before by going to the localhost or IP URL on port 3000.
If everything is OK, publish the image on Dockerhub.

docker login
docker push /mynodeapp

Helm chart

Now that we have the Docker image, we’ll create a Helm chart so we can deploy the web app to the Kubernetes cluster.
Create the necessary structure.

helm create mynodeapp
Creating mynodeapp

Go to the sub-directory mynodeapp and edit the values.yaml file first. Look for the image key first and replace it so it looks like this. Make sure you replace the with your Docker Hub username.

image:
  repository: /mynodeapp
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: "latest"

Then look for podAnnotations: {} line and replace it so it looks like this.

podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/path: "/metrics"
  prometheus.io/port: "3000"

Look for the service parameter and change it so it looks like this.

service:
  type: LoadBalancer
  port: 80
  targetPort: 3000
  name: mynodeapp-service

Look for the resources key and uncomment the defaults.

resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
limits:
  cpu: 100m
  memory: 128Mi
requests:
  cpu: 100m
  memory: 128Mi

Edit the templates/deployment.yaml file and find these lines.

ports:
  - name: http
    containerPort: {{ .Values.service.port }}

Replace the containerPort to look like this.

containerPort: 3000

Go back to the root of the mynodeapp folder and check the chart.

helm install mynodeapp --generate-name
NAME: mynodeapp-1674575241
LAST DEPLOYED: Tue Jan 24 10:47:22 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
     NOTE: It may take a few minutes for the LoadBalancer IP to be available.
           You can watch the status of by running 'kubectl get --namespace default svc -w mynodeapp-1674575241'
  export SERVICE_IP=$(kubectl get svc --namespace default mynodeapp-1674575241 --template "{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}")
  echo http://$SERVICE_IP:80

Execute the export SERVICE_IP command above and echo the IP. That’s your URL. If you go to that URL you’ll see the Hello World! greetings.
What we care is the pod. Get the pod running. It’s in the default namespace.

kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
mynodeapp-1674575241-5ff9d7469d-pddcz   1/1     Running   0          3m

Get the annotations. Replace with your pod name.

kubectl get pod mynodeapp-1674575241-5ff9d7469d-pddcz -o jsonpath='{.metadata.annotations}'
{"kubernetes.io/psp":"eks.privileged","prometheus.io/path":"/metrics","prometheus.io/port":"3000","prometheus.io/scrape":"true"}

You can see the prometheus annotations.
And if you go to Grafana and open up the NodeJS dash you’ll see the metrics.

AWS: Web Servers in HA config behind Application Load Balancer and Web Application Firewall (WAF)

Kliment Andreev — Mon, 06 Dec 2021 14:19:04 +0000

As the title says, I’ll create two web servers in a private subnet, put an Application Load Balancer in front and protect the content with Web Application Firewall (WAF). In addition, I’ll create a bastion host so I can access the web servers using SSH. Web Servers are not reachable from the Internet.

Elastic IP

First thing to do is to create an Elastic IP for the NAT gateway. NAT gateway is needed if you want to give the Web Servers Internet access so they can be patched. Go to Elastic IPs under Network & Security in EC2 menu and click on Allocate Elastic IP address in the top right corner. Accept the defaults and click Allocate.

VPC

We’ll create a VPC with two public and two private subnets. Minimum two public subnets are needed for the Application Load Balancer. You can’t create an ALB with one public subnet.
Click on Launch VPC Wizard in the VPC menu and select VPC with Public and Private Subnets. I’ll use this CIDR (192.168.100.0/22) which gives me 1024 hosts enough for 4 Class C /24 subnets, you can use whatever CIDR you want. As you can see I’ll use 192.168.100.0/24 for the first public subnet and 192.168.102.0/24 for the first private subnet. I’ll create the other public and private subnet later. You’ll have to associate the Elastic IP that you created in the previous step to the NAT gateway. Click Create VPC in the bottom right corner.

Go to Subnets menu click on the public subnet and look at the Availability Zone.

When you create the next subnet, you have to put it on another zone.
Click on the Route table tab. Remember/write-down the route table id (rtb-0e3cbsomething). It will be different for you.

Then click on Create subnet in the upper right corner. Choose your VPC, name the subnet (in my case subPublic2), assign an Availability Zone (e.g. us-east-1a) and add the CIDR 192.168.101.0/24. Make sure that your 2nd public Availability Zone is different than the 1st Availability Zone. Click Create subnet.
If you click on the new subnet now and then click on the Route table tab, you’ll see that AWS created a route table for you. But as you can notice below, the destination 0.0.0.0/0 goes through the NAT gateway.

We don’t want that. We want the public subnet to use the Internet gateway, not the NAT gateway. So click on the Edit route table association on the right. From the drop down choose the other route table, that’s the Internet gateway, that’s the one that you have wrote down earlier. When you change to the other route table, you’ll see that the destination for 0.0.0.0/0 changes to igw which is the Internet gateway. Click Save.

Now, do the same for the other private subnet. Click on the private subnet and look at the Availability Zone. In my case it’s us-east-1b. This means, I have to create my 2nd private subnet in another zone.

So, I am creating the 2nd private subnet within my VPC and I named my subnet subPvt2 in us-east-1a and 192.168.103.0/24 CIDR.

Check the route table for this new private subnet. The route table for 0.0.0.0/0 should be the NAT gateway which is correct, but just in case…
So these are my subnets now.

Bastion host

You use a bastion host as a jump server to get to the web servers. If you have any other means to get to them such as VPN or Systems Manager, then go for it. This is just an example of how to get to the web servers.
I’ll use the AWS Linux for this. It really doesn’t matter as long you have a SSH client installed on the bastion host. Launch a simple t2.micro instance in any public subnet. Make sure the bastion host has a public IP. Also make sure that this bastion host is accessible from a single IP or some dedicated subnet. You don’t want this bastion host to be exposed to the world. In my case, the security group for the bastion host is restricted to my home IP only on port 22 for SSH.

Web Servers

I’ll use two AWS Linux instances deployed in the private subnets. Do not assign a public IP for these. That’s what the bastion host is for. Create a security group that allows traffic for SSH from the security group that you created for the bastion host. In my case it looks like this. Start typing sg in the circled field and choose the security group that you create earlier. If you named it properly, you’ll see it. If not, go get the security group ID for the bastion host.

Launch two instances in two different private subnets.

Test connection

Log to the bastion host using the key you created. Now, we’ll connect to the web servers from the bastion host. If you used the same key for bastion host and web servers, copy and paste the PEM certificate in a file called key.pem or whatever. If you used different keys, it doesn’t matter, just make sure you have the keys as a file on the bastion host. I used the same key, which is not the best practice, but for the sake of simplicity, I have a file called key.pem on the bastion host. Log to each web server using ssh ec2-user@192.168.102.176 -i key.pem where ec2-user is the user for the AWS Linux (it differs for different Linux distros), 192.168.102.176 is the IP of the first web server and the parameter -i tells which key file certificate to use. Make sure you do chmod 600 key.pem so you can log in. You should be able to log in to both web servers.

Install Apache web server

Log to each web server and execute these commands to install Apache web server, start it and enable to start on boot.

sudo yum -y install httpd
sudo systemctl start httpd
sudo systemctl enable httpd

Switch to root.

sudo su

Then on each server do this. Do Server 2 on the second server. This is just an example. You need the same code on both web servers. The difference is because we’ll test the ALB later and see how the page changes.

echo "Server 1" > /var/www/html/index.html

For the 2nd server.

echo "Server 2" > /var/www/html/index.html

Application Load Balancer

Now, we are moving to the ALB. Before we create the ALB, let’s create a security group that will allow port 80 from everywhere. Looks like this. Create the security group from the EC2 menu. This is a security group that we’ll assign to the load balancer.

From the EC2 menu on the left side, go to Load Balancing and click on Load Balancers. Click on Create Load Balancer. Choose Application Load Balancer. Make sure you choose the public subnets and for the security group use the one that we created earlier for the load balancer.

If you scroll down, you’ll see that you need to create a target group for the ALB. The target is our web servers. Click on Create target group. It will open up a new browser window. Choose Instances, name the target group, leave HTTP1 for protocol and for the Health checks type /index.html. Remember, we created these files earlier.

Click Next, select the web servers and click on Include as pending below.

Finally, click on Create target group.
Go back to the original tab when you created the load balancer, click on the refresh button and choose the target group you just created.

Scroll down and click on Create load balancer.
Wait for a couple minutes until you see that the state changes to Active. Click Refresh on the right side.

While you are waiting, if you click on the Description tab for the load balancer and scroll a little bit down, you’ll see the security group that we create for the load balancer.

We need to add this security group to the web servers. Right now, the web servers are configured to accept traffic from the bastion host only, but we need for the web servers to accept HTTP traffic as well. Now, the security group for the load balancer allows HTTP from everywhere, but that doesn’t matter. When we add this security group to the web servers they’ll also allow HTTP request from everywhere. But the web servers have no public access so it really doesn’t matter. Does this mean that any other server in these public/private subnets will have access to HTTP to the web servers? Yes. If I create another server in the private/public subnet, it will have access to HTTP to the web servers.
Let’s add the ALB security group to the instances. In my case it looks like this where the bottom security group is the security group for allowing HTTP from everywhere, the one that we assigned to the ALB.

Before you do anything, go to the Target Groups menu under Load Balancing. You should see both instances as healthy.

If this looks good, go back to the load balancer and get the URL.

Open up a new browser window and go to that URL. Click refresh in your browser several times. You’ll see how it changes from Server 1 to Server 2.

If everything is OK, proceed further.

Web Application Firewall

The good news is that you can directly create your rules from the ALB. Click on the load balancer you created and click on the Integrated services tab. Click on Create Web ACL.

Name the ACL and click on Add AWS resources where we gonna add the ALB.

Here we are. Click Next after.

And now you are on your own. You have to add rules. There are some that are templated and there are some that you can buy from the AWS Marketplace.

Once you add these rules, you can see what’s blocked and what’s not under Cloud Watch | All metrics | WAFV2.

Azure, AWS: Use Azure AD with AWS SSO

Kliment Andreev — Sat, 12 Jun 2021 16:14:21 +0000

There is a newer version of this post but feel free to refer to this one in case you have any issues.
In this post, I’ll explain how to configure both Azure and AWS so when you log to AWS you’ll use your Azure AD credentials. In my lab, I have an Azure subscription and Azure AD (free tier). I also have a multi account AWS in AWS Organization structure. I want to be able to log to both Azure and AWS using the same Azure AD credentials. This way, I can use only Azure AD to control access to my cloud resources.
First, we’ll configure AWS for SSO. Log to your management/root account in AWS. Type sso in the search bar and choose Azure Single Sign-On.

Click on the button to Enable AWS SSO.

From the Dashboard, Choose your identity source.

You’ll be taken to the Settings pane. Click on Change under Identity source.

Select External identity provider and click on Download metadata file.

In Azure, go to Azure Active Directory and from the left, click on Enterprise applications.

Click on New application.

Search for AWS Single and choose AWS Single Sign-on. Make sure you choose this one if you have multi accounts in AWS. The setup is slightly different for the other single account access.

Click on the Create button on the right and wait for about 30 seconds. When this screen shows up, click on Single sign-on.

Choose SAML from the choices.

When this screen shows up, choose Upload metadata file.

Select the file that you just downloaded from the last step you did previously in AWS. You don’t have to do anything, just click Save and then close the blade on the right.

You’ll get something like this, just click there as instructed to fix it.

If you are asked if you want to test the SSO, just ignore the message and click on, No, I’ll test later.

Scroll down a bit to step 4 and click on Install the extension. This is the easiest way to do this. You can go the manual way, but it’s not worth it. Don’t worry, you can always remove the extension after. It’s for a one time setup. I use Firefox and it works fine, which means it will definitely work on Chrome and Edge.

Once you install the extension, click on it in the upper right corner.

Click on Sign in to get started button.

You should be automatically signed if you have the Azure session open in another tab. You’ll see something like this if everything is OK.

Click on the button Set up AWS Single Sign-on under step 5. Azure will prompt you to save the federation XML file and just click OK for the Azure prompt.

Azure will open a new AWS tab and ask you to confirm if it’s OK to make changes in AWS. Click OK.

You’ll see some action on the screen like someone clicks in your browser. Don’t do anything. Just click Yes or OK when asked.

The extension will ask you to upload the federation XML file from Azure that you saved earlier and then it will reconfigure AWS and you should see this at the end.

You’ll end up at the initial screen for step 1 in Azure. Just click Save and you are OK.
Now, if you don’t have any users or groups in Azure AD, go create some. In my case, I use free Azure AD so I am not gonna be able to use groups. It requires Azure AD P1 which is not free. But, I’ll show you how to assign a user in Azure AD. If you use Azure AD Connect to sync your on-prem AD with your Azure AD, you can choose that user as well. With your AWS Single Sign-on app selected, click on Users and groups.

Click on Add user/group.

As I mentioned earlier, I can’y choose groups because of my Azure AD free tier, so click on None Selected.

Select the user and just click Assign at the bottom.

I’ve assigned this guy from Azure AD (awsadmin). Just click on Provisioning on the left. We have to sync these users to AWS now.

Click on Get started and choose Automatic.

You’ll have to provide a Tenant URL and a Secret Token.
Go back to AWS, then SSO, click on Settings and click on Enable automatic provisioning under Provisioning.

Click on Show token. Copy both values on the clipboard, those are the Tenant URL and the token needed for Azure. Copy and paste them in Azure.

Click Test Connection and make sure you get OK as a result. Click Save.

You’ll be back here. Click on Edit attribute mappings.

Change the Provisioning status to On and click Save.

If you close that blade, you’ll see that one user synced successfully.

Go to AWS and under Users you’ll see the same user. Yay!

OK, if you look at the User portal, you’ll see the URL for your Single Sign-on. You definitely want to customize the URL to suit your needs.

Go to that URL, but use a separate browser or some private mode, so you don’t affect any open sessions. You’ll be redirected and prompted to log with your Azure AD. You might get a prompt to configure MFA, skip it for now, you can always configure it later.
This is what you’ll see.

But we assigned AWS app in Azure to this user. What’s going on?

Just go to AWS | SSO | AWS Accounts, select the accounts that you assign to a user and click Assign users.

Select the user (in my case awsadmin) and click Permission sets.

Click on Create new permission set. This will open a new tab.

This is where you assign permission to that user or group (if you have license). Use the second option for custom policies, e.g. access to only a set of S3 buckets or assign tagging policies only. I’ll use the first option for the pre-made policies. Click on Details.

I’ll use the Administrator access. Click Tags to proceed.

Finish the rest and go back to the first AWS tab. On the right click the Refresh button.

The permission set should show up. Select and click Finish.

If the permission set doesn’t show up, don’t worry. Repeat the same same procedure from the AWS accounts menu to select accounts and users and you’ll see that this time the permission set will show up.
Finally, go to the same URL where nothing showed up when we logged with that awsadmin user. Click Refresh in the browser and you’ll see the accounts now.

Click to log to the Management Console and you are all set!

AWS: DataSync using VPN and Internet

Kliment Andreev — Sun, 04 Oct 2020 14:54:06 +0000

In this post I’ll explain how to use AWS DataSync service. Per AWS, AWS DataSync makes it simple and fast to move large amounts of data online between on-premises storage and Amazon S3, Amazon Elastic File System (Amazon EFS), or Amazon FSx for Windows File Server.
You can used this service when you migrate your on-prem shares to Win FSx or EFS. In my case, I’ll migrate a Windows share that I have at home to an S3 bucket. In the first scenario, I’ll use the straight public Internet to migrate data and the second scenario, I’ll use a VPN connection. The config for VPN transfer is pretty much what you’ll have if you use Direct Connect instead. In both cases you’ll need a VM instance on-prem. I have a vCenter running, so I’ll download an OVA file from AWS, but they also offer VMs for KVM, Hyper-V even EC2.

Install a VM

No matter if you use public Internet or VPN/Direct Connect, you’ll need an on-prem VM that will act as a proxy between your on-prem data and AWS. There is no need to install any agents on any servers. For example, I’ll have a Windows share and the only thing this VM requires is access to that share with a domain or a workgroup user. Go to AWS DataSync service and click to download the image for your hypervisor.

The OVA file is about 600+ MBs compressed. Once downloaded, deploy it as with any OVA file. Specify the details for your vCenter, network, host, cluster… By default, the VM needs 80GB disk and eats up 4CPUs and 16GB RAM. Once deployed, open the console, log with admin/password which is admin and password and change the IP to static if you need and/or change the default password or whatever you want. The console looks like this, some modified Linux distro.

AWS DataSync over public Internet

Now, go back to AWS DataSync and enter the IP of the appliance. But before you click Get key, make sure you have selected the public service endpoint for the Service endpoint as in the screenshot below. This means we’ll use public Internet. No, worries, the traffic is encrypted. Click Get key.

Probably you wonder, wait – how can AWS access my internal IP when I don’t have any VPN or Direct Connect. Well, AWS doesn’t care about your internal IP. Your browser have access to the internal IP on port 80 to the VM appliance. AWS doesn’t initiate the traffic, the appliance does.
So, you have the appliance activated now. Name the agent and tag it if you want. Click Create agent.

Creating a task

You’ll see you agent online. Click on the Agent ID.

Now, you have to create a task. For each share, you need a task. Click on Create task.

Configure the source location on-prem.

I chose SMB which is a Windows share. Specify the agent, the server IP, the share name with “/” prefix and the username, password, domain (optional) in case the share is restricted, which probably is.

Click Next. Now, choose a destination. I am using some S3 bucket and choose to Autogenerate a role that will allow access to that bucket. Click Next again.

On this screen, you can choose options that suit your needs. If you scroll down, you can enable detailed logging in CloudWatch. Click Next.

Check if everything is OK and click to Create task.
You’ll see the task creating. It takes about 10-15 secs.

Click on Start in the upper right corner.

I put a single 10KB file in my share. After some init work, it took about… maybe 2-3 mins and the task status said “available”. I checked the bucket and it was there. Most likely, if you are migrating a share that’s changing on a daily basis, you want to schedule the transfer. If you edit the task, you can change the schedule.

AWS DataSync over VPN and Direct Connect

For DataSync to work over VPN and Direct Connect, you need a new endpoint and a new appliance. You can’t use the existing agent and appliance that was configured for public Internet. The endpoint allows you to privately connect to an AWS service. The traffic won’t go over direct Internet. Actually, with VPN it will, but the traffic will be tunneled. With Direct Connect, the traffic from on-prem to your endpoint is going over dedicated links.
Follow the first section above on how to deploy a new OVA appliance. Once completed, we need to create an endpoint now. But before doing that, create a new security group that allows TCP ports 22, 443 and 1024-1064 range from your appliance.

In AWS DataSync console, click to create a new agent, but this time choose VPC endpoints using PrivateLink and then click on Create VPC endpoint.

I’ll create a VPC endpoint for DataSync in a VPC that’s connected to my VPN. I have only one public subnet there and I’ll choose a security group that allows TCP 443, 1024–1064, and port 22.

Click on Create endpoint. It takes about 3-5 minutes for the endpoint to be created.
Go back to DataSync screen and you’ll see the parameters populated. Enter the IP of the VM appliance and click on Get Key.

Once you see the appliance activated, you can create the agent and follow the instructions in Creating a task section above.
NOTE: I created a task to transfer a 2GB file from my share to S3. When I checked my pfSense IPSec tunnel, I saw these values, which means that the transfer goes over the private link.