Here is the deal. Our vendor wants to access our Aurora database. We can’t do VPC peering because our CIDRs overlap. On top of that, we don’t want peering with 3rd party vendors and we don’t want to host our data in the vendor’s account. So, we have to use PrivateLink.
In this post I’ll guide you through creating an RDS database in one account and accessing it from another account over PrivateLink. In order to do this, we’ll need:

– Two AWS accounts (Provider and Customer). We are the provider, the vendor is the customer.
– One VPC with 2 public/private subnets in each account. The vendor’s (customer) VPC config doesn’t really matter.
– One EC2 instance in each account (public subnet) for testing access

We will create an endpoint service in the Provider VPC and an endpoint in the Customer VPC. The endpoint service requires a network load balancer as well. Here is a problem. When we create a NLB, the target of the NLB are IP addresses. The RDS database has DNS entries as endpoints. Technically, we can resolve the DNS URLs in IPs and use them as targets. But these IPs can change. So we have to put something in between the RDS and NLB. That’s the job of the RDS proxy. The DNS URLs of an RDS Proxy always resolve to the same IPs. Another problem is that RDS proxy can be deployed for MySQL and PostgreSQL databases only. So keep this in mind.

VPC

I’ll create two VPCs in each account. The first VPC will have a 192.168.100.0/22 CIDR and the second VPC will have 192.168.200.0/22 CIDR with 4 x /24 private and public subnets. Here is the breakdown of the subnets. It’s a standard build with two NAT gateways and one Internet gateway.
The first VPC is called vpc-Provider.

The second VPC is called vpc-Consumer.

Security groups

Create two security groups under vpc-Provider VPC, one called sgEC2 and the other called sgRDS.
For the sgEC2 group, allow SSH from some IP, e.g. your IP from work/home. The EC2 instance that we’ll create will be in a public subnet.
For the sgRDS group, allow port 3306 (MySQL) from the actual subnet ID for sgEC2 group and another rule to allow access to 3306 from itself. This is how the rules for sgRDS look like.

Now, do the same for the vpc-Consumer account, but create only the sgEC2 group.

RDS

For the RDS, use pretty much default settings. In my case, I used:

– Standard create
– Amazon Aurora
– Amazon Aurora MySQL-Compatible Edition
– Capacity type: Provisioned
– Aurora (MySQL 5.7) 2.07.2
– Templates: Dev/Test
– Name the cluster and enter your admin credentials (remember these)
– Burstable db.t3.small
– Don’t create an Aurora Replica
– Choose the VPC that we created
– Public access: No
– Security group: sgRDS
– Availability Zone: No preference

Then click to Create database.

EC2

While the database is creating, go to EC2 and create an instance that we’ll use for testing. It can be any Linux and any size. I’ve used t2.micro Amazon Linux with telnet/MySQL client installed (sudo yum install telnet mysql) in a public subnet. I’ve named the instance ec2Provider. Create another instance in the second account with the same specs as ec2Provider and name it ec2Customer.

Test 1

Go to the RDS and get the DNS record for the Aurora cluster. It’s at the bottom.

Log to the instance in the same account (ec2Provider) and try to telnet to that port. You should be able to. CTRL-C to get out of there and log to MySQL using the client.

mysql -h database-1.cluster-cdbzhqzxu6yp.us-east-2.rds.amazonaws.com -u admin -p

Replace the hostname with yours. This the output of my connection.

I’ll create a database while I am there.

create database test_db;

AWS Secrets Manager

In order to deploy an RDS proxy, we need a secret (user/password combo) stored in the AWS Secrets Manager. Use the same username and password that you used for the RDS database, which is admin and whatever password you used. Make sure you select the database that is listed at the bottom of the first screen.

Name the secret and proceed with the defaults.

RDS Proxy

Go to the RDS menu and select and click on the database cluster. Scroll all the way down to the RDS Proxy section. Click on Create proxy.

– Name the proxy (Proxy identifier)
– Engine – MySQL
– Choose the database
– Uncheck the Add reader point.
– Choose the secret that you created before and specify two private subnets
– Leave AWS to create the IAM role.
– Under Additional connectivity configuration make sure you assign the sgRDS security group and remove the default.

Click on Create proxy.

Test 2

Let’s try to access our database through the proxy now. In the main menu for the RDS, click on the Proxies on the left.

Click on the proxy and scroll down to see the proxy endpoint. Copy that value.

Go to the EC2 instance in the vpc-Provider, which is in the same account as the RDS and try to access the database over proxy.

Great job if you made it so far!

Network Load Balancer (NLB)

OK, so if everything works, we need to create an NLB so we can create the endpoint service. The NLB that we’ll create requires the IP targets of the RDS proxy. You got the DNS for the RDS proxy, the one that you just tested right above. Do an nslookup on it and get the IPs.

In my case the proxy RDS returns 192.168.103.210 and 192.168.102.160, which tells me they are in the private availability zones (AZs). Good.
Create an NLB, name it and make sure it’s internal.

Choose the VPC and the two internal subnets.

When you reach the Listeners and routing, change the port to 3306 and click on Create target group.

Choose IP addresses, name the Target group name and change the port to 3306. The correct VPC should be selected.

Leave the health checks, they are automatically populated for the right port and click Next.
On the next screen below, enter the IPs that you got from the nslookup for the RDS proxy. In my case it looks like this. Make sure the port is 3306 and click on Include as pending below and finally click on Create target group.

Go back to the original browser tab where you create the NLB and click the refresh button. You’ll be able to select the target group now.

Click on Create load balancer at the bottom. It takes a couple of minutes to provision the NLB. Once done, click on the Listeners tab and then the target group.

It will take another couple of minutes, but you should see that the targets are unhealthy. Why is that? Because our RDS proxy can get requests from the EC2 instances only. Thе NLB can’t access the RDS proxy. Per AWS, we have to allow the CIDRs of the NLB to access the RDS proxy. That’s easy. We created an internal NLB with 192.168.102.0/24 and 192.168.103.0/24 subnets, so we have to modify the sgRDS security group. Do the same. This is how my security group sgRDS looks like now.

And if you check the target group, you should see both RDS proxy instance as healthy.

Test 3

Let’s check the access on the NLB. Get the DNS name. In my case it’s rds-nlb-40d1b9bfc867b2fb.elb.us-east-2.amazonaws.com.
Go back to the EC2 instance in the same account and try to access the database.

Yep, we are doing great. Let’s recap. We can’t access the RDS directly from another account because – well, it’s another account that we don’t trust and our RDS DB is in a private subnet. So, we have to create an endpoint service that requires a network load balancer. But, the NLB requires IP targets and our RDS endpoints are DNS names. Which means that the DNS names can change its IPs, so we used an RDS proxy DNS names converted to IPs which AWS claims that they won’t change.

Endpoint service

The Provider (we) creates an Endpoint service, the Customer (the vendor) creates the Endpoint. Simple as that. It doesn’t have to be a vendor. Maybe the proper term is the Consumer. So, we (the provider) expose some service as Endpoint Service and someone else tries to access it.
An Endpoint service requires a network load balancer. We did that. So let’s create the endpoint service. It’s under the VPC menu.

Click on Endpoint Services and then Create endpoint service.
Name the endpoint service and check the network load balancer. Make sure that the Acceptance required is checked.

Once created (in a second), get the service name. Copy that URL highlighted.

Select the endpoint service and from the Actions menu, choose Allow principals. Add * for ARN which means everyone. Now this is really bad. For the sake of simplicity, I’ve added * (everyone) but you want to add an ARN role from the other account.

Endpoint

The endpoint is the Consumer, the vendor or some other VPC. Go to that account. We are no longer working on the original account, log to the second account and go to the Endpoint under VPC menu. Click on Create endpoint. Enter the name for endpoint, click on Other endpoint services and paste the endpoint that you got in the previous step. Click on Verify service and you should see that it’s OK.

Scroll down and choose the VPC where you want to create the endpoint. In our case, we want to use the EC2 instance that’s in the public subnet. Add the two public subnets.

And for the security group, add the EC2 security group.

NOTE: If you go to the Interfaces section under EC2, you’ll see three NICs. One is for the EC2 instance that we’ve created, the other two are the endpoints.

Anyway, once you create the endpoint, you’ll see this.

Go to the other account (Provider) and under the endpoint services (VPC menu, Endpoint connections tab), you’ll see a request for the connection. That’s what Accepance required means when we created the Endpoint Service. Accept the connection. It takes about 30 seconds to complete the acceptance.

Test 4

Go to the Consumer account and click on Endpoint and then select the endpoint. Under the Details tab you’ll see three DNS entries.

Copy one of them and log to the EC2 instance in that account and try to access the RDS.

mysql -h vpce-08e3fbbe49e5f241b-q4e8rrlt.vpce-svc-01cc20f911a4789ff.us-east-2.vpce.amazonaws.com -u admin -p

You won’t be able to do so. Finally, edit the security group sgEC2 that we assigned to the endpoint and add a reference to itself for port 3306. It looks like this.

And if you try to access the RDS now, you’ll be able to do so.

Endpoint service

You can also go the the Provider account and delete the principal under Allowed principals.

It’s not needed anymore.

I’ve started playing with Terraform in order to automate some server builds and found it to be very useful. It has its own quirks, strange JSON-like syntax and it takes some time to get fully on-board. I’ve noticed several bugs but there is always a workaround if you google it. At the time of this writing, Terraform is version 0.11.7 so it’s still some kind of beta. The team behind it are making a lot of changes so expect some things to break. The book that I was using to study was already out of date and some commands were deprecated.
I won’t explain how to install it and configure it. It’s very easy to get it up and running. In order for the following script to work, make sure that you have Terraform and AWS CLI installed and configured.
Create a separate folder/directory for this project and create three files. The first file is vars.tf. That’s where we define the variables.

variable "region" {
	description = "The region for the deployment"
	default = "us-west-2"
}
	
variable "vpc_id" {
	description = "The VPC ID where WordPress will reside"
	default = "vpc-a36ga9zd"
}

variable "ami_id" {
	description = "The AMI ID for AWS Linux 2 in us-west-2. In other regions, the ID is different"
	default = "ami-a9d09ed1"

}

variable "instance_type" {
	description = "AWS Instance type to be used for the WordPress instance"
	default = "t2.small"
}

variable "volume_size" {
	description = "EBS volume size in GBs for the instance"
	default = 8
}

variable "key_name" {
	description = "The key pair that will be used to log to the server using SSH"
	default = "MyKeyPair-Oregon"
}

variable "ssh_port" {
	description = "The SSH port for the server"
	default = 22
}

variable "http_port" {
	description = "The HTTP port for the server"
	default = 80
}

variable "mysql_port" {
	description = "The MySQL port for the database"
	default = 3306
}

variable "nfs_port" {
	description = "The NFS port for the shared filesystem"
	default = 2049
}

variable "allocated_storage" {
	description = "The size in GBs of the SQL database"
	default = 10
}

variable "instance_class" {
	description = "The size/type of the SQL instance"
	default = "db.t2.micro"
}

variable "db_admin" {
	description = "The dbadmin username"
	default = "dbadmin"
}

variable "db_password" {
	description = "The dbadmin password"
	default = "SuperSecret"
}

variable "db_name" {
	description = "The database name"
	default = "dbwordpress"
}

Make sure you specify your region, your VPC ID and the key name. That’s how you will login to your instance. You can find your key name if you go to Key Pairs menu in AWS console.

I tried to test the same setup on CentOS 7.x but for some reason the user data boot script was never executed, thus making this script unusable. It might work with Ubuntu or some other flavor of Linux, if you can make the user data script execute on boot and make some modifications, e.g. apt-get instead of yum etc…
The next file is outputs.tf. These are the variables that are needed at the end of the run of the script. BTW, the script takes around 4 minutes to complete.

output "efs_dns_name" {
	value = "${aws_efs_file_system.efsWordPress.dns_name}"
}

output "ip_address" {
    value = "${aws_instance.wordpress.public_ip}"
}

output "sql_hostname" {
    value = "${aws_db_instance.dbWordPress.address}"
}

At the end of the script, we’ll get the DNS name of the EFS filesystem, the external IP address of the instance and the MySQL hostname. We’ll need the last two.
And finally, the main script (main.tf) is where all the resources that are needed are specified.

provider "aws" {
	region = "${var.region}"
}

resource "aws_security_group" "sgWordPress" {
	name = "sgWordPress"
	vpc_id      = "${var.vpc_id}"

	ingress {
		from_port = "${var.ssh_port}"
		to_port = "${var.ssh_port}"
		protocol = "tcp"
		cidr_blocks =["0.0.0.0/0"]
	}

	ingress {
		from_port = "${var.http_port}"
		to_port = "${var.http_port}"
		protocol = "tcp"
		cidr_blocks = ["0.0.0.0/0"]
	}

	ingress {
		from_port = "${var.mysql_port}"
		to_port = "${var.mysql_port}"
		protocol = "tcp"
		self = true
	}

	ingress {
		from_port = "${var.nfs_port}"
		to_port = "${var.nfs_port}"
		protocol = "tcp"
		self = true
	}

	egress {
		from_port = 0
		to_port = 0
		protocol = "-1"
		cidr_blocks = ["0.0.0.0/0"]
	}

	tags {
		Name = "sgWordPress"
	}
}

resource "aws_efs_file_system" "efsWordPress" {
  creation_token = "EFS for WordPress"

  tags {
    Name = "EFS for WordPress"
  }
}

data "aws_subnet_ids" "suballIDs" {
	vpc_id = "${var.vpc_id}"
}

resource "aws_efs_mount_target" "mtWordPress" {
  count = "${length(data.aws_subnet_ids.suballIDs.ids)}"
  file_system_id = "${aws_efs_file_system.efsWordPress.id}"
  subnet_id      = "${element(data.aws_subnet_ids.suballIDs.ids, count.index)}"
  security_groups = ["${aws_security_group.sgWordPress.id}"]
}

resource "aws_instance" "wordpress" {
	ami = "${var.ami_id}"
	instance_type = "${var.instance_type}"
	vpc_security_group_ids = ["${aws_security_group.sgWordPress.id}"]
	key_name = "${var.key_name}" 
	ebs_block_device {
		device_name = "/dev/sdb"
    	volume_size = "${var.volume_size}"
    	delete_on_termination = "true"
  	}

	tags {
		Name = "WordPress Server"
	}

	user_data = <<EOF
		#!/bin/bash
		echo "${aws_efs_file_system.efsWordPress.dns_name}:/ /var/www/html nfs defaults,vers=4.1 0 0" >> /etc/fstab
		yum install -y php php-dom php-gd php-mysql
		for z in {0..120}; do
			echo -n .
			host "${aws_efs_file_system.efsWordPress.dns_name}" && break
		  	sleep 1
		done
		cd /tmp
		wget https://www.wordpress.org/latest.tar.gz
		mount -a
		tar xzvf /tmp/latest.tar.gz --strip 1 -C /var/www/html
		rm /tmp/latest.tar.gz
		chown -R apache:apache /var/www/html
		systemctl enable httpd
		sed -i 's/#ServerName www.example.com:80/ServerName www.myblog.com:80/' /etc/httpd/conf/httpd.conf
		sed -i 's/ServerAdmin root@localhost/ServerAdmin admin@myblog.com/' /etc/httpd/conf/httpd.conf
		#setsebool -P httpd_can_network_connect 1
		#setsebool -P httpd_can_network_connect_db 1
		systemctl start httpd
		#firewall-cmd --zone=public --permanent --add-service=http
		#firewall-cmd --reload
		#iptables -A INPUT -p tcp --dport 80 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
		#iptables -A OUTPUT -p tcp --sport 80 -m conntrack --ctstate ESTABLISHED -j ACCEPT
	EOF

}

resource "aws_db_instance" "dbWordPress" {
	identifier = "dbwordpress"
	engine = "mysql"
	engine_version = "5.7"
	allocated_storage = "${var.allocated_storage}"
	instance_class = "${var.instance_class}"
	vpc_security_group_ids = ["${aws_security_group.sgWordPress.id}"]
	name = "${var.db_name}"
	username = "${var.db_admin}"
	password = "${var.db_password}"
	parameter_group_name = "default.mysql5.7"
	skip_final_snapshot = true
	tags {
		Name = "WordPress DB"
	}
}

The script starts with the name of the provided that will be used (AWS), then a security group is being defined with ports 22 and 80 open to the world. Opening port 22 to the world is not a good idea, so you can restrict the access. In line 13, you can change the subnets allowed, e.g. instead of [“0.0.0.0/0”], you can restrict the SSH to a couple of IPs, e.g. [“1.2.3.4/32″,”12.13.14.15/32”]. Then, we define the ports for NFS and MySQL. If you can see these ports are open to themselves only. That means that only the resources having assigned that security group can talk between themselves on these two ports. And finally, we allow all outgoing traffic.
Further, we have an EFS file system created and mount targets. The mounts targets define in what availability zones the EFS system will be available. Since each region has different numbers of availability zones, we don’t know that number, so we have to enumerate the AZs for each region. If you look at lines 61 to 66 you’ll see how we do that. It’s pretty much a for..next loop in Terraform.
At line 68, we have our main resource defined, the instance. You’ll see that each resource has a lot of input parameters. Some of these are mandatory and some are optional. For each resource, you can find a detailed explanation on the Terraform site. For example, if you google “terraform aws_instance” the first link that shows up on terraform.io website will be about that.
At line 83 the custom script begins. It will add the EFS mount point to /etc/fstab so the NFS system is mounted on each reboot. At line 87 there is a short delay until EFS filesystem becomes available. Because the resources are created in parallel, it takes up to 90 seconds for the EFS DNS name to get propagated. If we don’t have the delay, the instance will boot up, try to mount the EFS system and fail, because the DNS mount point won’t be accessible. AWS Linux 2 comes with SELinux disabled and no firewalls, but if you have some other instance, you can uncomment the lines at the end of the user data script. Otherwise, SELinux will prevent Apache and MySQL to work with PHP. Lines 106 and 107 are if you use iptables instead of firewalld. Tailor to your needs. Finally, at line 112, the MySQL DB is defined.
Once you have all these three files (vars.tf, outputs.tf and main.tf) modified and saved in the same folder/directory, you can star the provisioning. First, initialize the script so Terraform can download the plugin for AWS.

terraform init

Then, check if everything is OK with the script.

terraform plan

And finally, execute the script.

terraform apply

It will prompt you to say “yes” and after the script completes, you’ll have something like this at the end.

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

Outputs:

efs_dns_name = fs-64307acd.efs.us-west-2.amazonaws.com
ip_address = 34.220.235.222
sql_hostname = dbwordpress.cbh9gck8kp6s.us-west-2.rds.amazonaws.com

Go to http://your_ip and you should have the welcome WordPress screen. Use the database name, username and the password for the database defined in vars.tf (dbwordpress, dbadmin, SuperSecret) and for the hostname use the SQL DB output variable (sql_hostname) above.

And that’s it! You have your WordPress up and running on RDS and EFS filesystem without even logging to AWS console or your Linux instance.

Once you are done playing you can destroy all of the resources with one command.

terraform destroy

I had to migrate a SQL 2014 database from our on-prem environment to AWS. The requirement was to use RDS. So, these are the steps that I’ve taken in order to successfully migrate the database. You can also check the official AWS link here.

First, take a backup of the existing database in BAK format. In my case for the purpose of this blog, I’ve backed up my vCenter database called VCDB in a file called VCDB.BAK. Once you have the file, create a S3 bucket in AWS with the default settings and upload the BAK file there. In my case the S3 bucket was named restoremybackup.
Next, make sure you have access to a Windows server with SQL Management Studio installed, either in AWS or on-prem, but this server will need to talk to the RDS database over port 1433 TCP. I built a server in AWS and installed the SQL Management Studio, which is a free download. Make sure that you have a security group in place that allows port 1433. We will use this security group to assign it to the RDS instance. I’ve created a new security group called sgAllowSQLAccess. For the source, type the IP of the server with SQL Management Studio or the security group itself.

Then, go to RDS and create an Option Group. Name the Option Group, in my case it’s ogSQL2014Restore, choose the engine and the version. I am using sqlserver-se which is SQL Standard Edition and the version is 12.00 which is MS SQL 2014.

Once created, select your Option Group and click on Add Option.

Choose SQLSERVER_BACKUP_RESTORE, then click on Create a New Role. For the role, use a name, in my case it’s iamRestoreSQLBackup. Then type the name of the S3 bucket that you just created and make sure that you select Apply Immediately – yes.
Then click on Add Option.

Now, it’s time to create the RDS database. Launch a new RDS instance and use the parameters that suit your needs.

For the next set of options, choose your preferred settings, but make sure that you assign the Security Group that we just created and the Option Group that we also created. In my case it looks like this.

Once the RDS it’s in available state, try to telnet to port 1433 to the RDS endpoint from the Windows server where you have the SQL Management Studio installed. If everything is OK, you are ready to restore the database. In SQL Management Studio, select a New Query and type the following.

exec msdb.dbo.rds_restore_database 
        @restore_db_name='database_name', 
        @s3_arn_to_restore_from='arn:aws:s3:::bucket_name/file_name_and_extension';

Replace ‘database_name’ and ‘bucket_name/file_name_and_extenstion’ with your values. In my case, they were VCDB and restoremybackup/VCDB.BAK.