Using VPC Peering to Connect an OpenShift Service on an AWS (ROSA) Cluster to an Amazon RDS MySQL Database in a Different VPC

Red Hat OpenShift Service on AWS (ROSA) provides an integrated experience to use OpenShift. If you are already familiar with OpenShift, you can accelerate your application development process by leveraging familiar OpenShift APIs and tools for deployments on AWS. With ROSA, you can use the wide range of AWS compute, database, analytics, machine learning, networking, mobile, and other services to build secure and scalable applications faster.

In this post, you will see how to connect an application/pod running on ROSA to an Amazon RDS for MySQL database, which like ROSA is also a managed service. In this case, both ROSA and the RDS instance are in their own dedicated VPC.

Here are the steps I followed to demonstrate connecting to an Amazon RDS MySQL database from an application running on a ROSA cluster:

1. Create the ROSA cluster in VPC A
2. Set up the Amazon RDSMySQL Database in VPC B
3. Connect the two VPC’s using VPC Peering

Create the ROSA Cluster

We will create a Multi AZ cluster using the rosa CLI as shown below:

mshetty@mshetty-mac ROSA % rosa create cluster --cluster-name mytest --multi-az --region us-west-2 --version 4.7.4 --enable-autoscaling --min-replicas 3 --max-replicas 3 --machine-cidr 10.0.0.0/16 --service-cidr 172.30.0.0/16 --pod-cidr 10.128.0.0/14 --host-prefix 23
mshetty@mshetty-mac ROSA % rosa list clusters
ID                                    NAME        STATE
1kbm9ul8ea2et1n3h2mg5jp0fk6k0h6f  mytest  ready
mshetty@mshetty-mac ROSA % rosa create admin --cluster=mytest              
W: It is recommended to add an identity provider to login to this cluster. See 'rosa create idp --help' for more information.
I: Admin account has been added to cluster 'mytest'.
I: Please securely store this generated password. If you lose this password you can delete and recreate the cluster admin user.
I: To login, run the following command:
   oc login https://api.mytest.ri4o.p1.openshiftapps.com:6443 --username cluster-admin --password <passwd>
I: It may take up to a minute for the account to become active.
mshetty@mshetty-mac ROSA %
mshetty@mshetty-mac ROSA % oc login https://api.mytest.ri4o.p1.openshiftapps.com:6443 --username cluster-admin --password <passswd>
Login successful.
You have access to 86 projects, the list has been suppressed. You can list all projects with 'oc projects'
Using project "default".
mshetty@mshetty-mac ROSA %

We will create a Kubernetes Service named mysql-service of type ExternalName aliasing the RDS endpoint:

mshetty@mshetty-mac ROSA % cat mysql-service.yaml
# mysql-service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
        app: mysql-service
  name: mysql-service
spec:
  externalName: demordsmyqldbinstance.chjd85ogsv2m.us-west-2.rds.amazonaws.com
  selector:
        app: mysql-service
  type: ExternalName
status:
  loadBalancer: {}

Run oc apply -f mysql-service.yaml to create the service:

mshetty@mshetty-mac ROSA % oc apply -f mysql-service.yaml
service/mysql-service created

We see that when we try to connect to the mysql-service from the pod, we are not able to connect:

mshetty@mshetty-mac ROSA % oc run -i --tty --rm debug --image=busybox --restart=Never -- sh

If you don't see a command prompt, try pressing enter.

/ # nc mysql-service 3306

Setup Amazon RDS for MySQL

We will create the RDS MySQL database using the AWS CLI by following the AWS documentation for RDS.

• Create the VPC

First, let us start by creating the VPC with the CIDR block 192.168.0.0/16 for RDS:

mshetty@mshetty-mac ROSA % aws ec2 create-vpc --cidr-block 192.168.0.0/16 | jq '{VpcId:.Vpc.VpcId,CidrBlock:.Vpc.CidrBlock}'
{
  "VpcId": "vpc-097bcbac03c3d8302",
  "CidrBlock": "192.168.0.0/16"
}
mshetty@mshetty-mac ROSA % export RDS_VPC_ID=vpc-097bcbac03c3d8302

• Create the Subnets

Next, we will create subnets in two Availability Zones. These subnets will be used when creating the DB subnet group. A DB subnet group is a collection of subnets (typically private) that you create for a VPC and that you then designate for your DB instances.

The RDS VPC (RDS_VPC_ID) is divided into two subnets: 192.168.0.0/19 and 192.168.32.0/19.

The first subnet is created in the availability zone us-west-2a:

And the second one is created in the availability zone us-west-2b:

mshetty@mshetty-mac ROSA % aws ec2 create-subnet --availability-zone "us-west-2b" --vpc-id ${RDS_VPC_ID} --cidr-block 192.168.32.0/19 | jq '{SubnetId:.Subnet.SubnetId,AvailabilityZone:.Subnet.AvailabilityZone,CidrBlock:.Subnet.CidrBlock,VpcId:.Subnet.VpcId}'
{
  "SubnetId": "subnet-08ec3f2379d718be5",
  "AvailabilityZone": "us-west-2b",
  "CidrBlock": "192.168.32.0/19",
  "VpcId": "vpc-097bcbac03c3d8302"
}

Each VPC has an implicit router and uses route tables to control where network traffic is directed. Each subnet in the VPC must be associated with a route table, which controls the routing for the subnet (subnet route table). We can explicitly associate a subnet with a particular route table. Otherwise, the subnet is implicitly associated with the main route table. For more information on VPC Route tables, please visit this document.

Here we associate these two subnets that we created, to the VPC's route table:

# Get the route table information and save it in a variable
mshetty@mshetty-mac ROSA % aws ec2 describe-route-tables --filters Name=vpc-id,Values=${RDS_VPC_ID} | jq '.RouteTables[0].RouteTableId'
"rtb-0c5cde34e0d406cee"
mshetty@mshetty-mac ROSA % export RDS_ROUTE_TABLE_ID=rtb-0c5cde34e0d406cee
# Associate the first subnet with the above route table:
mshetty@mshetty-mac ROSA % aws ec2 associate-route-table --route-table-id rtb-0c5cde34e0d406cee --subnet-id subnet-053ef2a03489e26e0
{
        "AssociationId": "rtbassoc-0bffb062e6500b28a",
        "AssociationState": {
            "State": "associated"
        }
}
# Associate the second subnet with the above route table:
mshetty@mshetty-mac ROSA % aws ec2 associate-route-table --route-table-id rtb-0c5cde34e0d406cee --subnet-id subnet-08ec3f2379d718be5
{
        "AssociationId": "rtbassoc-0dc11954068c8e915",
        "AssociationState": {
            "State": "associated"
        }
}

• Create DB Subnet Group

We will create the DB subnet group using the CLI as shown below, or you can use the AWS console as shown here.

mshetty@mshetty-mac ROSA % aws rds create-db-subnet-group --db-subnet-group-name  "DemoDBSubnetGroup" --db-subnet-group-description "Demo DB Subnet Group" --subnet-ids "subnet-053ef2a03489e26e0" "subnet-08ec3f2379d718be5" | jq '{DBSubnetGroupName:.DBSubnetGroup.DBSubnetGroupName,VpcId:.DBSubnetGroup.VpcId,Subnets:.DBSubnetGroup.Subnets[].SubnetIdentifier}'
{
  "DBSubnetGroupName": "demodbsubnetgroup",
  "VpcId": "vpc-097bcbac03c3d8302",
  "Subnets": "subnet-08ec3f2379d718be5"
}
{
  "DBSubnetGroupName": "demodbsubnetgroup",
  "VpcId": "vpc-097bcbac03c3d8302",
  "Subnets": "subnet-053ef2a03489e26e0"
}

• Create a VPC Security Group

Before we create the DB instance, we must create a VPC security group to associate with         the DB instance as shown below. For more information, please refer to the documentation.

Create the Security Group and save the Security Group ID in a variable:

mshetty@mshetty-mac ROSA % aws ec2 create-security-group \
--group-name DemoRDSSecurityGroup \
--description "Demo RDS security group" \
--vpc-id ${RDS_VPC_ID}
{
        "GroupId": "sg-0c1d4c274a427ea2b"
}
subsequent commands
mshetty@mshetty-mac ROSA % export RDS_VPC_SECURITY_GROUP_ID=sg-0c1d4c274a427ea2b

• Create the RDS DB Instance

Next, we create the DB instance using the aws CLI. For details on how to create a DB instance, see Creating an Amazon RDS DB instance.

mshetty@mshetty-mac ROSA % aws rds create-db-instance \
  --db-name demordsmyqldb \
  --db-instance-identifier demordsmyqldbinstance \
  --allocated-storage 10 \
  --db-instance-class db.t2.micro \
  --engine mysql \
  --engine-version "5.7.26" \
  --master-username mydbuser \
  --master-user-password mydbuserpasswd \
  --no-publicly-accessible \
  --vpc-security-group-ids ${RDS_VPC_SECURITY_GROUP_ID} \
  --db-subnet-group-name "demodbsubnetgroup" \
  --availability-zone us-west-2a \
  --port 3306 | jq '{DBInstanceIdentifier:.DBInstance.DBInstanceIdentifier,Engine:.DBInstance.Engine,DBName:.DBInstance.DBName,VpcSecurityGroups:.DBInstance.VpcSecurityGroups,EngineVersion:.DBInstance.EngineVersion,PubliclyAccessible:.DBInstance.PubliclyAccessible}'
{
  "DBInstanceIdentifier": "demordsmyqldbinstance",
  "Engine": "mysql",
  "DBName": "demordsmyqldb",
  "VpcSecurityGroups": [
        {
          "VpcSecurityGroupId": "sg-0c1d4c274a427ea2b",
          "Status": "active"
        }
  ],
  "EngineVersion": "5.7.26",
  "PubliclyAccessible": false
}

We can verify that the DB instance has been created in the UI as well:

Connecting the two VPC’s Using VPC Peering

A VPC peering connection is a networking connection between two VPCs that enables you to route traffic between them using private IPv4 addresses or IPv6 addresses. Instances in either VPC can communicate with each other as if they are within the same network. You can create a VPC peering connection between your own VPCs or with a VPC in another AWS account. The VPCs can be in different regions (also known as an inter-region VPC peering connection).

For more information on VPC peering, check the AWS documentation on it.

• Create and Accept a VPC Peering Connection

Create a VPC peering connection using the AWS console as follows:

1. VPC console: https://console.aws.amazon.com/vpc/
2. Select Peering Connections and click on Create Peering Connection.
3. Select the ROSA VPC as the Requester and the RDS VPC as the Accepter:

Click on Create Peering Connection

Select “rosa-peering-connection” that we just created. Then click on Actions drop down menu and navigate to Accept Request.

Accept the VPC Peering Connection Request by clicking on “Yes, Accept”.

The next screen capture shows that the VPC Peering Connection has been established, and that you must add a route to the peered VPC in one or more of your VPC route tables to be able to send and receive traffic across the VPC connection:

Save the Peering Connection ID in a variable:

mshetty@mshetty-mac ROSA % export VPC_PEERING_CONNECTION_ID=pcx-0330872bdb0c46775

• Update the ROSA Cluster VPC's Route Table

Get the route table associated with the three public subnets of the VPC created by rosactl:

mshetty@mshetty-mac ROSA % export ROSA_ROUTE_TABLE_ID=rtb-06e2ea2bcaa395dce

Add a route to the route table so that all traffic to the RDS VPC CIDR block is via the VPC Peering Connection:

mshetty@mshetty-mac ROSA % aws ec2 create-route --route-table-id ${ROSA_ROUTE_TABLE_ID} --destination-cidr-block 192.168.0.0/16 --vpc-peering-connection-id ${VPC_PEERING_CONNECTION_ID}
{
        "Return": true
}

• Update the RDS Instance's Security Group

Update the Security Group to allow all ingress traffic from the ROSA cluster to the RDS instance on port 3306:.

mshetty@mshetty-mac ROSA % aws ec2 authorize-security-group-ingress --group-id ${RDS_VPC_SECURITY_GROUP_ID} --protocol tcp --port 3306 --cidr 10.0.0.0/16

Test the Connection to RDS

We can now connect to the RDS MySQL database from a pod running on the ROSA cluster:

mshetty@mshetty-mac ROSA % oc run -i --tty --rm debug --image=busybox --restart=Never -- sh

If you don't see a command prompt, try pressing enter.

/ # nc mysql-service 3306
N
5.7.26-log?hQ6|on{???aB#h'
 amysql_native_password

Here we are connecting to the RDS MySQL database from a pod using mysql-57-rhel7 image:

mshetty@mshetty-mac ROSA % oc run -i --tty --rm debug --image=registry.access.redhat.com/rhscl/mysql-57-rhel7 --restart=Never -- sh

If you don't see a command prompt, try pressing enter.

sh-4.2$ which mysql
/opt/rh/rh-mysql57/root/usr/bin/mysql
sh-4.2$ mysql -h demordsmyqldbinstance.chjd85ogsv2m.us-west-2.rds.amazonaws.com -u mydbuser -p                        
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 90
Server version: 5.7.26-log Source distribution
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
mysql>
mysql> show databases;
+--------------------+
| Database               |
+--------------------+
| information_schema |
| demordsmyqldb          |
| innodb                 |
| mysql                  |
| performance_schema |
| sys                    |
+--------------------+
6 rows in set (0.01 sec)
mysql>

The below figure shows VPC Peering between VPC A with ROSA, and VPC B with B with Amazon RDS for MySQL.

Conclusion

We have seen how pods running on a ROSA cluster in one VPC can access an Amazon RDS database running on a different VPC using VPC peering. In this blog post we have created a VPC peering connection between my own VPCs, but the same procedure holds good for a VPC in another AWS account. Also, the VPCs can be in different regions (also known as an inter-region VPC peering connection).

In the next blog post, you will see how to use IAM roles- and create fine grained access to AWS services.