Configure AWS PrivateLink for RDS using an AWS CloudFormation Template

1. Tutorial overview

Last Updated: 2024-04-16

Background

AWS PrivateLink provides private connectivity between virtual private clouds (VPCs), supported AWS services, and on-premises networks without exposing traffic to the public internet.

Starburst Galaxy supports AWS PrivateLink for some of its catalogs. In this tutorial, you will learn how to configure PrivateLink for an Amazon RDS instance using an AWS CloudFormation Template (CFT).

What is an AWS CloudFormation Template (CFT)?

A CloudFormation template (CFT) is a JSON or YAML file used to define AWS resources and their configurations when deploying application or infrastructure stacks. It is a popular way of automating infrastructure deployments on AWS, and is particularly useful in multi-region clusters.

The CFT provided in this tutorial should be used whenever you are working with a multi-region RDS cluster or Aurora cluster. These clusters can failover from one availability zone to another, resulting in a change in the endpoint's IP address. The CFT includes a crucial Lambda script that monitors failovers and updates the IP address of the target group used by the endpoint service accordingly. This ensures seamless and continuous service availability even during failover events.

The CFT also deploys the target group, network load balancer, and endpoint service required for PrivateLink.

Scope of tutorial

In this tutorial, you will learn how to use a CFT to configure AWS PrivateLink for an Amazon RDS instance.

Learning objectives

Once you've completed this tutorial, you will be able to:

• Use a CFT to configure AWS PrivateLink for connectivity from Starburst Galaxy to an RDS instance.
• Use PrivateLink to securely connect Starburst Galaxy to an Amazon RDS instance.

Prerequisites

• You need a Starburst Galaxy account to complete this tutorial. Please see Starburst Galaxy: Getting started for instructions on setting up a free account.
• This tutorial comes with a bring your own storage requirement. Before proceeding with this lesson, you must already have an Amazon RDS instance or Aurora cluster set up.

About Starburst tutorials

Starburst tutorials are designed to get you up and running quickly by providing bite-sized, hands-on educational resources. Each tutorial explores a single feature or topic through a series of guided, step-by-step instructions.

As you navigate through the tutorial you should follow along using your own Starburst Galaxy account. This will help consolidate the learning process by mixing theory and practice.

2. Working with a Starburst technical resource

Background

If you are configuring PrivateLink for the first time you are encouraged to work with a Starburst technical resource. This individual will work with you to set up the environment needed to complete the tutorial.

Contacting your technical resource

To be assigned this resource, you should reach out to your regular Starburst account team for assistance.

Working together

Once assigned, your Starburst technical resource will work with you to set up an environment where you can complete the tutorial.

Please review the following overview of this process before beginning the tutorial.

Your responsibilities:

• Obtain your RDS instance details.
• Record the IP address of your RDS instance endpoint.
• Deploy the CFT to create a target group, a network load balancer, an endpoint service, and AWS Lambda.
  
  Note: AWS Lambda will automate network load balancer target group IP address synchronization in the event of an RDS failover.
• Provide your endpoint service name to Starburst technical support by opening a Starburst Galaxy support ticket.
• Starburst support will create the endpoint in the Starburst Galaxy AWS account.
• Monitor the endpoint connection in your AWS account.

3. Amazon RDS PrivateLink architecture

Background

Understanding the Amazon RDS PrivateLink architecture is important when completing the steps in this tutorial. In this section you will learn about this architecture and the way that Starburst Galaxy uses it to securely connect private clouds.

This tutorial also follows the corresponding GitHub readme on the topic. It is recommended that you review this alongside this tutorial for more information.

Reference architecture

The following diagram illustrates a PrivateLink connection between the Starburst Galaxy VPC and the Amazon RDS VPC.

Review the diagram and corresponding notes below for more information.

1. Once the PrivateLink configuration is complete, an endpoint is created in the Starburst Galaxy VPC (Source).
   
   This endpoint connects to a Network Load Balancer located inside an endpoint service situated in the Amazon RDS VPC (Destination).
   
   This establishes a private connection between Starburst Galaxy and Amazon RDS, enabling PrivateLink functionality.
2. In this reference architecture, the Starburst Galaxy VPC is the source.
3. In this reference architecture, the Amazon RDS VPC is the destination.

4. Obtain RDS instance details

Background

It's time to get started. In this section, you'll begin by obtaining some key information about your Amazon RDS instance, including:

• RDS endpoint
• Port number
• AWS Availability Zone
• VPC
• Subnet IDs
• VPC security group
• RDS endpoint IP address

You'll need this information to prepare the CloudFormation Template prior to deployment.

Step 1: Sign in to AWS console

You're going to start by signing in to your AWS console.

Remember that this should be the AWS account containing the RDS instance that you would like to connect using PrivateLink, so if you use multiple AWS accounts, ensure that you pick the correct one.

• Sign in to your AWS account.
• In the AWS console, enter RDS in the search field.
• Select RDS from the drop-down menu.

Step 2: Select RDS instance

Now it's time to find the right RDS instance. Depending on your workflow, you might have multiple instances in the same AWS account, so make sure you select the correct one.

• Using the Amazon RDS menu on the left, select Databases.
• Search for your RDS instance by name using the Database search field.
• Select the RDS instance to view its details.

Step 3: Record RDS details

Now it's time to record details about your RDS instance.

• In the Summary section, record the name of the RDS Engine.
• In the Connectivity & Security section, record the Endpoint, Port, Availability Zone, VPC, Subnets, and VPC security group.

For example:

• Engine: PostgreSQL
• Endpoint: erin-rosas-bootcamp-postgresql.cs8j7iukogcy.us-east-2.rds.amazonaws.com
• Port: 5432
• Availability Zone: us-east-2a
• VPC: bootcamp-vpc (vpc-0a354f3468f906f36)
• Subnets: subnet-0ba16d518af670462, subnet-079aebff5aec9ee59, subnet-011342a0c2d61823d

Step 4: Record RDS endpoint IP address

Next, you will use your RDS endpoint to determine its IP address.

To do this, you'll use a terminal window. Again, you will be copying information into your text editor.

• Open a Terminal window on your desktop.
• Run one of the following commands to retrieve the IP address.

Note: The command you choose will depend on your operating system. Be sure to replace [rds-endpoint] with your actual RDS endpoint.

• In Windows run the command nslookup [rds-endpoint]
• In Linux\MacOS run the command dig [rds-endpoint]
• Record the IP address of the RDS instance.

5. Deploy the CloudFormation Template

Background

Now it's time to work with the CloudFormation Template. You will be using a template file provided by AWS. This template simplifies resource creation by completing most of the steps automatically.

For this to work, you will need to enter the required information from your RDS instance. After that, the CFT will create a target group, load balancer, endpoint service, and AWS lambda automatically.

In production, this helps to save significant time.

Step 1: Download the prepared template file

The AWS-samples GitHub repository provides a template file outlining cross-account access methods using PrivateLink.

You'll download this file to simplify the CFT deployment.

• Open the GitHub repository in your browser.
• Open the options menu for the yml file by clicking the ellipses.
• In the Raw file content section, select Download.
• You will need the file soon, so make sure to take note of where it downloads.

Step 2: Begin CFT deployment

Now that you have your CFT template, it's time to deploy the CFT in the AWS console.

• In the AWS console, search for CloudFormation and select CloudFormation from the results list.
• Click Create stack.
• Select With new resources (standard) from the drop-down menu.

Step 3: Upload the CFT template

Next you'll upload the template that you obtained from GitHub.

• In the Prepare template section, select Template is ready.
• Select Upload a template file.
• Click Choose file.
• Select the CrossAccountRDSAccess.yml file that you downloaded earlier in this tutorial.
• Click the Next button.

Step 4: Configure stack parameters

The template you just uploaded has several parameters that must be entered.

You're going to use the information you recorded in the first section of this tutorial to complete the next few steps.

• In the stack name field, enter a meaningful Stack name.
• Add an extension to the existing Lambda Function Name to make it unique for you. We suggest using the stack name as the extension.
• Select RDS type using the following guidelines.
• If you are using a Single-AZ RDS instance, select db-instance.
• If you are using a Multi-AZ Aurora MySQL, Multi-AZ Aurora PostgreSQL, or Multi-AZ DB Cluster - new, select db-cluster.

• Enter your RDS endpoint.
• Enter your RDS port.

Step 5: Configure VPC input for lambda

Next, it's time to add additional parameters to the CFT. Specifically, you need to enter information into the VPC input for lambda section of the console.

• Select your RDS VPC using the drop-down menu.
• Select all of the subnets that you recorded from your Subnet group.
• Expand the Select Security Group for Lambda drop-down menu, and select a security group with inbound rules allowing the IP Cidr 172.16.0.0/16.

• Enter the following in the Role ARN of target Account/VPC field: arn:aws:iam::179619298502:root.
• Click the Next button.
• Check the required box to acknowledge IAM resources.
• Click the Submit button.

Step 6: Monitor stack deployment until complete

Your stack deployment will now be created. This process takes several minutes.

While you wait, you should monitor the progress of your stack deployment to ensure that the process has finished.

• Monitor your stack creation process.
• Confirm that the status of your stack is listed as CREATE_COMPLETE.

6. Configure stack resources

Background

The CFT you configured in the last section created a stack of available resources. You will now configure these resources to work with your RDS instance using PrivateLink.

To do this, you will update the target group by adding your RDS endpoint IP address to the registered targets list.

Step 1: Find your target group

In the last section, you ended on the information page listing the details of the new stack that you created using the CFT.

You will continue from this stage by locating your target group.

• In the new stack, select the Resources tab.
• Locate the rNetworkLoadBalancerTargetGroup resource from the list of available resources.
• Locate the link corresponding to this target group.
• Click the arrow at the bottom of the link.
• Confirm that a new browser tab has opened with a filtered list of the target group created by this stack.
• Select the Target group.
  
  For example, NLB-TG-RDS-pl-sqlserver.

Step 2: Provide your RDS endpoint IP address

Now it's time to add your RDS endpoint IP address to the list of targets for the target group.

• Select either of the Register targets buttons.
• Enter the IPv4 address of your RDS endpoint.
• Click Include as pending below.
• Click Register pending targets.
• Monitor the Health Status and wait for a status of healthy
  
  Note: You may need to click the refresh button.

7. Submit Starburst Galaxy support ticket

Background

Time to switch gears. You've completed all of the steps required on your own. Now it's time to contact the Starburst support team to finish the last steps.

Step 1: Locate your endpoint service name

Starburst support needs this information to complete the steps on their end.

• Return to your stack's Resources tab.
• Locate the rVPCEndpointService resource.
• Copy the name.
  
  For example, vpce-svc-035362fdf1ed4780c
• Click the arrow to open a new browser tab open to Endpoint services.
• Paste the name you copied into the filter box, and press enter.
• Click the Service ID
  
  For example, vpce-svc-035362fdf1ed4780c
• In the Details section, copy the Service name.

Step 2: Open support ticket

You are going to use the automated assistant in Starburst Galaxy to open a support ticket and provide support with the Service name that you just copied. You will also need to provide the port your database is listening on and your preferred Starburst Galaxy PrivateLink configuration name.

• Log in to Starburst Galaxy.
• Click the support icon located at the bottom right of the screen.
• Select Chat with technical support.
• Select Submit a Support Ticket.
• The automated assistant will ask you to provide your email address, first name, and last name.
• When you receive the prompt to describe your issue, note that you would like support to create a private endpoint connection for you. Be sure to include the Service name you just copied, the port your database is listening on, and your preferred Starburst Galaxy PrivateLink connection name.

• Wait for Starburst support to confirm that they have created the Endpoint in Starburst Galaxy. This should take no longer than 24 - 48 hours.

Step 3: Confirm endpoint connection

That's it. The connection is now being created. This process takes between 1 to 3 minutes to complete.

When this process is complete, you are finished and ready to start using PrivateLink.

• Select the Endpoint connections tab.
• Wait to see the connection listed, which should happen automatically.
• Wait for the State to change from Pending to Available.
• Click the Refresh button to view status updates.

8. Tutorial wrap-up

Tutorial complete

Congratulations! You have reached the end of this tutorial, and the end of this stage of your journey.

You're all set! Now you can use PrivateLink to configure access from Starburst Galaxy to data in your Amazon RDS instance.

Continuous learning

At Starburst, we believe in continuous learning. This tutorial provides the foundation for further training available on this platform, and you can return to it as many times as you like. Future tutorials will make use of the concepts used here.

Next steps

Starburst has lots of other tutorials to help you get up and running quickly. Each one breaks down an individual problem and guides you to a solution using a step-by-step approach to learning.

Tutorials available

Visit the Tutorials section to view the full list of tutorials and keep moving forward on your journey!