Last Updated: 2024-01-12

Background

Elasticsearch is a distributed, open-source search and analytics engine built on top of Apache Lucene. It is designed to provide a scalable search and analytics solution for large volumes of data. Elasticsearch is commonly used for various purposes, including full-text search, log and event data analysis, and real-time application monitoring.

Scope of tutorial

In this tutorial, you will learn how to configure a Elasticsearch catalog in Starburst Galaxy.

Learning objectives

Once you've completed this tutorial, you will be able to:

Configure a Elasticsearch catalog using Starburst Galaxy.
Access your own Elasticsearch data source using Starburst Galaxy.

Prerequisites

You need a Starburst Galaxy account to complete this tutorial. Please see Starburst Galaxy: Getting started for instructions on setting up a free account.
This tutorial comes with a bring your own storage requirement. Before proceeding with this lesson, you must already have an Elasticsearch account set up.

If this is not the case, please set this up first then return to this tutorial.

About Starburst tutorials

Starburst tutorials are designed to get you up and running quickly by providing bite-sized, hands-on educational resources. Each tutorial explores a single feature or topic through a series of guided, step-by-step instructions.

As you navigate through the tutorial you should follow along using your own Starburst Galaxy account. This will help consolidate the learning process by mixing theory and practice.

Background

Now it's time to pivot over to Starburst Galaxy to begin the process of connecting your Elasticsearch data source.

Starburst Galaxy separates users by role. Configuring a new catalog will require access to a role with appropriate privileges. Today, you'll be using the accountadmin role.

This is a quick step, but an important one.

Step 1: Sign into Starburst Galaxy

Sign into Starburst Galaxy in the usual way. If you have not already set up an account, you can do that here.

Step 2: Set your role

Your current role is listed in the top right-hand corner of the screen.

Check your role, to ensure that it is set to accountadmin.
If it is set to anything else, use the drop-down menu to select the correct role.

Background

Adding a new Elasticsearch catalog follows the same process as adding other data sources in Starburst Galaxy.

The steps below will show you how to start the process of configuring a new catalog.

Step 1: Create a new catalog

Create a new catalog for your Elasticsearch data source.

In the left-hand navigation bar, click Catalogs.
Click the Create catalog button.

Step 2: Select Elasticsearch

Starburst Galaxy allows the creation of catalogs for a number of different data sources. In this case, you are going to create a new catalog in the Elasticsearch category.

Click the Elasticsearch tile.

Step 3: Input cloud provider, name, and description

Elasticsearch works with all three major cloud providers: AWS, Azure, and GCP. You will need to select which one you want to use for this catalog.

Additionally, the catalog needs both a name and description. This ensures that you can find it later.

Select your cloud provider.
In the Catalog name field, enter the name of the new catalog.
In the Description field, input a description. This can be anything you want, so make it meaningful for you.
Scroll down to continue the configuration process.

Background

When you connect Starburst Galaxy to a new data source, it is necessary to undergo an authentication process. This helps ensure that you are connecting the right data source and that you have the appropriate permissions.

Step 1: Choose the connection type

Now it's time to choose a connection type. There are three options:

Connect directly: Choose this option if your database has a public IP that allows direct connection from Galaxy.
Connect via SSH tunnel: Choose this option if your database is protected by a firewall on a private network.
PrivateLink: Choose this option if you have enabled PrivateLink for your Galaxy account and have already worked with your Starburst technical resource to configure a PrivateLink between Galaxy and your AWS VPC.

Step 2: Enter Elasticsearch connection details

Now it's time to connect Starburst Galaxy to Elasticsearch. To do this, you're going to enter details from your Elasticsearch account to create the connection.

Enter your Elasticsearch endpoint in the Elasticsearch domain or cluster endpoint field.
Choose either the password based, Cross account IAM role, or AWS access key methods.

Step 2a: Password authentication option

The first option for authentication uses the Elasticsearch username and password. These should be retrieved Elasticsearch

Enter the database username and password.

Step 2b: Cross account IAM role authentication option

The other option is authentication using the Cross account IAM role.

Select Cross account IAM role.
Select the Elasticsearch endpoint region from the menu.
Select the Cross account IAM role from the menu.

Step 2c: AWS access key

The other option is authentication using the AWS access key.

Select AWS access key.
Select the Elasticsearch endpoint region from the menu.
Enter your AWS access key for Elasticsearch.
Enter your AWS secret key for Elasticsearch.

Background

Every new catalog connection includes a test before you connect it. This helps to ensure that you have input the correct credentials and allows you to quickly fix any problems before actually connecting.

Step 1: Test and Connect

You're almost there! Time to test the connection and then complete the process of creating your new Elasticsearch catalog.

Click the Test connection button.
Confirm that you see the Hooray! You can now add this catalog to a cluster message.
Click the Connect catalog button.

Background

Starburst Galaxy allows you to configure your catalog in a number of ways regarding access controls. The most important of these involves granting write access or restricting the catalog to read-only access.

Take some time to consider whether you require write access, or whether read-only access will be sufficient.

Step 1: Select read access

Select the appropriate read access for your situation.

If you want to restrict the write access, select the read-only catalog button.
If you want to grant write access, deselect the read-only catalog button.
Click the Save access controls button.

Background

The catalog is now created, but you are not able to use it until you add it to a cluster. Clusters can include many catalogs. You can either add the new Elasticsearch catalog to an existing cluster, or do this at a later stage.

Step 1: Add catalog to cluster or skip

You can add a catalog to a cluster at any time.

If you want to add the catalog to a cluster later, click Skip
If you want to add the catalog to a cluster now, select the cluster name from the drop-down menu and click Add to cluster.

Tutorial complete

Congratulations! You have reached the end of this tutorial, and the end of this stage of your journey.

You're all set! Now you can query the data in your Elasticsearch data source.

Continuous learning

At Starburst, we believe in continuous learning. This tutorial provides the foundation for further training available on this platform, and you can return to it as many times as you like. Future tutorials will make use of the concepts used here.

Next steps

Starburst has lots of other tutorials to help you get up and running quickly. Each one breaks down an individual problem and guides you to a solution using a step-by-step approach to learning.

Tutorials available

Visit the Tutorials section to view the full list of tutorials and keep moving forward on your journey!