Use universal search in Starburst Galaxy

1. Tutorial overview

Last Updated: 2023-12-15

Background

Starburst Galaxy includes a universal search feature. This assists data discoverability across catalogs, schemas, tables, views, columns, data products, tags, owners, or contacts. This added transparency is backed-up by role-based security access, ensuring maximum visibility to those with the necessary privileges, while restricting access to other users.

This helps break down organizational knowledge silos, freeing data consumers to efficiently find, query, and analyze datasets more easily. It also assists data engineers and platform administrators, by providing a global view across the data pipeline.

Prerequisites

You need a Starburst Galaxy account to complete this tutorial. Please be sure to complete the tutorial titled Starburst Galaxy: Getting started before attempting this tutorial.

Learning outcomes

Upon successful completion of this tutorial, you will be able to:

• Add metadata to tables and columns.
• Use universal search and catalog search to find data entities.
• Use metadata to filter universal search results.

About Starburst tutorials

Starburst tutorials are designed to get you up and running quickly by providing bite-sized, hands-on educational resources. Each tutorial explores a single feature or topic through a series of guided, step-by-step instructions.

As you navigate through the tutorial you should follow along using your own Starburst Galaxy account. This will help consolidate the learning process by mixing theory and practice.

Tutorial scenario

The data engineers at Chryse Corp. aim to enhance the discoverability of datasets used by data analysts. To achieve this, they plan to add both tags and metadata. This will make it easier for data analysts to find and access the relevant data, improving overall data exploration and analysis processes.

You'll help them start this process by adding tags and metadata to their Starburst Galaxy datasets, focusing on the astronauts and missions tables that are included in the demo catalog. Then, you'll use Starburst Galaxy's search features to find these datasets.

2. How does universal search work?

Background

Universal search uses metadata to create a searchable index of different data assets across all catalogs and clusters connected with your account. This synopsis is meant as a jumping-off point for further data discovery and querying activities.

Notably, this universality extends to assets held across different clouds, whether it be AWS, Azure, or GCP. Although you can see the location of this data, you cannot transfer data across clouds and the results are based on metadata rather than the data itself.

Universal search is continuously updated on a streaming basis, so any changes you make within Starburst Galaxy will take immediate effect. Changes made outside of Starburst Galaxy are updated less frequently. Those changes will be reflected in a batch process updated approximately once every 24 hours.

Video: Use tags in Starburst Galaxy

The following video walks through the first two sections in this tutorial. It shows you how to create tags and add metadata to tables and columns.

You can choose to watch the video and follow along using your own account. Alternatively, if you prefer, you can skip the video and proceed directly to the step-by-step instructions provided later in the tutorial.

3. Create tags

Background

In Starburst Galaxy, you can add tags to data entities, including catalogs, schemas, tables, views, columns, data products, tags, owners, or contacts.

Universal search works in tandem with this, allowing users to filter their results based on the types of tags involved.

In this section of the tutorial, you'll create a set of tags to help data consumers find data. Specifically, you'll use the astronaut dataset to create a missions tag and two additional tags nested under missions, called personnel and info.

Step 1: Sign into Starburst Galaxy

Sign into Starburst Galaxy in the usual way. If you have not already set up an account, you can do that here.

Step 2: Verify that your role is set to accountadmin

Only the data entity owner can add metadata to data entities. In this tutorial, you'll add metadata from the accountadmin role.

• Your current role appears below your email address in the top, right-hand corner of the browser.
• If your role is not set to accountadmin, click your username and choose accountadmin from the list under Your roles & privileges.

Step 3: Open the Access control menu

In Starburst Galaxy, tags are part of the Access control menu.

• Expand the Access control menu .
• Select Tags.

Step 4: Create a missions tag

In Starburst Galaxy, you can create tags nested within other tags. This is exactly what you're going to do in this tutorial.

You'll begin by creating a top-level tag called missions. It will help data consumers identify data entities that contain mission information. Afterwards, you'll create two other tags nested inside the missions tag.

• Click Create tag.
• In the Name field, enter missions.
• Type a meaningful Description of the tag. This is used to keep track of tags, so make it as clear as possible.
• Select a color to represent the tag, choosing from the available options below. For example, we have selected teal.
• Click the Create tag button.

Step 5: Create a nested tag under the missions tag

Now it's time to explore how to create nested tags.

You're going to start by creating a personnel tag nested under the missions tag that you created in the previous step.

• Click Create tag.
• In the Name field, enter personnel.
• Select Nested tag under:
• In the Select nesting drop-down menu, choose missions.
• Type a meaningful Description. This can be anything, but should mention its relation to missions.
• Select a color to represent the tag, choosing from the available options below. For example, we have selected yellow.
• Click the Create tag button.

Step 6: Create a second nested tag under the missions tag

Now it's time to create your second nested tag, info. This tag identifies columns that hold general mission information.

• Click Create tag.
• In the Name field, enter info.
• Select Nested tag under:
• In the Select nesting drop-down menu, choose missions.
• Add a meaningful Description. Again, this can be anything, but should mention its relation to missions.
• Select a color to represent the tag, choosing from the available options below. For example, we have selected orange.
• Click the Create tag button.

Step 7: View tags

You've created three tags, but you can only see the missions tag listed in the tags section.

That's because this is the only top-level tag you created, the other two were nested inside missions.

• Expand the missions tag.
• Notice that the personnel and info tags are now shown under missions.
• Notice that the tags pane provides other information, such as the number of times each tag is In-use.

4. Add metadata to tables and columns

Background

Universal search works by using metadata, but not all of your tables and columns have metadata from the outset. Luckily, Starburst Galaxy allows you to add metadata at any point to columns and tables.

In this section of the tutorial, you'll add metadata to the astronauts table and several of its columns. Later in this tutorial, you'll use this metadata with universal search.

Step 1: Select astronauts table in catalog explorer

To add metadata to a table, you need to select the table in the catalog explorer.

Remember that Starburst Galaxy uses the catalog.schema.table hierarchy. You're going to navigate down that hierarchy until you find the astronauts table.

• In the left-hand navigation menu, select Catalogs.
• Expand the sample catalog.
• Expand the demo schema.
• Select the astronauts table.

Step 2: View the metadata for the astronauts table

Starburst Galaxy displays important basic information about the astronauts table.

You can access additional information about the table's metadata by expanding these details.

• Select Show details to see details about this table's metadata.

Step 3: Add a description to the table

Starburst Galaxy shows you several metadata fields that you can edit. Each of these has a pencil icon next to it, allowing you to update or add additional metadata.

You're going to begin by adding to the table description, which is currently empty.

• Edit the Description field by selecting the pencil icon.
• Enter a meaningful Description. For example, "Information on completed space missions."
• Click the Save changes button.

Step 4: Add a tag to the table

Now you're going to add additional metadata to the astronauts table by adding a tag.

Just like before, you can do this by selecting the corresponding pencil icon, this time in the Tags row.

• Edit the Tags field by selecting the pencil icon.
• In the drop-down menu, select the missions.personnel tag.
• Close the drop-down menu by clicking outside it.
• Click the Save changes button.

Step 5: Add a contact to the table

You've added two types of metadata.

Now you're going to do the same thing by editing the Contacts field.

• Edit the Contacts field by selecting the pencil icon.
• In the Contacts drop-down menu, select your username.
• Click the Save changes button.
• Click Hide details to hide the table metadata.

Step 6: Add a tag to a column

Now it's time to pivot towards looking at columns.

Starburst Galaxy lists each of the columns in the table, making it easy to add metadata at the column level. For this tutorial, you're going to add metadata to specific columns in the astronauts table.

Notice that each column in this table already has one tag listed. This is because you added the missions.personnel tag to the whole table, and each column inside the table has inherited it.

Let's add a tag to the mission_number column.

• Expand the mission_number column by selecting the + icon.
• Select the missions.info tag.
• Click Save changes.

Step 7: Add a description to a column

You can add descriptions to columns just like you did with tables.

Let's add a description to the mission_number column to test it out.

• Edit the mission_number by selecting the pencil icon.
• Type a meaningful description in the box.
• Click the Save icon.

Step 8: View the tags summary at table level

Now that you have added tags to the astronauts table and the columns inside it, it's time to explore how Starburst Galaxy reports tag usage.

You're going to start by looking at tags at the table level first.

• In the left-hand navigation bar, expand the Access control menu.
• Select Tags.
• Notice that the missions tag is not listed as in use, denoted by a 0.
• Expand the missions tag by clicking the corresponding arrow.

Step 9: View the tags summary at column level

Notice that the nested tags missions.info and missions.personnel are listed as being in use, denoted by the 1.

Even though each column in the astronauts table inherited the missions.personnel tag, it is only considered to be in-use once because it was added to a single data entity.

Step 10: Manage tag settings

Starburst Galaxy also allows you to manage tag settings for column-level tags. This allows you to see where the tag is being used, remove the tag, and edit any of its settings.

• Click the 1 in the mission_number row.
• Edit some of the description text.
• Click the Save tag button.

5. Video: Universal search in Starburst Galaxy

The following video guides you through the remaining steps in this tutorial. Specifically, it shows you more information about using universal search.

You can choose to watch the video and follow along using your own account. Alternatively, if you prefer, you can skip the video and proceed directly to the step-by-step instructions provided later in the tutorial.

6. Search for data entities with universal search

Background

Universal search allows you to use keywords to find a number of different types of data entities. These include:

• Catalogs
• Schemas
• Tables
• Views
• Columns
• Data products
• Tags
• Owners
• Contacts

String matching

For the search to function properly, the keyword in the search term must match the first part of a data entity's name. It can also match the characters after an underscore.

For example, the search term ‘cust' would return a data entity named customer, but also an entity named profile_customer. However, a search for the term ‘omer' or ‘file' would not return either of these results because the matching occurs only on the first part of strings.

Filtering

When a keyword matches a data entity, you can further filter the results by:

• Datasets
• Data products
• Tags
• Owners and contacts
• Contact
• Owner

Universal search is improving rapidly, and keyword searches will match with more metadata in the future. For even more information, review the documentation.

Step 1: Access universal search menu

Universal search can be accessed in two different ways.

• Option 1 - Click the magnifying glass icon in menu navigation.
• Option 2 - Type / at any time.

Step 2: Begin searching keywords

It's time to test out universal search using the tags you added earlier in this tutorial.

Universal search works like many other search systems and involves keywords. You can choose how you want to filter the results. The default filter is datasets, but you can choose to filter by data products, tags, owners, or contacts.

• In the search field, enter missions.
• Notice that universal search has returned a list of datasets that match "missions".
• You can click any of these to view that data entity. You can also click the link at the bottom to View all results.

Step 3: Search for tags

Now it's time to search for one of the tags you added. This works in a similar way to searching for Datasets, but with one twist.

Let's explore that twist in more detail.

• In the search bar, enter missions.personnel.
• Notice the results section shows No matches. This is because the FILTER ASSETS BY is set to Datasets by default and no Datasets match that search term.
• However, missions.personnel isn't a Dataset, it's a Tag.
• Click the Tags bubble.
• Several results should now be visible.

Step 4: Exploring the search results menu

Universal search provides the best matches first, but sometimes it's necessary to dig deeper. This is when the Search results menu comes in, providing a number of search results and filter options.

• Open the search results menu by clicking View all results.
• Because you were filtering by the missions.personnel tag, the search results pane automatically applies that filter for you.
• You can toggle between a grid and list view.
• You can change the order in which results are displayed.
• You can navigate to the next set of results, if the number of assets found is greater than the number displayed on the first page.

Step 5: Change the search filter

Search filters can be updated. You can filter a search by asset type, catalog, tag, contact, and owner.

Right now you are filtering on the missions.personnel tag. Let's change the filter type to see how the results change.

• Deselect the mission.personnel tag to remove this filter.
• Select the missions.info tag to add this filter.
• Notice that fewer assets are now displayed.

Step 6: Using the catalog explorer search field

You can also use the Catalog explorer to search for catalogs, schemas, tables, views, and columns.

This type of search works in a similar way to universal search, and uses the same matching process.

• In the left-hand navigation menu, select Catalogs.
• Select the search bar at the top of the Catalog explorer.

Step 7: Enter a catalog explorer search

Time to try your first search in this field.

• In the search bar, type cust.
• Notice that results are displayed hierarchically, showing catalogs, schemas, tables, and columns as nested entities.
• Notice that any catalog without a match is omitted.
• Notice that matches are highlighted in yellow.
• Select any of these entities to view more information about it.
  
  Note: For columns, you must view the associated table.

7. Tutorial wrap-up

Tutorial complete

Congratulations! You have reached the end of this tutorial, and the end of this stage of your journey.

Now that you've completed this tutorial, you should have a better understanding of just how easy and convenient it is to use universal search in Starburst Galaxy.

Continuous learning

At Starburst, we believe in continuous learning. This tutorial provides the foundation for further training available on this platform, and you can return to it as many times as you like. Future tutorials will make use of the concepts used here.

Next steps

Starburst has lots of other tutorials to help you get up and running quickly. Each one breaks down an individual problem and guides you to a solution using a step-by-step approach to learning.

Tutorials available

Visit the Tutorials section to view the full list of tutorials and keep moving forward on your journey!