Scale beyond Snowflake with Starburst

Leverage the Starburst data lakehouse alongside Snowflake to increase productivity and decrease costs

Starburst empowers data teams to quickly support new workloads in a cost-efficient manner by letting teams performantly query data in object storage. With Starburst running alongside your cloud data warehouse, you can choose which workloads to push into your data warehouse, adding openness and flexibility to your data architecture while reducing overall costs by 50-75%.

Some additional exploration

What is Starburst?

Starburst Galaxy is a price-performant multi-cloud open data lakehouse powered by Trino, a leading open-source distributed MPP SQL query engine. Starburst Galaxy is used for interactive ad-hoc analytics, long-running workloads like batch and ETL/ELT, streaming, automated data maintenance, and offers high scalability and query completion rates even as the amount of data (petabytes of data), query volume, and query complexity increases. Starburst runs federated queries across data lakes, cloud data warehouses, on-premise databases, and relational data management systems like PostgreSQL and MySQL. Galaxy also supports enhanced fault-tolerant execution, smart indexing and caching, Data Products, machine learning (PyStarburst), universal search and schema discovery while truly separating compute and storage between Starburst and your cloud data lake.

What is Snowflake?

Snowflake is a data warehouse in the cloud brought to life in the Snowflake Data Cloud. It is built on top of Amazon Web Services and also runs on Microsoft Azure and Google Cloud. Snowflake offers a fully managed solution with no hardware or software needed to install, configure, or manage. Similar to Starburst, Snowflake powers multiple data workloads, from data warehousing, data Engineering, AI and machine learning, data applications, and cybersecurity across multiple cloud providers and regions from anywhere in the organization. Also, Snowflake separates compute and storage, but both are managed, billed, and executed within the Snowflake platform.

What does the Snowflake data cloud do?

At its core, Snowflake’s data cloud architecture is a data warehouse that is optimized for a cloud-based architecture. It allows the movement and transformation of data from storage (S3) to its proprietary and closed cloud data warehouse. This provides a holistic data cloud for elastic data management and consumption. Snowflake is not just a storage solution but a tool that provides performance, relational querying, security, and governance for data that resides within the confines of its digital walls. It can be used to consolidate structured and semi-structured data and power transformations, analytics, and reporting using a highly custom SQL.

In addition to handling structured and semi-structured data, Snowflake recently announced support for unstructured data in a data warehouse architecture. By incorporating unstructured data, Snowflake also aims to expand into data science and machine learning use cases within a data warehouse architecture.

Is Snowflake a cloud data platform?

Yes, Snowflake is indeed a cloud data platform. It provides a data warehouse as a service designed for the cloud. This platform allows businesses to store and analyze data using cloud-based hardware and software with both storage and compute within the Snowflake account. With Snowflake, businesses can handle several aspects of data storage, including performance, scalability, and security, once the data is loaded into Snowflake’s proprietary and closed storage.

Is Snowflake the same as AWS?

The Snowflake platform is similar to a combination of AWS services but does not provide the full breadth of functionality of all of AWS combined. Consider Snowflake similar to combining Amazon S3, Amazon Redshift, Amazon Glue, Amazon EMR, and Amazon Athena, along with several other security, governance, and management services in a highly closed and propriety platform.

How does Snowflake work?

Snowflake operates a cloud data warehouse architecture. At the heart of this are virtual warehouses, which are essentially clusters of compute resources. In order to use Snowflake, there are multiple checkpoints. It’s typically recommended to first stage your data in a cloud data lake and then ingest it into Snowflake. Once your data is in Snowflake’s proprietary storage, you can transform the data within the platform for your analytical needs.

Within the Snowflake cloud data warehouse, compute resources are separate from storage, meaning you can scale them independently based on your needs. This separation ensures that large queries won’t slow down smaller, more urgent ones and vice versa. It also means that you only pay for the compute resources you use. When queries require more nodes for processing, instead of adding more nodes to the existing cluster, Snowflake opts to add another cluster, doubling the total compute resources whether or not all the resources are needed.

Data outside of Snowflake can be queried using External tables, unmanaged Iceberg Tables, and their propiartaary managed Iceberg Tables.

Similar to Starburst, the Snowflake architecture is designed to be secure, fast, and easy to use.

What are the benefits of the Snowflake data cloud?

The Snowflake data cloud offers its users many benefits align with the approach taken by Starburst.io. For Snowflake customers, the ability to handle all types of data in one place simplifies data management and allows for more comprehensive analytics, a benefit that is also central to Starburst’s approach.

Metadata handling is another significant advantage of Snowflake, with automatic collection and management of metadata making it easier for users to understand and use their data effectively. Similarly, Starburst also emphasizes the importance of metadata in its approach.

The on-demand nature of Snowflake’s compute resources is a major advantage, allowing users to scale up or down instantly based on their needs. This mirrors Starburst’s emphasis on flexibility and cost-effectiveness.

Finally, Snowflake’s proprietary architecture allows for high levels of concurrency, meaning multiple queries can be run simultaneously without affecting each other’s performance. This is a capability that is also highlighted in Starburst’s approach, with its platform built for analyzing large amounts of distributed data with high concurrency using an open-source foundation.

In summary, both Snowflake and Starburst.io offer powerful solutions for data management with many similar benefits, including handling all types of data, effective metadata management, on-demand resources, and high levels of concurrency.

What challenges exist with the Snowflake data cloud for SQL workloads?

While Snowflake’s data cloud offers many benefits, it also presents certain challenges for SQL workloads. First off, Snowflake is difficult and excessively time-consuming to get started. It requires that all your data is first ingested into their closed and proprietary storage, which can take years. Furthermore, general Snowflake best practices, first have you move your data into a cloud data lake staging environment before it gets sucked up into Snowflake. This means you’re paying for data movement twice.

For Snowflake customers, one of the key challenges is building effective data pipelines. The scalability of Snowflake can lead to the ingestion of excessive amounts of data, which can increase storage costs and potentially degrade the quality of the data.

Another challenge is related to workload management. Snowflake’s multi-cluster architecture allows for high levels of concurrency, but it can also lead to increased complexity in managing and optimizing workloads. For instance, joining large tables of data directly from raw to presentation layers can cause workloads to run for hours and add significant costs to the warehouse.

Next, while Snowflake provides a lot of automation, it still requires some manual intervention. For example, decisions around whether to use Extract, Transform, Load (ETL) or Extract, Load, Transform (ELT) need to be made. Additionally, while Snowflake automates many aspects of data management, it still requires DBA efforts.

Customers of Snowflake of all sizes will also regularly cite costs going beyond forecasts very quickly as data volume increases, premium features are activated, and inefficient performance enhancement practices, i.e., doubling of clusters for autoscaling rather than adding the necessary amount of additional worker nodes to existing clusters.

Lastly, Snowflake is a highly closed and proprietary platform that is not only difficult to get all your data into, but once in the platform, the proprietary storage and custom SQL make it extremely difficult and costly to offload workloads from Snowflake as data and query volumes grow.

Is Snowflake ANSI SQL compliant?

Yes, Snowflake is ANSI SQL compliant. This means that all of the most common operations are usable within Snowflake. Snowflake also supports all of the operations that enable data warehousing operations, like create, update, insert, etc. In addition to that, Snowflake supports a subset of ANSI SQL:1999 and the SQL:2003 analytic extensions. However, it’s important to note that though Snowflake is ANSI SQL compliant, Snowflake operates its own highly proprietary version of SQL built on top of ANSI standards with 100+ custom functions, making the scripts difficult to migrate out of Snowflake as volumes increase.

How to do data sharing with Snowflake?

Data sharing is limited to within the Snowflake ecosystem, which allows you to share selected objects in a database in your account with other Snowflake accounts. Here’s how you can do it:

1. Create a Share: The provider creates a share of a database in their account and grants access to specific objects in the database. The provider can also share data from multiple databases, as long as these databases belong to the same account.
2. Add Accounts to the Share: One or more accounts are then added to the share, which can include your own accounts (if you have multiple Snowflake accounts).
3. Use Reader Accounts: If you want to share with people who don’t have Snowflake accounts, you can use Reader Accounts.

In addition to the above, Snowflake offers other options for data sharing:

• Listing: You can offer a listing privately to specific accounts, or publicly on the Snowflake Marketplace.
• Direct Share: Use a Direct Share to share data with one or more accounts in the same Snowflake region.
• Data Exchange: If creating listings that you offer privately to specific accounts isn’t an option, you can use a data exchange to share data with a selected group of accounts that you invite.

These features enable an ecosystem of data sharing and collaboration exclusively within Snowflake. At a high platform cost, they allow for the creation of data applications and apps that leverage shared data for reporting.

What are the layers of the Snowflake architecture?

Snowflake cloud data warehouse consists of three core layers:

1. Database Storage Layer: When data is loaded into Snowflake from your cloud data lake staging environment, it reorganizes that data into its proprietary internal optimized, compressed, columnar format that is built on top of a cloud data lake. Snowflake stores this optimized data in cloud storage. All aspects of how this data is stored — the organization, file size, structure, compression, metadata, statistics, and other aspects of data storage are controlled by Snowflake.
2. Query Processing Layer: This layer is responsible for executing SQL queries. Similar to Starburst and its use of OS Trino, Snowflake processes queries using MPP (massively parallel processing) compute clusters where each node in the cluster stores a portion of the entire data set locally. This approach offers the data management simplicity of a shared-disk architecture, but with the performance and scale benefits of a shared-nothing architecture.
3. Cloud Services Layer: This layer provides services such as infrastructure management, metadata management, query parsing and optimization, access control, and more. It coordinates and handles all transactions and sessions, ensuring that all operations are secure and ACID-compliant.

Beyond the three core components, there is also a built-in visualization component and data marketplace.