Databricks vs. Starburst

Some additional exploration

What is Starburst Galaxy?

Starburst Galaxy brings the open data lakehouse architecture to life using open file and table formats and an optimized open MPP SQL query engine with OS Trino. Starburst Galaxy is used for both interactive ad-hoc analytics and long-running workloads like batch and ETL/ELT, and offers high scalability and query completion rates even as the amount of data (terabytes of data), query volume, and query complexity increases. It runs federated queries across data lakes, cloud data warehouses, on-premises databases, and relational data management systems like PostgreSQL and MySQL. Galaxy also supports enhanced fault-tolerant execution, smart indexing and caching, Data Products, universal search and schema discovery while truly separating compute and storage between Starburst and your cloud storage.

What is Databricks SQL?

Databricks, traditionally known for its Apache Spark origins and data science and machine learning capabilities, introduced Databricks SQL, available as a serverless and non-serverless data warehouse in the Databricks Data Intelligence Platform (formerly known as the Lakehouse Platform) to run SQL and BI applications using its proprietary Photon engine. It offers a governance model via a proprietary data catalog, use of open formats (Delta Lake being the preferred for a first-class experience), APIs, and integrations with third party tools. It provides general compute resources for SQL queries, visualizations, and dashboards that are executed against the tables in data in cloud object stores. It also enables find and share insights with the built-in SQL editor, visualizations and dashboards.

What is Databricks SQL used for?

Unlike traditional SQL warehouses, Databricks SQL is a platform that provides SQL data warehousing capabilities on data stored in a cloud data lake. It is primarily used for data exploration, ad hoc analytics (without the need of data pipelines) and interactive big data analytics. The platform allows users to query data from a limited number of federated data sources using non-optimized connectors and requiring the use of their proprietary data catalog, Unity Catalog, for an optimal data federation experience.

Databricks SQL also offers its own visualization tooling and integration with various visualization BI tools such as Tableau, PowerBI, ThoughtSpot, Looker, and others to make it easier for data analyst to consume the data.

What is a Databricks workspace?

A workspace within the Databricks platform is an environment for data engineers, data science teams and others to access Databricks resources with the option of running a single or multiple workspaces within your Databricks account.

What type of SQL is used in Databricks?

Similar to Starburst, Databricks SQL uses the American National Standards Institute (ANSI) for SQL, which is a common standard for most SQL databases.

Can I write SQL in Databricks?

Yes. Similar to Starburst Galaxy, within Databricks SQL there is an environment via SQL endpoints for running SQL queries, creating dashboards with visualization tools, and sharing query results.

Can I write Databricks SQL in a text editor?

Yes, similar to Starburst Galaxy, you can write your SQL queries or scripts in any text editor of your choice and then copy and paste them into Databricks SQL or a Databricks notebook for execution.

Is Databricks SQL a program?

No, Databricks SQL is not a program. It is a SQL tool within the Databricks Lakehouse platform.

How does Azure Databricks differ from Databricks on AWS or GCP?

Databricks on Azure or Azure Databricks is the result of a collaboration between Microsoft and Databricks to bring Databricks on as a first party service to Microsoft Azure. The core functionality of Databricks remains the same across the three hyperscale cloud providers but there are some differences in how it is integrated with each cloud provider’s services.

• Azure Databricks
  • Azure treats Databricks as a first party service and is built and supported in collaboration with the Databricks team.
  • Azure Databricks can be integrated into the Azure portal and can be provisioned with a single click.
  • Azure Databricks is highly optimized for the Azure environment as a result of the shared engineering, is tightly integrated with other Azure services.
  • Time will tell how Azure Databricks evolves with Azure Synapse and Azure Fabric.
• Databricks on AWS and GCP
  • Databricks on AWS or GCP is not a first party service like Azure Databricks. Therefore, similar to other SaaS tools, it requires a bit more setup and understanding of AWS and GCP services like IAM (for AWS) and networking.