Maximizing the value of data and achieving over 100% faster time-to-insight

About

This pioneering cloud computing and virtualization technology company is a leading provider of multi-cloud services for all applications. The company offers leading enterprises a foundation to accelerate innovation and digitize their businesses so they can deliver a better customer experience. After moving away from an on-premises data warehouse to a complete SaaS platform, the cloud computing company wanted a more powerful, cost-effective way to query distributed data.

Challenge

The company faced several challenges that made querying petabytes of distributed data difficult. 

• Engineers had to write their own code to do a join across indexes 
• There wasn’t a viable solution for Elasticsearch indexing on the market
• There was no connectivity to Elasticsearch from Amazon Athena
• Every use case demanded the creation of custom logic in the system, forcing the engineering team to spend time processing data before storing it 
• The company didn’t have a good solution for querying the data stored in Amazon S3

Furthermore, Amazon increased Elastic Cloud Compute (EC2) costs, lacked role-based access control and Apache Ranger integration, and produced a high query failure rate. The cloud computing provider needed a platform that offered security, reduced costs, and successfully queried Elasticsearch, Redshift, and S3.

Solution

The cloud computing provider evaluated different solutions such as Open Distro, but found them to be too limited. Starburst Enterprise met all of the primary requirements and provided a user-friendly solution that included:

• High-performance connectors to Elasticsearch, Redshift, and Amazon S3
• Stronger security through Role-Based Access Control (RBAC) & Lightweight Directory Access Protocol (LDAP) support
• Support for 50+ production-tested, high-performance connectors to leading databases, data lakes, and other systems
• Separate storage and compute, allowing the company to scale each independently for better performance
• Ability to run joins across indexes, a functionality that ES doesn’t provide
• Fine-grained access control over all connected data sources, including column- and row-level masking, and Apache Ranger integration
• 24/7/365 support from leading Trino experts

Results

Starburst Enterprise is helping impact the cloud computing company’s larger mission by unlocking the value in its distributed data. Starburst provides the company with the following.

Enhanced analytics. The company gets new insights by quickly joining and querying across more data and by running analytics on the time series data and other data stored in Amazon S3.

Faster time-to-insight. Analysts and data scientists can engage their curiosity immediately and find answers to novel questions in seconds instead of days, reducing time-to-insight by over 100%.

Speed at scale. The company runs queries against tables with roughly 800 million records and sees results of thousands of documents in a second or two. 

Single point of data access. The company can access all of its distributed cloud data with Starburst, simplifying work for the engineers.

The company has also realized a:

• 100% faster time to insight – going from days query data to accessing novel insights in seconds
• 50% reduction in total cost of ownership through enhanced data access & elimination of complex ETL and processing operations
• 30% reduction in EC2 spend due to autoscaling and Starburst’s cost-based optimizer 
• 99.9% query success rate, compared to a high failure rate in Amazon Athena

Overall, Starburst Enterprise represents a huge win for the data architecture and engineering teams, who can process all of the company’s data and provide better, faster performance to the company’s users.