Delivering telemetry data in near-real time to detect cyber threats

About

This customer provides its clients with comprehensive cyber threat protection, rapid ransomware recovery, and secure data backup. Their solutions are aimed at improving administrative efficiency for Managed Service Providers (MSPs), which are third-party companies that remotely manage a customer’s IT infrastructure. Responding to triggers in Endpoint Detection and Response (EDR) technology is critical to this customer’s success and meeting their clients’ strict SLAs. With their EDR being slowed down by scalability limitations with AWS Athena, the company decided to switch to Starburst Galaxy.  

Challenge

The cybersecurity and data backup company provides an EDR solution that continually monitors endpoints, or computer systems, to mitigate malicious cyber threats in near-real time. The engineering team started to try AWS Athena as the analytics engine for its end point detection technology, and hired a specialized data architect to re-do the data platform. However, inherent limitations with Athena, such as concurrency limitations, query failures, and a poor user experience, led the data architect to seek out an alternative solution. Queries were taking anywhere between an hour to a half day to run and would then time out. “Any time I hear that a query is running for two hours and is not part of an ETL job, I know something is very wrong,” says the data architect team lead. If they continued with the way things were, they would need to hire an additional two to three full-time engineers to manage Athena, the data architect estimates. 

Solution

The engineering team decided to evaluate alternative options, including open-source Trino. As a data platform team of one, the data architect didn’t want to be solely responsible for maintaining the Trino cluster. That’s when the team came to evaluate and ultimately choose Starburst Galaxy, the cloud native and fully managed service of Starburst’s massively parallel processing (MPP), highly performant query engine. Starburst Galaxy offers the flexibility to run interactive and ELT workloads in one query engine with easy and fast access to the data lake and beyond, federating with other data sources. The deciding factors for choosing to deploy Starburst Galaxy were ease of use and total cost of ownership (TCO) savings. 

Results

The customer deployed and onboarded Starburst Galaxy within a few weeks, using it as an embedded analytics solution within their EDR application. “Starburst Galaxy is our single point of access for data, so it’s the one-stop shop for any kind of use case,” explains the data architect. “It’s not just for analytics, but it’s also supporting net new features in our application, and we’ll be implementing a new search functionality through Starburst Galaxy.” 

Within six weeks of deploying the solution, the sales team generated over $1 million in new revenue by onboarding thousands of new endpoints that leverage Starburst Galaxy to gain access to near-real time telemetry data. The revenue generated represents an overall 57% increase in annual recurring revenue (ARR) for the new EDR application. Additionally, the sales team is expecting a 10X customer growth before the end of the year as a result of the new solution. “Whenever there’s a ransomware or security breach, a client calls the us for help, and we point them to use the end point detection response to detect the threats that are happening in near-real time,” the data architect explains.

Beyond customer expansion and revenue generation, the customer notes other areas the solution is impacting the company: 

Cost savings

The cybersecurity company reduces costs by more than $450,000 annually through the implementation of Starburst Galaxy. The savings come from eliminating the need to hire two to three full-time equivalents (FTEs) to manage the query tool (~$350,000 per year, combined), and $100,000 reduction in resource costs by switching from Amazon Athena to Starburst Galaxy. By reducing the amount of database servers deployed for all of their tenants’ telemetry and usage data, they’ve also reduced their total AWS spend by 15%. Starburst Galaxy enables the customer to optimize their storage costs through their data lake in AWS S3, leveraging the analytic capabilities of Starburst on their telemetry data to achieve better insight, with less cost.

Faster time-to-insight 

Batch workloads running in Starburst Galaxy are 4-5X faster than Apache Spark, and overall they’ve seen a 500% accelerated query performance compared to the previous solution. “Aggregation queries that were being run previously [against Postgres] would time out after four and a half hours, and those are now completed in less than four and a half minutes,” shares the data architect. 

Better customer experience 

For the company’s largest customers, such as hospital systems or school districts, that have the EDR software installed on thousands of workstations to monitor security risks, having Starburst Galaxy as the data lake analytics platform allows customers to access and monitor their data in less than a minute. Sales now has the confidence to support new customers because they have the infrastructure in place to support it. At the end of the day, this allows the cybersecurity company to deliver a greater value proposition with faster threat detection. 

Self-service analytics 

Starburst Galaxy serves both users of the customer’s threat detection solution and internal ad-hoc analytical tools. By building a data platform that has roles and policies for data access, the engineering team is fostering an ecosystem where users can self-serve. The analytics solution enables an environment that serves all the different personas of data users, whether they prefer Excel, another visualization tool such as Apache Superset, or want to use the API directly.