Harnessing Trino for performance engineering and data analytics at Salesforce

Trino’s value: Reducing Saleforce’s analytics costs

Our performance  team’s access to various data sources — log events, transactional records, etc — has significantly decreased the overall cost to build advanced analytics and anomaly detection models. Moreover, log analysis provides crucial insights into potential issues before they occur, empowering teams to be proactive by identifying root causes and preventing downtime and performance issues.

By extracting more value from production data, accelerating access, we’ve been able to facilitate faster decision-making, simplifying operations, and automated anomaly detection, ultimately leading to improved cost efficiency. Salesforce has shortened the time required to understand user behavior, enhance application and infrastructure performance, proactively mitigate risks, and ensure compliance with security policies, audits, and regulations.

Trino: Faster than querying similar queries from Splunk

Additionally, after our team transitioned from Splunk to Trino, we observed a remarkable 194% improvement in query execution time. 

This significant enhancement is crucial for implementing production analytics as it enables rapid access to insights and actionable data, ensuring the maintenance of trust with customers. Trino has played a pivotal role in achieving these goals by providing quick and efficient access to vital information, benefiting the team in optimizing analytics for production processes.

Near real-time availability with extended historical data availability

Trino has proven to be a highly efficient tool, with Salesforce’s internal Trino team  offering an impressive Service Level Agreement (SLA) of 20 minutes latency on all production logs, providing near real-time data availability. This real-time data plays a crucial role in enhancing productivity and operational efficiency.

The analytics derived from Trino enable the generation of concise reports that cut through the noise generated by vast data collections. Performance engineers and service owners benefit from these easily digestible reports, allowing them to pinpoint the precise information they need for building prediction and forecasting models, as well as assessing performance improvements or degradations over time.

An essential aspect that Trino addresses is the need for historical data (remember the tens of millions of transactions per day?). Unlike the previous data source, Splunk, which had a 30-day retention policy, Trino enables retention and access to data for up to two years. This extended historical data availability facilitates the examination of past and current data, enabling the prediction of performance profiles, seasonality, and daily trends for Salesforce’s customers. By foreseeing scale, capacity, and performance issues well in advance, Trino helps the organization mitigate potential problems proactively.

Moreover, Trino simplifies the process of gathering data from different sources, including app logs, metrics, business logs, and audit trails. Previously, numerous queries were required to collect information about user activity across all production systems. However, with Trino, these queries have been efficiently replaced with just a few aggregated sequels, resulting in significant improvements and streamlining data analysis, saving time and resources for the team.

How Trino can help Salesforce achieve in the future

Trino holds tremendous potential to help Salesforce achieve various goals in the future. 

Performance assessment: Understanding our performance improvements and degradation over multiple releases

With its capability to store data for two years and provide easy and efficient access, Salesforce plans to assess performance changes over time for specific services across multiple releases. This analysis will involve exploring key performance indicators (KPIs) such as log volumes, JVM Heap usage, and resource usage on pods, enabling the identification of areas for optimization and improvement.

Quick and efficient Processing: Queries are able to run across more than one production pod at once

Moreover, since data is stored across all geographic regions for both first-party and high-performance cells, Salesforce can extend its models and analytics to include all customers, not just the prominent ones. This expansion will be valuable in understanding the impact of growing customer usage on underlying resources, facilitating proactive prediction and scaling without negatively impacting customers.

Integrating with Tableau dashboards

Our performance  team is also working on integrating the data set seamlessly with Tableau, Salesforce’s primary dashboard solution. This will simplify information sharing across teams and enhance data visualization capabilities for better insights.

Real time anomaly detection

With Salesforce’s internal Trino team providing an 20 minute SLA on log latency, we are able to achieve real-time anomaly detection.  20-minute lag Service Level Agreement (SLA) ensures near real-time anomaly detection. This empowers performance engineers with immediate data access for troubleshooting and resolving issues swiftly, without any adverse impact on customers.

Salesforce x Trino: Shaping the future of performance engineering and enhancing customer experiences

Salesforce’s integration to Trino would not have been possible without their internal teams, particularly the Huron team.   Our performance team’s collaboration with the Huron team has been crucial in enabling Salesforce to leverage Trino’s capabilities effectively.

In short, Trino has revolutionized the way Salesforce approaches performance engineering, analytics, and anomaly detection. Its efficiency, scalability, and real-time data access have been invaluable in delivering top-tier services to customers and driving continuous improvement. 

Salesforce’s partnership with Trino is set to shape the future of performance engineering at the company, bringing even more exciting opportunities for enhancing customer experiences.