Optimizing data operations with a lakehouse approach

About: 

Yello, a talent acquisition software company that works with Fortune 500 companies to attract and nurture top talent, embarked on a journey to modernize its data platform to provide faster insights for their internal and external clients. The previous platform failed to provide a single source of truth and had difficulty maintaining data stewardship. Yello aimed for a flexible and agile data architecture that could serve both clients and internal stakeholders efficiently. 

Challenge: 

Yello’s previous data infrastructure was an outdated legacy data warehouse built on a Postgres database within AWS. This setup was complex, prone to failures, and demanded constant maintenance. Their legacy warehouse also lacked standardization, with each client having its own data instance and varying data naming conventions.

Shawn Crenshaw, Director of Data at Yello, recalls, “The data was not fresh because it was only refreshed four times a month. It was very cumbersome. We couldn’t provide true insights to our product and customer areas. And we didn’t have a single source of truth, which is so important.” This hindered Yello’s ability to derive actionable insights and provide timely support to their clients.

Moreover, the absence of a semantic layer and self-serve capabilities limited data access and prevented the development of comprehensive analytics solutions.

Solution: 

Yello chose Starburst Galaxy as the end-to-end analytics platform for their data lakehouse, due to its powerful query engine and seamless integration with Iceberg’s open table format. Yello aimed to consolidate its data assets, improve data access, and accelerate insights delivery. Starburst Galaxy’s cloud integration and support for open standards aligned perfectly with Yello’s vision for a scalable, cost-effective solution.

The new architecture comprises separate databases for each client, change data capture, Amazon S3, and Starburst Galaxy as the computation layer. Tools like dbt and Dagster are deployed to transform and orchestrate, respectively. This architecture facilitates faster access to client data, improved dashboard rendering, and streamlined reporting processes.

Results: 

Shifting from a legacy data warehouse to a data lakehouse with Starburst Galaxy and Iceberg resulted in several outcomes for Yello:

Improved data health and accessibility: Reporting times have been cut from eight hours to just minutes, achieving a 98% improvement. Additionally, enhanced data storytelling capabilities allow for more compelling and comprehensible presentations.

Increased efficiency: The total combined compute run time has been reduced from 28 hours to 4 hours. These improvements ensure a faster time to production, enhancing client experience and satisfaction.

Enabled self-service capabilities: Starburst Galaxy allows Yello’s product and customer success teams with self-service analytics to make data-driven decisions without relying on the data team.

Customer-facing applications: To improve overall customer satisfaction, Yello plans to embed dashboards powered by Looker into their new architecture for both internal analytics and customer-facing experiences., .

Enhanced revenue generation: The new data platform elevates  Yello from a cost center to a revenue generator, allowing the company to monetize its data services and strengthen its financial sustainability and growth prospects.

Cost savings: “The move to Starburst and Iceberg has resulted in a 12X reduction in compute costs versus our previous data warehouse. This efficiency allows us to focus our attention on using analytics for revenue-generating opportunities,” shares Peter Lim, Sr. Data Engineer at Yello.

Ultimately, transitioning to Starburst Galaxy not only reduces tech debt and outages, improves scalability, and enhances data consistency but also elevates overall customer satisfaction. With the new solution in place, Yello is well positioned for future initiatives such as machine learning.