Accelerating data transformation, enhancing analytics, and future-proofing the organization

About:

The financial institution is one of the world’s largest banking and financial services organizations. The bank serves approximately 39 million customers worldwide in 62 countries and territories. The bank embarked on a data transformation project with Starburst and Google Cloud to tackle complex data landscapes, suboptimal Hadoop estates, stringent data lineage requirements, and multi-cloud data silos.

The result is a future-proofed data infrastructure that accelerates data processing and analysis, enhances compliance, reduces cost, and fosters innovation through self-service analytics, giving the bank a competitive edge.

Challenge: 

The financial institution was heavily relying on-premise Hive and Spark workloads which were great for ETL but not for analytics. Analytics teams relied on the Hadoop data lake for every use case that they were tasked to deliver – from product recommendations, risk profiles, pricing profiles, customer touch points, through bonding.

It took hours to perform analytics queries and there were high volumes of inconsistencies in the data copying which meant that data could not be read fast enough. Data engineers were duplicating and moving data unnecessarily, wasting storage space and licensing. Every time production requests were made to move the data, this resulted in wasted man hours and delayed insights (up to 6 weeks).

In addition to optimizing data processing for analysis, the bank also wanted to meet data sovereignty requirements, improve data lineage, and future proof their data infrastructure to fuel future innovation.

Solution: 

To remove the data bottleneck in Hadoop, the bank embarked on a modernization journey with Google Cloud Platform. It also needed a solution to facilitate federated queries between siloed systems and speed up ad-hoc analytics on the Hadoop data lakes. The institution ultimately chose Starburst Enterprise over Denodo and other vendors due to stronger performance with Starburst’s data lake analytics platform. 

The project incorporated several innovative solutions to tackle existing data management problems, including:

• Strategic data lakehouse design: Partnering with Starburst, the project has adopted a federated data virtualization strategy, transforming the on-premise platform into a strategic data lakehouse design across hybrid clouds. This design enables seamless data management, distribution, and analytics across various data sources, breaking down data silos and enhancing data accessibility.
• Starburst Stargate: The project has included Starburst Stargate to effectively manage the data distribution flow across geographical borders, utilizing regional clusters. This approach aligns with the latest data sovereignty requirements and allows the business to rapidly analyze critical data, while ensuring compliance with data localization laws across different countries.
• Scalability and performance: Starburst’s ability to scale effectively on-premise and in the cloud provides a robust foundation for the bank to meet increasing business demands. The materialized view feature delivers higher throughput for low-latency and critical applications, ensuring optimal performance.

End-to-end data lineage: The project has successfully integrated Starburst’s capabilities to complete the last mile of data lineage, from data assets to the consumption layer. This integration provides an end-to-end usage tracking solution at the user and use case level, enhancing the overall data governance and management process.

Results: 

The project successfully delivered on the desired business value – to enable better informed decision-making, meet data sovereignty compliance, and reduce the costs of their on-premise data infrastructure. By partnering with Starburst and GCP, the bank has substantially improved the ability to achieve their core goals in several ways: 

• 20X faster data querying: The accelerated data query feature has drastically reduced the time needed to generate insights from data (from over an hour to under three minutes). This improvement enables the organization to respond more quickly to business needs, adapt to market changes, and make data-driven decisions faster than ever before.
• Enhanced data accessibility: By breaking down data silos and integrating data across multiple sources, the bank has improved data accessibility for employees, leading to better-informed decision-making and more efficient business processes.
• Robust data governance and compliance: The project’s focus on end-to-end data lineage and alignment with data sovereignty requirements has strengthened the organization’s data governance practices. This enhancement ensures compliance with data localization laws and builds trust among customers, staff, and partners.
• Cost savings: The transition from an on-premise Hadoop estate to a hybrid cloud-based data distribution service has led to significant cost savings, including an estimated $200K savings per use case that’s run through Starburst*. The organization can now allocate resources more efficiently and invest in other critical areas to achieve its goals.

*Cost savings estimated from eliminating 6 weeks of work from each workload by no longer requiring data movement and data pipelines.

The success of the data modernization project has laid the groundwork for continuous innovation in data management and analytics, ensuring that the organization remains at the forefront of technological advancements and is well-equipped to embrace future opportunities and challenges.