Powering supply chain efficiency with a lakehouse architecture

About

7bridges is a leading AI data-driven supply chain management platform. The solution gives businesses the ability to understand the impact of every decision they make – gathering all their data in one powerful platform so they have full control and end-to-end visibility of their supply chain. Their AI lets businesses test out new ideas and adapt to real world disruptions, giving them more room to innovate. 7bridges is committed to reinventing supply chain management in a world that needs supply chains that are integrated, data first, responsive and resilient so they can adapt quickly and operate efficiently.

Challenge

As 7bridges continued to grow, the realization emerged that an increasing number of customers and internal teams were operating with individual databases, making it difficult to access data in one place. As incoming data volumes increased and became more diverse, the company also recognized that this could cause limitations that could negatively impact the customer experience and the smooth operations of the platform over time. Some of the limitations included:

• Slower query execution resulting in data accessibility issues.
• Non-technical clients and internal users struggling with data access and queries, leading to delays in obtaining insights.
• Depending on third party platform services made fault tolerance and debugging challenging.

The existing data infrastructure included an array of relational databases and a primary data platform (PostgreSQL). The platform worked for existing workloads; however, it was clear that 7bridges needed to take their services to the next level in order to meet the evolving demands of both their clients and internal users.

Solution

Deploying a lakehouse architecture with Starburst Galaxy

7bridges required a data management approach that could cater to both rigid, well-defined business questions and more unpredictable data science-oriented tasks. The traditional data warehouse approach and the unstructured data lake approach each had their strengths, but were difficult to reconcile when it came to exchanging information and handling complex scenarios. The lakehouse architecture emerged as a viable solution.

The company initially designed the lakehouse architecture in Delta format on Amazon S3; however, both read and write capabilities were needed to build a self-service environment. 7bridges initially ran a proof of concept with Trino and deployed Trino in Kubernetes. While the engineering team had the expertise required to manage the platform, limited time and resources made hosting and managing Trino unrealistic. That’s when they discovered Starburst Galaxy.

“We chose Galaxy because of the flexibility it offers to connect to so many different types of tools and data sources. Galaxy allows us to use Lakehouse tables for both transformations and reporting, and on top of that, Galaxy provides access to multiple data formats. This ensures that we can stay flexible and iterate quickly as the Lakehouse technology evolves,” shares Simon Thelin, lead data engineer at 7bridges.

Results

With the lakehouse platform deployed with Galaxy, 7bridges has realized significant value. The transition has enhanced the overall client experience by improving data accessibility, speeding up query execution, and streamlining decision-making processes.

Key outcomes and metrics:

1. Faster query execution. Reports now execute in minutes or less, representing a 98% improvement in reporting speed. This improvement allows clients to obtain critical insights swiftly, leading to more agile and data-driven decision-making.
2. Faster development cycles. The ease of data exploration and manipulation in Galaxy enables teams to test new ideas and features rapidly, accelerating the time-to-market for innovative solutions.
3. Enhanced client satisfaction. The time to access data has been significantly reduced, and clients have more options in visualizing historical data. These improvements result in increased satisfaction with the platform’s performance and usability.
4. Cost savings and efficiency. The lakehouse architecture, coupled with Galaxy’s fault tolerant execution and reliable compute capabilities, has optimized infrastructure costs while delivering insights faster.
5. Data accessibility for non-technical users. The dbt and Galaxy integration allows non-technical users and business analysts to interact with data efficiently for R&D and internal reports, reducing dependency on data engineers, and accelerating the data-to-insight process.
6. Flexibility for diverse use cases. Galaxy’s ability to connect to various data formats and sources has provided 7bridges with the flexibility needed to address their diverse supply chain use cases.

Overall, the adoption of Galaxy’s lakehouse analytics platform has contributed to the ease at which 7bridges’ clients are able to interact with their supply chain data. With faster and more accessible insights, clients can now make data-driven decisions swiftly, optimize supply chain processes, and gain a competitive edge. The successful deployment of Galaxy has solidified 7bridges’ position as a leader in the industry, providing cutting-edge solutions that elevate supply chain management to new heights.

More resources: Building a data analytics platform with a lakehouse at 7bridges