Implementing a data mesh architecture for faster data access and insights

About

Founded in 1999 as the UK’s first satellite TV service, Sky is Europe’s leading media and entertainment company. A subsidiary of Comcast Corporation, Sky connects 23 million customers across six countries through apps, entertainment, sports, news, original content, and the brand’s new streaming service, Sky Glass. This tremendous growth and reach generated petabytes of data that were distributed across a variety of cloud and on-prem storage systems.

Challenge

Sky’s data team was tasked with managing data access across the company’s distributed data. The company had data in GCP, Amazon S3, data associated with different Customer Relationship Management systems, and many different systems in the UK, Germany, and elsewhere. Sky also had additional petabytes of data in unstable legacy systems.

Distributed data meant slow data access. When analysts requested access to data stored in different databases, the engineering team would have to break the bad news that it would take a couple of months to deploy and operate an ETL pipeline. This lengthy turnaround time delayed analysis and decision-making, negatively impacting the business. 

The team had tried to centralize data in the past, but the process proved frustrating, difficult to manage, and far too expensive. 

Solution

To solve these issues, the data team embraced a decentralized data mesh architecture and chose Starburst as a single point of access to all of the company’s data. Starburst Enterprise makes this decentralized approach possible by providing a single point of access to all of Sky’s data. 

Data stays secure due to Starburst’s integration with Apache Ranger, which gives the company the ability to integrate groups from Active Directory to Ranger, set policies, and filter access at the table, column, or row level.

Starburst is an essential part of Sky’s overarching Data Mesh, giving the user the flexibility to access data through a single point rather than having to go around to multiple data sources.

Results

Starburst’s impact goes beyond supporting Sky’s data mesh initiative, with several features that the company finds advantageous.

Connectivity. Starburst offers 50+ enterprise connectors to core systems, and Sky uses Starburst to connect to all of its main data platforms. 

Faster insights. Business users that want to pursue a question requiring access to two or more siloed datasets can run lightning-fast queries.

Caching tables. Starburst allows Sky to cache tables from its legacy system in the cluster. When users query those tables, they access the stable Starburst layer, not the unreliable legacy system database.

BI tools. Starburst integrates with Tableau, the company’s preferred business intelligence (BI) reporting tool, giving users fast access to new data sources without having to learn a new tool or wait for ETL processes.

Data governance and GDPR. As a European business, Sky’s data is subject to GDPR and other requirements. Starburst Enterprise allows the company to deploy clusters in specific countries. With Starburst Stargate, the Starburst-to-Starburst connector, the company can extend access to datasets in different countries while remaining compliant with data governance regulations.

With the single point of secure data access that Starburst provides, the company is able to apply a product mindset to datasets and create data products. This allows Sky to give users access to the data they want and ultimately helps drive its business forward. 

More resources: Ritesh Ranjan’s Data Rebel profile