Decentralized Data

More than 75% of enterprises said they are decentralizing their data infrastructure — according to the State of Data & What’s Next survey. Clearly, data professionals have concluded that monolithic, centralized approaches to data management are not working.

This is not a surrender. Instead, enterprises have recognized that the volume of data and the demand for data insights are increasing too rapidly for centrally-managed architectures to keep pace.

Strategically-implemented data decentralization, on the other hand, promotes more efficient data management processes, makes data more accessible to more stakeholders, and accelerates time to insight.

This guide will explain data decentralization, how embracing distributed data benefits the enterprise, and how a modern approach to decentralization makes data accessible to everyone.

What is the difference between centralized and decentralized data?

Centralized architectures work for startups, simple data environments, or slower-moving organizations. In these cases, keeping the company’s data in a central location makes sense.

Modern enterprises are more complex. Organic growth, mergers, and acquisitions result in fragmented IT infrastructure comprising a decentralized network of on-premises systems in locations around the world and one or more cloud platforms.

Centralization in this context has meant adopting a series of technologies — from relational databases to data warehouses to data lakes — that try to aggregate critical data in a central, easily-accessed location.

Decentralization, on the other hand, considers domains as the best place for data to be. Business units and regional operations understand their data needs better than a central data team. Data remains at the source while becoming accessible to the rest of the organization.

Why doesn’t data centralization work?

For decades, companies have struggled under the belief that centrally-managed architectures would supercharge insight generation and empower decision-makers. Enterprise relational databases, data warehouses, data lakes, and other solutions sought to aggregate critical data into a single analytics resource.

That tactic hasn’t worked very well.

Centralization has not reduced data sprawl. Our survey found companies, on average, manage between four and six separate data platforms, with more than a tenth of respondents managing up to a dozen platforms.

Centralization delays business insights.

Typically, our respondents said it takes at least 48 hours to create a new extract, transform, and load (ETL) pipeline and get it into production. That could translate into a week or more elapsing between a decision-maker’s data request and the resulting analysis.

Centralization can’t keep up with advanced analytics.

Nearly three out of four companies told us that artificial intelligence, machine learning, and other data science projects are the top data workloads. Moving, preparing, and processing these massive datasets place enormous burdens on IT infrastructure and data teams.

Centralization undermines data-driven business cultures.

Successful companies are more effective and agile than their competitors because they empower executives, managers, and workers to make data-driven decisions. However, a centralized data architecture’s complexity requires data engineer involvement in any request for data. End users must wait as their requests compete in the queue for the data team’s limited resources.

Why is data decentralized? What are the benefits of decentralized data?

Decentralization addresses the challenges centralization strategies created. Decoupling data storage from data usage makes data more accessible to users throughout the company, unlocks data from more sources, reduces the pipeline development burden, and saves money.

Data democratization

In the decentralized model, domains manage the technical aspects of storing the data they create. A virtualized data consumption layer abstracts domain-by-domain variations in how each data source stores data.

Data democratization means business users no longer need the data team’s help to run their analyses. Instead, they interact with the data consumption layer through the SQL and business intelligence tools they already know how to use.

When decision-makers need answers, analysts can dive into the data immediately. No waiting for data teams to acknowledge their requests. No delays in pipeline development. Time to insight drops significantly as analysts produce results faster. And business leaders get better information more quickly to support their decisions.

Enabling access to decentralized data sources

The complexity of managing legacy systems, multi-cloud and hybrid-cloud architectures does not impact users in a decentralized system. Quite the opposite, decentralization unlocks data sources that were too difficult to access before.

A decentralized system grants each domain authority over managing their data sources while making them responsible for exposing their data to the rest of the company. Silos that hindered the flow of information between e-commerce, manufacturing, and other domains disappear.

Before, analysts would need the data team’s help to reconcile the different flavors of SQL implemented in each data platform. The data consumption layer abstracts those details so analysts can access every source directly.

As contradictory as it sounds, decentralization actually unifies on-premises, private cloud, and public cloud data stores. Unlocking enterprise data gives executives a real-time, holistic view of the company’s operations.

Simplifying the ETL/data integration process

Looking back after a decentralization initiative, it’s easy to think it all but eliminated data pipelines. Pipelines haven’t disappeared. They merely account for a smaller share of data activity. What really happens is that data use mushrooms as decentralization makes pipelines unnecessary for most requests. More people perform more analyses without needing data engineers’ help.

ETL pipelines and data integration still matter. They simply aren’t bottlenecks in daily operations. Freed from routine requests, data teams can focus on complex big data projects that depend on pipeline development.

Cost Efficiency

Leaving data at the source creates efficiencies and cost savings while making data infrastructures more agile and scalable. Domains can manage their storage architectures without worrying about how their decisions impact people using their data. Consequently, domains can cost-optimize storage strategies to handle ever-growing data volumes.

With the company’s data analysis less dependent on data warehouses and data lakes, there is less pressure on centralized data capacity. Rather than expanding the number and size of these repositories to accommodate all users, data teams can allocate storage expenses where speed, scalability, or other factors take priority.

What is an example of decentralized data?

Creating a data mesh is one way to decentralize a company’s data architecture. Data mesh federates storage and creates an access framework that enhances productivity company-wide.

Data federation: Connectors to all the data sources across your org

A federated architecture appears to users as a virtual database linked to every data source. Domains manage their data infrastructure, deciding how to store data. At the same time, they are responsible for sharing their data with the rest of the organization.

Users access this data through the virtual database without having to learn the nuances of each domain’s infrastructure.

Related reading: Delivering data virtualization and federation at enterprise scale

Data Product: Create and share curated data sets inside and outside your organization

Data mesh applies a product management paradigm to decentralize data ownership and accountability. Each domain owns its data sources and is responsible for data quality, availability, and usability.

By treating data as a product, domains become responsible for opening datasets to the enterprise by publishing application programming interfaces (APIs). These APIs let users in other domains query data for integration in their own data products.

Related eBook: Data Products For Dummies, Starburst Special Edition

Data lake analytics platform: Data lake is the most flexible and future-proof data storage layer to implement a data mesh

While decentralization addresses traditional data management challenges, Starburst believes that the data lakes remain at a data-driven organization’s center of gravity. Efficiently consolidating critical data in data lakes offers benefits in performance, efficiency, and time to insight that can’t be ignored.

At the same time, data lakes are not a complete solution. Analytics will always need data from elsewhere within and beyond the company’s infrastructure.

Starburst’s data lake analytics platform lets companies reap the benefits of their lakes while bringing all other data sources within a single point of access.

Built upon Trino’s massively parallel, open-source query engine, Starburst unlocks data across the organization and democratizes access by letting people use the tools they know. Yet, this decentralization does not compromise data governance. Starburst’s access control features allow you to apply fine-grained controls to every data product, helping to ensure compliance with data privacy and security frameworks.

Related reading: Data Lake Analytics for Smart, Modern Data Management