Data Complexity

The V’s of big data: Key factors driving data complexity

Having a lot of data may be enough to drive complexity as it impacts infrastructure spending, archive and backup systems, retention policies, and more. Yet big data is not complex just because it is big.

Enterprises generate and store structured, unstructured, and semi-structured data of varying quality levels throughout the organization. Formats and metadata are applied inconsistently. If managing this was not enough of a challenge, companies must ensure that data collection, use, and access comply with governance policies and regulatory requirements.

Before they can bring this complexity under control, data engineering teams use these data complexity measures to understand the scope and degree of the challenges they face.

Volume: Data’s rising tide

Data volumes increased dramatically as the variety of data sources proliferated. For example, retailers once generated sales data only from the registers in their stores. point-of-sale systems would batch process the day’s sales overnight. Today’s omnichannel retailers generate sales data in stores, apps, and online around the clock.

Complicating matters further is a company’s ability to find relationships between different data sources. Our retailer can use tracking cookies, loyalty cards, social media, and other sources to study customer behavior and predict purchasing intent.

Real-time streaming data sources also generate large amounts of data. Industrial Internet of Things (IIoT) devices such as sensors and cameras constantly flow data into enterprise systems. E-commerce produces enormous volumes of clickstream data that may yield business insights.

Collecting and storing this tidal wave of data is a monumental task that requires an ecosystem of storage locations and careful optimization for cost and performance.

Velocity: Accelerating data generation

Closely related to the rising tide of data is the steadily increasing rate at which enterprises generate data. The new sources of data discussed above contribute to this problem. A retailer accustomed to the steady, predictable data flows from its point-of-sale system must adjust to the faster, higher-volume data flows from apps and websites. Manufacturers generate more data faster when their factories add cameras and other sensors to inspect quality on the production line in real-time.

Business leaders inadvertently accelerate data growth as they adopt data-driven decision-making cultures. Since data insights could lie in any data source, their reasoning goes, we should keep every bit of data we generate. Just-in-case data strategies exacerbate the data volume challenge, resulting in higher storage costs and greater data management complexity.

Variety: disparate data and myriad data types

Number of disparate data sources

Enterprise data remains stubbornly distributed. Domains build IT infrastructure to address their field-specific requirements. Business divisions use their P&Ls to fund systems that support day-to-day operations. Beyond the organizational distribution, fragmented storage strategies result in data stored in various on-premises systems and cloud services.

These scattered data sources are valuable resources that could drive innovation. However, the disjointed development history and ownership of enterprise datasets often locks these sources of data-driven insights into inaccessible silos.

Traditional solutions to distributed data sources have involved consolidating data into central repositories, including enterprise database management systems, data warehouses, and data lakes. These solutions require significant investments in data engineering to develop and maintain data integration ETL pipelines.

Myriad types of data

The proliferation of data sources also expands the types of data companies must manage. Enterprise database management systems handle a business’ traditional structured data. But they cannot address today’s steady flow of text, audio, video, social media, and other data types. Semi-structured and unstructured data require different storage strategies, most commonly cloud-based data lakes.

Even structured data becomes complex due to inconsistencies in schemas, formats, metadata, and other properties. Date and time formats seem simple enough to end users. For an engineer integrating data from multiple sources, format variation is a constant frustration.

Veracity: Data quality and integrity

Data veracity refers to its trustworthiness and suitability for analysis. Not all sources are created equal. A dataset’s lineage and provenance could prevent its use. Alternatively, the data may be riddled with errors, omissions, and outliers, making it impossible to analyze. Engineers developing ingestion pipelines must address data quality issues by cleaning data before it reaches its destination.

Value: Making the most of data

Data complexity has as much to do with how companies use data as with any property of the data itself. No longer content with spreadsheets and line graphs, digital transformation leads companies to adopt advanced data analytics techniques that extract deeper insights from larger datasets. Artificial intelligence and machine learning algorithms are the latest additions to the data mining toolkit.

Despite the potential innovations big science offers, it adds to the demands placed on already over-burdened data teams.

Vigilance: Keeping data private and secure

A dynamic cybersecurity threat landscape and the advancement of data privacy regulation require constant vigilance. Security policies must prevent unauthorized access in the event of a breach.

Access control is even more challenging when websites make any business global. Data privacy and sovereignty regulations may dictate where the company may store data and which employees may analyze data.

How to use complex data in your business

The speed of business requires responsive, agile decision-making, which, in turn, requires democratized data access. Data-driven cultures are only possible when leaders at every level have rapid access to the data they need to make informed business decisions.

Data is too important to be left to the experts. Business intelligence analysts shouldn’t need a data engineer to run their queries. A self-service model lets users query data from sources across the enterprise for their data visualization and analysis needs.

Democratizing data access frees data teams from supporting routine requests so they can focus their skills on more complex data science initiatives.

3 ways to manage complex data in business

Traditional data management strategies cannot keep pace with data complexity. These modern approaches can help simplify enterprise data management.

1. Data virtualization for managing data complexity

Consolidating data within a data warehouse to create a single source of truth is a dream that has never come true. Enterprise data sources are too varied and distributed.

A better approach is to leave data at the source and unite them in a virtualized data access layer. This federated architecture decouples storage and compute, allowing the optimization of each, and enabling performant queries on diverse sources.

2. Data mesh for managing data at scale

Virtualization lets companies support a decentralized information architecture called a data mesh. This approach pushes ownership to domains, democratizes access, and enables a data-as-a-product mindset. At the same time, data mesh strategies can strengthen data governance to improve security and privacy compliance.

3. Leverage data products

The data-as-a-product paradigm is not exclusive to data mesh architectures. When data teams adopt a product mindset, they focus on meeting specific business needs through stakeholder collaboration and rapid iteration. The resulting data products are reusable and scalable, improving efficiency and supporting more effective decision-making.

How Starburst helps with data complexity

Starburst Galaxy’s modern data lake analytics platform tames complex enterprise data ecosystems. By leaving data at the source, Starburst’s virtualization layer provides a single point of access that lets users explore, discover, and query any dataset. Here are ways Starburst helps with the V’s of data complexity:

Volume: Starburst decouples storage from compute, letting companies choose the most efficient storage solutions and most performant compute solutions. A massively parallel query engine scales from routine business analysis to the largest data science projects.

Velocity: By leaving data at the source, Starburst eliminates the need for interim storage or data warehouses where data can go stale. Building data products with Starburst allows the creation of real-time dashboards, and letting users query sources directly means they always analyze current data.

Variety: Starburst offers connectors for over fifty enterprise data sources. These connectors abstract each source so data consumers do not deal with source-by-source variations. Data teams use Starburst to manage schema, metadata, and formats to ensure company-wide consistency for all users.

Veracity: Data teams can leverage Starburst’s discovery and management features to evaluate data quality and develop pipelines to clean and process data for analysis.

Value: Starburst lets users run analyses and create data products with little friction. For example, the connectors abstract each source’s SQL implementation so users can create queries with their existing analysis tools.

Vigilance: Starburst lets data teams define role-based and attribute-based access policies for granular control at the table, schema, row, and column levels. For example, American users accessing European datasets will only see aggregated data, while authorized European analysts will see individual records.