The data product lifecycle

8 Stages of the data product lifecycle

As there are stages in a data lifecycle, there are also stages in a data product lifecycle and it starts with the data product manager aligning data products with business goals and outcomes. 

The Data Product Manager Role 

If you look on LinkedIn, there are already thousands of data product managers and data product owners. Why is there a new role in the business, named data product manager or data product owner? There are several reasons why the Data Product Manager role is crucial. 

Let’s say a data consumer has a two reports from different systems — but they don’t match. It appears that there’s a data quality problem, but who is responsible and who will fix it? Maybe with one report, the data source was Oracle, another was SAP. Then there’s data from Salesforce and when you combine them with HubSpot, there are different admins for different platforms. There is no notion of who is the right data owner. 

#1 Start with business outcomes and ROI

The Data Product Manager owns and is responsible for data, builds data products with the data consumer informing them of their requirements, calculates the ROI, prioritizes consumer requests, and aligns resources. 

#2 Align resources to attend to the data product lifecycle

A team is critical to attend to the entire data product lifecycle: build, maintain, operate, retire. 

This team will have a number of people involved. Today when we build things, we have a team, the team builds, it moves to the next project, but the team won’t progress to the next project because the data product has a lifecycle. The Data Product Manager and team decides whether a data product is no longer useful and to retire it. 

#3 Define data contract: attributes, schema, KPIs, SLAs

Deciding what data product to build will encounter the concept of data contracts. A data contract is something that the consumers request the data producer. A common definition of a data contact is to is you’re document your schemas, SLAs, and KPIs.

Moreover, for a data product to be useful, there are baseline qualities that every data product must have. It must be: discoverable, understandable, addressable, secure, interoperable, trustworthy, natively accessible, and valuable on its own. All eight of these qualities need to be in the data contract. 

The domain-based Data Steward

In a hybrid approach, organizations will still have some data that require centralized domain ownership and data product definition, but you’ll also have data that will be decentralized with the domain’s business experts or domain-based data stewards. 

#4 A metadata plane for data products

The idea of a platform layer, metadata plane, or a data product catalog tracks the entire life cycle of data products and its usage, such as: who is using it, what is a version number, how fresh is my data, how high is the data product rated. All of this metadata for data products is now added into this metadata plane.

# 5 Build a reusable framework 

Building a reusable framework is a way of defining a standard. You’ll be addressing storage and compute: data lake, lakehouse, or data warehouse? What will be the analytical engine on top of that? Is the framework scalable? The idea of building a framework is very much like manufacturing and/or an assembly line, such that you can produce something that’s reusable. 

Data Engineer: business first, technology second

After the standards have been defined and created, you can integrate technology and involve engineering. You will notice how late the technology step shows up in this new data product lifecycle paradigm. This is the cultural shift we need to adopt.

# 6 Build and test for the data contract 

Because we’ve defined what the data consumer needs in the data contract, the contract becomes a focal point to build and test the data product. The data contracts has checks, agreements, and conditions that allow us to meet the consumer needs.

#7 Deploy with DataOps

Moreover, we need to hold data ops accountable. So we need automation everywhere, we need orchestration, we need to have observability, continuous testing, version control, all that goes into Dataops. Then, you’re ready to launch the data product.

#8 New Product Creation

At this point, the data product is created, but keep in mind, it also has a life of its own. Data products will change, so you’ll need a way to create new data products or modify existing data products. When that request comes in, you’ll need a way to update the data contracts, so a loop forms. 

The loop returns to the Data Product Manager

In this example, we  have an arrow going back to data contracts. You’d create a new version of the data contract and store it in your data product catalog. The Data Product catalog is essentially a marketplace, where users discover data products. And from that data product catalog, you’ll continue on your journey to create a new version of a data product. 

After reviewing this cycle, there are certainly specific industries that are better positioned to extract value from data products, primarily in the regulatory and governance industries like financial services. There are a few reasons why. 

Why Financial Services is eager to get started with data products

When industries such as financial services are looking at data to answer regulatory questions, they have metrics that they need to report out.

Financial services organizations especially and banks and capital markets, have been struggling with very complex business questions surrounding fraud, risk management, AML, Know Your Customer, all of which require organizational data, but in a form that is connected across various systems within the organization, and connects across various silos that exists in the organization. This is where speed matters. 

That’s why data products are appealing for organizations because there’s a framework that allows them to bring multiple data domains or data assets under the umbrella of data products to answer business questions in a relatively short time. 

And because building and optimizing data products takes a village, below we outline the Starburst and Deloitte partnership. 

Starburst and Deloitte partnership: helping organizations derive value from their data investments with data products

Starburst and Deloitte share a similar mindset and approach in how our clients drive value from their data investments. In fact, we began our journey together by educating our clients on the value of data mesh and data products.

Second, some of the industries — Healthcare and Financial Services— where Deloitte is seeing a greater demand for data products is where we are partnering with Starburst to drive that demand, together.

And then lastly, Deloitte is leveraging Starburst to build our own data products — we are also creating data and analytical products that we can take to the market. And Starburst is a platform that we use to bring our own data products and analytical products to market.