Data Mesh Book Bulletin: Principle of Self-Service Data Platform

Despite the investments and effort poured into next-generation data storage systems, data warehouses and data lakes have failed to provide data engineers, data analysts, and data leaders trustworthy and agile business insights to make intelligent business decisions. The answer is Data Mesh – a decentralized, distributed approach to enterprise data management. 

Founder of Data Mesh Zhamak Dehghani defines Data Mesh as “a sociotechnical approach to share, access, and manage analytical data in complex and large-scale environments – within or across organizations.” She’s authoring an O’Reilly book, Data Mesh: Delivering Data-Driven Value at Scale and Starburst, the ‘Analytics Engine for Data Mesh,’ happens to be the sole sponsor. In addition to providing a complimentary copy of the book, we’re also sharing chapter summaries so we can read along and educate our readers about this (r)evolutionary paradigm. Enjoy Chapter Six: Principle of Self-Service Data Platform!

So far, Zhamak has challenged current data architectures and suggested a brand new paradigm towards Data Mesh: a distributed data architecture with ownership based on business domains, and proposed a new architectural unit, data as a product.

However, these two principles on its own, raise a few concerns:

• Duplication of efforts in each domain
• Increased cost of operation, and
• Most importantly, large scale inconsistencies and incompatibilities across domains.

Zhamak acknowledges that “expecting domain engineering teams to own and share analytical data as a product, in addition to building and managing applications and products, raises legitimate concerns for both the practitioners and their leaders.”

These concerns gave way to Data Mesh’s third principle: self-serve data infrastructure as a platform. One key differentiation of data platforms that will enable a Data Mesh implementation: the ability to scale out sharing, accessing and using analytical data, in a decentralized manner. 

First, let’s understand how Data Mesh’s technical platform is different from many existing solutions we have today.

Data Mesh Platform’s Differentiating Characteristics

To elaborate on the needs of a Data Mesh self-serve platform and to differentiate it from today’s available existing platforms, technologies and approaches, the table below highlights the key differences:

In summary, below are a set of characteristics of a Data Mesh platform(left), in contrast to existing platforms and technologies(right).

Key Objectives of Data Mesh Platform Thinking

Now that we’ve outlined some of Data Mesh’s self-serve platform needs, let’s take a closer look at top-level objectives of Data Mesh adopting the concept of platform thinking.

Enable Autonomous Teams to Get Value from Data

Ideally, when a Data Mesh platform functions smoothly, it enables domain teams to independently get their work done and accomplish business goals with a sense of autonomy and without having to rely on another team. When designing domain teams, it is helpful to consider the roles within each team, and their data journey in sharing and consuming data products to ultimately create business value. There are many other roles (i.e. data product owner) we can consider and plan for when creating a frictionless experience, however, for now, let’s focus on the two main roles of Data Mesh: data product developers and data product consumers.

Enable Data Product Developers, Autonomously

A data product developer is responsible for managing the entire lifecycle of a data product: creating it, building it, testing it, deploying, monitoring, updating, and maintaining its integrity and security. This is all in concert with code, data and policy.

Accordingly, a successful Data Mesh platform will have versatile, domain-agnostic, and cross-functional capabilities that will enable a data product developer to build, test, deploy, secure and maintain a data product.

Enable Data Product Consumers, Autonomously

A frictionless and fully functioning Data Mesh platform facilitates a seamless data product journey and provides the capabilities needed for data product consumers to get their job done without a bottleneck or minimal manual intervention.

A data product consumer’s journey always starts with discovering the data. Once discovered, the consumer will need access to the data, so she can understand it and take a closer look. Upon inspection, if the data proves itself to be useful, it will remain in circulation. Using data isn’t a “once-and-done” process. Consumers should continuously receive and process novel data to produce up-to-date insights and/or maintain machine learning models.

Exchange Value with Autonomous and Interoperable Data Products

As a catalyst for organizational and cultural shifts, the Data Mesh platform promotes sharing and consuming autonomous data products  between two or more groups, for instance: data product developers, data product owners and data product consumers. The key performance indicator of a Data Mesh platform is measured by a frictionless exchange of value of data products between data consumers and data providers.

Zhamak guides us, “The value can be exchanged on the mesh, between data products, or at the edge of the mesh, between the end products such as an ML model, a report or a dashboard, and the data products. The mesh essentially becomes the organizational data marketplace.”

Accelerate Exchange of Value by Lowering the Cognitive Load

To simplify and accelerate the productivity of domain teams in delivering value, we need to hide technical and foundational Data Mesh platform complexity from domain teams. This reduces the team’s cognitive load and redirects their focus on what matters to the business: creating and sharing data products.

Platforms are increasingly popular as a way to reduce the cognitive load of developers to accelerate their tasks and reduce time-to-insight. This is done by hiding complex detail and information presented to the developer, also known as abstracting complexity.

How Data Mesh Scales Data Sharing

Data Mesh essentially solves the problem of organizational scale in getting value from their data. That’s why the platform must operationally scale and enable data sharing across domains within an organization, as well as outside the organization. We also know that data has limited value if it’s not shared, therefore a Data Mesh platform must design for secure interoperability across multiple platforms to share data products. Without a standard, we fall back to a single monolithic solution that limits the scale of our operation.

Zhamak uses observability as an example of a capability that a Data Mesh platform should offer: “the ability to monitor the behavior of all data products on the mesh and detect any disruptions, errors, and undesirable access; and notify the relevant teams to recover operation. Even for a simple capability such as observibility, there are multiple platform services that need to cooperate:

• the data products emitting and logging information about their operation
• the service that captures the emitted logs and metrics and provides a holistic mesh view
• the services that search, analyze, and detect anomalies and errors within those logs
• the services that notify the developers when things go wrong

The key to integration of these services is interoperability, a common language and APIs by which the logs and metrics are expressed and shared.”

How Data Mesh Creates a Culture of Innovation

As of late, continuous innovation must be a core competency of any business. A Data Mesh platform removes unnecessary manual labor, hides complexity, streamlines data workflows, and frees data product developers and consumers to perform data-driven experiments and innovate.

According to Zhamak, “To assess the efficacy of a Data Mesh platform: measure how long it takes for a domain team to ideate a data-driven experiment, get to use the data to run the experiment. The shorter the time, the more mature the Data Mesh platform has become.”

The image above depicts an overall Data Mesh platform objective applied to domain teams: sharing and using data products.

6 Ways to Get Started: Self-Serve Data Mesh Platform

If you’ve bought into the Data Mesh paradigm, here are a few steps you can take to build your Data Mesh platform today.

#1 First Design the APIs and Protocols

Whether you are buying and/or building a Data Mesh Platform, start with selecting, and designing the interfaces that platform exposes to its users. The interfaces might be programmatic APIs, shared as libraries, command line or graphic interfaces. Regardless, first decide on interfaces and then implementation.

#2 Prepare for Generalists’ Adoption

A Data Mesh platform must be designed for the generalist. Select platform technologies that are in line with a natural style of programming known to many developers. This approach will build resilient and maintainable data products.

#3 Create APIs to Manage Data Products

While many services from cloud data platforms include lower level utility APIs, a Data Mesh platform often introduces a new set of higher level APIs to manage data products as a way of abstraction.

#4 Converge Data and Operational Platforms, Where Possible

As data products integrate with the operational world, there is a real need to integrate their platforms. That’s why a Data Mesh platform is a catalyst in simplifying a technology environment and creates a collaborative effort between operational and analytical platforms.

#5 Build Experiences, Not Mechanisms

Platforms should enable experiences, not mechanisms. Your Data Mesh platform can start with a single experience of ‘discovering data products’. Then, build or buy the simplest tools, mechanisms, that enable this experience. Then iterate and recalibrate for the next experience.

#6 Start Building a Data Mesh Strategy Today

Zhamak has one simple piece of advice to get started, “Begin adopting a Data Mesh strategy today, even if you don’t have a Data Mesh platform.”

Your framework is most likely composed of the big data technologies that you’ve already adopted. As the number of your data products grow and the standards evolve, you’ll continue to iterate the platform by auditing and sharing common capabilities across data products and domain teams.

Read along with us!

Get your complimentary access to pre-release chapters from the O’Reilly book, Data Mesh: Delivering Data–Driven Value at Scale, authored by Zhamak Dehghani now.