How Annalect Super-Charged Cross-Company Collaboration with Data Mesh

New tools for data access

The main benefit of the data mesh approach is the liberated and governed data access for various data consumers. To implement a cross-team/cross-organization consumption pattern, we needed to leverage new tools that provide smooth, efficient data consumption across various data nodes with minimal data movement.

In the end, data consumers rarely care whether there is a data lake or data lakehouse under the hood, or which data pipeline architecture is used. What they do care about is that they can consume that data fast, without having to wait for access, and through the delivery mechanism of their choosing (dashboards, APIs, analytics workbenches).

Image 3 — Visualization of Distributed Compute functionality

Distributed compute

Tools like Trino, Starburst, Amazon Redshift Spectrum, and Amazon Athena have separate storage and compute. This creates a single-point of entry with data warehouse-like capabilities without building an actual warehouse. This separation of storage and compute allows us to scale each component independently, utilizing storage services like Amazon S3 (which provides a lower cost storage option), while scaling compute resources as needed.

The significance of distributed compute for easier access control and provisioning across multiple datasets cannot be overstated. In the past if you needed to join 10 different datasets in 10 different databases, you would need individual access to 10 of them. Then you would need a compute facility to join them. With distributed compute you can connect datasets in one place within matter of minutes and execute a simple SQL query in one place across all of them. Distributed compute is the practical query implementation of the theoretical part of a data mesh framework.

Image 4 – Visualization of AWS Clean Rooms inside data mesh

Data clean rooms

Data clean rooms allow us to securely query data from partners and clients, without moving the underlying data from their environment to ours. With AWS Clean Rooms, we are now able to join data across multiple entities while allowing each partner full control over how their data is shared, and how it can be queried.

Overcoming data-selfishness and collaborating externally

When Annalect decided to make the move to a data mesh approach, the main objective was to liberate ourselves from nuclear rewrites and achieve data sanity within our organization. However, one very unexpected benefit we found as a result of the transition was the ability to collaborate with other partners.

Marketing is a continuously evolving field. Being diligent about the privacy boundaries of consumers and working with clients and partners securely and dynamically, is becoming an essential concern for businesses. In the past companies would send data directly to each other to get a job done. In the marketing and media world that meant activation and performance optimization.

Image 5 – AWS Clean Rooms provides medium for secure collaboration.

With new technologies like data clean rooms and distributed compute platforms, agencies can work with client and partner data without having to copy the data from one place to another. They never see the underlying sensitive data in the first place.

For Annalect, we frequently find ourselves in need of combining performance marketing data from campaigns run by us with the additional metadata of our partners. Under the old model, we would need to get a copy of our partner’s data or have to send them our data to get the enriched version.

In the new world of collaborative data meshes, while leveraging clean rooms and distributed compute, we can join the data without disclosing full datasets or sending the data to each other. This provides all parties involved in the transaction the ability to easily collaborate without unnecessary data copies and data transfers.

Looking to the future, where there is a high need to open data assets to various LLM models, the data mesh framework future-proofs data assets and simplifies the process of providing data to new AI agents and models. Instead of restructuring to accommodate new LLM applications, one could seamlessly employ distributed compute technologies. They could integrate data from multiple domains and make it accessible to LLM models—treating LLM models as just another stakeholder.

Furthermore, if some data needs to be vectorized, converting structured data into embeddings can be conveniently incorporated as an additional step in the preparation of the data product. As we rapidly advance through new and inventive ways of utilizing retrieval-augmented generation (RAG), data mesh is a testament to the principle that data sharing and utilization are fundamental prerequisites for any data-driven environment.

Conclusion

The data mesh framework was a revolutionary concept that transformed how companies manage and utilize data assets across an organization. Data mesh enables agile and distributed architectures, establishes clear data ownership. It also allows data to be made widely available across the organization to various stakeholders for different types of consumption, both internally and externally.

Hopefully, you are now inspired to consider the benefits of cross-collaboration through new modern techniques of data management.

Originally published, on AWS.