Join the Team: Realizing the Promise Of Big Data

I have been in and around data since my days with Microsoft Access, Excel, and SQL Server circa 2000, and was fortunate to witness the phenomenal growth of big data. I’m not referring to the volume of data — that phenomena has been amply discussed — but the promise of data and its impact on the data practitioner.

In recent years, executives have gained a more strategic understanding that the organization’s data has the potential to increase revenue, enable the business to make smarter data-driven decisions, improve productivity and efficiency gains, stay up to date with current trends, and, with some effort, maybe even make the world a better place. As such, I’ve been fortunate enough to witness several large scale data lake creation projects, and led customers’ data platform teams.

Some of the projects I heard of, failed miserably. One project objective was to build a data lake that will hold the organization’s raw events data and the additional business generated data (marketing, sales, HR etc.), and that could be queried by all parties according to their role and responsibility. Sounds like a win-win. Spoiler alert: after a significant investment, the project failed and we didn’t build a centralized data lake. Large data projects that involve engineering are often challenging and complex.

However, there are projects that have succeeded, and indeed gave the organization the boost everyone expected. In both cases, I learned that an enterprise data platform or data access project is much more than the set of technological decisions we make — which cloud, what ETL mechanism, file formats, access control etc. These are all engineering problems that have viable solutions. What I did not expect to see was how differently various departments engaged with data. Why? Here’s my understanding of what makes data so magical.

Data as a regenerative resource

We‘ve all heard the phrase “data is the new oil.” Coined in 2006 by a British mathematician and architect of Tesco’s Clubcard, Clive Humby, the catchphrase means that data is valuable like oil, but if it’s unrefined, it cannot really be used.

Data must be transformed to drive any profitable activity; so data must be broken down and analyzed, for it to have value. In the same vein, Piero Scaruffi, cognitive scientist and author of “History of Silicon Valley,” said in 2016: “The difference between oil and data is that the product of oil does not generate more oil (unfortunately), whereas the product of data (self-driving cars, drones, wearables, etc) will generate more data (where do you normally drive, how fast/well you drive, who is with you, etc). So, it turns out, perhaps data isn’t the new oil because it is a regenerative resource.

Another way of thinking about data is to compare data with natural resources such as gold.  Everyone is mining data, similar to the California Gold Rush. Data-driven organizations offer resources to the data team and data-savvy employees. Each small data win is celebrated as the new growth engine of the organization.

Big data inspires us to be better

Transformation and change are always a locus of apprehension and uncertainty. Most people, at heart, prefer the known to the unknown, stability to innovation — even at the price of the gifts that are the inherent promise of progress. What’s more, “this is how we’ve always done it” is often synonymous with “this is the best way to do it.” Except, the good news is that big data has inspired us to be better. We can now avoid vendor lock-in, query and analyze data that lives anywhere, and data consumers now have the freedom to be curious.

At Starburst, our team includes dozens of very smart and enthusiastic engineers across the world that transform the future of data analytics every day. Whether that’s core SQL engine contributions to the open source project Trino, connectors to enterprise data sources, Starburst Data Products, or Starburst Galaxy, we have a number of opportunities for you to join the team.

In their own words: How does big data inspire you and your work?

“The ability to perform analytics on big data, and derive meaningful insights is critical. Enterprises need to gain insights on their data coming in at a high velocity and in all types of formats. We strive to improve business intelligence, empower data-driven decisions, and address the needs of organizations in a cloud-first world, without having to install or configure the hardware or software.” Anu Sudarsan, Co-Founder, Sr. Software Engineer

“Data is unapologetically blunt when it comes to evaluating decisions. You can’t hide behind a gut feeling. An uncompromising approach to making decisions from data is essential for choosing the best possible course of action. You can’t do that without access to data, however. We make it possible to deliver data to the right people, at the right time and in the right form.” Brian Luisi, Director Enterprise Presales, East

“The idea of using data to solve problems is both inspirational and aspirational in that it removes human biases and allows us to make better strategic decisions and ultimately be more successful in business. By simplifying the path from data to the value it creates, we make it easier for people to access the data they need, when they need it, and in the most straightforward way.” Colleen Tartow, Director of Engineering