6 ways to foster curiosity in data engineering and drive real results

You can’t teach curiosity or force people to ask questions. Yet curiosity is part of the human condition, according to Harvard cognitive scientist and researcher Elizabeth Bonawitz, and one of my roles as a technology leader is to nurture that drive to question and explore. After 30 years of experience in the technology realm, and now at El Toro, an adtech and advanced analytics firm, I’m a strong believer in fostering curiosity in data engineers. That’s one of the reasons I’m a big fan of Starburst – it gives my team the freedom to be curious.

Creating and nurturing this kind of environment requires more than the right tools, though. In this post, I’ll share my tips for fostering curiosity, and discuss how adopting these strategies with my own teams has directly impacted El Toro’s business.

#1 Make exploration easy

First, you need to make exploration easy and accessible. I’d encourage the use of interactive visualizations so the people analyzing the data and results (both the analysts as well as the project owners) can play around with the data themselves and uncover insights they may not have found otherwise. Do what you can to inspire them to work together and share their ideas without fear, both within and across teams and departments.

#2 Promote continuous learning

Training and mentorship programs for data engineers, analytics engineers and data analysts can all help your teams exercise their creativity and generally broaden their approach to data science and analytics. This emphasis on learning will back up your assertions that you want them to be curious, too. Yet education doesn’t have to be confined to specific training or programs. Encouraging open communication and an inclusive environment that welcomes different perspectives, backgrounds, and experiences can be hugely educational. Great ideas often spring out of groups composed of a broader mix of people.

#3 Don’t shame failure

I’m a professional screw-up. Over the years, I’ve learned enough from my failures that I’m pretty good at not breaking things – even as I prod and poke at their foundational structures and ideas. Basically, I learn from my mistakes and avoid letting a fear of failure determine my actions. And I wouldn’t be a very good leader if I didn’t cultivate this approach within my own teams. I don’t think we need to celebrate failure, exactly, but we should encourage it, because if you’re not failing now and then, you might not be pushing or thinking hard enough.

One simple way to get started with this is to remember to bring up the number of problems or failures you had along the way to get to a successful outcome. Everyone on your team should know that it is as important to bring those things up as it is to celebrate the success.

#4 Provide better tooling

Starburst is one of my favorite tools, and it really does give our data engineers and analysts that freedom to be curious. I like to see my team experimenting directly with Spark, Python, and SQL. I’m an advocate of BI tools like PowerBI and Tableau, and if you haven’t haven’t heard of scrollytelling, look it up now. It’s a powerful form of visual storytelling that helps you get to a deeper level of understanding of the material or idea you’re trying to get your head around.

#5 Do something creative (like building a haunted house)

In addition to my role as CTO at El Toro, I also operate a successful haunted house attraction. I have multiple patents, too, including one for a new kind of wearable exercise system. I’m not saying this to tout my credentials. I engage in this kind of work because it’s intellectually entertaining and fun, but I’m convinced it also makes me a better, more creative, and more curious CTO.

Whether you’re designing a haunted house or drawing up an invention, doing interesting work outside the lanes of your normal day-to-day life can be incredibly stimulating and impact your day job positively. I encourage hackathons, especially if you stretch them out instead of confining them to three- or four-hour sprints.

But at El Toro we’ll also tinker with Arduinos, or do ‘bug fixing exercises’ where we hunt and fix someone else’s problems in code they may not yet have been able to fix. It helps our teams expand their own knowledge and encourage other creative technical work as a team.

#6 Create boundaries, not stop signs

When I say that I encourage exploration and curiosity, that doesn’t mean I disregard accountability or production. You don’t want someone going so far down the rabbit hole that they become a drag on the team’s goals. At the same time, you don’t want them to be so focused on those goals that they don’t explore at all. So we promote the freedom to explore within boundaries. We create a sprint-like structure and allocate specific chunks of time to pursue those open questions, then have regular check-ins. My team knows that I’m going to say something if I think they’re on the wrong track. Yet they also know they are free to convince me otherwise – and if they present a reasonable argument, it’s my job to let them keep going.

How curiosity drives real results

As I mentioned above, we’re in the advertising analytics space. We log millions of transactions per second, collect this data in our data warehouse and data lakes, and connect it to our analytics and processing engines. The scale is outrageous, and one of our systems wasn’t fast enough. An engineer came to me and said he thought there might be a better way to build this system. I was skeptical, but I gave him a sprint to explore.

When we had our check-in, he came back with a prototype. The new solution didn’t work yet, but he knew it would work, and why. He explained all this to me and I gave him the green light. That exploratory sprint turned into an actual project and a few months later we had a working upgrade. The new system worked so well, in fact, that we went from utilizing 500 servers to 80 servers to do the exact same thing.

Another, more recent example: an engineer wanted to look into the new SQL engines built on chatGPT, to see if he could come up with a natural-language interface for our non-SQL-literate colleagues. He put together a system that allows someone to enter, in normal language, what they’d like to explore. The engine takes that and outputs a SQL query, then we connect that with our Starburst interface, which goes out and queries the appropriate datasets and provides fast results. Once again, a spark of curiosity led to a creative and very useful new tool.

Supporting curiosity at El Toro has generated real results. Amazing results, actually. We started in the advertising business and now we’re an analytics company with dozens of patented technologies. That’s what curiosity can do for your business.