Data applications: Architecture, examples, use cases

How Data applications power innovation and big data decision making

These applications are playing a transformative role across industries, powering data-driven decision making and paving the way for future innovation:

A healthcare analytics platform that offers predictive models for patient readmission rates, early detection of medical conditions, and personalized treatment plans
A financial analytics tool that offers anti-money laundering monitoring, fraud detection algorithms, real-time market analysis, and other risk assessments.
A retail analytics platform that provides real-time sales and inventory data, demand forecasting, and customer segmentation.

These applications exist amongst every industry and can take on many shapes and sizes, from in-product dashboards to next-best actions, to chat bots and machine-learning recommendation engines. However the look and feel, building efficient and scalable data applications requires strategic alignment across the business. With any new venture comes opportunities and challenges that require thorough research, planning, and the right technologies in place to achieve long-term product viability and success.

Today, we see customer building data applications for near-real-time (NRT) or real-time insights to business critical (and often revenue-generating) ventures. These include, but are not limited to:

log analysis
cyber threat detection
fraud detection
supply chain optimization
machine-generated data
clickstream data
customer 360

The challenges with building data applications

While the prospects of data applications are enticing, there are a number of challenges to face. These data applications need to be optimized for speed and scale, and flexible enough to meet the ever-changing demands of today’s enterprises. Some challenges to highlight are:

Legacy systems

Legacy systems create challenges when building data applications due to their outdated technology and lack of integration with modern software and platforms. These systems often have limited or no APIs, making it difficult to extract data and incorporate it into newer applications. Additionally, legacy systems may not support real-time data processing and lack the flexibility required to adapt to evolving business needs, hindering the performance and usability of data applications. As a result, developers often face complexities and inefficiencies when trying to build and maintain robust data-driven applications within such environments.

Build vs Buy

When building data applications, organizations face the “build vs. buy” challenge, where they must decide whether to develop the application in-house or purchase a third-party solution. Building the application in-house allows for customization and control but demands significant development resources, expertise, and time. On the other hand, buying a pre-built solution offers faster deployment and reduced development effort but might limit flexibility and require ongoing vendor support and licensing costs. Striking the right balance between customization and time-to-market is crucial when making this decision.

Scalability and high concurrency

Scalability and high concurrency challenges arise as businesses experience increased data volume and user demand. Ensuring that the application can handle a growing number of users and data inputs while maintaining optimal performance becomes critical. Proper database design, efficient big data processing, and load balancing techniques are essential to handle high concurrency and prevent performance bottlenecks. Scaling infrastructure and resources dynamically to meet demand is also crucial in addressing the challenges of scalability and high concurrency effectively.

Integrations

Integration challenges emerge when building data applications due to the need to seamlessly connect with various data sources, APIs, and existing systems. Ensuring smooth data flow and compatibility between different technologies can be complex, especially when dealing with legacy systems or disparate data formats. Developers must navigate big data silos, data mapping, data transformation, and potential data conflicts to achieve a cohesive and comprehensive integration that delivers accurate and up-to-date insights to users within the embedded application.

Security

Surfacing data within third-party systems increases the risk of unauthorized access and data breaches if not adequately protected. Developers must implement robust authentication and authorization mechanisms, encryption protocols, and data access controls to safeguard sensitive information. Regular security audits, updates, and vulnerability assessments are essential to mitigate potential risks and ensure the highest level of data protection within the embedded data application.

Seamless experience / user adoption

Designing intuitive user interfaces with clear data visualization and interactive features ensures users can easily access and interpret the data. Providing comprehensive user training and support during the adoption phase fosters user confidence and competence in utilizing the application effectively. By focusing on an intuitive design and comprehensive training, businesses can enhance user satisfaction, drive user adoption, and maximize the value derived from the embedded data application.

Bridging internal and external use cases with data applications

In a world of increasing competition and disruption, organizations are nearly required to build novel solutions to increase worker productivity and diversify business models. Data applications are steadfast approaches to both internal and external avenues.

Internally, In-product analytics drives operational efficiency within the organization. As data products become standardized, the business can streamline its internal processes, ensuring consistent and reliable delivery of insights to end users. This efficiency drives cost savings and optimized resource allocation so that teams can continue to focus on improvement and innovation across the business.

Externally, we exist in a fast-paced, ever-changing landscape. The ability to transform data analytics into marketable products is a game changer. It elevates businesses above competitors by offering tailored solutions, unlocking new revenue streams, and building unwavering customer loyalty. Rather than simply providing raw data or reports, businesses can deliver actionable solutions and tools that address specific pain points, deliver measurable results, and boost user adoption and retention.

Architecting a data application

Luckily for developers, cloud-native tech is playing a pivotal role in democratizing data application building for engineering teams. By providing scalable and easily manageable infrastructure, cloud-native tools eliminate the complexities associated with traditional OLTP setups. Regardless of the tools at your disposal, data apps at large are founded on the below principles.

There are several requirements to consider when researching, planning, and developing a data application:

Live data streams
1. Also known as real-time data streams, these enable data applications to process and analyze data as it arrives. These tools are becoming increasingly popular and less cost prohibitive with the proliferation of cloud based streaming providers.
Modern data stack
1. The modern data stack is set of technologies and tools used to manage and analyze data in today’s data-driven businesses. It encompasses data orchestration and transformation with a data pipeline (ETL or ELT), data storage (data warehouse or data lake), and an analytics tool (business intelligence and/or data visualization tool)
Multi-tenancy
1. Allows the application to efficiently serve many users at once, ensuring everyone gets their personalized data insights without interfering with others. This reduces costs, simplifies application management, and promotes scalability, making it easier to accommodate a growing user base, increasing data volume, and to provide a consistent and standardized experience across all tenants.
Fine-grained access control
1. Fine-grained access control requires the implementation of a robust authentication and authorization system to identify users and assign specific role-based access or permissions. Fine-grained data tagging is essential to associate each data row with attributes that define allowed user access. Efficient data filtering mechanisms must be in place to restrict access during data retrieval, and secure data storage and communication protocols are necessary to protect sensitive information.
APIs
1. Through various APIs (data APIs, SQL APIs) developers can access and manipulate data from various sources without needing to understand the underlying complexities of the data storage or processing systems. This simplifies data integration, accelerates development timelines, and fosters interoperability, enabling developers to focus on building the core features and functionalities of the data application.
Robust data visualization
1. Data visualizations shape the end user experience just as much as the upstream tools that fuel it. They aggregate data across datasets and facilitate better data exploration, identification of trends, patterns, and outliers. Whether custom built or fueled by business intelligence tools, this is a vital component for data-driven decision-making that enhances the overall application value.

Choosing the right tech stack

Navigating the tech stack selection is pivotal for creating successful data applications.

Open lakehouse approach

The open data lakehouse overcomes the limitations of legacy lakes, because it’s built with the understanding that center of gravity does not mean a single source of truth. It works with your other data sources in an open, scalable manner – creating a single, open system to access and govern the data in and around your lake.

Scalability

System must handle vast amounts of big data, from terabytes to exabytes, without compromising performance. Offers customers full control over data storage, management, and consumption to allow for optimization of their analytics environment. Scalable data applications enable seamless expansion and adaptability, ensuring they can meet the increasing demands of users and evolving business requirements.

Performance

Prioritize efficient handling of big data and complex computations. Look for robust data integration capabilities and seamless connections with various sources, and leverage in-memory processing and caching to reduce retrieval times. System should be able to provide a performant user experience no matter the load or concurrency.

Cost

Focus on cloud-based solutions to pay only for actual resource usage, utilizing serverless architectures and auto-scaling to efficiently scale resources as needed. Choose cost-effective big data storage options, compress data when possible, and implement caching to reduce retrieval costs. Emphasize data cleaning and transformation to minimize storage requirements. Regularly monitor resource usage and performance to identify areas for further cost optimization and avoid unnecessary expenses.

Example architecture

Based on our conversations across customers, below is an example reference architecture for a Starburst powered data application.

Starburst-powered data applications

A Modern, Self-Service Platform for Data Applications at Internet Scale

7bridges

Named one of the top 15 hottest AI startups in Europe in 2020, 7bridges is an AI-powered global supply chain platform that provides complete visibility into end-to-end operations all within one platform.

Problem:

All applications were connected to the main relational database – PostgreSQL. While the platform was functional, slow query execution, timeouts, and data accessibility issues created dissatisfaction among clients.

Solution:

7Bridges deployed Starburst Galaxy to overcome these data challenges. Reports that took >45 minutes now execute in minutes or less. Non-technical business users can now easily discover and interact with data on their own, reducing dependency on data engineering teams and accelerating time to insight across the business. With Starburst at the forefront of their lakehouse strategy, 7bridges optimized their infrastructure costs, improved data accessibility, sped up query execution, and greatly enhanced the overall client experience.

“We chose Galaxy because of the flexibility it offers to connect to so many different types of tools, data formats, and data sources we may need in the future. From a R&D perspective, it’s also extremely valuable to be able to spin clusters up and down and split clusters up in a quick and easy way,” shares Simon Thelin, Lead Data Engineer at 7Bridges.

Leading Cyber Security Company

A leading cyber security company provides products to customers that surface log-in anomalies and additional insights to mitigate cyberthreats.

Problem:

A leading cyber security company provides products to customers that surface log-in anomalies and additional insights to mitigate cyberthreats. With ElasticSearch and EMR, company ran into consistent query failure and were only able to query log data for up to 30 days. Each EMR environment required daily tuning, stealing valuable team members to focus on operations instead of feature development. These limitations were diminishing product value and prevented them from market expansion and upsell opportunities.

Solution:

Today, this company leverages Starburst Galaxy as an embedded query engine which enables their customers to expand their log data queries up to 90 days and unlocks new use cases within their existing customer base while offloading the management of Trino to Starburst Galaxy. With this expansion and new markets (9 total), this company believes they have an opportunity to IPO in the next 12-18 months.

Data applications shape the landscape of tomorrow’s insights

Data apps have revolutionized ways of work and have directly impacted how customers think about exceptional product experiences. While these applications have been difficult to build historically, continuous waves of innovation have allowed for streamlined development, management, and adoption of these tools. How we design today must plan for the future, and how we build for the future must be flexible enough to navigate unknowns. The future of data applications will be shaped by a combination of technological advancements, societal shifts, regulatory changes, and unforeseen breakthroughs.

If you are looking to kick-off your data application journey or want to take your existing application to the next level, sign up for Galaxy today.