Hadoop and Spark are big data processing frameworks. The former arrived when big data lived in the data center, while the latter emerged to meet the needs of data scientists processing data in the cloud. Although both remain in widespread use, open data lakehouses based on cloud object storage, Apache Iceberg, and the Trino distributed query engine offer more optionality across use cases.
What is Apache Hadoop?
Apache Hadoop is a distributed data processing framework designed to run on commodity hardware. When first released, it replaced expensive, proprietary data warehouses. Hadoop remains a fixture of data architectures despite its disadvantages against modern alternatives.
What is Apache Spark?
Apache Spark is a data processing engine that handles large datasets on single machines or scalable multi-node architectures. The open-source project’s core advantages are its machine learning and data science processing capabilities.
Hadoop vs. Spark: Big data frameworks
Big data analytics has changed dramatically since Hadoop’s initial release, eclipsing what was once a revolutionary framework for processing large datasets.
Hadoop’s rise in the data center
Amazon Web Services (AWS) was four years old at Hadoop’s release in 2006. Amazon S3 launched less than a month earlier, and neither Azure nor Google Cloud existed. For most companies, analytics relied on proprietary, on-premises data warehouses. Those solutions were expensive to acquire, run, and scale — especially for emerging dot-com companies like Yahoo! and Google. Developers at Yahoo! created what would become Hadoop as they sought a more efficient way to index and manage billions of web pages.
Today, four core modules comprise the Hadoop framework:
Hadoop Common – A set of utilities shared by the other modules.
Hadoop Distributed File System (HDFS) – A high throughput file system designed to run on commodity data storage hardware.
Hadoop Yet Another Resource Negotiator (Hadoop YARN) – Uses daemons to manage resources for Hadoop clusters and schedule jobs.
Hadoop MapReduce – A Java-based programming framework for distributing applications across thousands of nodes.
By making data processing at internet scale more efficient, Hadoop significantly reduced costs for a new wave of online companies like Facebook and Uber, spawning an entire Hadoop ecosystem. For example, Apache HBase is a distributed columnar database that runs on HDFS and offered BigTable-like capabilities years before Google made the technology publicly accessible.
From MapReduce to Hive
As Hadoop adoption grew, the framework’s limitations became more apparent. Few people understand MapReduce’s unique Java dialect, limiting Hadoop’s accessibility. In 2008, Facebook developers released their better data warehouse analytics platform running on HDFS — Apache Hive.
Hive is a translation layer between a user’s SQL statements and Hadoop MapReduce. Data engineering teams use HiveQL, a version of SQL, to build ETL batch pipelines for their HDFS-based data warehouses without needing deep expertise in MapReduce. Although not exactly ANSI-SQL, HiveQL is more accessible to analysts and data scientists.
Hadoop modernization
From Hive to Spark
Hive has an inherent limitation that limits its usefulness for modern analytics. As middleware for Hadoop, Hive needs time to translate each HiveQL statement into Hadoop-compatible execution plans and return results. This extra latency adds to the limitations of MapReduce itself. With multi-stage workflows, MapReduce writes each interim result to physical storage and reads the result as input for the next stage. This constant reading and writing makes iterative processing challenging to scale.
University of California, Berkeley, data scientists developed Spark, a faster system for their machine learning projects that uses in-memory processing. Spark reads data from storage when a job starts and only writes the final results.
The Spark framework is composed of the following elements:
Spark Core – Resource management utilities for the other elements.
Spark SQL – An ANSI-standard implementation of SQL.
Machine Learning library (MLlib) – Libraries of optimized machine learning algorithms for the Java, Scala, Python, and R programming languages.
Structured Streaming – Micro-batching APIs for processing streaming data in near real-time.
GraphX – Extends Spark’s resilient distributed dataset (RDD) to support graph processing.
Hadoop and Spark vs Trino
Engineers at Facebook experienced similar challenges with Hadoop and Hive. In addition to Hive’s inherent latency, using Hive to process petabytes of social media data incurred significant performance penalties due to how it manages data.
Hive organizes files at the folder level, meaning directories must perform file list operations to give queries their required metadata. The more file partitions, the more list operations, and the slower queries run. Another issue comes from running on Hadoop which forces rewrites of files and tables to accommodate changes.
Introducing Trino
Facebook’s internal project eventually became Trino, an open-source massively parallel SQL query engine that federates disparate data sources within a single interface. Like Spark, Trino queries data directly without using MapReduce. Also like Spark, Trino uses ANSI-standard SQL, which makes datasets more accessible to a broader range of users.
Trino’s strength is its ability to simultaneously process data from many different sources. Rather than limiting analytics to a monolithic data warehouse, Trino connectors can access structured and unstructured data in relational databases, real-time data processing systems, and other enterprise sources.
This flexibility lets users conduct large-scale data analysis projects that yield deeper, more nuanced insights to inform decision-making.
Related reading: Spark vs Trino
Open data lakehouses
Trino becomes the query engine and interface for modern open data lakehouse architectures when combined with cost-effective object storage services and the Iceberg open table format.
Object storage services use flat structures to store data more efficiently, eliminating the challenges imposed by Hive-like hierarchical folder structures.
Iceberg’s metadata-based table format greatly simplifies lakehouse data management. Table changes do not require data writes; instead, Iceberg records them as a new snapshot of the table metadata.
Trino provides the SQL-based interface for querying and managing Iceberg tables and the underlying data. The engine’s low latency, even when processing large volumes of data, makes Trino ideal for interactive, ad hoc queries and data science projects. Recent updates have added fault tolerance to support reliable batch processing as well as stream processing to ingest data from real-time data processing systems like Kafka.
Data migration strategy: Navigating the shift from Hadoop to cloud data lakehouses
Hadoop’s rapid adoption was the only way companies could keep pace with the explosion of data over the past twenty years. However, these legacy systems are increasingly complex and costly to maintain — especially when run on-premises. Moving to the cloud reduces costs, improves performance, and makes enterprise data architectures infinitely more scalable. However, any migration project risks significant business disruption should anything go wrong.
Starburst’s enterprise-enhanced Trino solution is ideal for migrating data from Hadoop to a cloud data lakehouse. Starburst features that streamline data migrations include:
Federation
Starburst adds performance and security features to Trino’s connectors, creating more robust integrations of on-premises and cloud-based data sources. These connectors let companies dissolve data silos so users can access data where it lives.
Unified interface
By abstracting all data sources within a single interface, Starburst becomes the single point of access and governance for a company’s data architecture. Engineers can build pipelines and manage analytics workloads from a single pane of glass. At the same time, data consumers can use the SQL tools they already know to query data across the company.
Migration transparency
Once users get used to accessing data through the Starburst interface, they no longer worry about where that data lives. Architecture becomes transparent. Engineers can migrate data from Hadoop to a data lakehouse without impacting the day-to-day business. Once the migration is complete, the data team switches connectors behind the scenes.
How Starburst helps with Hadoop modernization
Starburst gives companies the optionality they never had with proprietary analytics platforms. While pairing Trino and Iceberg provides the most robust, performant alternative to Hadoop, companies can use Starburst to migrate to any modern data platform.
1. Migration to Apache Iceberg
Apache Iceberg’s open table format enjoys widespread support in the enterprise space as well as among its deep pool of open-source contributors. Iceberg’s symbiotic relationship with Trino contributes to Starburst’s enhanced features, including automated Iceberg table maintenance.
2. Migration to Delta Lake
Although closely entwined with a single company, Delta Lake is an alternative open table format for companies committed to a Databricks solution. Starburst’s Delta Lake connector can support migrations from Hadoop.
3. Migration to Apache Hudi
Faced with the same Hadoop/Hive limitations as everyone else, developers at Uber created Apache Hudi, an incremental processing stack that accelerates integration workloads. The framework is compatible with several open query engines, including Trino, allowing companies to make Starburst their analytics interface for a Hudi architecture.
Cost and productivity insights
After dealing with the costs, limitations, and complexities of Hadoop management, Starburst customers migrated to open data lakehouses and realized significant improvements in costs and productivity.
Migration – Starburst’s integration with Hadoop implementations and object storage services streamline data migration while also minimizing the hardware and cloud resources needed to see the project through.
Operations – Starburst unifies enterprise architectures to simplify data operations and reduce maintenance demands.
Productivity – Starburst’s single point of access lets analysts get the data they need faster, resulting in richer insights and better, more agile decision-making.
Learn more about data migration with Starburst through our webinar and ebook.
