What’s the difference between HDFS and S3?

S3 architecture

Amazon S3, like competitors Azure Blob Storage and Google Cloud Storage, is an object storage service that lets customers manage large amounts of data on a scalable, accessible cloud platform. An object consists of a globally unique identifier, subsets of a larger data file, and object-level metadata.

S3 buckets are private object collections that authorized users and applications may access. Typically, an S3 bucket is specific to an AWS Region, allowing companies to optimize for latency, cost, and regulatory compliance. Although a bucket’s objects store data in a non-hierarchical structure, object metadata lets S3 present data to users in folder-like groupings.

Amazon S3 offers multiple storage options, allowing companies to store data more cost-effectively. High-performance, low-latency options cost more but deliver the best performance for mission-critical applications. At the other extreme, inexpensive archival options for infrequently accessed data come with retrieval times measured in minutes or hours.

Data processing & querying capabilities: Spark & Trino

HDFS and S3 cannot meet modern data processing and analytics demands by themselves. Hadoop’s legacy MapReduce technology is complex, relatively inaccessible, and imposes significant latency penalties. The Apache Hive project masks MapReduce complexity by adding a SQL-like layer to make Hadoop more accessible at the expense of even more latency. On the other hand, object stores like S3 cannot match a traditional data warehouse’s rich metadata and analytics capabilities.

Open-source analytics engines like Apache Spark and Trino can access either storage system to power large-scale analytics use cases.

Apache Spark

University of California, Berkeley researchers frustrated with Hadoop’s limitations developed Spark to support their data science and machine learning projects. The Spark framework consists of four elements:

• Spark Core provides scheduling, transformation, and optimization services to the other elements.
• Spark SQL gives users an ANSI-standard SQL language for querying datasets.
• Machine Learning library (MLlib) comprises optimized algorithms for machine learning projects.
• Structured Streaming lets Spark process real-time data streams.

Spark can use HDFS and YARN to query data without relying on MapReduce. As a result, Spark processes data significantly faster than a standard Hadoop implementation. Spark can also run on Amazon S3. However, this requires a separate cluster manager like Kubernetes.

Trino

Now an open-source project, Trino began as Facebook’s Presto initiative. The social media company’s data teams needed a faster way to run interactive queries on Hadoop data warehouses.

Trino’s massively parallel processing architecture decouples compute from storage, letting data teams scale performance independently of storage capacity.

Support for ANSI-standard SQL lets data consumers at any skill level access large datasets through the business intelligence applications they already know or by integrating SQL commands in their own code.

Rather than remaining confined to the Hadoop ecosystem, Trino connectors let users query data from any enterprise data source, including Amazon S3. This federated approach unifies a company’s data architecture within a single point of access to support multiple use cases beyond analytics. For example, data engineering teams can use Trino to run batch ETL pipelines more efficiently.

Amazon EMR: Hadoop in the cloud

Hadoop’s origins in the data center make implementations relatively inflexible. For example, expanding compute and storage capacity requires infrastructure investments that take time and money to deploy. Moreover, building infrastructure to meet peak demand requires an over-investment in capacity that will go unused.

In 2009, Amazon introduced Elastic MapReduce (now simply Amazon EMR) to let customers build Hadoop infrastructure in the cloud. The EMR File System extends HDFS to let EMR store data as Amazon S3 objects rather than blocks.

EMR clusters consist of EC2 instances, called nodes, that correspond to Hadoop’s NameNode and DataNodes. Latency-insensitive processing jobs can use MapReduce and Hive. However, Amazon EMR works with Spark and Trino to deliver more performant, low-latency results.

While Amazon EMR lets companies enjoy the scalability benefits of a cloud-based Hadoop platform, it also introduces several challenges:

Complexity: Managing Hadoop and, by extension, Amazon EMR requires highly specialized skills and resources.

Cost: Moving Hadoop to the cloud does not change how it couples compute and storage, making Amazon EMR a resource-intensive way to manage data.

Optionality: Moving from a Hadoop implementation in the data center to one in AWS prevents companies from designing flexible multi-cloud architectures.

Modernizing HDFS to S3 data lakehouses

The rise of cloud-based object storage solutions led many companies to adopt the modern open data lakehouse architecture. These platforms store raw data using open file formats like Parquet and open table formats like Iceberg to create richer metadata than object storage systems provide natively.

Trino’s massively parallel processing query engine is the open data lakehouse’s final component, delivering capabilities such as:

Dynamic scalability: Decoupling compute and storage lets open data lakehouses scale dynamically with analytics workloads without impacting storage capacity.

Streamlined data management: By federating enterprise data architectures, Trino reduces the need for data movement and duplication. Less frequently accessed data can remain at the source, allowing the lakehouse to store higher-value data.

Enterprise-wide accessibility: Trino’s SQL compatibility democratizes data access and powers data-driven decision-making.

Optionality: Parquet, Iceberg, and Trino make an open data lakehouse platform agnostic so companies are not locked into Amazon S3 storage.

What do you do after you move data to AWS S3?

Starburst offers an enterprise-ready analytics solution for building open data lakehouses based on Trino’s open-source technology. Advanced features like material views, smart indexing, and universal search and discovery make data more accessible. Security and governance features let data teams define role-based data access controls to create fine-grained permissions at the row and column levels.

A Starburst-based open data lakehouse makes Amazon S3 data easier to manage. For example, after landing data on S3, Starburst will automatically create and update Iceberg reference tables. Business intelligence applications and ad hoc queries get rapid, up-to-the-minute data access without incurring expensive data migration costs.