See How Data Engineering Gets Done on Our Do-It-Yourself Data Webcast Series

Start Free

Speed up your data preparation with Trifacta

Free Sign Up
Summer of SQL

A Q&A Series with Joe Hellerstein

See why SQL is Back
 
All Blog Posts

The History and Future of Hadoop Distributions

September 22, 2016

You have heard of Hadoop distributions, but what exactly is it? Apache Hadoop, or Hadoop for short, has forever changed the way we organize and compute big data. Considering how much big data is transforming the way we work, live, and play, it’s hard to believe Hadoop is just a decade old. We will examine the origins of Hadoop, dive deeper into the value of Hadoop distributions, as well as look at its future.

When Apache Hadoop Changed Big Data Forever

Apache Hadoop originated from a paper on the Google file system published in 2003. After more research, a formal project was born and work began on the Apache Nutch. The Hadoop subproject launched in January 2006, and by April, Hadoop 1.0 was released.

Apache Hadoop is an open-source software framework for distributed processing and storage of very large data sets on computer clusters. It is written mostly in Java, with C used in some of the native code. Hadoop was designed to run on commodity hardware, meaning that the framework was built to automatically handle expected hardware failures.

At the most basic level, Hadoop consists of a data processing component, called MapReduce, and data storage, known as the Hadoop Distributed File System (HDFS). Hadoop is considered a distributed system because the framework splits files into large data blocks and distributes them across nodes in a cluster. Hadoop then processes the data in parallel, where nodes only process data it has access to. This makes processing more efficient and faster than it would be in a more conventional architecture, such as RDBMS.

By innovating data analysis, Hadoop created the market for big data discovery over a very short period of time. Today, Hadoop is the go-to platform for big data.

What is Hadoop and Hadoop Distributions?

Since Hadoop is open source, many companies have developed proprietary distributions that go beyond the original open source code. To understand the many variants, we must first understand the standard Hadoop distributions, which include the following modules:

Hadoop Common. This is a collection of libraries and utilities leveraged by Hadoop modules.

Hadoop Distributed File System (HDFS). This scalable and portable file-system stores the data and delivers aggregate bandwidth across computer clusters.
Hadoop MapReduce. This is the processing model that computes in parallel for large-scale data processing.
Hadoop YARN. A newer addition to the base, YARN manages computing resources within clusters.

The full Hadoop ecosystem also includes software packages such as Apache Pig or Apache Hive that can be installed on top of Hadoop. When companies create new Hadoop distributions, the goal is to both provide additional value to customers and remedy issues with the original code. Typically, vendors focus on increasing reliability, stability, and technical support, as well as offering custom configurations to complete specific tasks.

The Future of Hadoop & Hadoop Distributions

Hadoop distributions continue to evolve. In the near term, we should see further adoption of the newer YARN module so that even larger data sets can be processed even more quickly. We will also see more integration with third party reporting systems such as Tableau. Performance improvements, while always happening with the open source platform, should markedly increase over the next few years. Finally, third parties will seek to address a broad range of data protection and security issues.

Trifacta and Hadoop: A Perfect Pair

Trifacta works seamlessly with Hadoop to support your data processing needs. Trifacta was built with careful consideration for Hadoop and its growing ecosystem. This not only makes for tighter integrations, but also better partner relationships with Hadoop distributions and applications.  Trifacta sits between the Hadoop platform’s data storage and processing, enabling the data visualization of analytics, while using machine learning in the analysis process for Hadoop distributions.

Sign up today for Trifacta.