Hadoop
Generated by GPT-5-miniExpansion Funnel Raw 86 → Dedup 9 → NER 8 → Enqueued 6
| Hadoop | |
|---|---|
| Name | Hadoop |
| Developer | Apache Software Foundation |
| Initial release | 2006 |
| Programming language | Java |
| Operating system | Cross-platform |
| License | Apache License 2.0 |
Hadoop Hadoop is an open-source framework for distributed storage and distributed processing of large data sets across clusters of computers. It originated from research at Yahoo! and development at the Apache Software Foundation, becoming central to early big data infrastructures used by companies such as Facebook, Twitter, LinkedIn, Amazon.com, and eBay. Hadoop enabled a shift from centralized data warehouses to scalable, commodity-cluster architectures adopted by Netflix, Spotify, Airbnb, Uber, and major financial institutions like JPMorgan Chase.
Overview
Hadoop provides a software platform enabling organizations including Walmart, Baidu, Alibaba Group, Intel, and Microsoft to store and process petabyte-scale data using clusters built from commodity servers. Influenced by technologies such as the Google File System and MapReduce (programming model), Hadoop became part of enterprise analytics strategies alongside systems like Teradata, Oracle Database, IBM Db2, and newer cloud services from Google Cloud Platform and Amazon Web Services. The project fostered an ecosystem that interconnects with projects from Apache Software Foundation siblings such as Apache Hive, Apache HBase, Apache Spark, and Apache ZooKeeper.
Architecture
Hadoop's architecture uses a master–worker model with components responsible for storage and resource management. The design emphasizes fault tolerance, replication, and locality of reference, concepts also central to systems like Cassandra, MongoDB, CockroachDB, and Redis. Clusters are commonly deployed in data centers operated by Equinix, Digital Realty, or cloud regions from providers like Amazon Web Services, Microsoft Azure, and Google Cloud Platform. Architects often compare Hadoop's design trade-offs with distributed systems theory exemplified by the CAP theorem and research from institutions such as MIT, Stanford University, and UC Berkeley.
Core Components
Hadoop's core components include a distributed file system and a resource scheduler. The distributed file system is conceptually similar to the Google File System and interoperates with object storage offerings from Amazon S3, Google Cloud Storage, and OpenStack Swift. The resource scheduler and job tracker role has evolved with projects like Apache YARN and integrations with Kubernetes. Processing engines historically include MapReduce (programming model) and later integrations with Apache Spark, Apache Tez, and Apache Flink. Storage and NoSQL capabilities are often provided through integrations with Apache HBase and connectors to Apache Cassandra and MongoDB.
Ecosystem and Integrations
The Hadoop ecosystem includes data warehousing, query, ingestion, and management projects. Query engines and SQL layers include Apache Hive, Apache Impala, Presto (SQL query engine), and Trino (software). Data ingestion and streaming are handled by Apache Kafka, Apache Flume, Apache Sqoop, and Apache NiFi. Coordination and metadata services involve Apache ZooKeeper, Apache Oozie, Apache Atlas, and Cloudera Manager. Commercial distributions and companies built around the ecosystem include Cloudera, Hortonworks, MapR Technologies, IBM, and Amazon Web Services offerings such as Amazon EMR. Integration patterns also link Hadoop stacks with analytics platforms sold by SAP, SAS Institute, Oracle Corporation, and cloud analytics from Google BigQuery.
Deployment and Operations
Hadoop deployments run on private clusters, colocation facilities, and cloud infrastructure from providers such as Amazon Web Services, Microsoft Azure, and Google Cloud Platform. Operations tasks draw on tooling and practices developed by enterprises like Yahoo!, Facebook, LinkedIn, and Netflix for monitoring, capacity planning, and failure recovery. Management tooling includes Cloudera Manager, Ambari, and proprietary orchestration from vendors like MapR Technologies and IBM. Automation and containerization strategies increasingly leverage Kubernetes, Docker, and configuration management from Ansible, Puppet, and Chef.
Use Cases and Industry Adoption
Hadoop has been applied to web indexing and search at companies such as Yahoo! and Baidu, log analytics at Facebook and Netflix, clickstream processing for Google Advertising-adjacent workflows, and recommendation systems at Amazon.com, Netflix, and Spotify. Telecom providers like Verizon and AT&T used Hadoop for network analytics; financial institutions including Goldman Sachs and JPMorgan Chase used Hadoop for risk analytics and fraud detection. Public sector and research institutions such as NASA, CERN, and National Institutes of Health applied Hadoop-style architectures for scientific workloads.
Security and Governance
Security and governance around Hadoop involve authentication, authorization, encryption, and metadata management. Authentication often uses Kerberos and integrates with directory services like Active Directory and protocols defined by organizations such as IETF. Authorization and auditing are supported by Apache projects and commercial offerings from Cloudera and Hortonworks, while metadata governance leverages Apache Atlas and catalog solutions tied to platforms offered by AWS, Google, and Microsoft. Regulatory compliance in industries overseen by bodies such as the Securities and Exchange Commission, European Commission, and national data protection authorities influences deployment patterns for encryption, data residency, and audit trails.