Hadoop YARN

Hadoop YARN
Name	Hadoop YARN
Developer	Apache Software Foundation
Initial release	2012
Programming language	Java
Operating system	Cross-platform
License	Apache License 2.0

Hadoop YARN Hadoop YARN is the resource-management and job-scheduling component of a widely used distributed data-processing platform. It decouples resource management from data processing, enabling diverse compute frameworks to run on top of a unified cluster managed by an open-source project led by the Apache Software Foundation. YARN evolved from early designs used at organizations such as Yahoo! and Facebook, and it integrates with ecosystem projects like Apache Spark, Apache Hive, Apache HBase, Apache Flink, and Apache Tez.

Overview

YARN serves as the cluster operating system for large-scale deployments operated by companies including LinkedIn, Netflix, Uber Technologies, Twitter, and Airbnb. It arose during architectural discussions that involved contributors from Cloudera, Hortonworks, and MapR Technologies and was formalized in a major release coordinated by the Apache Software Foundation. The design responded to scalability challenges observed in deployments at Yahoo! and Facebook, and it enabled integration with analytic engines developed at institutions like UC Berkeley and laboratories such as Lawrence Berkeley National Laboratory.

Architecture

The architecture separates a global ResourceManager and per-node NodeManagers, echoing patterns used by systems developed at Google and research from UC Berkeley AMPLab. The ResourceManager performs cluster-wide resource allocation while individual NodeManagers manage containers on hosts that may run on infrastructures provided by Amazon Web Services, Google Cloud Platform, Microsoft Azure, or on-premises data centers operated by organizations like IBM and Oracle Corporation. Applications are launched via ApplicationMasters, conceptually similar to schedulers designed at Stanford University and systems like Mesos. The architecture supports pluggable components influenced by projects from Apache ZooKeeper and coordination patterns used in Kubernetes.

Resource Management and Scheduling

YARN exposes resource containers and negotiates allocations using scheduling policies that mirror approaches studied at MIT and implemented in enterprise products by Cloudera and Hortonworks. Built-in schedulers include the CapacityScheduler and the FairScheduler, which echo resource sharing mechanisms used in cloud offerings from Google and in cluster managers at Facebook. Capacity allocation and queue management allow multi-tenant policies comparable to those implemented at Netflix and Airbnb, while preemption and priority features borrow concepts from scheduling research at Carnegie Mellon University and industrial systems from Intel Corporation.

Application Lifecycle and APIs

Applications submit jobs through client APIs inspired by frameworks from Apache Hadoop predecessors and research prototypes from UC Berkeley AMPLab. The ApplicationMaster negotiates container lifecycle via RPC interfaces, similar in pattern to remote invocation systems used at Amazon and RPC frameworks designed by Google. Developers integrate libraries from projects such as Apache Spark, Apache Flink, Apache Tez, and Apache Storm to implement streaming or batch workloads. Job histories and logs are compatible with monitoring stacks built with tools from Elastic NV (formerly ElasticSearch), Grafana Labs, and Prometheus.

Deployment and High Availability

High-availability patterns for YARN replicate leader election and failover techniques codified by Apache ZooKeeper and used in distributed databases like Apache Cassandra and Etcd. Production deployments often leverage container orchestration from Kubernetes or virtualization platforms provided by VMware, and integration tooling has been produced by vendors including Cloudera and Hortonworks (now part of Cloudera). Disaster recovery strategies align with backup and replication practices employed at cloud providers such as Amazon Web Services and Google Cloud Platform.

Performance, Scalability, and Monitoring

Performance engineering for YARN draws on benchmarking methodologies from SPEC and cluster measurements published by Yahoo! Research and Facebook Research. Scalability targets were demonstrated in large clusters operated by Yahoo! and Twitter, with tuning guidance adopted by enterprises like LinkedIn and Netflix. Observability integrates log aggregation and metrics pipelines using components from Fluentd, Logstash, Prometheus, and visualization using Grafana or Kibana. Profiling and resource tracing often use techniques from distributed-tracing systems developed by Google and projects like OpenTracing and Jaeger.

Security and Multitenancy

Security features in YARN build on authentication and authorization mechanisms from Kerberos deployments common at University of California campuses and enterprise identity systems from Microsoft Corporation and Okta, Inc.. Multi-tenant isolation strategies use Linux containerization and cgroups introduced by Linux Foundation collaborators and are augmented by access control lists and token-based delegation resembling models used at Amazon Web Services and Google Cloud Platform. Compliance and audit practices follow patterns used in regulated organizations such as Bank of America and Goldman Sachs.

Category:Apache Hadoop