Resilient Distributed Datasets

Resilient Distributed Datasets
Apache Software Foundation · Apache License 2.0 · source
Name	Resilient Distributed Datasets
Developer	Apache Software Foundation
Initial release	2012
Programming languages	Scala (programming language), Java (programming language), Python (programming language)
Operating system	Linux, Microsoft Windows, macOS
License	Apache License

Resilient Distributed Datasets are a programming abstraction introduced to enable fault-tolerant, distributed in-memory computation across clusters managed by systems such as Apache Hadoop, Apache Mesos, and Kubernetes (software). They were first presented by researchers affiliated with University of California, Berkeley, associated projects like AMPLab and collaborators including members linked to Berkeley RISELab and the Apache Software Foundation community. The abstraction influenced subsequent systems from organizations such as Intel, IBM, Microsoft, and Google (company) via research, products, and academic citations.

Overview

Resilient Distributed Datasets provide an immutable, partitioned collection abstraction designed for parallel operations across nodes in clusters managed by resource managers like Apache YARN, Mesosphere, and Kubernetes (software). The model emerged from work at University of California, Berkeley's AMPLab and was commercialized by companies such as Databricks while interacting with ecosystems including Apache Hadoop, Apache Spark, and Apache Flink. Key contributors include researchers from MIT, Stanford University, and industry teams at Intel and IBM Research whose publications were discussed at venues like SIGMOD, VLDB, and OSDI.

Design and Concepts

The core design centers on an immutable, distributed collection partitioned across worker nodes in clusters overseen by platforms such as Apache Mesos and Apache Hadoop YARN. The abstraction exposes coarse-grained transformations, lineage-based metadata, and deterministic recomputation strategies, connecting to storage systems like Hadoop Distributed File System and object stores used by Amazon Web Services, Microsoft Azure, and Google Cloud Platform. The concept draws on ideas from distributed systems research by groups at Carnegie Mellon University, University of California, Berkeley, and ETH Zurich, with ties to fault-tolerance theory explored at University of Cambridge and Princeton University.

Programming Model and APIs

APIs are provided in languages developed or used at institutions like University of California, Berkeley and companies including Databricks and Cloudera (company), notably in Scala (programming language), Java (programming language), and Python (programming language). The model supports transformations and actions that map to batch and interactive workloads familiar to practitioners from Facebook, Twitter, Netflix, and LinkedIn. Integration points and bindings have been discussed in engineering teams at Microsoft, Oracle Corporation, and research groups at University of Illinois Urbana–Champaign and Columbia University when adapting APIs for streaming, SQL, and machine learning libraries such as those developed by Google Research and OpenAI collaborators.

Fault Tolerance and Lineage

Fault tolerance relies on lineage graphs and deterministic recomputation rather than fine-grained replication, influenced by earlier research at Bell Labs, IBM Research, and academic work presented at PODC and SOSP. Lineage metadata permits recomputing lost partitions using task schedules coordinated by cluster managers like Apache YARN and Mesos, and storage systems such as Hadoop Distributed File System and Amazon S3. The techniques relate to checkpointing and recovery strategies explored by researchers at University of Illinois Urbana–Champaign and institutions such as Lawrence Berkeley National Laboratory.

Performance and Optimizations

Optimizations include in-memory caching, data locality scheduling aligned with cluster managers from Apache Mesos and Apache Hadoop, and operator fusion that parallels research from Stanford University and MIT. Performance engineering has been conducted by teams at Databricks, Uber, Airbnb, and LinkedIn to support large-scale workloads, and by cloud providers Amazon Web Services, Microsoft Azure, and Google Cloud Platform to tune networking and storage paths. Compiler-like physical planning and query optimization ideas were informed by work at University of Wisconsin–Madison and Carnegie Mellon University and by query execution engines from Oracle Corporation and IBM.

Use Cases and Applications

Applications span batch analytics, iterative machine learning, graph processing, and ETL used by companies such as Facebook, Twitter, Netflix, Airbnb, Uber, and LinkedIn. Scientific computing groups at Lawrence Berkeley National Laboratory, CERN, and NASA explored RDD-style abstractions for processing large datasets, while finance and telecom firms like Goldman Sachs and AT&T leveraged distributed in-memory primitives. Educational and research deployments occurred at University of California, Berkeley, Stanford University, and Massachusetts Institute of Technology where RDD patterns informed courses and publications.

Implementations and Ecosystem Integration

Implementations and integrations include engines and projects within the Apache Software Foundation ecosystem such as Apache Spark, connectors for Hadoop Distributed File System, cloud integrations for Amazon S3 and Google Cloud Storage, and enterprise support through vendors like Cloudera (company) and MapR. The abstraction influenced related systems and libraries from Google (company), Microsoft Research, IBM Research, and startups like Databricks and integrations with SQL-on-Hadoop projects and streaming systems including Apache Flink and Apache Kafka.

Category:Distributed computing