Apache Spark Streaming

Apache Spark Streaming
Name	Apache Spark Streaming
Developer	Apache Software Foundation
Initial release	2013
Programming language	Scala, Java
Operating system	Cross-platform
License	Apache License 2.0

Apache Spark Streaming Apache Spark Streaming is a scalable, fault-tolerant stream processing engine that extends Apache Spark for real-time data processing. It enables developers to build streaming applications that integrate with batch processing, machine learning, and graph processing from the Apache Software Foundation ecosystem. Used in production by companies and institutions such as Netflix (company), Uber Technologies, Inc., Airbnb, Inc., Yahoo!, and Spotify, it supports ingestion from systems like Apache Kafka, Amazon Kinesis Data Streams, and Flume.

Overview

Spark Streaming emerged to bridge real-time and batch analytics in big data platforms including Hadoop Distributed File System clusters and Amazon Web Services deployments. It processes live data using discretized streams (micro-batches) built on the core Apache Spark engine, interacting with projects such as Apache Flink in comparative evaluations and influencing research at universities like University of California, Berkeley and Massachusetts Institute of Technology. Production adopters include LinkedIn, Pinterest, Shopify, and Tencent, while vendors like Databricks offer managed services that integrate with Microsoft Azure and Google Cloud Platform.

Architecture and Components

The architecture centers on receivers, input streams, and DStream abstractions layered on Resilient Distributed Datasets, leveraging cluster managers such as Apache Mesos, Hadoop YARN, and Kubernetes. Components include sources (connectors to Apache Kafka, Amazon Kinesis Data Streams, RabbitMQ), processing stages influenced by libraries like MLlib for machine learning and GraphX for graph analytics, and sinks integrating with HBase, Cassandra (database), and Elasticsearch. The driver program coordinates executors across nodes provisioned by cloud providers such as Amazon EC2 and Google Compute Engine, and uses serialization frameworks like Kryo and Java (programming language) serialization.

Programming Model and APIs

APIs are provided in Scala (programming language), Java (programming language), and Python (programming language) with higher-level abstractions later paralleled by Structured Streaming and SQL interfaces inspired by Apache Hive and Apache Calcite. Developers implement transformations with functions and closures, integrating with libraries including TensorFlow, XGBoost, and scikit-learn via model serving systems like Seldon Core and TensorFlow Serving. Spark Streaming’s API supports operations similar to those in MapReduce and patterns found in Lambda architecture designs promoted by practitioners from Twitter, Inc. and Cloudera.

Windowing, State Management, and Fault Tolerance

Windowing semantics in micro-batch mode permit tumbling and sliding windows, enabling analytics comparable to approaches in Apache Flink and Google Cloud Dataflow. State management combines checkpointing to durable stores like HDFS and Amazon S3 with updateStateByKey and mapWithState primitives; these integrate with consistent storage systems such as Apache Zookeeper and PostgreSQL. Fault tolerance leverages lineage-based recomputation similar to concepts from Resilient Distributed Datasets and implements backpressure and rate control strategies informed by research from Berkeley AMPLab teams and industrial implementations at Netflix (company) and Goldman Sachs.

Deployment, Integration, and Ecosystem

Deployments occur on clusters managed by Kubernetes, Apache Mesos, or Hadoop YARN and are often packaged using tools from Ansible, Terraform, and Helm (software). Integrations span ingestion systems like Apache Kafka, Amazon Kinesis Data Streams, Flume, and Google Pub/Sub; storage targets such as HBase, Apache Cassandra, Delta Lake; and orchestration via Apache Airflow, Oozie, and Luigi (software). The ecosystem includes commercial distributions from Cloudera, MapR Technologies (historical), Hortonworks (historical), and managed offerings by Databricks and Amazon Web Services.

Use Cases and Performance Considerations

Common use cases include fraud detection for financial institutions like JPMorgan Chase, real-time recommendation engines used by Netflix (company) and Spotify, monitoring pipelines at telecom providers such as Verizon Communications and AT&T, and clickstream analytics for advertisers like Google LLC and Facebook, Inc.. Performance tuning addresses batch interval sizing, memory configuration, and serialization using tools from Ganglia, Prometheus, and Grafana for monitoring; benchmark studies compare throughput and latency against Apache Flink and Apache Storm in academic work from Carnegie Mellon University and industry labs at Yahoo!. Considerations include event time handling, watermarking approaches credited to researchers at Google Research, and integration with low-latency serving systems like Apache Druid and ClickHouse.

Category:Apache Spark Category:Stream processing