Hazelcast Jet

Hazelcast Jet
Name	Hazelcast Jet
Developer	Hazelcast, Inc.
Released	2016
Programming language	Java
Operating system	Cross-platform
Platform	Java Virtual Machine
Genre	Distributed stream processing, event processing
License	Apache License 2.0

Hazelcast Jet Hazelcast Jet is a distributed, high-performance stream and batch processing engine designed for low-latency, high-throughput data processing on the Java Virtual Machine. It integrates with in-memory data grids, message brokers, and storage systems to support real-time analytics, complex event processing, and ETL pipelines across clusters. Jet is used alongside technologies such as Apache Kafka, Apache Cassandra, Redis, MongoDB, Elasticsearch, and Kubernetes in production systems operated by enterprises and research organizations.

Overview

Hazelcast Jet provides a data-parallel processing model for event streams and bounded datasets, enabling continuous computation, windowing, and joins. It targets workloads similar to those addressed by Apache Flink, Apache Spark, Apache Storm, Google Dataflow, and Microsoft Azure Stream Analytics, while emphasizing in-memory processing and embedded deployment patterns. Jet interoperates with ecosystems including RabbitMQ, Amazon Kinesis, Apache Pulsar, Confluent Platform, and cloud platforms like Amazon Web Services, Microsoft Azure, and Google Cloud Platform.

Architecture

Jet's architecture centers on a distributed execution engine that schedules directed acyclic graphs (DAGs) of processing vertices and edges across a cluster of JVM-based nodes. The engine borrows concepts from MapReduce, MPI, and actor-model systems such as Akka to implement task distribution, backpressure, and fault isolation. Core components include the job coordinator, execution vertices, distributed snapshots, and local executors that run on nodes provisioned by orchestration systems like Docker Swarm and Kubernetes. Storage and state integration points connect to HDFS, Amazon S3, Azure Blob Storage, and databases including PostgreSQL and MySQL.

Programming Model and APIs

Jet exposes a fluent Java DSL and higher-level APIs for constructing DAGs, sources, sinks, and processors, comparable to APIs in Apache Beam and Spring Boot-based microservices. It offers connectors for JDBC, gRPC, Thrift, and messaging systems such as ActiveMQ and ZeroMQ to ingest and emit data. The API supports windowing semantics analogous to those in Google Cloud Dataflow, time semantics used in Event Sourcing patterns, and exactly-once semantics similar to implementations in Apache Flink and Kafka Streams. Jet applications can be embedded in JVM applications as libraries or deployed as standalone clusters managed alongside HashiCorp Consul and HashiCorp Nomad.

Deployment and Scalability

Jet can be deployed embedded within JVM processes, as a standalone cluster, or containerized for orchestration by Kubernetes, Mesos, or Docker. It scales horizontally by adding nodes and redistributing processing partitions, leveraging partitioning strategies familiar to users of Cassandra and Hazelcast IMDG. Resource management and autoscaling workflows often integrate with Prometheus, Grafana, and Kubernetes Horizontal Pod Autoscaler for telemetry and metrics. Large deployments coexist with service meshes like Istio and observability stacks including Jaeger and Zipkin for tracing.

Fault Tolerance and State Management

Jet implements snapshots and state persistence mechanisms compatible with checkpoints and distributed snapshots inspired by the Chandy–Lamport algorithm and systems such as Apache Flink. It persists operator state to durable stores like Amazon S3 or HDFS and supports recovery to ensure processing guarantees under node failure scenarios similar to those addressed in Zookeeper-coordinated systems. For coordination and leader election, Jet integrates with components like Apache ZooKeeper and etcd, while offering integration points for transaction managers used in Two-phase commit workflows and SAGA (software pattern) orchestration.

Use Cases and Performance

Typical use cases include real-time fraud detection for financial institutions interacting with SWIFT and Visa, telemetry and observability pipelines for Netflix-scale streaming architectures, sensor data processing for industrial Siemens deployments, and log analytics combined with Elasticsearch and Fluentd. Performance benchmarks position Jet alongside engines such as Apache Flink and Apache Spark Structured Streaming for low-latency stream processing workloads, with optimizations for in-memory joins, window aggregation, and stateful operations often compared to Google Spanner-related workloads in latency-sensitive contexts. Operational integrations include monitoring via Prometheus exporters and alerting with PagerDuty.

History and Development

Jet originated within the company Hazelcast, Inc., first released around 2016 as part of efforts to provide streaming capabilities adjacent to the Hazelcast IMDG product line. Its evolution reflects influences from academic research and industrial projects like Lambda architecture implementations, innovations from LinkedIn's stream processing initiatives such as Apache Samza, and open-source communities around Apache Software Foundation projects. Development and maintenance have been driven by engineering teams collaborating with users in sectors including telecommunications, finance, and e-commerce, and the project has participated in conferences and meetups alongside events like Strata Data Conference and Kafka Summit.

Category:Stream processing engines