ClickHouse

ClickHouse
Garethdafyddjones · CC0 · source
Name	ClickHouse
Developer	Yandex; ClickHouse, Inc.
Initial release	2009
Programming language	C++
License	Apache License 2.0
Operating system	Linux, macOS, Windows (WSL)
Genre	Columnar database, OLAP

ClickHouse ClickHouse is an open-source, column-oriented database management system designed for online analytical processing (OLAP) workloads. Developed initially at Yandex and later commercialized by ClickHouse, Inc., it emphasizes high-throughput analytical queries, real-time data ingestion, and efficient storage for large datasets. Used in conjunction with infrastructure projects and analytics platforms from organizations such as Facebook, Google, Amazon (company), and Microsoft, ClickHouse competes in environments alongside systems like Apache Hadoop, Apache Spark, and Snowflake (company).

History

ClickHouse began as an in-house project at Yandex to address web analytics and log processing needs at scale, emerging amidst work on other large-scale projects like MapReduce-inspired systems and Yandex.Metrica. Early development was influenced by columnar storage research such as C-Store and production systems like Vertica. The system was progressively open-sourced, attracting contributors from companies including Cloudflare, Booking.com, and Zalando. Following community growth and enterprise interest, commercial efforts were organized under ClickHouse, Inc., and the project saw adoption in deployments alongside Kubernetes clusters, Docker, and orchestration technologies used by Netflix and Airbnb.

Architecture

ClickHouse implements a columnar storage engine optimized for analytical scans and vectorized execution, drawing architectural parallels with systems like MonetDB and Apache Parquet. Its core components include a storage layer with merge-tree families inspired by concepts from Log-Structured Merge-tree work and a query execution engine that uses SIMD-friendly routines similar to optimizations explored by Intel and AMD. For replication and distributed queries, ClickHouse offers designs comparable to approaches in Apache Cassandra and Google Bigtable, using replicated tables, leaderless replication patterns, and cluster-aware query routing found in systems at Facebook (company) and Twitter.

ClickHouse integrates with ecosystem services: metadata and coordination can use ZooKeeper (software) or alternatives preferred by projects at Confluent, while data ingestion pipelines frequently involve Kafka (software), Fluentd, and Logstash. Storage tiers can be backed by object stores such as Amazon S3 or distributed filesystems like HDFS, echoing deployment patterns from Hortonworks and Cloudera.

Features

ClickHouse provides rich SQL support influenced by PostgreSQL and MySQL dialects, including window functions and approximate algorithms comparable to those in Apache Druid. It implements columnar compression codecs that benefit from techniques championed by Google researchers and hardware acceleration from Intel instruction sets. Secondary features include materialized views, user-defined functions (UDFs), and array types that mirror capabilities in TimescaleDB and CrateDB.

High-availability capabilities are achieved through replicated MergeTree engines and distributed tables, offering query sharding strategies akin to those in Cassandra and CockroachDB. Security and access control features draw on patterns used in Oracle Corporation and IBM enterprise databases, while observability integrates with tools like Prometheus and Grafana for performance metrics and alerting.

Performance and Scalability

ClickHouse is engineered for columnar scans, vectorized processing, and late materialization, yielding throughput competitive with analytical engines employed by Facebook and Google BigQuery. Benchmarks historically compare ClickHouse favorably against Apache Impala and Presto (SQL query engine), especially for low-latency, high-concurrency scenarios found in advertising tech stacks used by The Trade Desk. Scalability is achieved via horizontal sharding, replication, and data partitioning strategies comparable to those in HBase deployments.

Performance tuning often leverages IO and CPU optimizations discussed by vendors like Intel and cloud providers including Amazon Web Services and Google Cloud Platform. Use of adaptive compression, bloom filters, and primary key designs help sustain performance at petabyte scales—approaches similar to those in Teradata and Snowflake (company) whitepapers. Real-world deployments at companies such as Yandex, Cloudflare, and eBay demonstrate ClickHouse handling billions of rows with sub-second query latencies.

Use Cases and Deployments

Typical use cases include real-time analytics for web traffic and telemetry as practiced by Yandex.Metrica and Google Analytics, network monitoring in patterns used by Cloudflare and Akamai Technologies, and business intelligence workloads akin to those supported by Tableau Software and Looker. ClickHouse is used in security telemetry pipelines similar to architectures by Splunk and in observability backends analogous to Datadog.

Deployments range from on-premises clusters integrated with Kubernetes or OpenShift to managed offerings on cloud platforms like Amazon Web Services and Microsoft Azure. Enterprises in finance, ad tech, and e-commerce—examples include Mail.ru Group and Booking.com—use ClickHouse for cost-effective analytics and long-term event storage.

Ecosystem and Integrations

The ClickHouse ecosystem includes client drivers and connectors for languages and platforms such as Python (programming language), Java (programming language), Golang and Node.js, and BI integrations with tools like Tableau and Power BI. Data ingestion commonly uses Apache Kafka, Fluentd, and Logstash, while data exchange formats include Apache Parquet and ORC (file format). Orchestration and automation are supported through Ansible (software), Terraform, and Helm charts popular in Kubernetes deployments.

Community projects and commercial products provide backups, observability, and managed service offerings similar to cloud-native services from Google Cloud Platform and Amazon Web Services. The project collaborates with ecosystem contributors from companies such as Cloudflare, Altinity, and Yandex to extend integrations for data catalogs, security platforms, and machine learning pipelines used by Databricks and H2O.ai.

Category:Column-oriented database management systems