Apache Parquet

Apache Parquet
Name	Apache Parquet
Developer	Apache Software Foundation
Written in	C++, Java, Python
Initial release	2013
License	Apache License 2.0

Apache Parquet

Apache Parquet is a columnar storage file format designed for efficient data storage and retrieval in analytic workloads. It targets large-scale data processing systems such as Hadoop, Spark, Presto, Hive and Impala, offering compact representation and fast columnar reads for integration with platforms like Amazon S3, Google Cloud Storage, Azure Blob Storage, and HDFS. Parquet is widely used across ecosystems involving Cloudera, Databricks, Confluent, Snowflake, and Tableau.

Overview

Parquet is a free and open-source project incubated at the Apache Software Foundation that provides a columnar, self-describing storage format optimized for big data analytics. The format emphasizes efficient compression and encoding schemes to reduce I/O and storage costs on systems such as Apache Hadoop YARN, Kubernetes, and services offered by Amazon Web Services, Google Cloud Platform, and Microsoft Azure. Parquet complements row-based formats like Avro and ORC, and integrates with query engines and data processing frameworks including Apache Flink, Apache Beam, PrestoDB, and Dremio.

Design and Architecture

Parquet employs a columnar layout inspired by research from institutions like Google, with per-column chunking and encoding to optimize analytic queries executed by engines such as Apache Spark SQL, Apache Drill, and Trino. Its logical model maps complex nested types (maps, lists, structs) from systems like Protocol Buffers, Thrift, and Avro into a flat columnar representation amenable to vectorized execution in runtimes like LLVM and GraalVM. The architecture separates metadata (file footer, row group indexes) from column data, supporting predicate pushdown and projection pruning in connectors used by PrestoSQL and Amazon Athena. Parquet’s schema evolution mechanisms enable compatibility across versions of tools developed by Cloudera and Hortonworks, and coordination with metadata services such as Apache Hive Metastore and AWS Glue.

File Format and Compression

Parquet files contain row groups, column chunks, pages, and a footer with Thrift-serialized metadata; these components support encodings like dictionary encoding and bit-packing, and compression codecs such as Snappy, Zstandard, Gzip, and LZO. Column-oriented storage allows selective decompression for columns accessed by queries in Snowflake, Redshift, and BigQuery, reducing CPU overhead compared to row formats used in MySQL, PostgreSQL, and Oracle Database. The file footer and metadata enable tools like Apache Ranger and Apache Atlas to perform governance, while Parquet’s support for statistics and min/max indices improves pruning in systems like Presto and Trino.

Implementations and Ecosystem Integration

Parquet has reference implementations in Apache Arrow, Parquet-mr and native libraries in C++, Java, and Python wrappers used by projects such as pandas, Dask, Modin, and Koalas. It integrates with data ingestion technologies like Apache Kafka, Sqoop, and Apache Flume, and is supported by data catalog and governance tools from Collibra, Informatica, and Alation. Cloud-native analytics stacks from Databricks and Google BigQuery leverage Parquet alongside columnar in-memory formats such as Arrow for fast serialization across services like Looker and Power BI.

Performance and Use Cases

Parquet is optimized for read-heavy analytical workloads in data warehousing, business intelligence, machine learning pipelines, and batch ETL jobs executed by Apache Spark, Hadoop MapReduce, Apache Airflow, and Apache NiFi. Typical use cases include log analytics for platforms like ELK Stack, time-series aggregation for systems such as Prometheus, and feature storage in ML workflows using frameworks like TensorFlow and PyTorch. Performance benefits arise from reduced disk I/O, effective compression with codecs used by Facebook, Twitter, and Netflix, and improved cache efficiency in distributed query engines such as Dremio and PrestoDB.

History and Governance

Parquet originated from a collaboration between engineers at Twitter and Cloudera and was subsequently contributed to the Apache Software Foundation as an incubator project before becoming a top-level project. Governance follows Apache’s meritocratic model with community-driven development, contributors from companies like Netflix, Uber, Pinterest, and LinkedIn, and release cycles coordinated via the Apache Software Foundation infrastructure. The project’s public mailing lists, issue trackers, and governance documents adhere to community processes similar to other ASF projects such as Apache Hadoop, Apache Spark, and Apache Arrow.

Category:File formats