Apache Hive

Apache Hive
Name	Apache Hive
Developer	Apache Software Foundation
Released	2010
Programming language	Java
License	Apache License 2.0
Repository	Apache Hive Git

Apache Hive Apache Hive is a data warehousing and SQL-like query system built for large-scale data processing on distributed storage platforms. It provides a high-level language for querying and managing large datasets, integrates with popular Hadoop ecosystem projects, and is widely used in enterprise analytics, research institutions, and cloud services.

Overview

Hive originated at Facebook to enable ad-hoc and batch analytics on large datasets stored in Hadoop Distributed File System clusters, integrating with projects such as Apache Hadoop, Apache HBase, Apache Tez, Apache Spark, and Apache Parquet. The project was contributed to the Apache Software Foundation and developed by contributors from organizations including Cloudera, Hortonworks, MapR, Microsoft, and Amazon Web Services. Hive presents a table-oriented view over files and directories and exposes a language that resembles SQL to users familiar with Oracle Database, MySQL, PostgreSQL, and other relational systems. It has evolved to support interactive, batch, and streaming workloads, competing and complementing systems such as Presto (SQL query engine), Apache Impala, and Google BigQuery.

Architecture

Hive's architecture comprises a client, a compiler, a optimizer, an execution engine, and a metastore; these components interact with storage and compute projects like YARN, Apache Tez, Apache Spark, and MapReduce. The Hive Metastore is an essential service that stores metadata in backends such as MySQL, PostgreSQL, Apache Derby, and Oracle Database and integrates with catalog services like AWS Glue and Apache Ranger. Query compilation produces logical and physical plans which the optimizer transforms using rule-based and cost-based techniques influenced by research from System R, Volcano optimizer, and academic work at UC Berkeley and Stanford University. Execution interfaces enable tasks to run on engines including Hadoop MapReduce, Apache Tez, and the Spark SQL layer from Apache Spark.

Query Language and Compatibility

HiveQL, Hive's SQL-like language, provides DDL, DML, and SELECT capabilities similar to ANSI SQL and extensions inspired by systems like Teradata and Netezza. Hive supports CREATE TABLE, ALTER TABLE, INSERT, and distributed joins, window functions, and user-defined functions (UDFs) which developers implement in Java, Python, or Scala as in projects such as Apache Pig and Apache Mahout. Compatibility layers and connectors allow integration with ODBC and JDBC clients, business intelligence tools from Tableau Software, MicroStrategy, and Qlik, and orchestration platforms like Apache Airflow and Oozie.

Storage and File Formats

Hive stores data on distributed file systems like Hadoop Distributed File System and object stores including Amazon S3, Google Cloud Storage, and Azure Blob Storage via connectors maintained by Amazon Web Services, Google Cloud Platform, and Microsoft Azure. It supports columnar and row-oriented file formats such as Apache Parquet, Apache ORC, Avro (data serialization system), and Text file formats, and integrates compression codecs like Snappy, Zlib, and LZO. Hive also interoperates with table formats and transaction systems introduced in projects like Apache Hudi, Delta Lake, and Iceberg to enable ACID semantics and time-travel queries for workloads common at Uber, Netflix, and LinkedIn.

Performance and Optimization

Performance in Hive is improved through techniques such as partitioning, bucketing, vectorized execution, predicate pushdown, cost-based optimization (CBO), and materialized views, many of which draw on research implemented in Apache Calcite and optimizers from Google research papers. Execution via Apache Tez and Spark provides lower-latency queries compared with Hadoop MapReduce, while file format choices like ORC and Parquet enable predicate pushdown and column pruning used by companies like Facebook and Alibaba Group. Query tuning often involves statistics gathering in the metastore, join reordering, and using indexing and Bloom filters as seen in systems like Lucene and HBase integrations.

Use Cases and Adoption

Hive is used for ETL pipelines, interactive analytics, batch reporting, machine learning feature extraction, and log analysis by enterprises including Facebook, Yahoo!, Twitter, Netflix, Airbnb, and financial institutions such as JPMorgan Chase and Goldman Sachs. It underpins data lakes on cloud platforms like Amazon EMR, Google Dataproc, and Azure HDInsight and is integrated into data workflows alongside Apache Kafka, Apache NiFi, and Apache Flink. Academic and research institutions use Hive for large-scale genomics, astrophysics, and social science workloads conducted at CERN, Lawrence Berkeley National Laboratory, and major universities.

Security and Administration

Security and administration for Hive rely on integration with projects like Apache Ranger, Apache Sentry, Kerberos, and LDAP for authentication, authorization, auditing, and encryption; cloud offerings augment controls with AWS IAM, Google Cloud IAM, and Azure Active Directory. Administrators manage resource isolation using YARN queues, Apache Ambari and Cloudera Manager for provisioning, and backup/restore strategies tied to Hadoop Distributed File System snapshots and object store versioning used by Amazon S3. Governance and cataloging are supported through integration with Apache Atlas and metadata practices common in enterprises complying with regulations such as those enforced by Securities and Exchange Commission and General Data Protection Regulation.

Category:Apache Software Foundation projects Category:Data warehousing