TPC-H

TPC-H
Name	TPC-H
Developer	Transaction Processing Performance Council
Release	1999
Domain	decision support benchmark
Website	Transaction Processing Performance Council

TPC-H

TPC-H is a decision support benchmark created by the Transaction Processing Performance Council to evaluate the performance of database systems executing complex ad hoc queries and concurrent data modifications. It models a wholesale supplier scenario similar to workloads studied by IBM, Oracle Corporation, Microsoft, Amazon Web Services, and Google Cloud Platform, and it has been used by vendors such as SAP SE, Teradata, PostgreSQL Global Development Group, and MySQL to compare system behavior on analytic tasks. The benchmark influenced standards and comparisons in contexts involving YottaDB, Vertica, Snowflake, Cloudera, and research at institutions like Stanford University, Massachusetts Institute of Technology, and University of California, Berkeley.

Overview

TPC-H was defined by the Transaction Processing Performance Council to represent decision support workloads characterized by complex read-mostly queries, joins, and aggregations applied to large datasets derived from a wholesale supplier business model. The schema uses tables named after business artifacts analogous to those in systems marketed by IBM Db2, Oracle Database, Microsoft SQL Server, Amazon Redshift, and Greenplum Database. Data scaling is controlled by a scale factor similar to the practice in benchmarks such as TPC-C and TPC-DS, facilitating results reported by vendors like Intel Corporation, AMD, NVIDIA, and cloud providers including Google Cloud Platform, Amazon Web Services, and Microsoft Azure.

Benchmark Specification

The specification prescribes a fixed schema, data generation methods, query set, and workload rules defined and published by the Transaction Processing Performance Council. It mandates data generation tools and seed distributions used by implementers ranging from enterprise vendors like SAP SE and Teradata to open-source projects like PostgreSQL and MariaDB Corporation. The rules address allowed optimizations and measurement artifacts, analogous to compliance governance found in ISO/IEC standards, and are enforced in published result submissions similar to practices at SPEC, JDBC, and ODBC conformance efforts.

Benchmark Queries and Workload

The workload comprises a set of 22 SQL queries that exercise joins, subqueries, nested aggregations, window-like computations, and nontrivial predicate selectivity; these features are comparable to query patterns analyzed in research at Carnegie Mellon University, University of Wisconsin–Madison, University of Toronto, and ETH Zurich. The queries reference tables analogous to invoices, orders, customers, suppliers, and parts—entities also modeled in commercial schemas used by SAP SE, Oracle Corporation, and SAS Institute. The benchmark includes a concurrent update stream and a refresh function, which vendors such as Teradata and Vertica use in published comparisons alongside trials by IBM Research and Microsoft Research.

Performance Metrics and Measurement Methodology

Results are reported primarily as composite metrics: QphH@Size (queries per hour) and Power and Throughput figures, reflecting single-stream and multi-stream performance; these metrics parallel approaches used by SPEC, TPC-C, and TPC-DS. Measurement methodology prescribes load, warmup, execution, and audit phases enforced by registration with the Transaction Processing Performance Council, similar to compliance steps in ISO/IEC and reporting conventions used by IEEE publications. Audited reports often include hardware stack details referencing processors from Intel Corporation and AMD, storage from Dell Technologies and NetApp, and networking from Cisco Systems and Arista Networks.

Implementations and Results

Implementations have been published by vendors including IBM, Oracle Corporation, Microsoft, Amazon Web Services, Google Cloud Platform, Snowflake, Teradata, SAP SE, Vertica, Greenplum, and open-source systems such as PostgreSQL and MySQL. Research implementations and optimizations are described in papers from SIGMOD, VLDB, ICDE, and ACL-adjacent venues, with experimental comparisons at Stanford University, MIT, UC Berkeley, and ETH Zurich. Results appear in vendor whitepapers, academic articles, and TPC submissions, often comparing CPU families like Intel Xeon and AMD EPYC and storage technologies including NVMe, SCSI, SATA, and object storage offerings from Amazon S3.

Criticisms and Limitations

Critics from academia and industry such as authors affiliated with ACM SIGMOD, VLDB Endowment, University of California, Berkeley, and ETH Zurich have argued that the benchmark’s schema and workload may not reflect modern cloud-native analytics and streaming patterns found in systems such as Apache Kafka, Apache Spark, Flink, Druid, and Presto. Observers at Google Research and Microsoft Research have noted that optimizations tailored to the fixed query set can produce nonrepresentative tuning, similar to concerns raised about SPEC and other synthetic benchmarks. Limitations include the absence of semi-structured data types prevalent in deployments by Netflix, Facebook, Twitter, and LinkedIn, and limited coverage of concurrency models used by Kubernetes-orchestrated services and Hadoop ecosystems.

Historical Development and Impact

The benchmark was developed in the late 1990s and formalized by the Transaction Processing Performance Council; its adoption influenced performance reporting at vendors including IBM, Oracle Corporation, Microsoft, Teradata, SAP SE, and later cloud providers such as Amazon Web Services, Google Cloud Platform, and Microsoft Azure. It spurred academic work at institutions like Stanford University, MIT, UC Berkeley, Carnegie Mellon University, and ETH Zurich on query optimization, indexing, and storage architectures. The benchmark’s role in commercial and academic evaluations parallels the influence of other artifacts such as TPC-C, SPEC, TPC-DS, and standards promoted by ISO/IEC and has informed the design of analytic platforms used by enterprises like Walmart, Target Corporation, Procter & Gamble, and General Electric.

Category:Benchmarks