Criterion (software)

Criterion (software)
Name	Criterion
Developer	Criterion Developers
Released	2013
Programming language	Rust
Operating system	Cross-platform
License	MIT

Criterion (software) is a benchmarking and testing framework designed for reliable measurements of program performance. It provides statistical analysis, measurement isolation, and reproducible reports suitable for use with compilers, continuous integration systems, and academic benchmarking projects. Criterion integrates with build tools and version control platforms to track regressions across commits and releases.

Overview

Criterion aims to offer robust, reproducible performance measurement comparable to efforts such as SPEC CPU, Phoronix Test Suite, Google Benchmark, Apache JMeter, and Istanbul (software). It targets developers working with compilers like GCC, Clang, and Rustc as well as projects hosted on platforms such as GitHub, GitLab, Bitbucket, and Azure DevOps. The project emphasizes statistical rigor similar to methodologies used by ACM SIGPLAN papers, IEEE benchmarking guidelines, and analyses found in SIGMETRICS proceedings.

Features

Criterion provides features commonly sought in performance tools including automatic warmup, outlier detection, confidence interval estimation, and regression detection similar to Valgrind and Perf (Linux). It supports parameterized benchmarks, deterministic seeding, and result export formats compatible with JSON, CSV, and visualization tools such as Grafana and Kibana. Integration hooks exist for continuous integration solutions like Travis CI, CircleCI, and Jenkins (software), enabling automated performance tracking in workflows akin to those used by Mozilla, Google, or Dropbox engineering teams.

Architecture and Design

Criterion's architecture follows modular patterns employed by systems such as LLVM, Rust compiler, Node.js, and TensorFlow. It separates measurement harness, statistical engine, and reporting front ends, allowing reuse across languages and runtimes including Rust, C++, Python (programming language), and Java. The design uses process isolation comparable to techniques in Docker, systemd, and chroot to reduce noise from background services like systemd-journald or cron. Statistical computations draw on methods published in venues like Journal of the ACM and implemented in libraries inspired by Boost C++ Libraries and SciPy.

Usage and Tooling

Users invoke Criterion via command-line interfaces patterned after tools such as Cargo (software), CMake, Make (software), and Bazel. It offers plugins and adapters for editors and IDEs including Visual Studio Code, CLion, IntelliJ IDEA, and Emacs. Typical workflows mirror those used in projects hosted by Mozilla, Red Hat, and Canonical (company), combining benchmarking with continuous integration from GitHub Actions and automated alerting through integrations similar to PagerDuty or Slack (software).

Performance and Benchmarking

Benchmark results from Criterion are intended to be comparable to published evaluations in conferences like Eurosys, OSDI, SOSP, and USENIX ATC. The framework reports median, mean, standard deviation, and confidence intervals using statistical techniques found in Biometrika and Journal of Statistical Software. Performance comparisons often appear alongside profiling data produced by perf (Linux), DTrace, gprof, and sampling tools from Intel VTune or AMD μProf. Criterion supports microbenchmark suites that mirror workloads used by projects such as SQLite, Redis, and Nginx for realistic evaluation.

History and Development

Criterion's early development drew inspiration from benchmarking practices used at organizations including Mozilla Corporation, Google LLC, and Facebook. Contributions have come from individuals active in communities around Rust, C++, and Linux kernel development, with issue tracking and code review workflows modeled on GitHub Flow and governance resembling foundations like the Apache Software Foundation or Linux Foundation. Over time the project adopted continuous integration patterns influenced by Travis CI, CircleCI, and GitHub Actions and expanded support for export formats and reporting integrations popularized by Grafana Labs and Elastic NV.

Category:Benchmarking software