JMH

JMH
Name	JMH
Developer	Oracle Corporation
Programming language	Java (programming language)
Operating system	Linux, Windows, macOS
License	BSD license

JMH

JMH is a Java harness for creating, running, and analyzing benchmarks of microbenchmarks on the Java Virtual Machine platform. It provides a suite of annotations, runners, profilers, and utilities designed to measure execution time, throughput, and other performance characteristics while minimizing measurement bias introduced by Just-in-Time compilation, HotSpot optimizations, and G1 tuning. JMH is widely used by engineers working on projects such as OpenJDK, Spring Framework, Apache Kafka, Netty, and Hibernate ORM to produce repeatable, statistically robust performance data.

Overview

JMH was created to address pitfalls encountered by developers benchmarking code paths in environments like Oracle Corporation's HotSpot and the OpenJDK project. It exposes facilities for warmup iterations, measurement iterations, forked JVMs, and sample modes to guard against effects from Just-in-Time compilation, adaptive optimizations, and escape analysis. The framework integrates with build tools such as Maven, Gradle, and Ant to simplify continuous benchmarking workflows for projects like Apache Cassandra, Elasticsearch, and Akka.

History and Development

JMH originated from performance tooling efforts inside the OpenJDK community and contributors associated with Oracle Labs and the Java Community Process. Key contributors included engineers involved in HotSpot and JIT research who encountered unreliable microbenchmark results when using ad hoc timing code. The project matured through collaboration between teams from organizations such as Red Hat, Twitter, and LinkedIn that needed consistent benchmark methodologies for systems like Hibernate Search and Apache Lucene. JMH has evolved via releases coordinated through GitHub repositories, issue trackers, and discussions with maintainers of OpenJDK and related Java SE initiatives.

Architecture and Features

JMH is built in Java (programming language) and leverages annotation-driven configuration to define benchmark methods, setup and teardown, and various measurement modes. Core annotations include @Benchmark, @State, @Setup, and @TearDown, which interact with execution control for warmup and measurement iterations. JMH supports modes such as Throughput, AverageTime, SampleTime, SingleShotTime, and All, enabling comparisons across workloads found in systems like Apache Tomcat, Jetty, Spring Boot, and Play Framework.

The harness isolates benchmarks by forking new JVM processes with specified JVM arguments, minimizing cross-test contamination from class loading and JVM TI interactions present in long-running services like GlassFish or WildFly. It emits results in human-readable and machine-readable formats, including CSV, JSON, and SCSV, to integrate with CI systems used by projects like Jenkins, Travis CI, and GitLab CI/CD. JMH includes profilers and features for measuring memory allocations, lock contention, and assembly output via tools such as Async-profiler, perf (Linux), and Java Flight Recorder.

Usage and Examples

Typical usage involves annotating a class and method with @State and @Benchmark, configuring forks, warmups, and iterations either via annotations or command-line options. Example integrations demonstrate benchmarking algorithms used in Guava, serialization frameworks like Kryo, and JSON libraries such as Jackson and Gson. Users often compare implementations of data structures from Apache Commons Collections versus Eclipse Collections or concurrency primitives from java.util.concurrent against frameworks like Disruptor.

JMH supports parametric benchmarks through @Param fields to explore matrixed inputs similar to experiments performed for Apache Hadoop and Apache Spark. Developers commonly combine JMH with build pipelines for projects such as Gradle, Maven Central, and Bintray to run nightly performance suites, reporting regressions detected against baselines generated by JMH.

Performance and Benchmarking

JMH focuses on statistical rigor by providing options for warmup iterations, measurement iterations, forks, and confidence intervals to mitigate transient effects seen in platforms like OpenJDK builds and GraalVM. Benchmarks report metrics such as operations per second, average latency, standard deviation, and percentiles, helping performance engineers working on NetBeans or IntelliJ IDEA identify regressions. JMH’s support for Blackhole consumption and control over dead-code elimination aids in avoiding misleading results that plagued early benchmarks in projects like Scala (programming language) and Clojure implementations on the JVM.

Integration with profilers enables flamegraph generation and inlining analysis useful to contributors to OpenJ9 and Graal projects. JMH benchmarks have been instrumental in tuning garbage collectors across HotSpot options including Parallel, G1, and Z Garbage Collector to evaluate allocation patterns for services like Apache Kafka and Cassandra.

Adoption and Ecosystem

JMH is adopted across many notable Java ecosystems and projects including OpenJDK, Spring Framework, Apache Software Foundation projects like Kafka, Cassandra, Tomcat, and Elasticsearch, as well as commercial vendors such as Oracle Corporation and Red Hat. The ecosystem includes integrations, plugins, and educational materials from organizations like JetBrains, Lightbend, and Confluent. Community contributions have produced wrappers and visualizers compatible with CI platforms like CircleCI and dashboards used by teams at Google and Facebook that maintain JVM-based services.

Category:Java benchmarking tools