perf

perf
Name	perf
Developer	Linux kernel community
Released	2008
Operating system	Linux
License	GNU General Public License

perf perf is a performance analyzing tool for Linux kernel systems, used to measure and profile CPU, memory, and I/O behaviour on production and development hosts. It integrates with kernel tracing facilities and hardware performance counters to provide detailed sampling, event counting, and flame graphs for applications, subsystems, and firmware. perf is commonly used alongside other observability projects and distributions in enterprise and research environments.

Overview

perf is a command-line utility originating in the Linux kernel source tree to interface with kernel performance events, hardware counters from vendors such as Intel and AMD, and tracing frameworks like ftrace and eBPF. It supports statistical sampling, event-based profiling, and tracing for user-space binaries such as those produced by GCC, Clang, and runtime environments like JVM and Node.js. perf’s outputs are consumed by visualization tools and platforms including Flame Graphs creators, Grafana, and Kibana for observability workflows.

History and Development

perf was introduced to the Linux kernel development process in the mid-2000s and matured through contributions from kernel developers, companies such as Red Hat, Intel, Google, and academic groups at institutions like MIT and Stanford University. The tool evolved alongside kernel subsystems such as OProfile predecessors, ftrace, and the perf events infrastructure standardized via patches and discussions on the Linux Kernel Mailing List. Key milestones include integration of hardware event abstraction for Performance Monitoring Units, support for dynamic tracepoints associated with SystemTap and extensions accommodating eBPF tracing.

Features and Functionality

perf provides capabilities for event counting, sampling, call-graph capture, and context-switch tracing. Supported event sources include CPU hardware counters from Intel>

CPU line and AMD Ryzen microarchitectures, software events defined by the Linux kernel such as context-switch accounting, and tracepoints used by projects like systemd and OpenJDK. Features include per-thread and per-process measurement, system-wide profiling for services like Apache HTTP Server and Nginx, stack unwinding compatible with symbols generated by GDB and debuggers such as LLDB, and support for symbol resolution in binaries produced by Binutils toolchain.

Usage and Examples

Common invocations include sampling an application, recording events, and generating reports consumed by analysis tools. Example workflows pair perf record with perf report to inspect hotspots in applications like PostgreSQL, MySQL, Redis, web servers such as Apache HTTP Server and Nginx, and language runtimes including OpenJDK and Python interpreters. Integrations with continuous integration systems from Jenkins and GitLab enable automated regression detection, while containerized environments orchestrated by Kubernetes and packaged via Docker frequently use perf for container-level profiling and kernel tracing.

Architecture and Implementation

perf interfaces with the perf_events kernel subsystem implemented in the Linux kernel and uses system calls such as perf_event_open to configure and read performance counters. The userspace tool is written in C and links against libraries and symbol helpers from Glibc, libbfd (from Binutils), and can interoperate with systemtap and bpftrace via shared tracepoints. perf supports reading hardware registers on CPUs from Intel and AMD through standardized PMU abstractions and collaborates with scheduler hooks and tracepoints found in kernels used by distributions like Debian, Fedora, and Ubuntu.

Performance Metrics and Analysis

perf exposes metrics such as CPU cycles, instructions retired, cache misses, branch mispredictions, and context switches—metrics often analyzed in publications from ACM and IEEE conferences. The tool can produce flame graphs and annotated assembly views used in performance studies at labs such as Lawrence Berkeley National Laboratory and corporate performance groups at Facebook and Netflix. Analysis commonly involves correlating perf data with telemetry from monitoring systems like Prometheus and log aggregators such as Elasticsearch to attribute latency to code paths and external dependencies.

Integration and Tooling

perf is integrated into ecosystems via adapters and GUIs including perf-tools collections, visualization scripts for Flame Graphs by Brendan Gregg, and integrations for observability stacks like Prometheus exporters and Grafana dashboards. It complements other tracing and profiling utilities such as OProfile, SystemTap, bpftrace, and vendor tools from Intel VTune and AMD uProf. Packaging and distribution of perf binaries and kernel headers are maintained by distribution vendors including Debian, Red Hat, Ubuntu, and SUSE for use in development, CI pipelines with Travis CI and Jenkins, and research reproducibility efforts at universities like UC Berkeley.

Category:Linux software