perf (Linux tool)

perf (Linux tool)
Name	perf
Title	perf (Linux tool)
Developer	Linus Torvalds kernel community
Programming language	C (programming language)
Operating system	Linux
License	GNU General Public License
Website	Kernel.org

perf (Linux tool) perf is a performance analysis utility integrated into the Linux kernel toolchain that provides profiling, counting, and tracing for applications, subsystems, and the kernel itself. Originating within the Linux kernel development community, it is distributed alongside kernel sources and is used by contributors from projects such as Red Hat, Canonical, SUSE, and Intel for system tuning, benchmarking, and debugging. perf is commonly employed in environments involving Google, Facebook, Amazon, Microsoft, and Netflix to analyze production workloads and optimize performance.

Overview

perf provides command-line access to hardware and software performance counters exposed by the Linux kernel via the perf_event API, which was designed by contributors including developers from OProfile and LWN.net discussions. The tool interfaces with processor-specific PMU features present on platforms from Intel Corporation, Advanced Micro Devices, ARM Ltd., IBM POWER, and Fujitsu to collect events such as CPU cycles, cache misses, branch mispredictions, and context switches. perf integrates with build systems and continuous integration pipelines used by organizations like Jenkins, Travis CI, and GitLab to capture performance regressions and guide optimization efforts involving projects such as Kubernetes, Docker, PostgreSQL, and NGINX.

Features

perf supports a range of functions including sampling-based profiling, tracepoint recording, hardware counter aggregation, and callgraph collection used in performance investigations of systems running Systemd or Init-style init systems. Features include event multiplexing, per-thread and per-cgroup monitoring relevant to systemd-cgroup users, stack unwinding with support for DWARF and frame pointers, and the ability to export data for visualization tools like FlameGraph and Brendan Gregg’s tooling. perf records data in formats consumable by downstream projects such as BPF Compiler Collection and eBPF tools, and interplays with observability stacks like Prometheus, Grafana, and Elastic Stack.

Architecture and Components

perf relies on the perf_event_open syscall and kernel perf_event subsystem maintained in the Linux kernel tree overseen by maintainers and contributors from organizations like Red Hat and Intel. Core components include perf record, perf stat, perf top, perf script, and perf report, which interact with kernel facilities such as perf_events, perf tracepoints, and ftrace hooks introduced by developers active in Netdev and Kernel Summit discussions. perf uses kernel features like perf PMU drivers, tracepoints defined by subsystems (for example block layer, tcp stack, sched), and userland helpers that parse ELF symbols from binaries produced by toolchains like GCC, Clang (compiler), and Binutils. The toolchain integrates with debugging symbols from GDB and symbol servers maintained by projects such as Debian, Fedora, and Ubuntu.

Usage and Examples

Common workflows include using perf stat to gather aggregate counters during runs of workloads like Redis, MySQL, Apache HTTP Server, or TensorFlow training; using perf record and perf report to sample hotspots in microservices deployed on Kubernetes clusters; and using perf top to interactively inspect top-consuming functions on Linux nodes in data centers operated by Google Cloud Platform or Amazon Web Services. Examples: perf stat -e cycles,instructions -p to measure CPI for processes instrumented by CI systems managed within GitHub or GitLab CE; perf record -F 99 -g -- sleep 10 followed by perf report to generate callgraphs for workloads compiled with GCC and debug info from DWARF. Advanced users pipe perf script output into tools maintained by Brendan Gregg or into visualization projects such as Speedscope and FlameGraph for latency analysis of services like NGINX and HAProxy.

Performance Events and Counters

perf exposes hardware performance monitoring unit (PMU) events including cycles, instructions, cache references, cache misses, branch-instructions, and branch-misses standardized across processor vendors such as Intel, AMD, and ARM. It also exposes software events provided by the Linux kernel such as context-switches, cpu-migrations, page-faults, and minor/major fault counts relevant to subsystem developers in Kernel.org and enterprises like Red Hat and SUSE. perf supports custom events through PMU drivers for devices produced by NVIDIA, Qualcomm, Broadcom, and Marvell, and integrates with generic events defined by standards bodies and working groups active in IEEE and Open Source communities.

Development and Extensibility

perf is developed as part of the Linux kernel source tree and the perf tools repository hosted and reviewed by maintainers and contributors affiliated with Kernel.org, GitHub, and distribution vendors like Debian and Fedora. Extensions include BPF-based tracing integration via bcc and bpftrace, plugins for visualizers like Hotspot, and scripts maintained by performance engineers at Netflix and Facebook. Contributions follow workflows described in Linux kernel mailing list (LKML) discussions and are governed by the GNU General Public License and contributor agreements used by projects like The Linux Foundation.

Security and Permissions

perf interacts with low-level kernel interfaces and thus is subject to permission controls enforced by Linux capability bits, /proc and /sys interfaces, and security modules such as SELinux, AppArmor, and Smack. Access to raw PMU counters and perf_event_open may be restricted by sysctl settings and cgroup configurations used by container orchestration platforms like Docker, Kubernetes, and systemd-nspawn to prevent side-channel leakage exploited in research exhibited at conferences such as Black Hat and DEF CON. Distribution vendors like Ubuntu, Red Hat, and SUSE provide guidance on setting kernel.perf_event_paranoid and capability management to balance observability needs with attack surface reduction.

Category:Linux performance tools