perf script

perf script
Name	perf script
Developer	Linux kernel community
Released	2010s
Programming language	C, Python (supporting scripts)
Operating system	Linux
License	GNU General Public License

perf script

perf script is a command-line tool in the Linux performance analysis ecosystem that decodes recorded performance events into human-readable traces. It is part of the perf suite maintained in the Linux kernel community and is commonly used by contributors to Red Hat products, engineers at Intel, and researchers collaborating with institutions such as the University of Cambridge and the Massachusetts Institute of Technology. The tool bridges low-level kernel counters and user-facing interpretation, enabling analysis by teams from organizations like Google and Facebook as well as independent projects supported by the Linux Foundation.

Overview

perf script reads event data produced by the perf record subcommand and translates binary samples into textual trace events, stack traces, and annotated function information. Maintained alongside the perf subsystem in the Linux kernel source tree, it cooperates with other projects including BPF tooling maintained under the iovisor community and observability stacks from vendors such as Elastic and Datadog. Developers from companies like Netflix and Spotify use perf script output to investigate performance regressions, bottlenecks, and latency issues in applications deployed on infrastructure managed with orchestration platforms like Kubernetes.

Usage and Options

The basic invocation reads a perf.data file created by perf record and writes decoded events to standard output. Common options include specifying input files, applying scripting extensions, and adjusting symbol resolution behavior. Enterprises such as Canonical and SUSE document typical flag combinations for system-level investigations, while contributors to the GNU Project have influenced the command-line ergonomics and localization. perf script supports embedding Python or Perl scripts to transform event streams; these extension capabilities mirror similar extensibility found in tools developed at Oracle and research groups at ETH Zurich.

Output Formats and Interpretation

Output modes include flat event lists, threaded traces with timestamps, and verbose stacks enriched with symbol names and source-level offsets. For interpreting samples from complex environments, teams at Amazon and academic groups at Princeton University correlate perf script output with other artifacts such as kernel oops logs from Red Hat Enterprise Linux or telemetry from Grafana dashboards. Symbol resolution leverages debuginfo packages maintained by distributions like Fedora and Debian, and integration with the GNU Binutils and DWARF standards yields function names, inlined frames, and file/line annotations where available.

Examples and Common Workflows

A typical workflow begins with perf record sampling a workload on a server provisioned through services like Amazon EC2 or Google Cloud Platform, then running perf script to obtain a human-readable trace. Engineers at Microsoft have documented combining perf script output with flame graph generation techniques popularized by developers associated with Brendan Gregg and visualization projects that integrate with FlameGraph tools. Common pipelines use system automation frameworks from Ansible or Puppet to collect perf.data across fleets, then centralized analysis with platforms supported by Splunk or academic reproducibility efforts at Stanford University.

Integration with Other perf Tools

perf script interoperates closely with perf report for interactive analysis and with perf annotate for line-level inspection. It complements eBPF-based tracing utilities such as those in the bcc project and the successor bpftrace tooling fostered by contributors from Netflix and the Cloud Native Computing Foundation. Debugging workflows often combine perf script with symbol server techniques used by large vendors like Apple (for analogous macOS tooling) and with kernel debug facilities maintained by teams at Google and the Linux Foundation.

Implementation and Internals

Under the hood, the implementation parses perf.data event records, resolves addresses via in-kernel and userspace symbol resolution code paths, and applies Python/Perl script callbacks through an embedded interpreter interface. The design draws on standards and libraries from the GNU Project and utilities from the Binutils ecosystem; kernel-side event format evolution is governed by maintainers contributing to the Linux kernel and reviewed in community channels like the LKML. Contributions from corporate engineers at Intel and research prototypes from institutions such as Carnegie Mellon University have shaped perf script’s handling of stack unwinding, frame pointer omission, and the use of DWARF unwind tables.

Troubleshooting and Performance Considerations

Common issues include missing symbols due to stripped binaries, incomplete stack traces from frame-pointer-omitted builds, and long processing times on large perf.data files. Organizations like Red Hat and SUSE recommend installing debuginfo packages and using build configurations that retain unwind information, while cloud operators at Amazon and Google advise sampling strategies to limit overhead and data volume. Performance tuning often involves adjusting perf record sampling frequency, using kernel options contributed by teams at Intel to reduce jitter, and applying post-processing scripts inspired by case studies from Netflix and academic performance labs at UC Berkeley.

Category:Linux performance tools