trace-cmd

trace-cmd
Name	trace-cmd
Developer	Linux Kernel community
Released	2009
Programming language	C, Python
Operating system	Linux
License	GPL

trace-cmd

trace-cmd is a Linux tracing utility that records and analyzes kernel events by interfacing with the kernel's ftrace infrastructure. It provides command-line tools for recording, reporting, and extracting trace data, enabling developers and system administrators to diagnose performance issues and understand kernel behavior. trace-cmd is commonly used alongside other Linux performance tools and participates in debugging workflows in large-scale projects, embedded systems, and research environments.

Overview

trace-cmd acts as a front end to the kernel's ftrace facility, offering recording, reporting, and extraction commands. The tool is primarily associated with Linux kernel development, collaboration across communities like the Kernel.org project, and integration with performance analysis tools used in companies such as Intel, Red Hat, and Google. As a user-space utility, it bridges interactions between the kernel tracing APIs and tooling ecosystems including perf, SystemTap, and BPF-based frameworks.

History and Development

Development of trace-cmd began as part of efforts to make ftrace accessible to a broader set of contributors during the late 2000s. Key contributors and maintainers often collaborated with developers involved in the Linux Kernel Mailing List and projects sponsored by organizations such as Red Hat and Intel. Over time, trace-cmd evolved to support new kernel tracing features introduced in successive kernel releases, and contributors from companies like IBM and Samsung have incorporated support for platform-specific tracing needs. The project has been cited in presentations at conferences such as LinuxCon, Embedded Linux Conference, and KVM Forum.

Features and Components

trace-cmd provides several core components: a recorder that captures trace events, a report utility that formats traces into human-readable output, and an extract tool that produces per-CPU or per-thread trace data. It supports event types exposed by the kernel including function tracing, sched events, interrupt events, and custom tracepoints registered by subsystems in projects like the Linux kernel, KVM, and networking stacks maintained by teams at Cisco and Broadcom. Features include clock synchronization options used in distributed tracing scenarios, filtering by process identifiers common in debugging workflows at companies such as Facebook and Amazon, and integration paths to visualization tools employed by research groups at universities like MIT and Stanford.

Usage and Command Examples

Common workflows involve recording a trace, saving a data file, and generating a report. Examples include recording scheduling activity during a workload run, or capturing tracepoints during driver initialization. The recorder command is typically invoked with options to set buffer sizes and enable specific events. Users often combine trace-cmd with tools from projects like perf, SystemTap, and the LTTng stack when constructing end-to-end performance analysis pipelines in enterprise environments including Microsoft cloud infrastructure and telecommunications firms like Ericsson.

Integration with Kernel Tracing (ftrace)

trace-cmd interfaces directly with the kernel's ftrace API exposed through debugfs entries created by kernel subsystems. It manipulates ftrace controls such as tracer selection, event enabling, and buffer management that are central to kernel developers working on subsystems like the scheduler, virtual memory, and block layer. This tight integration allows trace-cmd to leverage kernel features added in successive Linux releases, and makes it a complementary tool for developers contributing to projects hosted on Git repositories managed by organizations like GitHub and GitLab.

File Formats and Output Processing

trace-cmd stores captured data in a binary format that encapsulates ftrace ring-buffer contents, metadata, and event descriptions. The report and extract utilities convert these binary files into text or per-thread traces suitable for further processing by analyzers and visualizers such as FlameGraph generators used in research at universities and industry performance teams. Outputs are often post-processed in scripting environments like Python and tools maintained by communities around projects such as perf-tools and FlameGraph authors.

Adoption and Use Cases

trace-cmd is used in kernel debugging, performance profiling, driver development, and system bring-up for platforms produced by companies like Intel, AMD, ARM, and NVIDIA. It supports development workflows in open-source projects hosted by organizations including the Linux Foundation, Apache Software Foundation, and various academic labs. Typical use cases include latency analysis for real-time systems in automotive projects, I/O tracing for storage vendors, and virtualization tracing for hypervisor development in projects like QEMU and KVM.

Category:Linux performance tools