Re: [RFC PATCH v2 0/5] Add perf.data tracepoint events to trace.dat conversion

[Tanushree Shah <tshah@linux.ibm.com>]

Hello Ian, Thank you very much for the feedback.

yes, v2 needed a rebase. I had already sent V3 patches on Monday.
Below is the link:
https://lore.kernel.org/all/20260803145958.299956-1-tshah@linux.ibm.com/.

Thanks.
Tanushree Shah


On 05/08/26 07:37, Ian Rogers wrote:
> On Sun, Aug 2, 2026 at 11:08 AM Tanushree Shah <tshah@linux.ibm.com> wrote:
>>
>> This RFC patch series introduces support for converting perf.data files
>> containing tracepoint events into trace.dat format, enabling seamless
>> visualization and analysis using KernelShark.
>>
>> ======================
>> Background and Motivation
>> ======================
>>
>> Currently, perf and trace-cmd operate as separate tracing ecosystems with
>> incompatible data formats. Users who collect tracepoint data with
>> 'perf record' cannot easily visualize it in KernelShark's graphical
>> timeline view or leverage trace-cmd's analysis capabilities.
>>
>> This creates workflow friction when users need to:
>>
>> - Visualize perf tracepoint data in KernelShark's interactive graphical
>>    timeline
>> - Share trace data between perf and trace-cmd workflows and toolchains
>> - Perform architecture-independent conversion and analysis of traces
>>
>> This conversion bridge eliminates these barriers by enabling seamless
>> data exchange between perf and trace-cmd ecosystems, allowing users to
>> choose the best tool for each analysis phase.
>>
>> ======================
>> Implementation Overview
>> ======================
>>
>> The series implements the trace.dat file format specification (version 7)
>> within perf's data conversion framework.
>>
>> **Patch 1/5: Core trace.dat Export Infrastructure**
>> Introduces util/trace-dat.c and util/trace-dat.h implementing:
>> - Per-CPU raw event buffer management (init, collect, free)
>> - Ftrace ring buffer page construction
>> - trace.dat section writers (strings, options, flyrecord sections)
>>
>> **Patch 2/5: Metadata Integration**
>> Extends util/trace-event-read.c to write trace.dat metadata during
>> perf.data
>> parsing:
>> - Initial format header (magic, version, endian, page size, compression)
>> - Section 16: HEADER INFO (header_page + header_event)
>> - Section 17: FTRACE EVENT FORMATS
>> - Section 18: EVENT FORMATS (per system/event format files)
>> - Section 19: KALLSYMS
>> - Section 21: CMDLINES
>> - Section 15: STRINGS (written last after all sections)
>>
>> **Patch 3/5: Conversion Backend**
>> Implements util/data-convert-trace.c with trace_convert__perf2dat()
>> function:
>> - Processes PERF_TYPE_TRACEPOINT samples via process_sample_event()
>> - Collects raw event data per-CPU using trace_dat__collect_cpu_event()
>> - Writes OPTIONS sections (CPUCOUNT, TRACECLOCK, metadata offsets)
>> - Writes FLYRECORD section with per-CPU ring buffer pages
>>
>> **Patch 4/5: User Interface**
>> Extends tools/perf/builtin-data.c with --to-trace-dat option:
>> - Adds command-line option for trace.dat output
>> - Mutually exclusive with --to-ctf and --to-json
>> - Calls trace_convert__perf2dat() to perform conversion
>>
>> **Patch 5/5: Shell Test**
>> Adds a shell test (tools/perf/tests/shell/) covering normal tracepoint
>> recordings, pipe mode and mixed tracepoint/non-tracepoint recording
>> conversions and --force flag behaviour.
>>
>> ======================
>> Current Implementation Details
>> ======================
>>
>> **trace.dat Format Version:**
>> The implementation currently targets trace.dat format version 7, which
>> is the stable version supported by current trace-cmd releases (v3.x).
>> This version is hardcoded to ensure compatibility with existing
>> trace-cmd and KernelShark installations. Future enhancements could add
>> version negotiation or support for newer format versions as they become
>> standardized.
>>
>> **Compression Strategy:**
>> Compression is explicitly disabled (set to NONE) in the generated
>> trace.dat files.
>> This design choice:
>> - Simplifies the initial implementation and testing
>> - Ensures maximum compatibility across trace-cmd versions
>> - Avoids external compression library dependencies
>>
>> Future work could add support for various compression algorithms (zlib,
>> zstd, lz4) with runtime selection via command-line options, significantly
>> reducing file sizes for large traces.
>>
>> ======================
>> Usage Example
>> ======================
>>
>> ```bash
>> *Record tracepoint events with perf*
>> perf record -e sched:sched_switch -e sched:sched_wakeup -a sleep 10
>>
>> *Convert to trace.dat format*
>> perf data convert --to-trace-dat=output.dat
>>
>> *Verify trace.dat structure*
>> trace-cmd dump --summary output.dat
>>
>> *Analyze with trace-cmd*
>> trace-cmd report output.dat
>>
>> *Visualize in KernelShark*
>> kernelshark output.dat
>> ```
>>
>> **Conversion Output:**
>> ```
>> [ perf data convert: Converted 'perf.data' into trace.dat format
>> 'output.dat' ]
>> [ perf data convert: Converted 2684 events ]
>> ```
>> **trace-cmd dump --summary Output:**
>> ```
>>   Tracing meta data in file output.dat:
>>          [Initial format]
>>                  7       [Version]
>>                  0       [Little endian]
>>                  8       [Bytes in a long]
>>                  65536   [Page size, bytes]
>>                  none    [Compression algorithm]
>>                          [Compression version]
>>          [buffer "", "local" clock, 65536 page size, 16 cpus, 1048576 bytes
>>      flyrecord data]
>>          [10 options]
>>          [Saved command lines, 0 bytes]
>>          [Kallsyms, 0 bytes]
>>          [Ftrace format, 0 events]
>>          [Header page, 206 bytes]
>>          [Header event, 205 bytes]
>>          [Events format, 1 systems]
>>          [9 sections]
>> ```
>> ======================
>> Testing and Verification
>> ======================
>>
>> The series has been extensively tested with:
>> - Various tracepoint events (sched, irq, syscalls, block I/O)
>> - Mixed recordings containing both tracepoint and non-tracepoint events
>>    only tracepoints converted)
>> - Verification with trace-cmd report and KernelShark visualization
>> - Memory leak testing with Valgrind (0 bytes leaked).
>> - Cross-architecture testing: v1 tested x86_64 and ppc64le. v2 adds
>>    s390 (big-endian) perf.data converted on both ppc64le
>>    (little-endian) and x86_64 (little-endian) hosts, in addition to
>>    same-arch x86_64 (LE->LE) and ppc64le (BE->BE) conversion.
>> - Pipe mode support has been tested end-to-end. (in v2)
>>
>> All generated trace.dat files successfully open in:
>> - trace-cmd report (v3.1+)
>> - KernelShark (v2.0+)
>>
>>
>> ======================
>> Next Steps
>> ======================
>>
>> We would highly appreciate reviews, comments, and feedback on:
>> - The overall architectural approach and integration points
>> - Compatibility considerations with trace-cmd ecosystem
>> - Performance characteristics for large-scale traces
>> - Additional use cases or workflow scenarios
>> - Future enhancement priorities
>>
>> ---
>> Changes in v2
>>
>> Addressing the Sashiko AI review findings on v1:
>>
>> Cross-arch correctness:
>>    - Introduce to_file_u16/u32/u64 helpers (wrapping tep_read_number())
>>      to write all multi-byte fields in the recorded machine's byte order;
>>      apply throughout metadata sections and flyrecord page/record headers
>>      (ts, commit, TIME_EXTEND, large-event data_len).
>>    - Fix flyrecord record header bit layout for big-endian files.
>>      The record header word bit layout differs by file endianness,
>>      matching kbuffer-parse.c type_len4host()/ts4host():
>>        LE: type_len in bits [4:0],   time_delta in bits [31:5]
>>        BE: type_len in bits [31:27], time_delta in bits [26:0]
>>
>> Pipe mode:
>>    - Add process_attr(), process_feature(), and process_tracing_data()
>>      callbacks required for pipe mode operation.
>>    - Defer CPU buffer initialisation until the first tracepoint sample,
>>      after process_feature()/process_tracing_data() have populated the
>>      session header. This ensures the recorded machine's CPU count is
>>      used rather than the host's - critical for cross-platform analysis.
>>
>> Format compliance:
>>    - Implement TIME_EXTEND records for timestamp deltas >27 bits to
>>      prevent silent truncation and maintain chronological ordering.
>>    - Fix large event encoding (>=29 words): use type_len=0 with a
>>      separate 32-bit length word, avoiding collision with reserved types
>>      (PADDING=29, TIME_EXTEND=30, TIME_STAMP=31).
>>    - Add bounds check rejecting records larger than a page payload before
>>      batching, preventing heap overflow in trace_dat__write_page().
>>    - Fix flyrecord section_size to exclude the 16-byte section header,
>>      matching trace.dat specification and trace-cmd behaviour.
>>
>> CLI behavior:
>>    - Fix --force flag: open with O_CREAT|O_EXCL when force is not set,
>>      failing with -EEXIST instead of silently overwriting existing files.
>>
>> Memory safety:
>>    - Fix realloc overwrite of cpu_events->events and page_records on
>>      failure: use temporary pointers, only commit on success.
>>    - Fix use-after-free/double-free in sequential page_records realloc
>>      failure: replace with malloc+memcpy+free pattern.
>>    - Fix section_size computed from before section header position.
>>    - Add NULL checks for get_tracing_file(), calloc() padding, and
>>      trace_dat_options_offset assignment on write failure.
>>    - Use goto out_free on record allocation failure to avoid leaking
>>      accumulated page_records entries.
>>    - Replace direct read() with do_read() in read_proc_kallsyms() to
>>      handle short reads correctly.
>>    - On fwrite failure, set trace_dat_write_failed and continue parsing
>>      so that perf.data processing completes normally.
>>
>> Testing (new in v2):
>>    - Add shell test covering conversion, trace-cmd dump validation,
>>      sched_switch event verification, and --force flag behaviour.
>>
>> Documentation (new in v2):
>>    - Add documentation for 'perf data convert --to-trace-dat', covering
>>      usage and supported options.
>>
>> v1: https://lore.kernel.org/linux-perf-users/20260608125951.90425-2-tshah@linux.ibm.com/
> 
> Thanks again Tanushree for pursuing this. I see on sashiko.dev that
> sashiko couldn't review the v2 patches because they failed to apply:
> https://sashiko.dev/#/log/baseline/53604/2
> The SHA of perf-tools-next that sashiko.dev was trying was
> d04ef71492fad7230d474efe33d05f4c0563d409, which corresponds to "perf
> machine: Check snprintf truncation for guest kallsyms path," added 4
> days ago:
> https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/commit/?h=perf-tools-next&id=d04ef71492fad7230d474efe33d05f4c0563d409
> Could you rebase the patches so that we can benefit from the Sashiko review?
> 
> Thanks,
> Ian
> 
>> Tanushree Shah (5):
>>    perf/trace-dat: Add trace.dat export infrastructure
>>    perf/trace-event: Write trace.dat metadata sections during parsing
>>    perf data-convert: Add perf.data to trace.dat conversion backend
>>    perf data: Add --to-trace-dat option for converting perf.data
>>      tracepoint events into trace.dat format
>>    perf test: Add test validating trace.dat generated by 'perf data
>>      convert --to-trace-dat'
>>
>>
>>   tools/perf/Documentation/perf-data.txt        |   7 +
>>   tools/perf/builtin-data.c                     |  40 +-
>>   ...rf_data_converter_tracepoints_trace_dat.sh | 169 ++++
>>   tools/perf/util/Build                         |   2 +
>>   tools/perf/util/data-convert-trace.c          | 240 +++++
>>   tools/perf/util/data-convert.h                |   4 +
>>   tools/perf/util/trace-dat.c                   | 876 ++++++++++++++++++
>>   tools/perf/util/trace-dat.h                   | 113 +++
>>   tools/perf/util/trace-event-read.c            | 307 +++++-
>>   9 files changed, 1749 insertions(+), 9 deletions(-)
>>   create mode 100755 tools/perf/tests/shell/test_perf_data_converter_tracepoints_trace_dat.sh
>>   create mode 100644 tools/perf/util/data-convert-trace.c
>>   create mode 100644 tools/perf/util/trace-dat.c
>>   create mode 100644 tools/perf/util/trace-dat.h
>>
>> --
>> 2.47.3
>>