trace-cmd No such device Errors

trace-cmd No such device Errors

[Vishal Chourasia]

When running trace-cmd record, I encounter multiple "No such device"
errors due to the splice syscall failing with ENODEV for
/sys/kernel/tracing/per_cpu/cpuX/trace_pipe_raw files corresponding to
CPUs that are possible but not present.

Environment:
- Operating System: Red Hat Enterprise Linux 9.5 (Plow)
- Kernel: Linux 6.12.0-55.9.1.el10_0.ppc64le
- Architecture: ppc64le
- Command Executed: trace-cmd record -o /dev/null -v -e sched_switch -- sleep 1
- System CPU Configuration:
  - /sys/devices/system/cpu/possible: 0-223
  - /sys/devices/system/cpu/present: 0-63
  - /sys/kernel/tracing/per_cpu/: Contains directories cpu0 to cpu223

Steps to Reproduce: 
1. On a system with a mismatch between possible and present CPUs (e.g.,
   possible: 0-223, present: 0-63). 
2. Ensure /sys/kernel/tracing/per_cpu/ contains directories for all possible 
   CPUs (e.g., cpu0 to cpu223). 
3. Run the command:
strace --decode-fds -kk -o strace/trace-cmd -ff -- trace-cmd record -o /dev/null -v -e sched_switch -- sleep 1
4. Observe the output and strace logs.

Expected Behavior: trace-cmd should only attempt to access
trace_pipe_raw files for CPUs that are present (e.g., cpu0 to cpu63),
completing the recording without errors.

Actual Behavior:
trace-cmd fails with multiple errors:
trace-cmd: No such device
recorder error in splice input

The strace output shows splice syscalls failing with ENODEV for
trace_pipe_raw files of non-present CPUs 
(e.g., /sys/kernel/tracing/per_cpu/cpu223/trace_pipe_raw). 

Example:
trace-cmd.47140:splice(5</sys/kernel/tracing/per_cpu/cpu223/trace_pipe_raw>, NULL, 7<pipe:[96498]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)


Analysis:
- trace-cmd iterates over all directories in
  /sys/kernel/tracing/per_cpu/ (which includes cpu0 to cpu223 for all 
  possible CPUs).
- It attempts to invoke splice on trace_pipe_raw for each CPU, including
  those not present (e.g., cpu64 to cpu223).
- The splice syscall fails with ENODEV for non-present CPUs.
- Stack trace (from strace -kk):
splice(5</sys/kernel/tracing/per_cpu/cpu223/trace_pipe_raw>, ...)
  > /usr/lib64/glibc-hwcaps/power10/libc.so.6(splice+0x74)
  > /usr/bin/trace-cmd(splice_data+0x3b)
  > /usr/bin/trace-cmd(tracecmd_start_recording+0x93)
  > /usr/bin/trace-cmd(create_recorder+0x14f)
  > /usr/bin/trace-cmd(start_threads+0x81b)
  > /usr/bin/trace-cmd(record_trace.isra.0+0xf5f)
  > /usr/bin/trace-cmd(trace_record+0x47)
  > /usr/bin/trace-cmd(main+0xff)

# strace --decode-fds -kk -o strace/trace-cmd -ff -- trace-cmd record -o /dev/null -v -e sched_switch -- sleep 1
trace-cmd: No such device
    trace-cmd: No such device
recorder error in splice input  recorder error in splice input
recorder error in splice inputtrace-cmd: No such device
recorder error in splice input

trace-cmd: No such device
<snipped>
recorder error in splice input
trace-cmd: No such device
recorder error in splice input
recorder error in splice inputtrace-cmd: No such device

  recorder error in splice inputtrace-cmd: No such device

  recorder error in splice input
trace-cmd: No such device
  trace-cmd: No such device
recorder error in splice input
  recorder error in splice inputtrace-cmd: No such device

  recorder error in splice input
trace-cmd: No such device
  recorder error in splice inputtrace-cmd: No such device

recorder error in splice inputtrace-cmd: No such device
<snipped>
recorder error in splice input
recorder error in splice inputtrace-cmd: No such device

  recorder error in splice inputtrace-cmd: No such device

  recorder error in splice inputtrace-cmd: No such device

  recorder error in splice input
CPU0 data recorded at offset=0x10000
    0 bytes in size
CPU1 data recorded at offset=0x10000
    0 bytes in size
<snipped>
CPU10 data recorded at offset=0x10000
    0 bytes in size
CPU11 data recorded at offset=0x10000
<snipped>
    0 bytes in size
CPU63 data recorded at offset=0x10000
    0 bytes in size
CPU64 data recorded at offset=0x10000
    0 bytes in size
<snipped?
CPU213 data recorded at offset=0x10000
    0 bytes in size
<snipped>
CPU222 data recorded at offset=0x10000
    0 bytes in size
CPU223 data recorded at offset=0x10000
    0 bytes in size

# grep -i "such device" * | grep splice
trace-cmd.46981:splice(5</sys/kernel/tracing/per_cpu/cpu64/trace_pipe_raw>, NULL, 7<pipe:[86367]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
trace-cmd.46982:splice(5</sys/kernel/tracing/per_cpu/cpu65/trace_pipe_raw>, NULL, 7<pipe:[82351]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
trace-cmd.46983:splice(5</sys/kernel/tracing/per_cpu/cpu66/trace_pipe_raw>, NULL, 7<pipe:[81717]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
trace-cmd.46984:splice(5</sys/kernel/tracing/per_cpu/cpu67/trace_pipe_raw>, NULL, 7<pipe:[1932]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
trace-cmd.46985:splice(5</sys/kernel/tracing/per_cpu/cpu68/trace_pipe_raw>, NULL, 7<pipe:[92520]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
trace-cmd.46986:splice(5</sys/kernel/tracing/per_cpu/cpu69/trace_pipe_raw>, NULL, 7<pipe:[93494]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
<snipped>
trace-cmd.47136:splice(5</sys/kernel/tracing/per_cpu/cpu219/trace_pipe_raw>, NULL, 7<pipe:[19742]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
trace-cmd.47137:splice(5</sys/kernel/tracing/per_cpu/cpu220/trace_pipe_raw>, NULL, 7<pipe:[29868]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
trace-cmd.47138:splice(5</sys/kernel/tracing/per_cpu/cpu221/trace_pipe_raw>, NULL, 7<pipe:[42069]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
trace-cmd.47139:splice(5</sys/kernel/tracing/per_cpu/cpu222/trace_pipe_raw>, NULL, 7<pipe:[81737]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)
trace-cmd.47140:splice(5</sys/kernel/tracing/per_cpu/cpu223/trace_pipe_raw>, NULL, 7<pipe:[96498]>, NULL, 65536, SPLICE_F_MOVE) = -1 ENODEV (No such device)

Re: trace-cmd No such device Errors

[Steven Rostedt]

On Thu, 15 May 2025 02:21:13 +0530
Vishal Chourasia <vishalc@linux.ibm.com> wrote:

> When running trace-cmd record, I encounter multiple "No such device"
> errors due to the splice syscall failing with ENODEV for
> /sys/kernel/tracing/per_cpu/cpuX/trace_pipe_raw files corresponding to
> CPUs that are possible but not present.
> 
> Environment:
> - Operating System: Red Hat Enterprise Linux 9.5 (Plow)
> - Kernel: Linux 6.12.0-55.9.1.el10_0.ppc64le
> - Architecture: ppc64le
> - Command Executed: trace-cmd record -o /dev/null -v -e sched_switch -- sleep 1
> - System CPU Configuration:
>   - /sys/devices/system/cpu/possible: 0-223
>   - /sys/devices/system/cpu/present: 0-63
>   - /sys/kernel/tracing/per_cpu/: Contains directories cpu0 to cpu223

Thanks for the report. Could you also submit a bug report here:

  https://bugzilla.kernel.org/buglist.cgi?component=Trace-cmd%2FKernelshark&list_id=1152443&product=Tools&resolution=---


> 
> Steps to Reproduce: 
> 1. On a system with a mismatch between possible and present CPUs (e.g.,
>    possible: 0-223, present: 0-63). 
> 2. Ensure /sys/kernel/tracing/per_cpu/ contains directories for all possible 
>    CPUs (e.g., cpu0 to cpu223). 
> 3. Run the command:
> strace --decode-fds -kk -o strace/trace-cmd -ff -- trace-cmd record -o /dev/null -v -e sched_switch -- sleep 1
> 4. Observe the output and strace logs.
> 
> Expected Behavior: trace-cmd should only attempt to access
> trace_pipe_raw files for CPUs that are present (e.g., cpu0 to cpu63),
> completing the recording without errors.

Can you show me what you see in /sys/kernel/tracing/tracing_cpumask

Thanks,

-- Steve

Re: trace-cmd No such device Errors

[Vishal Chourasia]

On Mon, May 19, 2025 at 10:39:14AM -0400, Steven Rostedt wrote:
> On Thu, 15 May 2025 02:21:13 +0530
> Vishal Chourasia <vishalc@linux.ibm.com> wrote:
> 
> > When running trace-cmd record, I encounter multiple "No such device"
> > errors due to the splice syscall failing with ENODEV for
> > /sys/kernel/tracing/per_cpu/cpuX/trace_pipe_raw files corresponding to
> > CPUs that are possible but not present.
> > 
> > Environment:
> > - Operating System: Red Hat Enterprise Linux 9.5 (Plow)
> > - Kernel: Linux 6.12.0-55.9.1.el10_0.ppc64le
> > - Architecture: ppc64le
> > - Command Executed: trace-cmd record -o /dev/null -v -e sched_switch -- sleep 1
> > - System CPU Configuration:
> >   - /sys/devices/system/cpu/possible: 0-223
> >   - /sys/devices/system/cpu/present: 0-63
> >   - /sys/kernel/tracing/per_cpu/: Contains directories cpu0 to cpu223
> 
> Thanks for the report. Could you also submit a bug report here:
> 
>   https://bugzilla.kernel.org/buglist.cgi?component=Trace-cmd%2FKernelshark&list_id=1152443&product=Tools&resolution=---
https://bugzilla.kernel.org/show_bug.cgi?id=220139
> 
> 
> > 
> > Steps to Reproduce: 
> > 1. On a system with a mismatch between possible and present CPUs (e.g.,
> >    possible: 0-223, present: 0-63). 
> > 2. Ensure /sys/kernel/tracing/per_cpu/ contains directories for all possible 
> >    CPUs (e.g., cpu0 to cpu223). 
> > 3. Run the command:
> > strace --decode-fds -kk -o strace/trace-cmd -ff -- trace-cmd record -o /dev/null -v -e sched_switch -- sleep 1
> > 4. Observe the output and strace logs.
> > 
> > Expected Behavior: trace-cmd should only attempt to access
> > trace_pipe_raw files for CPUs that are present (e.g., cpu0 to cpu63),
> > completing the recording without errors.
> 
> Can you show me what you see in /sys/kernel/tracing/tracing_cpumask
$ sudo cat /sys/kernel/tracing/tracing_cpumask
ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff,ffffffff
> 
> Thanks,
> 
> -- Steve
Thanks,

Vishal