Re: Question about the ipi_raise filter usage and output

Valentin Schneider
2024-02-06T12:01:12Z

On 06/02/24 16:42, richard clark wrote:
> On Tue, Feb 6, 2024 at 12:05 AM Valentin Schneider <vschneid@redhat.com> wrote:
>>
>> The CPUS{} thingie only works with an event field that is either declared as a
>> cpumask (__cpumask) or a scalar. That's not the case for ipi_raise, the
>> target_cpus event field is saved as a "raw" bitmask.
>>
>> There /should/ have been a warning about the event filter though, but I
>> think it's not happening because I'm allowing more than just FILTER_CPUMASK
>> in parse_pred() to make it work for scalars. I'll go poke around some more.
>>
>> Generally for this sort of IPI investigation I'd recommend using the newer
>> trace_ipi_send_cpu() and trace_ipi_send_cpumask() (for which CPUS{}
>> filtering works).
>> If it's only the function call interrupts you're interesting in, have a
>> look at trace_csd_queue_cpu().
>
> This should be supported by newer version kernels like v6.5, but I am
> using v6.1 and this trace event has not been supported yet... so ipi
> is more suitable for me. ipi_entry and ipi_exit is ok, but seems the
> filter doesn't support a specific cpu, maybe we need to add this?
>>

You should have access to the generic fields which include the CPU from
which the event happens. Any of "CPU", "cpu" or "common_cpu" would match
this.

So if you're on a recent enough kernel (v6.6 or above AFAICT), you should
be able to do something like so:

trace-cmd record -e 'ipi_raise' -f 'CPU & CPUS{7-42}' ./foo.sh

If you just want to match a single CPU, or are on an older kernel, this
should work as well:

trace-cmd record -e 'ipi_raise' -f 'CPU == 42' ./foo.sh

For example on a QEMU x86 environment:

# trace-cmd record -e 'call_function*' -f 'CPU & CPUS{3}' hackbench
Running in process mode with 10 groups using 40 file descriptors each (== 400 tasks)
Each sender will pass 100 messages of 100 bytes
Time: 0.396
CPU0 data recorded at offset=0x738000
0 bytes in size
CPU1 data recorded at offset=0x738000
0 bytes in size
CPU2 data recorded at offset=0x738000
0 bytes in size
CPU3 data recorded at offset=0x738000
4096 bytes in size

# trace-cmd report
CPU 0 is empty
CPU 1 is empty
CPU 2 is empty
cpus=4
<idle>-0 [003] 29.704387: call_function_single_entry: vector=251
<idle>-0 [003] 29.704388: call_function_single_exit: vector=251
<idle>-0 [003] 29.705950: call_function_single_entry: vector=251
<idle>-0 [003] 29.705951: call_function_single_exit: vector=251
<idle>-0 [003] 29.706462: call_function_single_entry: vector=251
<idle>-0 [003] 29.706463: call_function_single_exit: vector=251
hackbench-962 [003] 29.706501: call_function_single_entry: vector=251
hackbench-962 [003] 29.706502: call_function_single_exit: vector=251
hackbench-955 [003] 29.706521: call_function_single_entry: vector=251
hackbench-955 [003] 29.706522: call_function_single_exit: vector=251
<idle>-0 [003] 30.101812: call_function_single_entry: vector=251
<idle>-0 [003] 30.101814: call_function_single_exit: vector=251
<idle>-0 [003] 30.101897: call_function_single_entry: vector=251
<idle>-0 [003] 30.101898: call_function_single_exit: vector=251
<idle>-0 [003] 30.101985: call_function_single_entry: vector=251
<idle>-0 [003] 30.101986: call_function_single_exit: vector=251
<idle>-0 [003] 30.102072: call_function_single_entry: vector=251
<idle>-0 [003] 30.102072: call_function_single_exit: vector=251
<idle>-0 [003] 30.102161: call_function_single_entry: vector=251
<idle>-0 [003] 30.102161: call_function_single_exit: vector=251
<idle>-0 [003] 30.102250: call_function_single_entry: vector=251
<idle>-0 [003] 30.102251: call_function_single_exit: vector=251
<idle>-0 [003] 30.102372: call_function_single_entry: vector=251
<idle>-0 [003] 30.102372: call_function_single_exit: vector=251

CPU 0 is empty
CPU 1 is empty
CPU 2 is empty
cpus=4
<idle>-0 [003] 1067.718304: call_function_single_entry: vector=251
<idle>-0 [003] 1067.718309: call_function_single_exit: vector=251

and that behaves the same as

trace-cmd record -e 'call_function*' -f 'CPU == 3' hackbench