Re: [PATCH v4 1/5] perf: support specify vdso path in cmdline

[Adrian Hunter <adrian.hunter@intel.com>]

On 29/06/24 05:32, duchangbin wrote:
> On Fri, Jun 28, 2024 at 08:27:55PM +0300, Adrian Hunter wrote:
>> On 28/06/24 07:21, duchangbin wrote:
>>> On Thu, Jun 27, 2024 at 04:53:18PM -0700, Namhyung Kim wrote:
>>>> Hello guys,
>>>>
>>>> On Wed, Jun 26, 2024 at 01:32:42PM +0300, Adrian Hunter wrote:
>>>>> On 26/06/24 05:26, duchangbin wrote:
>>>>>> On Tue, Jun 25, 2024 at 04:20:49PM +0300, Adrian Hunter wrote:
>>>>>>> On 25/06/24 06:37, Changbin Du wrote:
>>>>>>>> The vdso dumped from process memory (in buildid-cache) lacks debugging
>>>>>>>> info. To annotate vdso symbols with source lines we need specify a
>>>>>>>> debugging version.
>>>>>>>>
>>>>>>>> For x86, we can find them from your local build as
>>>>>>>> arch/x86/entry/vdso/vdso{32,64}.so.dbg. Or they may reside in
>>>>>>>> /lib/modules/<version>/vdso/vdso{32,64}.so on Ubuntu. But notice that
>>>>>>>> the buildid has to match.
>>>>>>>>
>>>>>>>> $ sudo perf record -a
>>>>>>>> $ sudo perf report --objdump=llvm-objdump \
>>>>>>>>   --vdso arch/x86/entry/vdso/vdso64.so.dbg,arch/x86/entry/vdso/vdso32.so.dbg
>>>>>>>>
>>>>>>>> Samples: 17K of event 'cycles:P', 4000 Hz, Event count (approx.): 1760
>>>>>>>> __vdso_clock_gettime  /work/linux-host/arch/x86/entry/vdso/vdso64.so.d
>>>>>>>> Percent│       movq    -48(%rbp),%rsi
>>>>>>>>        │       testq   %rax,%rax
>>>>>>>>        │     ;               return vread_hvclock();
>>>>>>>>        │       movq    %rax,%rdx
>>>>>>>>        │     ;               if (unlikely(!vdso_cycles_ok(cycles)))
>>>>>>>>        │     ↑ js      eb
>>>>>>>>        │     ↑ jmp     74
>>>>>>>>        │     ;               ts->tv_sec = vdso_ts->sec;
>>>>>>>>   0.02 │147:   leaq    2(%rbx),%rax
>>>>>>>>        │       shlq    $4, %rax
>>>>>>>>        │       addq    %r10,%rax
>>>>>>>>        │     ;               while ((seq = READ_ONCE(vd->seq)) & 1) {
>>>>>>>>   9.38 │152:   movl    (%r10),%ecx
>>>>>>>>
>>>>>>>> When doing cross platform analysis, we also need specify the vdso path if
>>>>>>>> we are interested in its symbols.
>>>>>>>
>>>>>>> Would it be possible to add vdso and vdso debug to the build-id
>>>>>>> cache and ensure perf can find it there?
>>>>>>>
>>>>>>> Typically, getting dsos from another machine is handled via
>>>>>>> build-id cache e.g. what perf-archive does
>>>>>>>
>>>>>> Hmm. I agree this is better alternative approach for cross-machine analysis.
>>>>>> When collecting vdsos to buildid cache, I think we can use the local searched
>>>>>> objects (with debug symbols) instead if its build-id matches vdsos from process
>>>>>> dumping (the real code ran).
>>>>>>
>>>>>> Currently I just follow what perf does for vmlinux so to reuse most of existing
>>>>>> code. Maybe vmlinux is too big to add to buildid-cahce?
>>>>>>
>>>>>> Can we keep our current strategy for now? I'll think about above options when
>>>>>> I have more time.
>>>>>>
>>>>>
>>>>> I tried adding vdso via perf buildid-cache.  It doesn't work only
>>>>> because the lookup expects the basename to be "vdso" but it is
>>>>> "elf".
>>>>>
>>>>> Adding a link from "vdso" to "elf" made it work e.g.
>>>>>
>>>>> $ cat gettimeofday-test.c
>>>>> #include <stdio.h>
>>>>> #include <sys/time.h>
>>>>>
>>>>> int main()
>>>>> {
>>>>>         struct timeval tv;
>>>>>         int ret;
>>>>>
>>>>>         ret = gettimeofday(&tv, NULL);
>>>>>         if (ret == -1) {
>>>>>                 fprintf(stderr, "gettimeofday failed\n");
>>>>>                 return 1;
>>>>>         }
>>>>>
>>>>>         printf("%lu.%lu\n", (unsigned long)tv.tv_sec, (unsigned long)tv.tv_usec);
>>>>>
>>>>>         return 0;
>>>>> $ perf record -e intel_pt//u ./gettimeofday-test
>>>>> 1719397042.892837
>>>>> [ perf record: Woken up 1 times to write data ]
>>>>> [ perf record: Captured and wrote 0.026 MB perf.data ]
>>>>> $ perf script --itrace=e
>>>>> $ perf buildid-cache --remove /lib/modules/6.5.0-41-generic/vdso/vdso64.so
>>>>> $ perf script --itrace=e
>>>>> Warning:
>>>>> 2 instruction trace errors
>>>>>  instruction trace error type 1 time 525345.386424204 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e8e00 code 5: Failed to get instruction
>>>>>  instruction trace error type 1 time 525345.386424829 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e884d code 5: Failed to get instruction
>>>>> $ perf buildid-cache --add /lib/modules/6.5.0-41-generic/vdso/vdso64.so
>>>>> $ perf script --itrace=e
>>>>> Warning:
>>>>> 2 instruction trace errors
>>>>>  instruction trace error type 1 time 525345.386424204 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e8e00 code 5: Failed to get instruction
>>>>>  instruction trace error type 1 time 525345.386424829 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e884d code 5: Failed to get instruction
>>>>> $ cd ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8
>>>>> ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ ln -s elf vdso
>>>>> ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ ls -l
>>>>> total 36
>>>>> -rw-r--r-- 1 ahunter ahunter 33272 Jun 26 13:17 elf
>>>>> -rw-r----- 1 ahunter ahunter     0 Jun 26 13:17 probes
>>>>> lrwxrwxrwx 1 ahunter ahunter     3 Jun 26 13:18 vdso -> elf
>>>>> /.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ cd
>>>>> $ perf script --itrace=e
>>>>> $ 
>>>>>
>>>>> So maybe a change could be made to build_id_cache__add() to add
>>>>> the extra link if the file name matches vdso
>>>>  
>>>> Thanks for doing this!  I noticed buildid_cache__basename() will handle
>>>> the name properly once it realizes the file is a vdso.  Maybe we can
>>>> check the filepath with some patterns, or simply add an command line
>>>> option to say it's a vdso.
>>>>
>>>> Thanks,
>>>> Namhyung
>>>>
>>> I added some tricks for vdso in build_id_cache__add(). It replace vdso object
>>> with debugging one if found. Is this okay?
>>
>> perf has support for storing debug files in the build-id
>> cache using the base name "debug" instead of "elf", so really
>> it would be better to make that work
>>
> My thoughts are:
>   1. add debugging vdso file as "debug" in buildid cache.
>   2. add a new perf config item 'annotate.prefer_debug_file' (default false) to
>      annotate with debugging object files.

Is that needed?  Isn't the debug information read from
the "debug" file?  If not, does that need fixing?