Re: [PATCH v4 1/5] perf: support specify vdso path in cmdline

[Adrian Hunter <adrian.hunter@intel.com>]

On 28/06/24 07:21, duchangbin wrote:
> On Thu, Jun 27, 2024 at 04:53:18PM -0700, Namhyung Kim wrote:
>> Hello guys,
>>
>> On Wed, Jun 26, 2024 at 01:32:42PM +0300, Adrian Hunter wrote:
>>> On 26/06/24 05:26, duchangbin wrote:
>>>> On Tue, Jun 25, 2024 at 04:20:49PM +0300, Adrian Hunter wrote:
>>>>> On 25/06/24 06:37, Changbin Du wrote:
>>>>>> The vdso dumped from process memory (in buildid-cache) lacks debugging
>>>>>> info. To annotate vdso symbols with source lines we need specify a
>>>>>> debugging version.
>>>>>>
>>>>>> For x86, we can find them from your local build as
>>>>>> arch/x86/entry/vdso/vdso{32,64}.so.dbg. Or they may reside in
>>>>>> /lib/modules/<version>/vdso/vdso{32,64}.so on Ubuntu. But notice that
>>>>>> the buildid has to match.
>>>>>>
>>>>>> $ sudo perf record -a
>>>>>> $ sudo perf report --objdump=llvm-objdump \
>>>>>>   --vdso arch/x86/entry/vdso/vdso64.so.dbg,arch/x86/entry/vdso/vdso32.so.dbg
>>>>>>
>>>>>> Samples: 17K of event 'cycles:P', 4000 Hz, Event count (approx.): 1760
>>>>>> __vdso_clock_gettime  /work/linux-host/arch/x86/entry/vdso/vdso64.so.d
>>>>>> Percent│       movq    -48(%rbp),%rsi
>>>>>>        │       testq   %rax,%rax
>>>>>>        │     ;               return vread_hvclock();
>>>>>>        │       movq    %rax,%rdx
>>>>>>        │     ;               if (unlikely(!vdso_cycles_ok(cycles)))
>>>>>>        │     ↑ js      eb
>>>>>>        │     ↑ jmp     74
>>>>>>        │     ;               ts->tv_sec = vdso_ts->sec;
>>>>>>   0.02 │147:   leaq    2(%rbx),%rax
>>>>>>        │       shlq    $4, %rax
>>>>>>        │       addq    %r10,%rax
>>>>>>        │     ;               while ((seq = READ_ONCE(vd->seq)) & 1) {
>>>>>>   9.38 │152:   movl    (%r10),%ecx
>>>>>>
>>>>>> When doing cross platform analysis, we also need specify the vdso path if
>>>>>> we are interested in its symbols.
>>>>>
>>>>> Would it be possible to add vdso and vdso debug to the build-id
>>>>> cache and ensure perf can find it there?
>>>>>
>>>>> Typically, getting dsos from another machine is handled via
>>>>> build-id cache e.g. what perf-archive does
>>>>>
>>>> Hmm. I agree this is better alternative approach for cross-machine analysis.
>>>> When collecting vdsos to buildid cache, I think we can use the local searched
>>>> objects (with debug symbols) instead if its build-id matches vdsos from process
>>>> dumping (the real code ran).
>>>>
>>>> Currently I just follow what perf does for vmlinux so to reuse most of existing
>>>> code. Maybe vmlinux is too big to add to buildid-cahce?
>>>>
>>>> Can we keep our current strategy for now? I'll think about above options when
>>>> I have more time.
>>>>
>>>
>>> I tried adding vdso via perf buildid-cache.  It doesn't work only
>>> because the lookup expects the basename to be "vdso" but it is
>>> "elf".
>>>
>>> Adding a link from "vdso" to "elf" made it work e.g.
>>>
>>> $ cat gettimeofday-test.c
>>> #include <stdio.h>
>>> #include <sys/time.h>
>>>
>>> int main()
>>> {
>>>         struct timeval tv;
>>>         int ret;
>>>
>>>         ret = gettimeofday(&tv, NULL);
>>>         if (ret == -1) {
>>>                 fprintf(stderr, "gettimeofday failed\n");
>>>                 return 1;
>>>         }
>>>
>>>         printf("%lu.%lu\n", (unsigned long)tv.tv_sec, (unsigned long)tv.tv_usec);
>>>
>>>         return 0;
>>> $ perf record -e intel_pt//u ./gettimeofday-test
>>> 1719397042.892837
>>> [ perf record: Woken up 1 times to write data ]
>>> [ perf record: Captured and wrote 0.026 MB perf.data ]
>>> $ perf script --itrace=e
>>> $ perf buildid-cache --remove /lib/modules/6.5.0-41-generic/vdso/vdso64.so
>>> $ perf script --itrace=e
>>> Warning:
>>> 2 instruction trace errors
>>>  instruction trace error type 1 time 525345.386424204 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e8e00 code 5: Failed to get instruction
>>>  instruction trace error type 1 time 525345.386424829 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e884d code 5: Failed to get instruction
>>> $ perf buildid-cache --add /lib/modules/6.5.0-41-generic/vdso/vdso64.so
>>> $ perf script --itrace=e
>>> Warning:
>>> 2 instruction trace errors
>>>  instruction trace error type 1 time 525345.386424204 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e8e00 code 5: Failed to get instruction
>>>  instruction trace error type 1 time 525345.386424829 cpu 4 pid 198976 tid 198976 ip 0x7ffddb0e884d code 5: Failed to get instruction
>>> $ cd ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8
>>> ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ ln -s elf vdso
>>> ~/.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ ls -l
>>> total 36
>>> -rw-r--r-- 1 ahunter ahunter 33272 Jun 26 13:17 elf
>>> -rw-r----- 1 ahunter ahunter     0 Jun 26 13:17 probes
>>> lrwxrwxrwx 1 ahunter ahunter     3 Jun 26 13:18 vdso -> elf
>>> /.debug/.build-id/c3/530aed66e71bfd10af66039f58cc7c4d2eaba8$ cd
>>> $ perf script --itrace=e
>>> $ 
>>>
>>> So maybe a change could be made to build_id_cache__add() to add
>>> the extra link if the file name matches vdso
>>  
>> Thanks for doing this!  I noticed buildid_cache__basename() will handle
>> the name properly once it realizes the file is a vdso.  Maybe we can
>> check the filepath with some patterns, or simply add an command line
>> option to say it's a vdso.
>>
>> Thanks,
>> Namhyung
>>
> I added some tricks for vdso in build_id_cache__add(). It replace vdso object
> with debugging one if found. Is this okay?

perf has support for storing debug files in the build-id
cache using the base name "debug" instead of "elf", so really
it would be better to make that work

> 
> +static char *build_id_cache__find_debug_vdso(const char *sbuild_id)
> +{
> +       char sbuild_id_tmp[SBUILD_ID_SIZE];
> +       struct build_id bid;
> +       int i, ret = 0;
> +
> +       printf("Looking at the vdso_path (%d entries long)\n",
> +                vdso_paths.nr_entries + 1);
> +
> +       for (i = 0; i < vdso_paths.nr_entries; ++i) {
> +               ret = filename__read_build_id(vdso_paths.paths[i], &bid);
> +               if (ret < 0)
> +                       continue;
> +
> +               build_id__sprintf(&bid, sbuild_id_tmp);
> +               if (!strcmp(sbuild_id, sbuild_id_tmp)) {
> +                       printf("Found debugging vdso %s\n", vdso_paths.paths[i]);
> +                       return strdup(vdso_paths.paths[i]);
> +               }
> +       }
> +
> +       return NULL;
> +}
> +
>  int
>  build_id_cache__add(const char *sbuild_id, const char *name, const char *realname,
>                     struct nsinfo *nsi, bool is_kallsyms, bool is_vdso,
>                     const char *proper_name, const char *root_dir)
>  {
>         const size_t size = PATH_MAX;
> -       char *filename = NULL, *dir_name = NULL, *linkname = zalloc(size), *tmp;
> +       char *filename = NULL, *dir_name = NULL, *vdso_name = NULL, *linkname = zalloc(size), *tmp;
>         char *debugfile = NULL;
>         int err = -1;
> 
> +       /* replace vdso object with debugging one if found */
> +       if (is_vdso) {
> +               vdso_name = build_id_cache__find_debug_vdso(sbuild_id);
> +               if (vdso_name)
> +                       name = realname = vdso_name;
> +       }
> +
>         if (!proper_name)
>                 proper_name = name;
> 
>>
>