Symbol resolution when text in hugepages

Symbol resolution when text in hugepages

[Jayaramappa, Srilakshmi]

Hi all, 

We have an application that has its text section backed by hugepages. In our new build environment the text section of the binary is split up into read and read/execute sections with offsets into the segment. This is breaking the perf symbol resolution. Previously, everything was in one segment and there was no offset, so perf functioned fine. I know we can dump the symbols from readelf into /tmp/perf-PID.map file. I was wondering if anyone has a cleaner solution? I'd appreciate any pointers you have. 

Thanks
Sri

Re: Symbol resolution when text in hugepages

[Ian Rogers]

On Wed, Sep 13, 2023 at 5:21 PM Jayaramappa, Srilakshmi
<sjayaram@akamai.com> wrote:
>
> Hi all,
>
> We have an application that has its text section backed by hugepages. In our new build environment the text section of the binary is split up into read and read/execute sections with offsets into the segment. This is breaking the perf symbol resolution. Previously, everything was in one segment and there was no offset, so perf functioned fine. I know we can dump the symbols from readelf into /tmp/perf-PID.map file. I was wondering if anyone has a cleaner solution? I'd appreciate any pointers you have.

Hi Sri,

Since Linux 5.4 you can use MADV_HUGEPAGE with madvise for file backed
pages. It can still be profitable to use other sources of huge pages
so that you don't wait for the hugepage to be formed, in that case if
there is a file/memfd_create you can embed the information in the
filename. For anonymous pages you can merge them when scanning the
executable in perf_event__synthesize_mmap_events, but it can be
problematic if the program header is on an anonymous page.

Thanks,
Ian

> Thanks
> Sri

Re: Symbol resolution when text in hugepages

[Jayaramappa, Srilakshmi]

From: Ian Rogers <irogers@google.com>
Sent: Thursday, September 14, 2023 4:26 PM
To: Jayaramappa, Srilakshmi
Cc: linux-perf-users@vger.kernel.org
Subject: Re: Symbol resolution when text in hugepages
    
On Wed, Sep 13, 2023 at 5:21 PM Jayaramappa, Srilakshmi
<sjayaram@akamai.com> wrote:
>
> Hi all,
>
> We have an application that has its text section backed by hugepages. In our new build environment the text section of the binary is split up into read and read/execute sections with offsets into the segment. This is breaking the perf symbol resolution. Previously,  everything was in one segment and there was no offset, so perf functioned fine. I know we can dump the symbols from readelf into /tmp/perf-PID.map file. I was wondering if anyone has a cleaner solution? I'd appreciate any pointers you have.

Hi Sri,

Since Linux 5.4 you can use MADV_HUGEPAGE with madvise for file backed
pages. It can still be profitable to use other sources of huge pages
so that you don't wait for the hugepage to be formed, in that case if
there is a file/memfd_create you can embed the information in the
filename. For anonymous pages you can merge them when scanning the
executable in perf_event__synthesize_mmap_events, but it can be
problematic if the program header is on an anonymous page.

Thanks,
Ian

> Thanks
> Sri



[resending since my message bounced from the mailing list. Apologies for the spam, Ian]

Thank you for the response, Ian, appreciate it. The pointer to perf_event__synthesize_mmap_events  is super useful, we will need to do something like that to find if the address range is backed by hugepages and if so grab the offset from the executable. 


For more context - our application first loads and then we remap its text segment to hugepages via the libhugetlbfs. When that happens the memory map doesn't need any offset. But perf is grabbing the maps from the /proc/pid/maps and trying  to lookup the address in the binary, so it looks at the wrong place without the offset into the file. 


Thanks!
Sri      

Re: Symbol resolution when text in hugepages

[Ian Rogers]

On Tue, Sep 19, 2023 at 1:51 PM Jayaramappa, Srilakshmi
<sjayaram@akamai.com> wrote:
>
> From: Ian Rogers <irogers@google.com>
> Sent: Thursday, September 14, 2023 4:26 PM
> To: Jayaramappa, Srilakshmi
> Cc: linux-perf-users@vger.kernel.org
> Subject: Re: Symbol resolution when text in hugepages
>
> On Wed, Sep 13, 2023 at 5:21 PM Jayaramappa, Srilakshmi
> <sjayaram@akamai.com> wrote:
> >
> > Hi all,
> >
> > We have an application that has its text section backed by hugepages. In our new build environment the text section of the binary is split up into read and read/execute sections with offsets into the segment. This is breaking the perf symbol resolution. Previously,  everything was in one segment and there was no offset, so perf functioned fine. I know we can dump the symbols from readelf into /tmp/perf-PID.map file. I was wondering if anyone has a cleaner solution? I'd appreciate any pointers you have.
>
> Hi Sri,
>
> Since Linux 5.4 you can use MADV_HUGEPAGE with madvise for file backed
> pages. It can still be profitable to use other sources of huge pages
> so that you don't wait for the hugepage to be formed, in that case if
> there is a file/memfd_create you can embed the information in the
> filename. For anonymous pages you can merge them when scanning the
> executable in perf_event__synthesize_mmap_events, but it can be
> problematic if the program header is on an anonymous page.
>
> Thanks,
> Ian
>
> > Thanks
> > Sri
>
>
>
> [resending since my message bounced from the mailing list. Apologies for the spam, Ian]
>
> Thank you for the response, Ian, appreciate it. The pointer to perf_event__synthesize_mmap_events  is super useful, we will need to do something like that to find if the address range is backed by hugepages and if so grab the offset from the executable.
>
>
> For more context - our application first loads and then we remap its text segment to hugepages via the libhugetlbfs. When that happens the memory map doesn't need any offset. But perf is grabbing the maps from the /proc/pid/maps and trying  to lookup the address in the binary, so it looks at the wrong place without the offset into the file.

Hi Sri,

I think you'd be much better served using MADV_COLLAPSE or
MADV_HUGEPAGE, not least because hugetlbfs requires preallocation of
huge pages. Using these shouldn't alter perf's symbolization.

Thanks,
Ian

> Thanks!
> Sri