addr2line performance

addr2line performance

[Tony Jones]

Hi.

https://bugzilla.kernel.org/show_bug.cgi?id=218996#c1 mentions
"there are proposals to move this logic to being from a library again. 
I think the addr2line processing attempted is valiant but ultimately 
using a library will be best" (this was when LLVM addr2line support 
was already present),

Obviously the test patch mentioned in the above bug doesn't address 
the underlying performance issue, only better classifying a timeout 
error.

We have a customer bug complaining about the performance of addr2line.
I'm seeing cases where it's taking over 20 seconds wall clock to 
resolve certain addresses which obviously causes the io_timer to trip 
but the performance of the success path isn't great either.

Obviously BFD isn't an option and (for various reasons) we can't use 
LLVM for this specific distro so I'm looking at the performance
of addr2line.

Can someone point me to the prior "using a library" discussion 
referenced in the above bug or whatever the current thinking is.  I 
couldn't find anything searching the archive.

Thanks in advance!

Tony

-- 
Tony Jones
SUSE Kernel Performance Team

Re: addr2line performance

[Ian Rogers]

On Wed, Oct 22, 2025 at 11:30 AM Tony Jones <tonyj@suse.de> wrote:
>
> Hi.
>
> https://bugzilla.kernel.org/show_bug.cgi?id=218996#c1 mentions
> "there are proposals to move this logic to being from a library again.
> I think the addr2line processing attempted is valiant but ultimately
> using a library will be best" (this was when LLVM addr2line support
> was already present),
>
> Obviously the test patch mentioned in the above bug doesn't address
> the underlying performance issue, only better classifying a timeout
> error.
>
> We have a customer bug complaining about the performance of addr2line.
> I'm seeing cases where it's taking over 20 seconds wall clock to
> resolve certain addresses which obviously causes the io_timer to trip
> but the performance of the success path isn't great either.
>
> Obviously BFD isn't an option and (for various reasons) we can't use
> LLVM for this specific distro so I'm looking at the performance
> of addr2line.
>
> Can someone point me to the prior "using a library" discussion
> referenced in the above bug or whatever the current thinking is.  I
> couldn't find anything searching the archive.

I'm not sure there is an explicit discussion rather than discussion on
individual patch sets. It is something that comes up when fixing the
text parsing of addr2line, something that is particularly problematic
because of inlining. It comes up wrt removing non-distributable stuff,
most recently in:
https://lore.kernel.org/lkml/20250823003216.733941-1-irogers@google.com/

Wrt the bug, the --addr2line option for `perf script` is merged:
https://lore.kernel.org/r/eadc3e36-029d-4848-9d69-272fe5a83a26@foxlink.cz

There has also been clean up into how the addr2line implementation is
picked, see:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/srcline.c?h=tmp.perf-tools-next#n117
which came from:
https://lore.kernel.org/lkml/20251005212212.2892175-8-irogers@google.com/

Wrt dependencies, I have been pushing these patches that use dlopen
for libLLVM mean installing libLLVM can be done separately from
building perf against it:
https://lore.kernel.org/lkml/20251007163835.3152881-1-irogers@google.com/

So if addr2line is the problem, perhaps use the command line option to
switch to the llvm version. If that's not fast enough, and you can't
distribute linked against libLLVM, some feedback on the dlopen libLLVM
changes would be appreciated!

Thanks,
Ian

> Thanks in advance!
>
> Tony
>
> --
> Tony Jones
> SUSE Kernel Performance Team

Re: addr2line performance

[Tony Jones]

On Wed, Oct 29, 2025 at 08:51:12AM -0700, Ian Rogers wrote:
> 
> I'm not sure there is an explicit discussion rather than discussion on
> individual patch sets. 

Thanks for the reply.  I thought the original comment in that BZ
was talking about plans to remove the current forking of an "addr2line"
process and communicating via pipe with some library implementation.
Of course what that library would be if not BFD/LLVM was my question.
Clearly this was a misread on my part.

I'd seen the patches related to dlopen'ing llvm, backporting this is 
a possibility since we don't want a rpm requires.

Cheers!

Tony

-- 
Tony Jones
SUSE Kernel Performance Team

Re: addr2line performance

[Ian Rogers]

On Fri, Oct 31, 2025 at 10:01 AM Tony Jones <tonyj@suse.de> wrote:
>
> On Wed, Oct 29, 2025 at 08:51:12AM -0700, Ian Rogers wrote:
> >
> > I'm not sure there is an explicit discussion rather than discussion on
> > individual patch sets.
>
> Thanks for the reply.  I thought the original comment in that BZ
> was talking about plans to remove the current forking of an "addr2line"
> process and communicating via pipe with some library implementation.
> Of course what that library would be if not BFD/LLVM was my question.
> Clearly this was a misread on my part.
>
> I'd seen the patches related to dlopen'ing llvm, backporting this is
> a possibility since we don't want a rpm required.

That's great to know and motivates the dlopen work! I can resend a
rebase of that series, would it be okay to add an Acked-by tag[1] to
the series saying it'd be useful for SuSE ?

Thanks,
Ian

[1] https://www.kernel.org/doc/Documentation/process/submitting-patches.rst

> Cheers!
>
> Tony
>
> --
> Tony Jones
> SUSE Kernel Performance Team

Re: addr2line performance

[Namhyung Kim]

Hello,

On Wed, Oct 22, 2025 at 11:30:21AM -0700, Tony Jones wrote:
> Hi.
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=218996#c1 mentions
> "there are proposals to move this logic to being from a library again. 
> I think the addr2line processing attempted is valiant but ultimately 
> using a library will be best" (this was when LLVM addr2line support 
> was already present),
> 
> Obviously the test patch mentioned in the above bug doesn't address 
> the underlying performance issue, only better classifying a timeout 
> error.
> 
> We have a customer bug complaining about the performance of addr2line.
> I'm seeing cases where it's taking over 20 seconds wall clock to 
> resolve certain addresses which obviously causes the io_timer to trip 
> but the performance of the success path isn't great either.
> 
> Obviously BFD isn't an option and (for various reasons) we can't use 
> LLVM for this specific distro so I'm looking at the performance
> of addr2line.
> 
> Can someone point me to the prior "using a library" discussion 
> referenced in the above bug or whatever the current thinking is.  I 
> couldn't find anything searching the archive.

I think we can just use libdw.  I tried it in the past but never
finished. :(

Thanks,
Namhyung

Re: addr2line performance

[Tony Jones]

On Fri, Oct 31, 2025 at 12:39:48PM -0700, Namhyung Kim wrote:

> I think we can just use libdw.  I tried it in the past but never
> finished. :(

This would be interesting.  

I asked our toolchain folks if they were aware of libraries 
(other than libllvm and libbfd) that could replace the addr2line+pipe 
(and were license compatible) but I've not heard back from them yet.

I'm happy to look into implementing this if you lack the time. LMK.

Tony

-- 
Tony Jones
SUSE Kernel Performance Team

Re: addr2line performance

[Namhyung Kim]

Hello,

On Fri, Oct 31, 2025 at 01:40:55PM -0700, Tony Jones wrote:
> On Fri, Oct 31, 2025 at 12:39:48PM -0700, Namhyung Kim wrote:
> 
> > I think we can just use libdw.  I tried it in the past but never
> > finished. :(
> 
> This would be interesting.  
> 
> I asked our toolchain folks if they were aware of libraries 
> (other than libllvm and libbfd) that could replace the addr2line+pipe 
> (and were license compatible) but I've not heard back from them yet.
> 
> I'm happy to look into implementing this if you lack the time. LMK.

Sure, it'd be great if you can work on it.  I think it's basically
dwarf_lineaddr() followed by dwarf_linesrc() and dwarf_lineno() but
you also need to check inlined functions.

And probably you want to cache dbginfo of recent binaries to speed up
addr2line processing for the same DSOs.

Thanks,
Namhyung