Fwd: Re: [External] : Removal of 128MB limit for BPF JIT programs broke perf symbolication on aarch64

["Russell King (Oracle)" <linux@armlinux.org.uk>]

----- Forwarded message from Ivan Babrou <ivan@cloudflare.com> -----
Date: Tue, 4 Oct 2022 16:50:01 -0700
From: Ivan Babrou <ivan@cloudflare.com>
To: "Russell King (Oracle)" <linux@armlinux.org.uk>
Cc: Will Deacon <will@kernel.org>
Subject: Re: [External] : Removal of 128MB limit for BPF JIT programs broke
	perf symbolication on aarch64

Hi Russell,

Thanks for looking closely into this. Could you send this as a bug
report to the perf mailing list? Since it seems to be a bug in perf, I
hope people there will be able to figure out how to address this
properly.

Many thanks!

On Thu, Aug 18, 2022 at 4:33 AM Russell King (Oracle)
<linux@armlinux.org.uk> wrote:
>
> On Wed, Aug 17, 2022 at 05:46:03PM +0100, Russell King (Oracle) wrote:
> > So yes, I can reproduce the problem given exactly the right kernel
> > options, but I still don't see how it is caused by the commit you
> > claim it is. I think it's indicative of a deeper problem, one where
> > the kernel image is being placed in the vmalloc region, and /proc/kcore
> > then ends up with conflicting LOAD segments in its PHDR.
>
> It looks to be a bug in the perf tool:
>
> --- broken2     2022-08-18 12:11:29.219489345 +0100
> +++ working2    2022-08-18 12:11:06.716047129 +0100
> ...
> -replacement_map: ffffffc008000000-fffffffdf0000000 0000004008003000
> +replacement_map: ffffffd793200000-ffffffd794f20000 0000005793203000
>  New map:
>  Start            End              pgoff              start-pgoff
>  ffffff8000000000-ffffff807be00000 0000000000003000 = ffffff7fffffd000
>  ffffffa000000000-ffffffa700000000 0000002000003000 = ffffff7fffffd000
>  ffffffc000000000-ffffffc008000000 0000004000003000 = ffffff7fffffd000
> -ffffffc008000000-fffffffdf0000000 0000004008003000 = ffffff7fffffd000
> +ffffffc008000000-ffffffc0085a5890 0000004008003000 = ffffff7fffffd000
>  ffffffc0085a5890-ffffffc0085a5a60 0000000000000000 = ffffffc0085a5890
> +ffffffc0085a5a60-ffffffc0085b5b90 00000040085a8a60 = ffffff7fffffd000
>  ffffffc0085b5b90-ffffffc0085b5bf0 0000000000000000 = ffffffc0085b5b90
> +ffffffc0085b5bf0-ffffffc0085c58b0 00000040085b8bf0 = ffffff7fffffd000
>  ffffffc0085c58b0-ffffffc0085c5910 0000000000000000 = ffffffc0085c58b0
> +ffffffc0085c5910-ffffffc0086e5c64 00000040085c8910 = ffffff7fffffd000
>  ffffffc0086e5c64-ffffffc0086e5e30 0000000000000000 = ffffffc0086e5c64
> +ffffffc0086e5e30-ffffffc0086f5284 00000040086e8e30 = ffffff7fffffd000
>  ffffffc0086f5284-ffffffc0086f52e4 0000000000000000 = ffffffc0086f5284
> +ffffffc0086f52e4-ffffffc0087051c8 00000040086f82e4 = ffffff7fffffd000
>  ffffffc0087051c8-ffffffc008705228 0000000000000000 = ffffffc0087051c8
> +ffffffc008705228-ffffffc0093fdcbc 0000004008708228 = ffffff7fffffd000
>  ffffffc0093fdcbc-ffffffc0093fdd1c 0000000000000000 = ffffffc0093fdcbc
> +ffffffc0093fdd1c-ffffffc00940dc70 0000004009400d1c = ffffff7fffffd000
>  ffffffc00940dc70-ffffffc00940dcd0 0000000000000000 = ffffffc00940dc70
> +ffffffc00940dcd0-ffffffc0094257a0 0000004009410cd0 = ffffff7fffffd000
>  ffffffc0094257a0-ffffffc009425800 0000000000000000 = ffffffc0094257a0
> +ffffffc009425800-ffffffc00942f420 0000004009428800 = ffffff7fffffd000
>  ffffffc00942f420-ffffffc00942f480 0000000000000000 = ffffffc00942f420
> +ffffffc00942f480-ffffffc009435ae0 0000004009432480 = ffffff7fffffd000
>  ffffffc009435ae0-ffffffc009435d64 0000000000000000 = ffffffc009435ae0
> +ffffffc009435d64-ffffffd793200000 0000004009438d64 = ffffff7fffffd000
> +ffffffd793200000-ffffffd794f20000 0000005793203000 = ffffff7fffffd000
>
> The above is a red/black tree, which is ordered by start/end addresses.
> Overlaps are not allowed, yet we can see in the broken case that we end
> up with an overlap as this entry:
>
> -ffffffc008000000-fffffffdf0000000 0000004008003000 = ffffff7fffffd000
>
> overlaps all the following entries. This happens because the wrong entry
> is picked up from the sections in /proc/kcore, and is selected by:
>
>         /* Find the kernel map using the '_stext' symbol */
>         if (!kallsyms__get_function_start(kallsyms_filename, "_stext", &stext)) {
>                 list_for_each_entry(new_map, &md.maps, node) {
> //                      if (new_map->start == 0xffffffc008000000)
> //                              continue;
>                         if (stext >= new_map->start && stext < new_map->end) {
>                                 replacement_map = new_map;
>                                 break;
>                         }
>                 }
>         }
>
> The commented out code is my hack that makes perf work, by forcing it
> to use the proper entry in /proc/kcore for the kernel.
>
> The side effect of this is that in thread__find_map(), maps__find() is
> unable to find the right map entry to use, so al->map is NULL.
> Consequently, the address can't be translated to a symbol. We can see
> this in my debug output:
>
> rmk: ffffffd793e882e4: 0x7fc81bfd96 01 lvl k maps 0x55a3149bd0 map (nil)
>
> ffffffd793e882e4 is the address being looked up. The maps being used
> were:
>
> 0x55a3bcd1f0: ffffff8000000000-ffffff807be00000 0000000000003000
> 0x55a3bcd4f0: ffffffa000000000-ffffffa700000000 0000002000003000
> 0x55a3bcd070: ffffffc000000000-ffffffc008000000 0000004000003000
> 0x55a3150000: ffffffc008000000-fffffffdf0000000 0000004008003000
> 0x55a31507c0: ffffffc0085a5890-ffffffc0085a5a60 0000000000000000
> 0x55a3150fb0: ffffffc0085b5b90-ffffffc0085b5bf0 0000000000000000
> 0x55a3151370: ffffffc0085c58b0-ffffffc0085c5910 0000000000000000
> 0x55a3151730: ffffffc0086e5c64-ffffffc0086e5e30 0000000000000000
> 0x55a3151af0: ffffffc0086f5284-ffffffc0086f52e4 0000000000000000
> 0x55a3151eb0: ffffffc0087051c8-ffffffc008705228 0000000000000000
> 0x55a3152630: ffffffc0093fdcbc-ffffffc0093fdd1c 0000000000000000
> 0x55a31529f0: ffffffc00940dc70-ffffffc00940dcd0 0000000000000000
> 0x55a3152db0: ffffffc0094257a0-ffffffc009425800 0000000000000000
> 0x55a3153170: ffffffc00942f420-ffffffc00942f480 0000000000000000
> 0x55a3152270: ffffffc009435ae0-ffffffc009435d64 0000000000000000
>
> We can see this address would fall into the 0x55a3150000 map entry,
> but we can see "map (nil)" meaning it wasn't found.
>
> I've no idea what the right fix for this is.
>
> To be honest, I'm not sure I have a full enough understanding of the
> perf tool to really grasp what it's trying to do here, so I don't
> think I can progress this to a solution.
>
> I also think that the commit you refer to as causing this is just
> exposing a bug that's been there for quite some time, and it is
> unfortunate that changing the kernel layout slightly has provoked
> it in the perf tool. I'm not sure if this is a perf tool bug or
> a kernel bug (maybe the kernel shouldn't be exposing overlapping
> LOAD sections in /proc/kcore).
>
> Adding Will in case he has any ideas.
>
> --
> RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
> FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!


----- End forwarded message -----

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!