Re: [PATCH 01/11] objtool: Generic annotation infrastructure

Re: [PATCH 01/11] objtool: Generic annotation infrastructure

[Peter Zijlstra]

On Mon, Dec 04, 2023 at 10:37:03AM +0100, Peter Zijlstra wrote:
> Avoid endless .discard.foo sections for each annotation, create a
> single .discard.annotate section that takes an annotation type along
> with the instruction.
> 
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> ---
> --- a/include/linux/objtool.h
> +++ b/include/linux/objtool.h
> @@ -57,6 +57,13 @@
>  	".long 998b\n\t"						\
>  	".popsection\n\t"
>  
> +#define ASM_ANNOTATE(x)						\
> +	"911:\n\t"						\
> +	".pushsection .discard.annotate,\"M\",@progbits,8\n\t"	\
> +	".long 911b - .\n\t"					\
> +	".long " __stringify(x) "\n\t"				\
> +	".popsection\n\t"
> +
>  #else /* __ASSEMBLY__ */
>  
>  /*
> @@ -146,6 +153,14 @@
>  	.popsection
>  .endm
>  
> +.macro ANNOTATE type:req
> +.Lhere_\@:
> +	.pushsection .discard.annotate,"M",@progbits,8
> +	.long	.Lhere_\@ - .
> +	.long	\type
> +	.popsection
> +.endm
> +
>  #endif /* __ASSEMBLY__ */
>  
>  #else /* !CONFIG_OBJTOOL */
> @@ -167,6 +182,8 @@
>  .endm
>  .macro REACHABLE
>  .endm
> +.macro ANNOTATE
> +.endm
>  #endif
>  
>  #endif /* CONFIG_OBJTOOL */
> --- a/tools/objtool/check.c
> +++ b/tools/objtool/check.c
> @@ -2308,6 +2308,41 @@ static int read_unwind_hints(struct objt
>  	return 0;
>  }
>  
> +static int read_annotate(struct objtool_file *file, void (*func)(int type, struct instruction *insn))
> +{
> +	struct section *rsec, *sec;
> +	struct instruction *insn;
> +	struct reloc *reloc;
> +	int type;
> +
> +	rsec = find_section_by_name(file->elf, ".rela.discard.annotate");
> +	if (!rsec)
> +		return 0;
> +
> +	sec = find_section_by_name(file->elf, ".discard.annotate");
> +	if (!sec)
> +		return 0;
> +
> +	for_each_reloc(rsec, reloc) {
> +		insn = find_insn(file, reloc->sym->sec,
> +				 reloc->sym->offset + reloc_addend(reloc));
> +		if (!insn) {
> +			WARN("bad .discard.annotate entry: %d", reloc_idx(reloc));
> +			return -1;
> +		}
> +
> +		type = *(u32 *)(sec->data->d_buf + (reloc_idx(reloc) * sec->sh.sh_entsize) + 4);
> +
> +		func(type, insn);
> +	}
> +
> +	return 0;
> +}

So... ld.lld hates this :-(

From an LLVM=-19 build we can see that:

$ readelf -WS tmp-build/arch/x86/kvm/vmx/vmenter.o | grep annotate
  [13] .discard.annotate PROGBITS        0000000000000000 00028c 000018 08   M  0   0  1

$ readelf -WS tmp-build/arch/x86/kvm/kvm-intel.o | grep annotate
  [ 3] .discard.annotate PROGBITS        0000000000000000 069fe0 0089d0 00   M  0   0  1

Which tells us that the translation unit itself has a sh_entsize of 8,
while the linked object has sh_entsize of 0.

This then completely messes up the indexing objtool does, which relies
on it being a sane number.

GCC/binutils very much does not do this, it retains the 8.

Dear clang folks, help?

Re: [PATCH 01/11] objtool: Generic annotation infrastructure

[Nathan Chancellor]

On Fri, Nov 08, 2024 at 03:16:00PM +0100, Peter Zijlstra wrote:
> On Mon, Dec 04, 2023 at 10:37:03AM +0100, Peter Zijlstra wrote:
> > Avoid endless .discard.foo sections for each annotation, create a
> > single .discard.annotate section that takes an annotation type along
> > with the instruction.
> > 
> > Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> > ---
> > --- a/include/linux/objtool.h
> > +++ b/include/linux/objtool.h
> > @@ -57,6 +57,13 @@
> >  	".long 998b\n\t"						\
> >  	".popsection\n\t"
> >  
> > +#define ASM_ANNOTATE(x)						\
> > +	"911:\n\t"						\
> > +	".pushsection .discard.annotate,\"M\",@progbits,8\n\t"	\
> > +	".long 911b - .\n\t"					\
> > +	".long " __stringify(x) "\n\t"				\
> > +	".popsection\n\t"
> > +
> >  #else /* __ASSEMBLY__ */
> >  
> >  /*
> > @@ -146,6 +153,14 @@
> >  	.popsection
> >  .endm
> >  
> > +.macro ANNOTATE type:req
> > +.Lhere_\@:
> > +	.pushsection .discard.annotate,"M",@progbits,8
> > +	.long	.Lhere_\@ - .
> > +	.long	\type
> > +	.popsection
> > +.endm
> > +
> >  #endif /* __ASSEMBLY__ */
> >  
> >  #else /* !CONFIG_OBJTOOL */
> > @@ -167,6 +182,8 @@
> >  .endm
> >  .macro REACHABLE
> >  .endm
> > +.macro ANNOTATE
> > +.endm
> >  #endif
> >  
> >  #endif /* CONFIG_OBJTOOL */
> > --- a/tools/objtool/check.c
> > +++ b/tools/objtool/check.c
> > @@ -2308,6 +2308,41 @@ static int read_unwind_hints(struct objt
> >  	return 0;
> >  }
> >  
> > +static int read_annotate(struct objtool_file *file, void (*func)(int type, struct instruction *insn))
> > +{
> > +	struct section *rsec, *sec;
> > +	struct instruction *insn;
> > +	struct reloc *reloc;
> > +	int type;
> > +
> > +	rsec = find_section_by_name(file->elf, ".rela.discard.annotate");
> > +	if (!rsec)
> > +		return 0;
> > +
> > +	sec = find_section_by_name(file->elf, ".discard.annotate");
> > +	if (!sec)
> > +		return 0;
> > +
> > +	for_each_reloc(rsec, reloc) {
> > +		insn = find_insn(file, reloc->sym->sec,
> > +				 reloc->sym->offset + reloc_addend(reloc));
> > +		if (!insn) {
> > +			WARN("bad .discard.annotate entry: %d", reloc_idx(reloc));
> > +			return -1;
> > +		}
> > +
> > +		type = *(u32 *)(sec->data->d_buf + (reloc_idx(reloc) * sec->sh.sh_entsize) + 4);
> > +
> > +		func(type, insn);
> > +	}
> > +
> > +	return 0;
> > +}
> 
> So... ld.lld hates this :-(
> 
> From an LLVM=-19 build we can see that:
> 
> $ readelf -WS tmp-build/arch/x86/kvm/vmx/vmenter.o | grep annotate
>   [13] .discard.annotate PROGBITS        0000000000000000 00028c 000018 08   M  0   0  1
> 
> $ readelf -WS tmp-build/arch/x86/kvm/kvm-intel.o | grep annotate
>   [ 3] .discard.annotate PROGBITS        0000000000000000 069fe0 0089d0 00   M  0   0  1
> 
> Which tells us that the translation unit itself has a sh_entsize of 8,
> while the linked object has sh_entsize of 0.
> 
> This then completely messes up the indexing objtool does, which relies
> on it being a sane number.
> 
> GCC/binutils very much does not do this, it retains the 8.
> 
> Dear clang folks, help?

Perhaps Fangrui has immediate thoughts, since this appears to be an
ld.lld thing? Otherwise, I will see if I can dig into this in the next
couple of weeks (I have an LF webinar on Wednesday that I am still
prepping for). Is this reproducible with just defconfig or something
else?

Cheers,
Nathan

Re: [PATCH 01/11] objtool: Generic annotation infrastructure

[Peter Zijlstra]

On Fri, Nov 08, 2024 at 07:57:16AM -0700, Nathan Chancellor wrote:
> On Fri, Nov 08, 2024 at 03:16:00PM +0100, Peter Zijlstra wrote:
> > On Mon, Dec 04, 2023 at 10:37:03AM +0100, Peter Zijlstra wrote:
> > > Avoid endless .discard.foo sections for each annotation, create a
> > > single .discard.annotate section that takes an annotation type along
> > > with the instruction.
> > > 
> > > Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> > > ---
> > > --- a/include/linux/objtool.h
> > > +++ b/include/linux/objtool.h
> > > @@ -57,6 +57,13 @@
> > >  	".long 998b\n\t"						\
> > >  	".popsection\n\t"
> > >  
> > > +#define ASM_ANNOTATE(x)						\
> > > +	"911:\n\t"						\
> > > +	".pushsection .discard.annotate,\"M\",@progbits,8\n\t"	\
> > > +	".long 911b - .\n\t"					\
> > > +	".long " __stringify(x) "\n\t"				\
> > > +	".popsection\n\t"
> > > +
> > >  #else /* __ASSEMBLY__ */
> > >  
> > >  /*
> > > @@ -146,6 +153,14 @@
> > >  	.popsection
> > >  .endm
> > >  
> > > +.macro ANNOTATE type:req
> > > +.Lhere_\@:
> > > +	.pushsection .discard.annotate,"M",@progbits,8
> > > +	.long	.Lhere_\@ - .
> > > +	.long	\type
> > > +	.popsection
> > > +.endm
> > > +
> > >  #endif /* __ASSEMBLY__ */
> > >  
> > >  #else /* !CONFIG_OBJTOOL */
> > > @@ -167,6 +182,8 @@
> > >  .endm
> > >  .macro REACHABLE
> > >  .endm
> > > +.macro ANNOTATE
> > > +.endm
> > >  #endif
> > >  
> > >  #endif /* CONFIG_OBJTOOL */
> > > --- a/tools/objtool/check.c
> > > +++ b/tools/objtool/check.c
> > > @@ -2308,6 +2308,41 @@ static int read_unwind_hints(struct objt
> > >  	return 0;
> > >  }
> > >  
> > > +static int read_annotate(struct objtool_file *file, void (*func)(int type, struct instruction *insn))
> > > +{
> > > +	struct section *rsec, *sec;
> > > +	struct instruction *insn;
> > > +	struct reloc *reloc;
> > > +	int type;
> > > +
> > > +	rsec = find_section_by_name(file->elf, ".rela.discard.annotate");
> > > +	if (!rsec)
> > > +		return 0;
> > > +
> > > +	sec = find_section_by_name(file->elf, ".discard.annotate");
> > > +	if (!sec)
> > > +		return 0;
> > > +
> > > +	for_each_reloc(rsec, reloc) {
> > > +		insn = find_insn(file, reloc->sym->sec,
> > > +				 reloc->sym->offset + reloc_addend(reloc));
> > > +		if (!insn) {
> > > +			WARN("bad .discard.annotate entry: %d", reloc_idx(reloc));
> > > +			return -1;
> > > +		}
> > > +
> > > +		type = *(u32 *)(sec->data->d_buf + (reloc_idx(reloc) * sec->sh.sh_entsize) + 4);
> > > +
> > > +		func(type, insn);
> > > +	}
> > > +
> > > +	return 0;
> > > +}
> > 
> > So... ld.lld hates this :-(
> > 
> > From an LLVM=-19 build we can see that:
> > 
> > $ readelf -WS tmp-build/arch/x86/kvm/vmx/vmenter.o | grep annotate
> >   [13] .discard.annotate PROGBITS        0000000000000000 00028c 000018 08   M  0   0  1
> > 
> > $ readelf -WS tmp-build/arch/x86/kvm/kvm-intel.o | grep annotate
> >   [ 3] .discard.annotate PROGBITS        0000000000000000 069fe0 0089d0 00   M  0   0  1
> > 
> > Which tells us that the translation unit itself has a sh_entsize of 8,
> > while the linked object has sh_entsize of 0.
> > 
> > This then completely messes up the indexing objtool does, which relies
> > on it being a sane number.
> > 
> > GCC/binutils very much does not do this, it retains the 8.
> > 
> > Dear clang folks, help?
> 
> Perhaps Fangrui has immediate thoughts, since this appears to be an
> ld.lld thing? Otherwise, I will see if I can dig into this in the next
> couple of weeks (I have an LF webinar on Wednesday that I am still
> prepping for). Is this reproducible with just defconfig or something
> else?

I took the .config from the report you pointed me at yesterday.

  https://lore.kernel.org/202411071743.HZsCuurm-lkp@intel.com/

And specifically the kvm build targets from:

$ make O=tmp-build/ LLVM=-19 arch/x86/kvm/

show this problem.

I just ran a defconfig, and that seems to behave properly. Notably,
vmlinux.o (definitely a link target) has entsize=8 for the relevant
section.

I'm not sure how the kvm targets might be 'special'.

Re: [PATCH 01/11] objtool: Generic annotation infrastructure

[Peter Zijlstra]

On Fri, Nov 08, 2024 at 03:16:00PM +0100, Peter Zijlstra wrote:

> From an LLVM=-19 build we can see that:
> 
> $ readelf -WS tmp-build/arch/x86/kvm/vmx/vmenter.o | grep annotate
>   [13] .discard.annotate PROGBITS        0000000000000000 00028c 000018 08   M  0   0  1
> 
> $ readelf -WS tmp-build/arch/x86/kvm/kvm-intel.o | grep annotate
>   [ 3] .discard.annotate PROGBITS        0000000000000000 069fe0 0089d0 00   M  0   0  1
> 
> Which tells us that the translation unit itself has a sh_entsize of 8,
> while the linked object has sh_entsize of 0.
> 
> This then completely messes up the indexing objtool does, which relies
> on it being a sane number.
> 
> GCC/binutils very much does not do this, it retains the 8.

Anyway, for now I've added:

+       if (sec->sh.sh_entsize != 8) {
+               static bool warn = false;
+               if (!warn) {
+                       WARN("%s: dodgy linker, sh_entsize != 8", sec->name);
+                       warn = true;
+               }
+               sec->sh.sh_entsize = 8;
+       }

To objtool, this allows it function correctly and prints this reminder
to for us to figure out the linker story.