Re: [PATCH RFC] x86/lld: fix symbol map generation

[Jan Beulich <jbeulich@suse.com>]

On 02.05.2022 17:20, Roger Pau Monne wrote:
> The symbol map generation (and thus the debug info attached to Xen) is
> partially broken when using LLVM LD.  That's due to LLD converting
> almost all symbols from global to local in the last linking step, and

I'm puzzled by "almost" - is there a pattern of which ones aren't
converted?

Also "last linking step" is ambiguous, as we link three binaries and
aiui the issue is present on every of these passes. May I suggest
"... when linking actual executables" or (still somewhat ambiguous)
"... when linking final binaries"?

> thus confusing tools/symbols into adding a file prefix to all text
> symbols, the results looks like:
> 
> Xen call trace:
>    [<ffff82d040449fe8>] R xxhash64.c#__start_xen+0x3938/0x39c0
>    [<ffff82d040203734>] F __high_start+0x94/0xa0
> 
> In order to workaround this create a list of global symbols prior to
> the linking step, and use objcopy to convert the symbols in the final
> binary back to global before processing with tools/symbols.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> ---
> I haven't found a way to prevent LLD from converting the symbols, so
> I've come up with this rather crappy workaround.

Perhaps a map file (like we use for shared libraries in tools/) would
allow doing so? But of course this would want to be machine-generated,
not manually maintained.

Have you gained any insight into _why_ they are doing what they do?

> Not applied to EFI, partially because I don't have an environment with
> LLD capable of generating the EFI binary.
> 
> Obtaining the global symbol list could likely be a target on itself,
> if it is to be shared between the ELF and the EFI binary generation.

If, as the last paragraph of the description is worded, you did this
just once (as a prereq), I could see this working. Otherwise (as you
have it now, with it done 3 times) it would first require splitting
the linking rules into many separate ones (which has been the plan
anyway, but so far I didn't get to it).

> --- a/xen/arch/x86/Makefile
> +++ b/xen/arch/x86/Makefile
> @@ -134,24 +134,34 @@ $(TARGET): $(TARGET)-syms $(efi-y) $(obj)/boot/mkelf32
>  CFLAGS-$(XEN_BUILD_EFI) += -DXEN_BUILD_EFI
>  
>  $(TARGET)-syms: $(objtree)/prelink.o $(obj)/xen.lds
> +	# Dump global text symbols before the linking step
> +	$(NM) -pa --format=bsd $< | awk '{ if($$2 == "T") print $$3}' \
> +	    > $(@D)/.$(@F).global-syms
>  	$(LD) $(XEN_LDFLAGS) -T $(obj)/xen.lds -N $< $(build_id_linker) \
> -	    $(objtree)/common/symbols-dummy.o -o $(@D)/.$(@F).0
> +	    $(objtree)/common/symbols-dummy.o -o $(@D)/.$(@F).0.tmp
> +	# LLVM LD has converted global symbols into local ones as part of the
> +	# linking step, convert those back to global before using tools/symbols.
> +	$(OBJCOPY) --globalize-symbols=$(@D)/.$(@F).global-syms \
> +	    $(@D)/.$(@F).0.tmp $(@D)/.$(@F).0
>  	$(NM) -pa --format=sysv $(@D)/.$(@F).0 \
>  		| $(objtree)/tools/symbols $(all_symbols) --sysv --sort \
>  		>$(@D)/.$(@F).0.S
>  	$(MAKE) $(build)=$(@D) $(@D)/.$(@F).0.o
>  	$(LD) $(XEN_LDFLAGS) -T $(obj)/xen.lds -N $< $(build_id_linker) \
> -	    $(@D)/.$(@F).0.o -o $(@D)/.$(@F).1
> +	    $(@D)/.$(@F).0.o -o $(@D)/.$(@F).1.tmp
> +	$(OBJCOPY) --globalize-symbols=$(@D)/.$(@F).global-syms \
> +	    $(@D)/.$(@F).1.tmp $(@D)/.$(@F).1
>  	$(NM) -pa --format=sysv $(@D)/.$(@F).1 \
>  		| $(objtree)/tools/symbols $(all_symbols) --sysv --sort $(syms-warn-dup-y) \
>  		>$(@D)/.$(@F).1.S
>  	$(MAKE) $(build)=$(@D) $(@D)/.$(@F).1.o
>  	$(LD) $(XEN_LDFLAGS) -T $(obj)/xen.lds -N $< $(build_id_linker) \
> -	    $(orphan-handling-y) $(@D)/.$(@F).1.o -o $@
> +	    $(orphan-handling-y) $(@D)/.$(@F).1.o -o $@.tmp
> +	$(OBJCOPY) --globalize-symbols=$(@D)/.$(@F).global-syms $@.tmp $@

Is this very useful? It only affects ...

>  	$(NM) -pa --format=sysv $(@D)/$(@F) \
>  		| $(objtree)/tools/symbols --all-symbols --xensyms --sysv --sort \
>  		>$(@D)/$(@F).map

... the actual map file; what's in the binary and in this map file doesn't
depend on local vs global anymore (and you limit this to text symbols
anyway; I wonder in how far livepatching might also be affected by the
same issue with data symbols).

In any event I would like to ask that the objcopy invocations be tied to
lld being in use. No matter that it shouldn't, objcopy can alter binaries
even if no actual change is being made (I've just recently observed this
with xen.efi, see the thread rooted at "EFI: strip xen.efi when putting it
on the EFI partition", and recall that at least for GNU binutils objcopy
and strip are effectively [almost] the same binary).

Jan