[PATCH 0/2] Fix 'faddr2line' for LLVM arm64 builds

[PATCH 0/2] Fix 'faddr2line' for LLVM arm64 builds

[Will Deacon]

Hi Josh,

We recently noticed that 'faddr2line' isn't working very well on arm64
Android kernels built with LLVM, so I've hacked up this pair of fixes
which get it back into action. Please take a look!

I suck at shell, so apologies in advance for the patches.

Cheers,

Will

Cc: Josh Poimboeuf <jpoimboe@kernel.org>
Cc: John Stultz <jstultz@google.com>

--->8

Will Deacon (2):
  scripts/faddr2line: Ignore non-function symbols in readelf output
  scripts/faddr2line: Use LLVM addr2line and readelf if LLVM=1

 scripts/faddr2line | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

-- 
2.41.0.487.g6d72f3e995-goog

[PATCH 1/2] scripts/faddr2line: Ignore non-function symbols in readelf output

[Will Deacon]

Non-function symbols emitted in the readelf output can confuse the
'faddr2line' symbol size calculation, resulting in the erroneous
rejection of valid offsets. This is especially prevalent when building
an arm64 kernel with CONFIG_CFI_CLANG=y, where most functions are
prefixed with a 32-bit data value in a '$d.n' section. For example:

447538: ffff800080014b80   548 FUNC    GLOBAL DEFAULT    2 do_one_initcall
   104: ffff800080014c74     0 NOTYPE  LOCAL  DEFAULT    2 $x.73
   106: ffff800080014d30     0 NOTYPE  LOCAL  DEFAULT    2 $x.75
   111: ffff800080014da4     0 NOTYPE  LOCAL  DEFAULT    2 $d.78
   112: ffff800080014da8     0 NOTYPE  LOCAL  DEFAULT    2 $x.79
    36: ffff800080014de0   200 FUNC    LOCAL  DEFAULT    2 run_init_process

Adding a warning to do_one_initcall() results in:

  | WARNING: CPU: 0 PID: 1 at init/main.c:1236 do_one_initcall+0xf4/0x260

Which 'faddr2line' refuses to accept:

$ ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260
skipping do_one_initcall address at 0xffff800080014c74 due to size mismatch (0x260 != 0x224)
no match for do_one_initcall+0xf4/0x260

Filter out entries from readelf that are not "FUNC" type, so that the
size of a symbol is calculated as a delta to the next function address.

Cc: Josh Poimboeuf <jpoimboe@kernel.org>
Cc: John Stultz <jstultz@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
 scripts/faddr2line | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/faddr2line b/scripts/faddr2line
index 0e73aca4f908..60368a1cdaed 100755
--- a/scripts/faddr2line
+++ b/scripts/faddr2line
@@ -179,7 +179,7 @@ __faddr2line() {
 				found=2
 				break
 			fi
-		done < <(${READELF} --symbols --wide $objfile | sed 's/\[.*\]//' | ${AWK} -v sec=$sym_sec '$7 == sec' | sort --key=2)
+		done < <(${READELF} --symbols --wide $objfile | sed 's/\[.*\]//' | ${AWK} -v sec=$sym_sec '$4 == "FUNC" && $7 == sec' | sort --key=2)
 
 		if [[ $found = 0 ]]; then
 			warn "can't find symbol: sym_name: $sym_name sym_sec: $sym_sec sym_addr: $sym_addr sym_elf_size: $sym_elf_size"
-- 
2.41.0.487.g6d72f3e995-goog

Re: [PATCH 1/2] scripts/faddr2line: Ignore non-function symbols in readelf output

[Josh Poimboeuf]

On Mon, Jul 24, 2023 at 06:45:16PM +0100, Will Deacon wrote:
> Non-function symbols emitted in the readelf output can confuse the
> 'faddr2line' symbol size calculation, resulting in the erroneous
> rejection of valid offsets. This is especially prevalent when building
> an arm64 kernel with CONFIG_CFI_CLANG=y, where most functions are
> prefixed with a 32-bit data value in a '$d.n' section. For example:
> 
> 447538: ffff800080014b80   548 FUNC    GLOBAL DEFAULT    2 do_one_initcall
>    104: ffff800080014c74     0 NOTYPE  LOCAL  DEFAULT    2 $x.73
>    106: ffff800080014d30     0 NOTYPE  LOCAL  DEFAULT    2 $x.75
>    111: ffff800080014da4     0 NOTYPE  LOCAL  DEFAULT    2 $d.78
>    112: ffff800080014da8     0 NOTYPE  LOCAL  DEFAULT    2 $x.79
>     36: ffff800080014de0   200 FUNC    LOCAL  DEFAULT    2 run_init_process
> 
> Adding a warning to do_one_initcall() results in:
> 
>   | WARNING: CPU: 0 PID: 1 at init/main.c:1236 do_one_initcall+0xf4/0x260
> 
> Which 'faddr2line' refuses to accept:
> 
> $ ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260
> skipping do_one_initcall address at 0xffff800080014c74 due to size mismatch (0x260 != 0x224)
> no match for do_one_initcall+0xf4/0x260
> 
> Filter out entries from readelf that are not "FUNC" type, so that the
> size of a symbol is calculated as a delta to the next function address.

Problem is, I think the kernel's symbol printing code prints the nearest
kallsyms symbol, and there are some valid non-FUNC code symbols.  For
example, syscall_return_via_sysret.  This patch would cause those valid
symbols to get missed.

I presume these '$x.*' symbols don't end up in System.map?  How do they
get filtered out?  By the linker maybe?

We may want to use whatever criteria used there, e.g. have faddr2line
ignore symbols starting with '$' or so?

-- 
Josh

Re: [PATCH 1/2] scripts/faddr2line: Ignore non-function symbols in readelf output

[Will Deacon]

On Mon, Jul 24, 2023 at 04:47:34PM -0700, Josh Poimboeuf wrote:
> On Mon, Jul 24, 2023 at 06:45:16PM +0100, Will Deacon wrote:
> > Non-function symbols emitted in the readelf output can confuse the
> > 'faddr2line' symbol size calculation, resulting in the erroneous
> > rejection of valid offsets. This is especially prevalent when building
> > an arm64 kernel with CONFIG_CFI_CLANG=y, where most functions are
> > prefixed with a 32-bit data value in a '$d.n' section. For example:
> > 
> > 447538: ffff800080014b80   548 FUNC    GLOBAL DEFAULT    2 do_one_initcall
> >    104: ffff800080014c74     0 NOTYPE  LOCAL  DEFAULT    2 $x.73
> >    106: ffff800080014d30     0 NOTYPE  LOCAL  DEFAULT    2 $x.75
> >    111: ffff800080014da4     0 NOTYPE  LOCAL  DEFAULT    2 $d.78
> >    112: ffff800080014da8     0 NOTYPE  LOCAL  DEFAULT    2 $x.79
> >     36: ffff800080014de0   200 FUNC    LOCAL  DEFAULT    2 run_init_process
> > 
> > Adding a warning to do_one_initcall() results in:
> > 
> >   | WARNING: CPU: 0 PID: 1 at init/main.c:1236 do_one_initcall+0xf4/0x260
> > 
> > Which 'faddr2line' refuses to accept:
> > 
> > $ ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260
> > skipping do_one_initcall address at 0xffff800080014c74 due to size mismatch (0x260 != 0x224)
> > no match for do_one_initcall+0xf4/0x260
> > 
> > Filter out entries from readelf that are not "FUNC" type, so that the
> > size of a symbol is calculated as a delta to the next function address.
> 
> Problem is, I think the kernel's symbol printing code prints the nearest
> kallsyms symbol, and there are some valid non-FUNC code symbols.  For
> example, syscall_return_via_sysret.  This patch would cause those valid
> symbols to get missed.

Damn, yes, I can see a few of those in the arm64 vmlinux too. Thanks for
pointing it out.

> I presume these '$x.*' symbols don't end up in System.map?  How do they
> get filtered out?  By the linker maybe?

See the horrible ad-hoc list of ignored symbol types which get regexed
out by scripts/mksysmap.

> We may want to use whatever criteria used there, e.g. have faddr2line
> ignore symbols starting with '$' or so?

My first hack for this did exactly that (ignore symbols starting with '$'),
but then I tried to find ways to make it a bit more general. Lemme have a
crack at factoring out the stuff from mksysmap before I resort back to
checking for the '$' prefix.

Will

[tip: objtool/core] scripts/faddr2line: Don't filter out non-function symbols from readelf

[tip-bot2 for Will Deacon]

The following commit has been merged into the objtool/core branch of tip:

Commit-ID:     180af1a5bdaf8d4964837a46a9fce8c3a7fd2d97
Gitweb:        https://git.kernel.org/tip/180af1a5bdaf8d4964837a46a9fce8c3a7fd2d97
Author:        Will Deacon <will@kernel.org>
AuthorDate:    Mon, 02 Oct 2023 17:57:47 +01:00
Committer:     Josh Poimboeuf <jpoimboe@kernel.org>
CommitterDate: Mon, 23 Oct 2023 08:35:01 -07:00

scripts/faddr2line: Don't filter out non-function symbols from readelf

As Josh points out in 20230724234734.zy67gm674vl3p3wv@treble:

> Problem is, I think the kernel's symbol printing code prints the
> nearest kallsyms symbol, and there are some valid non-FUNC code
> symbols.  For example, syscall_return_via_sysret.

so we shouldn't be considering only 'FUNC'-type symbols in the output
from readelf.

Drop the function symbol type filtering from the faddr2line outer loop.

Suggested-by: Josh Poimboeuf <jpoimboe@kernel.org>
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
Link: https://lore.kernel.org/r/20230724234734.zy67gm674vl3p3wv@treble
Signed-off-by: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20231002165750.1661-2-will@kernel.org
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
---
 scripts/faddr2line | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/faddr2line b/scripts/faddr2line
index 0e73aca..a35a420 100755
--- a/scripts/faddr2line
+++ b/scripts/faddr2line
@@ -260,7 +260,7 @@ __faddr2line() {
 
 		DONE=1
 
-	done < <(${READELF} --symbols --wide $objfile | sed 's/\[.*\]//' | ${AWK} -v fn=$sym_name '$4 == "FUNC" && $8 == fn')
+	done < <(${READELF} --symbols --wide $objfile | sed 's/\[.*\]//' | ${AWK} -v fn=$sym_name '$8 == fn')
 }
 
 [[ $# -lt 2 ]] && usage

[PATCH 2/2] scripts/faddr2line: Use LLVM addr2line and readelf if LLVM=1

[Will Deacon]

GNU utilities cannot necessarily parse objects built by LLVM, which can
result in confusing errors when using 'faddr2line':

$ CROSS_COMPILE=aarch64-linux-gnu- ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260
aarch64-linux-gnu-addr2line: vmlinux: unknown type [0x13] section `.relr.dyn'
aarch64-linux-gnu-addr2line: DWARF error: invalid or unhandled FORM value: 0x25
do_one_initcall+0xf4/0x260:
aarch64-linux-gnu-addr2line: vmlinux: unknown type [0x13] section `.relr.dyn'
aarch64-linux-gnu-addr2line: DWARF error: invalid or unhandled FORM value: 0x25
$x.73 at main.c:?

Although this can be worked around by setting CROSS_COMPILE to "llvm=-",
it's cleaner to follow the same syntax as the top-level Makefile and
accept LLVM=1 as an indication to use the llvm- tools.

Cc: Josh Poimboeuf <jpoimboe@kernel.org>
Cc: John Stultz <jstultz@google.com>
Signed-off-by: Will Deacon <will@kernel.org>
---
 scripts/faddr2line | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/scripts/faddr2line b/scripts/faddr2line
index 60368a1cdaed..24845267efb2 100755
--- a/scripts/faddr2line
+++ b/scripts/faddr2line
@@ -58,8 +58,14 @@ die() {
 	exit 1
 }
 
-READELF="${CROSS_COMPILE:-}readelf"
-ADDR2LINE="${CROSS_COMPILE:-}addr2line"
+if [ "${LLVM:-}" == "1" ]; then
+	UTIL_PREFIX=llvm-
+else
+	UTIL_PREFIX=${CROSS_COMPILE:-}
+fi
+
+READELF="${UTIL_PREFIX}readelf"
+ADDR2LINE="${UTIL_PREFIX}addr2line"
 AWK="awk"
 GREP="grep"
 
-- 
2.41.0.487.g6d72f3e995-goog

Re: [PATCH 2/2] scripts/faddr2line: Use LLVM addr2line and readelf if LLVM=1

[Josh Poimboeuf]

On Mon, Jul 24, 2023 at 06:45:17PM +0100, Will Deacon wrote:
> GNU utilities cannot necessarily parse objects built by LLVM, which can
> result in confusing errors when using 'faddr2line':
> 
> $ CROSS_COMPILE=aarch64-linux-gnu- ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260
> aarch64-linux-gnu-addr2line: vmlinux: unknown type [0x13] section `.relr.dyn'
> aarch64-linux-gnu-addr2line: DWARF error: invalid or unhandled FORM value: 0x25
> do_one_initcall+0xf4/0x260:
> aarch64-linux-gnu-addr2line: vmlinux: unknown type [0x13] section `.relr.dyn'
> aarch64-linux-gnu-addr2line: DWARF error: invalid or unhandled FORM value: 0x25
> $x.73 at main.c:?
> 
> Although this can be worked around by setting CROSS_COMPILE to "llvm=-",
> it's cleaner to follow the same syntax as the top-level Makefile and
> accept LLVM=1 as an indication to use the llvm- tools.
> 
> Cc: Josh Poimboeuf <jpoimboe@kernel.org>
> Cc: John Stultz <jstultz@google.com>
> Signed-off-by: Will Deacon <will@kernel.org>

This one looks good to me, I'll go ahead and queue it.

-- 
Josh

Re: [PATCH 2/2] scripts/faddr2line: Use LLVM addr2line and readelf if LLVM=1

[Will Deacon]

On Mon, Jul 24, 2023 at 05:07:49PM -0700, Josh Poimboeuf wrote:
> On Mon, Jul 24, 2023 at 06:45:17PM +0100, Will Deacon wrote:
> > GNU utilities cannot necessarily parse objects built by LLVM, which can
> > result in confusing errors when using 'faddr2line':
> > 
> > $ CROSS_COMPILE=aarch64-linux-gnu- ./scripts/faddr2line vmlinux do_one_initcall+0xf4/0x260
> > aarch64-linux-gnu-addr2line: vmlinux: unknown type [0x13] section `.relr.dyn'
> > aarch64-linux-gnu-addr2line: DWARF error: invalid or unhandled FORM value: 0x25
> > do_one_initcall+0xf4/0x260:
> > aarch64-linux-gnu-addr2line: vmlinux: unknown type [0x13] section `.relr.dyn'
> > aarch64-linux-gnu-addr2line: DWARF error: invalid or unhandled FORM value: 0x25
> > $x.73 at main.c:?
> > 
> > Although this can be worked around by setting CROSS_COMPILE to "llvm=-",
> > it's cleaner to follow the same syntax as the top-level Makefile and
> > accept LLVM=1 as an indication to use the llvm- tools.
> > 
> > Cc: Josh Poimboeuf <jpoimboe@kernel.org>
> > Cc: John Stultz <jstultz@google.com>
> > Signed-off-by: Will Deacon <will@kernel.org>
> 
> This one looks good to me, I'll go ahead and queue it.

Thanks, Josh!

Will