[PATCH] scripts/decode_stacktrace.sh: optionally use LLVM utilities

[PATCH] scripts/decode_stacktrace.sh: optionally use LLVM utilities

[Carlos Llamas]

GNU's addr2line can have problems parsing a vmlinux built with LLVM,
particularly when LTO was used. In order to decode the traces correctly
this patch adds the ability to switch to LLVM's utilities readelf and
addr2line. The same approach is followed by Will in [1].

Before:
  $ scripts/decode_stacktrace.sh vmlinux < kernel.log
  [17716.240635] Call trace:
  [17716.240646] skb_cow_data (??:?)
  [17716.240654] esp6_input (ld-temp.o:?)
  [17716.240666] xfrm_input (ld-temp.o:?)
  [17716.240674] xfrm6_rcv (??:?)
  [...]

After:
  $ LLVM=1 scripts/decode_stacktrace.sh vmlinux < kernel.log
  [17716.240635] Call trace:
  [17716.240646] skb_cow_data (include/linux/skbuff.h:2172 net/core/skbuff.c:4503)
  [17716.240654] esp6_input (net/ipv6/esp6.c:977)
  [17716.240666] xfrm_input (net/xfrm/xfrm_input.c:659)
  [17716.240674] xfrm6_rcv (net/ipv6/xfrm6_input.c:172)
  [...]

Note that one could set CROSS_COMPILE=llvm- instead to hack around this
issue. However, doing so can break the decodecode routine as it will
force the selection of other LLVM utilities down the line e.g. llvm-as.

[1] https://lore.kernel.org/all/20230914131225.13415-3-will@kernel.org/

Cc: Will Deacon <will@kernel.org>
Cc: John Stultz <jstultz@google.com>
Cc: Masahiro Yamada <masahiroy@kernel.org>
Signed-off-by: Carlos Llamas <cmllamas@google.com>
---
 scripts/decode_stacktrace.sh | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/scripts/decode_stacktrace.sh b/scripts/decode_stacktrace.sh
index 564c5632e1a2..bfe5a4082d8e 100755
--- a/scripts/decode_stacktrace.sh
+++ b/scripts/decode_stacktrace.sh
@@ -16,6 +16,21 @@ elif type c++filt >/dev/null 2>&1 ; then
 	cppfilt_opts=-i
 fi
 
+UTIL_SUFFIX=
+if [[ -z ${LLVM:-} ]]; then
+	UTIL_PREFIX=${CROSS_COMPILE:-}
+else
+	UTIL_PREFIX=llvm-
+	if [[ ${LLVM} == */ ]]; then
+		UTIL_PREFIX=${LLVM}${UTIL_PREFIX}
+	elif [[ ${LLVM} == -* ]]; then
+		UTIL_SUFFIX=${LLVM}
+	fi
+fi
+
+READELF=${UTIL_PREFIX}readelf${UTIL_SUFFIX}
+ADDR2LINE=${UTIL_PREFIX}addr2line${UTIL_SUFFIX}
+
 if [[ $1 == "-r" ]] ; then
 	vmlinux=""
 	basepath="auto"
@@ -75,7 +90,7 @@ find_module() {
 
 	if [[ "$modpath" != "" ]] ; then
 		for fn in $(find "$modpath" -name "${module//_/[-_]}.ko*") ; do
-			if readelf -WS "$fn" | grep -qwF .debug_line ; then
+			if ${READELF} -WS "$fn" | grep -qwF .debug_line ; then
 				echo $fn
 				return
 			fi
@@ -169,7 +184,7 @@ parse_symbol() {
 	if [[ $aarray_support == true && "${cache[$module,$address]+isset}" == "isset" ]]; then
 		local code=${cache[$module,$address]}
 	else
-		local code=$(${CROSS_COMPILE}addr2line -i -e "$objfile" "$address" 2>/dev/null)
+		local code=$(${ADDR2LINE} -i -e "$objfile" "$address" 2>/dev/null)
 		if [[ $aarray_support == true ]]; then
 			cache[$module,$address]=$code
 		fi
-- 
2.42.0.582.g8ccd20d70d-goog

Re: [PATCH] scripts/decode_stacktrace.sh: optionally use LLVM utilities

[Nick Desaulniers]

On Thu, Sep 28, 2023 at 8:48 PM Carlos Llamas <cmllamas@google.com> wrote:
>
> GNU's addr2line can have problems parsing a vmlinux built with LLVM,
> particularly when LTO was used. In order to decode the traces correctly
> this patch adds the ability to switch to LLVM's utilities readelf and
> addr2line. The same approach is followed by Will in [1].
>
> Before:
>   $ scripts/decode_stacktrace.sh vmlinux < kernel.log
>   [17716.240635] Call trace:
>   [17716.240646] skb_cow_data (??:?)
>   [17716.240654] esp6_input (ld-temp.o:?)
>   [17716.240666] xfrm_input (ld-temp.o:?)
>   [17716.240674] xfrm6_rcv (??:?)
>   [...]
>
> After:
>   $ LLVM=1 scripts/decode_stacktrace.sh vmlinux < kernel.log
>   [17716.240635] Call trace:
>   [17716.240646] skb_cow_data (include/linux/skbuff.h:2172 net/core/skbuff.c:4503)
>   [17716.240654] esp6_input (net/ipv6/esp6.c:977)
>   [17716.240666] xfrm_input (net/xfrm/xfrm_input.c:659)
>   [17716.240674] xfrm6_rcv (net/ipv6/xfrm6_input.c:172)
>   [...]
>
> Note that one could set CROSS_COMPILE=llvm- instead to hack around this
> issue. However, doing so can break the decodecode routine as it will
> force the selection of other LLVM utilities down the line e.g. llvm-as.
>
> [1] https://lore.kernel.org/all/20230914131225.13415-3-will@kernel.org/
>
> Cc: Will Deacon <will@kernel.org>
> Cc: John Stultz <jstultz@google.com>
> Cc: Masahiro Yamada <masahiroy@kernel.org>
> Signed-off-by: Carlos Llamas <cmllamas@google.com>

Thanks for the patch!
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>

> ---
>  scripts/decode_stacktrace.sh | 19 +++++++++++++++++--
>  1 file changed, 17 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/decode_stacktrace.sh b/scripts/decode_stacktrace.sh
> index 564c5632e1a2..bfe5a4082d8e 100755
> --- a/scripts/decode_stacktrace.sh
> +++ b/scripts/decode_stacktrace.sh
> @@ -16,6 +16,21 @@ elif type c++filt >/dev/null 2>&1 ; then
>         cppfilt_opts=-i
>  fi
>
> +UTIL_SUFFIX=
> +if [[ -z ${LLVM:-} ]]; then
> +       UTIL_PREFIX=${CROSS_COMPILE:-}
> +else
> +       UTIL_PREFIX=llvm-
> +       if [[ ${LLVM} == */ ]]; then
> +               UTIL_PREFIX=${LLVM}${UTIL_PREFIX}
> +       elif [[ ${LLVM} == -* ]]; then
> +               UTIL_SUFFIX=${LLVM}
> +       fi
> +fi
> +
> +READELF=${UTIL_PREFIX}readelf${UTIL_SUFFIX}
> +ADDR2LINE=${UTIL_PREFIX}addr2line${UTIL_SUFFIX}
> +
>  if [[ $1 == "-r" ]] ; then
>         vmlinux=""
>         basepath="auto"
> @@ -75,7 +90,7 @@ find_module() {
>
>         if [[ "$modpath" != "" ]] ; then
>                 for fn in $(find "$modpath" -name "${module//_/[-_]}.ko*") ; do
> -                       if readelf -WS "$fn" | grep -qwF .debug_line ; then
> +                       if ${READELF} -WS "$fn" | grep -qwF .debug_line ; then
>                                 echo $fn
>                                 return
>                         fi
> @@ -169,7 +184,7 @@ parse_symbol() {
>         if [[ $aarray_support == true && "${cache[$module,$address]+isset}" == "isset" ]]; then
>                 local code=${cache[$module,$address]}
>         else
> -               local code=$(${CROSS_COMPILE}addr2line -i -e "$objfile" "$address" 2>/dev/null)
> +               local code=$(${ADDR2LINE} -i -e "$objfile" "$address" 2>/dev/null)
>                 if [[ $aarray_support == true ]]; then
>                         cache[$module,$address]=$code
>                 fi
> --
> 2.42.0.582.g8ccd20d70d-goog
>


-- 
Thanks,
~Nick Desaulniers

Re: [PATCH] scripts/decode_stacktrace.sh: optionally use LLVM utilities

[Elliot Berman]

On 9/28/2023 8:48 PM, Carlos Llamas wrote:
> GNU's addr2line can have problems parsing a vmlinux built with LLVM,
> particularly when LTO was used. In order to decode the traces correctly
> this patch adds the ability to switch to LLVM's utilities readelf and
> addr2line. The same approach is followed by Will in [1].
> 
> Before:
>   $ scripts/decode_stacktrace.sh vmlinux < kernel.log
>   [17716.240635] Call trace:
>   [17716.240646] skb_cow_data (??:?)
>   [17716.240654] esp6_input (ld-temp.o:?)
>   [17716.240666] xfrm_input (ld-temp.o:?)
>   [17716.240674] xfrm6_rcv (??:?)
>   [...]
> 
> After:
>   $ LLVM=1 scripts/decode_stacktrace.sh vmlinux < kernel.log
>   [17716.240635] Call trace:
>   [17716.240646] skb_cow_data (include/linux/skbuff.h:2172 net/core/skbuff.c:4503)
>   [17716.240654] esp6_input (net/ipv6/esp6.c:977)
>   [17716.240666] xfrm_input (net/xfrm/xfrm_input.c:659)
>   [17716.240674] xfrm6_rcv (net/ipv6/xfrm6_input.c:172)
>   [...]
> 
> Note that one could set CROSS_COMPILE=llvm- instead to hack around this
> issue. However, doing so can break the decodecode routine as it will
> force the selection of other LLVM utilities down the line e.g. llvm-as.
> 
> [1] https://lore.kernel.org/all/20230914131225.13415-3-will@kernel.org/
> 
> Cc: Will Deacon <will@kernel.org>
> Cc: John Stultz <jstultz@google.com>
> Cc: Masahiro Yamada <masahiroy@kernel.org>
> Signed-off-by: Carlos Llamas <cmllamas@google.com>

Reviewed-by: Elliot Berman <quic_eberman@quicinc.com>

> ---
>  scripts/decode_stacktrace.sh | 19 +++++++++++++++++--
>  1 file changed, 17 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/decode_stacktrace.sh b/scripts/decode_stacktrace.sh
> index 564c5632e1a2..bfe5a4082d8e 100755
> --- a/scripts/decode_stacktrace.sh
> +++ b/scripts/decode_stacktrace.sh
> @@ -16,6 +16,21 @@ elif type c++filt >/dev/null 2>&1 ; then
>  	cppfilt_opts=-i
>  fi
>  
> +UTIL_SUFFIX=
> +if [[ -z ${LLVM:-} ]]; then
> +	UTIL_PREFIX=${CROSS_COMPILE:-}
> +else
> +	UTIL_PREFIX=llvm-
> +	if [[ ${LLVM} == */ ]]; then
> +		UTIL_PREFIX=${LLVM}${UTIL_PREFIX}
> +	elif [[ ${LLVM} == -* ]]; then
> +		UTIL_SUFFIX=${LLVM}
> +	fi
> +fi
> +
> +READELF=${UTIL_PREFIX}readelf${UTIL_SUFFIX}
> +ADDR2LINE=${UTIL_PREFIX}addr2line${UTIL_SUFFIX}
> +
>  if [[ $1 == "-r" ]] ; then
>  	vmlinux=""
>  	basepath="auto"
> @@ -75,7 +90,7 @@ find_module() {
>  
>  	if [[ "$modpath" != "" ]] ; then
>  		for fn in $(find "$modpath" -name "${module//_/[-_]}.ko*") ; do
> -			if readelf -WS "$fn" | grep -qwF .debug_line ; then
> +			if ${READELF} -WS "$fn" | grep -qwF .debug_line ; then
>  				echo $fn
>  				return
>  			fi
> @@ -169,7 +184,7 @@ parse_symbol() {
>  	if [[ $aarray_support == true && "${cache[$module,$address]+isset}" == "isset" ]]; then
>  		local code=${cache[$module,$address]}
>  	else
> -		local code=$(${CROSS_COMPILE}addr2line -i -e "$objfile" "$address" 2>/dev/null)
> +		local code=$(${ADDR2LINE} -i -e "$objfile" "$address" 2>/dev/null)
>  		if [[ $aarray_support == true ]]; then
>  			cache[$module,$address]=$code
>  		fi

Re: [PATCH] scripts/decode_stacktrace.sh: optionally use LLVM utilities

[Carlos Llamas]

On Mon, Dec 18, 2023 at 12:08:14PM -0800, Elliot Berman wrote:
> 
> Reviewed-by: Elliot Berman <quic_eberman@quicinc.com>
> 

Oh, thanks Elliot for having a look. I had forgotten about this patch.
I'll RESEND if this doesn't get picked up next week.

--
Carlos Llamas

Re: [PATCH] scripts/decode_stacktrace.sh: optionally use LLVM utilities

[Justin Stitt]

Hi,

On Fri, Sep 29, 2023 at 03:48:17AM +0000, Carlos Llamas wrote:
> GNU's addr2line can have problems parsing a vmlinux built with LLVM,
> particularly when LTO was used. In order to decode the traces correctly
> this patch adds the ability to switch to LLVM's utilities readelf and
> addr2line. The same approach is followed by Will in [1].
>
> Before:
>   $ scripts/decode_stacktrace.sh vmlinux < kernel.log
>   [17716.240635] Call trace:
>   [17716.240646] skb_cow_data (??:?)
>   [17716.240654] esp6_input (ld-temp.o:?)
>   [17716.240666] xfrm_input (ld-temp.o:?)
>   [17716.240674] xfrm6_rcv (??:?)
>   [...]

Perhaps it is due to where I am dumping the stack, but I am getting good
stack traces on mainline (with a LLVM-built kernel) without this patch.

With that being said, applying this patch still has good stack traces
and it makes sense if it helps others.

>
> After:
>   $ LLVM=1 scripts/decode_stacktrace.sh vmlinux < kernel.log
>   [17716.240635] Call trace:
>   [17716.240646] skb_cow_data (include/linux/skbuff.h:2172 net/core/skbuff.c:4503)
>   [17716.240654] esp6_input (net/ipv6/esp6.c:977)
>   [17716.240666] xfrm_input (net/xfrm/xfrm_input.c:659)
>   [17716.240674] xfrm6_rcv (net/ipv6/xfrm6_input.c:172)
>   [...]
>
> Note that one could set CROSS_COMPILE=llvm- instead to hack around this
> issue. However, doing so can break the decodecode routine as it will
> force the selection of other LLVM utilities down the line e.g. llvm-as.
>
> [1] https://lore.kernel.org/all/20230914131225.13415-3-will@kernel.org/
>
> Cc: Will Deacon <will@kernel.org>
> Cc: John Stultz <jstultz@google.com>
> Cc: Masahiro Yamada <masahiroy@kernel.org>
> Signed-off-by: Carlos Llamas <cmllamas@google.com>

Tested-by: Justin Stitt <justinstitt@google.com>

Note that this patch is similar: https://lore.kernel.org/all/20231215-llvm-decode-stacktrace-v1-1-201cb86f4879@quicinc.com/

> ---
>  scripts/decode_stacktrace.sh | 19 +++++++++++++++++--
>  1 file changed, 17 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/decode_stacktrace.sh b/scripts/decode_stacktrace.sh
> index 564c5632e1a2..bfe5a4082d8e 100755
> --- a/scripts/decode_stacktrace.sh
> +++ b/scripts/decode_stacktrace.sh
> @@ -16,6 +16,21 @@ elif type c++filt >/dev/null 2>&1 ; then
>  	cppfilt_opts=-i
>  fi
>
> +UTIL_SUFFIX=
> +if [[ -z ${LLVM:-} ]]; then
> +	UTIL_PREFIX=${CROSS_COMPILE:-}
> +else
> +	UTIL_PREFIX=llvm-
> +	if [[ ${LLVM} == */ ]]; then
> +		UTIL_PREFIX=${LLVM}${UTIL_PREFIX}
> +	elif [[ ${LLVM} == -* ]]; then
> +		UTIL_SUFFIX=${LLVM}
> +	fi
> +fi
> +
> +READELF=${UTIL_PREFIX}readelf${UTIL_SUFFIX}
> +ADDR2LINE=${UTIL_PREFIX}addr2line${UTIL_SUFFIX}
> +
>  if [[ $1 == "-r" ]] ; then
>  	vmlinux=""
>  	basepath="auto"
> @@ -75,7 +90,7 @@ find_module() {
>
>  	if [[ "$modpath" != "" ]] ; then
>  		for fn in $(find "$modpath" -name "${module//_/[-_]}.ko*") ; do
> -			if readelf -WS "$fn" | grep -qwF .debug_line ; then
> +			if ${READELF} -WS "$fn" | grep -qwF .debug_line ; then
>  				echo $fn
>  				return
>  			fi
> @@ -169,7 +184,7 @@ parse_symbol() {
>  	if [[ $aarray_support == true && "${cache[$module,$address]+isset}" == "isset" ]]; then
>  		local code=${cache[$module,$address]}
>  	else
> -		local code=$(${CROSS_COMPILE}addr2line -i -e "$objfile" "$address" 2>/dev/null)
> +		local code=$(${ADDR2LINE} -i -e "$objfile" "$address" 2>/dev/null)
>  		if [[ $aarray_support == true ]]; then
>  			cache[$module,$address]=$code
>  		fi
> --
> 2.42.0.582.g8ccd20d70d-goog
>

Thanks
Justin

Re: [PATCH] scripts/decode_stacktrace.sh: optionally use LLVM utilities

[Carlos Llamas]

On Tue, Dec 19, 2023 at 12:23:18AM +0000, Justin Stitt wrote:
> Hi,
> 
> On Fri, Sep 29, 2023 at 03:48:17AM +0000, Carlos Llamas wrote:
> > GNU's addr2line can have problems parsing a vmlinux built with LLVM,
> > particularly when LTO was used. In order to decode the traces correctly
> > this patch adds the ability to switch to LLVM's utilities readelf and
> > addr2line. The same approach is followed by Will in [1].
> >
> > Before:
> >   $ scripts/decode_stacktrace.sh vmlinux < kernel.log
> >   [17716.240635] Call trace:
> >   [17716.240646] skb_cow_data (??:?)
> >   [17716.240654] esp6_input (ld-temp.o:?)
> >   [17716.240666] xfrm_input (ld-temp.o:?)
> >   [17716.240674] xfrm6_rcv (??:?)
> >   [...]
> 
> Perhaps it is due to where I am dumping the stack, but I am getting good
> stack traces on mainline (with a LLVM-built kernel) without this patch.

Maybe you have set CROSS_COMPILE=llvm- in your environment? This is an
alternative workaround. I write more about this in the commit log.

> 
> With that being said, applying this patch still has good stack traces
> and it makes sense if it helps others.
> 
> >
> > After:
> >   $ LLVM=1 scripts/decode_stacktrace.sh vmlinux < kernel.log
> >   [17716.240635] Call trace:
> >   [17716.240646] skb_cow_data (include/linux/skbuff.h:2172 net/core/skbuff.c:4503)
> >   [17716.240654] esp6_input (net/ipv6/esp6.c:977)
> >   [17716.240666] xfrm_input (net/xfrm/xfrm_input.c:659)
> >   [17716.240674] xfrm6_rcv (net/ipv6/xfrm6_input.c:172)
> >   [...]
> >
> > Note that one could set CROSS_COMPILE=llvm- instead to hack around this
> > issue. However, doing so can break the decodecode routine as it will
> > force the selection of other LLVM utilities down the line e.g. llvm-as.
> >
> > [1] https://lore.kernel.org/all/20230914131225.13415-3-will@kernel.org/
> >
> > Cc: Will Deacon <will@kernel.org>
> > Cc: John Stultz <jstultz@google.com>
> > Cc: Masahiro Yamada <masahiroy@kernel.org>
> > Signed-off-by: Carlos Llamas <cmllamas@google.com>
> 
> Tested-by: Justin Stitt <justinstitt@google.com>
> 
> Note that this patch is similar: https://lore.kernel.org/all/20231215-llvm-decode-stacktrace-v1-1-201cb86f4879@quicinc.com/

Ha, I see. Elliot's patch help revive this thread. Cool!

--
Thanks,
Carlos Llamas

Re: [PATCH] scripts/decode_stacktrace.sh: optionally use LLVM utilities

[Carlos Llamas]

On Fri, Sep 29, 2023 at 03:48:17AM +0000, Carlos Llamas wrote:
> GNU's addr2line can have problems parsing a vmlinux built with LLVM,
> particularly when LTO was used. In order to decode the traces correctly
> this patch adds the ability to switch to LLVM's utilities readelf and
> addr2line. The same approach is followed by Will in [1].
> 
> Before:
>   $ scripts/decode_stacktrace.sh vmlinux < kernel.log
>   [17716.240635] Call trace:
>   [17716.240646] skb_cow_data (??:?)
>   [17716.240654] esp6_input (ld-temp.o:?)
>   [17716.240666] xfrm_input (ld-temp.o:?)
>   [17716.240674] xfrm6_rcv (??:?)
>   [...]
> 
> After:
>   $ LLVM=1 scripts/decode_stacktrace.sh vmlinux < kernel.log
>   [17716.240635] Call trace:
>   [17716.240646] skb_cow_data (include/linux/skbuff.h:2172 net/core/skbuff.c:4503)
>   [17716.240654] esp6_input (net/ipv6/esp6.c:977)
>   [17716.240666] xfrm_input (net/xfrm/xfrm_input.c:659)
>   [17716.240674] xfrm6_rcv (net/ipv6/xfrm6_input.c:172)
>   [...]
> 
> Note that one could set CROSS_COMPILE=llvm- instead to hack around this
> issue. However, doing so can break the decodecode routine as it will
> force the selection of other LLVM utilities down the line e.g. llvm-as.
> 
> [1] https://lore.kernel.org/all/20230914131225.13415-3-will@kernel.org/
> 
> Cc: Will Deacon <will@kernel.org>
> Cc: John Stultz <jstultz@google.com>
> Cc: Masahiro Yamada <masahiroy@kernel.org>
> Signed-off-by: Carlos Llamas <cmllamas@google.com>
> ---

There is no clear maintainer for this script. However, I had a look at
git history and it seems this might need to be picked up by Andrew.

Cc: Andrew Morton <akpm@linux-foundation.org>

Andrew can you please pick this up?

--
Carlos Llamas