[PATCH] ci: display UBSan stack trace

[PATCH] ci: display UBSan stack trace

[David Marchand]

When UBSan raises an error, we get really few context.

Example, on a recently submitted patch:

RTE>>cksum_fuzz_autotest
../lib/net/rte_cksum.h:49:10: runtime error: load of misaligned address
	0x0001816c2e81 for type 'const unaligned_uint16_t' (aka 'const
	unsigned short'), which requires 2 byte alignment
0x0001816c2e81: note: pointer points here
 00 00 00  00 70 f2 00 00 00 00 00  00 00 00 00 00 00 00 00
              ^
	00 00 00 00 00 00 00 00  00 00 00 00 00
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior
	../lib/net/rte_cksum.h:49:10 in

Ask for the full stack.

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 .ci/linux-build.sh | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)

diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
index 084d9642fc..091d239fae 100755
--- a/.ci/linux-build.sh
+++ b/.ci/linux-build.sh
@@ -7,7 +7,11 @@ if [ -z "${DEF_LIB:-}" ]; then
 fi
 
 # Builds are run as root in containers, no need for sudo
-[ "$(id -u)" != '0' ] || alias sudo=
+if [ "$(id -u)" = '0' ]; then
+    run_as_root=""
+else
+    run_as_root="sudo -E"
+fi
 
 install_libabigail() {
     version=$1
@@ -28,15 +32,15 @@ configure_coredump() {
     # No point in configuring coredump without gdb
     which gdb >/dev/null || return 0
     ulimit -c unlimited
-    sudo sysctl -w kernel.core_pattern=/tmp/dpdk-core.%e.%p
+    $run_as_root sysctl -w kernel.core_pattern=/tmp/dpdk-core.%e.%p
 }
 
 catch_coredump() {
     ls /tmp/dpdk-core.*.* 2>/dev/null || return 0
     for core in /tmp/dpdk-core.*.*; do
-        binary=$(sudo readelf -n $core |grep $(pwd)/build/ 2>/dev/null |head -n1)
+        binary=$($run_as_root readelf -n $core |grep $(pwd)/build/ 2>/dev/null |head -n1)
         [ -x $binary ] || binary=
-        sudo gdb $binary -c $core \
+        $run_as_root gdb $binary -c $core \
             -ex 'info threads' \
             -ex 'thread apply all bt full' \
             -ex 'quit'
@@ -53,9 +57,9 @@ catch_ubsan() {
 
 check_traces() {
     which babeltrace >/dev/null || return 0
-    for file in $(sudo find $HOME -name metadata); do
-        ! sudo babeltrace $(dirname $file) >/dev/null 2>&1 || continue
-        sudo babeltrace $(dirname $file)
+    for file in $($run_as_root find $HOME -name metadata); do
+        ! $run_as_root babeltrace $(dirname $file) >/dev/null 2>&1 || continue
+        $run_as_root babeltrace $(dirname $file)
     done
 }
 
@@ -136,6 +140,7 @@ fi
 
 if [ "$UBSAN" = "true" ]; then
     sanitizer=${sanitizer:+$sanitizer,}undefined
+    export UBSAN_OPTIONS=print_stacktrace=1
     if [ "$RUN_TESTS" = "true" ]; then
         # UBSan takes too much memory with -O2
         buildtype=plain
@@ -218,7 +223,8 @@ fi
 if [ "$RUN_TESTS" = "true" ]; then
     failed=
     configure_coredump
-    sudo meson test -C build --suite fast-tests -t 3 --no-stdsplit --print-errorlogs || failed="true"
+    $run_as_root meson test -C build --suite fast-tests -t 3 --no-stdsplit --print-errorlogs ||
+        failed="true"
     catch_coredump
     catch_ubsan DPDK:fast-tests build/meson-logs/testlog.txt
     check_traces
-- 
2.52.0

RE: [PATCH] ci: display UBSan stack trace

[Marat Khalili]

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Wednesday 28 January 2026 11:05
> To: dev@dpdk.org
> Cc: scott.k.mitch1@gmail.com; Aaron Conole <aconole@redhat.com>
> Subject: [PATCH] ci: display UBSan stack trace
> 
> When UBSan raises an error, we get really few context.
> 
> Example, on a recently submitted patch:
> 
> RTE>>cksum_fuzz_autotest
> ../lib/net/rte_cksum.h:49:10: runtime error: load of misaligned address
> 	0x0001816c2e81 for type 'const unaligned_uint16_t' (aka 'const
> 	unsigned short'), which requires 2 byte alignment
> 0x0001816c2e81: note: pointer points here
>  00 00 00  00 70 f2 00 00 00 00 00  00 00 00 00 00 00 00 00
>               ^
> 	00 00 00 00 00 00 00 00  00 00 00 00 00
> SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior
> 	../lib/net/rte_cksum.h:49:10 in
> 
> Ask for the full stack.
> 
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
>  .ci/linux-build.sh | 22 ++++++++++++++--------
>  1 file changed, 14 insertions(+), 8 deletions(-)
> 
> diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
> index 084d9642fc..091d239fae 100755
> --- a/.ci/linux-build.sh
> +++ b/.ci/linux-build.sh
> @@ -7,7 +7,11 @@ if [ -z "${DEF_LIB:-}" ]; then
>  fi
> 
>  # Builds are run as root in containers, no need for sudo
> -[ "$(id -u)" != '0' ] || alias sudo=
> +if [ "$(id -u)" = '0' ]; then
> +    run_as_root=""
> +else
> +    run_as_root="sudo -E"

Just out of general principles, can we have --preserve-env=UBSAN_OPTIONS (or
necessary list) instead?

And/or, do we need to preserve environment for all commands below?

> +fi
> 
>  install_libabigail() {
>      version=$1
> @@ -28,15 +32,15 @@ configure_coredump() {
>      # No point in configuring coredump without gdb
>      which gdb >/dev/null || return 0
>      ulimit -c unlimited
> -    sudo sysctl -w kernel.core_pattern=/tmp/dpdk-core.%e.%p
> +    $run_as_root sysctl -w kernel.core_pattern=/tmp/dpdk-core.%e.%p
>  }
> 
>  catch_coredump() {
>      ls /tmp/dpdk-core.*.* 2>/dev/null || return 0
>      for core in /tmp/dpdk-core.*.*; do
> -        binary=$(sudo readelf -n $core |grep $(pwd)/build/ 2>/dev/null |head -n1)
> +        binary=$($run_as_root readelf -n $core |grep $(pwd)/build/ 2>/dev/null |head -n1)
>          [ -x $binary ] || binary=
> -        sudo gdb $binary -c $core \
> +        $run_as_root gdb $binary -c $core \
>              -ex 'info threads' \
>              -ex 'thread apply all bt full' \
>              -ex 'quit'
> @@ -53,9 +57,9 @@ catch_ubsan() {
> 
>  check_traces() {
>      which babeltrace >/dev/null || return 0
> -    for file in $(sudo find $HOME -name metadata); do
> -        ! sudo babeltrace $(dirname $file) >/dev/null 2>&1 || continue
> -        sudo babeltrace $(dirname $file)
> +    for file in $($run_as_root find $HOME -name metadata); do
> +        ! $run_as_root babeltrace $(dirname $file) >/dev/null 2>&1 || continue
> +        $run_as_root babeltrace $(dirname $file)
>      done
>  }
> 
> @@ -136,6 +140,7 @@ fi
> 
>  if [ "$UBSAN" = "true" ]; then
>      sanitizer=${sanitizer:+$sanitizer,}undefined
> +    export UBSAN_OPTIONS=print_stacktrace=1

Won't it replace options meson sets by default? I actually see
print_stacktrace=1 among them, although maybe depends on version and
environment.

Also, do we care only about UBSAN_OPTIONS, or also ASAN_OPTIONS and
LSAN_OPTIONS here?

I will share ones we are using in case you find some of them interesting:

ASAN_OPTIONS=abort_on_error=true:color=never:halt_on_error=true:print_summary=true:strict_string_checks=true:check_initialization_order=true:detect_stack_use_after_return=true:heap_profile=false:print_scariness=true:strict_init_order=true:verify_asan_link_order=false
LSAN_OPTIONS=abort_on_error=true:color=never:halt_on_error=true:print_summary=true:strict_string_checks=true
UBSAN_OPTIONS=abort_on_error=true:color=never:halt_on_error=true:print_summary=true:strict_string_checks=true:print_stacktrace=true:report_error_type=true

(Admittedly some of them might be unnecessary or nonsensical, they depend on
compiler and it's not easy to find documentation.)

>      if [ "$RUN_TESTS" = "true" ]; then
>          # UBSan takes too much memory with -O2
>          buildtype=plain
> @@ -218,7 +223,8 @@ fi
>  if [ "$RUN_TESTS" = "true" ]; then
>      failed=
>      configure_coredump
> -    sudo meson test -C build --suite fast-tests -t 3 --no-stdsplit --print-errorlogs || failed="true"
> +    $run_as_root meson test -C build --suite fast-tests -t 3 --no-stdsplit --print-errorlogs ||
> +        failed="true"
>      catch_coredump
>      catch_ubsan DPDK:fast-tests build/meson-logs/testlog.txt
>      check_traces
> --
> 2.52.0

With or without comments above addressed,

Acked-by: Marat Khalili <marat.khalili@huawei.com>

Re: [PATCH] ci: display UBSan stack trace

[David Marchand]

Hello,

On Wed, 28 Jan 2026 at 12:55, Marat Khalili <marat.khalili@huawei.com> wrote:
> > Subject: [PATCH] ci: display UBSan stack trace
> >
> > When UBSan raises an error, we get really few context.
> >
> > Example, on a recently submitted patch:
> >
> > RTE>>cksum_fuzz_autotest
> > ../lib/net/rte_cksum.h:49:10: runtime error: load of misaligned address
> >       0x0001816c2e81 for type 'const unaligned_uint16_t' (aka 'const
> >       unsigned short'), which requires 2 byte alignment
> > 0x0001816c2e81: note: pointer points here
> >  00 00 00  00 70 f2 00 00 00 00 00  00 00 00 00 00 00 00 00
> >               ^
> >       00 00 00 00 00 00 00 00  00 00 00 00 00
> > SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior
> >       ../lib/net/rte_cksum.h:49:10 in
> >
> > Ask for the full stack.
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
> >  .ci/linux-build.sh | 22 ++++++++++++++--------
> >  1 file changed, 14 insertions(+), 8 deletions(-)
> >
> > diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh
> > index 084d9642fc..091d239fae 100755
> > --- a/.ci/linux-build.sh
> > +++ b/.ci/linux-build.sh
> > @@ -7,7 +7,11 @@ if [ -z "${DEF_LIB:-}" ]; then
> >  fi
> >
> >  # Builds are run as root in containers, no need for sudo
> > -[ "$(id -u)" != '0' ] || alias sudo=
> > +if [ "$(id -u)" = '0' ]; then
> > +    run_as_root=""
> > +else
> > +    run_as_root="sudo -E"
>
> Just out of general principles, can we have --preserve-env=UBSAN_OPTIONS (or
> necessary list) instead?
>
> And/or, do we need to preserve environment for all commands below?

We could, but I wanted to keep the change simple.

In this patch, a single point in this script is used to trigger use of
sudo or not.
Otherwise, I would have to keep two separate variable/alias for
calling simple sudo, and running the unit tests.

And like this, there is no question about which variable must be preserved.


>
> > +fi
> >
> >  install_libabigail() {
> >      version=$1
> > @@ -28,15 +32,15 @@ configure_coredump() {
> >      # No point in configuring coredump without gdb
> >      which gdb >/dev/null || return 0
> >      ulimit -c unlimited
> > -    sudo sysctl -w kernel.core_pattern=/tmp/dpdk-core.%e.%p
> > +    $run_as_root sysctl -w kernel.core_pattern=/tmp/dpdk-core.%e.%p
> >  }
> >
> >  catch_coredump() {
> >      ls /tmp/dpdk-core.*.* 2>/dev/null || return 0
> >      for core in /tmp/dpdk-core.*.*; do
> > -        binary=$(sudo readelf -n $core |grep $(pwd)/build/ 2>/dev/null |head -n1)
> > +        binary=$($run_as_root readelf -n $core |grep $(pwd)/build/ 2>/dev/null |head -n1)
> >          [ -x $binary ] || binary=
> > -        sudo gdb $binary -c $core \
> > +        $run_as_root gdb $binary -c $core \
> >              -ex 'info threads' \
> >              -ex 'thread apply all bt full' \
> >              -ex 'quit'
> > @@ -53,9 +57,9 @@ catch_ubsan() {
> >
> >  check_traces() {
> >      which babeltrace >/dev/null || return 0
> > -    for file in $(sudo find $HOME -name metadata); do
> > -        ! sudo babeltrace $(dirname $file) >/dev/null 2>&1 || continue
> > -        sudo babeltrace $(dirname $file)
> > +    for file in $($run_as_root find $HOME -name metadata); do
> > +        ! $run_as_root babeltrace $(dirname $file) >/dev/null 2>&1 || continue
> > +        $run_as_root babeltrace $(dirname $file)
> >      done
> >  }
> >
> > @@ -136,6 +140,7 @@ fi
> >
> >  if [ "$UBSAN" = "true" ]; then
> >      sanitizer=${sanitizer:+$sanitizer,}undefined
> > +    export UBSAN_OPTIONS=print_stacktrace=1
>
> Won't it replace options meson sets by default? I actually see
> print_stacktrace=1 among them, although maybe depends on version and
> environment.

This change may overwrite anything that meson sets, but on the other
hand, we need it with the current meson version used in the CI.


>
> Also, do we care only about UBSAN_OPTIONS, or also ASAN_OPTIONS and
> LSAN_OPTIONS here?
>
> I will share ones we are using in case you find some of them interesting:
>
> ASAN_OPTIONS=abort_on_error=true:color=never:halt_on_error=true:print_summary=true:strict_string_checks=true:check_initialization_order=true:detect_stack_use_after_return=true:heap_profile=false:print_scariness=true:strict_init_order=true:verify_asan_link_order=false
> LSAN_OPTIONS=abort_on_error=true:color=never:halt_on_error=true:print_summary=true:strict_string_checks=true
> UBSAN_OPTIONS=abort_on_error=true:color=never:halt_on_error=true:print_summary=true:strict_string_checks=true:print_stacktrace=true:report_error_type=true
>
> (Admittedly some of them might be unnecessary or nonsensical, they depend on
> compiler and it's not easy to find documentation.)

Did those options help with understanding failures in unit tests?
I see "abort" or "halt" options in this list, does it stop execution
of all the tests?


Overall, the current script does everything, from detecting it needs
to run parts with sudo, to configuring ubsan stuff (for example here).
Maybe it could be interesting to better split the scripts, isolate
what needs special permissions, but that's a different topic.


-- 
David Marchand

RE: [PATCH] ci: display UBSan stack trace

[Marat Khalili]

> > I will share ones we are using in case you find some of them interesting:
> >
> >
> ASAN_OPTIONS=abort_on_error=true:color=never:halt_on_error=true:print_summary=true:strict_string_check
> s=true:check_initialization_order=true:detect_stack_use_after_return=true:heap_profile=false:print_sca
> riness=true:strict_init_order=true:verify_asan_link_order=false
> >
> LSAN_OPTIONS=abort_on_error=true:color=never:halt_on_error=true:print_summary=true:strict_string_check
> s=true
> >
> UBSAN_OPTIONS=abort_on_error=true:color=never:halt_on_error=true:print_summary=true:strict_string_chec
> ks=true:print_stacktrace=true:report_error_type=true
> >
> > (Admittedly some of them might be unnecessary or nonsensical, they depend on
> > compiler and it's not easy to find documentation.)
> 
> Did those options help with understanding failures in unit tests?

print_stacktrace is probably the most important for understanding failures.
Some of the rest detect additional problems, others are mostly nice-to-have
cosmetics (e.g. disabling ANSI sequences).

heap_profile was found to be useful for debugging but not for everyday tests
since it may cause process to hang on exit (may depend on other options).

verify_asan_link_order=false is generally not needed, it disables a self-check.

report_error_type is very useful for adding suppressions.

> I see "abort" or "halt" options in this list, does it stop execution
> of all the tests?

No, they only affects dpdk-test process, and meson runs each test in a separate
process here, so continues running other tests. Both of them get set by my
version of meson itself BTW. If set to false sanitizer complains to terminal
but the test may actually pass (depending on specific problem).