Re: [syzbot] [arm?] upstream test error: KASAN: invalid-access Write in setup_arch

Andrey Konovalov
2024-09-23T20:14:14Z

On Mon, Sep 23, 2024 at 12:46 PM Mark Rutland <mark.rutland@arm.com> wrote:
>
> > > > ==================================================================
> > > > BUG: KASAN: invalid-access in smp_build_mpidr_hash arch/arm64/kernel/setup.c:133 [inline]
> > > > BUG: KASAN: invalid-access in setup_arch+0x984/0xd60 arch/arm64/kernel/setup.c:356
> > > > Write of size 4 at addr 03ff800086867e00 by task swapper/0
> > > > Pointer tag: [03], memory tag: [fe]
> > > >
> > > > CPU: 0 UID: 0 PID: 0 Comm: swapper Not tainted 6.11.0-rc5-syzkaller-g33faa93bc856 #0
> > > > Hardware name: linux,dummy-virt (DT)
> > > > Call trace:
> > > > dump_backtrace+0x204/0x3b8 arch/arm64/kernel/stacktrace.c:317
> > > > show_stack+0x2c/0x3c arch/arm64/kernel/stacktrace.c:324
> > > > __dump_stack lib/dump_stack.c:93 [inline]
> > > > dump_stack_lvl+0x260/0x3b4 lib/dump_stack.c:119
> > > > print_address_description mm/kasan/report.c:377 [inline]
> > > > print_report+0x118/0x5ac mm/kasan/report.c:488
> > > > kasan_report+0xc8/0x108 mm/kasan/report.c:601
> > > > kasan_check_range+0x94/0xb8 mm/kasan/sw_tags.c:84
> > > > __hwasan_store4_noabort+0x20/0x2c mm/kasan/sw_tags.c:149
> > > > smp_build_mpidr_hash arch/arm64/kernel/setup.c:133 [inline]
> > > > setup_arch+0x984/0xd60 arch/arm64/kernel/setup.c:356
> > > > start_kernel+0xe0/0xff0 init/main.c:926
> > > > __primary_switched+0x84/0x8c arch/arm64/kernel/head.S:243
> > > >
> > > > The buggy address belongs to stack of task swapper/0
> > > >
> > > > Memory state around the buggy address:
> > > > ffff800086867c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > > > ffff800086867d00: 00 fe fe 00 00 00 fe fe fe fe fe fe fe fe fe fe
> > > > >ffff800086867e00: fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe
> > > > ^
> > > > ffff800086867f00: fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe
> > > > ffff800086868000: fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe
> > > > ==================================================================
> > >
> > > I can't spot the issue here. We have a couple of fixed-length
> > > (4 element) arrays on the stack and they're indexed by a simple loop
> > > counter that runs from 0-3.
> >
> > Having trimmed the config to the extreme, I can only trigger the
> > warning with CONFIG_KASAN_SW_TAGS (CONFIG_KASAN_GENERIC does not
> > scream). Same thing if I use gcc 14.2.0.
> >
> > However, compiling with clang 14 (Debian clang version 14.0.6) does
> > *not* result in a screaming kernel, even with KASAN_SW_TAGS.

Yeah, this is #1 from https://bugzilla.kernel.org/show_bug.cgi?id=218854.

> > So I can see two possibilities here:
> >
> > - either gcc is incompatible with KASAN_SW_TAGS and the generic
> > version is the only one that works
> >
> > - or we have a compiler bug on our hands.
> >
> > Frankly, I can't believe the later, as the code is so daft that I
> > can't imagine gcc getting it *that* wrong.
>
> It looks like what's happening here is:
>
> (1) With CONFIG_KASAN_SW_TAGS=y we pass the compiler
> `-fsanitize=kernel-hwaddress`.
>
> (2) When GCC is passed `-fsanitize=hwaddress` or
> `-fsanitize=kernel-hwaddress` it ignores
> `__attribute__((no_sanitize_address))`, and instruments functions we
> require are not instrumented.
>
> I believe this is a compiler bug, as there doesn't seem to be a
> separate attribute to prevent instrumentation in this mode.
>
> (3) In this config, smp_build_mpidr_hash() gets inlined into
> setup_arch(), and as setup_arch() is instrumented, all of the stack
> variables for smp_build_mpidr_hash() are initialized at the start of
> setup_arch(), with calls to __hwasan_tag_memory().
>
> At this point, we are using the early shadow (where a single page of
> shadow is used for all memory).
>
> (4) In setup_arch(), we call kasan_init() to transition from the early
> shadow to the runtime shadow. This replaces the early shadow memory
> with new shadow memory initialized to KASAN_SHADOW_INIT (0xFE AKA
> KASAN_TAG_INVALID), including the shadow for the stack.
>
> (5) Once the CPU returns back into setup_arch(), it's using the new
> shadow initialized to 0xFE. Subsequent stack accesses which check
> the shadow see 0xFE in the shadow, and fault. Note that in the dump
> of the shadow above, the shadow around ffff800086867d80 and above is
> all 0xFE, while below that functions have managed to clear the
> shadow.
>
> Compiler test case below. Note that this demonstrates the compiler
> ignores `__attribute__((no_sanitize_address))` regardless of
> KASAN_STACK, so KASAN_SW_TAGS is generally broken with GCC. All versions
> I tried were broken, from 11.3.0 to 14.2.0 inclusive.

Thank you for the detailed investigation report!

> I think we have to disable KASAN_SW_TAGS with GCC until this is fixed.

Sounds good to me.

Please reference https://bugzilla.kernel.org/show_bug.cgi?id=218854 if
you end up sending a patch for this.

Also the syzbot's kvm instance should probably be switched to Clang
(@Alexander).

Thank you!