[PATCH v1] LoongArch: Fix __smp_mb__{before,after}_atomic()

[PATCH v1] LoongArch: Fix __smp_mb__{before,after}_atomic()

[Tiezhu Yang]

When testing the BPF selftest "sudo ./test_progs -t timer_lockup", there
is a kernel lockup and panic:

  watchdog: BUG: soft lockup - CPU#1 stuck for 8s! [test_progs:39601]
  Kernel panic - not syncing: softlockup: hung tasks
  ...
  Call Trace:
  [<9000000000c6d2a4>] show_stack+0xf4/0x1c0
  [<9000000000c66ad4>] dump_stack_lvl+0x84/0xc8
  [<9000000000c40c10>] vpanic+0x278/0x4bc
  [<9000000000c40e98>] panic+0x44/0x48
  [<9000000000e8b000>] watchdog_timer_fn+0x500/0x520
  [<9000000000dfd9a4>] __hrtimer_run_queues+0xc4/0x530
  [<9000000000dffe90>] hrtimer_interrupt+0x140/0x320
  [<9000000000c701e4>] constant_timer_interrupt+0x34/0x50
  [<9000000000d956b8>] __handle_irq_event_percpu+0xe8/0x560
  [<9000000000d95b4c>] handle_irq_event_percpu+0x1c/0x80
  [<9000000000d9e4c4>] handle_percpu_irq+0x74/0xd0
  [<9000000000d945d4>] generic_handle_domain_irq+0x34/0x90
  [<9000000001c4ee84>] handle_cpu_irq+0x64/0xa0
  [<9000000002ac0fa8>] handle_loongarch_irq+0x28/0x40
  [<9000000002ac103c>] do_vint+0x7c/0xf0
  [<9000000002ad9e4c>] _raw_spin_unlock_irqrestore+0x8c/0xc0
  [<9000000000dfe9e0>] hrtimer_try_to_cancel.part.0+0x70/0x350
  [<9000000000dfed58>] hrtimer_cancel+0x38/0x80
  [<9000000000f6d944>] bpf_timer_cancel+0x94/0x1e0
  [<ffff80000200fad0>] bpf_prog_108ab87b32f22e44_timer_cb1+0xb0/0xfc
  [<9000000000f6b838>] bpf_timer_cb+0x98/0x170
  [<9000000000dfdaac>] __hrtimer_run_queues+0x1cc/0x530
  [<9000000000dfde94>] hrtimer_run_softirq+0x84/0xd0
  [<9000000000caa584>] handle_softirqs+0x154/0x7a0
  [<9000000000cab0a4>] do_softirq+0xb4/0x120
  [<9000000000cab27c>] __local_bh_enable_ip+0x16c/0x1c0
  [<900000000271d9e0>] bpf_test_run+0x1c0/0x5c0
  [<900000000271f548>] bpf_prog_test_run_skb+0x6e8/0xe20
  [<9000000000f39940>] __sys_bpf+0x1690/0x2c50
  [<9000000000f3af28>] sys_bpf+0x28/0x40
  [<9000000002ac2d68>] do_syscall+0x108/0x5e0
  [<9000000000c6a850>] handle_syscall+0xd0/0x170

In bpf_timer_cancel() of kernel/bpf/helpers.c, it explicitly notes that
"Need full barrier after relaxed atomic_inc" to expect a full hardware
barrier to ensure global visibility, but there is only an empty barrier
for smp_mb__after_atomic().

As LoongArch is a weakly-ordered architecture, without a data barrier,
the physical ordering of "store-before-load" cannot be guaranteed. As
a result, a subsequent read can bypass a prior relaxed atomic write
during lockless dependency checks. This allows both CPUs to experience
store-load reordering and simultaneously bypass the lockless deadlock
detection in the software, leading them to proceed into hrtimer_cancel()
and trigger a severe ABBA deadlock in the BPF core during the concurrent
test runs, resulting in a kernel panic.

Upgrade __smp_mb__after_atomic() to __smp_mb(), which generates the data
barrier instruction DBAR to provide full barrier after relaxed atomic_inc.
While at it, also upgrade __smp_mb__before_atomic() to __smp_mb().

With this patch, the lockless "store-before-load" ordering is enforced by
the DBAR instruction. The BPF timer_lockup selftest was stressed for 5000
consecutive loops on a physical LoongArch machine without encountering any
further lockups or warnings:

  for i in {1..5000}; do sudo ./test_progs -t timer_lockup; done

Reported-by: Vincent Li <vincent.mc.li@gmail.com>
Closes: https://lore.kernel.org/loongarch/CAK3+h2xOSEZUHhou7N2cRL-aGrZCNSm45g+P7thObMe+fpgYCA@mail.gmail.com/
Fixes: 46859ac8af52 ("LoongArch: Add multi-processor (SMP) support")

Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
---
 arch/loongarch/include/asm/barrier.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/loongarch/include/asm/barrier.h b/arch/loongarch/include/asm/barrier.h
index 4b663f197706..adfe343dfa65 100644
--- a/arch/loongarch/include/asm/barrier.h
+++ b/arch/loongarch/include/asm/barrier.h
@@ -57,8 +57,8 @@
 #define __WEAK_LLSC_MB		"			\n"
 #endif
 
-#define __smp_mb__before_atomic()	barrier()
-#define __smp_mb__after_atomic()	barrier()
+#define __smp_mb__before_atomic()	__smp_mb()
+#define __smp_mb__after_atomic()	__smp_mb()
 
 /**
  * array_index_mask_nospec() - generate a ~0 mask when index < size, 0 otherwise
-- 
2.42.0

Re: [PATCH v1] LoongArch: Fix __smp_mb__{before,after}_atomic()

[Tiezhu Yang]

On 2026/8/17 上午11:59, Tiezhu Yang wrote:
> When testing the BPF selftest "sudo ./test_progs -t timer_lockup", there
> is a kernel lockup and panic:

...

> With this patch, the lockless "store-before-load" ordering is enforced by
> the DBAR instruction. The BPF timer_lockup selftest was stressed for 5000
> consecutive loops on a physical LoongArch machine without encountering any
> further lockups or warnings:
> 
>    for i in {1..5000}; do sudo ./test_progs -t timer_lockup; done

...

> diff --git a/arch/loongarch/include/asm/barrier.h b/arch/loongarch/include/asm/barrier.h
> index 4b663f197706..adfe343dfa65 100644
> --- a/arch/loongarch/include/asm/barrier.h
> +++ b/arch/loongarch/include/asm/barrier.h
> @@ -57,8 +57,8 @@
>   #define __WEAK_LLSC_MB		"			\n"
>   #endif
>   
> -#define __smp_mb__before_atomic()	barrier()
> -#define __smp_mb__after_atomic()	barrier()
> +#define __smp_mb__before_atomic()	__smp_mb()
> +#define __smp_mb__after_atomic()	__smp_mb()
>   
>   /**
>    * array_index_mask_nospec() - generate a ~0 mask when index < size, 0 otherwise

Hi all,

To address the soft lockup while avoiding the performance overhead of
executing two separate instructions, I think there is a more elegant
way to fix this directly in the BPF core helper:

We can replace atomic_inc() and smp_mb__after_atomic() with a single
atomic_fetch_add() in bpf_timer_cancel(). This consolidates the logic
into a single native atomic operation with full ordering, which not
only guarantees the store-load order to eliminate the deadlock but
also improves the performance for weak memory model architectures.

Any thoughts on this approach?

```
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index c18f1e16edee..ee2b3a4dcc05 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -1591,9 +1591,7 @@ BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern 
*, async)
          */
         if (!cur_t)
                 goto drop;
-       atomic_inc(&t->cancelling);
-       /* Need full barrier after relaxed atomic_inc */
-       smp_mb__after_atomic();
+       atomic_fetch_add(1, &t->cancelling);
         inc = true;
         if (atomic_read(&cur_t->cancelling)) {
                 /* We're cancelling timer t, while some other timer 
callback is
```

I tested the above diff, the BPF timer_lockup selftest was stressed
for 5000 consecutive loops on a physical LoongArch machine without
encountering any further lockups or warnings.

If you are OK with this change of bpf code, I will send a bpf patch
later.

Thanks,
Tiezhu

Re: [PATCH v1] LoongArch: Fix __smp_mb__{before,after}_atomic()

[Kumar Kartikeya Dwivedi]

On Tue Aug 18, 2026 at 3:28 PM CEST, Tiezhu Yang wrote:
> On 2026/8/17 上午11:59, Tiezhu Yang wrote:
>> When testing the BPF selftest "sudo ./test_progs -t timer_lockup", there
>> is a kernel lockup and panic:
>
> ...
>
>> With this patch, the lockless "store-before-load" ordering is enforced by
>> the DBAR instruction. The BPF timer_lockup selftest was stressed for 5000
>> consecutive loops on a physical LoongArch machine without encountering any
>> further lockups or warnings:
>>
>>    for i in {1..5000}; do sudo ./test_progs -t timer_lockup; done
>
> ...
>
>> diff --git a/arch/loongarch/include/asm/barrier.h b/arch/loongarch/include/asm/barrier.h
>> index 4b663f197706..adfe343dfa65 100644
>> --- a/arch/loongarch/include/asm/barrier.h
>> +++ b/arch/loongarch/include/asm/barrier.h
>> @@ -57,8 +57,8 @@
>>   #define __WEAK_LLSC_MB		"			\n"
>>   #endif
>>
>> -#define __smp_mb__before_atomic()	barrier()
>> -#define __smp_mb__after_atomic()	barrier()
>> +#define __smp_mb__before_atomic()	__smp_mb()
>> +#define __smp_mb__after_atomic()	__smp_mb()
>>
>>   /**
>>    * array_index_mask_nospec() - generate a ~0 mask when index < size, 0 otherwise
>
> Hi all,
>
> To address the soft lockup while avoiding the performance overhead of
> executing two separate instructions, I think there is a more elegant
> way to fix this directly in the BPF core helper:
>
> We can replace atomic_inc() and smp_mb__after_atomic() with a single
> atomic_fetch_add() in bpf_timer_cancel(). This consolidates the logic
> into a single native atomic operation with full ordering, which not
> only guarantees the store-load order to eliminate the deadlock but
> also improves the performance for weak memory model architectures.
>
> Any thoughts on this approach?
>
> ```
> diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
> index c18f1e16edee..ee2b3a4dcc05 100644
> --- a/kernel/bpf/helpers.c
> +++ b/kernel/bpf/helpers.c
> @@ -1591,9 +1591,7 @@ BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern
> *, async)
>           */
>          if (!cur_t)
>                  goto drop;
> -       atomic_inc(&t->cancelling);
> -       /* Need full barrier after relaxed atomic_inc */
> -       smp_mb__after_atomic();
> +       atomic_fetch_add(1, &t->cancelling);
>          inc = true;
>          if (atomic_read(&cur_t->cancelling)) {
>                  /* We're cancelling timer t, while some other timer
> callback is
> ```
>
> I tested the above diff, the BPF timer_lockup selftest was stressed
> for 5000 consecutive loops on a physical LoongArch machine without
> encountering any further lockups or warnings.

There is a full barrier in both cases. I don't know what performance improvement
will be achieved by changing this. Don't you need to fix the lowering for
smp_mb__after_atomic() anyway? It's used in several other places in the kernel.

>
> If you are OK with this change of bpf code, I will send a bpf patch
> later.
>
> Thanks,
> Tiezhu

Re: [PATCH v1] LoongArch: Fix __smp_mb__{before,after}_atomic()

[Tiezhu Yang]

On 2026/8/18 下午9:35, Kumar Kartikeya Dwivedi wrote:
> On Tue Aug 18, 2026 at 3:28 PM CEST, Tiezhu Yang wrote:
>> On 2026/8/17 上午11:59, Tiezhu Yang wrote:
>>> When testing the BPF selftest "sudo ./test_progs -t timer_lockup", there
>>> is a kernel lockup and panic:
>>
>> ...
>>
>>> With this patch, the lockless "store-before-load" ordering is enforced by
>>> the DBAR instruction. The BPF timer_lockup selftest was stressed for 5000
>>> consecutive loops on a physical LoongArch machine without encountering any
>>> further lockups or warnings:
>>>
>>>     for i in {1..5000}; do sudo ./test_progs -t timer_lockup; done
>>
>> ...
>>
>>> diff --git a/arch/loongarch/include/asm/barrier.h b/arch/loongarch/include/asm/barrier.h
>>> index 4b663f197706..adfe343dfa65 100644
>>> --- a/arch/loongarch/include/asm/barrier.h
>>> +++ b/arch/loongarch/include/asm/barrier.h
>>> @@ -57,8 +57,8 @@
>>>    #define __WEAK_LLSC_MB		"			\n"
>>>    #endif
>>>
>>> -#define __smp_mb__before_atomic()	barrier()
>>> -#define __smp_mb__after_atomic()	barrier()
>>> +#define __smp_mb__before_atomic()	__smp_mb()
>>> +#define __smp_mb__after_atomic()	__smp_mb()
>>>
>>>    /**
>>>     * array_index_mask_nospec() - generate a ~0 mask when index < size, 0 otherwise
>>
>> Hi all,
>>
>> To address the soft lockup while avoiding the performance overhead of
>> executing two separate instructions, I think there is a more elegant
>> way to fix this directly in the BPF core helper:
>>
>> We can replace atomic_inc() and smp_mb__after_atomic() with a single
>> atomic_fetch_add() in bpf_timer_cancel(). This consolidates the logic
>> into a single native atomic operation with full ordering, which not
>> only guarantees the store-load order to eliminate the deadlock but
>> also improves the performance for weak memory model architectures.
>>
>> Any thoughts on this approach?
>>
>> ```
>> diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
>> index c18f1e16edee..ee2b3a4dcc05 100644
>> --- a/kernel/bpf/helpers.c
>> +++ b/kernel/bpf/helpers.c
>> @@ -1591,9 +1591,7 @@ BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern
>> *, async)
>>            */
>>           if (!cur_t)
>>                   goto drop;
>> -       atomic_inc(&t->cancelling);
>> -       /* Need full barrier after relaxed atomic_inc */
>> -       smp_mb__after_atomic();
>> +       atomic_fetch_add(1, &t->cancelling);
>>           inc = true;
>>           if (atomic_read(&cur_t->cancelling)) {
>>                   /* We're cancelling timer t, while some other timer
>> callback is
>> ```
>>
>> I tested the above diff, the BPF timer_lockup selftest was stressed
>> for 5000 consecutive loops on a physical LoongArch machine without
>> encountering any further lockups or warnings.
> 
> There is a full barrier in both cases. I don't know what performance improvement
> will be achieved by changing this.

We performed benchmarking using UnixBench on a physical LoongArch
machine, the UnixBench score is different (amadd.w + dbar < amadd_db.w).

> Don't you need to fix the lowering for
> smp_mb__after_atomic() anyway? It's used in several other places in the kernel.

The LoongArch architecture-level fix for smp_mb__after_atomic() is
a fundamental bug fix that will be pushed separately to the LoongArch
tree. This BPF-layer change is intended as a generic, cross-architecture
performance optimization.

Thanks,
Tiezhu

Re: [PATCH v1] LoongArch: Fix __smp_mb__{before,after}_atomic()

[Kumar Kartikeya Dwivedi]

On Tue Aug 18, 2026 at 3:50 PM CEST, Tiezhu Yang wrote:
> On 2026/8/18 下午9:35, Kumar Kartikeya Dwivedi wrote:
>> On Tue Aug 18, 2026 at 3:28 PM CEST, Tiezhu Yang wrote:
>>> On 2026/8/17 上午11:59, Tiezhu Yang wrote:
>>>> When testing the BPF selftest "sudo ./test_progs -t timer_lockup", there
>>>> is a kernel lockup and panic:
>>>
>>> ...
>>>
>>>> With this patch, the lockless "store-before-load" ordering is enforced by
>>>> the DBAR instruction. The BPF timer_lockup selftest was stressed for 5000
>>>> consecutive loops on a physical LoongArch machine without encountering any
>>>> further lockups or warnings:
>>>>
>>>>     for i in {1..5000}; do sudo ./test_progs -t timer_lockup; done
>>>
>>> ...
>>>
>>>> diff --git a/arch/loongarch/include/asm/barrier.h b/arch/loongarch/include/asm/barrier.h
>>>> index 4b663f197706..adfe343dfa65 100644
>>>> --- a/arch/loongarch/include/asm/barrier.h
>>>> +++ b/arch/loongarch/include/asm/barrier.h
>>>> @@ -57,8 +57,8 @@
>>>>    #define __WEAK_LLSC_MB		"			\n"
>>>>    #endif
>>>>
>>>> -#define __smp_mb__before_atomic()	barrier()
>>>> -#define __smp_mb__after_atomic()	barrier()
>>>> +#define __smp_mb__before_atomic()	__smp_mb()
>>>> +#define __smp_mb__after_atomic()	__smp_mb()
>>>>
>>>>    /**
>>>>     * array_index_mask_nospec() - generate a ~0 mask when index < size, 0 otherwise
>>>
>>> Hi all,
>>>
>>> To address the soft lockup while avoiding the performance overhead of
>>> executing two separate instructions, I think there is a more elegant
>>> way to fix this directly in the BPF core helper:
>>>
>>> We can replace atomic_inc() and smp_mb__after_atomic() with a single
>>> atomic_fetch_add() in bpf_timer_cancel(). This consolidates the logic
>>> into a single native atomic operation with full ordering, which not
>>> only guarantees the store-load order to eliminate the deadlock but
>>> also improves the performance for weak memory model architectures.
>>>
>>> Any thoughts on this approach?
>>>
>>> ```
>>> diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
>>> index c18f1e16edee..ee2b3a4dcc05 100644
>>> --- a/kernel/bpf/helpers.c
>>> +++ b/kernel/bpf/helpers.c
>>> @@ -1591,9 +1591,7 @@ BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern
>>> *, async)
>>>            */
>>>           if (!cur_t)
>>>                   goto drop;
>>> -       atomic_inc(&t->cancelling);
>>> -       /* Need full barrier after relaxed atomic_inc */
>>> -       smp_mb__after_atomic();
>>> +       atomic_fetch_add(1, &t->cancelling);
>>>           inc = true;
>>>           if (atomic_read(&cur_t->cancelling)) {
>>>                   /* We're cancelling timer t, while some other timer
>>> callback is
>>> ```
>>>
>>> I tested the above diff, the BPF timer_lockup selftest was stressed
>>> for 5000 consecutive loops on a physical LoongArch machine without
>>> encountering any further lockups or warnings.
>>
>> There is a full barrier in both cases. I don't know what performance improvement
>> will be achieved by changing this.
>
> We performed benchmarking using UnixBench on a physical LoongArch
> machine, the UnixBench score is different (amadd.w + dbar < amadd_db.w).
>

I mean, this function is already quite heavy. We have multiple fully ordered
atomics (refcount bumps/drops, xchg(), etc.) spread across the operation. Do you
observe any measurable speedup in the throughput of this function?  The second
question is whether timer cancellation is really frequent. In practice, I don't
think that is the case.  The actual hrtimer_cancel() in itself is heavy and
waits synchronously for the callback to finish.

I'm not opposed to it or anything, I just don't think it's worth it in this
case. If the latency of cancel is a problem there are other bigger opportunities
to pursue than this.

>> Don't you need to fix the lowering for
>> smp_mb__after_atomic() anyway? It's used in several other places in the kernel.
>
> The LoongArch architecture-level fix for smp_mb__after_atomic() is
> a fundamental bug fix that will be pushed separately to the LoongArch
> tree. This BPF-layer change is intended as a generic, cross-architecture
> performance optimization.
>
> Thanks,
> Tiezhu