* [PowerPC][linux-next-6.11-rc7-20240910] Kernel Warnings at kernel/jump_label.c:266
@ 2024-09-11 7:03 Venkat Rao Bagalkote
2024-09-11 9:49 ` Peter Zijlstra
2024-09-11 14:26 ` Michael Ellerman
0 siblings, 2 replies; 3+ messages in thread
From: Venkat Rao Bagalkote @ 2024-09-11 7:03 UTC (permalink / raw)
To: peterz, tglx; +Cc: linux-kernel, linuxppc-dev, Sachin Sant
Greetings!!!
I am observing kernel warnings while booting with
linux-next-6.11-rc7-20240910 kernel. Below are the warnings.
[ 63.613506] WARNING: CPU: 20 PID: 214 at kernel/jump_label.c:266
static_key_dec+0x78/0x8c
[ 63.613518] Modules linked in: ext4 mbcache jbd2 dm_service_time
sd_mod sg ibmvfc mlx5_core ibmvnic scsi_transport_fc ibmveth mlxfw
psample dm_multipath dm_mirror dm_region_hash dm_log dm_mod fuse
[ 63.613542] CPU: 20 UID: 0 PID: 214 Comm: kworker/20:1 Not tainted
6.11.0-rc7-next-20240910-auto #1
[ 63.613549] Hardware name: IBM,9080-HEX POWER10 (architected)
0x800200 0xf000006 of:IBM,FW1060.00 (NH1060_016) hv:phyp pSeries
[ 63.613556] Workqueue: cgroup_destroy css_free_rwork_fn
[ 63.613563] NIP: c00000000048ca20 LR: c00000000048d060 CTR:
c0000000005cb330
[ 63.613568] REGS: c00000005a1afa80 TRAP: 0700 Not tainted
(6.11.0-rc7-next-20240910-auto)
[ 63.613573] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>
CR: 24000884 XER: 20040001
[ 63.613586] CFAR: c00000000048c9cc IRQMASK: 0
[ 63.613586] GPR00: c00000000048d060 c00000005a1afd20 c0000000015f4300
c000000002faa440
[ 63.613586] GPR04: 0000000000000001 000000000000009e c0000000084bae00
000000000000009d
[ 63.613586] GPR08: 0000000000000000 0000000000000000 ffffffffffffffff
c00000005b852780
[ 63.613586] GPR12: c0000000005cb330 c000000c7db7df00 c0000000001a779c
c0000000084b27c0
[ 63.613586] GPR16: 0000000000000000 0000000000000000 0000000000000000
0000000000000000
[ 63.613586] GPR20: c000000c7b914300 c000000002de73c8 c0000000027b07a8
fffffffffffffef7
[ 63.613586] GPR24: 0000000000000402 c00000005b852780 c000000008eabc05
0000000000000010
[ 63.613586] GPR28: c000000002d2fac0 c00000000e9d6000 c000000070444000
c000000002faa440
[ 63.613633] NIP [c00000000048ca20] static_key_dec+0x78/0x8c
[ 63.613639] LR [c00000000048d060]
__static_key_slow_dec_cpuslocked+0x38/0xb4
[ 63.613645] Call Trace:
[ 63.613647] [c00000005a1afd20] [c00000005a1afd50] 0xc00000005a1afd50
(unreliable)
[ 63.613653] [c00000005a1afd50] [c00000000048d124]
static_key_slow_dec+0x48/0xb0
[ 63.613660] [c00000005a1afdc0] [c0000000005cb40c]
mem_cgroup_css_free+0xdc/0x104
[ 63.613667] [c00000005a1afdf0] [c0000000002c4d58]
css_free_rwork_fn+0x70/0x394
[ 63.613673] [c00000005a1afe40] [c00000000019a7dc]
process_one_work+0x1fc/0x4dc
[ 63.613680] [c00000005a1afef0] [c00000000019b73c]
worker_thread+0x340/0x504
[ 63.613686] [c00000005a1aff90] [c0000000001a78cc] kthread+0x138/0x140
[ 63.613691] [c00000005a1affe0] [c00000000000df98]
start_kernel_thread+0x14/0x18
[ 63.613697] Code: 7c0004ac 7c074800 40820018 38600001 5463063e
4e800020 60000000 60000000 7ce93b78 4bffffa4 60000000 60000000
<0fe00000> 38600001 7c60189e 5463063e
[ 63.613711] ---[ end trace 0000000000000000 ]---
Issue is introduced by the commit: de752774f38bb766941ed1bf910ba5a9f6cc6bf7
Reverting the above commit, issue is not seen.
Regards,
Venkat.
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PowerPC][linux-next-6.11-rc7-20240910] Kernel Warnings at kernel/jump_label.c:266
2024-09-11 7:03 [PowerPC][linux-next-6.11-rc7-20240910] Kernel Warnings at kernel/jump_label.c:266 Venkat Rao Bagalkote
@ 2024-09-11 9:49 ` Peter Zijlstra
2024-09-11 14:26 ` Michael Ellerman
1 sibling, 0 replies; 3+ messages in thread
From: Peter Zijlstra @ 2024-09-11 9:49 UTC (permalink / raw)
To: Venkat Rao Bagalkote; +Cc: tglx, linux-kernel, linuxppc-dev, Sachin Sant
On Wed, Sep 11, 2024 at 12:33:01PM +0530, Venkat Rao Bagalkote wrote:
> Greetings!!!
>
>
> I am observing kernel warnings while booting with
> linux-next-6.11-rc7-20240910 kernel. Below are the warnings.
>
>
> Issue is introduced by the commit: de752774f38bb766941ed1bf910ba5a9f6cc6bf7
That commit is now gone and should be replaced with the below on the
next next.
---
commit 1d7f856c2ca449f04a22d876e36b464b7a9d28b6
Author: Peter Zijlstra <peterz@infradead.org>
Date: Mon Sep 9 12:50:09 2024 +0200
jump_label: Fix static_key_slow_dec() yet again
While commit 83ab38ef0a0b ("jump_label: Fix concurrency issues in
static_key_slow_dec()") fixed one problem, it created yet another,
notably the following is now possible:
slow_dec
if (try_dec) // dec_not_one-ish, false
// enabled == 1
slow_inc
if (inc_not_disabled) // inc_not_zero-ish
// enabled == 2
return
guard((mutex)(&jump_label_mutex);
if (atomic_cmpxchg(1,0)==1) // false, we're 2
slow_dec
if (try-dec) // dec_not_one, true
// enabled == 1
return
else
try_dec() // dec_not_one, false
WARN
Use dec_and_test instead of cmpxchg(), like it was prior to
83ab38ef0a0b. Add a few WARNs for the paranoid.
Fixes: 83ab38ef0a0b ("jump_label: Fix concurrency issues in static_key_slow_dec()")
Reported-by: "Darrick J. Wong" <djwong@kernel.org>
Tested-by: Klara Modin <klarasmodin@gmail.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
diff --git a/kernel/jump_label.c b/kernel/jump_label.c
index 6dc76b590703..93a822d3c468 100644
--- a/kernel/jump_label.c
+++ b/kernel/jump_label.c
@@ -168,7 +168,7 @@ bool static_key_slow_inc_cpuslocked(struct static_key *key)
jump_label_update(key);
/*
* Ensure that when static_key_fast_inc_not_disabled() or
- * static_key_slow_try_dec() observe the positive value,
+ * static_key_dec_not_one() observe the positive value,
* they must also observe all the text changes.
*/
atomic_set_release(&key->enabled, 1);
@@ -250,7 +250,7 @@ void static_key_disable(struct static_key *key)
}
EXPORT_SYMBOL_GPL(static_key_disable);
-static bool static_key_slow_try_dec(struct static_key *key)
+static bool static_key_dec_not_one(struct static_key *key)
{
int v;
@@ -274,6 +274,14 @@ static bool static_key_slow_try_dec(struct static_key *key)
* enabled. This suggests an ordering problem on the user side.
*/
WARN_ON_ONCE(v < 0);
+
+ /*
+ * Warn about underflow, and lie about success in an attempt to
+ * not make things worse.
+ */
+ if (WARN_ON_ONCE(v == 0))
+ return true;
+
if (v <= 1)
return false;
} while (!likely(atomic_try_cmpxchg(&key->enabled, &v, v - 1)));
@@ -284,15 +292,27 @@ static bool static_key_slow_try_dec(struct static_key *key)
static void __static_key_slow_dec_cpuslocked(struct static_key *key)
{
lockdep_assert_cpus_held();
+ int val;
- if (static_key_slow_try_dec(key))
+ if (static_key_dec_not_one(key))
return;
guard(mutex)(&jump_label_mutex);
- if (atomic_cmpxchg(&key->enabled, 1, 0) == 1)
+ val = atomic_read(&key->enabled);
+ /*
+ * It should be impossible to observe -1 with jump_label_mutex held,
+ * see static_key_slow_inc_cpuslocked().
+ */
+ if (WARN_ON_ONCE(val == -1))
+ return;
+ /*
+ * Cannot already be 0, something went sideways.
+ */
+ if (WARN_ON_ONCE(val == 0))
+ return;
+
+ if (atomic_dec_and_test(&key->enabled))
jump_label_update(key);
- else
- WARN_ON_ONCE(!static_key_slow_try_dec(key));
}
static void __static_key_slow_dec(struct static_key *key)
@@ -329,7 +349,7 @@ void __static_key_slow_dec_deferred(struct static_key *key,
{
STATIC_KEY_CHECK_USE(key);
- if (static_key_slow_try_dec(key))
+ if (static_key_dec_not_one(key))
return;
schedule_delayed_work(work, timeout);
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PowerPC][linux-next-6.11-rc7-20240910] Kernel Warnings at kernel/jump_label.c:266
2024-09-11 7:03 [PowerPC][linux-next-6.11-rc7-20240910] Kernel Warnings at kernel/jump_label.c:266 Venkat Rao Bagalkote
2024-09-11 9:49 ` Peter Zijlstra
@ 2024-09-11 14:26 ` Michael Ellerman
1 sibling, 0 replies; 3+ messages in thread
From: Michael Ellerman @ 2024-09-11 14:26 UTC (permalink / raw)
To: Venkat Rao Bagalkote, peterz, tglx
Cc: linux-kernel, linuxppc-dev, Sachin Sant
Venkat Rao Bagalkote <venkat88@linux.vnet.ibm.com> writes:
> Greetings!!!
>
> I am observing kernel warnings while booting with
> linux-next-6.11-rc7-20240910 kernel. Below are the warnings.
>
> [ 63.613506] WARNING: CPU: 20 PID: 214 at kernel/jump_label.c:266 static_key_dec+0x78/0x8c
> [ 63.613518] Modules linked in: ext4 mbcache jbd2 dm_service_time sd_mod sg ibmvfc mlx5_core ibmvnic scsi_transport_fc ibmveth mlxfw psample dm_multipath dm_mirror dm_region_hash dm_log dm_mod fuse
> [ 63.613542] CPU: 20 UID: 0 PID: 214 Comm: kworker/20:1 Not tainted 6.11.0-rc7-next-20240910-auto #1
> [ 63.613549] Hardware name: IBM,9080-HEX POWER10 (architected) 0x800200 0xf000006 of:IBM,FW1060.00 (NH1060_016) hv:phyp pSeries
> [ 63.613556] Workqueue: cgroup_destroy css_free_rwork_fn
> [ 63.613563] NIP: c00000000048ca20 LR: c00000000048d060 CTR: c0000000005cb330
> [ 63.613568] REGS: c00000005a1afa80 TRAP: 0700 Not tainted (6.11.0-rc7-next-20240910-auto)
> [ 63.613573] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 24000884 XER: 20040001
> [ 63.613586] CFAR: c00000000048c9cc IRQMASK: 0
> [ 63.613586] GPR00: c00000000048d060 c00000005a1afd20 c0000000015f4300 c000000002faa440
> [ 63.613586] GPR04: 0000000000000001 000000000000009e c0000000084bae00 000000000000009d
> [ 63.613586] GPR08: 0000000000000000 0000000000000000 ffffffffffffffff c00000005b852780
> [ 63.613586] GPR12: c0000000005cb330 c000000c7db7df00 c0000000001a779c c0000000084b27c0
> [ 63.613586] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [ 63.613586] GPR20: c000000c7b914300 c000000002de73c8 c0000000027b07a8 fffffffffffffef7
> [ 63.613586] GPR24: 0000000000000402 c00000005b852780 c000000008eabc05 0000000000000010
> [ 63.613586] GPR28: c000000002d2fac0 c00000000e9d6000 c000000070444000 c000000002faa440
> [ 63.613633] NIP [c00000000048ca20] static_key_dec+0x78/0x8c
> [ 63.613639] LR [c00000000048d060] __static_key_slow_dec_cpuslocked+0x38/0xb4
> [ 63.613645] Call Trace:
> [ 63.613647] [c00000005a1afd20] [c00000005a1afd50] 0xc00000005a1afd50 (unreliable)
> [ 63.613653] [c00000005a1afd50] [c00000000048d124] static_key_slow_dec+0x48/0xb0
> [ 63.613660] [c00000005a1afdc0] [c0000000005cb40c] mem_cgroup_css_free+0xdc/0x104
> [ 63.613667] [c00000005a1afdf0] [c0000000002c4d58] css_free_rwork_fn+0x70/0x394
> [ 63.613673] [c00000005a1afe40] [c00000000019a7dc] process_one_work+0x1fc/0x4dc
> [ 63.613680] [c00000005a1afef0] [c00000000019b73c] worker_thread+0x340/0x504
> [ 63.613686] [c00000005a1aff90] [c0000000001a78cc] kthread+0x138/0x140
> [ 63.613691] [c00000005a1affe0] [c00000000000df98] start_kernel_thread+0x14/0x18
> [ 63.613697] Code: 7c0004ac 7c074800 40820018 38600001 5463063e 4e800020 60000000 60000000 7ce93b78 4bffffa4 60000000 60000000 <0fe00000> 38600001 7c60189e 5463063e
> [ 63.613711] ---[ end trace 0000000000000000 ]---
>
> Issue is introduced by the commit: de752774f38bb766941ed1bf910ba5a9f6cc6bf7
Which was:
de752774f38b ("jump_label: Fix static_key_slow_dec() yet again")
And was already reported:
https://lore.kernel.org/all/202409082005.393050e2-oliver.sang@intel.com/
https://lore.kernel.org/all/20240910111111.2591-1-spasswolf@web.de
In today's linux-next (20240911) it has become:
1d7f856c2ca4 ("jump_label: Fix static_key_slow_dec() yet again")
Can you test if the bug is fixed in today's linux-next?
cheers
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2024-09-11 14:26 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-09-11 7:03 [PowerPC][linux-next-6.11-rc7-20240910] Kernel Warnings at kernel/jump_label.c:266 Venkat Rao Bagalkote
2024-09-11 9:49 ` Peter Zijlstra
2024-09-11 14:26 ` Michael Ellerman
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).