[6.13-rc0 regression] workqueue throwing cpu affinity warnings during CPU hotplug

[6.13-rc0 regression] workqueue throwing cpu affinity warnings during CPU hotplug

[Dave Chinner]

Hi Tejun,

I just upgraded my test VMs from 6.12.0 to a current TOT kernel and
I got several of these warnings whilst running fstests whilst
running CPU hotplug online/offline concurrently with various tests:

[ 2508.109594] ------------[ cut here ]------------
[ 2508.115669] WARNING: CPU: 23 PID: 133 at kernel/kthread.c:76 kthread_set_per_cpu+0x33/0x50
[ 2508.125271] Modules linked in:
[ 2508.131000] CPU: 23 UID: 0 PID: 133 Comm: cpuhp/23 Tainted: G      D W          6.12.0-dgc+ #296
[ 2508.142954] Tainted: [D]=DIE, [W]=WARN
[ 2508.148153] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 2508.148158] RIP: 0010:kthread_set_per_cpu+0x33/0x50
[ 2508.169229] Code: f6 47 2e 20 74 24 48 8b 87 58 0a 00 00 48 85 c0 74 11 f6 47 2f 04 74 16 85 f6 78 18 89 70 08 f0 80 08 01 5d c3 cc cc cc cc cc <0f> 0b eb d8 0f 0b 85 f6 79 e8 f0 80 20 fe 5d c3 cc cc cc cc cc 0f
[ 2508.187728] RSP: 0018:ffffc900068c3de8 EFLAGS: 00010246
[ 2508.192758] RAX: 0000000000000000 RBX: ffff8888d4295680 RCX: 0b0ee35baf810600
[ 2508.199999] RDX: 0000000000000040 RSI: 00000000ffffffff RDI: ffff888866aa0000
[ 2508.208341] RBP: ffffc900068c3de8 R08: 0000000000000001 R09: 0000000000000001
[ 2508.218878] R10: 000000005b5d0000 R11: ffffffff8118d8b0 R12: 000000000002fef0
[ 2508.226572] R13: ffff88901fbefb88 R14: ffff88901fbef8c0 R15: 0000000000000017
[ 2508.233346] FS:  0000000000000000(0000) GS:ffff88901fbc0000(0000) knlGS:0000000000000000
[ 2508.239695] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2508.244123] CR2: 00007fbaf43c0004 CR3: 0000000003256000 CR4: 0000000000350ef0
[ 2508.249599] Call Trace:
[ 2508.253909]  <TASK>
[ 2508.311972]  unbind_worker+0x1b/0x70
[ 2508.315444]  workqueue_offline_cpu+0xd8/0x1f0
[ 2508.319554]  cpuhp_invoke_callback+0x13e/0x4f0
[ 2508.328936]  cpuhp_thread_fun+0xda/0x120
[ 2508.332746]  smpboot_thread_fn+0x132/0x1d0
[ 2508.336645]  kthread+0x147/0x170
[ 2508.347646]  ret_from_fork+0x3e/0x50
[ 2508.353845]  ret_from_fork_asm+0x1a/0x30
[ 2508.357773]  </TASK>
[ 2508.357776] ---[ end trace 0000000000000000 ]---

I have also seen similar traces from the CPUs coming on-line:

[ 2535.818771] WARNING: CPU: 23 PID: 133 at kernel/kthread.c:76 kthread_set_per_cpu+0x33/0x50
....
[ 2535.969004] RIP: 0010:kthread_set_per_cpu+0x33/0x50
....
[ 2508.249599] Call Trace:
[ 2508.253909]  <TASK>
[ 2535.969029]  workqueue_online_cpu+0xe6/0x2f0
[ 2535.969032]  cpuhp_invoke_callback+0x13e/0x4f0
[ 2535.969044]  cpuhp_thread_fun+0xda/0x120
[ 2535.969047]  smpboot_thread_fn+0x132/0x1d0
[ 2535.969053]  kthread+0x147/0x170
[ 2535.969066]  ret_from_fork+0x3e/0x50
[ 2535.969076]  ret_from_fork_asm+0x1a/0x30
[ 2508.357773]  </TASK>

I didn't see these on 6.12.0, so I'm guessing that there is
something in the merge window that has started triggering this.

-Dave.

-- 
Dave Chinner
david@fromorbit.com

Re: [6.13-rc0 regression] workqueue throwing cpu affinity warnings during CPU hotplug

[Tejun Heo]

Hello, Dave.

Sorry about the really late reply.

On Fri, Nov 22, 2024 at 11:38:19AM +1100, Dave Chinner wrote:
> Hi Tejun,
> 
> I just upgraded my test VMs from 6.12.0 to a current TOT kernel and
> I got several of these warnings whilst running fstests whilst
> running CPU hotplug online/offline concurrently with various tests:
> 
> [ 2508.109594] ------------[ cut here ]------------
> [ 2508.115669] WARNING: CPU: 23 PID: 133 at kernel/kthread.c:76 kthread_set_per_cpu+0x33/0x50
...
> [ 2508.253909]  <TASK>
> [ 2508.311972]  unbind_worker+0x1b/0x70
> [ 2508.315444]  workqueue_offline_cpu+0xd8/0x1f0
> [ 2508.319554]  cpuhp_invoke_callback+0x13e/0x4f0
> [ 2508.328936]  cpuhp_thread_fun+0xda/0x120
> [ 2508.332746]  smpboot_thread_fn+0x132/0x1d0
> [ 2508.336645]  kthread+0x147/0x170
> [ 2508.347646]  ret_from_fork+0x3e/0x50
> [ 2508.353845]  ret_from_fork_asm+0x1a/0x30
> [ 2508.357773]  </TASK>
> [ 2508.357776] ---[ end trace 0000000000000000 ]---

So, this is kthread saying that the thread passed to it doesn't have
PF_KTHREAD set. There hasn't been any related changes and the flag is never
cleared once set, so I don't see how that could be for a kworker.

> I have also seen similar traces from the CPUs coming on-line:
> 
> [ 2535.818771] WARNING: CPU: 23 PID: 133 at kernel/kthread.c:76 kthread_set_per_cpu+0x33/0x50
> ....
> [ 2535.969004] RIP: 0010:kthread_set_per_cpu+0x33/0x50
> ....
> [ 2508.249599] Call Trace:
> [ 2508.253909]  <TASK>
> [ 2535.969029]  workqueue_online_cpu+0xe6/0x2f0
> [ 2535.969032]  cpuhp_invoke_callback+0x13e/0x4f0
> [ 2535.969044]  cpuhp_thread_fun+0xda/0x120
> [ 2535.969047]  smpboot_thread_fn+0x132/0x1d0
> [ 2535.969053]  kthread+0x147/0x170
> [ 2535.969066]  ret_from_fork+0x3e/0x50
> [ 2535.969076]  ret_from_fork_asm+0x1a/0x30
> [ 2508.357773]  </TASK>

Yeah, this is the same.

> I didn't see these on 6.12.0, so I'm guessing that there is
> something in the merge window that has started triggering this.

I tried a few mixtures of stress-ng + continuous hot [un]plugging but can't
reproduce in the current linus#master. Do you still see this happening?

Thanks.

-- 
tejun

Re: [6.13-rc0 regression] workqueue throwing cpu affinity warnings during CPU hotplug

[Dave Chinner]

On Wed, Dec 11, 2024 at 12:57:26PM -1000, Tejun Heo wrote:
> Hello, Dave.
> 
> Sorry about the really late reply.
> 
> On Fri, Nov 22, 2024 at 11:38:19AM +1100, Dave Chinner wrote:
> > Hi Tejun,
> > 
> > I just upgraded my test VMs from 6.12.0 to a current TOT kernel and
> > I got several of these warnings whilst running fstests whilst
> > running CPU hotplug online/offline concurrently with various tests:
> > 
> > [ 2508.109594] ------------[ cut here ]------------
> > [ 2508.115669] WARNING: CPU: 23 PID: 133 at kernel/kthread.c:76 kthread_set_per_cpu+0x33/0x50
> ...
> > [ 2508.253909]  <TASK>
> > [ 2508.311972]  unbind_worker+0x1b/0x70
> > [ 2508.315444]  workqueue_offline_cpu+0xd8/0x1f0
> > [ 2508.319554]  cpuhp_invoke_callback+0x13e/0x4f0
> > [ 2508.328936]  cpuhp_thread_fun+0xda/0x120
> > [ 2508.332746]  smpboot_thread_fn+0x132/0x1d0
> > [ 2508.336645]  kthread+0x147/0x170
> > [ 2508.347646]  ret_from_fork+0x3e/0x50
> > [ 2508.353845]  ret_from_fork_asm+0x1a/0x30
> > [ 2508.357773]  </TASK>
> > [ 2508.357776] ---[ end trace 0000000000000000 ]---
> 
> So, this is kthread saying that the thread passed to it doesn't have
> PF_KTHREAD set. There hasn't been any related changes and the flag is never
> cleared once set, so I don't see how that could be for a kworker.

....

> > I didn't see these on 6.12.0, so I'm guessing that there is
> > something in the merge window that has started triggering this.
> 
> I tried a few mixtures of stress-ng + continuous hot [un]plugging but can't
> reproduce in the current linus#master. Do you still see this happening?

I don't know. The machine that I was using to reproduce this has
been down with hardware failure for the last two weeks. I might have
it back up today, and then I'll be able to start exercising this
again.

-Dave.

-- 
Dave Chinner
david@fromorbit.com