* Re: Linux 6.13-rc6
2025-01-06 13:18 ` Linux 6.13-rc6 Borislav Petkov
@ 2025-01-06 13:33 ` Greg KH
2025-01-06 13:56 ` Breno Leitao
` (2 subsequent siblings)
3 siblings, 0 replies; 7+ messages in thread
From: Greg KH @ 2025-01-06 13:33 UTC (permalink / raw)
To: Borislav Petkov
Cc: Linus Torvalds, Linux Kernel Mailing List, Koichiro Den,
Sebastian Andrzej Siewior, Andrew Morton, stable
On Mon, Jan 06, 2025 at 02:18:17PM +0100, Borislav Petkov wrote:
> On Sun, Jan 05, 2025 at 02:20:54PM -0800, Linus Torvalds wrote:
> > So we had a slight pickup in commits this last week, but as expected
> > and hoped for, things were still pretty quiet. About twice as many
> > commits as the holiday week, but that's still not all that many.
> >
> > I expect things will start becoming more normal now that people are
> > back from the holidays and are starting to recover and wake up from
> > their food comas.
> >
> > In the meantime, below is the shortlog for the last week. Nothing
> > particularly stands out, the changes are dominated by various driver
> > updates (gpu, rdma and networking), with a random smattering of fixes
> > elsewhere.
>
> Something not well baked managed to sneak in and it is tagged for stable:
>
> adcfb264c3ed ("vmstat: disable vmstat_work on vmstat_cpu_down_prep()")
>
> Reverting it fixes the warn splat below.
>
> [ 0.310373] smpboot: x86: Booting SMP configuration:
> [ 0.311074] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15
> [ 0.313798] ------------[ cut here ]------------
> [ 0.317530] workqueue: work disable count underflowed
> [ 0.317530] WARNING: CPU: 1 PID: 21 at kernel/workqueue.c:4317 enable_work+0xa4/0xb0
> [ 0.317530] Modules linked in:
> [ 0.317530] CPU: 1 UID: 0 PID: 21 Comm: cpuhp/1 Not tainted 6.13.0-rc6 #11
> [ 0.317530] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 2023.11-8 02/21/2024
> [ 0.317530] RIP: 0010:enable_work+0xa4/0xb0
> [ 0.317530] Code: c0 48 83 c4 18 5b 5d e9 ca 25 9f 00 80 3d 0a c7 48 08 00 75 b3 c6 05 01 c7 48 08 01 90 48 c7 c7 c8 eb 1d 82 e8 5d 77 fd ff 90 <0f> 0b 90 90 eb 98 90 0f 0b 90 eb b1 90 90 90 90 90 90 90 90 90 90
> [ 0.317530] RSP: 0018:ffffc90000137e18 EFLAGS: 00010082
> [ 0.317530] RAX: 0000000000000029 RBX: ffff88807d66dda0 RCX: 00000000ffefffff
> [ 0.317530] RDX: 0000000000000001 RSI: ffffc90000137ce0 RDI: 0000000000000001
> [ 0.317530] RBP: 0000000000000000 R08: 00000000ffefffff R09: 0000000000000058
> [ 0.317530] R10: 0000000000000000 R11: ffffffff8244df00 R12: 0000000000000000
> [ 0.317530] R13: ffff88807d6604e0 R14: ffffffff812439f0 R15: ffff88807d660508
> [ 0.317530] FS: 0000000000000000(0000) GS:ffff88807d640000(0000) knlGS:0000000000000000
> [ 0.317530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 0.317530] CR2: 0000000000000000 CR3: 0000000002c1a000 CR4: 00000000003506f0
> [ 0.317530] Call Trace:
> [ 0.317530] <TASK>
> [ 0.317530] ? __warn+0xa0/0x160
> [ 0.317530] ? enable_work+0xa4/0xb0
> [ 0.317530] ? report_bug+0x18c/0x1c0
> [ 0.317530] ? handle_bug+0x54/0x90
> [ 0.317530] ? exc_invalid_op+0x1b/0x80
> [ 0.317530] ? asm_exc_invalid_op+0x1a/0x20
> [ 0.317530] ? __pfx_vmstat_cpu_online+0x10/0x10
> [ 0.317530] ? enable_work+0xa4/0xb0
> [ 0.317530] ? enable_work+0xa3/0xb0
> [ 0.317530] vmstat_cpu_online+0x61/0x80
> [ 0.317530] cpuhp_invoke_callback+0x10f/0x480
> [ 0.317530] ? srso_return_thunk+0x5/0x5f
> [ 0.317530] cpuhp_thread_fun+0xd4/0x160
> [ 0.317530] smpboot_thread_fn+0xdd/0x1f0
> [ 0.317530] ? __pfx_smpboot_thread_fn+0x10/0x10
> [ 0.317530] kthread+0xca/0xf0
> [ 0.317530] ? __pfx_kthread+0x10/0x10
> [ 0.317530] ret_from_fork+0x50/0x60
> [ 0.317530] ? __pfx_kthread+0x10/0x10
> [ 0.317530] ret_from_fork_asm+0x1a/0x30
> [ 0.317530] </TASK>
> [ 0.317530] ---[ end trace 0000000000000000 ]---
> [ 0.377680] smp: Brought up 1 node, 16 CPUs
> [ 0.378345] smpboot: Total of 16 processors activated (118393.24 BogoMIPS)
I've just dropped it from the stable queues as well.
thanks,
greg k-h
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: Linux 6.13-rc6
2025-01-06 13:18 ` Linux 6.13-rc6 Borislav Petkov
2025-01-06 13:33 ` Greg KH
@ 2025-01-06 13:56 ` Breno Leitao
2025-01-06 14:07 ` Alexander Gordeev
2025-01-06 14:21 ` Koichiro Den
3 siblings, 0 replies; 7+ messages in thread
From: Breno Leitao @ 2025-01-06 13:56 UTC (permalink / raw)
To: Borislav Petkov, koichiro.den
Cc: Linus Torvalds, Linux Kernel Mailing List, Koichiro Den,
Sebastian Andrzej Siewior, Andrew Morton, stable
On Mon, Jan 06, 2025 at 02:18:17PM +0100, Borislav Petkov wrote:
> On Sun, Jan 05, 2025 at 02:20:54PM -0800, Linus Torvalds wrote:
> > So we had a slight pickup in commits this last week, but as expected
> > and hoped for, things were still pretty quiet. About twice as many
> > commits as the holiday week, but that's still not all that many.
> >
> > I expect things will start becoming more normal now that people are
> > back from the holidays and are starting to recover and wake up from
> > their food comas.
> >
> > In the meantime, below is the shortlog for the last week. Nothing
> > particularly stands out, the changes are dominated by various driver
> > updates (gpu, rdma and networking), with a random smattering of fixes
> > elsewhere.
>
> Something not well baked managed to sneak in and it is tagged for stable:
>
> adcfb264c3ed ("vmstat: disable vmstat_work on vmstat_cpu_down_prep()")
Thanks for bissecting it.
I've just confirmed that reverting adcfb264c3ed ("vmstat: disable
vmstat_work on vmstat_cpu_down_prep()") gets rid of the "workqueue: work
disable count underflow" WARNING.
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: Linux 6.13-rc6
2025-01-06 13:18 ` Linux 6.13-rc6 Borislav Petkov
2025-01-06 13:33 ` Greg KH
2025-01-06 13:56 ` Breno Leitao
@ 2025-01-06 14:07 ` Alexander Gordeev
2025-01-06 14:21 ` Koichiro Den
3 siblings, 0 replies; 7+ messages in thread
From: Alexander Gordeev @ 2025-01-06 14:07 UTC (permalink / raw)
To: Borislav Petkov
Cc: Linus Torvalds, Linux Kernel Mailing List, Koichiro Den,
Sebastian Andrzej Siewior, Andrew Morton, stable
On Mon, Jan 06, 2025 at 02:18:17PM +0100, Borislav Petkov wrote:
Hi All,
> Something not well baked managed to sneak in and it is tagged for stable:
>
> adcfb264c3ed ("vmstat: disable vmstat_work on vmstat_cpu_down_prep()")
>
> Reverting it fixes the warn splat below.
>
> [ 0.310373] smpboot: x86: Booting SMP configuration:
> [ 0.311074] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15
> [ 0.313798] ------------[ cut here ]------------
> [ 0.317530] workqueue: work disable count underflowed
> [ 0.317530] WARNING: CPU: 1 PID: 21 at kernel/workqueue.c:4317 enable_work+0xa4/0xb0
> [ 0.317530] Modules linked in:
> [ 0.317530] CPU: 1 UID: 0 PID: 21 Comm: cpuhp/1 Not tainted 6.13.0-rc6 #11
...
JFYI, hitting the same on s390:
[ 0.176304] smp: Bringing up secondary CPUs ...
[ 0.186596] ------------[ cut here ]------------
[ 0.186603] workqueue: work disable count underflowed
[ 0.186606] WARNING: CPU: 1 PID: 21 at kernel/workqueue.c:4317 enable_work+0x10a/0x120
[ 0.186613] Modules linked in:
[ 0.186615] CPU: 1 UID: 0 PID: 21 Comm: cpuhp/1 Not tainted 6.13.0-rc6 #206
[ 0.186619] Hardware name: IBM 3931 A01 701 (KVM/Linux)
[ 0.186621] Krnl PSW : 0404c00180000000 00000308470979ee (enable_work+0x10e/0x120)
[ 0.186626] R:0 T:1 IO:0 EX:0 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:0 EA:3
[ 0.186629] Krnl GPRS: 00000308484bf9b8 00000308484bf9b0 0000000000000029 00000308484bf9b8
[ 0.186632] 000002884717ba08 000002884717ba00 000003084726e5e0 000000000073e180
[ 0.186634] 000000007fffffff 0000000000000000 0000000000000000 000000007af24840
[ 0.186637] 0000000000e3a300 0000000000e3a300 00000308470979ea 000002884717bc60
[ 0.186685] Krnl Code: 00000308470979de: c02000846cd0 larl %r2,000003084812537e
[ 0.186685] 00000308470979e4: c0e5fffee146 brasl %r14,0000030847073c70
[ 0.186685] #00000308470979ea: af000000 mc 0,0
[ 0.186685] >00000308470979ee: a7080000 lhi %r0,0
[ 0.186685] 00000308470979f2: a7f4ffb2 brc 15,0000030847097956
[ 0.186685] 00000308470979f6: 0707 bcr 0,%r7
[ 0.186685] 00000308470979f8: 0707 bcr 0,%r7
[ 0.186685] 00000308470979fa: 0707 bcr 0,%r7
[ 0.186708] Call Trace:
[ 0.186710] [<00000308470979ee>] enable_work+0x10e/0x120
[ 0.186714] ([<00000308470979ea>] enable_work+0x10a/0x120)
[ 0.186718] [<000003084726e64c>] vmstat_cpu_online+0x6c/0x90
[ 0.186722] [<0000030847075d9a>] cpuhp_invoke_callback+0x16a/0x4d0
[ 0.186725] [<0000030847076832>] cpuhp_thread_fun+0xc2/0x250
[ 0.186728] [<00000308470a8424>] smpboot_thread_fn+0xe4/0x1b0
[ 0.186731] [<00000308470a0d46>] kthread+0x126/0x130
[ 0.186735] [<000003084702508c>] __ret_from_fork+0x3c/0x60
[ 0.186738] [<0000030847d3a582>] ret_from_fork+0xa/0x38
[ 0.186742] Last Breaking-Event-Address:
[ 0.186743] [<0000030847073d62>] __warn_printk+0xf2/0x100
[ 0.186746] Kernel panic - not syncing: kernel: panic_on_warn set ...
[ 0.186749] CPU: 1 UID: 0 PID: 21 Comm: cpuhp/1 Not tainted 6.13.0-rc6 #206
Thanks!
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: Linux 6.13-rc6
2025-01-06 13:18 ` Linux 6.13-rc6 Borislav Petkov
` (2 preceding siblings ...)
2025-01-06 14:07 ` Alexander Gordeev
@ 2025-01-06 14:21 ` Koichiro Den
2025-01-07 2:28 ` Andrew Morton
3 siblings, 1 reply; 7+ messages in thread
From: Koichiro Den @ 2025-01-06 14:21 UTC (permalink / raw)
To: Borislav Petkov
Cc: Linus Torvalds, Linux Kernel Mailing List,
Sebastian Andrzej Siewior, Andrew Morton, stable
On Mon, Jan 06, 2025 at 02:18:17PM +0100, Borislav Petkov wrote:
> On Sun, Jan 05, 2025 at 02:20:54PM -0800, Linus Torvalds wrote:
> > So we had a slight pickup in commits this last week, but as expected
> > and hoped for, things were still pretty quiet. About twice as many
> > commits as the holiday week, but that's still not all that many.
> >
> > I expect things will start becoming more normal now that people are
> > back from the holidays and are starting to recover and wake up from
> > their food comas.
> >
> > In the meantime, below is the shortlog for the last week. Nothing
> > particularly stands out, the changes are dominated by various driver
> > updates (gpu, rdma and networking), with a random smattering of fixes
> > elsewhere.
>
> Something not well baked managed to sneak in and it is tagged for stable:
>
> adcfb264c3ed ("vmstat: disable vmstat_work on vmstat_cpu_down_prep()")
>
> Reverting it fixes the warn splat below.
>
> [ 0.310373] smpboot: x86: Booting SMP configuration:
> [ 0.311074] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15
> [ 0.313798] ------------[ cut here ]------------
> [ 0.317530] workqueue: work disable count underflowed
> [ 0.317530] WARNING: CPU: 1 PID: 21 at kernel/workqueue.c:4317 enable_work+0xa4/0xb0
> [ 0.317530] Modules linked in:
> [ 0.317530] CPU: 1 UID: 0 PID: 21 Comm: cpuhp/1 Not tainted 6.13.0-rc6 #11
> [ 0.317530] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 2023.11-8 02/21/2024
> [ 0.317530] RIP: 0010:enable_work+0xa4/0xb0
> [ 0.317530] Code: c0 48 83 c4 18 5b 5d e9 ca 25 9f 00 80 3d 0a c7 48 08 00 75 b3 c6 05 01 c7 48 08 01 90 48 c7 c7 c8 eb 1d 82 e8 5d 77 fd ff 90 <0f> 0b 90 90 eb 98 90 0f 0b 90 eb b1 90 90 90 90 90 90 90 90 90 90
> [ 0.317530] RSP: 0018:ffffc90000137e18 EFLAGS: 00010082
> [ 0.317530] RAX: 0000000000000029 RBX: ffff88807d66dda0 RCX: 00000000ffefffff
> [ 0.317530] RDX: 0000000000000001 RSI: ffffc90000137ce0 RDI: 0000000000000001
> [ 0.317530] RBP: 0000000000000000 R08: 00000000ffefffff R09: 0000000000000058
> [ 0.317530] R10: 0000000000000000 R11: ffffffff8244df00 R12: 0000000000000000
> [ 0.317530] R13: ffff88807d6604e0 R14: ffffffff812439f0 R15: ffff88807d660508
> [ 0.317530] FS: 0000000000000000(0000) GS:ffff88807d640000(0000) knlGS:0000000000000000
> [ 0.317530] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 0.317530] CR2: 0000000000000000 CR3: 0000000002c1a000 CR4: 00000000003506f0
> [ 0.317530] Call Trace:
> [ 0.317530] <TASK>
> [ 0.317530] ? __warn+0xa0/0x160
> [ 0.317530] ? enable_work+0xa4/0xb0
> [ 0.317530] ? report_bug+0x18c/0x1c0
> [ 0.317530] ? handle_bug+0x54/0x90
> [ 0.317530] ? exc_invalid_op+0x1b/0x80
> [ 0.317530] ? asm_exc_invalid_op+0x1a/0x20
> [ 0.317530] ? __pfx_vmstat_cpu_online+0x10/0x10
> [ 0.317530] ? enable_work+0xa4/0xb0
> [ 0.317530] ? enable_work+0xa3/0xb0
> [ 0.317530] vmstat_cpu_online+0x61/0x80
> [ 0.317530] cpuhp_invoke_callback+0x10f/0x480
> [ 0.317530] ? srso_return_thunk+0x5/0x5f
> [ 0.317530] cpuhp_thread_fun+0xd4/0x160
> [ 0.317530] smpboot_thread_fn+0xdd/0x1f0
> [ 0.317530] ? __pfx_smpboot_thread_fn+0x10/0x10
> [ 0.317530] kthread+0xca/0xf0
> [ 0.317530] ? __pfx_kthread+0x10/0x10
> [ 0.317530] ret_from_fork+0x50/0x60
> [ 0.317530] ? __pfx_kthread+0x10/0x10
> [ 0.317530] ret_from_fork_asm+0x1a/0x30
> [ 0.317530] </TASK>
> [ 0.317530] ---[ end trace 0000000000000000 ]---
> [ 0.377680] smp: Brought up 1 node, 16 CPUs
> [ 0.378345] smpboot: Total of 16 processors activated (118393.24 BogoMIPS)
Thanks for letting me know, and apologies for the inconvenience caused.
In the thread [1], I'm working on a follow-up fix with help from Lorenzo
and others. Please feel free to take any necessary action (e.g. revert).
Also, thank you, Greg, for [2].
[1]: https://lore.kernel.org/all/20241221033321.4154409-1-koichiro.den@canonical.com/T/#m11d983715699d3cea295b8618aba7b6ccec4db55
[2]: https://lore.kernel.org/stable/2025010603-tabasco-laziness-db0e@gregkh/
-Koichiro Den
>
> --
> Regards/Gruss,
> Boris.
>
> https://people.kernel.org/tglx/notes-about-netiquette
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: Linux 6.13-rc6
2025-01-06 14:21 ` Koichiro Den
@ 2025-01-07 2:28 ` Andrew Morton
2025-01-07 7:23 ` Borislav Petkov
0 siblings, 1 reply; 7+ messages in thread
From: Andrew Morton @ 2025-01-07 2:28 UTC (permalink / raw)
To: Koichiro Den
Cc: Borislav Petkov, Linus Torvalds, Linux Kernel Mailing List,
Sebastian Andrzej Siewior, stable
On Mon, 6 Jan 2025 23:21:23 +0900 Koichiro Den <koichiro.den@canonical.com> wrote:
> > [ 0.377680] smp: Brought up 1 node, 16 CPUs
> > [ 0.378345] smpboot: Total of 16 processors activated (118393.24 BogoMIPS)
>
> Thanks for letting me know, and apologies for the inconvenience caused.
> In the thread [1], I'm working on a follow-up fix with help from Lorenzo
> and others. Please feel free to take any necessary action (e.g. revert).
Thanks all, I have queued a revert.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Linux 6.13-rc6
2025-01-07 2:28 ` Andrew Morton
@ 2025-01-07 7:23 ` Borislav Petkov
0 siblings, 0 replies; 7+ messages in thread
From: Borislav Petkov @ 2025-01-07 7:23 UTC (permalink / raw)
To: Andrew Morton
Cc: Koichiro Den, Linus Torvalds, Linux Kernel Mailing List,
Sebastian Andrzej Siewior, stable
On Mon, Jan 06, 2025 at 06:28:00PM -0800, Andrew Morton wrote:
> Thanks all, I have queued a revert.
Linus zapped it already.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
^ permalink raw reply [flat|nested] 7+ messages in thread