[PATCH] cpu/hotplug: Prevent offlining the last domain housekeeping CPU

[PATCH] cpu/hotplug: Prevent offlining the last domain housekeeping CPU

[Guopeng Zhang]

From: Guopeng Zhang <zhangguopeng@kylinos.cn>

_cpu_down() is supposed to leave at least one HK_TYPE_DOMAIN CPU online.
However, cpumask_any_and() includes the CPU being offlined. The target
therefore satisfies the check when it is the last online CPU in that
housekeeping mask.

Offlining it leaves no online CPU in the HK_TYPE_DOMAIN mask. The next
scheduler-domain rebuild passes an empty span to build_sched_domains()
and triggers its WARN_ON().

Use cpumask_any_and_but() to look for another online HK_TYPE_DOMAIN CPU
and reject the offline request with -EBUSY if none exists.

Testing:
Tested on a 32-vCPU Ubuntu 24.04 guest with:

    isolcpus=domain,0,3-31

On the unpatched kernel, offlining the last HK_TYPE_DOMAIN CPU triggered
a kernel panic.

With this patch, the same operation was rejected with -EBUSY (errno 16),
the target CPU remained online, and no scheduler-domain warning was
observed.

Fixes: de715325cc47 ("cpu: Revert "cpu/hotplug: Prevent self deadlock on CPU hot-unplug"")
Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn>
---
 kernel/cpu.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/kernel/cpu.c b/kernel/cpu.c
index b3c8553d7bd6..81e961964e59 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -1418,8 +1418,9 @@ static int __ref _cpu_down(unsigned int cpu, int tasks_frozen,
 	 * Keep at least one housekeeping cpu onlined to avoid generating
 	 * an empty sched_domain span.
 	 */
-	if (cpumask_any_and(cpu_online_mask,
-			    housekeeping_cpumask(HK_TYPE_DOMAIN)) >= nr_cpu_ids) {
+	if (cpumask_any_and_but(cpu_online_mask,
+				housekeeping_cpumask(HK_TYPE_DOMAIN),
+				cpu) >= nr_cpu_ids) {
 		ret = -EBUSY;
 		goto out;
 	}
-- 
2.43.0

Re: [PATCH] cpu/hotplug: Prevent offlining the last domain housekeeping CPU

[Bradley Morgan]

Hi Guopeng,

On Fri, 18 Jul 2026, Guopeng Zhang wrote:
> _cpu_down() is supposed to leave at least one HK_TYPE_DOMAIN CPU online.
> However, cpumask_any_and() includes the CPU being offlined. The target
> therefore satisfies the check when it is the last online CPU in that
> housekeeping mask.

The bug is real. Note the code before the revert got this right,
38685e2a0476
("cpu/hotplug: Don't offline the last non-isolated CPU") had

	for_each_cpu_and(cpu, cpu_online_mask, housekeeping_cpumask(HK_TYPE_DOMAIN)) {
		if (cpu != work.cpu)

and the exclusion got lost when de715325cc47 moved the check into
_cpu_down(). please say that in the changelog, it makes the Fixes: tag
obvious.

> Offlining it leaves no online CPU in the HK_TYPE_DOMAIN mask. The next
> scheduler-domain rebuild passes an empty span to build_sched_domains()
> and triggers its WARN_ON().

[...]

> On the unpatched kernel, offlining the last HK_TYPE_DOMAIN CPU triggered
> a kernel panic.

WARN_ON() or panic, pick one. if that was panic_on_warn, say so. if it
kept going and crashed later, paste the splat. The changelog should say
what actually happens on a default config.

> -	if (cpumask_any_and(cpu_online_mask,
> -			    housekeeping_cpumask(HK_TYPE_DOMAIN)) >= nr_cpu_ids) {
> +	if (cpumask_any_and_but(cpu_online_mask,
> +				housekeeping_cpumask(HK_TYPE_DOMAIN),
> +				cpu) >= nr_cpu_ids) {

The check itself is correct for the sysfs path. cpumask_any_and_but()
returns >= nr_cpu_ids when nothing is left, and excluding a cpu that is
not in the housekeeping mask is harmless. Placement is fine too,
de715325cc47 put this under cpus_write_lock() specifically to serialize
against cpuset partition updates, so it should stay where it is.

but this breaks suspend. _cpu_down() is also the frozen path,
freeze_secondary_cpus() does

	error = _cpu_down(cpu, 1, CPUHP_OFFLINE);

for every CPU except the boot CPU. With your own cmdline
(HK_TYPE_DOMAIN = {1,2}, primary = CPU0) suspend now dies at the last
housekeeping CPU:

	Error taking CPU1 down: -16
	Non-boot CPUs are not disabled

The old check passed there because it counted the outgoing CPU. I did not
check whether the frozen path can hit the same empty span WARN today, but
turning a suspend that used to work into -EBUSY is a regression either
way. do e.g:

	if (!tasks_frozen &&
	    cpumask_any_and_but(cpu_online_mask,
				housekeeping_cpumask(HK_TYPE_DOMAIN),
				cpu) >= nr_cpu_ids) {

and test suspend/resume with the same isolcpus cmdline for v2, not just
the sysfs offline.

The Fixes: tag LGTM, the revert is where the exclusion went missing. If
de715325cc47 is in a released kernel by now, CC stable. iirc cpu hotplug
is root only and this needs isolcpus on the cmdline, so put that in the
impact line and let the stable folks judge.
Thanks!

Re: [PATCH] cpu/hotplug: Prevent offlining the last domain housekeeping CPU

[Guopeng Zhang]

在 2026/7/18 22:53, Bradley Morgan 写道:
> Hi Guopeng,
> 
> On Fri, 18 Jul 2026, Guopeng Zhang wrote:
>> _cpu_down() is supposed to leave at least one HK_TYPE_DOMAIN CPU online.
>> However, cpumask_any_and() includes the CPU being offlined. The target
>> therefore satisfies the check when it is the last online CPU in that
>> housekeeping mask.

Hi Bradley,

Thanks for the detailed review. I verified the points you raised.

> 
> The bug is real. Note the code before the revert got this right,
> 38685e2a0476
> ("cpu/hotplug: Don't offline the last non-isolated CPU") had
> 
> 	for_each_cpu_and(cpu, cpu_online_mask, housekeeping_cpumask(HK_TYPE_DOMAIN)) {
> 		if (cpu != work.cpu)
> 
> and the exclusion got lost when de715325cc47 moved the check into
> _cpu_down(). please say that in the changelog, it makes the Fixes: tag
> obvious.
> 

Agreed. I will update the changelog accordingly.

>> Offlining it leaves no online CPU in the HK_TYPE_DOMAIN mask. The next
>> scheduler-domain rebuild passes an empty span to build_sched_domains()
>> and triggers its WARN_ON().
> 
> [...]
> 
>> On the unpatched kernel, offlining the last HK_TYPE_DOMAIN CPU triggered
>> a kernel panic.
> 
> WARN_ON() or panic, pick one. if that was panic_on_warn, say so. if it
> kept going and crashed later, paste the splat. The changelog should say
> what actually happens on a default config.
> 

The panic was not caused by panic_on_warn. After the empty-span warning,
the scheduler-domain rebuild continued and hit a general protection
fault in build_perf_domains(), which was followed by a kernel panic.

The relevant splat is:
[   69.172549] Oops: general protection fault, probably for non-canonical address 0xfc894905e5c15114: 0000 [#1] SMP NOPTI
[   69.173143] CPU: 2 UID: 0 PID: 29 Comm: cpuhp/2 Tainted: G        W   E       7.2.0-rc3-next-20260716-sdaf+ #4 PREEMPT(lazy) 
[   69.173966] Tainted: [W]=WARN, [E]=UNSIGNED_MODULE
[   69.174306] Hardware name: Red Hat KVM, BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[   69.174944] RIP: 0010:build_perf_domains+0x40/0x230
[   69.175312] Code: fd 53 e8 83 e1 8e 00 48 c7 c2 80 7c 20 92 48 98 48 8b 04 c5 80 0d 57 90 4c 8b a4 10 48 10 00 00 8b 05 6c 9f 4e 02 85 c0 75 3a <49> 8b 84 24 c0 07 00 00 49 c7 84 24 c0 07 00 00 00 00 00 00 48 85
[   69.176881] RSP: 0018:ffffd1be401efc38 EFLAGS: 00010246
[   69.177496] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[   69.178328] RDX: 0000000000000040 RSI: 0000000000000020 RDI: ffff88ed8813ab30
[   69.178821] RBP: ffff88ed8813ab30 R08: 0000000000000001 R09: ffffffff80000000
[   69.179226] R10: 0000000000000000 R11: 0000000000000000 R12: fc894905e5c14954
[   69.179622] R13: 0000000000000008 R14: 0000000000000000 R15: ffff88ed8813a690
[   69.180017] FS:  0000000000000000(0000) GS:ffff88f9bad3e000(0000) knlGS:0000000000000000
[   69.180465] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   69.180786] CR2: 00007f6cfb06a010 CR3: 0000000115c49001 CR4: 0000000000772ef0
[   69.181182] PKRU: 55555554
[   69.181340] Call Trace:
[   69.181484]  <TASK>
[   69.181608]  partition_sched_domains_locked+0x3d9/0x710
[   69.182013]  ? partition_sched_domains_locked+0x16d/0x710
[   69.182418]  partition_sched_domains+0x30/0x40
[   69.182769]  rebuild_sched_domains_locked+0xe3/0x110
[   69.183151]  cpuset_handle_hotplug+0x578/0xbc0
[   69.183507]  sched_cpu_deactivate+0x1dc/0x300
[   69.183856]  ? cpuhp_thread_fun+0x5a/0x230
[   69.184187]  cpuhp_invoke_callback+0x1a4/0x780
[   69.184542]  ? cpuhp_thread_fun+0x5a/0x230
[   69.184872]  cpuhp_thread_fun+0x1b4/0x230
[   69.185202]  ? __pfx_smpboot_thread_fn+0x10/0x10
[   69.185562]  smpboot_thread_fn+0xf3/0x220
[   69.185889]  kthread+0xe6/0x120
[   69.186168]  ? __pfx_kthread+0x10/0x10
[   69.186482]  ret_from_fork+0x29f/0x330
[   69.186793]  ? __pfx_kthread+0x10/0x10
[   69.187102]  ret_from_fork_asm+0x1a/0x30
[   69.187424]  </TASK>

I will describe this sequence more accurately in the v2 changelog.

>> -	if (cpumask_any_and(cpu_online_mask,
>> -			    housekeeping_cpumask(HK_TYPE_DOMAIN)) >= nr_cpu_ids) {
>> +	if (cpumask_any_and_but(cpu_online_mask,
>> +				housekeeping_cpumask(HK_TYPE_DOMAIN),
>> +				cpu) >= nr_cpu_ids) {
> 
> The check itself is correct for the sysfs path. cpumask_any_and_but()
> returns >= nr_cpu_ids when nothing is left, and excluding a cpu that is
> not in the housekeeping mask is harmless. Placement is fine too,
> de715325cc47 put this under cpus_write_lock() specifically to serialize
> against cpuset partition updates, so it should stay where it is.
> 
> but this breaks suspend. _cpu_down() is also the frozen path,
> freeze_secondary_cpus() does
> 
> 	error = _cpu_down(cpu, 1, CPUHP_OFFLINE);
> 
> for every CPU except the boot CPU. With your own cmdline
> (HK_TYPE_DOMAIN = {1,2}, primary = CPU0) suspend now dies at the last
> housekeeping CPU:
> 
> 	Error taking CPU1 down: -16
> 	Non-boot CPUs are not disabled

Confirmed. I tested suspend with the same isolcpus configuration and
reproduced this failure.

> The old check passed there because it counted the outgoing CPU. I did not
> check whether the frozen path can hit the same empty span WARN today, but
> turning a suspend that used to work into -EBUSY is a regression either
> way. do e.g:
> 
> 	if (!tasks_frozen &&
> 	    cpumask_any_and_but(cpu_online_mask,
> 				housekeeping_cpumask(HK_TYPE_DOMAIN),
> 				cpu) >= nr_cpu_ids) {
> 
> and test suspend/resume with the same isolcpus cmdline for v2, not just
> the sysfs offline.

I will update the patch as suggested and retest both the sysfs CPU
offline path and the suspend/resume path with the same isolcpus command
line before sending v2.

> 
> The Fixes: tag LGTM, the revert is where the exclusion went missing. If
> de715325cc47 is in a released kernel by now, CC stable. iirc cpu hotplug
> is root only and this needs isolcpus on the cmdline, so put that in the
> impact line and let the stable folks judge.

I will add the root-only and isolcpus requirements to the impact
description. I will also check whether de715325cc47 is present in a
released kernel and Cc stable if appropriate.

Thanks again for catching the suspend regression.

Thanks,
Guopeng