From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out-187.mta1.migadu.com (out-187.mta1.migadu.com [95.215.58.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 076EF3CF1E6 for ; Mon, 20 Jul 2026 08:29:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.187 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784536175; cv=none; b=F6k3CQYMVrNP+jdgQg4PhHty5Fjd4OellMGr4DJoaHYfKR7Ig+ESSL6jkHPSygp+IXIGnFVAKffVryIIAgD1cZbq8en2GFBhBPjnDMQ5/rcZvKvqMycfYG1mZc9VMgLw/15y9hSfDyQmft+VYVOKY7H6VcKUqHJbn4QCkkki/H0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784536175; c=relaxed/simple; bh=I2k0/mLBXI3WZxds8jjSvTJA+TLI5PdtF7G9ncaSwNM=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=MJJuq4sp217KccmLccR/RqG7zxr1YCw0+IgQ5UcL6fX/6RYVEkZKlTFmP4pPmbwR8E08vNceT53tdEm01YKnIop6CX800OIWnFT+43XyFVFdHa1jUHqJ8pSxfskCVb/wRhB6U5UrRyC/kz4mygh8HAa1T8/MOEzMmg7vMyH78Bk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=AiCxWrVJ; arc=none smtp.client-ip=95.215.58.187 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="AiCxWrVJ" Message-ID: <5a38dec5-9120-497d-aeb6-aa20ad87b7d5@linux.dev> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1784536168; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vwF+n1XoE/OxpY5imhDLSIWlovWmOJF+JJ7hq35yR6o=; b=AiCxWrVJRVGCkbpylFXFdTX4KLI5YNnyL9uQzj2mlijs3RbQabWwWK5RwVEXbG3urfjj/w EosLYonsPknc+BEE9h0mLdEQe1OzaOKET+NhSXyiXS8IgS9wXy/zx5kBAT3FatmC1ZaBGp Ft3VgCsbSvYfe9roczbdd44u75xj/Uc= Date: Mon, 20 Jul 2026 16:29:14 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Subject: Re: [PATCH] cpu/hotplug: Prevent offlining the last domain housekeeping CPU To: Bradley Morgan Cc: frederic@kernel.org, linux-kernel@vger.kernel.org, peterz@infradead.org, tglx@kernel.org, zhangguopeng@kylinos.cn References: <4DE64100-BF83-4591-B554-4E6F7982FDD9@grrlz.net> Content-Language: en-US X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Guopeng Zhang In-Reply-To: <4DE64100-BF83-4591-B554-4E6F7982FDD9@grrlz.net> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT 在 2026/7/18 22:53, Bradley Morgan 写道: > Hi Guopeng, > > On Fri, 18 Jul 2026, Guopeng Zhang wrote: >> _cpu_down() is supposed to leave at least one HK_TYPE_DOMAIN CPU online. >> However, cpumask_any_and() includes the CPU being offlined. The target >> therefore satisfies the check when it is the last online CPU in that >> housekeeping mask. Hi Bradley, Thanks for the detailed review. I verified the points you raised. > > The bug is real. Note the code before the revert got this right, > 38685e2a0476 > ("cpu/hotplug: Don't offline the last non-isolated CPU") had > > for_each_cpu_and(cpu, cpu_online_mask, housekeeping_cpumask(HK_TYPE_DOMAIN)) { > if (cpu != work.cpu) > > and the exclusion got lost when de715325cc47 moved the check into > _cpu_down(). please say that in the changelog, it makes the Fixes: tag > obvious. > Agreed. I will update the changelog accordingly. >> Offlining it leaves no online CPU in the HK_TYPE_DOMAIN mask. The next >> scheduler-domain rebuild passes an empty span to build_sched_domains() >> and triggers its WARN_ON(). > > [...] > >> On the unpatched kernel, offlining the last HK_TYPE_DOMAIN CPU triggered >> a kernel panic. > > WARN_ON() or panic, pick one. if that was panic_on_warn, say so. if it > kept going and crashed later, paste the splat. The changelog should say > what actually happens on a default config. > The panic was not caused by panic_on_warn. After the empty-span warning, the scheduler-domain rebuild continued and hit a general protection fault in build_perf_domains(), which was followed by a kernel panic. The relevant splat is: [ 69.172549] Oops: general protection fault, probably for non-canonical address 0xfc894905e5c15114: 0000 [#1] SMP NOPTI [ 69.173143] CPU: 2 UID: 0 PID: 29 Comm: cpuhp/2 Tainted: G W E 7.2.0-rc3-next-20260716-sdaf+ #4 PREEMPT(lazy) [ 69.173966] Tainted: [W]=WARN, [E]=UNSIGNED_MODULE [ 69.174306] Hardware name: Red Hat KVM, BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 [ 69.174944] RIP: 0010:build_perf_domains+0x40/0x230 [ 69.175312] Code: fd 53 e8 83 e1 8e 00 48 c7 c2 80 7c 20 92 48 98 48 8b 04 c5 80 0d 57 90 4c 8b a4 10 48 10 00 00 8b 05 6c 9f 4e 02 85 c0 75 3a <49> 8b 84 24 c0 07 00 00 49 c7 84 24 c0 07 00 00 00 00 00 00 48 85 [ 69.176881] RSP: 0018:ffffd1be401efc38 EFLAGS: 00010246 [ 69.177496] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 69.178328] RDX: 0000000000000040 RSI: 0000000000000020 RDI: ffff88ed8813ab30 [ 69.178821] RBP: ffff88ed8813ab30 R08: 0000000000000001 R09: ffffffff80000000 [ 69.179226] R10: 0000000000000000 R11: 0000000000000000 R12: fc894905e5c14954 [ 69.179622] R13: 0000000000000008 R14: 0000000000000000 R15: ffff88ed8813a690 [ 69.180017] FS: 0000000000000000(0000) GS:ffff88f9bad3e000(0000) knlGS:0000000000000000 [ 69.180465] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 69.180786] CR2: 00007f6cfb06a010 CR3: 0000000115c49001 CR4: 0000000000772ef0 [ 69.181182] PKRU: 55555554 [ 69.181340] Call Trace: [ 69.181484] [ 69.181608] partition_sched_domains_locked+0x3d9/0x710 [ 69.182013] ? partition_sched_domains_locked+0x16d/0x710 [ 69.182418] partition_sched_domains+0x30/0x40 [ 69.182769] rebuild_sched_domains_locked+0xe3/0x110 [ 69.183151] cpuset_handle_hotplug+0x578/0xbc0 [ 69.183507] sched_cpu_deactivate+0x1dc/0x300 [ 69.183856] ? cpuhp_thread_fun+0x5a/0x230 [ 69.184187] cpuhp_invoke_callback+0x1a4/0x780 [ 69.184542] ? cpuhp_thread_fun+0x5a/0x230 [ 69.184872] cpuhp_thread_fun+0x1b4/0x230 [ 69.185202] ? __pfx_smpboot_thread_fn+0x10/0x10 [ 69.185562] smpboot_thread_fn+0xf3/0x220 [ 69.185889] kthread+0xe6/0x120 [ 69.186168] ? __pfx_kthread+0x10/0x10 [ 69.186482] ret_from_fork+0x29f/0x330 [ 69.186793] ? __pfx_kthread+0x10/0x10 [ 69.187102] ret_from_fork_asm+0x1a/0x30 [ 69.187424] I will describe this sequence more accurately in the v2 changelog. >> - if (cpumask_any_and(cpu_online_mask, >> - housekeeping_cpumask(HK_TYPE_DOMAIN)) >= nr_cpu_ids) { >> + if (cpumask_any_and_but(cpu_online_mask, >> + housekeeping_cpumask(HK_TYPE_DOMAIN), >> + cpu) >= nr_cpu_ids) { > > The check itself is correct for the sysfs path. cpumask_any_and_but() > returns >= nr_cpu_ids when nothing is left, and excluding a cpu that is > not in the housekeeping mask is harmless. Placement is fine too, > de715325cc47 put this under cpus_write_lock() specifically to serialize > against cpuset partition updates, so it should stay where it is. > > but this breaks suspend. _cpu_down() is also the frozen path, > freeze_secondary_cpus() does > > error = _cpu_down(cpu, 1, CPUHP_OFFLINE); > > for every CPU except the boot CPU. With your own cmdline > (HK_TYPE_DOMAIN = {1,2}, primary = CPU0) suspend now dies at the last > housekeeping CPU: > > Error taking CPU1 down: -16 > Non-boot CPUs are not disabled Confirmed. I tested suspend with the same isolcpus configuration and reproduced this failure. > The old check passed there because it counted the outgoing CPU. I did not > check whether the frozen path can hit the same empty span WARN today, but > turning a suspend that used to work into -EBUSY is a regression either > way. do e.g: > > if (!tasks_frozen && > cpumask_any_and_but(cpu_online_mask, > housekeeping_cpumask(HK_TYPE_DOMAIN), > cpu) >= nr_cpu_ids) { > > and test suspend/resume with the same isolcpus cmdline for v2, not just > the sysfs offline. I will update the patch as suggested and retest both the sysfs CPU offline path and the suspend/resume path with the same isolcpus command line before sending v2. > > The Fixes: tag LGTM, the revert is where the exclusion went missing. If > de715325cc47 is in a released kernel by now, CC stable. iirc cpu hotplug > is root only and this needs isolcpus on the cmdline, so put that in the > impact line and let the stable folks judge. I will add the root-only and isolcpus requirements to the impact description. I will also check whether de715325cc47 is present in a released kernel and Cc stable if appropriate. Thanks again for catching the suspend regression. Thanks, Guopeng