From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from SA9PR02CU001.outbound.protection.outlook.com (mail-southcentralusazon11013049.outbound.protection.outlook.com [40.93.196.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2AAF447013C for ; Tue, 25 Aug 2026 13:16:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.196.49 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787663780; cv=fail; b=gvAl4qMNxrhXlVpwpwjnBjgAafTJ48wZzqMiMTOuTHYY2HZit/l+tcjn8R1FZpzyH+SQ0jKDecwlP3wtcTXfp32M717S3mtE2xlWonyYXD4OQhtGF6tSH4ZN7UpVDngi+tmGaFeTo1vqT2MHYzNbEzrT4+qLLMHj0JPM7i0kzZ4= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787663780; c=relaxed/simple; bh=LxQ7DEYm3V1pvF/nMx84/bxqakp/29Gbjk9dd/ngr8Y=; h=Message-ID:Date:MIME-Version:Subject:To:CC:References:From: In-Reply-To:Content-Type; b=N1NNb9dMRjbnev+SRzH5yKLVxHwGE05Oo+wxgBnZfQEqu++Fvsf8b7JeBeymRBaLPDHeH66fox8B/v6gvYJQ2ofbTHhlfOnarZcPQt85GxgeRr6sE24W2B12PWL3vO0GI2/6TQvssOL8tMDlYc6Ea7JzrHcFy/MeGlACfQXuS5U= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=dODsgSOa; arc=fail smtp.client-ip=40.93.196.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="dODsgSOa" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=QubBJJsD5pRqVZMdi1rEFiYGHxfAkqpaqPjtvTRAefSBYnxSg8eHU53+N6RcHI5O/v5mtVEPATpJXRABpPXBkR8UuHKptO6ehxZf/rQhQDorbkgY5WdC/ruzNfNhuZ6beAOeK5IeFK45mry1itIncUanRx1b6+u5dNmM6wnPJXA8ewTS23eYHEZP0GIXmo/KQ7gNTVtDLfkYTYHuJdTIDpBcYNd4FQbLugnF/GTNcrOCXm8XqlRJm1QItLg74E0oBH/yvy3o9VIYOJzG/qll46zjWrRaeAtzmpa82Rsefc98co4xfHF7P2Mk8jVNpWIA6mhquAN9NG83K3M1KN5xWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=XrgQtGi2ty9oR7vPrS7ddWjYE1ZM6BT3minNauHNowE=; b=GtsxaTxkYoSBE8iCfsJfpLfh9A2MGtBGbqs8EG2U60Kb47kuGQ6o7VSV2bdLd3JMVsbgHznozIvHYkunBlUo2jzUfxDaooWIhTqGQrOCG4Sjf4GGlGt/V8Zs95XlMXSRyg2ncOCmj0lIvdXf7I+d0URryjCsodSUMCI20lHEk5x/Jpx25KoeIQIFa2Nn/kXYS3RyFfz/bYwmvjeaAsL1MP5yXgoJ+onm4YzLuY80GmM95pGJCBVw1qm/ZM0Qf3y6+5xEMJj7oINmHgBEfSLJ+wIq3rvMzXZXeMLusTcZu/Hgxl8+dAZ+ogZditYDhqiD3H/wZAeOGwbWtiadqGNOEA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 149.199.90.133) smtp.rcpttodomain=huawei.com smtp.mailfrom=amd.com; dmarc=fail (p=quarantine sp=quarantine pct=100) action=quarantine header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XrgQtGi2ty9oR7vPrS7ddWjYE1ZM6BT3minNauHNowE=; b=dODsgSOanELEzWXvPCa62ImQY+CS7oHoKgEQi6q6h6CJo2eZAz5TymkyJzqX7zRDrzVCwh03C5nZ6dppmnO86EOu7wt4g9SUAPIGpeSAVtmR4oXMj8y7KJePMjuamw3vCUIHqiF7Cdhm/Q9Bg8WMPmlipDTMQSW1tvFoa44QUGU= Received: from BN9PR03CA0111.namprd03.prod.outlook.com (2603:10b6:408:fd::26) by DS7PR12MB8417.namprd12.prod.outlook.com (2603:10b6:8:eb::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.6; Tue, 25 Aug 2026 13:16:10 +0000 Received: from LV8PEPF00000069.namprd03.prod.outlook.com (2603:10b6:408:fd:cafe::20) by BN9PR03CA0111.outlook.office365.com (2603:10b6:408:fd::26) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.339.12 via Frontend Transport; Tue, 25 Aug 2026 13:16:07 +0000 X-MS-Exchange-Authentication-Results: spf=softfail (sender IP is 149.199.90.133) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=fail action=quarantine header.from=amd.com; Received-SPF: SoftFail (protection.outlook.com: domain of transitioning amd.com discourages use of 149.199.90.133 as permitted sender) Received: from satlexmb08.amd.com (149.199.90.133) by LV8PEPF00000069.mail.protection.outlook.com (10.167.248.42) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.3 via Frontend Transport; Tue, 25 Aug 2026 13:16:06 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by satlexmb08.amd.com (10.181.42.217) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Tue, 25 Aug 2026 08:16:06 -0500 Received: from satlexmb07.amd.com (10.181.42.216) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Tue, 25 Aug 2026 08:16:05 -0500 Received: from [192.168.0.226] (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server id 15.2.2562.45 via Frontend Transport; Tue, 25 Aug 2026 08:16:02 -0500 Message-ID: <24236a3a-e23e-491d-86eb-c17fe88cbd04@amd.com> Date: Tue, 25 Aug 2026 18:45:55 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [Question] Userspace throttling + "sched/fair: Combine detach into dequeue when migrating task" causes guest boot hang To: Chen Jinghuang , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot CC: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Aaron Lu , References: <20260825120629.2472938-1-chenjinghuang2@huawei.com> Content-Language: en-US From: K Prateek Nayak In-Reply-To: <20260825120629.2472938-1-chenjinghuang2@huawei.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV8PEPF00000069:EE_|DS7PR12MB8417:EE_ X-MS-Office365-Filtering-Correlation-Id: 4eef3b8c-5f83-4ba0-1a63-08df02ab0587 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|82310400026|1800799024|36860700016|23010399003|10067099003|56012099006|11063799006|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: EOioxmMlw3aFAOpA/CFvZT5dv/skaFUabxTEUFvCVfuEpq5UbrBI+vj+upghq77kTlUGMNjVug3KY80uKAiKvfws9SPUXNKIUHrbLQAZ0YuVVWfEtKpfhVAbJfK7euHuP0PxNohEjatQHLC7+d65tvoG6mDAI/dSpXzLvYWcxUu/NuzRI9aPGQ2LE8CZwnjd1MzsNkH4uDqHofU5whnNfZ2jIpN9NnOnYVfCHF/yVWGJ1JGTRcKIPzyQlSR3UfVZagqcWQAelX0HQuodrVHMc1lvZbCEIY/bNnXku0knnN+hSeofTZk/VQb+nJRin5XRrl2Ef8hF33X522wkjRfHV3DlCgpriQPh6FOpWL5ZPCZenEcXbTGIEmDfhPlSqAsA3G14NsvYovDmxPhQFsmR5H7gDeC9Qd7QajtOr1kvY+zt9/6uW8P8djPcviKMXCLwFt/hyF1rFc9IliLoV6aLouA8kgbw+88muau6fThinX836XXqxGswP0yBdB0kcxQhlETKS0Io98Gm3SL0AttRlrH1ElePp7AkKjO1IPATKwF3SFF0Ip2p6HVz091Nb3jWonOWFxvmZyuht8imBXAI/n9kU0p11GRKI1SZlx5CRMSDKdGstft/7pVJafyDqHKWzCsCW11hROfNN7g0dcSm9CWEsGz+blVz3nGi2W0BhSbumo2GtLC+RHDfvMG6UbpJ5yIMjznIgp/q4ZJZ57hXTQ== X-Forefront-Antispam-Report: CIP:149.199.90.133;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb08.amd.com;PTR:unknown-90-133.xilinx.com;CAT:NONE;SFS:(13230040)(7416014)(376014)(82310400026)(1800799024)(36860700016)(23010399003)(10067099003)(56012099006)(11063799006)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: YsbrVdQUxTIZkeTNil4JQ70n4YTYlQlsWk+lfgUxm7wztOYJqZAi+jUn33uwc5LM0iXURSNI71VnC7rSfpf1PVlV2gNS7fiJpI/w/lg/jE2NxLOvXHyy/03bs1dDa5/+vJf5HVR6lpOSDBXwxDyWvEQpKRUSft8V9QTIwO8N8xyFc7rGUAam8PD0LHVwQ9D/xp9XkrDzwJu3eiPATEHMo6R1cVaVOkb9+O2Q2cLAYIaeo56ajz8K1uZv4hHxqC5qOl5QGU/+HzYh53h5z22aaQ3Bi8RhxcI7i23lKHegTR2T342UFXHU0APrQed2hD+BiiNTkEGEgMrinprTYfWNISK4Xn/sFrpuOdOIItB4hRoOr7fV3gpa0Skd6LR24M1GUzPK/N7G+3N36tmypG7PscVM9Tzllno4Ostsj5OdDpHr4z+r3VvDfzHV1lUY17AF X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2026 13:16:06.4028 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4eef3b8c-5f83-4ba0-1a63-08df02ab0587 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[149.199.90.133];Helo=[satlexmb08.amd.com] X-MS-Exchange-CrossTenant-AuthSource: LV8PEPF00000069.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB8417 Hello Chen, Thank you for your report. On 8/25/2026 5:36 PM, Chen Jinghuang wrote: > Hi, I'm seeing a VM boot hang on mainline, and I'd like to understand the > interaction between userspace throttling and the scheduler patch > "e1f078f50478 sched/fair: Combine detach into dequeue when migrating > task". > > Host: > aarch64, 96 CPUs (0-95), 4 NUMA nodes: > node0: 0-23, node1: 24-47, node2: 48-71, node3: 72-95 > Mainline kernel tag: 7.2-rc1. > > Guest(libvirt/KVM) - described in words: > An aarch64 (virt-6.2) UEFI VM launched with `virsh create`; key config: > > - 128 vCPUs (statically placed, oversubscribed — the host has only 96 > physical CPUs). > - host-passthrough CPU model; GICv3; 64 GiB RAM. > - has set to 400000; all and > entries are commented out, so there is no vCPU pinning. > - Storage: qcow2 on virtio-scsi (cache=none, io=native). HPET disabled. > > Userspace throttling: > The VM runs under a CPU-quota cap applied on the host. The actual values > from the cgroup controller are: > > cpu.cfs_period_us = 100000 > cpu.cfs_quota_us = 400000 400ms across 96 CPUs per 100ms seems awfully low. Let me go see if I can reproduce this. > > I also found that if I set cpu.cfs_quota_us to -1, or enlarge it beyond a > certain point, the guest boots fine. Sounds a lot like guest side lock-holder preemption stalling the guest. If you give it enough time, does the guest progress? > > Symptom: > The guest hangs at some command early in boot and never reaches the login > prompt. What happens if you allow it to boot and then enforce the more the aggressive limits later? Do you see RCU stalls / lockups? > > Observations: > Only reverting both of the following together makes it boot (neither one > alone suffices): > > 1. The kernel patch for userspace throttling. Are these Aaron's patches too or just the recent rework that I did? Could you please paste a log of all the reverts. > 2. The scheduler patch: > e1f078f50478 ("sched/fair: Combine detach into dequeue when migrating > task"). > > Reverting only one of them still hangs; reverting both together boots fine. Can you check your cgroup stats to see how much time the vCPUs are getting before and after the revert? Very surprising that e1f078f50478 has some effect here. > > Question: > I don't fully understand how these two interact. My rough guess: e1f078f50478 > ("sched/fair: Combine detach into dequeue when migrating task") affects the > PELT accounting, and the userspace throttling also has logic that affects PELT > accounting. When both are combined, load balancing and subsequent scheduling > behavior may end up misbehaving, stalling the guest. > > This looks like a real regression on mainline in the 128-vCPU oversubscribed > VM on a 96-core/4-NUMA host scenario. Any pointer to the correct mechanism or > a fix direction would be very much appreciated. Both, with exit-to-user throttling, and the legacy method, we would have preempted the vCPU in xfer_to_guest_mode_work(): if (ti_work & (_TIF_NEED_RESCHED | _TIF_NEED_RESCHED_LAZY)) schedule(); if (ti_work & _TIF_NOTIFY_RESUME) resume_user_mode_work(NULL); Previously, task would have taken the schedule() route out, and now it is done via resume_user_mode_work() -> schedule() / preempt_schedule() Since e1f078f50478 only takes effect at migration, does 1:1 pinning help progress the boot? Are there any splats in your dmesg? -- Thanks and Regards, Prateek