From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 87CC2C004D4 for ; Thu, 19 Jan 2023 21:19:00 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 14218418C3; Thu, 19 Jan 2023 21:19:00 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 14218418C3 Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.a=rsa-sha256 header.s=2020 header.b=mdQ7CD0w; dkim=fail reason="signature verification failed" header.d=linutronix.de header.i=@linutronix.de header.a=ed25519-sha256 header.s=2020e header.b=0BJ5w/p0 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JsEnE3y3h_hr; Thu, 19 Jan 2023 21:18:59 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id 5717D41C66; Thu, 19 Jan 2023 21:18:58 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 5717D41C66 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 175F0C0033; Thu, 19 Jan 2023 21:18:58 +0000 (UTC) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 30706C002D for ; Thu, 19 Jan 2023 21:18:57 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 04087405B5 for ; Thu, 19 Jan 2023 21:18:57 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 04087405B5 Authentication-Results: smtp2.osuosl.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.a=rsa-sha256 header.s=2020 header.b=mdQ7CD0w; dkim=pass header.d=linutronix.de header.i=@linutronix.de header.a=ed25519-sha256 header.s=2020e header.b=0BJ5w/p0 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VBsPYntkPIFH for ; Thu, 19 Jan 2023 21:18:56 +0000 (UTC) X-Greylist: delayed 00:06:19 by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org DABDB405A7 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by smtp2.osuosl.org (Postfix) with ESMTPS id DABDB405A7 for ; Thu, 19 Jan 2023 21:18:55 +0000 (UTC) From: Thomas Gleixner DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1674162752; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=rRUXLP0p0WQ+Qd4xVEuRKMl25ggID9YoYcEY3DCIiik=; b=mdQ7CD0wRJvfaZDOr0f82Dc4967iwC75I1+s5lDCu1fHHwtrp+IBba8UPSoQ/va3j4Qar2 ATGu2nZLLKa0vJ8CheBHLvt0I89xHSG06ZUusQKDPY104va7qqUjwYfiuIUukk+LaqO7Gu N+aHnCbtSnW+e8Smx0L+A2DiKy1AOSyLQ5QGRIN4vgJHVE1sRkt9jT/0k9LahiDV7T+AiK jKmIL8dmtVDNgeoMh+UAXcZ37mv49o4hNCz9KdPCGAyDUP9ZfpqnymsWZ0f3dG4LxhFiyw RTndkgdz2BKS/FhRmtxFB9hP9YbenuT2IOAVALIMh4inh4bZRqxisS45FivGfA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1674162752; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=rRUXLP0p0WQ+Qd4xVEuRKMl25ggID9YoYcEY3DCIiik=; b=0BJ5w/p0x778LsQ1MznxzGrIdB3OE1QGABAUfN3vpOmJUgdDINUgQsr9tLqZGxSzWuC+ct ivl26is4VR39BBBw== To: Igor Mammedov , "Srivatsa S. Bhat" Subject: Re: [PATCH v2] x86/hotplug: Do not put offline vCPUs in mwait idle state In-Reply-To: <20230116155526.05d37ff9@imammedo.users.ipa.redhat.com> References: <20230116060134.80259-1-srivatsa@csail.mit.edu> <20230116155526.05d37ff9@imammedo.users.ipa.redhat.com> Date: Thu, 19 Jan 2023 22:12:31 +0100 Message-ID: <87bkmui5z4.ffs@tglx> MIME-Version: 1.0 Cc: x86@kernel.org, kvm@vger.kernel.org, Peter Zijlstra , Dave Hansen , virtualization@lists.linux-foundation.org, ganb@vmware.com, ankitja@vmware.com, namit@vmware.com, "H. Peter Anvin" , "Rafael J. Wysocki" , VMware PV-Drivers Reviewers , Ingo Molnar , bordoloih@vmware.com, xen-devel@lists.xenproject.org, Wyes Karny , Tom Lendacky , "Paul E. McKenney" , keerthanak@vmware.com, Borislav Petkov , Juergen Gross , amakhalov@vmware.com, linux-kernel@vger.kernel.org, Lewis Caroll , blamoreaux@vmware.com X-BeenThere: virtualization@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Linux virtualization List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: virtualization-bounces@lists.linux-foundation.org Sender: "Virtualization" On Mon, Jan 16 2023 at 15:55, Igor Mammedov wrote: > "Srivatsa S. Bhat" wrote: >> Fix this by preventing the use of mwait idle state in the vCPU offline >> play_dead() path for any hypervisor, even if mwait support is >> available. > > if mwait is enabled, it's very likely guest to have cpuidle > enabled and using the same mwait as well. So exiting early from > mwait_play_dead(), might just punt workflow down: > native_play_dead() > ... > mwait_play_dead(); > if (cpuidle_play_dead()) <- possible mwait here > hlt_play_dead(); > > and it will end up in mwait again and only if that fails > it will go HLT route and maybe transition to VMM. Good point. > Instead of workaround on guest side, > shouldn't hypervisor force VMEXIT on being uplugged vCPU when it's > actually hot-unplugging vCPU? (ex: QEMU kicks vCPU out from guest > context when it is removing vCPU, among other things) For a pure guest side CPU unplug operation: guest$ echo 0 >/sys/devices/system/cpu/cpu$N/online the hypervisor is not involved at all. The vCPU is not removed in that case. So to ensure that this ends up in HLT something like the below is required. Note, the removal of the comment after mwait_play_dead() is intentional because the comment is completely bogus. Not having MWAIT is not a failure. But that wants to be a seperate patch. Thanks, tglx --- diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 55cad72715d9..3f1f20f71ec5 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -1833,7 +1833,10 @@ void native_play_dead(void) play_dead_common(); tboot_shutdown(TB_SHUTDOWN_WFS); - mwait_play_dead(); /* Only returns on failure */ + if (this_cpu_has(X86_FEATURE_HYPERVISOR)) + hlt_play_dead(); + + mwait_play_dead(); if (cpuidle_play_dead()) hlt_play_dead(); } _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5206CC004D4 for ; Thu, 19 Jan 2023 21:13:04 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.481338.746148 (Exim 4.92) (envelope-from ) id 1pIcD6-0002hh-Ec; Thu, 19 Jan 2023 21:12:36 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 481338.746148; Thu, 19 Jan 2023 21:12:36 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1pIcD6-0002ha-Bb; Thu, 19 Jan 2023 21:12:36 +0000 Received: by outflank-mailman (input) for mailman id 481338; Thu, 19 Jan 2023 21:12:34 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1pIcD4-0002Ux-7s for xen-devel@lists.xenproject.org; Thu, 19 Jan 2023 21:12:34 +0000 Received: from galois.linutronix.de (galois.linutronix.de [193.142.43.55]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id fd594469-983d-11ed-91b6-6bf2151ebd3b; Thu, 19 Jan 2023 22:12:33 +0100 (CET) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: fd594469-983d-11ed-91b6-6bf2151ebd3b From: Thomas Gleixner DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1674162752; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=rRUXLP0p0WQ+Qd4xVEuRKMl25ggID9YoYcEY3DCIiik=; b=mdQ7CD0wRJvfaZDOr0f82Dc4967iwC75I1+s5lDCu1fHHwtrp+IBba8UPSoQ/va3j4Qar2 ATGu2nZLLKa0vJ8CheBHLvt0I89xHSG06ZUusQKDPY104va7qqUjwYfiuIUukk+LaqO7Gu N+aHnCbtSnW+e8Smx0L+A2DiKy1AOSyLQ5QGRIN4vgJHVE1sRkt9jT/0k9LahiDV7T+AiK jKmIL8dmtVDNgeoMh+UAXcZ37mv49o4hNCz9KdPCGAyDUP9ZfpqnymsWZ0f3dG4LxhFiyw RTndkgdz2BKS/FhRmtxFB9hP9YbenuT2IOAVALIMh4inh4bZRqxisS45FivGfA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1674162752; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=rRUXLP0p0WQ+Qd4xVEuRKMl25ggID9YoYcEY3DCIiik=; b=0BJ5w/p0x778LsQ1MznxzGrIdB3OE1QGABAUfN3vpOmJUgdDINUgQsr9tLqZGxSzWuC+ct ivl26is4VR39BBBw== To: Igor Mammedov , "Srivatsa S. Bhat" Cc: linux-kernel@vger.kernel.org, amakhalov@vmware.com, ganb@vmware.com, ankitja@vmware.com, bordoloih@vmware.com, keerthanak@vmware.com, blamoreaux@vmware.com, namit@vmware.com, Peter Zijlstra , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , "Rafael J. Wysocki" , "Paul E. McKenney" , Wyes Karny , Lewis Caroll , Tom Lendacky , Juergen Gross , x86@kernel.org, VMware PV-Drivers Reviewers , virtualization@lists.linux-foundation.org, kvm@vger.kernel.org, xen-devel@lists.xenproject.org Subject: Re: [PATCH v2] x86/hotplug: Do not put offline vCPUs in mwait idle state In-Reply-To: <20230116155526.05d37ff9@imammedo.users.ipa.redhat.com> References: <20230116060134.80259-1-srivatsa@csail.mit.edu> <20230116155526.05d37ff9@imammedo.users.ipa.redhat.com> Date: Thu, 19 Jan 2023 22:12:31 +0100 Message-ID: <87bkmui5z4.ffs@tglx> MIME-Version: 1.0 Content-Type: text/plain On Mon, Jan 16 2023 at 15:55, Igor Mammedov wrote: > "Srivatsa S. Bhat" wrote: >> Fix this by preventing the use of mwait idle state in the vCPU offline >> play_dead() path for any hypervisor, even if mwait support is >> available. > > if mwait is enabled, it's very likely guest to have cpuidle > enabled and using the same mwait as well. So exiting early from > mwait_play_dead(), might just punt workflow down: > native_play_dead() > ... > mwait_play_dead(); > if (cpuidle_play_dead()) <- possible mwait here > hlt_play_dead(); > > and it will end up in mwait again and only if that fails > it will go HLT route and maybe transition to VMM. Good point. > Instead of workaround on guest side, > shouldn't hypervisor force VMEXIT on being uplugged vCPU when it's > actually hot-unplugging vCPU? (ex: QEMU kicks vCPU out from guest > context when it is removing vCPU, among other things) For a pure guest side CPU unplug operation: guest$ echo 0 >/sys/devices/system/cpu/cpu$N/online the hypervisor is not involved at all. The vCPU is not removed in that case. So to ensure that this ends up in HLT something like the below is required. Note, the removal of the comment after mwait_play_dead() is intentional because the comment is completely bogus. Not having MWAIT is not a failure. But that wants to be a seperate patch. Thanks, tglx --- diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 55cad72715d9..3f1f20f71ec5 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -1833,7 +1833,10 @@ void native_play_dead(void) play_dead_common(); tboot_shutdown(TB_SHUTDOWN_WFS); - mwait_play_dead(); /* Only returns on failure */ + if (this_cpu_has(X86_FEATURE_HYPERVISOR)) + hlt_play_dead(); + + mwait_play_dead(); if (cpuidle_play_dead()) hlt_play_dead(); }