From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.0 required=3.0 tests=BAYES_00,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5862C433ED for ; Thu, 6 May 2021 06:33:31 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id 1432161401 for ; Thu, 6 May 2021 06:33:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1432161401 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvmarm-bounces@lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 7EC204B4A1; Thu, 6 May 2021 02:33:30 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tF0GxJFI2J+o; Thu, 6 May 2021 02:33:28 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 7840F4B4B2; Thu, 6 May 2021 02:33:28 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 8C8FA4B4A1 for ; Thu, 6 May 2021 02:33:27 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vEqZDZnA6fNk for ; Thu, 6 May 2021 02:33:26 -0400 (EDT) Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 28FA34B4A0 for ; Thu, 6 May 2021 02:33:26 -0400 (EDT) Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 1BE88613F9; Thu, 6 May 2021 06:33:25 +0000 (UTC) Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1leXZa-00BBpF-RQ; Thu, 06 May 2021 07:33:22 +0100 Date: Thu, 06 May 2021 07:33:22 +0100 Message-ID: <87zgx8mkwd.wl-maz@kernel.org> From: Marc Zyngier To: Zenghui Yu Subject: Re: [PATCH v2 03/11] KVM: arm64: Make kvm_skip_instr() and co private to HYP In-Reply-To: <875yzxnn5w.wl-maz@kernel.org> References: <20201102164045.264512-1-maz@kernel.org> <20201102164045.264512-4-maz@kernel.org> <875yzxnn5w.wl-maz@kernel.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: yuzenghui@huawei.com, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, kernel-team@android.com, will@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com, ascull@google.com, mark.rutland@arm.com, qperret@google.com, dbrazdil@google.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Cc: kvm@vger.kernel.org, kernel-team@android.com, Will Deacon , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Wed, 05 May 2021 17:46:51 +0100, Marc Zyngier wrote: > > Hi Zenghui, > > On Wed, 05 May 2021 15:23:02 +0100, > Zenghui Yu wrote: > > > > Hi Marc, > > > > On 2020/11/3 0:40, Marc Zyngier wrote: > > > In an effort to remove the vcpu PC manipulations from EL1 on nVHE > > > systems, move kvm_skip_instr() to be HYP-specific. EL1's intent > > > to increment PC post emulation is now signalled via a flag in the > > > vcpu structure. > > > > > > Signed-off-by: Marc Zyngier > > > > [...] > > > > > @@ -133,6 +134,8 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) > > > __load_guest_stage2(vcpu->arch.hw_mmu); > > > __activate_traps(vcpu); > > > + __adjust_pc(vcpu); > > > > If the INCREMENT_PC flag was set (e.g., for WFx emulation) while we're > > handling PSCI CPU_ON call targetting this VCPU, the *target_pc* (aka > > entry point address, normally provided by the primary VCPU) will be > > unexpectedly incremented here. That's pretty bad, I think. > > How can you online a CPU using PSCI if that CPU is currently spinning > on a WFI? Or is that we have transitioned via userspace to perform the > vcpu reset? I can imagine it happening in that case. > > > This was noticed with a latest guest kernel, at least with commit > > dccc9da22ded ("arm64: Improve parking of stopped CPUs"), which put the > > stopped VCPUs in the WFx loop. The guest kernel shouted at me that > > > > "CPU: CPUs started in inconsistent modes" > > Ah, the perks of running guests with "quiet"... Well caught. > > > *after* rebooting. The problem is that the secondary entry point was > > corrupted by KVM as explained above. All of the secondary processors > > started from set_cpu_boot_mode_flag(), with w0=0. Oh well... > > > > I write the below diff and guess it will help. But I have to look at all > > other places where we adjust PC directly to make a right fix. Please let > > me know what do you think. > > > > > > Thanks, > > Zenghui > > > > ---->8---- > > diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c > > index 956cdc240148..ed647eb387c3 100644 > > --- a/arch/arm64/kvm/reset.c > > +++ b/arch/arm64/kvm/reset.c > > @@ -265,7 +265,12 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu) > > if (vcpu->arch.reset_state.be) > > kvm_vcpu_set_be(vcpu); > > > > + /* > > + * Don't bother with the KVM_ARM64_INCREMENT_PC flag while > > + * using this version of __adjust_pc(). > > + */ > > *vcpu_pc(vcpu) = target_pc; > > + vcpu->arch.flags &= ~KVM_ARM64_INCREMENT_PC; Actually, this is far worse than it looks, and this only papers over one particular symptom. We need to resolve all pending PC updates *before* returning to userspace, or things like live migration can observe an inconsistent state. I'll try and cook something up. Thanks, M. -- Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A53ECC433ED for ; Thu, 6 May 2021 06:35:24 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0E343613F0 for ; Thu, 6 May 2021 06:35:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0E343613F0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Subject:Cc:To: From:Message-ID:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=pgzLBI+v/duLf6S+I5e9yyM2k0EpXE3qSCklthIlJXA=; b=HbFVIRBpTz7vf9Bva/9jo1xo4 IxBblVfLTqTYneeJljn15wmF7VAsehXpda9XVxVF7kdXPctDrJWKJIMBkRCmXDYUZ0xNsFEJ3lODA 7zxyxTqnPklkibiGpVK8rzCMjK2zc6Zzb508mQB833IFbBjnsC79FBcSLlxRh2DZwjR0zVBmFNcXP 3H75HZIHBtvv1ENgKlwnLw1lIcY8MUP1Fl21vk0qyMrIsB2CmcFMCRvC9FnEa4EX+qqFdlqEM4P+2 z1GSBUQtvsWCkJEP4VmesTwQXePmpNz1RW7kc7SSDvNIpz5RhUUxsprx7Nh/31R0BjzS/RGG6NLgI Xzgf/kJGA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1leXZl-003MI9-Os; Thu, 06 May 2021 06:33:34 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1leXZg-003MGm-BP for linux-arm-kernel@desiato.infradead.org; Thu, 06 May 2021 06:33:28 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Type:MIME-Version:References: In-Reply-To:Subject:Cc:To:From:Message-ID:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=EqGBlC9Pz7UsFoNcaelWhEHsIdJBZK4h882dmVUvG/E=; b=QKkYPnJmi/sv0agSPHwygS9tQM Emea7ctPzZ/xIcBHMvcrTEiPbczWHJmA7ZgQFlYaC/PBhWtdUk1YRst3Q1gxwshHEVSiqaJ507q+U PoaTO5IyH7gbWwXhom+cIJ+EuQwjVFLIQLdpu/zZ1qto8d3MmNk1x141FWRUnRI9pnn364yuS07ax uW4nAe19zN0g/W5pbQc/U/wMIv++TY6rEtUddr7D9KGMmk6d9kzRGEf2D2pX6GBby+jz5USejbzlj lzlMk38StHWveQhkYSNRCYh34p/GXELO4aGniIJU45tjDIcAW8o81tFv/ulIF/4NInqMMlXdbIuMk ZjNfvOwQ==; Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1leXZd-005dXk-N2 for linux-arm-kernel@lists.infradead.org; Thu, 06 May 2021 06:33:27 +0000 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 1BE88613F9; Thu, 6 May 2021 06:33:25 +0000 (UTC) Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1leXZa-00BBpF-RQ; Thu, 06 May 2021 07:33:22 +0100 Date: Thu, 06 May 2021 07:33:22 +0100 Message-ID: <87zgx8mkwd.wl-maz@kernel.org> From: Marc Zyngier To: Zenghui Yu Cc: , , , , Will Deacon , James Morse , Julien Thierry , Suzuki K Poulose , Andrew Scull , Mark Rutland , Quentin Perret , David Brazdil Subject: Re: [PATCH v2 03/11] KVM: arm64: Make kvm_skip_instr() and co private to HYP In-Reply-To: <875yzxnn5w.wl-maz@kernel.org> References: <20201102164045.264512-1-maz@kernel.org> <20201102164045.264512-4-maz@kernel.org> <875yzxnn5w.wl-maz@kernel.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: yuzenghui@huawei.com, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, kernel-team@android.com, will@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com, ascull@google.com, mark.rutland@arm.com, qperret@google.com, dbrazdil@google.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210505_233325_838169_88471E3E X-CRM114-Status: GOOD ( 35.32 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, 05 May 2021 17:46:51 +0100, Marc Zyngier wrote: > > Hi Zenghui, > > On Wed, 05 May 2021 15:23:02 +0100, > Zenghui Yu wrote: > > > > Hi Marc, > > > > On 2020/11/3 0:40, Marc Zyngier wrote: > > > In an effort to remove the vcpu PC manipulations from EL1 on nVHE > > > systems, move kvm_skip_instr() to be HYP-specific. EL1's intent > > > to increment PC post emulation is now signalled via a flag in the > > > vcpu structure. > > > > > > Signed-off-by: Marc Zyngier > > > > [...] > > > > > @@ -133,6 +134,8 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) > > > __load_guest_stage2(vcpu->arch.hw_mmu); > > > __activate_traps(vcpu); > > > + __adjust_pc(vcpu); > > > > If the INCREMENT_PC flag was set (e.g., for WFx emulation) while we're > > handling PSCI CPU_ON call targetting this VCPU, the *target_pc* (aka > > entry point address, normally provided by the primary VCPU) will be > > unexpectedly incremented here. That's pretty bad, I think. > > How can you online a CPU using PSCI if that CPU is currently spinning > on a WFI? Or is that we have transitioned via userspace to perform the > vcpu reset? I can imagine it happening in that case. > > > This was noticed with a latest guest kernel, at least with commit > > dccc9da22ded ("arm64: Improve parking of stopped CPUs"), which put the > > stopped VCPUs in the WFx loop. The guest kernel shouted at me that > > > > "CPU: CPUs started in inconsistent modes" > > Ah, the perks of running guests with "quiet"... Well caught. > > > *after* rebooting. The problem is that the secondary entry point was > > corrupted by KVM as explained above. All of the secondary processors > > started from set_cpu_boot_mode_flag(), with w0=0. Oh well... > > > > I write the below diff and guess it will help. But I have to look at all > > other places where we adjust PC directly to make a right fix. Please let > > me know what do you think. > > > > > > Thanks, > > Zenghui > > > > ---->8---- > > diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c > > index 956cdc240148..ed647eb387c3 100644 > > --- a/arch/arm64/kvm/reset.c > > +++ b/arch/arm64/kvm/reset.c > > @@ -265,7 +265,12 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu) > > if (vcpu->arch.reset_state.be) > > kvm_vcpu_set_be(vcpu); > > > > + /* > > + * Don't bother with the KVM_ARM64_INCREMENT_PC flag while > > + * using this version of __adjust_pc(). > > + */ > > *vcpu_pc(vcpu) = target_pc; > > + vcpu->arch.flags &= ~KVM_ARM64_INCREMENT_PC; Actually, this is far worse than it looks, and this only papers over one particular symptom. We need to resolve all pending PC updates *before* returning to userspace, or things like live migration can observe an inconsistent state. I'll try and cook something up. Thanks, M. -- Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.0 required=3.0 tests=BAYES_00,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF62AC433B4 for ; Thu, 6 May 2021 06:33:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B11A9613FC for ; Thu, 6 May 2021 06:33:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231927AbhEFGee (ORCPT ); Thu, 6 May 2021 02:34:34 -0400 Received: from mail.kernel.org ([198.145.29.99]:54992 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231192AbhEFGeY (ORCPT ); Thu, 6 May 2021 02:34:24 -0400 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 1BE88613F9; Thu, 6 May 2021 06:33:25 +0000 (UTC) Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1leXZa-00BBpF-RQ; Thu, 06 May 2021 07:33:22 +0100 Date: Thu, 06 May 2021 07:33:22 +0100 Message-ID: <87zgx8mkwd.wl-maz@kernel.org> From: Marc Zyngier To: Zenghui Yu Cc: , , , , Will Deacon , James Morse , Julien Thierry , Suzuki K Poulose , Andrew Scull , Mark Rutland , Quentin Perret , David Brazdil Subject: Re: [PATCH v2 03/11] KVM: arm64: Make kvm_skip_instr() and co private to HYP In-Reply-To: <875yzxnn5w.wl-maz@kernel.org> References: <20201102164045.264512-1-maz@kernel.org> <20201102164045.264512-4-maz@kernel.org> <875yzxnn5w.wl-maz@kernel.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: yuzenghui@huawei.com, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, kernel-team@android.com, will@kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com, ascull@google.com, mark.rutland@arm.com, qperret@google.com, dbrazdil@google.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Wed, 05 May 2021 17:46:51 +0100, Marc Zyngier wrote: > > Hi Zenghui, > > On Wed, 05 May 2021 15:23:02 +0100, > Zenghui Yu wrote: > > > > Hi Marc, > > > > On 2020/11/3 0:40, Marc Zyngier wrote: > > > In an effort to remove the vcpu PC manipulations from EL1 on nVHE > > > systems, move kvm_skip_instr() to be HYP-specific. EL1's intent > > > to increment PC post emulation is now signalled via a flag in the > > > vcpu structure. > > > > > > Signed-off-by: Marc Zyngier > > > > [...] > > > > > @@ -133,6 +134,8 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) > > > __load_guest_stage2(vcpu->arch.hw_mmu); > > > __activate_traps(vcpu); > > > + __adjust_pc(vcpu); > > > > If the INCREMENT_PC flag was set (e.g., for WFx emulation) while we're > > handling PSCI CPU_ON call targetting this VCPU, the *target_pc* (aka > > entry point address, normally provided by the primary VCPU) will be > > unexpectedly incremented here. That's pretty bad, I think. > > How can you online a CPU using PSCI if that CPU is currently spinning > on a WFI? Or is that we have transitioned via userspace to perform the > vcpu reset? I can imagine it happening in that case. > > > This was noticed with a latest guest kernel, at least with commit > > dccc9da22ded ("arm64: Improve parking of stopped CPUs"), which put the > > stopped VCPUs in the WFx loop. The guest kernel shouted at me that > > > > "CPU: CPUs started in inconsistent modes" > > Ah, the perks of running guests with "quiet"... Well caught. > > > *after* rebooting. The problem is that the secondary entry point was > > corrupted by KVM as explained above. All of the secondary processors > > started from set_cpu_boot_mode_flag(), with w0=0. Oh well... > > > > I write the below diff and guess it will help. But I have to look at all > > other places where we adjust PC directly to make a right fix. Please let > > me know what do you think. > > > > > > Thanks, > > Zenghui > > > > ---->8---- > > diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c > > index 956cdc240148..ed647eb387c3 100644 > > --- a/arch/arm64/kvm/reset.c > > +++ b/arch/arm64/kvm/reset.c > > @@ -265,7 +265,12 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu) > > if (vcpu->arch.reset_state.be) > > kvm_vcpu_set_be(vcpu); > > > > + /* > > + * Don't bother with the KVM_ARM64_INCREMENT_PC flag while > > + * using this version of __adjust_pc(). > > + */ > > *vcpu_pc(vcpu) = target_pc; > > + vcpu->arch.flags &= ~KVM_ARM64_INCREMENT_PC; Actually, this is far worse than it looks, and this only papers over one particular symptom. We need to resolve all pending PC updates *before* returning to userspace, or things like live migration can observe an inconsistent state. I'll try and cook something up. Thanks, M. -- Without deviation from the norm, progress is not possible.