From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 32F4CE7E635 for ; Tue, 26 Sep 2023 13:52:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=KHJXYdG2EDZnd/ZRZAwnOgaxdLvWNFC7DcxkGLLv9EU=; b=JbtYfPXzUneNp2 4bS3pBoBLX3WGta7j4z0q98I/592cW30MrL9AZLAIlCuc2iRLAxPmTB1p2/is5lqnW2mFMffVEtNv VP64uidXc9NXAXBNf6DEW9OPpT+to6ygQ1EA0U7CDCsA+c99GiWEXTSd0YS3Rsk771dt/sYEWv6P3 TudHXTPqgKk5a5cH4cyrLsdjqql8IDQW15d0tW7doYwcJhmk3ILAV3KbhUyQWPcEE/1n9wP66e3Sq t0clertS9xglxcXxZFe5EkntoTQhPEqWaAczh4ovMbt8quPYpc4/TR8DP3hudYzC40SnwHNArbzi0 9BFbXooiDG9GGqZP2MPA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1ql8UD-00GVQd-12; Tue, 26 Sep 2023 13:52:25 +0000 Received: from sin.source.kernel.org ([2604:1380:40e1:4800::1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1ql8U9-00GVPs-2P for linux-arm-kernel@lists.infradead.org; Tue, 26 Sep 2023 13:52:23 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id E6F28CE13B8; Tue, 26 Sep 2023 13:52:19 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 24545C433C7; Tue, 26 Sep 2023 13:52:15 +0000 (UTC) Date: Tue, 26 Sep 2023 14:52:13 +0100 From: Catalin Marinas To: Lorenzo Pieralisi Cc: Jason Gunthorpe , ankita@nvidia.com, maz@kernel.org, oliver.upton@linux.dev, will@kernel.org, aniketa@nvidia.com, cjia@nvidia.com, kwankhede@nvidia.com, targupta@nvidia.com, vsethi@nvidia.com, acurrid@nvidia.com, apopple@nvidia.com, jhubbard@nvidia.com, danw@nvidia.com, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org Subject: Re: [PATCH v1 2/2] KVM: arm64: allow the VM to select DEVICE_* and NORMAL_NC for IO memory Message-ID: References: <20230907181459.18145-1-ankita@nvidia.com> <20230907181459.18145-3-ankita@nvidia.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230926_065222_121339_26540381 X-CRM114-Status: GOOD ( 24.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Sep 26, 2023 at 10:31:38AM +0200, Lorenzo Pieralisi wrote: > Currently, KVM for ARM64 maps at stage 2 memory that is > considered device (ie using pfn_is_map_memory() to discern > between device memory and memory itself) with DEVICE_nGnRE > memory attributes; this setting overrides (as per the ARM > architecture [1]) any device MMIO mapping present at stage > 1, resulting in a set-up whereby a guest operating system > can't determine device MMIO mapping memory attributes on its > own but it is always overriden by the KVM stage 2 default. > = > This set-up does not allow guest operating systems to map > device memory on a page by page basis with combined attributes > other than DEVICE_nGnRE, Well, it also has the option of DEVICE_nGnRnE ;). > which turns out to be an issue in that > guest operating systems (eg Linux) may request to map > devices MMIO regions with memory attributes that guarantee > better performance (eg gathering attribute - that for some > devices can generate larger PCIe memory writes TLPs) > and specific operations (eg unaligned transactions) such as > the NormalNC memory type. > = > The default device stage 2 mapping was chosen in KVM > for ARM64 since it was considered safer (ie it would > not allow guests to trigger uncontained failures > ultimately crashing the machine) but this turned out > to be imprecise. > = > Failures containability is a property of the platform > and is independent from the memory type used for MMIO > device memory mappings (ie DEVICE_nGnRE memory type is > even more problematic than NormalNC in terms of containability > since eg aborts triggered on loads cannot be made synchronous, > which make them harder to contain); this means that, > regardless of the combined stage1+stage2 mappings a > platform is safe if and only if device transactions cannot trigger > uncontained failures; reworded, the default KVM device > stage 2 memory attributes play no role in making device > assignment safer for a given platform and therefore can > be relaxed. > = > For all these reasons, relax the KVM stage 2 device > memory attributes from DEVICE_nGnRE to NormalNC. > = > This puts guests in control (thanks to stage1+stage2 > combined memory attributes rules [1]) of device MMIO > regions memory mappings, according to the rules > described in [1] and summarized here ([(S1) =3D Stage1][(S2) =3D Stage2]): > = > =EF=BF=BDS1=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF= =BD=EF=BF=BD=EF=BF=BD=EF=BF=BD |=EF=BF=BD=EF=BF=BD S2=EF=BF=BD=EF=BF=BD=EF= =BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD |=EF=BF=BD Res= ult > =EF=BF=BDNORMAL-WB=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD|=EF=BF=BD NORMAL-N= C=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD|=EF=BF=BD NORMAL-NC > =EF=BF=BDNORMAL-WT=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD|=EF=BF=BD NORMAL-N= C=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD|=EF=BF=BD NORMAL-NC > =EF=BF=BDNORMAL-NC=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD|=EF=BF=BD NORMAL-N= C=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD|=EF=BF=BD NORMAL-NC > =EF=BF=BDDEVICE=EF=BF=BD|=EF=BF=BD NORMAL-NC=EF=BF=BD=EF=BF=BD=EF= =BF=BD=EF=BF=BD|=EF=BF=BD DEVICE Not sure what's wrong with my font setup as I can't see the above table but I know it from the Arm ARM. Anyway, the text looks fine to me. Thanks for putting it together Lorenzo. One thing not mentioned here is that pci-vfio still maps such memory as Device-nGnRnE in user space and relaxing this potentially creates an alias. But such alias is only relevant of both the VMM and the VM try to access the same device which I doubt is a realistic scenario. -- = Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel