From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D7E5CE7D25F for ; Tue, 26 Sep 2023 08:32:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=P+dTtvNDzR6GjKGKgtvzFw0r2BftYcFQO2GhHn2fDRc=; b=Ckb60MNWLt5Poh MZIvhxBTEP7d3KNNjCnOCuwQY7oda+tBSQcpRQVqbaFzuDgLeNBLTBSfUh5A4mctcVNWcznmdL8U4 n5MbooEtQc/E6sKihXJw+YjH8mfmsyStKyeOf7NW9SssNx0IZHXxx/612W7f8jCkwapF637nJnpi5 hnr7fM680T5ZmtsWRelE933rL6vDi/1ErUmTAeAUrzKCrOAOB6PIe7dekCd5R6uu9IRqvPH/EB3K6 eU5jjVWhGSn4oSBKOghRgi7KVK2w1a1n+Sz2IuGaZDsJtjMv83BtYvmIsUN2ewFDRq68C7tCyG6jY w8gvFYAnin327HGr/XDg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1ql3Tz-00FsrY-1j; Tue, 26 Sep 2023 08:31:51 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1ql3Tw-00Fsr8-1C for linux-arm-kernel@lists.infradead.org; Tue, 26 Sep 2023 08:31:49 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 2BA4461357; Tue, 26 Sep 2023 08:31:47 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9CF5BC433C8; Tue, 26 Sep 2023 08:31:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695717106; bh=Fc48TGFGlIvNgT5qj7OMuZkiOQBxoSbh/YnlAwsL9Uc=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=gJ5n8dkm6AT3cuVGUxgXBM0mMuUixfCKzrCGnbeNxoqGWgmxJR5SdFe7U6reKp1yx pa7R/A9x381CenK03NcXBpJbqW7/Wqo+tJ8JHTimsjhSTtn3jPUWmrLL/MtDhVKIji R0xvu18gp3g9eocn2oW2lCWbzPs519LEpf0Zm/BJ1jZTXQ4rpoLmrrTAZMgifY58ht bgAfVL+vUPzbW6u4EjBgGL0fZwMlOgA2/jOFYwOcMEwsPmQQbQzIH0a5MtW//y/lvk snppNh8bAbHQpZGy3l5zwWIxs8FFPrYYCdMyeqVYdyUlr3OsOYM5p5oI2ZqSKRzGkl GnaCa76+elYsg== Date: Tue, 26 Sep 2023 10:31:38 +0200 From: Lorenzo Pieralisi To: Jason Gunthorpe Cc: ankita@nvidia.com, maz@kernel.org, oliver.upton@linux.dev, catalin.marinas@arm.com, will@kernel.org, aniketa@nvidia.com, cjia@nvidia.com, kwankhede@nvidia.com, targupta@nvidia.com, vsethi@nvidia.com, acurrid@nvidia.com, apopple@nvidia.com, jhubbard@nvidia.com, danw@nvidia.com, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org Subject: Re: [PATCH v1 2/2] KVM: arm64: allow the VM to select DEVICE_* and NORMAL_NC for IO memory Message-ID: References: <20230907181459.18145-1-ankita@nvidia.com> <20230907181459.18145-3-ankita@nvidia.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230926_013148_519278_30F74176 X-CRM114-Status: GOOD ( 18.57 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Sep 13, 2023 at 03:54:54PM -0300, Jason Gunthorpe wrote: > On Wed, Sep 13, 2023 at 05:26:01PM +0200, Lorenzo Pieralisi wrote: [...] > > I can write up the commit log and post it if I manage to summarize > > it any better - more important the review on the code (that was already > > provided), I will try to write something up asap. > = > Thank you! > = > Jason FWIW, I have come up with the commit log below - please review and scrutinize/change it as deemed fit - it is not necessarily clearer than this one and it definitely requires MarcZ/Catalin/Will attention before it can be considered: --- Currently, KVM for ARM64 maps at stage 2 memory that is considered device (ie using pfn_is_map_memory() to discern between device memory and memory itself) with DEVICE_nGnRE memory attributes; this setting overrides (as per the ARM architecture [1]) any device MMIO mapping present at stage 1, resulting in a set-up whereby a guest operating system can't determine device MMIO mapping memory attributes on its own but it is always overriden by the KVM stage 2 default. This set-up does not allow guest operating systems to map device memory on a page by page basis with combined attributes other than DEVICE_nGnRE, which turns out to be an issue in that guest operating systems (eg Linux) may request to map devices MMIO regions with memory attributes that guarantee better performance (eg gathering attribute - that for some devices can generate larger PCIe memory writes TLPs) and specific operations (eg unaligned transactions) such as the NormalNC memory type. The default device stage 2 mapping was chosen in KVM for ARM64 since it was considered safer (ie it would not allow guests to trigger uncontained failures ultimately crashing the machine) but this turned out to be imprecise. Failures containability is a property of the platform and is independent from the memory type used for MMIO device memory mappings (ie DEVICE_nGnRE memory type is even more problematic than NormalNC in terms of containability since eg aborts triggered on loads cannot be made synchronous, which make them harder to contain); this means that, regardless of the combined stage1+stage2 mappings a platform is safe if and only if device transactions cannot trigger uncontained failures; reworded, the default KVM device stage 2 memory attributes play no role in making device assignment safer for a given platform and therefore can be relaxed. For all these reasons, relax the KVM stage 2 device memory attributes from DEVICE_nGnRE to NormalNC. This puts guests in control (thanks to stage1+stage2 combined memory attributes rules [1]) of device MMIO regions memory mappings, according to the rules described in [1] and summarized here ([(S1) =3D Stage1][(S2) =3D Stage2]): =A0S1=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 |=A0=A0 S2=A0=A0=A0=A0=A0=A0=A0=A0=A0 |= =A0 Result =A0NORMAL-WB=A0=A0=A0=A0|=A0 NORMAL-NC=A0=A0=A0=A0|=A0 NORMAL-NC =A0NORMAL-WT=A0=A0=A0=A0|=A0 NORMAL-NC=A0=A0=A0=A0|=A0 NORMAL-NC =A0NORMAL-NC=A0=A0=A0=A0|=A0 NORMAL-NC=A0=A0=A0=A0|=A0 NORMAL-NC =A0DEVICE=A0|=A0 NORMAL-NC=A0=A0=A0=A0|=A0 DEVICE [1] section D8.5 - DDI0487_I_a_a-profile_architecture_reference_manual.pdf _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel